Merge branch '31833-add-database-guides-on-not-null-constraints' into 'master'

Documentation: Add database guides for NOT NULL constraints

Closes #31833

See merge request gitlab-org/gitlab!32375

doc/development/database/index.md

@@ -27,6 +27,7 @@
 - [Adding database indexes](../adding_database_indexes.md)
 - [Foreign keys & associations](../foreign_keys.md)
 - [Adding a foreign key constraint to an existing column](add_foreign_key_to_existing_column.md)
+- [`NOT NULL` constraints](not_null_constraints.md)
 - [Strings and the Text data type](strings_and_the_text_data_type.md)
 - [Single table inheritance](../single_table_inheritance.md)
 - [Polymorphic associations](../polymorphic_associations.md)

doc/development/database/not_null_constraints.md

+# `NOT NULL` constraints
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/38358) in GitLab 13.0.
+
+All attributes that should not have `NULL` as a value, should be defined as `NOT NULL`
+columns in the database.
+
+Depending on the application logic, `NOT NULL` columns should either have a `presence: true`
+validation defined in their Model or have a default value as part of their database definition.
+As an example, the latter can be true for boolean attributes that should always have a non-`NULL`
+value, but have a well defined default value that the application does not need to enforce each
+time (for example, `active=true`).
+
+## Create a new table with `NOT NULL` columns
+
+When adding a new table, all `NOT NULL` columns should be defined as such directly inside `create_table`.
+
+For example, consider a migration that creates a table with two `NOT NULL` columns,
+`db/migrate/20200401000001_create_db_guides.rb`:
+
+```ruby
+class CreateDbGuides < ActiveRecord::Migration[6.0]
+  DOWNTIME = false
+
+  def change
+    create_table :db_guides do |t|
+      t.bigint :stars, default: 0, null: false
+      t.bigint :guide, null: false
+    end
+  end
+end
+```
+
+## Add a `NOT NULL` column to an existing table
+
+With PostgreSQL 11 being the minimum version since GitLab 13.0, adding columns with `NULL` and/or
+default values has become much easier and the standard `add_column` helper should be used in all cases.
+
+For example, consider a migration that adds a new `NOT NULL` column `active` to table `db_guides`,
+`db/migrate/20200501000001_add_active_to_db_guides.rb`:
+
+```ruby
+class AddExtendedTitleToSprints < ActiveRecord::Migration[6.0]
+  DOWNTIME = false
+
+  def change
+    add_column :db_guides, :active, :boolean, default: true, null: false
+  end
+end
+```
+
+## Add a `NOT NULL` constraint to an existing column
+
+Adding `NOT NULL` to existing database columns requires multiple steps split into at least two
+different releases:
+
+1. Release `N.M` (current release)
+
+   - Ensure the constraint is enforced at the application level (i.e. add a model validation).
+   - Add a post-deployment migration to add the `NOT NULL` constraint with `validate: false`.
+   - Add a post-deployment migration to fix the existing records.
+
+     NOTE: **Note:**
+     Depending on the size of the table, a background migration for cleanup could be required in the next release.
+     See the [`NOT NULL` constraints on large tables](not_null_constraints.md#not-null-constraints-on-large-tables) section for more information.
+
+   - Create an issue for the next milestone to validate the `NOT NULL` constraint.
+
+1. Release `N.M+1` (next release)
+
+   - Validate the `NOT NULL` constraint using a post-deployment migration.
+
+### Example
+
+Considering a given release milestone, such as 13.0, a model validation has been added into `epic.rb`
+to require a description:
+
+```ruby
+class Epic < ApplicationRecord
+  validates :description, presence: true
+end
+```
+
+The same constraint should be added at the database level for consistency purposes.
+We only want to enforce the `NOT NULL` constraint without setting a default, as we have decided
+that all epics should have a user-generated description.
+
+After checking our production database, we know that there are `epics` with `NULL` descriptions,
+so we can not add and validate the constraint in one step.
+
+NOTE: **Note:**
+Even if we did not have any epic with a `NULL` description, another instance of GitLab could have
+such records, so we would follow the same process either way.
+
+#### Prevent new invalid records (current release)
+
+We first add the `NOT NULL` constraint with a `NOT VALID` parameter, which enforces consistency
+when new records are inserted or current records are updated.
+
+In the example above, the existing epics with a `NULL` description will not be affected and you'll
+still be able to update records in the `epics` table. However, when you try to update or insert
+an epic without providing a description, the constraint causes a database error.
+
+Adding or removing a `NOT NULL` clause requires that any application changes are deployed _first_.
+Thus, adding a `NOT NULL` constraint to an existing column should happen in a post-deployment migration.
+
+Still in our example, for the 13.0 milestone example (current), we add the `NOT NULL` constraint
+with `validate: false` in a post-deployment migration,
+`db/post_migrate/20200501000001_add_not_null_constraint_to_epics_description.rb`:
+
+```ruby
+class AddNotNullConstraintToEpicsDescription < ActiveRecord::Migration[6.0]
+  include Gitlab::Database::MigrationHelpers
+  DOWNTIME = false
+
+  disable_ddl_transaction!
+
+  def up
+    # This will add the `NOT NULL` constraint WITHOUT validating it
+    add_not_null_constraint :epics, :description, validate: false
+  end
+
+  def down
+    # Down is required as `add_not_null_constraint` is not reversible
+    remove_not_null_constraint :epics, :description
+  end
+end
+```
+
+#### Data migration to fix existing records (current release)
+
+The approach here depends on the data volume and the cleanup strategy. The number of records that
+must be fixed on GitLab.com is a nice indicator that will help us decide whether to use a
+post-deployment migration or a background data migration:
+
+- If the data volume is less than `1000` records, then the data migration can be executed within the post-migration.
+- If the data volume is higher than `1000` records, it's advised to create a background migration.
+
+When unsure about which option to use, please contact the Database team for advice.
+
+Back to our example, the epics table is not considerably large nor frequently accessed,
+so we are going to add a post-deployment migration for the 13.0 milestone (current),
+`db/post_migrate/20200501000002_cleanup_epics_with_null_description.rb`:
+
+```ruby
+class CleanupEpicsWithNullDescription < ActiveRecord::Migration[6.0]
+  include Gitlab::Database::MigrationHelpers
+
+  # With BATCH_SIZE=1000 and epics.count=29500 on GitLab.com
+  # - 30 iterations will be run
+  # - each requires on average ~150ms
+  # Expected total run time: ~5 seconds
+  BATCH_SIZE = 1000
+
+  disable_ddl_transaction!
+
+  class Epic < ActiveRecord::Base
+    include EachBatch
+
+    self.table_name = 'epics'
+  end
+
+  def up
+    Epic.each_batch(of: BATCH_SIZE) do |relation|
+      relation.
+        where('description IS NULL').
+        update_all(description: 'No description')
+    end
+  end
+
+  def down
+    # no-op : can't go back to `NULL` without first dropping the `NOT NULL` constraint
+  end
+end
+```
+
+#### Validate the text limit (next release)
+
+Validating the `NOT NULL` constraint will scan the whole table and make sure that each record is correct.
+
+Still in our example, for the 13.1 milestone (next), we run the `validate_not_null_constraint`
+migration helper in a final post-deployment migration,
+`db/post_migrate/20200601000001_validate_not_null_constraint_on_epics_description.rb`:
+
+```ruby
+class ValidateNotNullConstraintOnEpicsDescription < ActiveRecord::Migration[6.0]
+  include Gitlab::Database::MigrationHelpers
+  DOWNTIME = false
+
+  disable_ddl_transaction!
+
+  def up
+    validate_not_null_constraint :epics, :description
+  end
+
+  def down
+    # no-op
+  end
+end
+```
+
+## `NOT NULL` constraints on large tables
+
+If you have to clean up a text column for a really [large table](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/migration_helpers.rb#L12)
+(for example, the `artifacts` in `ci_builds`), your background migration will go on for a while and
+it will need an additional [background migration cleaning up](../background_migrations.md#cleaning-up)
+in the release after adding the data migration.
+
+In that rare case you will need 3 releases end-to-end:
+
+1. Release `N.M` - Add the `NOT NULL` constraint and the background-migration to fix the existing records.
+1. Release `N.M+1` - Cleanup the background migration.
+1. Release `N.M+2` - Validate the `NOT NULL` constraint.
+
+For these cases, please consult the database team early in the update cycle. The `NOT NULL`
+constraint may not be required or other options could exist that do not affect really large
+or frequently accessed tables.

doc/development/database/strings_and_the_text_data_type.md

@@ -34,7 +34,7 @@ When adding a new table, the limits for all text columns should be added in the
 the table creation.
 
 For example, consider a migration that creates a table with two text columns,
-**db/migrate/20200401000001_create_db_guides.rb**:
+`db/migrate/20200401000001_create_db_guides.rb`:
 
 ```ruby
 class CreateDbGuides < ActiveRecord::Migration[6.0]
@@ -80,7 +80,7 @@ frequently accessed table may take minutes in GitLab.com and requires the use of
 For these reasons, it is advised to add the text limit on a separate migration than the `add_column` one.
 
 For example, consider a migration that adds a new text column `extended_title` to table `sprints`,
-**db/migrate/20200501000001_add_extended_title_to_sprints.rb**:
+`db/migrate/20200501000001_add_extended_title_to_sprints.rb`:
 
 ```ruby
 class AddExtendedTitleToSprints < ActiveRecord::Migration[6.0]
@@ -96,7 +96,7 @@ end
 ```
 
 A second migration should follow the first one with a limit added to `extended_title`,
-**db/migrate/20200501000002_add_text_limit_to_sprints_extended_title.rb**:
+`db/migrate/20200501000002_add_text_limit_to_sprints_extended_title.rb`:
 
 ```ruby
 class AddTextLimitToSprintsExtendedTitle < ActiveRecord::Migration[6.0]
@@ -174,7 +174,7 @@ validates :title_html, length: { maximum: 1024 }
 
 We can also update the database in the same milestone by adding the text limit with `validate: false`
 in a post-deployment migration,
-**db/post_migrate/20200501000001_add_text_limit_migration.rb**:
+`db/post_migrate/20200501000001_add_text_limit_migration.rb`:
 
 ```ruby
 class AddTextLimitMigration < ActiveRecord::Migration[6.0]
@@ -208,7 +208,7 @@ When unsure about which option to use, please contact the Database team for advi
 
 Back to our example, the issues table is considerably large and frequently accessed, so we are going
 to add a background migration for the 13.0 milestone (current),
-**db/post_migrate/20200501000002_schedule_cap_title_length_on_issues.rb**:
+`db/post_migrate/20200501000002_schedule_cap_title_length_on_issues.rb`:
 
 ```ruby
 class ScheduleCapTitleLengthOnIssues < ActiveRecord::Migration[6.0]
@@ -255,7 +255,7 @@ Validating the text limit will scan the whole table and make sure that each reco
 
 Still in our example, for the 13.1 milestone (next), we run the `validate_text_limit` migration
 helper in a final post-deployment migration,
-**db/post_migrate/20200601000001_validate_text_limit_migration.rb**:
+`db/post_migrate/20200601000001_validate_text_limit_migration.rb`:
 
 ```ruby
 class ValidateTextLimitMigration < ActiveRecord::Migration[6.0]

doc/development/migration_style_guide.md

@@ -552,6 +552,12 @@ operations that don't require `disable_ddl_transaction!`.
 
 You can read more about adding [foreign key constraints to an existing column](database/add_foreign_key_to_existing_column.md).
 
+## `NOT NULL` constraints
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/38358) in GitLab 13.0.
+
+See the style guide on [`NOT NULL` constraints](database/not_null_constraints.md) for more information.
+
 ## Adding Columns With Default Values
 
 With PostgreSQL 11 being the minimum version since GitLab 13.0, adding columns with default values has become much easier and

doc/development/what_requires_downtime.md

@@ -135,7 +135,7 @@ With [Canary](https://about.gitlab.com/handbook/engineering/infrastructure/libra
 
 ## Changing Column Constraints
 
-Adding or removing a NOT NULL clause (or another constraint) can typically be
+Adding or removing a `NOT NULL` clause (or another constraint) can typically be
 done without requiring downtime. However, this does require that any application
 changes are deployed _first_. Thus, changing the constraints of a column should
 happen in a post-deployment migration.
@@ -143,35 +143,11 @@ happen in a post-deployment migration.
 NOTE: Avoid using `change_column` as it produces an inefficient query because it re-defines
 the whole column type.
 
-To add a NOT NULL constraint, use the `add_not_null_constraint` migration helper:
+You can check the following guides for each specific use case:
 
-```ruby
-# A post-deployment migration in db/post_migrate
-class AddNotNull < ActiveRecord::Migration[4.2]
-  include Gitlab::Database::MigrationHelpers
-
-  disable_ddl_transaction!
-
-  def up
-    add_not_null_constraint :users, :username
-  end
-
-  def down
-    remove_not_null_constraint :users, :username
-  end
-end
-```
-
-If the column to be updated requires cleaning first (e.g. there are `NULL` values), you should:
-
-1. Add the `NOT NULL` constraint with `validate: false`
-
-   `add_not_null_constraint :users, :username, validate: false`
-
-1. Clean up the data with a data migration
-1. Validate the `NOT NULL` constraint with a followup migration
-
-   `validate_not_null_constraint :users, :username`
+- [Adding foreign-key constraints](migration_style_guide.md#adding-foreign-key-constraints)
+- [Adding `NOT NULL` constraints](database/not_null_constraints.md)
+- [Adding limits to text columns](database/strings_and_the_text_data_type.md)
 
 ## Changing Column Types