Before beginning, you should already have a working GitLab instance. [Learn how
to install GitLab](https://about.gitlab.com/install/).
Provision a PostgreSQL server (PostgreSQL 11 or newer).
Provision a PostgreSQL server. We recommend using the PostgreSQL that is shipped
with Omnibus GitLab and use it to configure the PostgreSQL database. You can use an
external PostgreSQL server (version 11 or newer) but you must set it up [manually](#manual-database-setup).
Prepare all your new nodes by [installing
GitLab](https://about.gitlab.com/install/).
Prepare all your new nodes by [installing GitLab](https://about.gitlab.com/install/). You need:
- 1 PostgreSQL node
- 1 PgBouncer node (optional)
- At least 1 Praefect node (minimal storage required)
- 3 Gitaly nodes (high CPU, high memory, fast storage)
- 1 GitLab server
You need the IP/host address for each node.
You also need the IP/host address for each node:
1.`LOAD_BALANCER_SERVER_ADDRESS`: the IP/host address of the load balancer
1.`POSTGRESQL_SERVER_ADDRESS`: the IP/host address of the PostgreSQL server
1.`PRAEFECT_LOADBALANCER_HOST`: the IP/host address of Praefect load balancer
1.`POSTGRESQL_HOST`: the IP/host address of the PostgreSQL server
1.`PGBOUNCER_HOST`: the IP/host address of the PostgreSQL server
1.`PRAEFECT_HOST`: the IP/host address of the Praefect server
1.`GITALY_HOST_*`: the IP or host address of each Gitaly server
1.`GITLAB_HOST`: the IP/host address of the GitLab server
If you are using a cloud provider, you can look up the addresses for each server through your cloud provider's management console.
If you are using Google Cloud Platform, SoftLayer, or any other vendor that provides a virtual private cloud (VPC) you can use the private addresses for each cloud instance (corresponds to "internal address" for Google Cloud Platform) for `PRAEFECT_HOST`, `GITALY_HOST_*`, and `GITLAB_HOST`.
#### Secrets
...
...
@@ -98,6 +100,14 @@ with secure tokens as you complete the setup process.
Praefect cluster directly; that could lead to data loss.
1.`PRAEFECT_SQL_PASSWORD`: this password is used by Praefect to connect to
PostgreSQL.
1.`PRAEFECT_SQL_PASSWORD_HASH`: the hash of password of the Praefect user.
Use `gitlab-ctl pg-password-md5 praefect` to generate the hash. The command
asks for the password for `praefect` user. Enter `PRAEFECT_SQL_PASSWORD`
plaintext password. By default, Praefect uses `praefect` user, but you can
change it.
1.`PGBOUNCER_SQL_PASSWORD_HASH`: the hash of password of the PgBouncer user.
PgBouncer uses this password to connect to PostgreSQL. For more details
see [bundled PgBouncer](../postgresql/pgbouncer.md) documentation.
We note in the instructions below where these secrets are required.
...
...
@@ -108,63 +118,81 @@ Omnibus GitLab installations can use `gitlab-secrets.json` for `GITLAB_SHELL_SEC
NOTE:
Do not store the GitLab application database and the Praefect
database on the same PostgreSQL server if using
[Geo](../geo/index.md). The replication state is internal to each instance
of GitLab and should not be replicated.
database on the same PostgreSQL server if using[Geo](../geo/index.md).
The replication state is internal to each instance of GitLab and should
not be replicated.
These instructions help set up a single PostgreSQL database, which creates a single point of
failure. The following options are available:
failure. Alternatively, [you can use PostgreSQL replication and failover](../postgresql/replication_and_failover.md).
The following options are available:
- For non-Geo installations, either:
- Use one of the documented [PostgreSQL setups](../postgresql/index.md).
- Use your own third-party database setup, if fault tolerance is required.
- Use your own third-party database setup. This will require [manual setup](#manual-database-setup).
- For Geo instances, either:
- Set up a separate [PostgreSQL instance](https://www.postgresql.org/docs/11/high-availability.html).
- Use a cloud-managed PostgreSQL service. AWS
[Relational Database Service](https://aws.amazon.com/rds/) is recommended.
#### Manual database setup
To complete this section you need:
-1 Praefect node
-1 PostgreSQL server (PostgreSQL 11 or newer)
- An SQL user with permissions to create databases
-One Praefect node
-One PostgreSQL node (version 11 or newer)
- A PostgreSQL user with permissions to manage the database server
During this section, we configure the PostgreSQL server, from the Praefect
node, using `psql` which is installed by Omnibus GitLab.
In this section, we configure the PostgreSQL database. This can be used for both external
and Omnibus-provided PostgreSQL server.
1. SSH into the **Praefect** node and login as root:
To run the following instructions, you can use the Praefect node, where `psql` is installed
by Omnibus GitLab (`/opt/gitlab/embedded/bin/psql`). If you are using the Omnibus-provided
PostgreSQL you can use `gitlab-psql` on the PostgreSQL node instead:
```shell
sudo-i
```
1. Connect to the PostgreSQL server with administrative access. This is likely
the `postgres` user. The database `template1` is used because it is created
by default on all PostgreSQL servers.
1. Create a new user `praefect` to be used by Praefect:
@@ -415,6 +415,10 @@ which is used to track vulnerabilities
as new commits are pushed to the repository.
The attributes used to generate the location fingerprint also depend on the type of scanning.
### Details
The `details` field is an object that supports many different content elements that are displayed when viewing vulnerability information. An example of the various data elements can be seen in the [security-reports repository](https://gitlab.com/gitlab-examples/security/security-reports/-/tree/master/samples/details-example).
#### Dependency Scanning
The `location` of a Dependency Scanning vulnerability is composed of a `dependency` and a `file`.
info:To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Iterating Tables In Batches
# Iterating tables in batches
Rails provides a method called `in_batches` that can be used to iterate over
rows in batches. For example:
...
...
@@ -15,7 +15,7 @@ User.in_batches(of: 10) do |relation|
end
```
Unfortunately this method is implemented in a way that is not very efficient,
Unfortunately, this method is implemented in a way that is not very efficient,
both query and memory usage wise.
To work around this you can include the `EachBatch` module into your models,
...
...
@@ -44,17 +44,18 @@ all of the arguments that `in_batches` supports. You should always use
## Avoid iterating over non-unique columns
One should proceed with extra caution, and possibly avoid iterating over a column that can contain duplicate values.
When you iterate over an attribute that is not unique, even with the applied max batch size, there is no guarantee that the resulting batches will not surpass it.
The following snippet demonstrates this situation, when one attempt to select `Ci::Build` entries for users with `id` between `1` and `10,s000`, database returns `1 215 178`
matching rows
One should proceed with extra caution, and possibly avoid iterating over a column that can contain
duplicate values. When you iterate over an attribute that is not unique, even with the applied max
batch size, there is no guarantee that the resulting batches will not surpass it. The following
snippet demonstrates this situation when one attempt to select `Ci::Build` entries for users with
`id` between `1` and `10,000`, the database returns `1 215 178` matching rows.
@@ -62,12 +63,16 @@ SELECT "ci_builds".* FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND
=>nil
```
And queries which filters non-unique column by range `WHERE "ci_builds"."user_id" BETWEEN ? AND ?`, even though the range size is limited to certain threshold (`10,000` in previous example) this threshold does not translates to the size of returned dataset. That happens because when taking `n` possible values of attributes,
one can't tell for sure that the number of records that contains them will be less than `n`.
`And` queries which filter non-unique column by range `WHERE "ci_builds"."user_id" BETWEEN ? AND ?`,
even though the range size is limited to a certain threshold (`10,000` in the previous example) this
threshold does not translate to the size of the returned dataset. That happens because when taking
`n` possible values of attributes, one can't tell for sure that the number of records that contains
them will be less than `n`.
## Column definition
`EachBatch` uses the primary key of the model by default for the iteration. This works most of the cases, however in some cases, you might want to use a different column for the iteration.
`EachBatch` uses the primary key of the model by default for the iteration. This works most of the
cases, however in some cases, you might want to use a different column for the iteration.
The query above iterates over the project creators and prints them out without duplications.
NOTE:
In case the column is not unique (no unique index definition), calling the `distinct` method on the relation is necessary. Using not unique column without `distinct` may result in `each_batch` falling into endless loop as described at following [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/285097)
In case the column is not unique (no unique index definition), calling the `distinct` method on
the relation is necessary. Using not unique column without `distinct` may result in `each_batch`
falling into an endless loop as described in following
When dealing with data migrations the preferred way to iterate over large volume of data is using `EachBatch`.
When dealing with data migrations the preferred way to iterate over a large volume of data is using
`EachBatch`.
A special case of data migration is a [background migration](background_migrations.md#scheduling)
where the actual data modification is executed in a background job. The migration code that determines
the data ranges (slices) and schedules the background jobs uses `each_batch`.
where the actual data modification is executed in a background job. The migration code that
determines the data ranges (slices) and schedules the background jobs uses `each_batch`.
## Efficient usage of `each_batch`
`EachBatch` helps iterating over large tables. It's important to highlight that `EachBatch` is not going to magically solve all iteration related performance problems and it might not help at all in some scenarios. From the database point of view, correctly configured database indexes are also necessary to make `EachBatch` perform well.
`EachBatch` helps to iterate over large tables. It's important to highlight that `EachBatch` is
not going to magically solve all iteration related performance problems and it might not help at
all in some scenarios. From the database point of view, correctly configured database indexes are
also necessary to make `EachBatch` perform well.
### Example 1: Simple iteration
Let's consider that we want to iterate over the `users` table and print the `User` records to the standard output. The `users` table contains millions of records, thus running one query to fetch the users will likely time out.
Let's consider that we want to iterate over the `users` table and print the `User` records to the
standard output. The `users` table contains millions of records, thus running one query to fetch
This is a simplified version of the `users` table which contains several rows. We have a few smaller gaps in the `id` column to make the example a bit more realistic (a few records were already deleted). Currently we have one index on the `id` field.
This is a simplified version of the `users` table which contains several rows. We have a few
smaller gaps in the `id` column to make the example a bit more realistic (a few records were
already deleted). Currently, we have one index on the `id` field.
Loading all users into memory (avoid):
...
...
@@ -117,9 +133,10 @@ User.each_batch(of: 5) do |relation|
end
```
#### How does `each_batch` work?
#### How `each_batch` works
As the first step, it finds the lowest `id` (start `id`) in the table by executing the following database query:
As the first step, it finds the lowest `id` (start `id`) in the table by executing the following
@@ -127,9 +144,12 @@ SELECT "users"."id" FROM "users" ORDER BY "users"."id" ASC LIMIT 1
Notice that the query only reads data from the index (`INDEX ONLY SCAN`), the table is not accessed. Database indexes are sorted so taking out the first item is a very cheap operation.
Notice that the query only reads data from the index (`INDEX ONLY SCAN`), the table is not
accessed. Database indexes are sorted so taking out the first item is a very cheap operation.
The next step is to find the next `id` (end `id`) which should respect the batch size configuration. In this example we used batch size of 5. `EachBatch` uses the `OFFSET` clause to get a "shifted" `id` value.
The next step is to find the next `id` (end `id`) which should respect the batch size
configuration. In this example we used a batch size of 5. `EachBatch` uses the `OFFSET` clause
@@ -137,19 +157,25 @@ SELECT "users"."id" FROM "users" WHERE "users"."id" >= 1 ORDER BY "users"."id" A
Again, the query only looks into the index. The `OFFSET 5` takes out the sixth `id` value: this query reads a maximum of six items from the index regardless of the table size or the iteration count.
Again, the query only looks into the index. The `OFFSET 5` takes out the sixth `id` value: this
query reads a maximum of six items from the index regardless of the table size or the iteration
count.
At this point we know the `id` range for the first batch. Now it's time to construct the query for the `relation` block.
At this point, we know the `id` range for the first batch. Now it's time to construct the query
Notice the `<` sign. Previously six items were read from the index and in this query the last value is "excluded". The query will look at the index to get the location of the five `user` rows on the disk and read the rows from the table. The returned array is processed in Ruby.
Notice the `<` sign. Previously six items were read from the index and in this query, the last
value is "excluded". The query will look at the index to get the location of the five `user`
rows on the disk and read the rows from the table. The returned array is processed in Ruby.
The first iteration is done. For the next iteration, the last `id` value is reused from the previous iteration in order to find out the next end `id` value.
The first iteration is done. For the next iteration, the last `id` value is reused from the
previous iteration in order to find out the next end `id` value.
@@ -167,7 +193,8 @@ SELECT "users".* FROM "users" WHERE "users"."id" >= 302 AND "users"."id" < 353
### Example 2: Iteration with filters
Building on top of the previous example, we want to print users with zero sign-in count. We keep track of the number of sign-ins in the `sign_in_count` column so we write the following code:
Building on top of the previous example, we want to print users with zero sign-in count. We keep
track of the number of sign-ins in the `sign_in_count` column so we write the following code:
Selecting only the `id` column and ordering by `id` is going to "force" the database to use the index on the `id` (primary key index) column, however we also have an extra condition on the `sign_in_count` column. The column is not part of the index, so the database needs to look into the actual table to find the first matching row.
Selecting only the `id` column and ordering by `id` is going to "force" the database to use the
index on the `id` (primary key index) column however, we also have an extra condition on the
`sign_in_count` column. The column is not part of the index, so the database needs to look into
the actual table to find the first matching row.
...
...
@@ -193,7 +223,11 @@ The number of scanned rows depends on the data distribution in the table.
- Best case scenario: the first user was never logged in. The database reads only one row.
- Worst case scenario: all users were logged in at least once. The database reads all rows.
In this particular example the database had to read 10 rows (regardless of our batch size setting) to determine the first `id` value. In a "real-world" application it's hard to predict whether the filtering is going to cause problems or not. In case of GitLab, verifying the data on a production replica is a good start, but keep in mind that data distribution on GitLab.com can be different from self-managed instances.
In this particular example, the database had to read 10 rows (regardless of our batch size setting)
to determine the first `id` value. In a "real-world" application it's hard to predict whether the
filtering is going to cause problems or not. In the case of GitLab, verifying the data on a
production replica is a good start, but keep in mind that data distribution on GitLab.com can be
different from self-managed instances.
#### Improve filtering with `each_batch`
...
...
@@ -207,21 +241,26 @@ This is how our table and the newly created index looks like:
This index definition covers the conditions on the `id` and `sign_in_count` columns thus makes the `each_batch` queries very effective (similar to the simple iteration example).
This index definition covers the conditions on the `id` and `sign_in_count` columns thus makes the
`each_batch` queries very effective (similar to the simple iteration example).
It's rare when a user was never signed in so we anticipate small index size. Including only the `id` in the index definition also helps keeping the index size small.
It's rare when a user was never signed in so we a anticipate small index size. Including only the
`id` in the index definition also helps to keep the index size small.
##### Index on columns
Later on we might want to iterate over the table filtering for different `sign_in_count` values, in those cases we cannot use the previously suggested conditional index because the `WHERE` condition does not match with our new filter (`sign_in_count > 10`).
Later on, we might want to iterate over the table filtering for different `sign_in_count` values, in
those cases we cannot use the previously suggested conditional index because the `WHERE` condition
does not match with our new filter (`sign_in_count > 10`).
To address this problem, we have two options:
- Create another, conditional index to cover the new query.
- Replace the index with more generalized configuration.
- Replace the index with a more generalized configuration.
NOTE:
Having multiple indexes on the same table and on the same columns could be a performance bottleneck when writing data.
Having multiple indexes on the same table and on the same columns could be a performance bottleneck
when writing data.
Let's consider the following index (avoid):
...
...
@@ -229,15 +268,18 @@ Let's consider the following index (avoid):
Executing the query above results in an `INDEX ONLY SCAN`. However, the query still needs to iterate over unknown number of entries in the index, and then find the first item where the `sign_in_count` is `0`.
Executing the query above results in an `INDEX ONLY SCAN`. However, the query still needs to
iterate over an unknown number of entries in the index, and then find the first item where the
`sign_in_count` is `0`.
We can improve the query significantly by swapping the columns in the index definition (prefer).
...
...
@@ -253,11 +295,14 @@ The following index definition is not going to work well with `each_batch` (avoi
Since `each_batch` builds range queries based on the `id` column, this index cannot be used efficiently. The DB reads the rows from the table or uses a bitmap search where the primary key index is also read.
Since `each_batch` builds range queries based on the `id` column, this index cannot be used
efficiently. The DB reads the rows from the table or uses a bitmap search where the primary
key index is also read.
##### "Slow" iteration
Slow iteration means that we use a good index configuration to iterate over the table and apply filtering on the yielded relation.
Slow iteration means that we use a good index configuration to iterate over the table and
apply filtering on the yielded relation.
```ruby
User.each_batch(of: 5)do|relation|
...
...
@@ -266,7 +311,8 @@ end
```
The iteration uses the primary key index (on the `id` column) which makes it safe from statement
timeouts. The filter (`sign_in_count: 0`) is applied on the `relation` where the `id` is already constrained (range). The number of rows are limited.
timeouts. The filter (`sign_in_count: 0`) is applied on the `relation` where the `id` is already
constrained (range). The number of rows is limited.
Slow iteration generally takes more time to finish. The iteration count is higher and
one iteration could yield fewer records than the batch size. Iterations may even yield
...
...
@@ -285,18 +331,19 @@ projects.each_batch do |relation|
end
```
The iteration uses the `id` column of the `projects` table. The batching does not affect the subquery.
This means for each iteration, the subquery is executed by the database. This adds a constant "load"
on the query which often ends up in statement timeouts. We have an unknown number of confidential
issues, the execution time and the accessed database rows depends on the data distribution in the
`issues` table.
The iteration uses the `id` column of the `projects` table. The batching does not affect the
subquery. This means for each iteration, the subquery is executed by the database. This adds a
constant "load" on the query which often ends up in statement timeouts. We have an unknown number
of confidential issues, the execution time and the accessed database rows depend on the data
distribution in the `issues` table.
NOTE:
Using subqueries works only when the subquery returns a small number of rows.
#### Improving Subqueries
When dealing with subqueries, a slow iteration approach could work: the filter on `creator_id` can be part of the generated `relation` object.
When dealing with subqueries, a slow iteration approach could work: the filter on `creator_id`
can be part of the generated `relation` object.
```ruby
projects=Project.all
...
...
@@ -306,7 +353,8 @@ projects.each_batch do |relation|
end
```
If the query on the `issues` table itself is not performant enough, a nested loop could be constructed. Try to avoid it when possible.
If the query on the `issues` table itself is not performant enough, a nested loop could be
constructed. Try to avoid it when possible.
```ruby
projects=Project.all
...
...
@@ -320,7 +368,8 @@ projects.each_batch do |relation|
end
```
If we know that the `issues` table has many more rows than `projects`, it would make sense to flip the queries, where the `issues` table is batched first.
If we know that the `issues` table has many more rows than `projects`, it would make sense to flip
the queries, where the `issues` table is batched first.
### Using `JOIN` and `EXISTS`
...
...
@@ -331,7 +380,8 @@ When to use `JOINS`:
-`projects` - `project_settings`
-`users` - `user_details`
-`users` - `user_statuses`
-`LEFT JOIN` works well in this case. Conditions on the joined table need to go to the yielded relation so the iteration is not affected by the data distribution in the joined table.
-`LEFT JOIN` works well in this case. Conditions on the joined table need to go to the yielded
relation so the iteration is not affected by the data distribution in the joined table.
Example:
...
...
@@ -353,7 +403,8 @@ end
### Complex queries on the relation object
When the `relation` object has several extra conditions, the execution plans might become "unstable".
When the `relation` object has several extra conditions, the execution plans might become
"unstable".
Example:
...
...
@@ -370,10 +421,11 @@ end
Here, we expect that the `relation` query reads the `BATCH_SIZE` of user records and then
filters down the results according to the provided queries. The planner might decide that
using a bitmap index lookup with the index on the `confidential` column is a better way to
execute the query. This can cause unexpectedly high amount of rows to be read and the query
could time out.
execute the query. This can cause an unexpectedly high amount of rows to be read and the
query could time out.
Problem: we know for sure that the relation is returning maximum `BATCH_SIZE` of records, however the planner does not know this.
Problem: we know for sure that the relation is returning maximum `BATCH_SIZE` of records
however, the planner does not know this.
Common table expression (CTE) trick to force the range query to execute first:
...
...
@@ -394,4 +446,132 @@ end
### `EachBatch` vs `BatchCount`
When adding new counters for Service Ping, the preferred way to count records is using the `Gitlab::Database::BatchCount` class. The iteration logic implemented in `BatchCount` has similar performance characteristics like `EachBatch`. Most of the tips and suggestions for improving `BatchCount` mentioned above applies to `BatchCount` as well.
When adding new counters for Service Ping, the preferred way to count records is using the
`Gitlab::Database::BatchCount` class. The iteration logic implemented in `BatchCount`
has similar performance characteristics like `EachBatch`. Most of the tips and suggestions
for improving `BatchCount` mentioned above applies to `BatchCount` as well.
## Iterate with keyset pagination
There are a few special cases where iterating with `EachBatch` does not work. `EachBatch`
requires one distinct column (usually the primary key), which makes the iteration impossible
for timestamp columns and tables with composite primary keys.
Where `EachBatch` does not work, you can use
[keyset pagination](database/pagination_guidelines.md#keyset-pagination) to iterate over the
table or a range of rows. The scaling and performance characteristics are very similar to
`EachBatch`.
Examples:
- Iterate over the table in a specific order (timestamp columns) in combination with a tie-breaker
if column user to sort by does not contain unique values.
- Iterate over the table with composite primary keys.
### Iterate over the issues in a project by creation date
You can use keyset pagination to iterate over any database column in a specific order (for example,
`created_at DESC`). To ensure consistent order of the returned records with the same values for
`created_at`, use a tie-breaker column with unique values (for example, `id`).
Assume you have the following index in the `issues` table:
and [`Gitlab::Pagination::Keyset::ColumnOrderDefinition`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/pagination/keyset/column_order_definition.rb)
classes give an overview of the possible options for configuring the `ORDER BY` clause. You can
