Commit 44712324 authored by Marcel Amirault's avatar Marcel Amirault

Merge branch 'docs-bprescott/20210316-joblogs' into 'master'

Remove beta flag from incremental logging

See merge request gitlab-org/gitlab!56789
parents 78a478be 6d9ce97c
...@@ -130,12 +130,26 @@ find /var/opt/gitlab/gitlab-rails/shared/artifacts -name "job.log" -mtime +60 -d ...@@ -130,12 +130,26 @@ find /var/opt/gitlab/gitlab-rails/shared/artifacts -name "job.log" -mtime +60 -d
## Incremental logging architecture ## Incremental logging architecture
NOTE: > - [Deployed behind a feature flag](../user/feature_flags.md), disabled by default.
This beta feature is off by default. See below for how to [enable or disable](#enabling-incremental-logging) it. > - Enabled on GitLab.com.
> - [Recommended for production use](https://gitlab.com/groups/gitlab-org/-/epics/4275) in GitLab 13.6.
> - [Recommended for production use with AWS S3](https://gitlab.com/gitlab-org/gitlab/-/issues/273498) in GitLab 13.7.
> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-incremental-logging). **(FREE SELF)**
By combining the process with object storage settings, we can completely bypass Job logs are sent from the GitLab Runner in chunks and cached temporarily on disk
the local file storage. This is a useful option if GitLab is installed as in `/var/opt/gitlab/gitlab-ci/builds` by Omnibus GitLab. After the job completes,
cloud-native, for example on Kubernetes. a background job archives the job log. The log is moved to `/var/opt/gitlab/gitlab-rails/shared/artifacts/`
by default, or to object storage if configured.
In a [scaled-out architecture](reference_architectures/index.md) with Rails and Sidekiq running on more than one
server, these two locations on the filesystem have to be shared using NFS.
To eliminate both filesystem requirements:
- Enable the incremental logging feature, which uses Redis instead of disk space for temporary caching of job logs.
- Configure [object storage](job_artifacts.md#object-storage-settings) for storing archived job logs.
### Technical details
The data flow is the same as described in the [data flow section](#data-flow) The data flow is the same as described in the [data flow section](#data-flow)
with one change: _the stored path of the first two phases is different_. This incremental with one change: _the stored path of the first two phases is different_. This incremental
...@@ -157,67 +171,39 @@ Here is the detailed data flow: ...@@ -157,67 +171,39 @@ Here is the detailed data flow:
1. The Sidekiq worker archives the log to object storage and cleans up the log 1. The Sidekiq worker archives the log to object storage and cleans up the log
in Redis and a persistent store (object storage or the database). in Redis and a persistent store (object storage or the database).
### Enabling incremental logging ### Limitations
The following commands are to be issued in a Rails console: - [Redis cluster is not supported](https://gitlab.com/gitlab-org/gitlab/-/issues/224171).
- You must configure [object storage for CI/CD artifacts, logs, and builds](job_artifacts.md#object-storage-settings)
before you enable the feature flag. After the flag is enabled, files cannot be written
to disk, and there is no protection against misconfiguration.
- There is [an epic tracking other potential limitations and improvements](https://gitlab.com/groups/gitlab-org/-/epics/3791).
```shell ### Enable or disable incremental logging **(FREE SELF)**
# Omnibus GitLab
gitlab-rails console
# Installation from source Incremental logging is under development, but ready for production use. It is
cd /home/git/gitlab deployed behind a feature flag that is **disabled by default**.
sudo -u git -H bin/rails console -e production [GitLab administrators with access to the GitLab Rails console](feature_flags.md)
``` can enable it.
**To check if incremental logging (trace) is enabled:** Before you enable the feature flag:
```ruby - Review [the limitations of incremental logging](#limitations).
Feature.enabled?(:ci_enable_live_trace) - [Enable object storage](job_artifacts.md#object-storage-settings).
```
**To enable incremental logging (trace):** To enable incremental logging:
```ruby ```ruby
Feature.enable(:ci_enable_live_trace) Feature.enable(:ci_enable_live_trace)
``` ```
NOTE: Running jobs' logs continue to be written to disk, but new jobs use
The transition period is handled gracefully. Upcoming logs are incremental logging.
generated with the incremental architecture, and on-going logs stay with the
legacy architecture, which means that on-going logs aren't forcibly
re-generated with the incremental architecture.
**To disable incremental logging (trace):** To disable incremental logging:
```ruby ```ruby
Feature.disable('ci_enable_live_trace') Feature.disable(:ci_enable_live_trace)
``` ```
NOTE: Running jobs continue to use incremental logging, but new jobs write to the disk.
The transition period is handled gracefully. Upcoming logs are generated
with the legacy architecture, and on-going incremental logs stay with the incremental
architecture, which means that on-going incremental logs aren't forcibly re-generated
with the legacy architecture.
### Potential implications
In some cases, having data stored on Redis could incur data loss:
1. **Case 1: When all data in Redis are accidentally flushed**
- On going incremental logs could be recovered by re-sending logs (this is
supported by all versions of GitLab Runner).
- Finished jobs which have not archived incremental logs lose the last part
(~128KB) of log data.
1. **Case 2: When Sidekiq workers fail to archive (e.g., there was a bug that
prevents archiving process, Sidekiq inconsistency, etc.)**
- All log data in Redis is deleted after one week. If the
Sidekiq workers can't finish by the expiry date, the part of log data is lost.
Another issue that might arise is that it could consume all memory on the Redis
instance. If the number of jobs is 1000, 128MB (128KB * 1000) is consumed.
Also, it could pressure the database replication lag. `INSERT`s are generated to
indicate that we have log chunk. `UPDATE`s with 128KB of data is issued once we
receive multiple chunks.
...@@ -843,4 +843,4 @@ If you see this page when trying to set a password via the web interface, make s ...@@ -843,4 +843,4 @@ If you see this page when trying to set a password via the web interface, make s
When the GitLab deployment is scaled up to more than one node, some job logs may not be uploaded to [object storage](../../administration/object_storage.md) properly. [Incremental logging is required](../../administration/object_storage.md#incremental-logging-is-required-for-ci-to-use-object-storage) for CI to use object storage. When the GitLab deployment is scaled up to more than one node, some job logs may not be uploaded to [object storage](../../administration/object_storage.md) properly. [Incremental logging is required](../../administration/object_storage.md#incremental-logging-is-required-for-ci-to-use-object-storage) for CI to use object storage.
Enable [incremental logging](../../administration/job_logs.md#enabling-incremental-logging) if it has not already been enabled. Enable [incremental logging](../../administration/job_logs.md#enable-or-disable-incremental-logging) if it has not already been enabled.
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment