Commit 23cdf953 authored by Suzanne Selhorn's avatar Suzanne Selhorn

Merge branch 'bbodenmiller-master-patch-68437' into 'master'

Improve incremental logging architecture details

See merge request gitlab-org/gitlab!64862
parents 42300942 2f7dcc25
...@@ -108,7 +108,7 @@ See "Phase 4: uploading" in [Data flow](#data-flow) to learn about the process. ...@@ -108,7 +108,7 @@ See "Phase 4: uploading" in [Data flow](#data-flow) to learn about the process.
If you want to avoid any local disk usage for job logs, If you want to avoid any local disk usage for job logs,
you can do so using one of the following options: you can do so using one of the following options:
- Enable the [beta incremental logging](#incremental-logging-architecture) feature. - Enable the [incremental logging](#incremental-logging-architecture) feature.
- Set the [job logs location](#changing-the-job-logs-local-location) - Set the [job logs location](#changing-the-job-logs-local-location)
to an NFS drive. to an NFS drive.
...@@ -140,7 +140,7 @@ For more information, see [delete references to missing artifacts](raketasks/che ...@@ -140,7 +140,7 @@ For more information, see [delete references to missing artifacts](raketasks/che
> - [Recommended for production use with AWS S3](https://gitlab.com/gitlab-org/gitlab/-/issues/273498) in GitLab 13.7. > - [Recommended for production use with AWS S3](https://gitlab.com/gitlab-org/gitlab/-/issues/273498) in GitLab 13.7.
> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-incremental-logging). **(FREE SELF)** > - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-incremental-logging). **(FREE SELF)**
Job logs are sent from the GitLab Runner in chunks and cached temporarily on disk By default job logs are sent from the GitLab Runner in chunks and cached temporarily on disk
in `/var/opt/gitlab/gitlab-ci/builds` by Omnibus GitLab. After the job completes, in `/var/opt/gitlab/gitlab-ci/builds` by Omnibus GitLab. After the job completes,
a background job archives the job log. The log is moved to `/var/opt/gitlab/gitlab-rails/shared/artifacts/` a background job archives the job log. The log is moved to `/var/opt/gitlab/gitlab-rails/shared/artifacts/`
by default, or to object storage if configured. by default, or to object storage if configured.
...@@ -150,7 +150,7 @@ server, these two locations on the filesystem have to be shared using NFS. ...@@ -150,7 +150,7 @@ server, these two locations on the filesystem have to be shared using NFS.
To eliminate both filesystem requirements: To eliminate both filesystem requirements:
- Enable the incremental logging feature, which uses Redis instead of disk space for temporary caching of job logs. - [Enable the incremental logging feature](#enable-or-disable-incremental-logging), which uses Redis instead of disk space for temporary caching of job logs.
- Configure [object storage](job_artifacts.md#object-storage-settings) for storing archived job logs. - Configure [object storage](job_artifacts.md#object-storage-settings) for storing archived job logs.
### Technical details ### Technical details
...@@ -185,7 +185,7 @@ Here is the detailed data flow: ...@@ -185,7 +185,7 @@ Here is the detailed data flow:
### Enable or disable incremental logging **(FREE SELF)** ### Enable or disable incremental logging **(FREE SELF)**
Incremental logging is under development, but ready for production use. It is Incremental logging is under development, but [ready for production use as of GitLab 13.6](https://gitlab.com/groups/gitlab-org/-/epics/4275). It is
deployed behind a feature flag that is **disabled by default**. deployed behind a feature flag that is **disabled by default**.
[GitLab administrators with access to the GitLab Rails console](feature_flags.md) [GitLab administrators with access to the GitLab Rails console](feature_flags.md)
can enable it. can enable it.
......
...@@ -591,12 +591,6 @@ with the Fog library that GitLab uses. Symptoms include an error in `production. ...@@ -591,12 +591,6 @@ with the Fog library that GitLab uses. Symptoms include an error in `production.
411 Length Required 411 Length Required
``` ```
### Incremental logging is required for CI to use object storage
If you configure GitLab to use object storage for CI logs and artifacts,
you can avoid [local disk usage for job logs](job_logs.md#data-flow) by enabling
[beta incremental logging](job_logs.md#incremental-logging-architecture).
### Proxy Download ### Proxy Download
Clients can download files in object storage by receiving a pre-signed, time-limited URL, Clients can download files in object storage by receiving a pre-signed, time-limited URL,
......
...@@ -841,6 +841,6 @@ If you see this page when trying to set a password via the web interface, make s ...@@ -841,6 +841,6 @@ If you see this page when trying to set a password via the web interface, make s
### Some job logs are not uploaded to object storage ### Some job logs are not uploaded to object storage
When the GitLab deployment is scaled up to more than one node, some job logs may not be uploaded to [object storage](../../administration/object_storage.md) properly. [Incremental logging is required](../../administration/object_storage.md#incremental-logging-is-required-for-ci-to-use-object-storage) for CI to use object storage. When the GitLab deployment is scaled up to more than one node, some job logs may not be uploaded to [object storage](../../administration/object_storage.md) properly. [Incremental logging is required](../../administration/object_storage.md#other-alternatives-to-file-system-storage) for CI to use object storage.
Enable [incremental logging](../../administration/job_logs.md#enable-or-disable-incremental-logging) if it has not already been enabled. Enable [incremental logging](../../administration/job_logs.md#enable-or-disable-incremental-logging) if it has not already been enabled.
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment