Merge branch 'docs-job-trace-to-log' into 'master'

Rename job traces to job logs in docs

See merge request gitlab-org/gitlab!17740

doc/administration/gitaly/index.md

@@ -549,7 +549,7 @@ a few things that you need to do:
 1. Configure [database lookup of SSH keys](../operations/fast_ssh_key_lookup.md)
    to eliminate the need for a shared authorized_keys file.
 1. Configure [object storage for job artifacts](../job_artifacts.md#using-object-storage)
-   including [live tracing](../job_traces.md#new-live-trace-architecture).
+   including [incremental logging](../job_logs.md#new-incremental-logging-architecture).
 1. Configure [object storage for LFS objects](../../workflow/lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage).
 1. Configure [object storage for uploads](../uploads.md#using-object-storage-core-only).
 

doc/administration/index.md

@@ -154,7 +154,7 @@ Learn how to install, configure, update, and maintain your GitLab instance.
 - [Enable/disable GitLab CI/CD](../ci/enable_or_disable_ci.md#site-wide-admin-setting): Enable or disable GitLab CI/CD for your instance.
 - [GitLab CI/CD admin settings](../user/admin_area/settings/continuous_integration.md): Enable or disable Auto DevOps site-wide and define the artifacts' max size and expiration time.
 - [Job artifacts](job_artifacts.md): Enable, disable, and configure job artifacts (a set of files and directories which are outputted by a job when it completes successfully).
-- [Job traces](job_traces.md): Information about the job traces (logs).
+- [Job logs](job_logs.md): Information about the job logs.
 - [Register Shared and specific Runners](../ci/runners/README.md#registering-a-shared-runner): Learn how to register and configure Shared and specific Runners to your own instance.
 - [Shared Runners pipelines quota](../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota-starter-only): Limit the usage of pipeline minutes for Shared Runners. **(STARTER ONLY)**
 - [Enable/disable Auto DevOps](../topics/autodevops/index.md#enablingdisabling-auto-devops): Enable or disable Auto DevOps for your instance.

doc/administration/job_artifacts.md

@@ -90,7 +90,7 @@ This configuration relies on valid AWS credentials to be configured already.
 Use an object storage option like AWS S3 to store job artifacts.
 
 DANGER: **Danger:**
-If you're enabling S3 in [GitLab HA](high_availability/README.md), you will need to have an [NFS mount set up for CI traces and artifacts](high_availability/nfs.md#a-single-nfs-mount) or enable [live tracing](job_traces.md#new-live-trace-architecture). If these settings are not set, you will risk job traces disappearing or not being saved.
+If you're enabling S3 in [GitLab HA](high_availability/README.md), you will need to have an [NFS mount set up for CI logs and artifacts](high_availability/nfs.md#a-single-nfs-mount) or enable [incremental logging](job_logs.md#new-incremental-logging-architecture). If these settings are not set, you will risk job logs disappearing or not being saved.
 
 #### Object Storage Settings
 

doc/administration/job_logs.md

+# Job logs
+
+> [Renamed from Job Traces to Job logs](https://gitlab.com/gitlab-org/gitlab/issues/29121) in 12.4.
+
+Job logs (traces) are sent by GitLab Runner while it's processing a job. You can see
+logs in job pages, pipelines, email notifications, etc.
+
+## Data flow
+
+In general, there are two states for job logs: `log` and `archived log`.
+In the following table you can see the phases a log goes through:
+
+| Phase          | State        | Condition               | Data flow                                | Stored path |
+| -------------- | ------------ | ----------------------- | -----------------------------------------| ----------- |
+| 1: patching    | log          | When a job is running   | GitLab Runner => Unicorn => file storage | `#{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log` |
+| 2: overwriting | log          | When a job is finished  | GitLab Runner => Unicorn => file storage | `#{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log` |
+| 3: archiving   | archived log | After a job is finished | Sidekiq moves log to artifacts folder    | `#{ROOT_PATH}/gitlab-rails/shared/artifacts/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log` |
+| 4: uploading   | archived log | After a log is archived | Sidekiq moves archived log to [object storage](#uploading-logs-to-object-storage) (if configured) | `#{bucket_name}/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log` |
+
+The `ROOT_PATH` varies per environment. For Omnibus GitLab it
+would be `/var/opt/gitlab`, and for installations from source
+it would be `/home/git/gitlab`.
+
+## Changing the job logs local location
+
+To change the location where the job logs will be stored, follow the steps below.
+
+**In Omnibus installations:**
+
+1. Edit `/etc/gitlab/gitlab.rb` and add or amend the following line:
+
+   ```ruby
+   gitlab_ci['builds_directory'] = '/mnt/to/gitlab-ci/builds'
+   ```
+
+1. Save the file and [reconfigure GitLab][] for the changes to take effect.
+
+---
+
+**In installations from source:**
+
+1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines:
+
+   ```yaml
+   gitlab_ci:
+     # The location where build logs are stored (default: builds/).
+     # Relative paths are relative to Rails.root.
+     builds_path: path/to/builds/
+   ```
+
+1. Save the file and [restart GitLab][] for the changes to take effect.
+
+[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab"
+[restart gitlab]: restart_gitlab.md#installations-from-source "How to restart GitLab"
+
+## Uploading logs to object storage
+
+Archived logs are considered as [job artifacts](job_artifacts.md).
+Therefore, when you [set up the object storage integration](job_artifacts.md#object-storage-settings),
+job logs are automatically migrated to it along with the other job artifacts.
+
+See "Phase 4: uploading" in [Data flow](#data-flow) to learn about the process.
+
+## How to remove job logs
+
+There isn't a way to automatically expire old job logs, but it's safe to remove
+them if they're taking up too much space. If you remove the logs manually, the
+job output in the UI will be empty.
+
+## New incremental logging architecture
+
+> [Introduced][ce-18169] in GitLab 10.4.
+> [Announced as General availability][ce-46097] in GitLab 11.0.
+
+NOTE: **Note:**
+This feature is off by default. See below for how to [enable or disable](#enabling-incremental-logging) it.
+
+By combining the process with object storage settings, we can completely bypass
+the local file storage. This is a useful option if GitLab is installed as
+cloud-native, for example on Kubernetes.
+
+The data flow is the same as described in the [data flow section](#data-flow)
+with one change: _the stored path of the first two phases is different_. This incremental
+log architecture stores chunks of logs in Redis and a persistent store (object storage or database) instead of
+file storage. Redis is used as first-class storage, and it stores up-to 128KB
+of data. Once the full chunk is sent, it is flushed to a persistent store, either object storage(temporary directory) or database.
+After a while, the data in Redis and a persitent store will be archived to [object storage](#uploading-logs-to-object-storage).
+
+The data are stored in the following Redis namespace: `Gitlab::Redis::SharedState`.
+
+Here is the detailed data flow:
+
+1. GitLab Runner picks a job from GitLab
+1. GitLab Runner sends a piece of log to GitLab
+1. GitLab appends the data to Redis
+1. Once the data in Redis reach 128KB, the data is flushed to a persistent store (object storage or the database).
+1. The above steps are repeated until the job is finished.
+1. Once the job is finished, GitLab schedules a Sidekiq worker to archive the log.
+1. The Sidekiq worker archives the log to object storage and cleans up the log
+   in Redis and a persistent store (object storage or the database).
+
+### Enabling incremental logging
+
+The following commands are to be issued in a Rails console:
+
+```sh
+# Omnibus GitLab
+gitlab-rails console
+
+# Installation from source
+cd /home/git/gitlab
+sudo -u git -H bin/rails console RAILS_ENV=production
+```
+
+**To check if incremental logging (trace) is enabled:**
+
+```ruby
+Feature.enabled?('ci_enable_live_trace')
+```
+
+**To enable incremental logging (trace):**
+
+```ruby
+Feature.enable('ci_enable_live_trace')
+```
+
+NOTE: **Note:**
+The transition period will be handled gracefully. Upcoming logs will be
+generated with the incremental architecture, and on-going logs will stay with the
+legacy architecture, which means that on-going logs won't be forcibly
+re-generated with the incremental architecture.
+
+**To disable incremental logging (trace):**
+
+```ruby
+Feature.disable('ci_enable_live_trace')
+```
+
+NOTE: **Note:**
+The transition period will be handled gracefully. Upcoming logs will be generated
+with the legacy architecture, and on-going incremental logs will stay with the incremental
+architecture, which means that on-going incremental logs won't be forcibly re-generated
+with the legacy architecture.
+
+### Potential implications
+
+In some cases, having data stored on Redis could incur data loss:
+
+1. **Case 1: When all data in Redis are accidentally flushed**
+   - On going incremental logs could be recovered by re-sending logs (this is
+     supported by all versions of the GitLab Runner).
+   - Finished jobs which have not archived incremental logs will lose the last part
+     (~128KB) of log data.
+
+1. **Case 2: When Sidekiq workers fail to archive (e.g., there was a bug that
+   prevents archiving process, Sidekiq inconsistency, etc.)**
+   - Currently all log data in Redis will be deleted after one week. If the
+     Sidekiq workers can't finish by the expiry date, the part of log data will be lost.
+
+Another issue that might arise is that it could consume all memory on the Redis
+instance. If the number of jobs is 1000, 128MB (128KB * 1000) is consumed.
+
+Also, it could pressure the database replication lag. `INSERT`s are generated to
+indicate that we have log chunk. `UPDATE`s with 128KB of data is issued once we
+receive multiple chunks.
+
+[ce-18169]: https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/18169
+[ce-21193]: https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/21193
+[ce-46097]: https://gitlab.com/gitlab-org/gitlab-foss/issues/46097

doc/administration/job_traces.md

-# Job traces (logs)
-
-Job traces are sent by GitLab Runner while it's processing a job. You can see
-traces in job pages, pipelines, email notifications, etc.
-
-## Data flow
-
-In general, there are two states in job traces: "live trace" and "archived trace".
-In the following table you can see the phases a trace goes through.
-
-| Phase          | State          | Condition                 | Data flow                                       |  Stored path |
-| -----          | -----          | ---------                 | ---------                                       |  ----------- |
-| 1: patching    | Live trace     | When a job is running     | GitLab Runner => Unicorn => file storage        |`#{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log`|
-| 2: overwriting | Live trace     | When a job is finished    | GitLab Runner => Unicorn => file storage        |`#{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log`|
-| 3: archiving   | Archived trace | After a job is finished   | Sidekiq moves live trace to artifacts folder    |`#{ROOT_PATH}/gitlab-rails/shared/artifacts/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log`|
-| 4: uploading   | Archived trace | After a trace is archived | Sidekiq moves archived trace to [object storage](#uploading-traces-to-object-storage) (if configured)  |`#{bucket_name}/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log`|
-
-The `ROOT_PATH` varies per your environment. For Omnibus GitLab it
-would be `/var/opt/gitlab`, whereas for installations from source
-it would be `/home/git/gitlab`.
-
-## Changing the job traces local location
-
-To change the location where the job logs will be stored, follow the steps below.
-
-**In Omnibus installations:**
-
-1. Edit `/etc/gitlab/gitlab.rb` and add or amend the following line:
-
-   ```ruby
-   gitlab_ci['builds_directory'] = '/mnt/to/gitlab-ci/builds'
-   ```
-
-1. Save the file and [reconfigure GitLab][] for the changes to take effect.
-
+---
+redirect_to: 'job_logs.md'
 ---
 
-**In installations from source:**
-
-1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines:
-
-   ```yaml
-   gitlab_ci:
-     # The location where build traces are stored (default: builds/).
-     # Relative paths are relative to Rails.root.
-     builds_path: path/to/builds/
-   ```
-
-1. Save the file and [restart GitLab][] for the changes to take effect.
-
-[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab"
-[restart gitlab]: restart_gitlab.md#installations-from-source "How to restart GitLab"
-
-## Uploading traces to object storage
-
-Archived traces are considered as [job artifacts](job_artifacts.md).
-Therefore, when you [set up the object storage integration](job_artifacts.md#object-storage-settings),
-job traces are automatically migrated to it along with the other job artifacts.
-
-See "Phase 4: uploading" in [Data flow](#data-flow) to learn about the process.
-
-## How to remove job traces
-
-There isn't a way to automatically expire old job logs, but it's safe to remove
-them if they're taking up too much space. If you remove the logs manually, the
-job output in the UI will be empty.
-
-## New live trace architecture
-
-> [Introduced][ce-18169] in GitLab 10.4.
-> [Announced as General availability][ce-46097] in GitLab 11.0.
-
-NOTE: **Note:**
-This feature is off by default. Check below how to [enable/disable](#enabling-live-trace) it.
-
-By combining the process with object storage settings, we can completely bypass
-the local file storage. This is a useful option if GitLab is installed as
-cloud-native, for example on Kubernetes.
-
-The data flow is the same as described in the [data flow section](#data-flow)
-with one change: _the stored path of the first two phases is different_. This new live
-trace architecture stores chunks of traces in Redis and a persistent store (object storage or database) instead of
-file storage. Redis is used as first-class storage, and it stores up-to 128KB
-of data. Once the full chunk is sent, it is flushed a persistent store, either object storage(temporary directory) or database.
-After a while, the data in Redis and a persitent store will be archived to [object storage](#uploading-traces-to-object-storage).
-
-The data are stored in the following Redis namespace: `Gitlab::Redis::SharedState`.
-
-Here is the detailed data flow:
-
-1. GitLab Runner picks a job from GitLab
-1. GitLab Runner sends a piece of trace to GitLab
-1. GitLab appends the data to Redis
-1. Once the data in Redis reach 128KB, the data is flushed to a persistent store (object storage or the database).
-1. The above steps are repeated until the job is finished.
-1. Once the job is finished, GitLab schedules a Sidekiq worker to archive the trace.
-1. The Sidekiq worker archives the trace to object storage and cleans up the trace
-   in Redis and a persistent store (object storage or the database).
-
-### Enabling live trace
-
-The following commands are to be issues in a Rails console:
-
-```sh
-# Omnibus GitLab
-gitlab-rails console
-
-# Installation from source
-cd /home/git/gitlab
-sudo -u git -H bin/rails console RAILS_ENV=production
-```
-
-**To check if live trace is enabled:**
-
-```ruby
-Feature.enabled?('ci_enable_live_trace')
-```
-
-**To enable live trace:**
-
-```ruby
-Feature.enable('ci_enable_live_trace')
-```
-
-NOTE: **Note:**
-The transition period will be handled gracefully. Upcoming traces will be
-generated with the new architecture, and on-going live traces will stay with the
-legacy architecture, which means that on-going live traces won't be forcibly
-re-generated with the new architecture.
-
-**To disable live trace:**
-
-```ruby
-Feature.disable('ci_enable_live_trace')
-```
-
-NOTE: **Note:**
-The transition period will be handled gracefully. Upcoming traces will be generated
-with the legacy architecture, and on-going live traces will stay with the new
-architecture, which means that on-going live traces won't be forcibly re-generated
-with the legacy architecture.
-
-### Potential implications
-
-In some cases, having data stored on Redis could incur data loss:
-
-1. **Case 1: When all data in Redis are accidentally flushed**
-   - On going live traces could be recovered by re-sending traces (this is
-     supported by all versions of the GitLab Runner).
-   - Finished jobs which have not archived live traces will lose the last part
-     (~128KB) of trace data.
-
-1. **Case 2: When Sidekiq workers fail to archive (e.g., there was a bug that
-   prevents archiving process, Sidekiq inconsistency, etc.)**
-   - Currently all trace data in Redis will be deleted after one week. If the
-     Sidekiq workers can't finish by the expiry date, the part of trace data will be lost.
-
-Another issue that might arise is that it could consume all memory on the Redis
-instance. If the number of jobs is 1000, 128MB (128KB * 1000) is consumed.
-
-Also, it could pressure the database replication lag. `INSERT`s are generated to
-indicate that we have trace chunk. `UPDATE`s with 128KB of data is issued once we
-receive multiple chunks.
-
-[ce-18169]: https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/18169
-[ce-21193]: https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/21193
-[ce-46097]: https://gitlab.com/gitlab-org/gitlab-foss/issues/46097
+This document was moved to [another location](job_logs.md).

doc/api/jobs.md

@@ -537,9 +537,9 @@ Possible response status codes:
 | 400       | Invalid path provided                |
 | 404       | Build not found or no file/artifacts |
 
-## Get a trace file
+## Get a log file
 
-Get a trace of a specific job of a project
+Get a log (trace) of a specific job of a project:
 
 ```
 GET /projects/:id/jobs/:job_id/trace
@@ -557,9 +557,9 @@ curl --location --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.ex
 Possible response status codes:
 
 | Status    | Description                   |
-|-----------|-----------------------------------|
-| 200       | Serves the trace file             |
-| 404       | Build not found or no trace file  |
+|-----------|-------------------------------|
+| 200       | Serves the log file           |
+| 404       | Job not found or no log file  |
 
 ## Cancel a job
 
@@ -661,7 +661,7 @@ Example of response
 
 ## Erase a job
 
-Erase a single job of a project (remove job artifacts and a job trace)
+Erase a single job of a project (remove job artifacts and a job log)
 
 ```
 POST /projects/:id/jobs/:job_id/erase

doc/ci/pipelines.md

@@ -283,11 +283,11 @@ You can also access pipelines for a merge request by navigating to its **Pipelin
 
 When you access a pipeline, you can see the related jobs for that pipeline.
 
-Clicking on an individual job will show you its job trace, and allow you to:
+Clicking on an individual job will show you its job log, and allow you to:
 
 - Cancel the job.
 - Retry the job.
-- Erase the job trace.
+- Erase the job log.
 
 ### Seeing the failure reason for jobs
 

doc/ci/ssh_keys/README.md

@@ -35,8 +35,8 @@ with any type of [executor](https://docs.gitlab.com/runner/executors/)
    if you are accessing a private GitLab repository.
 
 NOTE: **Note:**
-The private key will not be displayed in the job trace, unless you enable
-[debug tracing](../variables/README.md#debug-tracing). You might also want to
+The private key will not be displayed in the job log, unless you enable
+[debug logging](../variables/README.md#debug-logging). You might also want to
 check the [visibility of your pipelines](../../user/project/pipelines/settings.md#visibility-of-pipelines).
 
 ## SSH keys when using the Docker executor

doc/ci/variables/README.md

@@ -568,7 +568,7 @@ Below you can find supported syntax reference:
    Precedence of operators follows standard Ruby 2.5 operation
    [precedence](https://ruby-doc.org/core-2.5.0/doc/syntax/precedence_rdoc.html).
 
-## Debug tracing
+## Debug logging
 
 > Introduced in GitLab Runner 1.7.
 
@@ -576,24 +576,24 @@ CAUTION: **Warning:**
 Enabling debug tracing can have severe security implications. The
 output **will** contain the content of all your variables and any other
 secrets! The output **will** be uploaded to the GitLab server and made visible
-in job traces!
+in job logs!
 
 By default, GitLab Runner hides most of the details of what it is doing when
-processing a job. This behavior keeps job traces short, and prevents secrets
-from being leaked into the trace unless your script writes them to the screen.
+processing a job. This behavior keeps job logs short, and prevents secrets
+from being leaked into the log unless your script writes them to the screen.
 
 If a job isn't working as expected, this can make the problem difficult to
 investigate; in these cases, you can enable debug tracing in `.gitlab-ci.yml`.
 Available on GitLab Runner v1.7+, this feature enables the shell's execution
-trace, resulting in a verbose job trace listing all commands that were run,
+log, resulting in a verbose job log listing all commands that were run,
 variables that were set, etc.
 
 Before enabling this, you should ensure jobs are visible to
 [team members only](../../user/permissions.md#project-features). You should
-also [erase](../pipelines.md#accessing-individual-jobs) all generated job traces
+also [erase](../pipelines.md#accessing-individual-jobs) all generated job logs
 before making them visible again.
 
-To enable debug traces, set the `CI_DEBUG_TRACE` variable to `true`:
+To enable debug logs (traces), set the `CI_DEBUG_TRACE` variable to `true`:
 
 ```yaml
 job_name:
@@ -601,7 +601,7 @@ job_name:
     CI_DEBUG_TRACE: "true"
 ```
 
-Example truncated output with debug trace set to true:
+Example truncated output with `CI_DEBUG_TRACE` set to `true`:
 
 ```bash
 ...

doc/ci/variables/predefined_variables.md

@@ -40,7 +40,7 @@ future GitLab releases.**
 | `CI_COMMIT_TAG`                         | 9.0    | 0.5    | The commit tag name. Present only when building tags. |
 | `CI_COMMIT_TITLE`                       | 10.8   | all    | The title of the commit - the full first line of the message |
 | `CI_CONFIG_PATH`                        | 9.4    | 0.5    | The path to CI config file. Defaults to `.gitlab-ci.yml` |
-| `CI_DEBUG_TRACE`                        | all    | 1.7    | Whether [debug tracing](README.md#debug-tracing) is enabled |
+| `CI_DEBUG_TRACE`                        | all    | 1.7    | Whether [debug logging (tracing)](README.md#debug-logging) is enabled |
 | `CI_DEPLOY_PASSWORD`                    | 10.8   | all    | Authentication password of the [GitLab Deploy Token][gitlab-deploy-token], only present if the Project has one related.|
 | `CI_DEPLOY_USER`                        | 10.8   | all    | Authentication username of the [GitLab Deploy Token][gitlab-deploy-token], only present if the Project has one related.|
 | `CI_DISPOSABLE_ENVIRONMENT`             | all    | 10.1   | Marks that the job is executed in a disposable environment (something that is created only for this job and disposed of/destroyed after the execution - all executors except `shell` and `ssh`). If the environment is disposable, it is set to true, otherwise it is not defined at all. |

doc/install/installation.md

@@ -448,7 +448,7 @@ sudo -u git -H mkdir -p public/uploads/
 # now that files in public/uploads are served by gitlab-workhorse
 sudo chmod 0700 public/uploads
 
-# Change the permissions of the directory where CI job traces are stored
+# Change the permissions of the directory where CI job logs are stored
 sudo chmod -R u+rwX builds/
 
 # Change the permissions of the directory where CI artifacts are stored