@@ -35,7 +35,7 @@ We're hiring developers, support people, and production engineers all the time,
There are two editions of GitLab:
- GitLab Community Edition (CE) is available freely under the MIT Expat license.
- GitLab Enterprise Edition (EE) includes [extra features](https://about.gitlab.com/products/#compare-options) that are more useful for organizations with more than 100 users. To use EE and get official support please [become a subscriber](https://about.gitlab.com/products/).
- GitLab Enterprise Edition (EE) includes [extra features](https://about.gitlab.com/pricing/#compare-options) that are more useful for organizations with more than 100 users. To use EE and get official support please [become a subscriber](https://about.gitlab.com/pricing/).
add_index"projects",["last_repository_check_at"],name: "index_projects_on_last_repository_check_at",where: "(last_repository_check_at IS NOT NULL)",using: :btree
With GitLab self-hosted, you deploy your own GitLab instance on-premises or on a private cloud of your choice. GitLab self-hosted is available for [free and with paid subscriptions](https://about.gitlab.com/products/): Core, Starter, Premium, and Ultimate.
With GitLab self-hosted, you deploy your own GitLab instance on-premises or on a private cloud of your choice. GitLab self-hosted is available for [free and with paid subscriptions](https://about.gitlab.com/pricing/): Core, Starter, Premium, and Ultimate.
Every feature available in Core is also available in Starter, Premium, and Ultimate.
Starter features are also available in Premium and Ultimate, and Premium features are also
| 1: patching | Live trace | When a job is running | GitLab Runner => Unicorn => file storage |`#{ROOT_PATH}/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log`|
| 2: overwriting | Live trace | When a job is finished | GitLab Runner => Unicorn => file storage |`#{ROOT_PATH}/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log`|
| 3: archiving | Archived trace | After a job is finished | Sidekiq moves live trace to artifacts folder |`#{ROOT_PATH}/shared/artifacts/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/trace.log`|
| 4: uploading | Archived trace | After a trace is archived | Sidekiq moves archived trace to [object storage](#uploading-traces-to-object-storage)(if configured) |`#{bucket_name}/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/trace.log`|
The `ROOT_PATH` varies per your environment. For Omnibus GitLab it
would be `/var/opt/gitlab/gitlab-ci`, whereas for installations from source
it would be `/home/git/gitlab`.
## Changing the job traces local location
To change the location where the job logs will be stored, follow the steps below.
...
...
@@ -41,97 +55,110 @@ To change the location where the job logs will be stored, follow the steps below
[reconfigure gitlab]:restart_gitlab.md#omnibus-gitlab-reconfigure"How to reconfigure Omnibus GitLab"
[restart gitlab]:restart_gitlab.md#installations-from-source"How to restart GitLab"
## Uploading traces to object storage
An archived trace is considered as a [job artifact](job_artifacts.md).
Therefore, when you [set up an object storage](job_artifacts.md#object-storage-settings),
job traces are automatically migrated to it along with the other job artifacts.
See [Data flow](#data-flow) to learn about the process.
## New live trace architecture
> [Introduced][ce-18169] in GitLab 10.4.
> [Announced as General availability][ce-46097] in GitLab 11.0.
> **Notes**:
- This feature is still Beta, which could impact GitLab.com/on-premises instances, and in the worst case scenario, traces will be lost.
- This feature is still being discussed in [an issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/46097) for the performance improvements.
- This feature is off by default. Please check below how to enable/disable this featrue.
NOTE: **Note:**
This feature is off by default. Check below how to [enable/disable](#enabling-live-trace) it.
**What is "live trace"?**
By combining the process with object storage settings, we can completely bypass
the local file storage. This is a useful option if GitLab is installed as
cloud-native, for example on Kubernetes.
Job trace that is sent by runner while jobs are running. You can see live trace in job pages UI.
The live traces are archived once job finishes.
The data flow is the same as described in the [data flow section](#data-flow)
with one change: _the stored path of the first two phases is different_. This new live
trace architecture stores chunks of traces in Redis and the database instead of
file storage. Redis is used as first-class storage, and it stores up-to 128KB
of data. Once the full chunk is sent, it is flushed to database. After a while,
the data in Redis and database will be archived to [object storage](#uploading-traces-to-object-storage).
**What is new architecture?**
The data are stored in the following Redis namespace: `Gitlab::Redis::SharedState`.
So far, when GitLab Runner sends a job trace to GitLab-Rails, traces have been saved to file storage as text files.
This was a problem for [Cloud Native-compatible GitLab application](https://gitlab.com/gitlab-com/migration/issues/23) where GitLab had to rely on File Storage.
Here is the detailed data flow:
This new live trace architecture stores chunks of traces in Redis and database instead of file storage.
Redis is used as first-class storage, and it stores up-to 128kB. Once the full chunk is sent it will be flushed to database. Afterwhile, the data in Redis and database will be archived to ObjectStorage.
1. GitLab Runner picks a job from GitLab
1. GitLab Runner sends a piece of trace to GitLab
1. GitLab appends the data to Redis
1. Once the data in Redis reach 128KB, the data is flushed to the database.
1. The above steps are repeated until the job is finished.
1. Once the job is finished, GitLab schedules a Sidekiq worker to archive the trace.
1. The Sidekiq worker archives the trace to object storage and cleans up the trace
in Redis and the database.
Here is the detailed data flow.
### Enabling live trace
1. GitLab Runner picks a job from GitLab-Rails
1. GitLab Runner sends a piece of trace to GitLab-Rails
1. GitLab-Rails appends the data to Redis
1. If the data in Redis is fulfilled 128kB, the data is flushed to Database.
1. 2.~4. is continued until the job is finished
1. Once the job is finished, GitLab-Rails schedules a sidekiq worker to archive the trace
1. The sidekiq worker archives the trace to Object Storage, and cleanup the trace in Redis and Database
The following commands are to be issues in a Rails console:
The transition period will be handled gracefully. Upcoming traces will be generated with the new architecture, and on-going live traces will stay with the legacy architecture (i.e. on-going live traces won't be re-generated forcibly with the new architecture).
NOTE: **Note:**
The transition period will be handled gracefully. Upcoming traces will be
generated with the new architecture, and on-going live traces will stay with the
legacy architecture, which means that on-going live traces won't be forcibly
re-generated with the new architecture.
**How to disable?**
**To disable live trace:**
```ruby
Feature.disable('ci_enable_live_trace')
```
>**Note:**
The transition period will be handled gracefully. Upcoming traces will be generated with the legacy architecture, and on-going live traces will stay with the new architecture (i.e. on-going live traces won't be re-generated forcibly with the legacy architecture).
**Redis namespace:**
`Gitlab::Redis::SharedState`
**Potential impact:**
NOTE: **Note:**
The transition period will be handled gracefully. Upcoming traces will be generated
with the legacy architecture, and on-going live traces will stay with the new
architecture, which means that on-going live traces won't be forcibly re-generated
with the legacy architecture.
- This feature could incur data loss:
- Case 1: When all data in Redis are accidentally flushed.
- On-going live traces could be recovered by re-sending traces (This is supported by all versions of GitLab Runner)
- Finished jobs which has not archived live traces will lose the last part (~128kB) of trace data.
- Case 2: When sidekiq workers failed to archive (e.g. There was a bug that prevents archiving process, Sidekiq inconsistancy, etc):
- Currently all trace data in Redis will be deleted after one week. If the sidekiq workers can't finish by the expiry date, the part of trace data will be lost.
- This feature could consume all memory on Redis instance. If the number of jobs is 1000, 128MB (128kB * 1000) is consumed.
- This feature could pressure Database replication lag. `INSERT` are generated to indicate that we have trace chunk. `UPDATE` with 128kB of data is issued once we receive multiple chunks.
- and so on
### Potential implications
**How to test?**
In some cases, having data stored on Redis could incur data loss:
We're currently evaluating this feature on dev.gitalb.org or staging.gitlab.com to verify this features. Here is the list of tests/measurements.
1.**Case 1: When all data in Redis are accidentally flushed**
- On going live traces could be recovered by re-sending traces (this is
supported by all versions of the GitLab Runner).
- Finished jobs which have not archived live traces will lose the last part
(~128KB) of trace data.
- Features:
- Live traces should be visible on job pages
- Archived traces should be visible on job pages
- Live traces should be archived to Object storage
- Live traces should be cleaned up after archived
- etc
- Performance:
- Schedule 1000~10000 jobs and let GitLab-runners process concurrently. Measure memoery presssure, IO load, etc.
- etc
- Failover:
- Simulate Redis outage
- etc
1.**Case 2: When Sidekiq workers fail to archive (e.g., there was a bug that
There are two software distributions of GitLab: the open source [Community Edition](https://gitlab.com/gitlab-org/gitlab-ce/)(CE), and the open core [Enterprise Edition](https://gitlab.com/gitlab-org/gitlab-ee/)(EE). GitLab is available under [different subscriptions](https://about.gitlab.com/products/).
There are two software distributions of GitLab: the open source [Community Edition](https://gitlab.com/gitlab-org/gitlab-ce/)(CE), and the open core [Enterprise Edition](https://gitlab.com/gitlab-org/gitlab-ee/)(EE). GitLab is available under [different subscriptions](https://about.gitlab.com/pricing/).
New versions of GitLab are released in stable branches and the master branch is for bleeding edge development.
@@ -71,7 +71,7 @@ For most installations, only two parameters are required:
Other common configuration options:
-`baseIP`: the desired [external IP address](#external-ip-recommended)
-`gitlab`: Choose the [desired edition](https://about.gitlab.com/products), either `ee` or `ce`. `ce` is the default.
-`gitlab`: Choose the [desired edition](https://about.gitlab.com/pricing), either `ee` or `ce`. `ce` is the default.
-`gitlabEELicense`: For Enterprise Edition, the [license](https://docs.gitlab.com/ee/user/admin_area/license.html) can be installed directly via the Chart
-`provider`: Optimizes the deployment for a cloud provider. The default is `gke` for [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/), with `acs` also supported for the [Azure Container Service](https://azure.microsoft.com/en-us/services/container-service/).
@@ -42,7 +42,7 @@ Set up your project's merge request settings:
### Service Desk
Enable [Service Desk](https://docs.gitlab.com/ee/user/project/service_desk.html) for your project to offer customer support. Service Desk is available in [GitLab Premium](https://about.gitlab.com/products/).
Enable [Service Desk](https://docs.gitlab.com/ee/user/project/service_desk.html) for your project to offer customer support. Service Desk is available in [GitLab Premium](https://about.gitlab.com/pricing/).
