@@ -10,39 +10,182 @@ For information, see the [GitLab Release Process](https://gitlab.com/gitlab-org/
...
@@ -10,39 +10,182 @@ For information, see the [GitLab Release Process](https://gitlab.com/gitlab-org/
Both EE and CE require some add-on components called gitlab-shell and Gitaly. These components are available from the [gitlab-shell](https://gitlab.com/gitlab-org/gitlab-shell/tree/master) and [gitaly](https://gitlab.com/gitlab-org/gitaly/tree/master) repositories respectively. New versions are usually tags but staying on the master branch will give you the latest stable version. New releases are generally around the same time as GitLab CE releases with exception for informal security updates deemed critical.
Both EE and CE require some add-on components called gitlab-shell and Gitaly. These components are available from the [gitlab-shell](https://gitlab.com/gitlab-org/gitlab-shell/tree/master) and [gitaly](https://gitlab.com/gitlab-org/gitaly/tree/master) repositories respectively. New versions are usually tags but staying on the master branch will give you the latest stable version. New releases are generally around the same time as GitLab CE releases with exception for informal security updates deemed critical.
## Physical office analogy
## GitLab Omnibus Component by Component
You can imagine GitLab as a physical office.
This document is designed to be consumed by systems adminstrators and GitLab Support Engineers who want to understand more about the internals of GitLab and how they work together.
**The repositories** are the goods GitLab handles.
When deployed, GitLab should be considered the amalgamation of the below processes. When troubleshooting or debugging, be as specific as possible as to which component you are referencing. That should increase clarity and reduce confusion.
They can be stored in a warehouse.
This can be either a hard disk, or something more complex, such as a NFS filesystem;
**Nginx** acts like the front-desk.
### GitLab Process Descriptions
Users come to Nginx and request actions to be done by workers in the office;
**The database** is a series of metal file cabinets with information on:
As of this writing, a fresh GitLab 11.3.0 install will show the following processes with `gitlab-ctl status`:
- The goods in the warehouse (metadata, issues, merge requests etc);
- The users coming to the front desk (permissions)
**Redis** is a communication board with “cubby holes” that can contain tasks for office workers;
GitLab can be considered to have two layers from a process perspective:
-**Monitoring**: Anything from this layer is not required to deliver GitLab the application, but will allow administrators more insight into their infrastructure and what the service as a whole is doing.
-**Core**: Any process that is vital for the delivery of GitLab as as platform. If any of these processes halt there will be a GitLab outage. For the Core layer, you can further divide into:
-**Processors**: These processes are responsible for actually performing operations and presenting the service.
-**Data**: These services store/expose structured data for the GitLab service.
### alertmanager
- Omnibus configuration options
- Layer: Monitoring
[Alert manager](https://prometheus.io/docs/alerting/alertmanager/) is a tool provided by prometheus that _"handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts."_ You can read more in [issue gitlab-ce#45740](https://gitlab.com/gitlab-org/gitlab-ce/issues/45740) about what we will be alerting on.
Gitaly is a service designed by GitLab to remove our need for NFS for Git storage in distributed deployments of GitLab. (Think GitLab.com or High Availablity Deployments) As of 11.3.0, This service handles all Git level access in GitLab. You can read more about the project [in the project's readme](https://gitlab.com/gitlab-org/gitaly).
### gitlab-monitor
- Omnibus configuration options
- Layer: Monitoring
GitLab Monitor is a process disigned in house that allows us to export metrics about GitLab application internals to prometheus. You can read more [in the project's readme](https://gitlab.com/gitlab-org/gitlab-monitor)
### gitlab-workhorse
- Omnibus configuration options
- Layer: Core Service (Processor)
[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) is a program designed at GitLab to help alieviate pressure from unicorn. You can read more about the [historical reasons for developing](https://about.gitlab.com/2016/04/12/a-brief-history-of-gitlab-workhorse/). It's designed to act as a smart reverse proxy to help speed up GitLab as a whole.
GitLab is comprised of a large number of services that all log. We started bundling our own logrotate as of 7.4 to make sure we were logging responsibly. This is just a packaged version of the common opensource offering.
Nginx as as an ingress port for all HTTP requests and routes them to the approriate sub-systems within GitLab. We are bundling an unmodified version of the popular open source webserver.
[Node Exporter](https://github.com/prometheus/node_exporter) is a Prometheus tool that gives us metrics on the underlying machine. (Think CPU/Disk/Load) It's just a packaged version of the common open source offering from the Prometheus project.
[Postgres-exporter](https://github.com/wrouesnel/postgres_exporter) is the community provided Prometheus exporter that will deliver data about Postgres to prometheus for use in Grafana Dashboards.
[Redis Exporter](https://github.com/oliver006/redis_exporter) is designed to give specific metrics about the Redis process to Prometheus so that we can graph these metrics in Graphana.
### sidekiq
- Omnibus configuration options
- Layer: Core Service (Processor)
Sidekiq is a Ruby background job processor that pulls jobs from the redis queue and processes them. Background jobs allow GitLab to provide a faster request/response cycle by moving work into the background.
[Unicorn](https://bogomips.org/unicorn/) is a Ruby application server that is used to run the core Rails Application that provides the user facing features in GitLab. Often process output you will see this as `bundle` or `config.ru` depending on the GitLab version.
### Additional Processes
### GitLab Pages
TODO
### Mattermost
TODO
## GitLab by Request Type
GitLab provides two "interfaces" for end users to access the service:
It's important to understand the distinction as some processes are used in both and others are exclusive to a specific request type.
### GitLab Web HTTP Request Cycle
When making a request to an HTTP Endpoint (Think `/users/sign_in`) the request will take the following path through the GitLab Service:
- nginx - Acts as our first line reverse proxy
- gitlab-workhorse - This determines if it needs to go to the Rails application or somewhere else to reduce load on unicorn.
- unicorn - Since this is a web request, and it needs to access the application it will go to Unicorn.
- Postgres/Gitaly/Redis - Depending on the type of request, it may hit these services to store or retreive data.
**Sidekiq** is a worker that primarily handles sending out emails.
It takes tasks from the Redis communication board;
**A Unicorn worker** is a worker that handles quick/mundane tasks.
### GitLab Git Request Cycle
They work with the communication board (Redis).
Their job description:
- check permissions by checking the user session stored in a Redis “cubby hole”;
- make tasks for Sidekiq;
- fetch stuff from the warehouse or move things around in there;
**GitLab-shell** is a third kind of worker that takes orders from a fax machine (SSH) instead of the front desk (HTTP).
Below we describe the different pathing that HTTP vs. SSH Git requests will take. There is some overlap with the Web Request Cycle but also some differences.
GitLab-shell communicates with Sidekiq via the “communication board” (Redis), and asks quick questions of the Unicorn workers either directly or via the front desk.
**Gitaly** is a back desk that is specialized on reaching the disks to perform git operations efficiently and keep a copy of the result of costly operations. All git operations go through Gitaly.
### Web Request (80/443)
TODO
**GitLab Enterprise Edition (the application)** is the collection of processes and business practices that the office is run by.
