- Redis - Key/Value store (User sessions, cache, queue for Sidekiq)
- Redis - Key/Value store (User sessions, cache, queue for Sidekiq)
- Sentinel - Redis health check/failover manager
- Sentinel - Redis health check/failover manager
- Gitaly - Provides high-level storage and RPC access to Git repositories
- Gitaly - Provides high-level storage and RPC access to Git repositories
- S3 Object Storage service[^3] and / or NFS storage servers[^4] for entities such as Uploads, Artifacts, LFS Objects, etc...
- S3 Object Storage service[^4] and / or NFS storage servers[^5] for entities such as Uploads, Artifacts, LFS Objects, etc...
- Load Balancer[^2] - Main entry point and handles load balancing for the GitLab application nodes.
- Load Balancer[^6] - Main entry point and handles load balancing for the GitLab application nodes.
- Monitor - Prometheus and Grafana monitoring with auto discovery.
- Monitor - Prometheus and Grafana monitoring with auto discovery.
## Scalable Architecture Examples
## Scalable Architecture Examples
...
@@ -72,9 +72,9 @@ larger one.
...
@@ -72,9 +72,9 @@ larger one.
- 1 PostgreSQL node
- 1 PostgreSQL node
- 1 Redis node
- 1 Redis node
- 1 Gitaly node
- 1 Gitaly node
- 1 or more Object Storage services[^3] and / or NFS storage server[^4]
- 1 or more Object Storage services[^4] and / or NFS storage server[^5]
- 2 or more GitLab application nodes (Unicorn / Puma, Workhorse, Sidekiq)
- 2 or more GitLab application nodes (Unicorn / Puma, Workhorse, Sidekiq)
- 1 or more Load Balancer nodes[^2]
- 1 or more Load Balancer nodes[^6]
- 1 Monitoring node (Prometheus, Grafana)
- 1 Monitoring node (Prometheus, Grafana)
#### Installation Instructions
#### Installation Instructions
...
@@ -83,13 +83,13 @@ Complete the following installation steps in order. A link at the end of each
...
@@ -83,13 +83,13 @@ Complete the following installation steps in order. A link at the end of each
section will bring you back to the Scalable Architecture Examples section so
section will bring you back to the Scalable Architecture Examples section so
you can continue with the next step.
you can continue with the next step.
1.[Load Balancer(s)](load_balancer.md)[^2]
1.[Load Balancer(s)](load_balancer.md)[^6]
1.[Consul](consul.md)
1.[Consul](consul.md)
1.[PostgreSQL](database.md#postgresql-in-a-scaled-environment) with [PgBouncer](https://docs.gitlab.com/ee/administration/high_availability/pgbouncer.html)
1.[PostgreSQL](database.md#postgresql-in-a-scaled-environment) with [PgBouncer](pgbouncer.md)
1.[Redis](redis.md#redis-in-a-scaled-environment)
1.[Redis](redis.md#redis-in-a-scaled-environment)
1.[Gitaly](gitaly.md)(recommended) and / or [NFS](nfs.md)[^4]
1.[Gitaly](gitaly.md)(recommended) and / or [NFS](nfs.md)[^5]
1.[GitLab application nodes](gitlab.md)
1.[GitLab application nodes](gitlab.md)
- With [Object Storage service enabled](../gitaly/index.md#eliminating-nfs-altogether)[^3]
- With [Object Storage service enabled](../gitaly/index.md#eliminating-nfs-altogether)[^4]
1.[Monitoring node (Prometheus and Grafana)](monitoring_node.md)
1.[Monitoring node (Prometheus and Grafana)](monitoring_node.md)
### Full Scaling
### Full Scaling
...
@@ -103,10 +103,10 @@ in size, indicating that there is contention or there are not enough resources.
...
@@ -103,10 +103,10 @@ in size, indicating that there is contention or there are not enough resources.
- 1 or more PostgreSQL nodes
- 1 or more PostgreSQL nodes
- 1 or more Redis nodes
- 1 or more Redis nodes
- 1 or more Gitaly storage servers
- 1 or more Gitaly storage servers
- 1 or more Object Storage services[^3] and / or NFS storage server[^4]
- 1 or more Object Storage services[^4] and / or NFS storage server[^5]
- 2 or more Sidekiq nodes
- 2 or more Sidekiq nodes
- 2 or more GitLab application nodes (Unicorn / Puma, Workhorse, Sidekiq)
- 2 or more GitLab application nodes (Unicorn / Puma, Workhorse, Sidekiq)
- 1 or more Load Balancer nodes[^2]
- 1 or more Load Balancer nodes[^6]
- 1 Monitoring node (Prometheus, Grafana)
- 1 Monitoring node (Prometheus, Grafana)
## High Availability Architecture Examples
## High Availability Architecture Examples
...
@@ -117,17 +117,17 @@ page mentions, there is a tradeoff between cost/complexity and uptime. Be sure
...
@@ -117,17 +117,17 @@ page mentions, there is a tradeoff between cost/complexity and uptime. Be sure
this complexity is absolutely required before taking the step into full
this complexity is absolutely required before taking the step into full
high availability.
high availability.
For all examples below, we recommend running Consul and Redis Sentinel on
For all examples below, we recommend running Consul and Redis Sentinel separately
dedicated nodes. If Consul is running on PostgreSQL nodes or Sentinel on
from the services they monitor. If Consul is running on PostgreSQL nodes or Sentinel on
Redis nodes, there is a potential that high resource usage by PostgreSQL or
Redis nodes, there is a potential that high resource usage by PostgreSQL or
Redis could prevent communication between the other Consul and Sentinel nodes.
Redis could prevent communication between the other Consul and Sentinel nodes.
This may lead to the other nodes believing a failure has occurred and initiating
This may lead to the other nodes believing a failure has occurred and initiating
automated failover. Isolating Redis and Consul from the services they monitor
automated failover. Isolating Consul and Redis Sentinel from the services they monitor
reduces the chances of a false positive that a failure has occurred.
reduces the chances of a false positive that a failure has occurred.
The examples below do not address high availability of NFS for objects. We recommend a
The examples below do not address high availability of NFS for objects. We recommend a
S3 Object Storage service[^3] is used where possible over NFS but it's still required in
S3 Object Storage service[^4] is used where possible over NFS but it's still required in
certain cases[^4]. Where NFS is to be used some enterprises have access to NFS appliances
certain cases[^5]. Where NFS is to be used some enterprises have access to NFS appliances
that manage availability and this would be best case scenario.
that manage availability and this would be best case scenario.
There are many options in between each of these examples. Work with GitLab Support
There are many options in between each of these examples. Work with GitLab Support
...
@@ -147,12 +147,12 @@ moving to a hybrid or fully distributed architecture depending on what is causin
...
@@ -147,12 +147,12 @@ moving to a hybrid or fully distributed architecture depending on what is causin
the contention.
the contention.
- 3 PostgreSQL nodes
- 3 PostgreSQL nodes
-2 Redis nodes
-3 Redis nodes
- 3 Consul/Sentinel nodes
- 3 Consul / Sentinel nodes
- 2 or more GitLab application nodes (Unicorn / Puma, Workhorse, Sidekiq)
- 2 or more GitLab application nodes (Unicorn / Puma, Workhorse, Sidekiq)
- 1 Gitaly storage servers
- 1 Gitaly storage servers
- 1 Object Storage service[^3] and / or NFS storage server[^4]
- 1 Object Storage service[^4] and / or NFS storage server[^5]
> - Server hooks were [introduced](https://gitlab.com/gitlab-org/gitlab/issues/196051) in GitLab 12.8 replacing Custom Hooks.
> - Server hooks must be configured on the filesystem of the GitLab server. Only GitLab server administrators will be able to complete these tasks. Please explore [webhooks](../user/project/integrations/webhooks.md) and [GitLab CI/CD](../ci/README.md) as an option if you do not have filesystem access. For a user-configurable Git hook interface, see [Push Rules](../push_rules/push_rules.md), available in GitLab Starter **(STARTER)**.
> - Server hooks won't be replicated to secondary nodes if you use [GitLab Geo](geo/replication/index.md).
Git natively supports hooks that are executed on different actions.
Examples of server-side Git hooks include pre-receive, post-receive, and update.
See [Git SCM Server-Side Hooks](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks#Server-Side-Hooks) for more information about each hook type.
As of GitLab Shell version 2.2.0 (which requires GitLab 7.5+), GitLab
administrators can add custom Git hooks to any GitLab project.
## Create a server hook for a repository
Server-side Git hooks are typically placed in the repository's `hooks`
subdirectory. In GitLab, hook directories are symlinked to the GitLab Shell
`hooks` directory for ease of maintenance between GitLab Shell upgrades.
Server hooks are implemented differently, but the behavior is exactly the same
once the hook is created. Follow the steps below to set up a server hook for a
repository:
1. Pick a project that needs a server hook.
1. On the GitLab server, navigate to the project's repository directory.
For an installation from source the path is usually
`/home/git/repositories/<group>/<project>.git`. For Omnibus installs the path is
usually `/var/opt/gitlab/git-data/repositories/<group>/<project>.git`.
1. Create a new directory in this location called `custom_hooks`.
1. Inside the new `custom_hooks` directory, create a file with a name matching
the hook type. For a pre-receive hook the file name should be `pre-receive`
with no extension.
1. Make the hook file executable and make sure it's owned by Git.
1. Write the code to make the server hook function as expected. Hooks can be
in any language. Ensure the 'shebang' at the top properly reflects the language
type. For example, if the script is in Ruby the shebang will probably be
`#!/usr/bin/env ruby`.
That's it! Assuming the hook code is properly implemented the hook will fire
as appropriate.
## Set a global server hook for all repositories
To create a Git hook that applies to all of your repositories in
your instance, set a global server hook. Since GitLab will look inside the GitLab Shell
`hooks` directory for global hooks, adding any hook there will apply it to all repositories.
Follow the steps below to properly set up a server hook for all repositories:
1. On the GitLab server, navigate to the configured custom hook directory. The
default is in the GitLab Shell directory. The GitLab Shell `hook` directory
for an installation from source the path is usually
`/home/git/gitlab-shell/hooks`. For Omnibus installs the path is usually
@@ -256,12 +256,18 @@ Do not include the same information in multiple places. [Link to a SSOT instead.
...
@@ -256,12 +256,18 @@ Do not include the same information in multiple places. [Link to a SSOT instead.
Some features are also objects. For example, "GitLab's Merge Requests support X" and
Some features are also objects. For example, "GitLab's Merge Requests support X" and
"Create a new merge request for Z."
"Create a new merge request for Z."
- Use common contractions when it helps create a friendly and informal tone, especially in tutorials and [UIs](https://design.gitlab.com/content/punctuation/#contractions).
- Do use contractions like: _it's_, _can't_, _wouldn't_, _you're_, _you've_, _haven't_, don't, _we're_, _that's_, and _won't_. Contractions in instructional documentation such as tutorials can help create a friendly and informal tone.
- Avoid less common contractions such as: _he'd_, _it'll_, _should've_, and _there'd_.
- Do not use contractions in reference documentation. Examples:
- You cannot set a limit higher than 1000.
- For `parameter1`, the default is 10.
- Do not use contractions with a proper noun and a verb, such as _GitLab's creating X_.
- Avoid using contractions when you need to emphasize a negative, such as "Do **not** install X with Y."
- Avoid use of the future tense:
- Avoid use of the future tense:
- Instead of "after you execute this command, GitLab will display the result", use "after you execute this command, GitLab displays the result".
- Instead of "after you execute this command, GitLab will display the result", use "after you execute this command, GitLab displays the result".
- Only use the future tense to convey when the action or result will actually occur at a future time.
- Only use the future tense to convey when the action or result will actually occur at a future time.
- Do not use contractions:
- Instead of "don't," "can't," "doesn't," and so on, use "do not," "cannot," or "does not."
- Possible exceptions are cases when a more familiar tone is desired, such as a blog post or other casual context.
- Do not use slashes to clump different words together or as a replacement for the word "or":
- Do not use slashes to clump different words together or as a replacement for the word "or":
- Instead of "and/or," consider using "or," or use another sensible construction.
- Instead of "and/or," consider using "or," or use another sensible construction.
- Other examples include "clone/fetch," author/assignee," and "namespace/repository name." Break apart any such instances in an appropriate way.
- Other examples include "clone/fetch," author/assignee," and "namespace/repository name." Break apart any such instances in an appropriate way.
@@ -4,57 +4,31 @@ You can integrate your GitLab instance with GitHub.com as well as GitHub Enterpr
...
@@ -4,57 +4,31 @@ You can integrate your GitLab instance with GitHub.com as well as GitHub Enterpr
## Enabling GitHub OAuth
## Enabling GitHub OAuth
To enable GitHub OmniAuth provider, you must use GitHub's credentials for your GitLab instance.
To enable the GitHub OmniAuth provider, you'll need an OAuth 2 Client ID and Client Secret from GitHub. To get these credentials, sign into GitHub and follow their procedure for [Creating an OAuth App](https://developer.github.com/apps/building-oauth-apps/creating-an-oauth-app/).
To get the credentials (a pair of Client ID and Client Secret), you must register an application as an OAuth App on GitHub.
1. Sign in to GitHub.
When you create an OAuth 2 app in GitHub, you'll need the following information:
1. Navigate to your individual user or organization settings, depending on how you want the application registered. It does not matter if the application is registered as an individual or an organization - that is entirely up to you.
- The URL of your GitLab instance, such as `https://gitlab.example.com`.
- The authorization callback URL; in this case, `https://gitlab.example.com/users/auth`. Include the port number if your GitLab instance uses a non-default port.
- For individual accounts, select **Developer settings** from the left menu, then select **OAuth Apps**.
NOTE: **Note:**
- For organization accounts, directly select **OAuth Apps** from the left menu.
To prevent an [OAuth2 covert redirect](http://tetraph.com/covert_redirect/) vulnerability, append `/users/auth` to the end of the GitHub authorization callback URL.
1. Select **Register an application** (if you don't have any OAuth App) or **New OAuth App** (if you already have OAuth Apps).
See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings.
1. Provide the required details.
Once you have configured the GitHub provider, you'll need the following information, which you'll need to substitute in the GitLab configuration file, in the steps shown next.
- Application name: This can be anything. Consider something like `<Organization>'s GitLab` or `<Your Name>'s GitLab` or something else descriptive.
- Homepage URL: The URL of your GitLab installation. For example, `https://gitlab.example.com`.
- Application description: Fill this in if you wish.
- Authorization callback URL: `http(s)://${YOUR_DOMAIN}/users/auth`. Please make sure the port is included if your GitLab instance is not configured on default port.
support for it. The [Cilium](https://cilium.io/) network plugin can be
installed as a [cluster application](../../user/clusters/applications.md#install-cilium-using-gitlab-ci)
to enable support for network policies.
You can enable deployment of a network policy by setting the following
in the `.gitlab/auto-deploy-values.yaml` file:
```yml
networkPolicy:
enabled:true
```
The default policy deployed by the auto deploy pipeline will allow
traffic within a local namespace and from the `gitlab-managed-apps`
namespace, all other inbound connection will be blocked. Outbound
traffic is not affected by the default policy.
You can also provide a custom [policy specification](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.16/#networkpolicyspec-v1-networking-k8s-io)
via the `.gitlab/auto-deploy-values.yaml` file, for example:
```yml
networkPolicy:
enabled:true
spec:
podSelector:
matchLabels:
app.gitlab.com/env:staging
ingress:
-from:
-podSelector:
matchLabels:{}
-namespaceSelector:
matchLabels:
app.gitlab.com/managed_by:gitlab
```
#### Running commands in the container
#### Running commands in the container
Applications built with [Auto Build](#auto-build) using Herokuish, the default
Applications built with [Auto Build](#auto-build) using Herokuish, the default
Installation and removal of the Cilium [requires restart](https://cilium.readthedocs.io/en/stable/gettingstarted/k8s-install-gke/#restart-remaining-pods)
of all affected pods in all namespaces to ensure that they are
@@ -10,7 +10,7 @@ Error tracking allows developers to easily discover and view the errors that the
...
@@ -10,7 +10,7 @@ Error tracking allows developers to easily discover and view the errors that the
### Deploying Sentry
### Deploying Sentry
You may sign up to the cloud hosted <https://sentry.io> or deploy your own [on-premise instance](https://docs.sentry.io/server/installation/).
You may sign up to the cloud hosted <https://sentry.io>, deploy your own [on-premise instance](https://docs.sentry.io/server/installation/) or use GitLab to [install Sentry to a Kubernetes cluster](../../clusters/applications.md#install-sentry-using-gitlab-ci).
Note: If you want to run tests requiring SSH against GDK, you
Note: If you want to run tests requiring SSH against GDK, you
will need to [modify your GDK setup](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/run_qa_against_gdk.md).
will need to [modify your GDK setup](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/run_qa_against_gdk.md).
#### Running EE tests
When running EE tests you'll need to have a license available. GitLab engineers can [request a license](https://about.gitlab.com/handbook/developer-onboarding/#working-on-gitlab-ee).
Once you have the license file you can export it as an environment variable and then the framework can use it. If you do so it will be installed automatically.
```
export EE_LICENSE=$(cat /path/to/gitlab_license)
```
### Writing tests
### Writing tests
-[Writing tests from scratch tutorial](../doc/development/testing_guide/end_to_end/quick_start_guide.md)
-[Writing tests from scratch tutorial](../doc/development/testing_guide/end_to_end/quick_start_guide.md)
