Skip to content

G
gitlab-ce
Commit ab85ad6d authored by Achilleas Pipinellis's avatar Achilleas Pipinellis
Browse files
Options

Merge branch 'docs/refactor-registry' into 'master' 

Container Registry refactor docs

From https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6516

See merge request !790
parents e37ed543 ecfba52f
Hide whitespace changes
Inline Side-by-side
Showing with 318 additions and 274 deletions
doc/README.md
View file @ ab85ad6d
......@@ -7,7 +7,7 @@
- [CI/CD](ci/README.md) GitLab Continuous Integration (CI) and Continuous Delivery (CD) getting started, `.gitlab-ci.yml` options, and examples.
- [Custom templates for issues and merge requests](customization/issue_and_merge_request_template.md) Pre-fill the description of issues and merge requests to your liking.
- [GitLab as OAuth2 authentication service provider](integration/oauth_provider.md). It allows you to login to other applications from GitLab.
- [Container Registry](container_registry/README.md) Learn how to use GitLab Container Registry.
- [Container Registry](user/project/container_registry.md) Learn how to use GitLab Container Registry.
- [GitLab Basics](gitlab-basics/README.md) Find step by step how to start working on your commandline and on GitLab.
- [GitLab Pages](pages/README.md) Using GitLab Pages.
- [Importing to GitLab](workflow/importing/README.md).
......
doc/administration/container_registry.md
View file @ ab85ad6d
# GitLab Container Registry Administration
# GitLab Container Registry administration
> [Introduced][ce-4040] in GitLab 8.8.
With the Docker Container Registry integrated into GitLab, every project can
have its own space to store its Docker images.
You can read more about Docker Registry at https://docs.docker.com/registry/introduction/.
---
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
> **Notes:**
- Container Registry manifest `v1` support was added in GitLab 8.9 to support
Docker versions earlier than 1.10.
- This document is about the admin guide. To learn how to use GitLab Container
Registry [user documentation](../user/project/container_registry.md).
- [Enable the Container Registry](#enable-the-container-registry)
- [Container Registry domain configuration](#container-registry-domain-configuration)
- [Configure Container Registry under an existing GitLab domain](#configure-container-registry-under-an-existing-gitlab-domain)
- [Configure Container Registry under its own domain](#configure-container-registry-under-its-own-domain)
- [Disable Container Registry site-wide](#disable-container-registry-site-wide)
- [Disable Container Registry per project](#disable-container-registry-per-project)
- [Disable Container Registry for new projects site-wide](#disable-container-registry-for-new-projects-site-wide)
- [Container Registry storage path](#container-registry-storage-path)
- [Container Registry storage driver](#container-registry-storage-driver)
- [Storage limitations](#storage-limitations)
- [Changelog](#changelog)
With the Container Registry integrated into GitLab, every project can have its
own space to store its Docker images.
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
You can read more about the Container Registry at
https://docs.docker.com/registry/introduction/.
## Enable the Container Registry
**Omnibus GitLab installations**
All you have to do is configure the domain name under which the Container
Registry will listen to. Read [#container-registry-domain-configuration](#container-registry-domain-configuration)
Registry will listen to. Read
[#container-registry-domain-configuration](#container-registry-domain-configuration)
and pick one of the two options that fits your case.
>**Note:**
The container Registry works under HTTPS by default. Using HTTP is possible
The container registry works under HTTPS by default. Using HTTP is possible
but not recommended and out of the scope of this document.
Read the [insecure Registry documentation][docker-insecure] if you want to
implement this.
......@@ -47,7 +37,7 @@ implement this.
If you have installed GitLab from source:
1. You will have to [install Docker Registry][registry-deploy] by yourself.
1. You will have to [install Registry][registry-deploy] by yourself.
1. After the installation is complete, you will have to configure the Registry's
settings in `gitlab.yml` in order to enable it.
1. Use the sample NGINX configuration file that is found under
......@@ -80,11 +70,13 @@ where:
| `issuer` | This should be the same value as configured in Registry's `issuer`. Read the [token auth configuration documentation][token-config]. |
>**Note:**
GitLab does not ship with a Registry init file. Hence, [restarting GitLab][restart gitlab]
will not restart the Registry should you modify its settings. Read the upstream
documentation on how to achieve that.
A Registry init file is not shipped with GitLab if you install it from source.
Hence, [restarting GitLab][restart gitlab] will not restart the Registry should
you modify its settings. Read the upstream documentation on how to achieve that.
The Docker Registry configuration will need `container_registry` as the service and `https://gitlab.example.com/jwt/auth` as the realm:
At the absolute minimum, make sure your [Registry configuration][registry-auth]
has `container_registry` as the service and `https://gitlab.example.com/jwt/auth`
as the realm:
```
auth:
......@@ -275,12 +267,6 @@ Registry application itself.
1. Save the file and [restart GitLab][] for the changes to take effect.
## Disable Container Registry per project
If Registry is enabled in your GitLab instance, but you don't need it for your
project, you can disable it from your project's settings. Read the user guide
on how to achieve that.
## Disable Container Registry for new projects site-wide
If the Container Registry is enabled, then it will be available on all new
......@@ -436,6 +422,46 @@ storage:
enabled: true
```
## Change the registry's internal port
> **Note:**
This is not to be confused with the port that GitLab itself uses to expose
the Registry to the world.
The Registry server listens on localhost at port `5000` by default,
which is the address for which the Registry server should accept connections.
In the examples below we set the Registry's port to `5001`.
**Omnibus GitLab**
1. Open `/etc/gitlab/gitlab.rb` and set `registry['registry_http_addr']`:
```ruby
registry['registry_http_addr'] = "localhost:5001"
```
1. Save the file and [reconfigure GitLab][] for the changes to take effect.
---
**Installations from source**
1. Open the configuration file of your Registry server and edit the
[`http:addr`][registry-http-config] value:
```
http
addr: localhost:5001
```
1. Save the file and restart the Registry server.
## Disable Container Registry per project
If Registry is enabled in your GitLab instance, but you don't need it for your
project, you can disable it from your project's settings. Read the user guide
on how to achieve that.
## Storage limitations
Currently, there is no storage limitation, which means a user can upload an
......@@ -455,6 +481,8 @@ configurable in future releases.
[docker-insecure]: https://docs.docker.com/registry/insecure/
[registry-deploy]: https://docs.docker.com/registry/deploying/
[storage-config]: https://docs.docker.com/registry/configuration/#storage
[registry-http-config]: https://docs.docker.com/registry/configuration/#http
[registry-auth]: https://docs.docker.com/registry/configuration/#auth
[token-config]: https://docs.docker.com/registry/configuration/#token
[8-8-docs]: https://gitlab.com/gitlab-org/gitlab-ce/blob/8-8-stable/doc/administration/container_registry.md
[registry-ssl]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/support/nginx/registry-ssl
......
doc/container_registry/README.md
View file @ ab85ad6d
# GitLab Container Registry
> [Introduced][ce-4040] in GitLab 8.8. Docker Registry manifest
`v1` support was added in GitLab 8.9 to support Docker versions earlier than 1.10.
> **Note:**
This document is about the user guide. To learn how to enable GitLab Container
Registry across your GitLab instance, visit the
[administrator documentation](../administration/container_registry.md).
With the Docker Container Registry integrated into GitLab, every project can
have its own space to store its Docker images.
You can read more about Docker Registry at https://docs.docker.com/registry/introduction/.
---
## Enable the Container Registry for your project
1. First, ask your system administrator to enable GitLab Container Registry
following the [administration documentation](../administration/container_registry.md).
If you are using GitLab.com, this is enabled by default so you can start using
the Registry immediately.
1. Go to your project's settings and enable the **Container Registry** feature
on your project. For new projects this might be enabled by default. For
existing projects you will have to explicitly enable it.
![Enable Container Registry](img/project_feature.png)
## Build and push images
After you save your project's settings, you should see a new link in the
sidebar called **Container Registry**. Following this link will get you to
your project's Registry panel where you can see how to login to the Container
Registry using your GitLab credentials.
For example if the Registry's URL is `registry.example.com`, the you should be
able to login with:
```
docker login registry.example.com
```
Building and publishing images should be a straightforward process. Just make
sure that you are using the Registry URL with the namespace and project name
that is hosted on GitLab:
```
docker build -t registry.example.com/group/project .
docker push registry.example.com/group/project
```
## Use images from GitLab Container Registry
To download and run a container from images hosted in GitLab Container Registry,
use `docker run`:
```
docker run [options] registry.example.com/group/project [arguments]
```
For more information on running Docker containers, visit the
[Docker documentation][docker-docs].
## Control Container Registry from within GitLab
GitLab offers a simple Container Registry management panel. Go to your project
and click **Container Registry** in the left sidebar.
This view will show you all tags in your project and will easily allow you to
delete them.
![Container Registry panel](img/container_registry.png)
## Build and push images using GitLab CI
> **Note:**
This feature requires GitLab 8.8 and GitLab Runner 1.2.
Make sure that your GitLab Runner is configured to allow building Docker images by
following the [Using Docker Build](../ci/docker/using_docker_build.md)
and [Using the GitLab Container Registry documentation](../ci/docker/using_docker_build.md#using-the-gitlab-container-registry).
## Limitations
In order to use a container image from your private project as an `image:` in
your `.gitlab-ci.yml`, you have to follow the
[Using a private Docker Registry][private-docker]
documentation. This workflow will be simplified in the future.
## Troubleshooting
See [the GitLab Docker registry troubleshooting guide](troubleshooting.md).
[ce-4040]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4040
[docker-docs]: https://docs.docker.com/engine/userguide/intro/
[private-docker]: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/configuration/advanced-configuration.md#using-a-private-docker-registry
This document was moved in [user/project/container_registry](../user/project/container_registry.md).
doc/container_registry/troubleshooting.md
View file @ ab85ad6d
# Troubleshooting the GitLab Container Registry
## Basic Troubleshooting
1. Check to make sure that the system clock on your Docker client and GitLab server have
been synchronized (e.g. via NTP).
2. If you are using an S3-backed Registry, double check that the IAM
permissions and the S3 credentials (including region) are correct. See [the
sample IAM policy](https://docs.docker.com/registry/storage-drivers/s3/)
for more details.
3. Check the Registry logs (e.g. `/var/log/gitlab/registry/current`) and the GitLab production logs
for errors (e.g. `/var/log/gitlab/gitlab-rails/production.log`). You may be able to find clues
there.
## Advanced Troubleshooting
>**NOTE:** The following section is only recommended for experts.
Sometimes it's not obvious what is wrong, and you may need to dive deeper into
the communication between the Docker client and the Registry to find out
what's wrong. We will use a concrete example in the past to illustrate how to
diagnose a problem with the S3 setup.
### Unexpected 403 error during push
A user attempted to enable an S3-backed Registry. The `docker login` step went
fine. However, when pushing an image, the output showed:
```
The push refers to a repository [s3-testing.myregistry.com:4567/root/docker-test]
dc5e59c14160: Pushing [==================================================>] 14.85 kB
03c20c1a019a: Pushing [==================================================>] 2.048 kB
a08f14ef632e: Pushing [==================================================>] 2.048 kB
228950524c88: Pushing 2.048 kB
6a8ecde4cc03: Pushing [==> ] 9.901 MB/205.7 MB
5f70bf18a086: Pushing 1.024 kB
737f40e80b7f: Waiting
82b57dbc5385: Waiting
19429b698a22: Waiting
9436069b92a3: Waiting
error parsing HTTP 403 response body: unexpected end of JSON input: ""
```
This error is ambiguous, as it's not clear whether the 403 is coming from the
GitLab Rails application, the Docker Registry, or something else. In this
case, since we know that since the login succeeded, we probably need to look
at the communication between the client and the Registry.
The REST API between the Docker client and Registry is [described
here](https://docs.docker.com/registry/spec/api/). Normally, one would just
use Wireshark or tcpdump to capture the traffic and see where things went
wrong. However, since all communication between Docker clients and servers
are done over HTTPS, it's a bit difficult to decrypt the traffic quickly even
if you know the private key. What can we do instead?
One way would be to disable HTTPS by setting up an [insecure
Registry](https://docs.docker.com/registry/insecure/). This could introduce a
security hole and is only recommended for local testing. If you have a
production system and can't or don't want to do this, there is another way:
use mitmproxy, which stands for Man-in-the-Middle Proxy.
### mitmproxy
[mitmproxy](https://mitmproxy.org/) allows you to place a proxy between your
client and server to inspect all traffic. One wrinkle is that your system
needs to trust the mitmproxy SSL certificates for this to work.
The following installation instructions assume you are running Ubuntu:
1. Install mitmproxy (see http://docs.mitmproxy.org/en/stable/install.html)
1. Run `mitmproxy --port 9000` to generate its certificates.
Enter <kbd>CTRL</kbd>-<kbd>C</kbd> to quit.
1. Install the certificate from `~/.mitmproxy` to your system:
```sh
sudo cp ~/.mitmproxy/mitmproxy-ca-cert.pem /usr/local/share/ca-certificates/mitmproxy-ca-cert.crt
sudo update-ca-certificates
```
If successful, the output should indicate that a certificate was added:
```sh
Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d....done.
```
To verify that the certificates are properly installed, run:
```sh
mitmproxy --port 9000
```
This will run mitmproxy on port `9000`. In another window, run:
```sh
curl --proxy http://localhost:9000 https://httpbin.org/status/200
```
If everything is setup correctly, you will see information on the mitmproxy window and
no errors from the curl commands.
### Running the Docker daemon with a proxy
For Docker to connect through a proxy, you must start the Docker daemon with the
proper environment variables. The easiest way is to shutdown Docker (e.g. `sudo initctl stop docker`)
and then run Docker by hand. As root, run:
```sh
export HTTP_PROXY="http://localhost:9000"
export HTTPS_PROXY="https://localhost:9000"
docker daemon --debug
```
This will launch the Docker daemon and proxy all connections through mitmproxy.
### Running the Docker client
Now that we have mitmproxy and Docker running, we can attempt to login and push
a container image. You may need to run as root to do this. For example:
```sh
docker login s3-testing.myregistry.com:4567
docker push s3-testing.myregistry.com:4567/root/docker-test
```
In the example above, we see the following trace on the mitmproxy window:
![mitmproxy output from Docker](img/mitmproxy-docker.png)
The above image shows:
* The initial PUT requests went through fine with a 201 status code.
* The 201 redirected the client to the S3 bucket.
* The HEAD request to the AWS bucket reported a 403 Unauthorized.
What does this mean? This strongly suggests that the S3 user does not have the right
[permissions to perform a HEAD request](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectHEAD.html).
The solution: check the [IAM permissions again](https://docs.docker.com/registry/storage-drivers/s3/).
Once the right permissions were set, the error will go away.
This document was moved to [user/project/container_registry](../user/project/container_registry.md).
doc/user/project/container_registry.md 0 → 100644
View file @ ab85ad6d
# GitLab Container Registry
> [Introduced][ce-4040] in GitLab 8.8.
---
> **Note**
Docker Registry manifest `v1` support was added in GitLab 8.9 to support Docker
versions earlier than 1.10.
>
This document is about the user guide. To learn how to enable GitLab Container
Registry across your GitLab instance, visit the
[administrator documentation](../../administration/container_registry.md).
With the Docker Container Registry integrated into GitLab, every project can
have its own space to store its Docker images.
You can read more about Docker Registry at https://docs.docker.com/registry/introduction/.
---
## Enable the Container Registry for your project
1. First, ask your system administrator to enable GitLab Container Registry
following the [administration documentation](../../administration/container_registry.md).
If you are using GitLab.com, this is enabled by default so you can start using
the Registry immediately.
1. Go to your project's settings and enable the **Container Registry** feature
on your project. For new projects this might be enabled by default. For
existing projects (prior GitLab 8.8), you will have to explicitly enable it.
![Enable Container Registry](img/container_registry_enable.png)
1. Hit **Save changes** for the changes to take effect. You should now be able
to see the **Registry** link in the project menu.
![Container Registry tab](img/container_registry_tab.png)
## Build and push images
If you visit the **Registry** link under your project's menu, you can see the
explicit instructions to login to the Container Registry using your GitLab
credentials.
For example if the Registry's URL is `registry.example.com`, the you should be
able to login with:
```
docker login registry.example.com
```
Building and publishing images should be a straightforward process. Just make
sure that you are using the Registry URL with the namespace and project name
that is hosted on GitLab:
```
docker build -t registry.example.com/group/project .
docker push registry.example.com/group/project
```
Your image will be named after the following scheme:
```
<registry URL>/<namespace>/<project>
```
As such, the name of the image is unique, but you can differentiate the images
using tags.
## Use images from GitLab Container Registry
To download and run a container from images hosted in GitLab Container Registry,
use `docker run`:
```
docker run [options] registry.example.com/group/project [arguments]
```
For more information on running Docker containers, visit the
[Docker documentation][docker-docs].
## Control Container Registry from within GitLab
GitLab offers a simple Container Registry management panel. Go to your project
and click **Registry** in the project menu.
This view will show you all tags in your project and will easily allow you to
delete them.
![Container Registry panel](img/container_registry_panel.png)
## Build and push images using GitLab CI
> **Note:**
This feature requires GitLab 8.8 and GitLab Runner 1.2.
Make sure that your GitLab Runner is configured to allow building Docker images by
following the [Using Docker Build](../ci/docker/using_docker_build.md)
and [Using the GitLab Container Registry documentation](../ci/docker/using_docker_build.md#using-the-gitlab-container-registry).
## Limitations
In order to use a container image from your private project as an `image:` in
your `.gitlab-ci.yml`, you have to follow the
[Using a private Docker Registry][private-docker]
documentation. This workflow will be simplified in the future.
## Troubleshooting the GitLab Container Registry
### Basic Troubleshooting
1. Check to make sure that the system clock on your Docker client and GitLab server have
been synchronized (e.g. via NTP).
2. If you are using an S3-backed Registry, double check that the IAM
permissions and the S3 credentials (including region) are correct. See [the
sample IAM policy](https://docs.docker.com/registry/storage-drivers/s3/)
for more details.
3. Check the Registry logs (e.g. `/var/log/gitlab/registry/current`) and the GitLab production logs
for errors (e.g. `/var/log/gitlab/gitlab-rails/production.log`). You may be able to find clues
there.
### Advanced Troubleshooting
>**NOTE:** The following section is only recommended for experts.
Sometimes it's not obvious what is wrong, and you may need to dive deeper into
the communication between the Docker client and the Registry to find out
what's wrong. We will use a concrete example in the past to illustrate how to
diagnose a problem with the S3 setup.
#### Unexpected 403 error during push
A user attempted to enable an S3-backed Registry. The `docker login` step went
fine. However, when pushing an image, the output showed:
```
The push refers to a repository [s3-testing.myregistry.com:4567/root/docker-test]
dc5e59c14160: Pushing [==================================================>] 14.85 kB
03c20c1a019a: Pushing [==================================================>] 2.048 kB
a08f14ef632e: Pushing [==================================================>] 2.048 kB
228950524c88: Pushing 2.048 kB
6a8ecde4cc03: Pushing [==> ] 9.901 MB/205.7 MB
5f70bf18a086: Pushing 1.024 kB
737f40e80b7f: Waiting
82b57dbc5385: Waiting
19429b698a22: Waiting
9436069b92a3: Waiting
error parsing HTTP 403 response body: unexpected end of JSON input: ""
```
This error is ambiguous, as it's not clear whether the 403 is coming from the
GitLab Rails application, the Docker Registry, or something else. In this
case, since we know that since the login succeeded, we probably need to look
at the communication between the client and the Registry.
The REST API between the Docker client and Registry is [described
here](https://docs.docker.com/registry/spec/api/). Normally, one would just
use Wireshark or tcpdump to capture the traffic and see where things went
wrong. However, since all communication between Docker clients and servers
are done over HTTPS, it's a bit difficult to decrypt the traffic quickly even
if you know the private key. What can we do instead?
One way would be to disable HTTPS by setting up an [insecure
Registry](https://docs.docker.com/registry/insecure/). This could introduce a
security hole and is only recommended for local testing. If you have a
production system and can't or don't want to do this, there is another way:
use mitmproxy, which stands for Man-in-the-Middle Proxy.
#### mitmproxy
[mitmproxy](https://mitmproxy.org/) allows you to place a proxy between your
client and server to inspect all traffic. One wrinkle is that your system
needs to trust the mitmproxy SSL certificates for this to work.
The following installation instructions assume you are running Ubuntu:
1. Install mitmproxy (see http://docs.mitmproxy.org/en/stable/install.html)
1. Run `mitmproxy --port 9000` to generate its certificates.
Enter <kbd>CTRL</kbd>-<kbd>C</kbd> to quit.
1. Install the certificate from `~/.mitmproxy` to your system:
```sh
sudo cp ~/.mitmproxy/mitmproxy-ca-cert.pem /usr/local/share/ca-certificates/mitmproxy-ca-cert.crt
sudo update-ca-certificates
```
If successful, the output should indicate that a certificate was added:
```sh
Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d....done.
```
To verify that the certificates are properly installed, run:
```sh
mitmproxy --port 9000
```
This will run mitmproxy on port `9000`. In another window, run:
```sh
curl --proxy http://localhost:9000 https://httpbin.org/status/200
```
If everything is setup correctly, you will see information on the mitmproxy window and
no errors from the curl commands.
#### Running the Docker daemon with a proxy
For Docker to connect through a proxy, you must start the Docker daemon with the
proper environment variables. The easiest way is to shutdown Docker (e.g. `sudo initctl stop docker`)
and then run Docker by hand. As root, run:
```sh
export HTTP_PROXY="http://localhost:9000"
export HTTPS_PROXY="https://localhost:9000"
docker daemon --debug
```
This will launch the Docker daemon and proxy all connections through mitmproxy.
#### Running the Docker client
Now that we have mitmproxy and Docker running, we can attempt to login and push
a container image. You may need to run as root to do this. For example:
```sh
docker login s3-testing.myregistry.com:4567
docker push s3-testing.myregistry.com:4567/root/docker-test
```
In the example above, we see the following trace on the mitmproxy window:
![mitmproxy output from Docker](img/mitmproxy-docker.png)
The above image shows:
* The initial PUT requests went through fine with a 201 status code.
* The 201 redirected the client to the S3 bucket.
* The HEAD request to the AWS bucket reported a 403 Unauthorized.
What does this mean? This strongly suggests that the S3 user does not have the right
[permissions to perform a HEAD request](http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectHEAD.html).
The solution: check the [IAM permissions again](https://docs.docker.com/registry/storage-drivers/s3/).
Once the right permissions were set, the error will go away.
[ce-4040]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4040
[docker-docs]: https://docs.docker.com/engine/userguide/intro/
[private-docker]: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/configuration/advanced-configuration.md#using-a-private-docker-registry
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7