Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
G
gitlab-ce
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
1
Merge Requests
1
Analytics
Analytics
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Commits
Issue Boards
Open sidebar
nexedi
gitlab-ce
Commits
c91168c0
Commit
c91168c0
authored
Aug 03, 2016
by
Achilleas Pipinellis
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Small refactor on Registry troubleshooting
[ci skip]
parent
2c9cce0f
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
23 additions
and
26 deletions
+23
-26
doc/container_registry/troubleshooting.md
doc/container_registry/troubleshooting.md
+23
-26
No files found.
doc/container_registry/troubleshooting.md
View file @
c91168c0
...
@@ -5,27 +5,27 @@
...
@@ -5,27 +5,27 @@
1.
Check to make sure that the system clock on your Docker client and GitLab server have
1.
Check to make sure that the system clock on your Docker client and GitLab server have
been synchronized (e.g. via NTP).
been synchronized (e.g. via NTP).
2.
If you are using an S3-backed
r
egistry, double check that the IAM
2.
If you are using an S3-backed
R
egistry, double check that the IAM
permissions and the S3 credentials (including region) are correct. See
[
the
permissions and the S3 credentials (including region) are correct. See
[
the
sample IAM policy
](
https://docs.docker.com/registry/storage-drivers/s3/
)
sample IAM policy
](
https://docs.docker.com/registry/storage-drivers/s3/
)
for more details.
for more details.
3.
Check the
r
egistry logs (e.g.
`/var/log/gitlab/registry/current`
) and the GitLab production logs
3.
Check the
R
egistry logs (e.g.
`/var/log/gitlab/registry/current`
) and the GitLab production logs
for errors (e.g.
`/var/log/gitlab/gitlab-rails/production.log`
). You may be able to find clues
for errors (e.g.
`/var/log/gitlab/gitlab-rails/production.log`
). You may be able to find clues
there.
there.
# Advanced Troubleshooting
#
#
Advanced Troubleshooting
NOTE:
The following section is only recommended for experts.
>**NOTE:**
The following section is only recommended for experts.
Sometimes it's not obvious what is wrong, and you may need to dive deeper into
Sometimes it's not obvious what is wrong, and you may need to dive deeper into
the communication between the Docker client and the
r
egistry to find out
the communication between the Docker client and the
R
egistry to find out
what's wrong. We will use a concrete example in the past to illustrate how to
what's wrong. We will use a concrete example in the past to illustrate how to
diagnose a problem with the S3 setup.
diagnose a problem with the S3 setup.
##
Example:
Unexpected 403 error during push
##
#
Unexpected 403 error during push
A user attempted to enable an S3-backed
r
egistry. The
`docker login`
step went
A user attempted to enable an S3-backed
R
egistry. The
`docker login`
step went
fine. However, when pushing an image, the output showed:
fine. However, when pushing an image, the output showed:
```
```
...
@@ -44,11 +44,11 @@ error parsing HTTP 403 response body: unexpected end of JSON input: ""
...
@@ -44,11 +44,11 @@ error parsing HTTP 403 response body: unexpected end of JSON input: ""
```
```
This error is ambiguous, as it's not clear whether the 403 is coming from the
This error is ambiguous, as it's not clear whether the 403 is coming from the
GitLab Rails application, the Docker
r
egistry, or something else. In this
GitLab Rails application, the Docker
R
egistry, or something else. In this
case, since we know that since the login succeeded, we probably need to look
case, since we know that since the login succeeded, we probably need to look
at the communication between the client and the
r
egistry.
at the communication between the client and the
R
egistry.
The REST API between the Docker client and
r
egistry is
[
described
The REST API between the Docker client and
R
egistry is
[
described
here
](
https://docs.docker.com/registry/spec/api/
)
. Normally, one would just
here
](
https://docs.docker.com/registry/spec/api/
)
. Normally, one would just
use Wireshark or tcpdump to capture the traffic and see where things went
use Wireshark or tcpdump to capture the traffic and see where things went
wrong. However, since all communication between Docker clients and servers
wrong. However, since all communication between Docker clients and servers
...
@@ -56,12 +56,12 @@ are done over HTTPS, it's a bit difficult to decrypt the traffic quickly even
...
@@ -56,12 +56,12 @@ are done over HTTPS, it's a bit difficult to decrypt the traffic quickly even
if you know the private key. What can we do instead?
if you know the private key. What can we do instead?
One way would be to disable HTTPS by setting up an
[
insecure
One way would be to disable HTTPS by setting up an
[
insecure
r
egistry
](
https://docs.docker.com/registry/insecure/
)
. This could introduce a
R
egistry
](
https://docs.docker.com/registry/insecure/
)
. This could introduce a
security hole and is only recommended for local testing. If you have a
security hole and is only recommended for local testing. If you have a
production system and can't or don't want to do this, there is another way:
production system and can't or don't want to do this, there is another way:
use mitmproxy, which stands for Man-in-the-Middle Proxy.
use mitmproxy, which stands for Man-in-the-Middle Proxy.
## mitmproxy
##
#
mitmproxy
[
mitmproxy
](
https://mitmproxy.org/
)
allows you to place a proxy between your
[
mitmproxy
](
https://mitmproxy.org/
)
allows you to place a proxy between your
client and server to inspect all traffic. One wrinkle is that your system
client and server to inspect all traffic. One wrinkle is that your system
...
@@ -70,10 +70,9 @@ needs to trust the mitmproxy SSL certificates for this to work.
...
@@ -70,10 +70,9 @@ needs to trust the mitmproxy SSL certificates for this to work.
The following installation instructions assume you are running Ubuntu:
The following installation instructions assume you are running Ubuntu:
1.
Install mitmproxy (see http://docs.mitmproxy.org/en/stable/install.html)
1.
Install mitmproxy (see http://docs.mitmproxy.org/en/stable/install.html)
1.
Run
`mitmproxy --port 9000`
to generate its certificates.
2.
Run
`mitmproxy --port 9000`
to generate its certificates. Enter CTRL-C to quit.
Enter
<kbd>
CTRL
</kbd>
-
<kbd>
C
</kbd>
to quit.
1.
Install the certificate from
`~/.mitmproxy`
to your system:
3.
Install the certificate from ~/.mitmproxy to your system:
```sh
```sh
sudo cp ~/.mitmproxy/mitmproxy-ca-cert.pem /usr/local/share/ca-certificates/mitmproxy-ca-cert.crt
sudo cp ~/.mitmproxy/mitmproxy-ca-cert.pem /usr/local/share/ca-certificates/mitmproxy-ca-cert.crt
...
@@ -87,24 +86,22 @@ Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done.
...
@@ -87,24 +86,22 @@ Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done.
Running hooks
in
/etc/ca-certificates/update.d....done.
Running hooks
in
/etc/ca-certificates/update.d....done.
```
```
## Verifying mitmproxy certifiactes
To verify that the certificates are properly installed, run:
To verify that the certificates are properly install, run:
```
sh
```
sh
mitmproxy
--port
9000
mitmproxy
--port
9000
```
```
This will run mitmproxy on port
9000
. In another window, run:
This will run mitmproxy on port
`9000`
. In another window, run:
```
sh
```
sh
curl
--proxy
http://localhost:9000 https://httpbin.org/status/200
curl
--proxy
http://localhost:9000 https://httpbin.org/status/200
```
```
If everything is setup correctly,
then
you will see information on the mitmproxy window and
If everything is setup correctly, you will see information on the mitmproxy window and
no errors from the curl commands.
no errors from the curl commands.
## Running the Docker daemon with a proxy
##
#
Running the Docker daemon with a proxy
For Docker to connect through a proxy, you must start the Docker daemon with the
For Docker to connect through a proxy, you must start the Docker daemon with the
proper environment variables. The easiest way is to shutdown Docker (e.g.
`sudo initctl stop docker`
)
proper environment variables. The easiest way is to shutdown Docker (e.g.
`sudo initctl stop docker`
)
...
@@ -118,10 +115,10 @@ docker daemon --debug
...
@@ -118,10 +115,10 @@ docker daemon --debug
This will launch the Docker daemon and proxy all connections through mitmproxy.
This will launch the Docker daemon and proxy all connections through mitmproxy.
## Running the Docker client
##
#
Running the Docker client
Now that we have mitmproxy and Docker running, we can
now attempt to login and push a container
Now that we have mitmproxy and Docker running, we can
attempt to login and push
image. You may need to run as root to do this. For example:
a container
image. You may need to run as root to do this. For example:
```
sh
```
sh
docker login s3-testing.myregistry.com:4567
docker login s3-testing.myregistry.com:4567
...
@@ -141,4 +138,4 @@ The above image shows:
...
@@ -141,4 +138,4 @@ The above image shows:
What does this mean? This strongly suggests that the S3 user does not have the right
What does this mean? This strongly suggests that the S3 user does not have the right
[
permissions to perform a HEAD request
](
http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectHEAD.html
)
.
[
permissions to perform a HEAD request
](
http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectHEAD.html
)
.
The solution: check the
[
IAM permissions again
](
https://docs.docker.com/registry/storage-drivers/s3/
)
.
The solution: check the
[
IAM permissions again
](
https://docs.docker.com/registry/storage-drivers/s3/
)
.
Once the right permissions were set, the error w
ent
away.
Once the right permissions were set, the error w
ill go
away.
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment