Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
G
gitlab-ce
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
1
Merge Requests
1
Analytics
Analytics
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Commits
Issue Boards
Open sidebar
nexedi
gitlab-ce
Commits
e21dc168
Commit
e21dc168
authored
Apr 15, 2022
by
Katrin Leinweber
Committed by
Achilleas Pipinellis
Apr 15, 2022
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Crosslink Geo errors example
A follow-up to
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/83004
parent
7482ed82
Changes
3
Hide whitespace changes
Inline
Side-by-side
Showing
3 changed files
with
24 additions
and
9 deletions
+24
-9
doc/administration/geo/replication/troubleshooting.md
doc/administration/geo/replication/troubleshooting.md
+3
-0
doc/administration/gitaly/troubleshooting.md
doc/administration/gitaly/troubleshooting.md
+2
-1
doc/administration/troubleshooting/log_parsing.md
doc/administration/troubleshooting/log_parsing.md
+19
-8
No files found.
doc/administration/geo/replication/troubleshooting.md
View file @
e21dc168
...
...
@@ -111,6 +111,9 @@ http://secondary.example.com/
Last status report was: 2 minutes ago
```
To find more details about failed items, check
[
the `gitlab-rails/geo.log` file
](
../../troubleshooting/log_parsing.md#find-most-common-geo-sync-errors
)
### Check if PostgreSQL replication is working
To check if PostgreSQL replication is working, check if:
...
...
doc/administration/gitaly/troubleshooting.md
View file @
e21dc168
...
...
@@ -16,7 +16,8 @@ Before troubleshooting, see the Gitaly and Gitaly Cluster
The following sections provide possible solutions to Gitaly errors.
See also
[
Gitaly timeout
](
../../user/admin_area/settings/gitaly_timeouts.md
)
settings.
See also
[
Gitaly timeout
](
../../user/admin_area/settings/gitaly_timeouts.md
)
settings,
and our advice on
[
parsing the `gitaly/current` file
](
../troubleshooting/log_parsing.md#parsing-gitalycurrent
)
.
### Check versions when using standalone Gitaly servers
...
...
doc/administration/troubleshooting/log_parsing.md
View file @
e21dc168
...
...
@@ -11,6 +11,11 @@ but if they are not available you can still quickly parse
[
GitLab logs
](
../logs.md
)
in JSON format
(the default in GitLab 12.0 and later) using
[
`jq`
](
https://stedolan.github.io/jq/
)
.
NOTE:
Spefically for summarising error events and basic usage statistics,
the GitLab Support Team provides the specialised
[
`fast-stats` tool
](
https://gitlab.com/gitlab-com/support/toolbox/fast-stats/#when-to-use-it
)
.
## What is JQ?
As noted in its
[
manual
](
https://stedolan.github.io/jq/manual/
)
,
`jq`
is a command-line JSON processor. The following examples
...
...
@@ -18,6 +23,10 @@ include use cases targeted for parsing GitLab log files.
## Parsing Logs
The examples listed below address their respective log files by
their relative Omnibus paths and default filenames.
Find the respective full paths in the
[
GitLab logs sections
](
../logs.md#production_jsonlog
)
.
### General Commands
#### Pipe colorized `jq` output into `less`
...
...
@@ -61,7 +70,7 @@ zcat some_json.log.25.gz | (head -1; tail -1) | jq '.time'
grep
-hR
<correlationID> | jq
-c
-R
'fromjson?'
| jq
-C
-s
'sort_by(.time)'
| less
-R
```
### Parsing `
production_json.log` and `
api_json.log`
### Parsing `
gitlab-rails/production_json.log` and `gitlab-rails/
api_json.log`
#### Find all requests with a 5XX status code
...
...
@@ -111,7 +120,7 @@ jq 'select(.queue_duration > 10000)' <FILE>
jq
-s
'map(select(.gitaly_calls != null)) | sort_by(-.gitaly_calls) | limit(10; .[])'
<FILE>
```
### Parsing `production_json.log`
### Parsing `
gitlab-rails/
production_json.log`
#### Print the top three controller methods by request volume and their three longest durations
...
...
@@ -127,7 +136,7 @@ CT: 2435 METHOD: MetricsController#index DURS: 299.29, 284.01, 158.57
CT: 1328 METHOD: Projects::NotesController#index DURS: 403.99, 386.29, 384.39
```
### Parsing `api_json.log`
### Parsing `
gitlab-rails/
api_json.log`
#### Print top three routes with request count and their three longest durations
...
...
@@ -157,6 +166,8 @@ jq --raw-output 'select(.severity == "ERROR") | [.project_path, .message] | @tsv
### Parsing `gitaly/current`
The following examples are useful to
[
troubleshoot Gitaly
](
../gitaly/troubleshooting.md
)
.
#### Find all Gitaly requests sent from web UI
```
shell
...
...
@@ -197,7 +208,7 @@ jq --raw-output --slurp '
.[2]."grpc.time_ms",
.[0]."grpc.request.glProjectPath"
]
| @sh'
/var/log/gitlab/gitaly/
current
\
| @sh'
current
\
|
awk
'BEGIN { printf "%7s %10s %10s %10s\t%s\n", "CT", "MAX DURS", "", "", "PROJECT" }
{ printf "%7u %7u ms, %7u ms, %7u ms\t%s\n", $1, $2, $3, $4, $5 }'
```
...
...
@@ -215,12 +226,12 @@ jq --raw-output --slurp '
#### Find all projects affected by a fatal Git problem
```
shell
grep
"fatal: "
/var/log/gitlab/gitaly/
current |
\
grep
"fatal: "
current |
\
jq
'."grpc.request.glProjectPath"'
|
\
sort
|
uniq
```
### Parsing `gitlab-shell.log`
### Parsing `gitlab-shell
/gitlab-shell
.log`
For investigating Git calls via SSH, from
[
GitLab 12.10
](
https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/367
)
.
...
...
@@ -238,7 +249,7 @@ jq --raw-output --slurp '
| sort_by(-length)
| limit(20; .[])
| "count: \(length)\tuser: \(.[0].username)\tproject: \(.[0].gl_project_path)" '
\
/var/log/gitlab/gitlab-shell/
gitlab-shell.log
gitlab-shell.log
```
Find the top 20 calls by project, user, and command:
...
...
@@ -256,5 +267,5 @@ jq --raw-output --slurp '
| sort_by(-length)
| limit(20; .[])
| "count: \(length)\tcommand: \(.[0].command)\tuser: \(.[0].username)\tproject: \(.[0].gl_project_path)" '
\
/var/log/gitlab/gitlab-shell/
gitlab-shell.log
gitlab-shell.log
```
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment