@@ -118,7 +118,7 @@ The connection settings match those provided by [Fog](https://github.com/fog), a
...
@@ -118,7 +118,7 @@ The connection settings match those provided by [Fog](https://github.com/fog), a
| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true |
| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true |
| `region` | AWS region | us-east-1 |
| `region` | AWS region | us-east-1 |
| `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com |
| `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com |
| `endpoint` | Can be used when configuring an S3 compatible service such as [Minio](https://www.minio.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) |
| `endpoint` | Can be used when configuring an S3 compatible service such as [MinIO](https://www.minio.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) |
| `path_style` | Set to true to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as false for AWS S3 | false |
| `path_style` | Set to true to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as false for AWS S3 | false |
| `use_iam_profile` | Set to true to use IAM profile instead of access keys | false
| `use_iam_profile` | Set to true to use IAM profile instead of access keys | false
B1[Check Admin Area > Integrations<br>to ensure the settings are correct]
B1[Check Admin Area > Integrations<br>to ensure the settings are correct]
B2[Perform a search via<br>the rails console]
B2[Perform a search via<br>the rails console]
B3[If all settings are correct<br>and it still doesn't show ElasticSearch<br>doing the searches, escalate<br>to GitLab support.]
B3[If all settings are correct<br>and it still doesn't show Elasticsearch<br>doing the searches, escalate<br>to GitLab support.]
B4[Perform<br>the same search via the<br>ElasticSearch API]
B4[Perform<br>the same search via the<br>Elasticsearch API]
B5{Are the results<br>the same?}
B5{Are the results<br>the same?}
B6[This means it is working as intended.<br>Speak with GitLab support<br>to confirm if the issue lies with<br>the filters.]
B6[This means it is working as intended.<br>Speak with GitLab support<br>to confirm if the issue lies with<br>the filters.]
B7[Check the index status of the project<br>containing the missing search<br>results.]
B7[Check the index status of the project<br>containing the missing search<br>results.]
...
@@ -55,7 +55,7 @@ graph TD;
...
@@ -55,7 +55,7 @@ graph TD;
### Indexing workflow
### Indexing workflow
The following workflow is for ElasticSearch indexing issues:
The following workflow is for Elasticsearch indexing issues:
```mermaid
```mermaid
graph TD;
graph TD;
...
@@ -67,7 +67,7 @@ graph TD;
...
@@ -67,7 +67,7 @@ graph TD;
C --> |No| C6
C --> |No| C6
C6 --> |No| C10
C6 --> |No| C10
C7 --> |GitLab| C8
C7 --> |GitLab| C8
C7 --> |ElasticSearch| C9
C7 --> |Elasticsearch| C9
C6 --> |Yes| C7
C6 --> |Yes| C7
C10 --> |No| C12
C10 --> |No| C12
C10 --> |Yes| C11
C10 --> |Yes| C11
...
@@ -76,27 +76,27 @@ graph TD;
...
@@ -76,27 +76,27 @@ graph TD;
C14 --> |Yes| C15
C14 --> |Yes| C15
C14 --> |No| C16
C14 --> |No| C16
C{Is the problem with<br>creating an empty<br>index?}
C{Is the problem with<br>creating an empty<br>index?}
C1{Does the gitlab-production<br>index exist on the<br>ElasticSearch instance?}
C1{Does the gitlab-production<br>index exist on the<br>Elasticsearch instance?}
C2(Try to manually<br>delete the index on the<br>ElasticSearch instance and<br>retry creating an empty index.)
C2(Try to manually<br>delete the index on the<br>Elasticsearch instance and<br>retry creating an empty index.)
C3{Can indices be made<br>manually on the ElasticSearch<br>instance?}
C3{Can indices be made<br>manually on the Elasticsearch<br>instance?}
C4(Retry the creation of an empty index)
C4(Retry the creation of an empty index)
C5(It is best to speak with an<br>ElasticSearch admin concerning the<br>instance's inability to create indices.)
C5(It is best to speak with an<br>Elasticsearch admin concerning the<br>instance's inability to create indices.)
C6{Is the indexer presenting<br>errors during indexing?}
C6{Is the indexer presenting<br>errors during indexing?}
C7{Is the error a GitLab<br>error or an ElasticSearch<br>error?}
C7{Is the error a GitLab<br>error or an Elasticsearch<br>error?}
C8[Escalate to<br>GitLab support]
C8[Escalate to<br>GitLab support]
C9[You will want<br>to speak with an<br>ElasticSearch admin.]
C9[You will want<br>to speak with an<br>Elasticsearch admin.]
C10{Does the index status<br>show 100%?}
C10{Does the index status<br>show 100%?}
C11[Escalate to<br>GitLab support]
C11[Escalate to<br>GitLab support]
C12{Does re-indexing the project<br> present any GitLab errors?}
C12{Does re-indexing the project<br> present any GitLab errors?}
C13[Rectify the GitLab errors and<br>restart troubleshooting, or<br>escalate to GitLab support.]
C13[Rectify the GitLab errors and<br>restart troubleshooting, or<br>escalate to GitLab support.]
C14{Does re-indexing the project<br>present errors on the <br>ElasticSearch instance?}
C14{Does re-indexing the project<br>present errors on the <br>Elasticsearch instance?}
C15[It would be best<br>to speak with an<br>ElasticSearch admin.]
C15[It would be best<br>to speak with an<br>Elasticsearch admin.]
C16[This is likely a bug/issue<br>in GitLab and will require<br>deeper investigation. Escalate<br>to GitLab support.]
C16[This is likely a bug/issue<br>in GitLab and will require<br>deeper investigation. Escalate<br>to GitLab support.]
```
```
### Integration workflow
### Integration workflow
The following workflow is for ElasticSearch integration issues:
The following workflow is for Elasticsearch integration issues:
```mermaid
```mermaid
graph TD;
graph TD;
...
@@ -107,7 +107,7 @@ graph TD;
...
@@ -107,7 +107,7 @@ graph TD;
D4 --> |No| D5
D4 --> |No| D5
D4 --> |Yes| D6
D4 --> |Yes| D6
D{Is the error concerning<br>the beta indexer?}
D{Is the error concerning<br>the beta indexer?}
D1[It would be best<br>to speak with an<br>ElasticSearch admin.]
D1[It would be best<br>to speak with an<br>Elasticsearch admin.]
D2{Is the ICU development<br>package installed?}
D2{Is the ICU development<br>package installed?}
D3>This package is required.<br>Install the package<br>and retry.]
D3>This package is required.<br>Install the package<br>and retry.]
D4{Is the error stemming<br>from the indexer?}
D4{Is the error stemming<br>from the indexer?}
...
@@ -117,7 +117,7 @@ graph TD;
...
@@ -117,7 +117,7 @@ graph TD;
### Performance workflow
### Performance workflow
The following workflow is for ElasticSearch performance issues:
The following workflow is for Elasticsearch performance issues:
```mermaid
```mermaid
graph TD;
graph TD;
...
@@ -128,19 +128,19 @@ graph TD;
...
@@ -128,19 +128,19 @@ graph TD;
F4 --> F5
F4 --> F5
F5 --> |No| F6
F5 --> |No| F6
F5 --> |Yes| F7
F5 --> |Yes| F7
F{Is the ElasticSearch instance<br>running on the same server<br>as the GitLab instance?}
F{Is the Elasticsearch instance<br>running on the same server<br>as the GitLab instance?}
F1(This is not advised and will cause issues.<br>We recommend moving the ElasticSearch<br>instance to a different server.)
F1(This is not advised and will cause issues.<br>We recommend moving the Elasticsearch<br>instance to a different server.)
F2{Does the ElasticSearch<br>server have at least 8<br>GB of RAM and 2 CPU<br>cores?}
F2{Does the Elasticsearch<br>server have at least 8<br>GB of RAM and 2 CPU<br>cores?}
F3(According to ElasticSearch, a non-prod<br>server needs these as a base requirement.<br>Production often requires more. We recommend<br>you increase the server specifications.)
F3(According to Elasticsearch, a non-prod<br>server needs these as a base requirement.<br>Production often requires more. We recommend<br>you increase the server specifications.)
F4(Obtain the <br>cluster health information)
F4(Obtain the <br>cluster health information)
F5(Does it show the<br>status as green?)
F5(Does it show the<br>status as green?)
F6(We recommend you speak with<br>an ElasticSearch admin<br>about implementing sharding.)
F6(We recommend you speak with<br>an Elasticsearch admin<br>about implementing sharding.)
F7(Escalate to<br>GitLab support.)
F7(Escalate to<br>GitLab support.)
```
```
## Troubleshooting walkthrough
## Troubleshooting walkthrough
Most ElasticSearch troubleshooting can be broken down into 4 categories:
Most Elasticsearch troubleshooting can be broken down into 4 categories:
If all the settings look correct and it is still not using ElasticSearch for the search function, it is best to escalate to GitLab support. This could be a bug/issue.
If all the settings look correct and it is still not using Elasticsearch for the search function, it is best to escalate to GitLab support. This could be a bug/issue.
Moving past that, it is best to attempt the same search using the [ElasticSearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) and compare the results from what you see in GitLab.
Moving past that, it is best to attempt the same search using the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) and compare the results from what you see in GitLab.
If the results:
If the results:
- Sync up, then there is not a technical "issue" per se. Instead, it might be a problem
- Sync up, then there is not a technical "issue" per se. Instead, it might be a problem
with the ElasticSearch filters we are using. This can be complicated, so it is best to
with the Elasticsearch filters we are using. This can be complicated, so it is best to
escalate to GitLab support to check these and guide you on the potential on whether or
escalate to GitLab support to check these and guide you on the potential on whether or
not a feature request is needed.
not a feature request is needed.
- Do not match up, this indicates a problem with the documents generated from the
- Do not match up, this indicates a problem with the documents generated from the
...
@@ -197,20 +197,20 @@ If the results:
...
@@ -197,20 +197,20 @@ If the results:
### Troubleshooting indexing
### Troubleshooting indexing
Troubleshooting indexing issues can be tricky. It can pretty quickly go to either GitLab
Troubleshooting indexing issues can be tricky. It can pretty quickly go to either GitLab
support or your ElasticSearch admin.
support or your Elasticsearch admin.
The best place to start is to determine if the issue is with creating an empty index.
The best place to start is to determine if the issue is with creating an empty index.
If it is, check on the ElasticSearch side to determine if the `gitlab-production` (the
If it is, check on the Elasticsearch side to determine if the `gitlab-production` (the
name for the GitLab index) exists. If it exists, manually delete it on the ElasticSearch
name for the GitLab index) exists. If it exists, manually delete it on the Elasticsearch
If you still encounter issues, try creating an index manually on the ElasticSearch
If you still encounter issues, try creating an index manually on the Elasticsearch
instance. The details of the index aren't important here, as we want to test if indices
instance. The details of the index aren't important here, as we want to test if indices
can be made. If the indices:
can be made. If the indices:
- Cannot be made, speak with your ElasticSearch admin.
- Cannot be made, speak with your Elasticsearch admin.
- Can be made, Escalate this to GitLab support.
- Can be made, Escalate this to GitLab support.
If the issue is not with creating an empty index, the next step is to check for errors
If the issue is not with creating an empty index, the next step is to check for errors
...
@@ -218,7 +218,7 @@ during the indexing of projects. If errors do occur, they will either stem from
...
@@ -218,7 +218,7 @@ during the indexing of projects. If errors do occur, they will either stem from
- On the GitLab side. You need to rectify those. If they are not
- On the GitLab side. You need to rectify those. If they are not
something you are familiar with, contact GitLab support for guidance.
something you are familiar with, contact GitLab support for guidance.
- Within the ElasticSearch instance itself. See if the error is [documented and has a fix](../../integration/elasticsearch.md#troubleshooting). If not, speak with your ElasticSearch admin.
- Within the Elasticsearch instance itself. See if the error is [documented and has a fix](../../integration/elasticsearch.md#troubleshooting). If not, speak with your Elasticsearch admin.
If the indexing process does not present errors, you will want to check the status of the indexed projects. You can do this via the following rake tasks:
If the indexing process does not present errors, you will want to check the status of the indexed projects. You can do this via the following rake tasks:
...
@@ -235,8 +235,8 @@ If:
...
@@ -235,8 +235,8 @@ If:
If reindexing the project shows:
If reindexing the project shows:
- Errors on the GitLab side, escalate those to GitLab support.
- Errors on the GitLab side, escalate those to GitLab support.
- ElasticSearch errors or doesn't present any errors at all, reach out to your
- Elasticsearch errors or doesn't present any errors at all, reach out to your
ElasticSearch admin to check the instance.
Elasticsearch admin to check the instance.
### Troubleshooting integration
### Troubleshooting integration
...
@@ -246,7 +246,7 @@ much to "integrate" here.
...
@@ -246,7 +246,7 @@ much to "integrate" here.
If the issue is:
If the issue is:
- Not concerning the beta indexer, it is almost always an
- Not concerning the beta indexer, it is almost always an
ElasticSearch-side issue. This means you should reach out to your ElasticSearch admin
Elasticsearch-side issue. This means you should reach out to your Elasticsearch admin
regarding the error(s) you are seeing. If you are unsure here, it never hurts to reach
regarding the error(s) you are seeing. If you are unsure here, it never hurts to reach
out to GitLab support.
out to GitLab support.
- With the beta indexer, check if the ICU development package is installed.
- With the beta indexer, check if the ICU development package is installed.
...
@@ -260,48 +260,48 @@ Beyond that, you will want to review the error. If it is:
...
@@ -260,48 +260,48 @@ Beyond that, you will want to review the error. If it is:
### Troubleshooting performance
### Troubleshooting performance
Troubleshooting performance can be difficult on ElasticSearch. There is a ton of tuning
Troubleshooting performance can be difficult on Elasticsearch. There is a ton of tuning
that *can* be done, but the majority of this falls on shoulders of a skilled
that *can* be done, but the majority of this falls on shoulders of a skilled
ElasticSearch administrator.
Elasticsearch administrator.
Generally speaking, ensure:
Generally speaking, ensure:
- The ElasticSearch server **is not** running on the same node as GitLab.
- The Elasticsearch server **is not** running on the same node as GitLab.
- The ElasticSearch server have enough RAM and CPU cores.
- The Elasticsearch server have enough RAM and CPU cores.
- That sharding **is** being used.
- That sharding **is** being used.
Going into some more detail here, if ElasticSearch is running on the same server as GitLab, resource contention is **very** likely to occur. Ideally, ElasticSearch, which requires ample resources, should be running on its own server (maybe coupled with logstash and kibana).
Going into some more detail here, if Elasticsearch is running on the same server as GitLab, resource contention is **very** likely to occur. Ideally, Elasticsearch, which requires ample resources, should be running on its own server (maybe coupled with logstash and kibana).
When it comes to ElasticSearch, RAM is the key resource. ElasticSearch themselves recommend:
When it comes to Elasticsearch, RAM is the key resource. Elasticsearch themselves recommend:
-**At least** 8 GB of RAM for a non-production instance.
-**At least** 8 GB of RAM for a non-production instance.
-**At least** 16 GB of RAM for a production instance.
-**At least** 16 GB of RAM for a production instance.
- Ideally, 64 GB of RAM.
- Ideally, 64 GB of RAM.
For CPU, ElasticSearch recommends at least 2 CPU cores, but ElasticSearch states common
For CPU, Elasticsearch recommends at least 2 CPU cores, but Elasticsearch states common
setups use up to 8 cores. For more details on server specs, check out
setups use up to 8 cores. For more details on server specs, check out