Commit 77350a46 authored by Russell Dickenson's avatar Russell Dickenson

Merge branch '38283-doc-update-es-indexer' into 'master'

Doc: Mark Elasticsearch Go indexer as stable

See merge request gitlab-org/gitlab!21291
parents 268d1ff6 0f87bdcb
...@@ -47,10 +47,11 @@ But since 11.8 the indexer uses Gitaly for data access as well. NFS can still ...@@ -47,10 +47,11 @@ But since 11.8 the indexer uses Gitaly for data access as well. NFS can still
be leveraged for redudancy on block level of the Git data. But only has to be leveraged for redudancy on block level of the Git data. But only has to
be mounted on the Gitaly server. be mounted on the Gitaly server.
Starting with GitLab 11.8, it is possible to use Elasticsearch in conjunction with From GitLab v11.8 to v12.2, it is possible to use Elasticsearch in conjunction with
a Gitaly setup that isn't utilising NFS. In order to use Elasticsearch in this a Gitaly setup that isn't utilising NFS. In order to use Elasticsearch in this
scenario, the [new repository indexer](../../integration/elasticsearch.md#elasticsearch-repository-indexer-beta) scenario, the [new repository indexer](../../integration/elasticsearch.md#elasticsearch-repository-indexer)
needs to be enabled in your GitLab configuration. needs to be enabled in your GitLab configuration. [Since GitLab v12.3](https://gitlab.com/gitlab-org/gitlab/issues/6481),
the new indexer becomes the default and no configuration is required.
NOTE: **Note:** While Gitaly can be used as a replacement for NFS, it's not recommended NOTE: **Note:** While Gitaly can be used as a replacement for NFS, it's not recommended
to use EFS as it may impact GitLab's performance. Review the [relevant documentation](../high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs) to use EFS as it may impact GitLab's performance. Review the [relevant documentation](../high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs)
......
...@@ -106,7 +106,7 @@ graph TD; ...@@ -106,7 +106,7 @@ graph TD;
D2 --> |Yes| D4 D2 --> |Yes| D4
D4 --> |No| D5 D4 --> |No| D5
D4 --> |Yes| D6 D4 --> |Yes| D6
D{Is the error concerning<br>the beta indexer?} D{Is the error concerning<br>the Go indexer?}
D1[It would be best<br>to speak with an<br>Elasticsearch admin.] D1[It would be best<br>to speak with an<br>Elasticsearch admin.]
D2{Is the ICU development<br>package installed?} D2{Is the ICU development<br>package installed?}
D3>This package is required.<br>Install the package<br>and retry.] D3>This package is required.<br>Install the package<br>and retry.]
...@@ -245,12 +245,13 @@ much to "integrate" here. ...@@ -245,12 +245,13 @@ much to "integrate" here.
If the issue is: If the issue is:
- Not concerning the beta indexer, it is almost always an - With the Go indexer, check if the ICU development package is installed.
This is a required package so make sure you install it.
Go indexer was a beta indexer which can be optionally turned on/off, but in 12.3 it reached stable status and is now the default.
- Not concerning the Go indexer, it is almost always an
Elasticsearch-side issue. This means you should reach out to your Elasticsearch admin Elasticsearch-side issue. This means you should reach out to your Elasticsearch admin
regarding the error(s) you are seeing. If you are unsure here, it never hurts to reach regarding the error(s) you are seeing. If you are unsure here, it never hurts to reach
out to GitLab support. out to GitLab support.
- With the beta indexer, check if the ICU development package is installed.
This is a required package so make sure you install it.
Beyond that, you will want to review the error. If it is: Beyond that, you will want to review the error. If it is:
......
...@@ -37,20 +37,6 @@ brew install elasticsearch@5.6 ...@@ -37,20 +37,6 @@ brew install elasticsearch@5.6
There is no need to install any plugins There is no need to install any plugins
## New repo indexer (beta)
If you're interested on working with the new beta repo indexer, all you need to do is:
```sh
git clone git@gitlab.com:gitlab-org/gitlab-elasticsearch-indexer.git
make
make install
```
this adds `gitlab-elasticsearch-indexer` to `$GOPATH/bin`, please make sure that is in your `$PATH`. After that GitLab will find it and you'll be able to enable it in the admin settings area.
**note:** `make` will not recompile the executable unless you do `make clean` beforehand
## Helpful rake tasks ## Helpful rake tasks
- `gitlab:elastic:test:index_size`: Tells you how much space the current index is using, as well as how many documents are in the index. - `gitlab:elastic:test:index_size`: Tells you how much space the current index is using, as well as how many documents are in the index.
......
...@@ -40,14 +40,13 @@ Once the data is added to the database or repository and [Elasticsearch is ...@@ -40,14 +40,13 @@ Once the data is added to the database or repository and [Elasticsearch is
enabled in the admin area](#enabling-elasticsearch) the search index will be enabled in the admin area](#enabling-elasticsearch) the search index will be
updated automatically. updated automatically.
## Elasticsearch repository indexer (beta) ## Elasticsearch repository indexer
In order to improve Elasticsearch indexing performance, GitLab has made available a [new indexer written in Go](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). For indexing Git repository data, GitLab uses an [indexer written in Go](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer).
This will replace the included Ruby indexer in the future but should be considered beta software for now, so there may be some bugs.
The Elasticsearch Go indexer is included in Omnibus for GitLab 11.8 and newer. The Go indexer was included in Omnibus GitLab 11.8 as an optional replacement to a
Ruby-based indexer. [Since GitLab v12.3](https://gitlab.com/gitlab-org/gitlab/issues/6481),
To use the new Elasticsearch indexer included in Omnibus, check the box "Use the new repository indexer (beta)" when [enabling the Elasticsearch integration](#enabling-elasticsearch). all indexing is done by the Go indexer, and the Ruby indexer is removed.
If you would like to use the Elasticsearch Go indexer with a source installation or an older version of GitLab, please follow the instructions below. If you would like to use the Elasticsearch Go indexer with a source installation or an older version of GitLab, please follow the instructions below.
...@@ -139,7 +138,6 @@ The following Elasticsearch settings are available: ...@@ -139,7 +138,6 @@ The following Elasticsearch settings are available:
| Parameter | Description | | Parameter | Description |
| ----------------------------------------------------- | ----------- | | ----------------------------------------------------- | ----------- |
| `Elasticsearch indexing` | Enables/disables Elasticsearch indexing. You may want to enable indexing but disable search in order to give the index time to be fully completed, for example. Also, keep in mind that this option doesn't have any impact on existing data, this only enables/disables background indexer which tracks data changes. So by enabling this you will not get your existing data indexed, use special rake task for that as explained in [Adding GitLab's data to the Elasticsearch index](#adding-gitlabs-data-to-the-elasticsearch-index). | | `Elasticsearch indexing` | Enables/disables Elasticsearch indexing. You may want to enable indexing but disable search in order to give the index time to be fully completed, for example. Also, keep in mind that this option doesn't have any impact on existing data, this only enables/disables background indexer which tracks data changes. So by enabling this you will not get your existing data indexed, use special rake task for that as explained in [Adding GitLab's data to the Elasticsearch index](#adding-gitlabs-data-to-the-elasticsearch-index). |
| `Use the new repository indexer (beta)` | Perform repository indexing using [GitLab Elasticsearch Indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). |
| `Search with Elasticsearch enabled` | Enables/disables using Elasticsearch in search. | | `Search with Elasticsearch enabled` | Enables/disables using Elasticsearch in search. |
| `URL` | The URL to use for connecting to Elasticsearch. Use a comma-separated list to support clustering (e.g., `http://host1, https://host2:9200`). If your Elasticsearch instance is password protected, pass the `username:password` in the URL (e.g., `http://<username>:<password>@<elastic_host>:9200/`). | | `URL` | The URL to use for connecting to Elasticsearch. Use a comma-separated list to support clustering (e.g., `http://host1, https://host2:9200`). If your Elasticsearch instance is password protected, pass the `username:password` in the URL (e.g., `http://<username>:<password>@<elastic_host>:9200/`). |
| `Number of Elasticsearch shards` | Elasticsearch indexes are split into multiple shards for performance reasons. In general, larger indexes need to have more shards. Changes to this value do not take effect until the index is recreated. You can read more about tradeoffs in the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#create-index-settings) | | `Number of Elasticsearch shards` | Elasticsearch indexes are split into multiple shards for performance reasons. In general, larger indexes need to have more shards. Changes to this value do not take effect until the index is recreated. You can read more about tradeoffs in the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#create-index-settings) |
......
...@@ -94,11 +94,9 @@ sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_PAGES_VERSION) ...@@ -94,11 +94,9 @@ sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_PAGES_VERSION)
sudo -u git -H make sudo -u git -H make
``` ```
### 8. Install/Update `gitlab-elasticsearch-indexer` (optional) **(STARTER ONLY)** ### 8. Install/Update `gitlab-elasticsearch-indexer` **(STARTER ONLY)**
If you're interested in using GitLab's new [Elasticsearch repository indexer](../integration/elasticsearch.md#elasticsearch-repository-indexer-beta) (currently in beta) Please follow the [install instruction](../integration/elasticsearch.md#installation).
please follow the instructions on the document linked above and enable the
indexer usage in the GitLab admin settings.
### 9. Start application ### 9. Start application
......
...@@ -77,11 +77,9 @@ sudo -u git -H bundle exec rake yarn:install gitlab:assets:clean gitlab:assets:c ...@@ -77,11 +77,9 @@ sudo -u git -H bundle exec rake yarn:install gitlab:assets:clean gitlab:assets:c
sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production
``` ```
### 4. Install `gitlab-elasticsearch-indexer` (optional) **(STARTER ONLY)** ### 4. Install `gitlab-elasticsearch-indexer` **(STARTER ONLY)**
If you're interested in using GitLab's new [Elasticsearch repository indexer](../integration/elasticsearch.md) Please follow the [install instruction](../integration/elasticsearch.md#installation).
(currently in beta) please follow the instructions on the
document linked above and enable the indexer usage in the GitLab admin settings.
### 5. Start application ### 5. Start application
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment