Merge branch '328-other-ee' into 'master'

Port of EE "Elasticsearch versioned schema for other ActiveRecord models"

See merge request gitlab-org/gitlab-ce!31660

config/initializers/zz_metrics.rb

@@ -100,11 +100,7 @@ def instrument_classes(instrumentation)
     instrumentation.instrument_instance_methods(Gitlab::Elastic::SnippetSearchResults)
     instrumentation.instrument_methods(Gitlab::Elastic::Helper)
 
-    instrumentation.instrument_instance_methods(Elastic::ApplicationSearch)
-    instrumentation.instrument_instance_methods(Elastic::IssuesSearch)
-    instrumentation.instrument_instance_methods(Elastic::MergeRequestsSearch)
-    instrumentation.instrument_instance_methods(Elastic::MilestonesSearch)
-    instrumentation.instrument_instance_methods(Elastic::NotesSearch)
+    instrumentation.instrument_instance_methods(Elastic::ApplicationVersionedSearch)
     instrumentation.instrument_instance_methods(Elastic::ProjectsSearch)
     instrumentation.instrument_instance_methods(Elastic::RepositoriesSearch)
     instrumentation.instrument_instance_methods(Elastic::SnippetsSearch)

doc/development/elasticsearch.md

@@ -148,26 +148,49 @@ Uses an [Edge NGram token filter](https://www.elastic.co/guide/en/elasticsearch/
 - Searches can have their own analyzers. Remember to check when editing analyzers
 - `Character` filters (as opposed to token filters) always replace the original character, so they're not a good choice as they can hinder exact searches
 
-## Architecture
+## Zero downtime reindexing with multiple indices
 
-GitLab uses `elasticsearch-rails` for handling communication with Elasticsearch server. However, in order to achieve zero-downtime deployment during schema changes, an extra abstraction layer is built to allow:
+Currently GitLab can only handle a single version of setting. Any setting/schema changes would require reindexing everything from scratch. Since reindexing can take a long time, this can cause search functionality downtime.
 
-* Indexing (writes) to multiple indexes, with different mappings
-* Switching to different index for searches (reads) on the fly
+To avoid downtime, GitLab is working to support multiple indices that
+can function at the same time. Whenever the schema changes, the admin
+will be able to create a new index and reindex to it, while searches
+continue to go to the older, stable index. Any data updates will be
+forwarded to both indices. Once the new index is ready, an admin can
+mark it active, which will direct all searches to it, and remove the old
+index.
 
-Currently we are on the process of migrating models to this new design (e.g. `Snippet`), and it is hardwired to work with a single version for now.
+This is also helpful for migrating to new servers, e.g. moving to/from AWS.
 
-Traditionally, `elasticsearch-rails` provides class and instance level `__elasticsearch__` proxy methods. If you call `Issue.__elasticsearch__`, you will get an instance of `Elasticsearch::Model::Proxy::ClassMethodsProxy`, and if you call `Issue.first.__elasticsearch__`, you will get an instance of `Elasticsearch::Model::Proxy::InstanceMethodsProxy`. These proxy objects would talk to Elasticsearch server directly.
+Currently we are on the process of migrating to this new design. Everything is hardwired to work with one single version for now.
 
-In the new design, `__elasticsearch__` instead represents one extra layer of proxy. It would keep multiple versions of the actual proxy objects, and it would forward read and write calls to the proxy of the intended version.
+### Architecture
 
-The `elasticsearch-rails`'s way of specifying each model's mappings and other settings is to create a module for the model to include. However in the new design, each model would have its own corresponding subclassed proxy object, where the settings reside in. For example, snippet related setting in the past reside in `SnippetsSearch` module, but in the new design would reside in `SnippetClassProxy` (which is a subclass of `Elasticsearch::Model::Proxy::ClassMethodsProxy`). This reduces namespace pollution in model classes.
+The traditional setup, provided by `elasticsearch-rails`, is to communicate through its internal proxy classes. Developers would write model-specific logic in a module for the model to include in (e.g. `SnippetsSearch`). The `__elasticsearch__` methods would return a proxy object, e.g.:
+
+- `Issue.__elasticsearch__` returns an instance of `Elasticsearch::Model::Proxy::ClassMethodsProxy`
+- `Issue.first.__elasticsearch__` returns an instance of `Elasticsearch::Model::Proxy::InstanceMethodsProxy`.
+
+These proxy objects would talk to Elasticsearch server directly (see top half of the diagram).
+
+![Elasticsearch Architecture](img/elasticsearch_architecture.svg)
+
+In the planned new design, each model would have a pair of corresponding subclassed proxy objects, in which model-specific logic is located. For example, `Snippet` would have `SnippetClassProxy` and `SnippetInstanceProxy` (being subclass of `Elasticsearch::Model::Proxy::ClassMethodsProxy` and `Elasticsearch::Model::Proxy::InstanceMethodsProxy`, respectively).
+
+`__elasticsearch__` would represent another layer of proxy object, keeping track of multiple actual proxy objects. It would forward method calls to the appropriate index. For example:
+
+- `model.__elasticsearch__.search` would be forwarded to the one stable index, since it is a read operation.
+- `model.__elasticsearch__.update_document` would be forwarded to all indices, to keep all indices up-to-date.
 
 The global configurations per version are now in the `Elastic::(Version)::Config` class. You can change mappings there.
 
 ### Creating new version of schema
 
-Currently GitLab would still work with a single version of setting. Once it is implemented, multiple versions of setting can exists in different folders (e.g. `ee/lib/elastic/v12p1` and `ee/lib/elastic/v12p3`). To keep a continuous git history, the latest version lives under the `/latest` folder, but is aliased as the latest version.
+NOTE: **Note:** this is not applicable yet as multiple indices functionality is not fully implemented.
+
+Folders like `ee/lib/elastic/v12p1` contain snapshots of search logic from different versions. To keep a continuous git history, the latest version lives under `ee/lib/elastic/latest`, but its classes are aliased under an actual version (e.g. `ee/lib/elastic/v12p3`). When referencing these classes, never use the `Latest` namespace directly, but use the actual version (e.g. `V12p3`).
+
+The version name basically follows GitLab's release version. If setting is changed in 12.3, we will create a new namespace called `V12p3` (p stands for "point"). Raise an issue if there is a need to name a version differently.
 
 If the current version is `v12p1`, and we need to create a new version for `v12p3`, the steps are as follows:
 
@@ -176,7 +199,7 @@ If the current version is `v12p1`, and we need to create a new version for `v12p
 1. Delete `v12p1` folder
 1. Copy the entire folder of `latest` as `v12p1`
 1. Change the namespace for files under `v12p1` folder from `Latest` to `V12p1`
-1. Make changes to `Latest` as needed
+1. Make changes to files under the `latest` folder as needed
 
 ## Troubleshooting