Skip to content

G
gitlab-ce
Commit 8438243e authored by Stan Hu's avatar Stan Hu
Browse files
Options

Merge branch '328-other' into 'master' 

Elasticsearch versioned schema for other ActiveRecord models

See merge request gitlab-org/gitlab-ee!14522
parents f49fe157 1c53564d
Hide whitespace changes
Inline Side-by-side
Showing with 415 additions and 603 deletions
config/initializers/zz_metrics.rb
View file @ 8438243e
......@@ -100,11 +100,7 @@ def instrument_classes(instrumentation)
instrumentation.instrument_instance_methods(Gitlab::Elastic::SnippetSearchResults)
instrumentation.instrument_methods(Gitlab::Elastic::Helper)
instrumentation.instrument_instance_methods(Elastic::ApplicationSearch)
instrumentation.instrument_instance_methods(Elastic::IssuesSearch)
instrumentation.instrument_instance_methods(Elastic::MergeRequestsSearch)
instrumentation.instrument_instance_methods(Elastic::MilestonesSearch)
instrumentation.instrument_instance_methods(Elastic::NotesSearch)
instrumentation.instrument_instance_methods(Elastic::ApplicationVersionedSearch)
instrumentation.instrument_instance_methods(Elastic::ProjectsSearch)
instrumentation.instrument_instance_methods(Elastic::RepositoriesSearch)
instrumentation.instrument_instance_methods(Elastic::SnippetsSearch)
......
doc/development/elasticsearch.md
View file @ 8438243e
......@@ -148,26 +148,49 @@ Uses an [Edge NGram token filter](https://www.elastic.co/guide/en/elasticsearch/
- Searches can have their own analyzers. Remember to check when editing analyzers
- `Character` filters (as opposed to token filters) always replace the original character, so they're not a good choice as they can hinder exact searches
## Architecture
## Zero downtime reindexing with multiple indices
GitLab uses `elasticsearch-rails` for handling communication with Elasticsearch server. However, in order to achieve zero-downtime deployment during schema changes, an extra abstraction layer is built to allow:
Currently GitLab can only handle a single version of setting. Any setting/schema changes would require reindexing everything from scratch. Since reindexing can take a long time, this can cause search functionality downtime.
* Indexing (writes) to multiple indexes, with different mappings
* Switching to different index for searches (reads) on the fly
To avoid downtime, GitLab is working to support multiple indices that
can function at the same time. Whenever the schema changes, the admin
will be able to create a new index and reindex to it, while searches
continue to go to the older, stable index. Any data updates will be
forwarded to both indices. Once the new index is ready, an admin can
mark it active, which will direct all searches to it, and remove the old
index.
Currently we are on the process of migrating models to this new design (e.g. `Snippet`), and it is hardwired to work with a single version for now.
This is also helpful for migrating to new servers, e.g. moving to/from AWS.
Traditionally, `elasticsearch-rails` provides class and instance level `__elasticsearch__` proxy methods. If you call `Issue.__elasticsearch__`, you will get an instance of `Elasticsearch::Model::Proxy::ClassMethodsProxy`, and if you call `Issue.first.__elasticsearch__`, you will get an instance of `Elasticsearch::Model::Proxy::InstanceMethodsProxy`. These proxy objects would talk to Elasticsearch server directly.
Currently we are on the process of migrating to this new design. Everything is hardwired to work with one single version for now.
In the new design, `__elasticsearch__` instead represents one extra layer of proxy. It would keep multiple versions of the actual proxy objects, and it would forward read and write calls to the proxy of the intended version.
### Architecture
The `elasticsearch-rails`'s way of specifying each model's mappings and other settings is to create a module for the model to include. However in the new design, each model would have its own corresponding subclassed proxy object, where the settings reside in. For example, snippet related setting in the past reside in `SnippetsSearch` module, but in the new design would reside in `SnippetClassProxy` (which is a subclass of `Elasticsearch::Model::Proxy::ClassMethodsProxy`). This reduces namespace pollution in model classes.
The traditional setup, provided by `elasticsearch-rails`, is to communicate through its internal proxy classes. Developers would write model-specific logic in a module for the model to include in (e.g. `SnippetsSearch`). The `__elasticsearch__` methods would return a proxy object, e.g.:
- `Issue.__elasticsearch__` returns an instance of `Elasticsearch::Model::Proxy::ClassMethodsProxy`
- `Issue.first.__elasticsearch__` returns an instance of `Elasticsearch::Model::Proxy::InstanceMethodsProxy`.
These proxy objects would talk to Elasticsearch server directly (see top half of the diagram).
![Elasticsearch Architecture](img/elasticsearch_architecture.svg)
In the planned new design, each model would have a pair of corresponding subclassed proxy objects, in which model-specific logic is located. For example, `Snippet` would have `SnippetClassProxy` and `SnippetInstanceProxy` (being subclass of `Elasticsearch::Model::Proxy::ClassMethodsProxy` and `Elasticsearch::Model::Proxy::InstanceMethodsProxy`, respectively).
`__elasticsearch__` would represent another layer of proxy object, keeping track of multiple actual proxy objects. It would forward method calls to the appropriate index. For example:
- `model.__elasticsearch__.search` would be forwarded to the one stable index, since it is a read operation.
- `model.__elasticsearch__.update_document` would be forwarded to all indices, to keep all indices up-to-date.
The global configurations per version are now in the `Elastic::(Version)::Config` class. You can change mappings there.
### Creating new version of schema
Currently GitLab would still work with a single version of setting. Once it is implemented, multiple versions of setting can exists in different folders (e.g. `ee/lib/elastic/v12p1` and `ee/lib/elastic/v12p3`). To keep a continuous git history, the latest version lives under the `/latest` folder, but is aliased as the latest version.
NOTE: **Note:** this is not applicable yet as multiple indices functionality is not fully implemented.
Folders like `ee/lib/elastic/v12p1` contain snapshots of search logic from different versions. To keep a continuous git history, the latest version lives under `ee/lib/elastic/latest`, but its classes are aliased under an actual version (e.g. `ee/lib/elastic/v12p3`). When referencing these classes, never use the `Latest` namespace directly, but use the actual version (e.g. `V12p3`).
The version name basically follows GitLab's release version. If setting is changed in 12.3, we will create a new namespace called `V12p3` (p stands for "point"). Raise an issue if there is a need to name a version differently.
If the current version is `v12p1`, and we need to create a new version for `v12p3`, the steps are as follows:
......@@ -176,7 +199,7 @@ If the current version is `v12p1`, and we need to create a new version for `v12p
1. Delete `v12p1` folder
1. Copy the entire folder of `latest` as `v12p1`
1. Change the namespace for files under `v12p1` folder from `Latest` to `V12p1`
1. Make changes to `Latest` as needed
1. Make changes to files under the `latest` folder as needed
## Troubleshooting
......
doc/development/img/elasticsearch_architecture.svg 0 → 100644
View file @ 8438243e
<svg version="1.2" width="210mm" height="297mm" viewBox="0 0 21000 29700" preserveAspectRatio="xMidYMid" fill-rule="evenodd" stroke-width="28.222" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg"><defs class="ClipPathGroup"><clipPath id="a" clipPathUnits="userSpaceOnUse"><path d="M0 0h21000v29700H0z"/></clipPath></defs><g class="SlideGroup"><g class="Slide" clip-path="url(#a)"><g class="Page"><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1975 5575h3051v1651H1975z"/><path fill="#FFF" d="M3500 7200H2000V5600h3000v1600H3500z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3500 7200H2000V5600h3000v1600H3500z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="2778" y="6311"><tspan>Snippet</tspan></tspan><tspan class="TextPosition" x="2099" y="6785"><tspan>(ActiveRecord)</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1475 3975h4051v3551H1475z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3500 7500H1500V4000h4000v3500H3500z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="1788" y="5048"><tspan>ApplicationSearch</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M5975 4675h8051v701H5975z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M6000 5350h4000v-650h4000"/></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M5975 5325h8051v1101H5975z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M6000 5350h4000v1050h4000"/></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1075 2875h4951v4951H1075z"/><path fill="none" stroke="#F33" stroke-width="50" d="M3550 7800H1100V2900h4900v4900H3550z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="1946" y="3514"><tspan fill="#C9211E">SnippetsSearch</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1975 12175h3051v1651H1975z"/><path fill="#FFF" d="M3500 13800H2000v-1600h3000v1600H3500z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3500 13800H2000v-1600h3000v1600H3500z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="2778" y="12911"><tspan>Snippet</tspan></tspan><tspan class="TextPosition" x="2099" y="13385"><tspan>(ActiveRecord)</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1075 10775h4951v3251H1075z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3550 14000H1100v-3200h4900v3200H3550z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="2511" y="11461"><tspan>Application</tspan></tspan><tspan class="TextPosition" x="1933" y="11935"><tspan>VersionedSearch</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M3525 13975h4501v7451H3525z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3550 14000v7400h4450"/></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M14008 14075h4985v851h-4985z"/><path fill="none" stroke="#999" stroke-width="50" d="M16500 14900h-2467v-800h4934v800h-2467z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14720" y="14648"><tspan fill="gray">ClassMethodProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13375 13075h6251v2151h-6251z"/><path fill="none" stroke="#F33" stroke-width="50" d="M16500 15200h-3100v-2100h6200v2100h-3100z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="13799" y="13731"><tspan fill="#C9211E">V12p1::SnippetClassProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M7975 14575h3051v1851H7975z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M9500 16400H8000v-1800h3000v1800H9500z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="8277" y="15411"><tspan>MultiVersion-</tspan></tspan><tspan class="TextPosition" x="8429" y="15885"><tspan>ClassProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M14008 16875h4985v851h-4985z"/><path fill="none" stroke="#999" stroke-width="50" d="M16500 17700h-2467v-800h4934v800h-2467z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14720" y="17448"><tspan fill="gray">ClassMethodProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13375 15875h6251v2151h-6251z"/><path fill="none" stroke="#F33" stroke-width="50" d="M16500 18000h-3100v-2100h6200v2100h-3100z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="13799" y="16531"><tspan fill="#C9211E">V12p2::SnippetClassProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10975 14125h2451v1401h-2451z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M11000 15500h1463v-1350h937"/></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10975 15475h2451v1501h-2451z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M11000 15500h1463v1450h937"/></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M3525 13975h4501v1551H3525z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M3550 14000v1500h4450"/></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M14008 19975h4985v851h-4985z"/><path fill="none" stroke="#999" stroke-width="50" d="M16500 20800h-2467v-800h4934v800h-2467z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14445" y="20548"><tspan fill="gray">InstanceMethodProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13375 18975h6251v2151h-6251z"/><path fill="none" stroke="#F33" stroke-width="50" d="M16500 21100h-3100v-2100h6200v2100h-3100z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="13505" y="19631"><tspan fill="#C9211E">V12p1::SnippetInstanceProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M7975 20275h3051v2251H7975z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M9500 22500H8000v-2200h3000v2200H9500z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="8277" y="21311"><tspan>MultiVersion-</tspan></tspan><tspan class="TextPosition" x="8154" y="21785"><tspan>InstanceProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M14008 22775h4985v851h-4985z"/><path fill="none" stroke="#999" stroke-width="50" d="M16500 23600h-2467v-800h4934v800h-2467z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14445" y="23348"><tspan fill="gray">InstanceMethodProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13375 21775h6251v2151h-6251z"/><path fill="none" stroke="#F33" stroke-width="50" d="M16500 23900h-3100v-2100h6200v2100h-3100z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="13505" y="22431"><tspan fill="#C9211E">V12p2::SnippetInstanceProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10975 20025h2451v1401h-2451z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M11000 21400h1463v-1350h937"/></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10975 21375h2451v1501h-2451z"/><path fill="none" stroke="#3465A4" stroke-width="50" d="M11000 21400h1463v1450h937"/></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M900 1600h10697v879H900z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="564" font-weight="400"><tspan class="TextPosition" x="1150" y="2233"><tspan>Standard elasticsearch-rails setup</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M900 9300h7683v879H900z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="564" font-weight="400"><tspan class="TextPosition" x="1150" y="9933"><tspan>GitLab multi-indices setup</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M3400 21300h4821v1197H3400z"/><text class="TextShape"><tspan class="TextParagraph" font-size="388" font-weight="400"><tspan class="TextPosition" x="4250" y="21840"><tspan fill="gray">(instance method)</tspan></tspan><tspan class="TextPosition" x="3651" y="22264"><tspan font-family="Courier" font-size="423">__elasticsearch__</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M3380 15400h4821v1197H3380z"/><text class="TextShape"><tspan class="TextParagraph" font-size="388" font-weight="400"><tspan class="TextPosition" x="4512" y="15940"><tspan fill="gray">(class method)</tspan></tspan><tspan class="TextPosition" x="3631" y="16364"><tspan font-family="Courier" font-size="423">__elasticsearch__</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M9000 3500h4821v1197H9000z"/><text class="TextShape"><tspan class="TextParagraph" font-size="388" font-weight="400"><tspan class="TextPosition" x="10132" y="4040"><tspan fill="gray">(class method)</tspan></tspan><tspan class="TextPosition" x="9251" y="4464"><tspan font-family="Courier" font-size="423">__elasticsearch__</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M9000 6400h4821v1197H9000z"/><text class="TextShape"><tspan class="TextParagraph" font-size="388" font-weight="400"><tspan class="TextPosition" x="9850" y="6940"><tspan fill="gray">(instance method)</tspan></tspan><tspan class="TextPosition" x="9251" y="7364"><tspan font-family="Courier" font-size="423">__elasticsearch__</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1975 25175h2051v851H1975z"/><path fill="none" stroke="#999" stroke-width="50" d="M3000 26000H2000v-800h2000v800H3000z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="2634" y="25748"><tspan fill="gray">Foo</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M4400 25200h7101v726H4400z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="4650" y="25710"><tspan>elasticsearch-rails’ internal class</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M4400 26400h8601v1200H4400z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="4650" y="26910"><tspan>where model-specific logic is</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M1975 26275h2051v851H1975z"/><path fill="none" stroke="#F33" stroke-width="50" d="M3000 27100H2000v-800h2000v800H3000z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="700"><tspan class="TextPosition" x="2613" y="26848"><tspan fill="#C9211E">Foo</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M4900 17289h5901v2312H4900z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="370" font-weight="400"><tspan class="TextPosition" x="7236" y="17748"><tspan fill="gray">Write operations like </tspan></tspan><tspan class="TextPosition" x="5323" y="18159"><tspan fill="gray">indexing/updating are forwarded </tspan></tspan><tspan class="TextPosition" x="8024" y="18570"><tspan fill="gray">to all instances.</tspan></tspan></tspan><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="370" font-weight="400"><tspan class="TextPosition" x="5501" y="18981"><tspan fill="gray">Read operations are forwarded </tspan></tspan><tspan class="TextPosition" x="7126" y="19392"><tspan fill="gray">to specified instance.</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10785 15769h1422v2691h-1422z"/><path fill="none" stroke="#999" stroke-width="30" d="M10800 18444c1429 0 934-1618 1119-2337"/><path fill="#999" d="M12206 15769l-460 293 267 217 193-510z"/></g><g class="com.sun.star.drawing.ConnectorShape"><path class="BoundingBox" fill="none" d="M10785 18429h1528v2862h-1528z"/><path fill="none" stroke="#999" stroke-width="30" d="M10800 18444c1509 0 970 1782 1200 2526"/><path fill="#999" d="M12312 21290l-227-496-252 235 479 261z"/></g><g class="com.sun.star.drawing.TextShape"><path class="BoundingBox" fill="none" d="M1800 24000h7101v807H1800z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="494" font-weight="700"><tspan class="TextPosition" x="2050" y="24574"><tspan>Legend</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13975 4275h5085v851h-5085z"/><path fill="none" stroke="#999" stroke-width="50" d="M16517 5100h-2517v-800h5034v800h-2517z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14737" y="4848"><tspan fill="gray">ClassMethodProxy</tspan></tspan></tspan></text></g><g class="com.sun.star.drawing.CustomShape"><path class="BoundingBox" fill="none" d="M13975 5975h5085v851h-5085z"/><path fill="none" stroke="#999" stroke-width="50" d="M16517 6800h-2517v-800h5034v800h-2517z"/><text class="TextShape"><tspan class="TextParagraph" font-family="Arial, sans-serif" font-size="423" font-weight="400"><tspan class="TextPosition" x="14462" y="6548"><tspan fill="gray">InstanceMethodProxy</tspan></tspan></tspan></text></g></g></g></g></svg>
\ No newline at end of file
ee/app/models/concerns/elastic/application_search.rb deleted 100644 → 0
View file @ f49fe157
# frozen_string_literal: true
module Elastic
module ApplicationSearch
extend ActiveSupport::Concern
# Defer evaluation from class-definition time to index-creation time
class AsJSON
def initialize(&blk)
@blk = blk
end
def call
@blk.call
end
def as_json(*args, &blk)
call
end
end
included do
include Elasticsearch::Model
index_name [Rails.application.class.parent_name.downcase, Rails.env].join('-')
# ES6 requires a single type per index
document_type 'doc'
# A temp solution to keep only one copy of setting,
# will be removed in https://gitlab.com/gitlab-org/gitlab-ee/issues/12548
__elasticsearch__.instance_variable_set(:@settings, Elastic::Latest::Config.settings)
__elasticsearch__.instance_variable_set(:@mapping, Elastic::Latest::Config.mappings)
after_commit on: :create do
if Gitlab::CurrentSettings.elasticsearch_indexing? && self.searchable?
ElasticIndexerWorker.perform_async(:index, self.class.to_s, self.id, self.es_id)
end
end
after_commit on: :update do
if Gitlab::CurrentSettings.elasticsearch_indexing? && self.searchable?
ElasticIndexerWorker.perform_async(
:update,
self.class.to_s,
self.id,
self.es_id,
changed_fields: self.previous_changes.keys
)
end
end
after_commit on: :destroy do
if Gitlab::CurrentSettings.elasticsearch_indexing? && self.searchable?
ElasticIndexerWorker.perform_async(
:delete,
self.class.to_s,
self.id,
self.es_id,
es_parent: self.es_parent
)
end
end
# Should be overridden in the models where some records should be skipped
def searchable?
self.use_elasticsearch?
end
def use_elasticsearch?
self.project&.use_elasticsearch?
end
def generic_attributes
{
'join_field' => {
'name' => es_type,
'parent' => es_parent
},
'type' => es_type
}
end
def es_parent
"project_#{project_id}" unless is_a?(Project) || self&.project_id.nil?
end
def es_type
self.class.es_type
end
def es_id
"#{es_type}_#{id}"
end
# Some attributes are actually complicated methods. Bad data can cause
# them to raise exceptions. When this happens, we still want the remainder
# of the object to be saved, so silently swallow the errors
def safely_read_attribute_for_elasticsearch(attr_name)
send(attr_name) # rubocop:disable GitlabSecurity/PublicSend
rescue => err
logger.warn("Elasticsearch failed to read #{attr_name} for #{self.class} #{self.id}: #{err}")
nil
end
end
class_methods do
# Support STI models
def inherited(subclass)
super
# Avoid SystemStackError in Model.import
# See https://github.com/elastic/elasticsearch-rails/issues/144
subclass.include Elasticsearch::Model
# Use ES configuration from parent model
# TODO: Revisit after upgrading to elasticsearch-model 7.0.0
# See https://github.com/elastic/elasticsearch-rails/commit/b8455db186664e21927bfb271bab6390853e7ff3
subclass.__elasticsearch__.index_name = self.index_name
subclass.__elasticsearch__.document_type = self.document_type
subclass.__elasticsearch__.instance_variable_set(:@mapping, self.mapping.dup)
end
# Should be overridden for all nested models
def nested?
false
end
def es_type
name.underscore
end
def highlight_options(fields)
es_fields = fields.map { |field| field.split('^').first }.each_with_object({}) do |field, memo|
memo[field.to_sym] = {}
end
{ fields: es_fields }
end
def es_import(**options)
transform = lambda do |r|
{ index: { _id: r.es_id, data: r.__elasticsearch__.as_indexed_json } }.tap do |data|
data[:index][:routing] = r.es_parent if r.es_parent
end
end
options[:transform] = transform
self.import(options)
end
def basic_query_hash(fields, query)
query_hash = if query.present?
{
query: {
bool: {
must: [{
simple_query_string: {
fields: fields,
query: query,
default_operator: :and
}
}],
filter: [{
term: { type: self.es_type }
}]
}
}
}
else
{
query: {
bool: {
must: { match_all: {} }
}
},
track_scores: true
}
end
query_hash[:sort] = [
{ updated_at: { order: :desc } },
:_score
]
query_hash[:highlight] = highlight_options(fields)
query_hash
end
def iid_query_hash(iid)
{
query: {
bool: {
filter: [{ term: { iid: iid } }]
}
}
}
end
# Builds an elasticsearch query that will select child documents from a
# set of projects, taking user access rules into account.
def project_ids_filter(query_hash, options)
project_query = project_ids_query(
options[:current_user],
options[:project_ids],
options[:public_and_internal_projects],
options[:features]
)
query_hash[:query][:bool][:filter] ||= []
query_hash[:query][:bool][:filter] << {
has_parent: {
parent_type: "project",
query: {
bool: project_query
}
}
}
query_hash
end
# Builds an elasticsearch query that will select projects the user is
# granted access to.
#
# If a project feature(s) is specified, it indicates interest in child
# documents gated by that project feature - e.g., "issues". The feature's
# visibility level must be taken into account.
def project_ids_query(user, project_ids, public_and_internal_projects, features = nil)
# When reading cross project is not allowed, only allow searching a
# a single project, so the `:read_*` ability is only checked once.
unless Ability.allowed?(user, :read_cross_project)
project_ids = [] if project_ids.is_a?(Array) && project_ids.size > 1
end
# At least one condition must be present, so pick no projects for
# anonymous users.
# Pick private, internal and public projects the user is a member of.
# Pick all private projects for admins & auditors.
conditions = [pick_projects_by_membership(project_ids, features)]
if public_and_internal_projects
# Skip internal projects for anonymous and external users.
# Others are given access to all internal projects. Admins & auditors
# get access to internal projects where the feature is private.
conditions << pick_projects_by_visibility(Project::INTERNAL, user, features) if user && !user.external?
# All users, including anonymous, can access public projects.
# Admins & auditors get access to public projects where the feature is
# private.
conditions << pick_projects_by_visibility(Project::PUBLIC, user, features)
end
{ should: conditions }
end
private
# Most users come with a list of projects they are members of, which may
# be a mix of public, internal or private. Grant access to them all, as
# long as the project feature is not disabled.
#
# Admins & auditors are given access to all private projects. Access to
# internal or public projects where the project feature is private is not
# granted here.
def pick_projects_by_membership(project_ids, features = nil)
condition =
if project_ids == :any
{ term: { visibility_level: Project::PRIVATE } }
else
{ terms: { id: project_ids } }
end
limit_by_feature(condition, features, include_members_only: true)
end
# Grant access to projects of the specified visibility level to the user.
#
# If a project feature is specified, access is only granted if the feature
# is enabled or, for admins & auditors, private.
def pick_projects_by_visibility(visibility, user, features)
condition = { term: { visibility_level: visibility } }
limit_by_feature(condition, features, include_members_only: user&.full_private_access?)
end
# If a project feature(s) is specified, access is dependent on its visibility
# level being enabled (or private if `include_members_only: true`).
#
# This method is a no-op if no project feature is specified.
# It accepts an array of features or a single feature, when an array is provided
# it queries if any of the features is enabled.
#
# Always denies access to projects when the features are disabled - even to
# admins & auditors - as stale child documents may be present.
def limit_by_feature(condition, features, include_members_only:)
return condition unless features
features = Array(features)
features.map do |feature|
limit =
if include_members_only
{ terms: { "#{feature}_access_level" => [::ProjectFeature::ENABLED, ::ProjectFeature::PRIVATE] } }
else
{ term: { "#{feature}_access_level" => ::ProjectFeature::ENABLED } }
end
{ bool: { filter: [condition, limit] } }
end
end
end
end
end
ee/app/models/concerns/elastic/issues_search.rb deleted 100644 → 0
View file @ f49fe157
# frozen_string_literal: true
module Elastic
module IssuesSearch
extend ActiveSupport::Concern
included do
include ApplicationSearch
def as_indexed_json(options = {})
data = {}
# We don't use as_json(only: ...) because it calls all virtual and serialized attributtes
# https://gitlab.com/gitlab-org/gitlab-ee/issues/349
[:id, :iid, :title, :description, :created_at, :updated_at, :state, :project_id, :author_id, :confidential].each do |attr|
data[attr.to_s] = safely_read_attribute_for_elasticsearch(attr)
end
data['assignee_id'] = safely_read_attribute_for_elasticsearch(:assignee_ids)
data.merge(generic_attributes)
end
def self.nested?
true
end
def self.elastic_search(query, options: {})
query_hash =
if query =~ /#(\d+)\z/
iid_query_hash(Regexp.last_match(1))
else
basic_query_hash(%w(title^2 description), query)
end
options[:features] = 'issues'
query_hash = project_ids_filter(query_hash, options)
query_hash = confidentiality_filter(query_hash, options[:current_user])
self.__elasticsearch__.search(query_hash)
end
def self.confidentiality_filter(query_hash, current_user)
return query_hash if current_user && current_user.full_private_access?
filter = if current_user
{
bool: {
should: [
{ term: { confidential: false } },
{
bool: {
must: [
{ term: { confidential: true } },
{
bool: {
should: [
{ term: { author_id: current_user.id } },
{ term: { assignee_id: current_user.id } },
{ terms: { project_id: current_user.authorized_projects(Gitlab::Access::REPORTER).pluck(:id) } }
]
}
}
]
}
}
]
}
}
else
{ term: { confidential: false } }
end
query_hash[:query][:bool][:filter] << filter
query_hash
end
end
end
end
ee/app/models/concerns/elastic/projects_search.rb
View file @ 8438243e
......@@ -4,13 +4,7 @@ module Elastic
module ProjectsSearch
extend ActiveSupport::Concern
TRACKED_FEATURE_SETTINGS = %w(
issues_access_level
merge_requests_access_level
snippets_access_level
wiki_access_level
repository_access_level
).freeze
include ApplicationVersionedSearch
INDEXED_ASSOCIATIONS = [
:issues,
......@@ -21,97 +15,10 @@ module Elastic
].freeze
included do
include ApplicationSearch
def use_elasticsearch?
::Gitlab::CurrentSettings.elasticsearch_indexes_project?(self)
end
def as_indexed_json(options = {})
# We don't use as_json(only: ...) because it calls all virtual and serialized attributtes
# https://gitlab.com/gitlab-org/gitlab-ee/issues/349
data = {}
[
:id,
:name,
:path,
:description,
:namespace_id,
:created_at,
:updated_at,
:archived,
:visibility_level,
:last_activity_at,
:name_with_namespace,
:path_with_namespace
].each do |attr|
data[attr.to_s] = safely_read_attribute_for_elasticsearch(attr)
end
# Set it as a parent in our `project => child` JOIN field
data['join_field'] = es_type
# ES6 is now single-type per index, so we implement our own typing
data['type'] = 'project'
TRACKED_FEATURE_SETTINGS.each do |feature|
data[feature] = project_feature.public_send(feature) # rubocop:disable GitlabSecurity/PublicSend
end
data
end
def self.elastic_search(query, options: {})
options[:in] = %w(name^10 name_with_namespace^2 path_with_namespace path^9 description)
query_hash = basic_query_hash(options[:in], query)
filters = []
if options[:namespace_id]
filters << {
terms: {
namespace_id: [options[:namespace_id]].flatten
}
}
end
if options[:non_archived]
filters << {
terms: {
archived: [!options[:non_archived]].flatten
}
}
end
if options[:visibility_levels]
filters << {
terms: {
visibility_level: [options[:visibility_levels]].flatten
}
}
end
if options[:project_ids]
filters << {
bool: project_ids_query(options[:current_user], options[:project_ids], options[:public_and_internal_projects])
}
end
query_hash[:query][:bool][:filter] = filters
query_hash[:sort] = [:_score]
self.__elasticsearch__.search(query_hash)
end
def self.indexed_association_classes
INDEXED_ASSOCIATIONS.map do |association_name|
reflect_on_association(association_name).klass
end
end
def each_indexed_association
INDEXED_ASSOCIATIONS.each do |association_name|
association = self.association(association_name)
......
ee/app/models/ee/issue.rb
View file @ 8438243e
......@@ -11,7 +11,7 @@ module EE
WEIGHT_ANY = 'Any'.freeze
WEIGHT_NONE = 'None'.freeze
include Elastic::IssuesSearch
include Elastic::ApplicationVersionedSearch
scope :order_weight_desc, -> { reorder ::Gitlab::Database.nulls_last_order('weight', 'DESC') }
scope :order_weight_asc, -> { reorder ::Gitlab::Database.nulls_last_order('weight') }
......
ee/app/models/ee/legacy_diff_note.rb
View file @ 8438243e
......@@ -5,7 +5,7 @@ module EE
extend ActiveSupport::Concern
prepended do
include Elastic::NotesSearch
include Elastic::ApplicationVersionedSearch
end
end
end
ee/app/models/ee/merge_request.rb
View file @ 8438243e
......@@ -11,7 +11,7 @@ module EE
include FromUnion
prepended do
include Elastic::MergeRequestsSearch
include Elastic::ApplicationVersionedSearch
has_many :reviews, inverse_of: :merge_request
has_many :approvals, dependent: :delete_all # rubocop:disable Cop/ActiveRecordDependent
......
ee/app/models/ee/milestone.rb
View file @ 8438243e
......@@ -5,7 +5,7 @@ module EE
extend ActiveSupport::Concern
prepended do
include Elastic::MilestonesSearch
include Elastic::ApplicationVersionedSearch
has_many :boards
end
......
ee/app/models/ee/note.rb
View file @ 8438243e
......@@ -7,7 +7,7 @@ module EE
prepended do
include ::ObjectStorage::BackgroundMove
include Elastic::NotesSearch
include Elastic::ApplicationVersionedSearch
belongs_to :review, inverse_of: :notes
......
ee/lib/elastic/latest/issue_class_proxy.rb 0 → 100644
View file @ 8438243e
# frozen_string_literal: true
module Elastic
module Latest
class IssueClassProxy < ApplicationClassProxy
def nested?
true
end
def elastic_search(query, options: {})
query_hash =
if query =~ /#(\d+)\z/
iid_query_hash(Regexp.last_match(1))
else
basic_query_hash(%w(title^2 description), query)
end
options[:features] = 'issues'
query_hash = project_ids_filter(query_hash, options)
query_hash = confidentiality_filter(query_hash, options[:current_user])
search(query_hash)
end
private
def confidentiality_filter(query_hash, current_user)
return query_hash if current_user && current_user.full_private_access?
filter =
if current_user
{
bool: {
should: [
{ term: { confidential: false } },
{
bool: {
must: [
{ term: { confidential: true } },
{
bool: {
should: [
{ term: { author_id: current_user.id } },
{ term: { assignee_id: current_user.id } },
{ terms: { project_id: current_user.authorized_projects(Gitlab::Access::REPORTER).pluck_primary_key } }
]
}
}
]
}
}
]
}
}
else
{ term: { confidential: false } }
end
query_hash[:query][:bool][:filter] << filter
query_hash
end
end
end
end
ee/lib/elastic/latest/issue_instance_proxy.rb 0 → 100644
View file @ 8438243e
# frozen_string_literal: true
module Elastic
module Latest
class IssueInstanceProxy < ApplicationInstanceProxy
def as_indexed_json(options = {})
data = {}
# We don't use as_json(only: ...) because it calls all virtual and serialized attributtes
# https://gitlab.com/gitlab-org/gitlab-ee/issues/349
[:id, :iid, :title, :description, :created_at, :updated_at, :state, :project_id, :author_id, :confidential].each do |attr|
data[attr.to_s] = safely_read_attribute_for_elasticsearch(attr)
end
data['assignee_id'] = safely_read_attribute_for_elasticsearch(:assignee_ids)
data.merge(generic_attributes)
end
end
end
end
ee/lib/elastic/latest/merge_request_class_proxy.rb 0 → 100644
View file @ 8438243e
# frozen_string_literal: true
module Elastic
module Latest
class MergeRequestClassProxy < ApplicationClassProxy
def nested?
true
end
def elastic_search(query, options: {})
query_hash =
if query =~ /\!(\d+)\z/
iid_query_hash(Regexp.last_match(1))
else
basic_query_hash(%w(title^2 description), query)
end
options[:features] = 'merge_requests'
query_hash = project_ids_filter(query_hash, options)
search(query_hash)
end
end
end
end
ee/app/models/concerns/elastic/merge_requests_search.rb → ee/lib/elastic/latest/merge_request_instance_proxy.rb
View file @ 8438243e
# frozen_string_literal: true
module Elastic
module MergeRequestsSearch
extend ActiveSupport::Concern
included do
include ApplicationSearch
module Latest
class MergeRequestInstanceProxy < ApplicationInstanceProxy
def as_indexed_json(options = {})
# We don't use as_json(only: ...) because it calls all virtual and serialized attributtes
# https://gitlab.com/gitlab-org/gitlab-ee/issues/349
......@@ -32,28 +28,6 @@ module Elastic
data.merge(generic_attributes)
end
def es_parent
"project_#{target_project_id}"
end
def self.nested?
true
end
def self.elastic_search(query, options: {})
query_hash =
if query =~ /\!(\d+)\z/
iid_query_hash(Regexp.last_match(1))
else
basic_query_hash(%w(title^2 description), query)
end
options[:features] = 'merge_requests'
query_hash = project_ids_filter(query_hash, options)
self.__elasticsearch__.search(query_hash)
end
end
end
end
ee/lib/elastic/latest/milestone_class_proxy.rb 0 → 100644
View file @ 8438243e
# frozen_string_literal: true
module Elastic
module Latest
class MilestoneClassProxy < ApplicationClassProxy
def nested?
true
end
def elastic_search(query, options: {})
options[:in] = %w(title^2 description)
query_hash = basic_query_hash(options[:in], query)
query_hash = project_ids_filter(query_hash, options)
search(query_hash)
end
end
end
end
ee/app/models/concerns/elastic/milestones_search.rb → ee/lib/elastic/latest/milestone_instance_proxy.rb
View file @ 8438243e
# frozen_string_literal: true
module Elastic
module MilestonesSearch
extend ActiveSupport::Concern
included do
include ApplicationSearch
module Latest
class MilestoneInstanceProxy < ApplicationInstanceProxy
def as_indexed_json(options = {})
# We don't use as_json(only: ...) because it calls all virtual and serialized attributtes
# https://gitlab.com/gitlab-org/gitlab-ee/issues/349
......@@ -18,20 +14,6 @@ module Elastic
data.merge(generic_attributes)
end
def self.nested?
true
end
def self.elastic_search(query, options: {})
options[:in] = %w(title^2 description)
query_hash = basic_query_hash(options[:in], query)
query_hash = project_ids_filter(query_hash, options)
self.__elasticsearch__.search(query_hash)
end
end
end
end
ee/app/models/concerns/elastic/notes_search.rb → ee/lib/elastic/latest/note_class_proxy.rb
View file @ 8438243e
# frozen_string_literal: true
module Elastic
module NotesSearch
extend ActiveSupport::Concern
included do
include ApplicationSearch
module Latest
class NoteClassProxy < ApplicationClassProxy
def es_type
'note'
end
def as_indexed_json(options = {})
data = {}
# We don't use as_json(only: ...) because it calls all virtual and serialized attributtes
# https://gitlab.com/gitlab-org/gitlab-ee/issues/349
[:id, :note, :project_id, :noteable_type, :noteable_id, :created_at, :updated_at].each do |attr|
data[attr.to_s] = safely_read_attribute_for_elasticsearch(attr)
end
if noteable.is_a?(Issue)
data['issue'] = {
assignee_id: noteable.assignee_ids,
author_id: noteable.author_id,
confidential: noteable.confidential
}
end
data.merge(generic_attributes)
end
def self.nested?
def nested?
true
end
def self.elastic_search(query, options: {})
def elastic_search(query, options: {})
options[:in] = ['note']
query_hash = basic_query_hash(%w[note], query)
......@@ -49,11 +25,13 @@ module Elastic
query_hash[:highlight] = highlight_options(options[:in])
self.__elasticsearch__.search(query_hash)
search(query_hash)
end
def self.confidentiality_filter(query_hash, current_user)
return query_hash if current_user && current_user.full_private_access?
private
def confidentiality_filter(query_hash, current_user)
return query_hash if current_user&.full_private_access?
filter = {
bool: {
......@@ -74,7 +52,7 @@ module Elastic
should: [
{ term: { "issue.author_id" => current_user.id } },
{ term: { "issue.assignee_id" => current_user.id } },
{ terms: { "project_id" => current_user.authorized_projects(Gitlab::Access::REPORTER).pluck(:id) } }
{ terms: { "project_id" => current_user.authorized_projects(Gitlab::Access::REPORTER).pluck_primary_key } }
]
}
}
......
ee/lib/elastic/latest/note_instance_proxy.rb 0 → 100644
View file @ 8438243e
# frozen_string_literal: true
module Elastic
module Latest
class NoteInstanceProxy < ApplicationInstanceProxy
delegate :noteable, to: :target
def as_indexed_json(options = {})
data = {}
# We don't use as_json(only: ...) because it calls all virtual and serialized attributtes
# https://gitlab.com/gitlab-org/gitlab-ee/issues/349
[:id, :note, :project_id, :noteable_type, :noteable_id, :created_at, :updated_at].each do |attr|
data[attr.to_s] = safely_read_attribute_for_elasticsearch(attr)
end
if noteable.is_a?(Issue)
data['issue'] = {
assignee_id: noteable.assignee_ids,
author_id: noteable.author_id,
confidential: noteable.confidential
}
end
data.merge(generic_attributes)
end
end
end
end
ee/lib/elastic/latest/project_class_proxy.rb 0 → 100644
View file @ 8438243e
# frozen_string_literal: true
module Elastic
module Latest
class ProjectClassProxy < ApplicationClassProxy
def elastic_search(query, options: {})
options[:in] = %w(name^10 name_with_namespace^2 path_with_namespace path^9 description)
query_hash = basic_query_hash(options[:in], query)
filters = []
if options[:namespace_id]
filters << {
terms: {
namespace_id: [options[:namespace_id]].flatten
}
}
end
if options[:non_archived]
filters << {
terms: {
archived: [!options[:non_archived]].flatten
}
}
end
if options[:visibility_levels]
filters << {
terms: {
visibility_level: [options[:visibility_levels]].flatten
}
}
end
if options[:project_ids]
filters << {
bool: project_ids_query(options[:current_user], options[:project_ids], options[:public_and_internal_projects])
}
end
query_hash[:query][:bool][:filter] = filters
query_hash[:sort] = [:_score]
search(query_hash)
end
end
end
end
ee/lib/elastic/latest/project_instance_proxy.rb 0 → 100644
View file @ 8438243e
# frozen_string_literal: true
module Elastic
module Latest
class ProjectInstanceProxy < ApplicationInstanceProxy
TRACKED_FEATURE_SETTINGS = %w(
issues_access_level
merge_requests_access_level
snippets_access_level
wiki_access_level
repository_access_level
).freeze
def as_indexed_json(options = {})
# We don't use as_json(only: ...) because it calls all virtual and serialized attributtes
# https://gitlab.com/gitlab-org/gitlab-ee/issues/349
data = {}
[
:id,
:name,
:path,
:description,
:namespace_id,
:created_at,
:updated_at,
:archived,
:visibility_level,
:last_activity_at,
:name_with_namespace,
:path_with_namespace
].each do |attr|
data[attr.to_s] = safely_read_attribute_for_elasticsearch(attr)
end
# Set it as a parent in our `project => child` JOIN field
data['join_field'] = es_type
# ES6 is now single-type per index, so we implement our own typing
data['type'] = 'project'
TRACKED_FEATURE_SETTINGS.each do |feature|
data[feature] = target.project_feature.public_send(feature) # rubocop:disable GitlabSecurity/PublicSend
end
data
end
end
end
end
ee/lib/elastic/multi_version_util.rb
View file @ 8438243e
......@@ -36,6 +36,19 @@ module Elastic
klass < ActiveRecord::Base ? klass.base_class : klass
end
# Handles which method calls should be forwarded to all targets,
# and which calls should be forwarded to just one target.
#
# This first sets up forwarding for methods which should go to all targets.
# This is specified by implementing `methods_for_all_write_targets`.
# Examples include document indexing/updating operations.
#
# Then other methods are forwarded to just the single read target.
# Examples include user searches.
#
# Special write operations specified in `methods_for_one_write_target` are left out.
# The caller must always specify the version the call should be triggered.
# Examples include deleting the whole index.
def generate_forwarding
methods_for_all_write_targets = elastic_writing_targets.first.real_class.methods_for_all_write_targets
methods_for_one_write_target = elastic_writing_targets.first.real_class.methods_for_one_write_target
......
ee/lib/elastic/v12p1/issue_class_proxy.rb 0 → 100644
View file @ 8438243e
# frozen_string_literal: true
module Elastic
module V12p1
IssueClassProxy = Elastic::Latest::IssueClassProxy
end
end
ee/lib/elastic/v12p1/issue_instance_proxy.rb 0 → 100644
View file @ 8438243e
# frozen_string_literal: true
module Elastic
module V12p1
IssueInstanceProxy = Elastic::Latest::IssueInstanceProxy
end
end
ee/lib/elastic/v12p1/merge_request_class_proxy.rb 0 → 100644
View file @ 8438243e
# frozen_string_literal: true
module Elastic
module V12p1
MergeRequestClassProxy = Elastic::Latest::MergeRequestClassProxy
end
end
ee/lib/elastic/v12p1/merge_request_instance_proxy.rb 0 → 100644
View file @ 8438243e
# frozen_string_literal: true
module Elastic
module V12p1
MergeRequestInstanceProxy = Elastic::Latest::MergeRequestInstanceProxy
end
end
ee/lib/elastic/v12p1/milestone_class_proxy.rb 0 → 100644
View file @ 8438243e
# frozen_string_literal: true
module Elastic
module V12p1
MilestoneClassProxy = Elastic::Latest::MilestoneClassProxy
end
end
ee/lib/elastic/v12p1/milestone_instance_proxy.rb 0 → 100644
View file @ 8438243e
# frozen_string_literal: true
module Elastic
module V12p1
MilestoneInstanceProxy = Elastic::Latest::MilestoneInstanceProxy
end
end
ee/lib/elastic/v12p1/note_class_proxy.rb 0 → 100644
View file @ 8438243e
# frozen_string_literal: true
module Elastic
module V12p1
NoteClassProxy = Elastic::Latest::NoteClassProxy
end
end
ee/lib/elastic/v12p1/note_instance_proxy.rb 0 → 100644
View file @ 8438243e
# frozen_string_literal: true
module Elastic
module V12p1
NoteInstanceProxy = Elastic::Latest::NoteInstanceProxy
end
end
ee/lib/elastic/v12p1/project_class_proxy.rb 0 → 100644
View file @ 8438243e
# frozen_string_literal: true
module Elastic
module V12p1
ProjectClassProxy = Elastic::Latest::ProjectClassProxy
end
end
ee/lib/elastic/v12p1/project_instance_proxy.rb 0 → 100644
View file @ 8438243e
# frozen_string_literal: true
module Elastic
module V12p1
ProjectInstanceProxy = Elastic::Latest::ProjectInstanceProxy
end
end
ee/lib/gitlab/elastic/helper.rb
View file @ 8438243e
......@@ -4,8 +4,7 @@ module Gitlab
module Elastic
class Helper
# rubocop: disable CodeReuse/ActiveRecord
def self.create_empty_index
index_name = Project.index_name
def self.create_empty_index(version = ::Elastic::MultiVersionUtil::TARGET_VERSION)
settings = {}
mappings = {}
......@@ -23,7 +22,9 @@ module Gitlab
mappings.deep_merge!(klass.__elasticsearch__.mappings.to_hash)
end
client = Project.__elasticsearch__.client
proxy = Project.__elasticsearch__.version(version)
client = proxy.client
index_name = proxy.index_name
# ES5.6 needs a setting enabled to support JOIN datatypes that ES6 does not support...
if Gitlab::VersionInfo.parse(client.info['version']['number']) < Gitlab::VersionInfo.new(6)
......@@ -42,16 +43,16 @@ module Gitlab
end
# rubocop: enable CodeReuse/ActiveRecord
def self.delete_index
Project.__elasticsearch__.delete_index!
def self.delete_index(version = ::Elastic::MultiVersionUtil::TARGET_VERSION)
Project.__elasticsearch__.version(version).delete_index!
end
def self.refresh_index
Project.__elasticsearch__.refresh_index!
end
def self.index_size
Project.__elasticsearch__.client.indices.stats['indices'][Project.__elasticsearch__.index_name]['total']
def self.index_size(version = ::Elastic::MultiVersionUtil::TARGET_VERSION)
Project.__elasticsearch__.version(version).client.indices.stats['indices'][Project.__elasticsearch__.index_name]['total']
end
end
end
......
ee/lib/gitlab/elastic/search_results.rb
View file @ 8438243e
......@@ -199,7 +199,7 @@ module Gitlab
def milestones
strong_memoize(:milestones) do
# Must pass 'issues' and 'merge_requests' to check
# if any of the features is available for projects in Elastic::ApplicationSearch#project_ids_query
# if any of the features is available for projects in ApplicationClassProxy#project_ids_query
# Otherwise it will ignore project_ids and return milestones
# from projects with milestones disabled.
options = base_options
......
ee/lib/tasks/gitlab/elastic/test.rake
View file @ 8438243e
......@@ -4,7 +4,7 @@ namespace :gitlab do
desc 'GitLab | Elasticsearch | Test | Measure space taken by ES indices'
task index_size: :environment do
puts "===== Size stats for index: #{Project.__elasticsearch__.index_name} ====="
pp Gitlab::Elastic::Helper.index_size.slice(*%w(docs store))
pp Gitlab::Elastic::Helper.index_size(::Elastic::MultiVersionUtil::TARGET_VERSION).slice(*%w(docs store))
end
desc 'GitLab | Elasticsearch | Test | Measure space taken by ES indices, reindex, and measure space taken again'
......
ee/spec/models/concerns/elastic/issue_spec.rb
View file @ 8438243e
......@@ -89,7 +89,7 @@ describe Issue, :elastic do
expected_hash['assignee_id'] = [assignee.id]
expect(issue.as_indexed_json).to eq(expected_hash)
expect(issue.__elasticsearch__.as_indexed_json).to eq(expected_hash)
end
it_behaves_like 'no results when the user cannot read cross project' do
......
ee/spec/models/concerns/elastic/merge_request_spec.rb
View file @ 8438243e
......@@ -61,7 +61,7 @@ describe MergeRequest, :elastic do
'type' => merge_request.es_type
})
expect(merge_request.as_indexed_json).to eq(expected_hash)
expect(merge_request.__elasticsearch__.as_indexed_json).to eq(expected_hash)
end
it_behaves_like 'no results when the user cannot read cross project' do
......
ee/spec/models/concerns/elastic/milestone_spec.rb
View file @ 8438243e
......@@ -53,7 +53,7 @@ describe Milestone, :elastic do
'type' => milestone.es_type
})
expect(milestone.as_indexed_json).to eq(expected_hash)
expect(milestone.__elasticsearch__.as_indexed_json).to eq(expected_hash)
end
it_behaves_like 'no results when the user cannot read cross project' do
......
ee/spec/models/concerns/elastic/note_spec.rb
View file @ 8438243e
......@@ -88,7 +88,7 @@ describe Note, :elastic do
type
)
expect(note.as_indexed_json.keys).to eq(expected_hash_keys)
expect(note.__elasticsearch__.as_indexed_json.keys).to eq(expected_hash_keys)
end
it "does not create ElasticIndexerWorker job for system messages" do
......@@ -105,7 +105,7 @@ describe Note, :elastic do
Note.subclasses.each do |note_class|
expect(note_class.index_name).to eq(Note.index_name)
expect(note_class.document_type).to eq(Note.document_type)
expect(note_class.mappings.to_hash).to eq(Note.mappings.to_hash)
expect(note_class.__elasticsearch__.mappings.to_hash).to eq(Note.__elasticsearch__.mappings.to_hash)
end
end
......
ee/spec/models/concerns/elastic/project_spec.rb
View file @ 8438243e
......@@ -159,6 +159,6 @@ describe Project, :elastic do
expected_hash['name_with_namespace'] = project.full_name
expected_hash['path_with_namespace'] = project.full_path
expect(project.as_indexed_json).to eq(expected_hash)
expect(project.__elasticsearch__.as_indexed_json).to eq(expected_hash)
end
end
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7