@@ -220,7 +220,6 @@ are listed in the descriptions of the relevant settings.
| `elasticsearch_aws` | boolean | no | **(PREMIUM)** Enable the use of AWS hosted Elasticsearch |
| `elasticsearch_aws_region` | string | no | **(PREMIUM)** The AWS region the elasticsearch domain is configured |
| `elasticsearch_aws_secret_access_key` | string | no | **(PREMIUM)** AWS IAM secret access key |
| `elasticsearch_experimental_indexer` | boolean | no | **(PREMIUM)** Use the experimental elasticsearch indexer. More info: <https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer> |
| `elasticsearch_limit_indexing` | boolean | no | **(PREMIUM)** Limit Elasticsearch to index certain namespaces and projects |
| `elasticsearch_namespace_ids` | array of integers | no | **(PREMIUM)** The namespaces to index via Elasticsearch if `elasticsearch_limit_indexing` is enabled. |
@@ -59,7 +59,7 @@ Additionally, if you need large repos or multiple forks for testing, please cons
## How does it work?
The Elasticsearch integration depends on an external indexer. We ship a[ruby indexer](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/bin/elastic_repo_indexer) by default but are also working on an [indexer written in Go](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). The user must trigger the initial indexing via a rake task, but after this is done GitLab itself will trigger reindexing when required via `after_` callbacks on create, update, and destroy that are inherited from [/ee/app/models/concerns/elastic/application_search.rb](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/app/models/concerns/elastic/application_search.rb).
The Elasticsearch integration depends on an external indexer. We ship an [indexer written in Go](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). The user must trigger the initial indexing via a rake task but, after this is done, GitLab itself will trigger reindexing when required via `after_` callbacks on create, update, and destroy that are inherited from [/ee/app/models/concerns/elastic/application_search.rb](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/app/models/concerns/elastic/application_search.rb).
All indexing after the initial one is done via `ElasticIndexerWorker` (sidekiq jobs).
@@ -10,24 +10,30 @@ Review Apps are automatically deployed by each pipeline, both in
```mermaid
graph TD
build-qa-image -.->|once the `prepare` stage is done| gitlab:assets:compile
review-build-cng -->|triggers a CNG-mirror pipeline and wait for it to be done| CNG-mirror
review-build-cng -.->|once the `test` stage is done| review-deploy
review-deploy -.->|once the `review` stage is done| review-qa-smoke
build-qa-image -->|once the `prepare` stage is done| gitlab:assets:compile
gitlab:assets:compile -->|once the `gitlab:assets:compile` job is done| review-build-cng
review-build-cng -.->|triggers a CNG-mirror pipeline and wait for it to be done| CNG-mirror
CNG-mirror -.->|polls until completed| review-build-cng
review-build-cng -->|once the `review-build-cng` job is done| review-deploy
review-deploy -->|once the `review-deploy` job is done| review-qa-smoke
subgraph "1. gitlab-ce/ee `prepare` stage"
build-qa-image
end
subgraph "2. gitlab-ce/ee `test` stage"
gitlab:assets:compile -->|plays dependent job once done| review-build-cng
gitlab:assets:compile
end
subgraph "3. gitlab-ce/ee `review` stage"
subgraph "3. gitlab-ce/ee `review-prepare` stage"
review-build-cng
end
subgraph "4. gitlab-ce/ee `review` stage"
review-deploy["review-deploy<br><br>Helm deploys the Review App using the Cloud<br/>Native images built by the CNG-mirror pipeline.<br><br>Cloud Native images are deployed to the `review-apps-ce` or `review-apps-ee`<br>Kubernetes (GKE) cluster, in the GCP `gitlab-review-apps` project."]
end
subgraph "4. gitlab-ce/ee `qa` stage"
subgraph "5. gitlab-ce/ee `qa` stage"
review-qa-smoke[review-qa-smoke<br><br>gitlab-qa runs the smoke suite against the Review App.]
If [Review App Stability](https://gitlab.com/gitlab-org/quality/team-tasks/issues/93) dips this may be a signal
that the `review-apps-ce/ee` cluster is unhealthy. Leading indicators may be healthcheck failures leading to restarts or majority failure for Review App deployments.
The following items may help diagnose this:
-[Instance group CPU Utilization in GCP](https://console.cloud.google.com/compute/instanceGroups/details/us-central1-a/gke-review-apps-ce-preemp-n1-standard-a4c9571c-grp?project=gitlab-review-apps&tab=monitoring&graph=GCE_CPU&duration=PT12H) - helpful to identify if nodes are problematic or the entire cluster is trending towards unhealthy
-[Instance Group size in GCP](https://console.cloud.google.com/compute/instanceGroups/details/us-central1-a/gke-review-apps-ce-preemp-n1-standard-a4c9571c-grp?project=gitlab-review-apps&tab=monitoring&graph=GCE_SIZE&duration=PT12H) - aids in identifying load spikes on the cluster. Kubernetes will add nodes up to 220 based on total resource requests.
-`kubectl top nodes --sort-by=cpu` - can identify if node spikes are common or load on specific nodes which may get rebalanced by the Kubernetes scheduler.
-`kubectl top pods --sort-by=cpu` -
-[K9s] - K9s is a powerful command line dashboard which allows you to filter by labels. This can help identify trends with apps exceeding the [review-app resource requests](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/scripts/review_apps/base-config.yaml). Kubernetes will schedule pods to nodes based on resource requests and allow for CPU usage up to the limits.
- In K9s you can sort or add filters by typing the `/` character
-`-lrelease=<review-app-slug>` - filters down to all pods for a release. This aids in determining what is having issues in a single deployment
-`-lapp=<app>` - filters down to all pods for a specific app. This aids in determining resource usage by app.
- You can scroll to a Kubernetes resource and hit `d`(describe), `s`(shell), `l`(logs) for a deeper inspection
![K9s](img/k9s.png)
### Troubleshoot a pending `dns-gitlab-review-app-external-dns` Deployment
#### Finding the problem
...
...
@@ -266,6 +291,12 @@ find a way to limit it to only us.**
## Other resources
-[Review Apps integration for CE/EE (presentation)](https://docs.google.com/presentation/d/1QPLr6FO4LduROU8pQIPkX1yfGvD13GEJIBOenqoKxR8/edit?usp=sharing)
The source code will first be fetched to the path specified by the first parameter. Then a binary will be built under its `bin` directory.
You will then need to update `gitlab.yml`'s `production -> elasticsearch -> indexer_path` setting to point to that binary.
### Install GitLab Pages
GitLab Pages uses [GNU Make](https://www.gnu.org/software/make/). This step is optional and only needed if you wish to host static sites from within GitLab. The following commands will install GitLab Pages in `/home/git/gitlab-pages`. For additional setup steps, consult the [administration guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/administration/pages/source.md) for your version of GitLab as the GitLab Pages daemon can be run several different ways.
| Go ([Golang](https://golang.org/)) | not currently ([issue](https://gitlab.com/gitlab-org/gitlab-ee/issues/7132"Dependency Scanning for Go")) | not available |
| Python ([Pipfile](https://docs.pipenv.org/en/latest/basics/)) | not currently ([issue](https://gitlab.com/gitlab-org/gitlab-ee/issues/11756"Pipfile.lock support for Dependency Scanning"))| not available |
| Python ([poetry](https://poetry.eustace.io/)) | not currently ([issue](https://gitlab.com/gitlab-org/gitlab-ee/issues/7006"Support Poetry in Dependency Scanning")) | not available |
context'when developers and maintainers are not allowed to push to a protected branch'do
it'user without push rights fails to push to the protected branch'do
create_protected_branch(allow_to_push: false)
create_protected_branch(allowed_to_push: {
roles: Resource::ProtectedBranch::Roles::NO_ONE
})
expect{push_new_file(branch_name)}.toraise_error(QA::Git::Repository::RepositoryCommandError,/remote: GitLab: You are not allowed to push code to protected branches on this project\.([\s\S]+)\[remote rejected\] #{branch_name} -> #{branch_name} \(pre-receive hook declined\)/)
it'returns an informational message with rules that can run'do
expect(described_class.local_warning_message).toeq('==> Only the following Danger rules can be run locally: changes_size, gemfile, documentation, frozen_string, duplicate_yarn_dependencies, prettier, eslint, database')
expect(described_class.local_warning_message).toeq('==> Only the following Danger rules can be run locally: changes_size, gemfile, documentation, frozen_string, duplicate_yarn_dependencies, prettier, eslint, database, commit_messages')