Commit 68a56658 authored by Kushal Pandya's avatar Kushal Pandya

Merge branch '354845-aqualls-workhorse1' into 'master'

Workhorse docs: fix linting errors

See merge request gitlab-org/gitlab!83782
parents e11d499a 885e674d
---
stage: Create
group: Source Code
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# GitLab Workhorse # GitLab Workhorse
GitLab Workhorse is a smart reverse proxy for GitLab. It handles GitLab Workhorse is a smart reverse proxy for GitLab. It handles
...@@ -10,26 +16,25 @@ GitLab](doc/architecture/gitlab_features.md) that would not work efficiently wit ...@@ -10,26 +16,25 @@ GitLab](doc/architecture/gitlab_features.md) that would not work efficiently wit
## Canonical source ## Canonical source
The canonical source for Workhorse is The canonical source for Workhorse is
[gitlab-org/gitlab/workhorse](https://gitlab.com/gitlab-org/gitlab/tree/master/workhorse). [`gitlab-org/gitlab/workhorse`](https://gitlab.com/gitlab-org/gitlab/tree/master/workhorse).
Prior to https://gitlab.com/groups/gitlab-org/-/epics/4826, it was Prior to [epic #4826](https://gitlab.com/groups/gitlab-org/-/epics/4826), it was
[gitlab-org/gitlab-workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse/tree/master), [`gitlab-org/gitlab-workhorse`](https://gitlab.com/gitlab-org/gitlab-workhorse/tree/master),
but that repository is no longer used for development. but that repository is no longer used for development.
## Documentation ## Documentation
Workhorse documentation is available in the [`doc` folder of this repository](doc/). Workhorse documentation is available in the [`doc` folder of this repository](doc/).
* Architectural overview - Architectural overview
* [GitLab features that rely on Workhorse](doc/architecture/gitlab_features.md) - [GitLab features that rely on Workhorse](doc/architecture/gitlab_features.md)
* [Websocket channel support](doc/architecture/channel.md) - [Websocket channel support](doc/architecture/channel.md)
* Operating Workhorse - Operating Workhorse
* [Source installation](doc/operations/install.md) - [Source installation](doc/operations/install.md)
* [Workhorse configuration](doc/operations/configuration.md) - [Workhorse configuration](doc/operations/configuration.md)
* [Contributing](CONTRIBUTING.md) - [Contributing](CONTRIBUTING.md)
* [Adding new features](doc/development/new_features.md) - [Adding new features](doc/development/new_features.md)
* [Testing your code](doc/development/tests.md) - [Testing your code](doc/development/tests.md)
## License ## License
This code is distributed under the MIT license, see the [LICENSE](LICENSE) file. This code is distributed under the MIT license, see the [LICENSE](LICENSE) file.
---
stage: Create
group: Source Code
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Websocket channel support # Websocket channel support
In some cases, GitLab can provide in-browser terminal access to an In some cases, GitLab can provide in-browser terminal access to an
...@@ -22,7 +28,7 @@ responses. WebSocket URLs have schemes like `ws://` (unencrypted) or ...@@ -22,7 +28,7 @@ responses. WebSocket URLs have schemes like `ws://` (unencrypted) or
When requesting an upgrade to WebSocket, the browser sends a HTTP/1.1 When requesting an upgrade to WebSocket, the browser sends a HTTP/1.1
request that looks like this: request that looks like this:
``` ```plaintext
GET /path.ws HTTP/1.1 GET /path.ws HTTP/1.1
Connection: upgrade Connection: upgrade
Upgrade: websocket Upgrade: websocket
...@@ -116,10 +122,10 @@ This returns a JSON response containing details of where the ...@@ -116,10 +122,10 @@ This returns a JSON response containing details of where the
terminal can be found, and how to connect it. In particular, terminal can be found, and how to connect it. In particular,
the following details are returned in case of success: the following details are returned in case of success:
* WebSocket URL to **connect** to, e.g.: `wss://example.com/terminals/1.ws?tty=1` - WebSocket URL to **connect** to, e.g.: `wss://example.com/terminals/1.ws?tty=1`
* WebSocket subprotocols to support, e.g.: `["channel.k8s.io"]` - WebSocket subprotocols to support, e.g.: `["channel.k8s.io"]`
* Headers to send, e.g.: `Authorization: Token xxyyz..` - Headers to send, e.g.: `Authorization: Token xxyyz..`
* Certificate authority to verify `wss` connections with (optional) - Certificate authority to verify `wss` connections with (optional)
Workhorse periodically re-checks this endpoint, and if it gets an Workhorse periodically re-checks this endpoint, and if it gets an
error response, or the details of the terminal change, it will error response, or the details of the terminal change, it will
...@@ -186,9 +192,8 @@ written to that fd. ...@@ -186,9 +192,8 @@ written to that fd.
Also used by Kubernetes, this subprotocol defines a similar multiplexed Also used by Kubernetes, this subprotocol defines a similar multiplexed
channel to `channel.k8s.io`. The main differences are: channel to `channel.k8s.io`. The main differences are:
* `TextMessage` frames are valid, rather than `BinaryMessage` frames. - `TextMessage` frames are valid, rather than `BinaryMessage` frames.
* The first byte of each `TextMessage` frame represents the file - The first byte of each `TextMessage` frame represents the file
descriptor as a numeric UTF-8 character, so the character `U+0030`, descriptor as a numeric UTF-8 character, so the character `U+0030`,
or "0", is fd 0, STDIN). or "0", is fd 0, STDIN).
* The remaining bytes represent base64-encoded arbitrary data. - The remaining bytes represent base64-encoded arbitrary data.
---
stage: Create
group: Source Code
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Features that rely on Workhorse # Features that rely on Workhorse
Workhorse itself is not a feature, but there are several features in Workhorse itself is not a feature, but there are several features in
GitLab that would not work efficiently without Workhorse. GitLab that would not work efficiently without Workhorse.
To put the efficiency benefit in context, consider that in 2020Q3 on To put the efficiency benefit in context, consider that in 2020Q3 on
GitLab.com [we see][thanos] Rails application threads using on average GitLab.com [we see][https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=sum(ruby_process_resident_memory_bytes%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)%20%2F%20sum(puma_max_threads%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)&g0.tab=1&g1.range_input=1h&g1.max_source_resolution=0s&g1.expr=sum(go_memstats_sys_bytes%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)%2Fsum(go_goroutines%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)&g1.tab=1]
Rails application threads using on average
about 200MB of RSS vs about 200KB for Workhorse goroutines. about 200MB of RSS vs about 200KB for Workhorse goroutines.
Examples of features that rely on Workhorse: Examples of features that rely on Workhorse:
...@@ -63,7 +70,4 @@ memory than it costs to have Workhorse look after it. ...@@ -63,7 +70,4 @@ memory than it costs to have Workhorse look after it.
- Workhorse does not clean up idle client connections. - Workhorse does not clean up idle client connections.
- We assume that all requests to Rails pass through Workhorse. - We assume that all requests to Rails pass through Workhorse.
For more information see ['A brief history of GitLab Workhorse'][brief-history-blog]. For more information see ['A brief history of GitLab Workhorse'](https://about.gitlab.com/2016/04/12/a-brief-history-of-gitlab-workhorse/).
[thanos]: https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=sum(ruby_process_resident_memory_bytes%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)%20%2F%20sum(puma_max_threads%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)&g0.tab=1&g1.range_input=1h&g1.max_source_resolution=0s&g1.expr=sum(go_memstats_sys_bytes%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)%2Fsum(go_goroutines%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)&g1.tab=1
[brief-history-blog]: https://about.gitlab.com/2016/04/12/a-brief-history-of-gitlab-workhorse/
## Adding new features to Workhorse ---
stage: Create
group: Source Code
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Adding new features to Workhorse
GitLab Workhorse is a smart reverse proxy for GitLab. It handles GitLab Workhorse is a smart reverse proxy for GitLab. It handles
"long" HTTP requests such as file downloads, file uploads, Git "long" HTTP requests such as file downloads, file uploads, Git
...@@ -16,14 +22,14 @@ We suggest to follow this route only if absolutely necessary and no other option ...@@ -16,14 +22,14 @@ We suggest to follow this route only if absolutely necessary and no other option
Splitting a feature between the Rails code-base and Workhorse is deliberately choosing to introduce technical debt. It adds complexity to the system and coupling between the two components. Splitting a feature between the Rails code-base and Workhorse is deliberately choosing to introduce technical debt. It adds complexity to the system and coupling between the two components.
* Building features using Workhorse has a considerable complexity cost, so you should prefer designs based on Rails requests and Sidekiq jobs. - Building features using Workhorse has a considerable complexity cost, so you should prefer designs based on Rails requests and Sidekiq jobs.
* Even when using Rails+Sidekiq is "more work" than using Rails+Workhorse, Rails+Sidekiq is easier to maintain in the long term because Workhorse is unique to GitLab while Rails+Sidekiq is an industry standard. - Even when using Rails+Sidekiq is "more work" than using Rails+Workhorse, Rails+Sidekiq is easier to maintain in the long term because Workhorse is unique to GitLab while Rails+Sidekiq is an industry standard.
* For "global" behaviors around web requests consider using a Rack middleware instead of Workhorse. - For "global" behaviors around web requests consider using a Rack middleware instead of Workhorse.
* Generally speaking, we should only use Rails+Workhorse if the HTTP client expects behavior that is not reasonable to implement in Rails, like "long" requests. - Generally speaking, we should only use Rails+Workhorse if the HTTP client expects behavior that is not reasonable to implement in Rails, like "long" requests.
## What is a "long" request? ## What is a "long" request?
There is one order of magnitude between Workhorse and puma RAM usage. Having connection open for a period longer than milliseconds is a problem because of the amount of RAM it monopolizes once it reaches the Ruby on Rails controller. There is one order of magnitude between Workhorse and Puma RAM usage. Having connection open for a period longer than milliseconds is a problem because of the amount of RAM it monopolizes once it reaches the Ruby on Rails controller.
So far we identified two classes of "long" requests: data transfers and HTTP long polling. So far we identified two classes of "long" requests: data transfers and HTTP long polling.
...@@ -37,5 +43,4 @@ You can watch the recording for more details on the history of Workhorse and the ...@@ -37,5 +43,4 @@ You can watch the recording for more details on the history of Workhorse and the
[Uploads development documentation]( https://docs.gitlab.com/ee/development/uploads.html) [Uploads development documentation]( https://docs.gitlab.com/ee/development/uploads.html)
contains the most common use-cases for adding a new type of upload and may answer all of your questions. contains the most common use-cases for adding a new type of upload and may answer all of your questions.
If you still think we should add a new feature to Workhorse, please open an issue explaining **what you want to implement** and **why it can't be implemented in our ruby code-base**. Workhorse maintainers will be happy to help you assessing the situation. If you still think we should add a new feature to Workhorse, please open an issue explaining **what you want to implement** and **why it can't be implemented in our Ruby code-base**. Workhorse maintainers will be happy to help you assessing the situation.
---
stage: Create
group: Source Code
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Testing your code # Testing your code
Run the tests with: Run the tests with:
``` ```plaintext
make clean test make clean test
``` ```
......
---
stage: Create
group: Source Code
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Workhorse configuration # Workhorse configuration
For historical reasons Workhorse uses both command line flags, a configuration file and environment variables. For historical reasons Workhorse uses both command line flags, a configuration file and environment variables.
...@@ -6,7 +12,7 @@ All new configuration options that get added to Workhorse should go into the con ...@@ -6,7 +12,7 @@ All new configuration options that get added to Workhorse should go into the con
## CLI options ## CLI options
``` ```plaintext
gitlab-workhorse [OPTIONS] gitlab-workhorse [OPTIONS]
Options: Options:
...@@ -60,10 +66,10 @@ HTTP. ...@@ -60,10 +66,10 @@ HTTP.
GitLab Workhorse can listen on either a TCP or a Unix domain socket. It GitLab Workhorse can listen on either a TCP or a Unix domain socket. It
can also open a second listening TCP listening socket with the Go can also open a second listening TCP listening socket with the Go
[net/http/pprof profiler server](http://golang.org/pkg/net/http/pprof/). [`net/http/pprof` profiler server](http://golang.org/pkg/net/http/pprof/).
GitLab Workhorse can listen on redis events (currently only builds/register GitLab Workhorse can listen on Redis events (currently only builds/register
for runners). This requires you to pass a valid TOML config file via for runners). This requires you to pass a valid TOML configuration file via
`-config` flag. `-config` flag.
For regular setups it only requires the following (replacing the string For regular setups it only requires the following (replacing the string
with the actual socket) with the actual socket)
...@@ -73,19 +79,19 @@ with the actual socket) ...@@ -73,19 +79,19 @@ with the actual socket)
GitLab Workhorse integrates with Redis to do long polling for CI build GitLab Workhorse integrates with Redis to do long polling for CI build
requests. This is configured via two things: requests. This is configured via two things:
- Redis settings in the TOML config file - Redis settings in the TOML configuration file
- The `-apiCiLongPollingDuration` command line flag to control polling - The `-apiCiLongPollingDuration` command line flag to control polling
behavior for CI build requests behavior for CI build requests
It is OK to enable Redis in the config file but to leave CI polling It is OK to enable Redis in the configuration file but to leave CI polling
disabled; this just results in an idle Redis pubsub connection. The disabled; this just results in an idle Redis pubsub connection. The
opposite is not possible: CI long polling requires a correct Redis opposite is not possible: CI long polling requires a correct Redis
configuration. configuration.
Below we discuss the options for the `[redis]` section in the config Below we discuss the options for the `[redis]` section in the configuration
file. file.
``` ```plaintext
[redis] [redis]
URL = "unix:///var/run/gitlab/redis.sock" URL = "unix:///var/run/gitlab/redis.sock"
Password = "my_awesome_password" Password = "my_awesome_password"
...@@ -95,12 +101,15 @@ SentinelMaster = "mymaster" ...@@ -95,12 +101,15 @@ SentinelMaster = "mymaster"
- `URL` takes a string in the format `unix://path/to/redis.sock` or - `URL` takes a string in the format `unix://path/to/redis.sock` or
`tcp://host:port`. `tcp://host:port`.
- `Password` is only required if your redis instance is password-protected - `Password` is only required if your Redis instance is password-protected
- `Sentinel` is used if you are using Sentinel. - `Sentinel` is used if you are using Sentinel.
*NOTE* that if both `Sentinel` and `URL` are given, only `Sentinel` will be used
NOTE:
If both `Sentinel` and `URL` are given, only `Sentinel` will be used.
Optional fields are as follows: Optional fields are as follows:
```
```plaintext
[redis] [redis]
DB = 0 DB = 0
MaxIdle = 1 MaxIdle = 1
...@@ -108,7 +117,7 @@ MaxActive = 1 ...@@ -108,7 +117,7 @@ MaxActive = 1
``` ```
- `DB` is the Database to connect to. Defaults to `0` - `DB` is the Database to connect to. Defaults to `0`
- `MaxIdle` is how many idle connections can be in the redis-pool at once. Defaults to 1 - `MaxIdle` is how many idle connections can be in the Redis pool at once. Defaults to 1
- `MaxActive` is how many connections the pool can keep. Defaults to 1 - `MaxActive` is how many connections the pool can keep. Defaults to 1
## Relative URL support ## Relative URL support
...@@ -117,7 +126,7 @@ If you are mounting GitLab at a relative URL, e.g. ...@@ -117,7 +126,7 @@ If you are mounting GitLab at a relative URL, e.g.
`example.com/gitlab`, then you should also use this relative URL in `example.com/gitlab`, then you should also use this relative URL in
the `authBackend` setting: the `authBackend` setting:
``` ```plaintext
gitlab-workhorse -authBackend http://localhost:8080/gitlab gitlab-workhorse -authBackend http://localhost:8080/gitlab
``` ```
...@@ -151,7 +160,7 @@ development. ...@@ -151,7 +160,7 @@ development.
Omnibus (`/etc/gitlab/gitlab.rb`): Omnibus (`/etc/gitlab/gitlab.rb`):
``` ```ruby
gitlab_workhorse['env'] = { gitlab_workhorse['env'] = {
'GITLAB_WORKHORSE_SENTRY_DSN' => 'https://foobar' 'GITLAB_WORKHORSE_SENTRY_DSN' => 'https://foobar'
'GITLAB_WORKHORSE_SENTRY_ENVIRONMENT' => 'production' 'GITLAB_WORKHORSE_SENTRY_ENVIRONMENT' => 'production'
...@@ -160,16 +169,16 @@ gitlab_workhorse['env'] = { ...@@ -160,16 +169,16 @@ gitlab_workhorse['env'] = {
Source installations (`/etc/default/gitlab`): Source installations (`/etc/default/gitlab`):
``` ```plaintext
export GITLAB_WORKHORSE_SENTRY_DSN='https://foobar' export GITLAB_WORKHORSE_SENTRY_DSN='https://foobar'
export GITLAB_WORKHORSE_SENTRY_ENVIRONMENT='production' export GITLAB_WORKHORSE_SENTRY_ENVIRONMENT='production'
``` ```
## Distributed Tracing ## Distributed Tracing
Workhorse supports distributed tracing through [LabKit][] using [OpenTracing APIs](https://opentracing.io). Workhorse supports distributed tracing through [LabKit](https://gitlab.com/gitlab-org/labkit/) using [OpenTracing APIs](https://opentracing.io).
By default, no tracing implementation is linked into the binary, but different OpenTracing providers can be linked in using [build tags][build-tags]/[build constraints][build-tags]. This can be done by setting the `BUILD_TAGS` make variable. By default, no tracing implementation is linked into the binary, but different OpenTracing providers can be linked in using [build tags](https://golang.org/pkg/go/build/#hdr-Build_Constraints) or build constraints. This can be done by setting the `BUILD_TAGS` make variable.
For more details of the supported providers, see LabKit, but as an example, for Jaeger tracing support, include the tags: `BUILD_TAGS="tracer_static tracer_static_jaeger"`. For more details of the supported providers, see LabKit, but as an example, for Jaeger tracing support, include the tags: `BUILD_TAGS="tracer_static tracer_static_jaeger"`.
...@@ -187,9 +196,9 @@ GITLAB_TRACING=opentracing://jaeger ./gitlab-workhorse ...@@ -187,9 +196,9 @@ GITLAB_TRACING=opentracing://jaeger ./gitlab-workhorse
## Continuous Profiling ## Continuous Profiling
Workhorse supports continuous profiling through [LabKit][] using [Stackdriver Profiler](https://cloud.google.com/profiler). Workhorse supports continuous profiling through [LabKit](https://gitlab.com/gitlab-org/labkit/) using [Stackdriver Profiler](https://cloud.google.com/profiler).
By default, the Stackdriver Profiler implementation is linked in the binary using [build tags][build-tags], though it's not By default, the Stackdriver Profiler implementation is linked in the binary using [build tags](https://golang.org/pkg/go/build/#hdr-Build_Constraints), though it's not
required and can be skipped. required and can be skipped.
For example: For example:
...@@ -207,7 +216,4 @@ For example: ...@@ -207,7 +216,4 @@ For example:
GITLAB_CONTINUOUS_PROFILING="stackdriver?service=workhorse&service_version=1.0.1&project_id=test-123 ./gitlab-workhorse" GITLAB_CONTINUOUS_PROFILING="stackdriver?service=workhorse&service_version=1.0.1&project_id=test-123 ./gitlab-workhorse"
``` ```
More information about see the [LabKit monitoring docs](https://gitlab.com/gitlab-org/labkit/-/blob/master/monitoring/doc.go). More information about see the [LabKit monitoring documentation](https://gitlab.com/gitlab-org/labkit/-/blob/master/monitoring/doc.go).
[LabKit]: https://gitlab.com/gitlab-org/labkit/
[build-tags]: https://golang.org/pkg/go/build/#hdr-Build_Constraints
...@@ -6,13 +6,13 @@ Make](https://www.gnu.org/software/make/). ...@@ -6,13 +6,13 @@ Make](https://www.gnu.org/software/make/).
To install into `/usr/local/bin` run `make install`. To install into `/usr/local/bin` run `make install`.
``` ```plaintext
make install make install
``` ```
To install into `/foo/bin` set the PREFIX variable. To install into `/foo/bin` set the PREFIX variable.
``` ```plaintext
make install PREFIX=/foo make install PREFIX=/foo
``` ```
...@@ -26,19 +26,19 @@ On some operating systems, such as FreeBSD, you may have to use ...@@ -26,19 +26,19 @@ On some operating systems, such as FreeBSD, you may have to use
### Exiftool ### Exiftool
Workhorse uses [exiftool](https://www.sno.phy.queensu.ca/~phil/exiftool/) for Workhorse uses [Exiftool](https://www.sno.phy.queensu.ca/~phil/exiftool/) for
removing EXIF data (which may contain sensitive information) from uploaded removing EXIF data (which may contain sensitive information) from uploaded
images. If you installed GitLab: images. If you installed GitLab:
- Using the Omnibus package, you're all set. - Using the Omnibus package, you're all set.
*NOTE* that if you are using CentOS Minimal, you may need to install `perl` *NOTE* that if you are using CentOS Minimal, you may need to install `perl`
package: `yum install perl` package: `yum install perl`
- From source, make sure `exiftool` is installed: - From source, make sure `exiftool` is installed:
```sh ```shell
# Debian/Ubuntu # Debian/Ubuntu
sudo apt-get install libimage-exiftool-perl sudo apt-get install libimage-exiftool-perl
# RHEL/CentOS # RHEL/CentOS
sudo yum install perl-Image-ExifTool sudo yum install perl-Image-ExifTool
``` ```
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment