Commit 68a56658 authored by Kushal Pandya's avatar Kushal Pandya

Merge branch '354845-aqualls-workhorse1' into 'master'

Workhorse docs: fix linting errors

See merge request gitlab-org/gitlab!83782
parents e11d499a 885e674d
---
stage: Create
group: Source Code
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# GitLab Workhorse
GitLab Workhorse is a smart reverse proxy for GitLab. It handles
......@@ -10,26 +16,25 @@ GitLab](doc/architecture/gitlab_features.md) that would not work efficiently wit
## Canonical source
The canonical source for Workhorse is
[gitlab-org/gitlab/workhorse](https://gitlab.com/gitlab-org/gitlab/tree/master/workhorse).
Prior to https://gitlab.com/groups/gitlab-org/-/epics/4826, it was
[gitlab-org/gitlab-workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse/tree/master),
[`gitlab-org/gitlab/workhorse`](https://gitlab.com/gitlab-org/gitlab/tree/master/workhorse).
Prior to [epic #4826](https://gitlab.com/groups/gitlab-org/-/epics/4826), it was
[`gitlab-org/gitlab-workhorse`](https://gitlab.com/gitlab-org/gitlab-workhorse/tree/master),
but that repository is no longer used for development.
## Documentation
Workhorse documentation is available in the [`doc` folder of this repository](doc/).
* Architectural overview
* [GitLab features that rely on Workhorse](doc/architecture/gitlab_features.md)
* [Websocket channel support](doc/architecture/channel.md)
* Operating Workhorse
* [Source installation](doc/operations/install.md)
* [Workhorse configuration](doc/operations/configuration.md)
* [Contributing](CONTRIBUTING.md)
* [Adding new features](doc/development/new_features.md)
* [Testing your code](doc/development/tests.md)
- Architectural overview
- [GitLab features that rely on Workhorse](doc/architecture/gitlab_features.md)
- [Websocket channel support](doc/architecture/channel.md)
- Operating Workhorse
- [Source installation](doc/operations/install.md)
- [Workhorse configuration](doc/operations/configuration.md)
- [Contributing](CONTRIBUTING.md)
- [Adding new features](doc/development/new_features.md)
- [Testing your code](doc/development/tests.md)
## License
This code is distributed under the MIT license, see the [LICENSE](LICENSE) file.
---
stage: Create
group: Source Code
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Websocket channel support
In some cases, GitLab can provide in-browser terminal access to an
......@@ -22,7 +28,7 @@ responses. WebSocket URLs have schemes like `ws://` (unencrypted) or
When requesting an upgrade to WebSocket, the browser sends a HTTP/1.1
request that looks like this:
```
```plaintext
GET /path.ws HTTP/1.1
Connection: upgrade
Upgrade: websocket
......@@ -116,10 +122,10 @@ This returns a JSON response containing details of where the
terminal can be found, and how to connect it. In particular,
the following details are returned in case of success:
* WebSocket URL to **connect** to, e.g.: `wss://example.com/terminals/1.ws?tty=1`
* WebSocket subprotocols to support, e.g.: `["channel.k8s.io"]`
* Headers to send, e.g.: `Authorization: Token xxyyz..`
* Certificate authority to verify `wss` connections with (optional)
- WebSocket URL to **connect** to, e.g.: `wss://example.com/terminals/1.ws?tty=1`
- WebSocket subprotocols to support, e.g.: `["channel.k8s.io"]`
- Headers to send, e.g.: `Authorization: Token xxyyz..`
- Certificate authority to verify `wss` connections with (optional)
Workhorse periodically re-checks this endpoint, and if it gets an
error response, or the details of the terminal change, it will
......@@ -186,9 +192,8 @@ written to that fd.
Also used by Kubernetes, this subprotocol defines a similar multiplexed
channel to `channel.k8s.io`. The main differences are:
* `TextMessage` frames are valid, rather than `BinaryMessage` frames.
* The first byte of each `TextMessage` frame represents the file
- `TextMessage` frames are valid, rather than `BinaryMessage` frames.
- The first byte of each `TextMessage` frame represents the file
descriptor as a numeric UTF-8 character, so the character `U+0030`,
or "0", is fd 0, STDIN).
* The remaining bytes represent base64-encoded arbitrary data.
- The remaining bytes represent base64-encoded arbitrary data.
---
stage: Create
group: Source Code
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Features that rely on Workhorse
Workhorse itself is not a feature, but there are several features in
GitLab that would not work efficiently without Workhorse.
To put the efficiency benefit in context, consider that in 2020Q3 on
GitLab.com [we see][thanos] Rails application threads using on average
GitLab.com [we see][https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=sum(ruby_process_resident_memory_bytes%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)%20%2F%20sum(puma_max_threads%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)&g0.tab=1&g1.range_input=1h&g1.max_source_resolution=0s&g1.expr=sum(go_memstats_sys_bytes%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)%2Fsum(go_goroutines%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)&g1.tab=1]
Rails application threads using on average
about 200MB of RSS vs about 200KB for Workhorse goroutines.
Examples of features that rely on Workhorse:
......@@ -63,7 +70,4 @@ memory than it costs to have Workhorse look after it.
- Workhorse does not clean up idle client connections.
- We assume that all requests to Rails pass through Workhorse.
For more information see ['A brief history of GitLab Workhorse'][brief-history-blog].
[thanos]: https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=sum(ruby_process_resident_memory_bytes%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)%20%2F%20sum(puma_max_threads%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)&g0.tab=1&g1.range_input=1h&g1.max_source_resolution=0s&g1.expr=sum(go_memstats_sys_bytes%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)%2Fsum(go_goroutines%7Bapp%3D%22webservice%22%2Cenv%3D%22gprd%22%2Crelease%3D%22gitlab%22%7D)&g1.tab=1
[brief-history-blog]: https://about.gitlab.com/2016/04/12/a-brief-history-of-gitlab-workhorse/
For more information see ['A brief history of GitLab Workhorse'](https://about.gitlab.com/2016/04/12/a-brief-history-of-gitlab-workhorse/).
## Adding new features to Workhorse
---
stage: Create
group: Source Code
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Adding new features to Workhorse
GitLab Workhorse is a smart reverse proxy for GitLab. It handles
"long" HTTP requests such as file downloads, file uploads, Git
......@@ -16,14 +22,14 @@ We suggest to follow this route only if absolutely necessary and no other option
Splitting a feature between the Rails code-base and Workhorse is deliberately choosing to introduce technical debt. It adds complexity to the system and coupling between the two components.
* Building features using Workhorse has a considerable complexity cost, so you should prefer designs based on Rails requests and Sidekiq jobs.
* Even when using Rails+Sidekiq is "more work" than using Rails+Workhorse, Rails+Sidekiq is easier to maintain in the long term because Workhorse is unique to GitLab while Rails+Sidekiq is an industry standard.
* For "global" behaviors around web requests consider using a Rack middleware instead of Workhorse.
* Generally speaking, we should only use Rails+Workhorse if the HTTP client expects behavior that is not reasonable to implement in Rails, like "long" requests.
- Building features using Workhorse has a considerable complexity cost, so you should prefer designs based on Rails requests and Sidekiq jobs.
- Even when using Rails+Sidekiq is "more work" than using Rails+Workhorse, Rails+Sidekiq is easier to maintain in the long term because Workhorse is unique to GitLab while Rails+Sidekiq is an industry standard.
- For "global" behaviors around web requests consider using a Rack middleware instead of Workhorse.
- Generally speaking, we should only use Rails+Workhorse if the HTTP client expects behavior that is not reasonable to implement in Rails, like "long" requests.
## What is a "long" request?
There is one order of magnitude between Workhorse and puma RAM usage. Having connection open for a period longer than milliseconds is a problem because of the amount of RAM it monopolizes once it reaches the Ruby on Rails controller.
There is one order of magnitude between Workhorse and Puma RAM usage. Having connection open for a period longer than milliseconds is a problem because of the amount of RAM it monopolizes once it reaches the Ruby on Rails controller.
So far we identified two classes of "long" requests: data transfers and HTTP long polling.
......@@ -37,5 +43,4 @@ You can watch the recording for more details on the history of Workhorse and the
[Uploads development documentation]( https://docs.gitlab.com/ee/development/uploads.html)
contains the most common use-cases for adding a new type of upload and may answer all of your questions.
If you still think we should add a new feature to Workhorse, please open an issue explaining **what you want to implement** and **why it can't be implemented in our ruby code-base**. Workhorse maintainers will be happy to help you assessing the situation.
If you still think we should add a new feature to Workhorse, please open an issue explaining **what you want to implement** and **why it can't be implemented in our Ruby code-base**. Workhorse maintainers will be happy to help you assessing the situation.
---
stage: Create
group: Source Code
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Testing your code
Run the tests with:
```
```plaintext
make clean test
```
......
---
stage: Create
group: Source Code
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
---
# Workhorse configuration
For historical reasons Workhorse uses both command line flags, a configuration file and environment variables.
......@@ -6,7 +12,7 @@ All new configuration options that get added to Workhorse should go into the con
## CLI options
```
```plaintext
gitlab-workhorse [OPTIONS]
Options:
......@@ -60,10 +66,10 @@ HTTP.
GitLab Workhorse can listen on either a TCP or a Unix domain socket. It
can also open a second listening TCP listening socket with the Go
[net/http/pprof profiler server](http://golang.org/pkg/net/http/pprof/).
[`net/http/pprof` profiler server](http://golang.org/pkg/net/http/pprof/).
GitLab Workhorse can listen on redis events (currently only builds/register
for runners). This requires you to pass a valid TOML config file via
GitLab Workhorse can listen on Redis events (currently only builds/register
for runners). This requires you to pass a valid TOML configuration file via
`-config` flag.
For regular setups it only requires the following (replacing the string
with the actual socket)
......@@ -73,19 +79,19 @@ with the actual socket)
GitLab Workhorse integrates with Redis to do long polling for CI build
requests. This is configured via two things:
- Redis settings in the TOML config file
- Redis settings in the TOML configuration file
- The `-apiCiLongPollingDuration` command line flag to control polling
behavior for CI build requests
It is OK to enable Redis in the config file but to leave CI polling
It is OK to enable Redis in the configuration file but to leave CI polling
disabled; this just results in an idle Redis pubsub connection. The
opposite is not possible: CI long polling requires a correct Redis
configuration.
Below we discuss the options for the `[redis]` section in the config
Below we discuss the options for the `[redis]` section in the configuration
file.
```
```plaintext
[redis]
URL = "unix:///var/run/gitlab/redis.sock"
Password = "my_awesome_password"
......@@ -95,12 +101,15 @@ SentinelMaster = "mymaster"
- `URL` takes a string in the format `unix://path/to/redis.sock` or
`tcp://host:port`.
- `Password` is only required if your redis instance is password-protected
- `Password` is only required if your Redis instance is password-protected
- `Sentinel` is used if you are using Sentinel.
*NOTE* that if both `Sentinel` and `URL` are given, only `Sentinel` will be used
NOTE:
If both `Sentinel` and `URL` are given, only `Sentinel` will be used.
Optional fields are as follows:
```
```plaintext
[redis]
DB = 0
MaxIdle = 1
......@@ -108,7 +117,7 @@ MaxActive = 1
```
- `DB` is the Database to connect to. Defaults to `0`
- `MaxIdle` is how many idle connections can be in the redis-pool at once. Defaults to 1
- `MaxIdle` is how many idle connections can be in the Redis pool at once. Defaults to 1
- `MaxActive` is how many connections the pool can keep. Defaults to 1
## Relative URL support
......@@ -117,7 +126,7 @@ If you are mounting GitLab at a relative URL, e.g.
`example.com/gitlab`, then you should also use this relative URL in
the `authBackend` setting:
```
```plaintext
gitlab-workhorse -authBackend http://localhost:8080/gitlab
```
......@@ -151,7 +160,7 @@ development.
Omnibus (`/etc/gitlab/gitlab.rb`):
```
```ruby
gitlab_workhorse['env'] = {
'GITLAB_WORKHORSE_SENTRY_DSN' => 'https://foobar'
'GITLAB_WORKHORSE_SENTRY_ENVIRONMENT' => 'production'
......@@ -160,16 +169,16 @@ gitlab_workhorse['env'] = {
Source installations (`/etc/default/gitlab`):
```
```plaintext
export GITLAB_WORKHORSE_SENTRY_DSN='https://foobar'
export GITLAB_WORKHORSE_SENTRY_ENVIRONMENT='production'
```
## Distributed Tracing
Workhorse supports distributed tracing through [LabKit][] using [OpenTracing APIs](https://opentracing.io).
Workhorse supports distributed tracing through [LabKit](https://gitlab.com/gitlab-org/labkit/) using [OpenTracing APIs](https://opentracing.io).
By default, no tracing implementation is linked into the binary, but different OpenTracing providers can be linked in using [build tags][build-tags]/[build constraints][build-tags]. This can be done by setting the `BUILD_TAGS` make variable.
By default, no tracing implementation is linked into the binary, but different OpenTracing providers can be linked in using [build tags](https://golang.org/pkg/go/build/#hdr-Build_Constraints) or build constraints. This can be done by setting the `BUILD_TAGS` make variable.
For more details of the supported providers, see LabKit, but as an example, for Jaeger tracing support, include the tags: `BUILD_TAGS="tracer_static tracer_static_jaeger"`.
......@@ -187,9 +196,9 @@ GITLAB_TRACING=opentracing://jaeger ./gitlab-workhorse
## Continuous Profiling
Workhorse supports continuous profiling through [LabKit][] using [Stackdriver Profiler](https://cloud.google.com/profiler).
Workhorse supports continuous profiling through [LabKit](https://gitlab.com/gitlab-org/labkit/) using [Stackdriver Profiler](https://cloud.google.com/profiler).
By default, the Stackdriver Profiler implementation is linked in the binary using [build tags][build-tags], though it's not
By default, the Stackdriver Profiler implementation is linked in the binary using [build tags](https://golang.org/pkg/go/build/#hdr-Build_Constraints), though it's not
required and can be skipped.
For example:
......@@ -207,7 +216,4 @@ For example:
GITLAB_CONTINUOUS_PROFILING="stackdriver?service=workhorse&service_version=1.0.1&project_id=test-123 ./gitlab-workhorse"
```
More information about see the [LabKit monitoring docs](https://gitlab.com/gitlab-org/labkit/-/blob/master/monitoring/doc.go).
[LabKit]: https://gitlab.com/gitlab-org/labkit/
[build-tags]: https://golang.org/pkg/go/build/#hdr-Build_Constraints
More information about see the [LabKit monitoring documentation](https://gitlab.com/gitlab-org/labkit/-/blob/master/monitoring/doc.go).
......@@ -6,13 +6,13 @@ Make](https://www.gnu.org/software/make/).
To install into `/usr/local/bin` run `make install`.
```
```plaintext
make install
```
To install into `/foo/bin` set the PREFIX variable.
```
```plaintext
make install PREFIX=/foo
```
......@@ -26,7 +26,7 @@ On some operating systems, such as FreeBSD, you may have to use
### Exiftool
Workhorse uses [exiftool](https://www.sno.phy.queensu.ca/~phil/exiftool/) for
Workhorse uses [Exiftool](https://www.sno.phy.queensu.ca/~phil/exiftool/) for
removing EXIF data (which may contain sensitive information) from uploaded
images. If you installed GitLab:
......@@ -35,7 +35,7 @@ images. If you installed GitLab:
package: `yum install perl`
- From source, make sure `exiftool` is installed:
```sh
```shell
# Debian/Ubuntu
sudo apt-get install libimage-exiftool-perl
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment