Development guide for Kubernetes

Start writing some development guidelines in relation to our Kubernetes
integration

doc/development/README.md

@@ -49,6 +49,7 @@ description: 'Learn how to contribute to GitLab.'
 - [Working with the GitHub importer](github_importer.md)
 - [Import/Export development documentation](import_export.md)
 - [Working with Merge Request diffs](diffs.md)
+- [Kubernetes integration guidelines](kubernetes.md)
 - [Permissions](permissions.md)
 - [Prometheus metrics](prometheus_metrics.md)
 - [Guidelines for reusing abstractions](reusing_abstractions.md)

doc/development/kubernetes.md

+# Kubernetes integration - development guidelines
+
+This document provides various guidelines when developing for GitLab's
+[Kubernetes integration](../user/project/clusters/index.md).
+
+## Development
+
+### Architecture
+
+Some Kubernetes operations, such as creating restricted project
+namespaces are performed on the GitLab Rails application. These
+operations are performed using a [client library](#client-library).
+These operations will carry an element of risk as the operations will be
+run as the same user running the GitLab Rails application, see the
+[security](#security) section below.
+
+Some Kubernetes operations, such as installing cluster applications are
+performed on one-off pods on the Kubernetes cluster itself. These
+installation pods are currently named `install-<application_name>` and
+are created within the `gitlab-managed-apps` namespace.
+
+In terms of code organization, we generally add objects that represent
+Kubernetes resources in
+[`lib/gitlab/kubernetes`](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/lib/gitlab/kubernetes).
+
+### Client library
+
+We use the [`kubeclient`](https://rubygems.org/gems/kubeclient) gem to
+perform Kubernetes API calls. As the `kubeclient` gem does not support
+different API Groups (e.g. `apis/rbac.authorization.k8s.io`) from a
+single client, we have created a wrapper class,
+[`Gitlab::Kubernetes::KubeClient`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/kubernetes/kube_client.rb)
+that will enable you to achieve this.
+
+Selected Kubernetes API groups are currently supported. Do add support
+for new API groups or methods to
+[`Gitlab::Kubernetes::KubeClient`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/kubernetes/kube_client.rb)
+if you need to use them. New API groups or API group versions can be
+added to `SUPPORTED_API_GROUPS` - internally, this will create an
+internal client for that group. New methods can be added as a delegation
+to the relevant internal client.
+
+### Performance considerations
+
+All calls to the Kubernetes API must be in a background process. Do not
+perform Kubernetes API calls within a web request as this will block
+unicorn and can easily lead to a Denial Of Service (DoS) attack in GitLab as
+the Kubernetes cluster response times are outside of our control.
+
+The easiest way to ensure your calls happen a background process is to
+delegate any such work to happen in a [sidekiq
+worker](sidekiq_style_guide.md).
+
+There are instances where you would like to make calls to Kubernetes and
+return the response and as such a background worker does not seem to be
+a good fit. For such cases you should make use of [reactive
+caching](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/models/concerns/reactive_caching.rb).
+For example:
+
+```ruby
+  def calculate_reactive_cache!
+    { pods: cluster.platform_kubernetes.kubeclient.get_pods }
+  end
+
+  def pods
+    with_reactive_cache do |data|
+      data[:pods]
+    end
+  end
+```
+
+### Testing
+
+We have some Webmock stubs in
+[`KubernetesHelpers`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/support/helpers/kubernetes_helpers.rb)
+which can help with mocking out calls to Kubernetes API in your tests.
+
+## Security
+
+### SSRF
+
+As URLs for Kubernetes clusters are user controlled it is easily
+susceptible to Server Side Request Forgery (SSRF) attacks. You should
+understand the mitigation strategies if you are adding more API calls to
+a cluster.
+
+Mitigation strategies include:
+
+1. Not allowing redirects to attacker controller resources:
+   [`Kubeclient::KubeClient`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/kubernetes/kube_client.rb#)
+   can be configured to disallow any redirects by passing in
+   `http_max_redirects: 0` as an option.
+1. Not exposing error messages: by doing so, we
+   prevent attackers from triggering errors to expose results from
+   attacker controlled requests. For example, we do not expose (or store)
+   raw error messages:
+
+   ```ruby
+   rescue Kubernetes::HttpError => e
+     # bad
+     # app.make_errored!("Kubernetes error: #{e.message}")
+
+     # good
+     app.make_errored!("Kubernetes error: #{e.error_code}")
+   ```
+
+## Debugging
+
+Logs related to the Kubernetes integration can be found in
+[kubernetes.log](../administration/logs.md#kuberneteslog). On a local
+GDK install, this will be present in `log/kubernetes.log`.
+
+Some services such as
+[`Clusters::Applications::InstallService`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/services/clusters/applications/install_service.rb#L18)
+rescues `StandardError` which can make it harder to debug issues in an
+development environment. The current workaround is to temporarily
+comment out the `rescue` in your local development source.
+
+You can also follow the installation pod logs to debug issues related to
+installation. Once the installation/upgrade is underway, wait for the
+pod to be created. Then run the following to obtain the pods logs as
+they are written:
+
+```bash
+kubectl logs <pod_name> --follow -n gitlab-managed-apps
+```