Merge branch 'fern-defend-tutorial' into 'master'

Create Guide on installation/usage of Defend WAF

See merge request gitlab-org/gitlab!18676

doc/topics/web_application_firewall/index.md

+# Web Application Firewall - ModSecurity
+
+A web application firewall (or WAF) filters, monitors, and blocks HTTP traffic to
+and from a web application. By inspecting HTTP traffic, it can prevent attacks
+stemming from web application security flaws. It can be used to detect SQL injection,
+Cross-Site Scripting (XSS), Remote File Inclusion, Security Misconfigurations, and
+much more.
+
+## Overview
+
+GitLab provides a WAF out of the box after Ingress is deployed.
+All you need to do is deploy your application along with a service
+and Ingress resource.
+
+In GitLab's [Ingress](../../user/clusters/applications.md#ingress) deployment, the [ModSecurity](https://modsecurity.org/) module is loaded
+into Ingress-NGINX by default and monitors the traffic going to the
+applications which have an Ingress.
+
+The ModSecurity module runs with the [OWASP Core Rule Set (CRS)](https://coreruleset.org/) by default. The OWASP CRS will detect and log a wide range of common attacks.
+
+NOTE: **Note**
+The WAF is deployed in "Detection-only mode" by default and will only log attack
+attempts.
+
+## Requirements
+
+The Web Application Firewall requires:
+
+- **Kubernetes**
+
+  To enable the WAF, you need:
+
+  - Kubernetes 1.12+.
+  - A load balancer. You can use NGINX-Ingress by deploying it to your
+    Kubernetes cluster by either:
+    - Using the [`nginx-ingress` Helm chart](https://github.com/helm/charts/tree/master/stable/nginx-ingress).
+    - Installing the [Ingress GitLab Managed App](../../user/clusters/applications.md#ingress) with WAF enabled.
+
+- **Configured Kubernetes objects**
+
+  To use the WAF on an application, you need to deploy the following Kubernetes resources:
+
+  - [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/)
+  - [Service](https://kubernetes.io/docs/concepts/services-networking/service/)
+  - [Ingress Resource](https://kubernetes.io/docs/concepts/services-networking/ingress/)
+
+## Quick start
+
+If you are using GitLab.com, see the [quick start guide](quick_start_guide.md) for
+how to use the WAF with GitLab.com and a Kubernetes cluster on Google Kubernetes Engine (GKE).
+
+If you are using a self-hosted instance of GitLab, you need to configure the
+[Google OAuth2 OmniAuth Provider](../../integration/google.md) before
+you can configure a cluster on GKE. Once this is set up, you can follow the steps on the [quick start guide](quick_start_guide.md) to get started.
+
+NOTE: **Note**
+This guide shows how the WAF can be deployed using Auto DevOps. The WAF
+is avaliable by default to all applications no matter how they are deployed,
+as long as they are using Ingress.
+
+## Network firewall vs. Web Application Firewall
+
+A network firewall or packet filter looks at traffic at the Network (L3) and Transport (L4) layers
+of the [OSI Model](https://en.wikipedia.org/wiki/OSI_model), and denies packets from entry based on
+a set of rules regarding the network in general.
+
+A Web Application Firewall operates at the Application (L7) layer of the OSI Model and can
+examine all the packets traveling to and from a specific application. A WAF can set
+more advanced rules around threat detection.
+
+## Features
+
+ModSecurity is enabled with the [OWASP Core Rule Set (CRS)](https://modsecurity.org/crs/) by
+default. The OWASP CRS logs attempts to the following attacks:
+
+- [SQL Injection](https://www.owasp.org/index.php/OWASP_Periodic_Table_of_Vulnerabilities_-_SQL_Injection)
+- [Cross-Site Scripting](https://www.owasp.org/index.php/OWASP_Periodic_Table_of_Vulnerabilities_-_Cross-Site_Scripting_(XSS))
+- [Local File Inclusion](https://www.owasp.org/index.php/Testing_for_Local_File_Inclusion)
+- [Remote File Inclusion](https://www.owasp.org/index.php/OWASP_Periodic_Table_of_Vulnerabilities_-_Remote_File_Inclusion)
+- [Code Injection](https://www.owasp.org/index.php/Code_Injection)
+- [Session Fixation](https://www.owasp.org/index.php/Session_fixation)
+- [Scanner Detection](https://www.owasp.org/index.php/Category:Vulnerability_Scanning_Tools)
+- [Metadata/Error Leakages](https://www.owasp.org/index.php/Improper_Error_Handling)
+
+It is good to have a basic knowledge of the following:
+
+- [Kubernetes](https://kubernetes.io/docs/home/)
+- [Ingress](https://kubernetes.github.io/ingress-nginx/)
+- [ModSecurity](https://www.modsecurity.org/)
+- [OWASP Core Rule Set](https://modsecurity.org/crs/)
+
+## Roadmap
+
+More information on the direction of the WAF can be
+found in [Product Vision - Defend](https://about.gitlab.com/direction/defend/#waf)

doc/topics/web_application_firewall/quick_start_guide.md

+# Getting started with the Web Application Firewall
+
+This is a step-by-step guide that will help you use GitLab's [Web Application Firewall](index.md) after
+deploying a project hosted on GitLab.com to Google Kubernetes Engine using [Auto DevOps](../autodevops/index.md).
+
+We will use GitLab's native Kubernetes integration, so you will not need
+to create a Kubernetes cluster manually using the Google Cloud Platform console.
+We will create and deploy a simple application that we create from a GitLab template.
+
+These instructions will also work for a self-hosted GitLab instance. However, you will
+need to ensure your own [Runners are configured](../../ci/runners/README.md) and
+[Google OAuth is enabled](../../integration/google.md).
+
+**Note**: GitLab's Web Application Firewall is deployed with [Ingress](../../user/clusters/applications.md#Ingress),
+so it will be avaliable to your applications no matter how you deploy them to Kubernetes.
+
+## Enable or disable ModSecurity
+
+ModSecurity is enabled by default on GitLab.com. You can toggle the feature flag to false by running the following command in the Rails console:
+
+```ruby
+Feature.disable(:ingress_modsecurity)
+```
+
+Once disabled, you must uninstall and reinstall your Ingress application for the changes to take effect. See the [Feature Flag](../../user/project/operations/feature_flags.md) documentation for more information.
+
+## Configuring your Google account
+
+Before creating and connecting your Kubernetes cluster to your GitLab project,
+you need a Google Cloud Platform account. If you do not already have one,
+sign up at <https://console.cloud.google.com>. You will need to either sign in with an existing
+Google account (for example, one that you use to access Gmail, Drive, etc.) or create a new one.
+
+1. To enable the required APIs and related services, follow the steps in the ["Before you begin" section of the Kubernetes Engine docs](https://cloud.google.com/kubernetes-engine/docs/quickstart#before-you-begin).
+1. Make sure you have created a [billing account](https://cloud.google.com/billing/docs/how-to/manage-billing-account).
+
+TIP: **Tip:**
+Every new Google Cloud Platform (GCP) account receives [$300 in credit](https://console.cloud.google.com/freetrial),
+and in partnership with Google, GitLab is able to offer an additional $200 for new GCP accounts to get started with GitLab's
+Google Kubernetes Engine integration. All you have to do is [follow this link](https://cloud.google.com/partners/partnercredit/?PCN=a0n60000006Vpz4AAC) and apply for credit.
+
+## Creating a new project from a template
+
+We will use one of GitLab's project templates to get started. As the name suggests,
+those projects provide a barebones application built on some well-known frameworks.
+
+1. In GitLab, click the plus icon (**+**) at the top of the navigation bar and select
+   **New project**.
+1. Go to the **Create from template** tab where you can choose for example a Ruby on
+   Rails, Spring, or NodeJS Express project.
+   We will use the Ruby on Rails template.
+
+   ![Select project template](../autodevops/img/guide_project_template_v12_3.png)
+
+1. Give your project a name, optionally a description, and make it public so that
+   you can take advantage of the features available in the
+   [GitLab Gold plan](https://about.gitlab.com/pricing/#gitlab-com).
+
+   ![Create project](../autodevops/img/guide_create_project_v12_3.png)
+
+1. Click **Create project**.
+
+Now that the project is created, the next step is to create the Kubernetes cluster
+under which this application will be deployed.
+
+## Creating a Kubernetes cluster from within GitLab
+
+1. On the project's landing page, click **Add Kubernetes cluster**
+   (note that this option is also available when you navigate to **Operations > Kubernetes**).
+
+   ![Project landing page](../autodevops/img/guide_project_landing_page_v12_3.png)
+
+1. On the **Create new cluster on GKE** tab, click **Sign in with Google**.
+
+   ![Google sign in](../autodevops/img/guide_google_signin_v12_3.png)
+
+1. Connect with your Google account and click **Allow** when asked (this
+   appears only the first time you connect GitLab with your Google account).
+
+   ![Google auth](../autodevops/img/guide_google_auth_v12_3.png)
+
+1. The last step is to provide the cluster details.
+    1. Give it a name, leave the environment scope as is, and choose the GCP project under which the cluster
+   will be created (per the instructions to [configure your Google account](#configuring-your-google-account), a project should have already been created for you).
+    1. Choose the [region/zone](https://cloud.google.com/compute/docs/regions-zones/) under which the cluster will be created.
+    1. Enter the number of nodes you want it to have.
+    1. Choose the [machine type](https://cloud.google.com/compute/docs/machine-types).
+
+   ![GitLab GKE cluster details](../autodevops/img/guide_gitlab_gke_details_v12_3.png)
+
+1. Click **Create Kubernetes cluster**.
+
+After a couple of minutes, the cluster is created. You can also see its
+status on your [GCP dashboard](https://console.cloud.google.com/kubernetes).
+
+The next step is to install some applications on your cluster that are needed
+to take full advantage of Auto DevOps.
+
+## Installing Helm and Ingress
+
+GitLab's Kubernetes integration comes with some
+[pre-defined applications](../../user/project/clusters/index.md#installing-applications)
+for you to install.
+
+![Cluster applications](../autodevops/img/guide_cluster_apps_v12_3.png)
+
+The first one to install is Helm Tiller, a package manager for Kubernetes, which
+is needed in order to install the rest of the applications. Go ahead and click
+its **Install** button.
+Once it is installed, the other applications that rely on it will each have their
+**Install** buttons enabled.
+
+For this guide, we need to install Ingress. Ingress provides load balancing,
+SSL termination, and name-based virtual hosting, using NGINX behind
+the scenes. Make sure that the **Enable Web Application Firewall** button is checked
+before installing.
+
+![Cluster applications](./img/guide_waf_ingress_installation.png)
+
+After Ingress is installed, wait a few seconds and copy the IP address that
+is displayed in order to add in your base **Domain** at the top of the page. For
+the purpose of this guide, we will use the one suggested by GitLab. Once you have
+filled in the domain, click **Save changes**.
+
+![Cluster Base Domain](../autodevops/img/guide_base_domain_v12_3.png)
+
+Prometheus should also be installed. It is an open-source monitoring and
+alerting system that we will use to supervise the deployed application.
+We will not install GitLab Runners as we will use the shared Runners that
+GitLab.com provides.
+
+## Enabling Auto DevOps (optional)
+
+Starting with GitLab 11.3, Auto DevOps is enabled by default. However, it is possible to disable
+Auto DevOps at both the instance-level (for self-managed instances) and the group-level.
+Follow these steps if Auto DevOps has been manually disabled:
+
+1. Navigate to **Settings > CI/CD > Auto DevOps**.
+1. Select **Default to Auto DevOps pipeline**.
+1. Select the [continuous deployment strategy](../autodevops/index.md#deployment-strategy)
+   which automatically deploys the application to production once the pipeline
+   successfully runs on the `master` branch.
+1. Click **Save changes**.
+
+   ![Auto DevOps settings](../autodevops/img/guide_enable_autodevops_v12_3.png)
+
+Once you complete all the above and save your changes, a new pipeline is
+automatically created. To view the pipeline, go to **CI/CD > Pipelines**.
+
+![First pipeline](../autodevops/img/guide_first_pipeline_v12_3.png)
+
+The next section explains what each pipeline job does.
+
+## Deploying the application
+
+By now you should see the pipeline running, but what is it running exactly?
+
+To navigate inside the pipeline, click its status badge (its status should be "Running").
+The pipeline is split into a few stages, each running a couple of jobs.
+
+![Pipeline stages](../autodevops/img/guide_pipeline_stages_v12_3.png)
+
+In the **build** stage, the application is built into a Docker image and then
+uploaded to your project's [Container Registry](../../user/packages/container_registry/index.md) ([Auto Build](../autodevops/index.md#auto-build)).
+
+In the **test** stage, GitLab runs various checks on the application.
+
+The **production** stage is run after the tests and checks finish, and it automatically
+deploys the application in Kubernetes ([Auto Deploy](../autodevops/index.md#auto-deploy)).
+
+The **production** stage creates Kubernetes objects
+like a Deployment, Service, and Ingress resource. The
+application will be monitored by the WAF automatically.
+
+## Validating Ingress is running ModSecurity
+
+Now we can make sure that Ingress is running properly with ModSecurity and send
+a request to ensure our application is responding correctly. You must connect to
+your cluster either using [Cloud Shell](https://cloud.google.com/shell/) or the [Google Cloud SDK](https://cloud.google.com/sdk/install).
+
+1. After connecting to your cluster, check if the Ingress-NGINX controller is running and ModSecurity is enabled.
+
+    This is done by running the following commands:
+
+    ```bash
+    $ kubectl get pods -n gitlab-managed-apps | grep 'ingress-controller'
+    ingress-nginx-ingress-controller-55f9cf6584-dxljn        2/2     Running
+
+    $ kubectl -n gitlab-managed-apps exec -it $(kubectl get pods -n gitlab-managed-apps | grep 'ingress-controller' | awk '{print $1}') -- cat /etc/nginx/nginx.conf | grep 'modsecurity on;'
+            modsecurity on;
+    ```
+
+1. Verify the Rails application has been installed properly.
+
+    ```bash
+    $ kubectl get ns
+    auto-devv-2-16730183-production     Active
+
+    $ kubectl get pods -n auto-devv-2-16730183-production
+    NAME                                   READY   STATUS    RESTARTS
+    production-5778cfcfcd-nqjcm            1/1     Running   0
+    production-postgres-6449f8cc98-r7xgg   1/1     Running   0
+    ```
+
+1. To make sure the Rails application is responding, send a request to it by running:
+
+    ```bash
+    $ kubectl get ing -n auto-devv-2-16730183-production
+    NAME  HOSTS  PORTS
+    production-auto-deploy  fjdiaz-auto-devv-2.34.68.60.207.nip.io,le-16730183.34.68.60.207.nip.io  80, 443
+
+    $ curl --location --insecure fjdiaz-auto-devv-2.34.68.60.207.nip.io | grep 'Rails!' --after 2 --before 2
+    <body>
+        <p>You're on Rails!</p>
+    </body>
+    ```
+
+Now that we have confirmed our system is properly setup, we can go ahead and test
+the WAF with OWASP CRS!
+
+## Testing out the OWASP Core Rule Set
+
+Now let's send a potentially malicious request, as if we were a scanner,
+checking for vulnerabilities within our application and examine the modsecurity logs:
+
+```bash
+$ curl --location --insecure fjdiaz-auto-devv-2.34.68.60.207.nip.io --header "User-Agent: absinthe" | grep 'Rails!' --after 2 --before 2
+<body>
+    <p>You're on Rails!</p>
+</body>
+
+$ kubectl -n gitlab-managed-apps exec -it $(kubectl get pods -n gitlab-managed-apps | grep 'ingress-controller' | awk '{print $1}') -- cat /var/log/modsec/audit.log | grep 'absinthe'
+{
+    "message": "Found User-Agent associated with security scanner",
+    "details": {
+        "match": "Matched \"Operator `PmFromFile' with parameter `scanners-user-agents.data' against variable `REQUEST_HEADERS:user-agent' (Value: `absinthe' )",
+        "reference": "o0,8v84,8t:lowercase",
+        "ruleId": "913100",
+        "file": "/etc/nginx/owasp-modsecurity-crs/rules/REQUEST-913-SCANNER-DETECTION.conf",
+        "lineNumber": "33",
+        "data": "Matched Data: absinthe found within REQUEST_HEADERS:user-agent: absinthe",
+        "severity": "2",
+        "ver": "OWASP_CRS/3.2.0",
+        "rev": "",
+        "tags": ["application-multi", "language-multi", "platform-multi", "attack-reputation-scanner", "OWASP_CRS", "OWASP_CRS/AUTOMATION/SECURITY_SCANNER", "WASCTC/WASC-21", "OWASP_TOP_10/A7", "PCI/6.5.10"],
+        "maturity": "0",
+        "accuracy": "0"
+    }
+}
+```
+
+You can see that ModSecurity logs the suspicous behavior. By sending a request
+with the `User Agent: absinthe` header, which [absinthe](https://github.com/cameronhotchkies/Absinthe), a tool for testing for SQL injections uses, we can detect that someone was
+searching for vulnerabilities on our system. Detecting scanners is useful, because we
+can learn if someone is trying to exploit our system.
+
+## Conclusion
+
+You can now see the benefits of a using a Web Application Firewall.
+ModSecurity and the OWASP Core Rule Set, offer many more benefits.
+You can explore them in more detail:
+
+- [GitLab Defend Vision](https://about.gitlab.com/direction/defend/#waf)
+- [ModSecurity](https://www.modsecurity.org/)
+- [OWASP Core Rule Set](https://modsecurity.org/crs/)
+- [AutoDevOps](../autodevops/index.md)

doc/user/clusters/applications.md

@@ -139,10 +139,12 @@ file. Customizing installation by modifying this file is not supported.
 > - Introduced in GitLab 10.2 for project-level clusters.
 > - Introduced in GitLab 11.6 for group-level clusters.
 
-[Ingress](https://kubernetes.github.io/ingress-nginx/) can provide load
-balancing, SSL termination, and name-based virtual hosting. It acts as a
-web proxy for your applications and is useful if you want to use [Auto
-DevOps](../../topics/autodevops/index.md) or deploy your own web apps.
+[Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) provides load balancing, SSL termination, and name-based virtual hosting
+out of the box. It acts as a web proxy for your applications and is useful
+if you want to use [Auto DevOps](../../topics/autodevops/index.md) or deploy your own web apps.
+
+The Ingress Controller installed is [Ingress-NGINX](https://kubernetes.io/docs/concepts/services-networking/ingress/),
+which is supported by the Kubernetes community.
 
 NOTE: **Note:**
 With the following procedure, a load balancer must be installed in your cluster
@@ -255,12 +257,20 @@ use an A record. If your external endpoint is a hostname, use a CNAME record.
 
 > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/21966) in GitLab 12.7.
 
-Out of the box, GitLab provides you real-time security monitoring with
-[ModSecurity](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#modsecurity).
+A Web Application Firewall (WAF) is able to examine traffic being sent/received
+and can block malicious traffic before it reaches your application. The benefits
+of a WAF are:
+
+- Real-time security monitoring for your application
+- Logging of all your HTTP traffic to the application
+- Access control for your application
+- Highly configurable logging and blocking rules
+
+Out of the box, GitLab provides you with a WAF known as [`ModSecurity`](https://www.modsecurity.org/)
 
-Modsecurity is a toolkit for real-time web application monitoring, logging,
-and access control. With GitLab's offering, the [OWASP's Core Rule Set](https://www.modsecurity.org/CRS/Documentation/), which provides generic attack detection capabilities,
-is automatically applied.
+ModSecurity is a toolkit for real-time web application monitoring, logging,
+and access control. With GitLab's offering, the [OWASP's Core Rule Set](https://www.modsecurity.org/CRS/Documentation/),
+which provides generic attack detection capabilities, is automatically applied.
 
 This feature:
 
@@ -275,6 +285,12 @@ This feature:
 To enable ModSecurity, check the **Enable Web Application Firewall** checkbox
 when installing your [Ingress application](#ingress).
 
+If this is your first time using GitLab's WAF, we recommend you follow the
+[quick start guide](../../topics/web_application_firewall/quick_start_guide.md).
+
+There is a small performance overhead by enabling ModSecurity. However,
+if this is considered significant for your application, you can disable it.
+
 There is a small performance overhead by enabling ModSecurity. If this is
 considered significant for your application, you can disable ModSecurity's
 rule engine for your deployed application by setting