Merge branch 'jramsay-github-ci-docs' into 'master'

Add step by step guides for GitLab CI/CD for GitHub and Bitbucket

Closes #5108

See merge request gitlab-org/gitlab-ee!5025

doc/ci/README.md

@@ -22,6 +22,7 @@ The first steps towards your GitLab CI/CD journey.
 - [Pipelines and jobs](pipelines.md): configure your GitLab CI/CD pipelines to build, test, and deploy your application.
 - Runners: The [GitLab Runner](https://docs.gitlab.com/runner/) is responsible by running the jobs in your CI/CD pipeline. On GitLab.com, Shared Runners are enabled by default, so
 you don't need to set up anything to start to use them with GitLab CI/CD.
+- [Using GitLab CI/CD with GitHub and other external repositories](ci_cd_for_external_repos/index.md)
 
 ### Introduction to GitLab CI/CD
 

doc/ci/ci_cd_for_external_repos/bitbucket_integration.md

+# Using GitLab CI/CD with a Bitbucket Cloud repository
+
+GitLab CI/CD can be used with Bitbucket Cloud by creating a
+[CI/CD project](../../user/project/ci_cd_for_external_repo.md) and connecting
+your Git repository via URL.
+
+1. In GitLab create a **CI/CD for external repo**, select **Repo by URL** and
+   create the project.
+
+    ![Create project](img/external_repository.png)
+
+    GitLab will import the repository and enable [Pull Mirroring][pull-mirroring].
+    
+1. In GitLab create a
+   [Personal Access Token](../../user/profile/personal_access_tokens.md)
+   with `api` scope. This will be used to authenticate requests from the web
+   hook that will be created in Bitbucket to notify GitLab of new commits.
+
+1. In Bitbucket from **Settings > Webhooks** create a new web hook to notify
+   GitLab of new commits.
+
+    The web hook URL should be set to the GitLab API to trigger pull mirroring,
+    using the Personal Access Token we just generated for authentication.
+
+    ```
+    https://gitlab.com/api/v4/projects/<NAMESPACE>%2F<PROJECT>/mirror/pull?private_token=<PERSONAL_ACCESS_TOKEN>
+    ```
+
+    The web hook Trigger should be set to 'Repository Push'.
+
+    ![Bitbucket Cloud webhook](img/bitbucket_webhook.png)
+
+    After saving, test the web hook by pushing a change to your Bitbucket
+    repository.
+
+1. In Bitbucket create an **App Password** from **Bitbucket Settings > App
+   Passwords** to authenticate the build status script setting commit build
+   statuses in Bitbucket. Repository write permissions are required.
+
+    ![Bitbucket Cloud webhook](img/bitbucket_app_password.png)
+
+1. In GitLab from **Settings > CI/CD > Secret variables** add variables to allow
+   communication with Bitbucket via the Bitbucket API.
+
+    `BITBUCKET_ACCESS_TOKEN`: the Bitbucket app password created above
+
+    `BITBUCKET_USERNAME`: the username of the Bitbucket account
+
+    `BITBUCKET_NAMESPACE`: set this if your GitLab and Bitbucket namespaces differ
+
+    `BITBUCKET_REPOSITORY`: set this if your GitLab and Bitbucket project names differ
+
+1. In Bitbucket add a script to push the pipeline status to Bitbucket.
+
+    > Note: changes made in GitLab will be overwritten by any changes made
+    upstream in Bitbucket.
+
+    Create a file `build_status` and insert the script below and run
+    `chmod +x build_status` in your terminal to make the script executable.
+
+    ```bash
+    #!/usr/bin/env bash
+
+    # Push GitLab CI/CD build status to Bitbucket Cloud
+
+    if [ -z "$BITBUCKET_ACCESS_TOKEN" ]; then
+       echo "ERROR: BITBUCKET_ACCESS_TOKEN is not set"
+    exit 1
+    fi
+    if [ -z "$BITBUCKET_USERNAME" ]; then
+        echo "ERROR: BITBUCKET_USERNAME is not set"
+    exit 1
+    fi
+    if [ -z "$BITBUCKET_NAMESPACE" ]; then
+        echo "Setting BITBUCKET_NAMESPACE to $CI_PROJECT_NAMESPACE"
+        BITBUCKET_NAMESPACE=$CI_PROJECT_NAMESPACE
+    fi
+    if [ -z "$BITBUCKET_REPOSITORY" ]; then
+        echo "Setting BITBUCKET_REPOSITORY to $CI_PROJECT_NAME"
+        BITBUCKET_REPOSITORY=$CI_PROJECT_NAME
+    fi
+
+    BITBUCKET_API_ROOT="https://api.bitbucket.org/2.0"
+    BITBUCKET_STATUS_API="$BITBUCKET_API_ROOT/repositories/$BITBUCKET_NAMESPACE/$BITBUCKET_REPOSITORY/commit/$CI_COMMIT_SHA/statuses/build"
+    BITBUCKET_KEY="ci/gitlab-ci/$CI_JOB_NAME"
+
+    case "$BUILD_STATUS" in
+    running)
+       BITBUCKET_STATE="INPROGRESS"
+       BITBUCKET_DESCRIPTION="The build is running!"
+       ;;
+    passed)
+       BITBUCKET_STATE="SUCCESSFUL"
+       BITBUCKET_DESCRIPTION="The build passed!"
+       ;;
+    failed)
+       BITBUCKET_STATE="FAILED"
+       BITBUCKET_DESCRIPTION="The build failed."
+       ;;
+    esac
+
+    echo "Pushing status to $BITBUCKET_STATUS_API..."
+    curl --request POST $BITBUCKET_STATUS_API \
+    --user $BITBUCKET_USERNAME:$BITBUCKET_ACCESS_TOKEN \
+    --header "Content-Type:application/json" \
+    --silent \
+    --data "{ \"state\": \"$BITBUCKET_STATE\", \"key\": \"$BITBUCKET_KEY\", \"description\":
+    \"$BITBUCKET_DESCRIPTION\",\"url\": \"$CI_PROJECT_URL/-/jobs/$CI_JOB_ID\" }"
+    ```
+
+1. Still in Bitbucket, create a `.gitlab-ci.yml` file to use the script to push
+   pipeline success and failures to Bitbucket.
+
+    ```
+    stages:
+      - test
+      - ci_status
+
+    unit-tests
+      script:
+        - echo "Success. Add your tests!"
+
+    success:
+      stage: ci_status
+      before_script:
+        - ""
+      after_script:
+        - ""
+      script:
+        - BUILD_STATUS=passed BUILD_KEY=push ./build_status
+      when: on_success
+
+    failure:
+      stage: ci_status
+      before_script:
+        - ""
+      after_script:
+        - ""
+      script:
+        - BUILD_STATUS=failed BUILD_KEY=push ./build_status
+      when: on_failure
+    ```
+
+GitLab is now configured to mirror changes from Bitbucket, run CI/CD pipelines
+configured in `.gitlab-ci.yml` and push the status to Bitbucket.
+
+[pull-mirroring]: ../../workflow/repository_mirroring.md#pulling-from-a-remote-repository

doc/ci/ci_cd_for_external_repos/github_integration.md

+# Using GitLab CI/CD with a GitHub repository
+
+GitLab CI/CD can be used with **GitHub.com** and **GitHub Enterprise** by
+creating a [CI/CD project][cicd-projects] to connect your GitHub repository to
+GitLab.
+
+> Note: to use **GitHub Enterprise** with **GitLab.com** you should use the
+> manual method.
+
+## Connect with GitHub integration
+
+If the [GitHub integration][github-integration] has been enabled by your GitLab
+administrator:
+
+1. In GitLab create a **CI/CD for external repo** project and select
+   **GitHub**.
+
+    ![Create project](img/github_omniauth.png)
+
+1. Once authenticated, you will be redirected to a list of your repositories to
+   connect. Click **Connect** to select the repository.
+
+    ![Create project](img/github_repo_list.png)
+
+1. In GitHub, add a `.gitlab-ci.yml` to [configure GitLab CI/CD][ci-quickstart].
+
+GitLab will import the project, enable [Pull Mirroring][pull-mirroring], enable
+[GitHub project integration][github-project-integration], and create a web hook
+on GitHub to notify GitLab of new commits.
+
+## Connect with Personal Access Token
+
+> Note: Personal Access Tokens can only be used to connect GitHub.com
+repositories to GitLab.
+
+If you are not using the [GitHub integration][github-integration], you can
+still perform a one-off authorization with GitHub to grant GitLab access your
+repositories:
+
+1. Open https://github.com/settings/tokens/new to create a **Personal Access
+   Token**. This token with be used to access your repository and push commit
+   statuses to GitHub.
+   
+    The `repo` and `admin:repo_hook` should be enable to allow GitLab access to
+    your project, update commit statuses, and create a web hook to notify
+    GitLab of new commits.
+
+1. In GitLab create a **CI/CD for external repo** project and select
+   **GitHub**.
+
+    ![Create project](img/github_omniauth.png)
+
+1. Paste the token into the **Personal access token** field and click **List
+   Repositories**. Click **Connect** to select the repository.
+
+1. In GitHub, add a `.gitlab-ci.yml` to [configure GitLab CI/CD][ci-quickstart].
+
+GitLab will import the project, enable [Pull Mirroring][pull-mirroring], enable
+[GitHub project integration][github-project-integration], and create a web hook
+on GitHub to notify GitLab of new commits.
+
+## Connect manually
+
+If the [GitHub integration][github-integration] is not enabled, or is enabled
+for a different GitHub instance, you GitLab CI/CD can be manually enabled for
+your repository.
+
+1. In GitHub open https://github.com/settings/tokens/new create a **Personal
+   Access Token.** GitLab will use this token to access your repository and
+   push commit statuses.
+   
+    Enter a **Token description** and update the scope to allow:
+
+    `repo` so that GitLab can access your project and update commit statuses
+
+1. In GitLab create a **CI/CD project** using the Git URL option and the HTTPS
+   URL for your GitHub repository. If your project if private, use the personal
+   access token you just created for authentication.
+
+    GitLab will automatically configure polling-based pull mirroring.
+
+1. Still in GitLab, enable the [GitHub project integration][github-project-integration]
+   from **Settings > Integrations.**
+
+    Check the **Active** checkbox to enable the integration, paste your
+    personal access token and HTTPS repository URL into the form, and **Save.**
+
+1. Still in GitLab create a **Personal Access Token** with `API` scope to
+   authenticate the GitHub web hook notifying GitLab of new commits.
+
+1. In GitHub from **Settings > Webhooks** create a web hook to notify GitLab of
+   new commits.
+
+    The web hook URL should be set to the GitLab API to
+    [trigger pull mirroring][pull-mirroring-trigger],
+    using the GitLab personal access token we just created.
+
+    ```
+    https://gitlab.com/api/v4/projects/<NAMESPACE>%2F<PROJECT>/mirror/pull?private_token=<PERSONAL_ACCESS_TOKEN>
+    ```
+
+    ![Create web hook](img/github_push_webhook.png)
+
+1. In GitHub add a `.gitlab-ci.yml` to configure GitLab CI/CD.
+
+[ci-quickstart]: ../quick_start/README.md
+[cicd-projects]: ../../user/project/ci_cd_for_external_repo.md
+[github-integration]: ../../integration/github.md
+[github-project-integration]: ../../user/project/integrations/github.md
+[pull-mirroring]: ../../workflow/repository_mirroring.md#pulling-from-a-remote-repository
+[pull-mirroring-trigger]: ../../api/projects.md#start-the-pull-mirroring-process-for-a-project

doc/ci/ci_cd_for_external_repos/index.md

+# GitLab CI/CD for external repositories
+
+>[Introduced][ee-4642] in [GitLab Premium][eep] 10.6.
+
+GitLab CI/CD can be used with GitHub or any other Git server.
+Instead of moving your entire project to GitLab, you can connect your
+external repository to get the benefits of GitLab CI/CD. 
+
+- [GitHub](github_integration.md)
+- [Bitbucket Cloud](bitbucket_integration.md)
+
+Connecting an external repository will set up [repository mirroring][mirroring]
+and create a lightweight project where issues, merge requests, container
+registry, wiki, and snippets disabled. These features
+[can be re-enabled later][settings].
+
+1. From your GitLab dashboard click **New project**
+1. Switch to the **CI/CD for external repo** tab
+1. Choose **GitHub** or **Repo by URL**
+1. The next steps are similar to the [import flow](../../user/project/import/index.md)
+
+![CI/CD for external repository project creation](img/ci_cd_for_external_repo.png)
+
+
+[ee-4642]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4642
+[eep]: https://about.gitlab.com/products/
+[mirroring]: ../../workflow/repository_mirroring.md
+[settings]: ../../user/project/settings/index.md#sharing-and-permissions

doc/user/project/ci_cd_for_external_repo.md

-## CI/CD for external repositories
-
->[Introduced][ee-4642] in [GitLab Premium][eep] 10.6.
-
-Instead of importing the repo directly to GitLab, you can connect your
-external repository to get GitLab CI/CD benefits.
-
-This will set up [repository mirroring](../../workflow/repository_mirroring.md)
-and create a stripped-down version of a project that has issues, merge requests,
-container registry, wiki, and snippets disabled but
-[can be re-enabled later on](settings/index.md#sharing-and-permissions).
-
-1. From your GitLab dashboard click **New project**
-1. Switch to the **CI/CD for external repo** tab
-1. Choose **GitHub** or **Repo by URL**
-1. The next steps are similar to the [import flow](import/index.md)
-
-![CI/CD for external repository project creation](img/ci_cd_for_external_repo.png)
-
-
-[ee-4642]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4642
-[eep]: https://about.gitlab.com/products/
+This document was moved to [another location](../../ci/ci_cd_for_external_repos/index.md).

doc/user/project/index.md

@@ -120,7 +120,7 @@ Read through the documentation on [project settings](settings/index.md).
 Instead of importing a repository directly to GitLab, you can connect your repository
 as a CI/CD project.
 
-Read through the documentation on [CI/CD for external repositories](ci_cd_for_external_repo.md).
+Read through the documentation on [CI/CD for external repositories](../../ci/ci_cd_for_external_repos/index.md).
 
 ## Project's members