Merge branch 'docs/git-submodules-with-ci' into 'master'

Refactor the Git submodules with CI docs

Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/22567

See merge request !7856

doc/ci/README.md

@@ -21,6 +21,7 @@
 - [CI services (linked docker containers)](services/README.md)
 - [CI/CD pipelines settings](../user/project/pipelines/settings.md)
 - [Review Apps](review_apps/index.md)
+- [Git submodules](git_submodules.md) Using Git submodules in your CI jobs
 
 ## Breaking changes
 

doc/ci/git_submodules.md

+# Using Git submodules with GitLab CI
+
+> **Notes:**
+- GitLab 8.12 introduced a new [CI build permissions model][newperms] and you
+  are encouraged to upgrade your GitLab instance if you haven't done already.
+  If you are **not** using GitLab 8.12 or higher, you would need to work your way
+  around submodules in order to access the sources of e.g., `gitlab.com/group/project`
+  with the use of [SSH keys](ssh_keys/README.md).
+- With GitLab 8.12 onward, your permissions are used to evaluate what a CI build
+  can access. More information about how this system works can be found in the
+  [Build permissions model](../user/permissions.md#builds-permissions).
+- The HTTP(S) Git protocol [must be enabled][gitpro] in your GitLab instance.
+
+## Configuring the `.gitmodules` file
+
+If dealing with [Git submodules][gitscm], your project will probably have a file
+named `.gitmodules`.
+
+Let's consider the following example:
+
+1. Your project is located at `https://gitlab.com/secret-group/my-project`.
+1. To checkout your sources you usually use an SSH address like
+   `git@gitlab.com:secret-group/my-project.git`.
+1. Your project depends on `https://gitlab.com/group/project`, which you want
+   to include as a submodule.
+
+If you are using GitLab 8.12+ and your submodule is on the same GitLab server,
+you must update your `.gitmodules` file to use **relative URLs**.
+Since Git allows the usage of relative URLs for your `.gitmodules` configuration,
+this easily allows you to use HTTP(S) for cloning all your CI builds and SSH
+for all your local checkouts. The `.gitmodules` would look like:
+
+```ini
+[submodule "project"]
+  path = project
+  url = ../../group/project.git
+```
+
+The above configuration will instruct Git to automatically deduce the URL that
+should be used when cloning sources. Whether you use HTTP(S) or SSH, Git will use
+that same channel and it will allow to make all your CI builds use HTTP(S)
+(because GitLab CI only uses HTTP(S) for cloning your sources), and all your local
+clones will continue using SSH.
+
+For all other submodules not located on the same GitLab server, use the full
+HTTP(S) protocol URL:
+
+```ini
+[submodule "project-x"]
+  path = project-x
+  url = https://gitserver.com/group/project-x.git
+```
+
+Once `.gitmodules` is correctly configured, you can move on to
+[configuring your `.gitlab-ci.yml`](#using-git-submodules-in-your-ci-jobs).
+
+## Using Git submodules in your CI jobs
+
+There are a few steps you need to take in order to make submodules work
+correctly with your CI builds:
+
+1. First, make sure you have used [relative URLs](#configuring-the-gitmodules-file)
+   for the submodules located in the same GitLab server.
+1. Then, use `git submodule sync/update` in `before_script`:
+
+    ```yaml
+    before_script:
+      - git submodule sync --recursive
+      - git submodule update --init --recursive
+    ```
+
+    `--recursive` should be used in either both or none (`sync/update`) depending on
+    whether you have recursive submodules.
+
+The rationale to set the `sync` and `update` in `before_script` is because of
+the way Git submodules work. On a fresh Runner workspace, Git will set the
+submodule URL including the token in `.git/config`
+(or `.git/modules/<submodule>/config`) based on `.gitmodules` and the current
+remote URL. On subsequent builds on the same Runner, `.git/config` is cached
+and already contains a full URL for the submodule, corresponding to the previous
+build, and to **a token from a previous build**. `sync` allows to force updating
+the full URL.
+
+[gitpro]: ../user/admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols
+[gitscm]: https://git-scm.com/book/en/v2/Git-Tools-Submodules "Git submodules documentation"
+[newperms]: ../user/project/new_ci_build_permissions_model.md

doc/user/permissions.md

@@ -141,10 +141,9 @@ instance and project. In addition, all admins can use the admin interface under
 | See events in the system              |                 |             |          | ✓      |
 | Admin interface                       |                 |             |          | ✓      |
 
-### Build permissions
-
-> Changed in GitLab 8.12.
+### Builds permissions
 
+>**Note:**
 GitLab 8.12 has a completely redesigned build permissions system.
 Read all about the [new model and its implications][new-mod].
 

doc/user/project/new_ci_build_permissions_model.md

@@ -34,9 +34,9 @@ as created be the pusher (local push or via the UI) and any build created in thi
 pipeline will have the permissions of the pusher.
 
 This allows us to make it really easy to evaluate the access for all projects
-that have Git submodules or are using container images that the pusher would
-have access too. **The permission is granted only for time that build is running.
-The access is revoked after the build is finished.**
+that have [Git submodules][gitsub] or are using container images that the pusher
+would have access too. **The permission is granted only for time that build is
+running. The access is revoked after the build is finished.**
 
 ## Types of users
 
@@ -141,7 +141,7 @@ with GitLab 8.12.
 With the new build permissions model, there is now an easy way to access all
 dependent source code in a project. That way, we can:
 
-1. Access a project's Git submodules
+1. Access a project's [Git submodules][gitsub]
 1. Access private container images
 1. Access project's and submodule LFS objects
 
@@ -179,108 +179,8 @@ As a user:
 
 ### Git submodules
 
->
-It often happens that while working on one project, you need to use another
-project from within it; perhaps it’s a library that a third party developed or
-you’re developing a project separately and are using it in multiple parent
-projects.
-A common issue arises in these scenarios: you want to be able to treat the two
-projects as separate yet still be able to use one from within the other.
->
-_Excerpt from the [Git website][git-scm] about submodules._
-
-If dealing with submodules, your project will probably have a file named
-`.gitmodules`. And this is how it usually looks like:
-
-```
-[submodule "tools"]
-	path = tools
-	url = git@gitlab.com/group/tools.git
-```
-
-> **Note:**
-If you are **not** using GitLab 8.12 or higher, you would need to work your way
-around this issue in order to access the sources of `gitlab.com/group/tools`
-(e.g., use [SSH keys](../ssh_keys/README.md)).
->
-With GitLab 8.12 onward, your permissions are used to evaluate what a CI build
-can access. More information about how this system works can be found in the
-[Build permissions model](../../user/permissions.md#builds-permissions).
-
-To make use of the new changes, you have to update your `.gitmodules` file to
-use a relative URL.
-
-Let's consider the following example:
-
-1. Your project is located at `https://gitlab.com/secret-group/my-project`.
-1. To checkout your sources you usually use an SSH address like
-   `git@gitlab.com:secret-group/my-project.git`.
-1. Your project depends on `https://gitlab.com/group/tools`.
-1. You have the `.gitmodules` file with above content.
-
-Since Git allows the usage of relative URLs for your `.gitmodules` configuration,
-this easily allows you to use HTTP for cloning all your CI builds and SSH
-for all your local checkouts.
-
-For example, if you change the `url` of your `tools` dependency, from
-`git@gitlab.com/group/tools.git` to `../../group/tools.git`, this will instruct
-Git to automatically deduce the URL that should be used when cloning sources.
-Whether you use HTTP or SSH, Git will use that same channel and it will allow
-to make all your CI builds use HTTPS (because GitLab CI uses HTTPS for cloning
-your sources), and all your local clones will continue using SSH.
-
-Given the above explanation, your `.gitmodules` file should eventually look
-like this:
-
-```
-[submodule "tools"]
-	path = tools
-	url = ../../group/tools.git
-```
-
-However, you have to explicitly tell GitLab CI to clone your submodules as this
-is not done automatically. You can achieve that by adding a `before_script`
-section to your `.gitlab-ci.yml`:
-
-```
-before_script:
-  - git submodule update --init --recursive
-
-test:
-  script:
-    - run-my-tests
-```
-
-This will make GitLab CI initialize (fetch) and update (checkout) all your
-submodules recursively.
-
-If Git does not use the newly added relative URLs but still uses your old URLs,
-you might need to add `git submodule sync --recursive` to your `.gitlab-ci.yml`,
-prior to running `git submodule update --init --recursive`. This transfers the
-changes from your `.gitmodules` file into the `.git` folder, which is kept by
-runners between runs.
-
-In case your environment or your Docker image doesn't have Git installed,
-you have to either ask your Administrator or install the missing dependency
-yourself:
-
-```
-# Debian / Ubuntu
-before_script:
-  - apt-get update -y
-  - apt-get install -y git-core
-  - git submodule update --init --recursive
-
-# CentOS / RedHat
-before_script:
-  - yum install git
-  - git submodule update --init --recursive
-
-# Alpine
-before_script:
-  - apk add -U git
-  - git submodule update --init --recursive
-```
+To properly configure submodules with GitLab CI, read the
+[Git submodules documentation][gitsub].
 
 ### Container Registry
 
@@ -310,7 +210,7 @@ test:
 [build permissions]: ../permissions.md#builds-permissions
 [comment]: https://gitlab.com/gitlab-org/gitlab-ce/issues/22484#note_16648302
 [ext]: ../permissions.md#external-users
-[git-scm]: https://git-scm.com/book/en/v2/Git-Tools-Submodules
+[gitsub]: ../../ci/git_submodules.md
 [https]: ../admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols
 [triggers]: ../../ci/triggers/README.md
 [update-docs]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update