Merge branch 'enable-parent-child-pipelines' into 'master'

Enable parent/child pipelines

See merge request gitlab-org/gitlab!21830

changelogs/unreleased/enable-parent-child-pipelines.yml

+---
+title: Allow a pipeline (parent) to create a child pipeline as downstream pipeline within the same project
+merge_request: 21830
+author:
+type: added

doc/ci/parent_child_pipelines.md

+---
+type: reference
+---
+
+# Parent-child pipelines
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/16094) in GitLab Starter 12.7.
+
+As pipelines grow more complex, a few related problems start to emerge:
+
+- The staged structure, where all steps in a stage must be completed before the first
+  job in next stage begins, causes arbitrary waits, slowing things down.
+- Configuration for the single global pipeline becomes very long and complicated,
+  making it hard to manage.
+- Imports with [`include`](yaml/README.md#include) increase the complexity of the configuration, and create the potential
+  for namespace collisions where jobs are unintentionally duplicated.
+- Pipeline UX can become unwieldy with so many jobs and stages to work with.
+
+Additionally, sometimes the behavior of a pipeline needs to be more dynamic. The ability
+to choose to start sub-pipelines (or not) is a powerful ability, especially if the
+YAML is dynamically generated.
+
+![Parent pipeline graph expanded](img/parent_pipeline_graph_expanded_v12_6.png)
+
+Similarly to [multi-project pipelines](multi_project_pipelines.md), a pipeline can trigger a
+set of concurrently running child pipelines, but within the same project:
+
+- Child pipelines still execute each of their jobs according to a stage sequence, but
+  would be free to continue forward through their stages without waiting for unrelated
+  jobs in the parent pipeline to finish.
+- The configuration is split up into smaller child pipeline configurations, which are
+  easier to understand. This reduces the cognitive load to understand the overall configuration.
+- Imports are done at the child pipeline level, reducing the likelihood of collisions.
+- Each pipeline has only the steps relevant steps, making it easier to understand what's going on.
+
+Child pipelines work well with other GitLab CI features:
+
+- Use [`only: changes`](yaml/README.md#onlychangesexceptchanges) to trigger pipelines only when
+  certain files change. This is useful for monorepos, for example.
+- Since the parent pipeline in `.gitlab-ci.yml` and the child pipeline run as normal
+  pipelines, they can have their own behaviors and sequencing in relation to triggers.
+
+All of this will work with [`include:`](yaml/README.md#include) feature so you can compose
+the child pipeline configuration.
+
+## Examples
+
+The simplest case is [triggering a child pipeline](yaml/README.md#trigger-premium) using a
+local YAML file to define the pipeline configuration. In this case, the parent pipeline will
+trigger the child pipeline, and continue without waiting:
+
+```yaml
+microservice_a:
+  trigger:
+    include: path/to/microservice_a.yml
+```
+
+You can include multiple files when composing a child pipeline:
+
+```yaml
+microservice_a:
+  trigger:
+    include:
+      - local: path/to/microservice_a.yml
+      - template: SAST.gitlab-ci.yml
+```
+
+NOTE: **Note:**
+The max number of entries that are accepted for `trigger:include:` is three.
+
+Similar to [multi-project pipelines](multi_project_pipelines.md#mirroring-status-from-triggered-pipeline),
+we can set the parent pipeline to depend on the status of the child pipeline upon completion:
+
+```yaml
+microservice_a:
+  trigger:
+    include:
+      - local: path/to/microservice_a.yml
+      - template: SAST.gitlab-ci.yml
+    strategy: depend
+```
+
+## Limitations
+
+A parent pipeline can trigger many child pipelines, but a child pipeline cannot trigger
+further child pipelines. See the [related issue](https://gitlab.com/gitlab-org/gitlab/issues/29651) for discussion on possible future improvements.

doc/ci/pipelines.md

@@ -246,6 +246,13 @@ Pipelines for different projects can be combined and visualized together.
 
 For more information, see [Multi-project pipelines](multi_project_pipelines.md).
 
+## Parent-child pipelines
+
+Complex pipelines can be broken down into one parent pipeline that can trigger
+multiple child sub-pipelines, which all run in the same project and with the same SHA.
+
+For more information, see [Parent-Child pipelines](parent_child_pipelines.md).
+
 ## Working with pipelines
 
 In general, pipelines are executed automatically and require no intervention once created.

doc/ci/yaml/README.md

@@ -2600,14 +2600,17 @@ job split into three separate jobs.
 from `trigger` definition is started by GitLab, a downstream pipeline gets
 created.
 
-Learn more about [multi-project pipelines](../multi_project_pipelines.md#creating-multi-project-pipelines-from-gitlab-ciyml).
+This keyword allows the creation of two different types of downstream pipelines:
+
+- [Multi-project pipelines](../multi_project_pipelines.md#creating-multi-project-pipelines-from-gitlab-ciyml)
+- [Child pipelines](../parent_child_pipelines.md)
 
 NOTE: **Note:**
 Using a `trigger` with `when:manual` together results in the error `jobs:#{job-name}
 when should be on_success, on_failure or always`, because `when:manual` prevents
 triggers being used.
 
-#### Simple `trigger` syntax
+#### Simple `trigger` syntax for multi-project pipelines
 
 The simplest way to configure a downstream trigger is to use `trigger` keyword
 with a full path to a downstream project:
@@ -2622,7 +2625,7 @@ staging:
   trigger: my/deployment
 ```
 
-#### Complex `trigger` syntax
+#### Complex `trigger` syntax for multi-project pipelines
 
 It is possible to configure a branch name that GitLab will use to create
 a downstream pipeline with:
@@ -2657,6 +2660,28 @@ upstream_bridge:
     pipeline: other/project
 ```
 
+#### `trigger` syntax for child pipeline
+
+To create a [child pipeline](../parent_child_pipelines.md), specify the path to the
+YAML file containing the CI config of the child pipeline:
+
+```yaml
+trigger_job:
+  trigger:
+    include: path/to/child-pipeline.yml
+```
+
+Similar to [multi-project pipelines](../multi_project_pipelines.md#mirroring-status-from-triggered-pipeline),
+it is possible to mirror the status from a triggered pipeline:
+
+```yaml
+trigger_job:
+  trigger:
+    include:
+      - local: path/to/child-pipeline.yml
+    strategy: depend
+```
+
 ### `interruptible`
 
 > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/23464) in GitLab 12.3.

ee/lib/ee/gitlab/ci/config/entry/trigger.rb

@@ -26,7 +26,7 @@ module EE
               strategy :CrossProjectTrigger, if: -> (config) { !config.key?(:include) }
 
               strategy :SameProjectTrigger, if: -> (config) do
-                ::Feature.enabled?(:ci_parent_child_pipeline) &&
+                ::Feature.enabled?(:ci_parent_child_pipeline, default_enabled: true) &&
                   config.key?(:include)
               end
 
@@ -73,7 +73,7 @@ module EE
 
               class UnknownStrategy < ::Gitlab::Config::Entry::Node
                 def errors
-                  if ::Feature.enabled?(:ci_parent_child_pipeline)
+                  if ::Feature.enabled?(:ci_parent_child_pipeline, default_enabled: true)
                     ['config must specify either project or include']
                   else
                     ['config must specify project']