Merge branch 'ld-feature-flag-rollout-communication' into 'master'

Clarify communicating a feature flag rollout

See merge request gitlab-org/gitlab!29219

doc/development/experiment_guide/index.md

@@ -56,7 +56,7 @@ The author then adds a comment to this piece of code and adds a link to the issu
 
 - Track necessary events. See the [telemetry guide](../../telemetry/index.md) for details.
 - After the merge request is merged, use [`chatops`](../../ci/chatops/README.md) in the
-[appropriate channel](../feature_flags/controls.md#where-to-run-commands) to start the experiment for 10% of the users.
+[appropriate channel](../feature_flags/controls.md#communicate-the-change) to start the experiment for 10% of the users.
 The feature flag should have the name of the experiment with the `_experiment_percentage` suffix appended.
 For visibility, please also share any commands run against production in the `#s_growth` channel:
 

doc/development/feature_flags/controls.md

@@ -16,32 +16,6 @@ run:
 /chatops run feature --help
 ```
 
-## Where to run commands
-
-To increase visibility, we recommend that GitLab team members run feature flag
-related Chatops commands within certain Slack channels based on the environment
-and related feature. For the [staging](https://staging.gitlab.com)
-and [development](https://dev.gitlab.org) environments of GitLab.com,
-the commands should run in a channel for the stage the feature is relevant too.
-
-For example, use the `#s_monitor` channel for features developed by the
-Monitor stage, Health group.
-
-For all production environment Chatops commands, use the `#production` channel.
-
-As per the template, where a feature would have a (potentially) significant user
-impact and the feature is being enabled instance wide prior to release, please copy
-the Slack message and repost in the `#support_gitlab-com` channel for added visibility
-and awareness, preferably with a link to the issue, MR, or docs.
-
-Regardless of the channel in which the Chatops command is ran, any feature flag change that affects GitLab.com will automatically be logged in an issue.
-
-The issue is created in the [gl-infra/feature-flag-log](https://gitlab.com/gitlab-com/gl-infra/feature-flag-log/issues?scope=all&utf8=%E2%9C%93&state=closed) project, and it will at minimum log the Slack handle of person enabling a feature flag, the time, and the name of the flag being changed.
-
-The issue is then also posted to GitLab Inc. internal [Grafana dashboard](https://dashboards.gitlab.net/) as an annotation marker to make the change even more visible.
-
-Changes to the issue format can be submitted in the [Chatops project](https://gitlab.com/gitlab-com/chatops).
-
 ## Rolling out changes
 
 When the changes are deployed to the environments it is time to start
@@ -67,19 +41,11 @@ If you get an error "Whoops! This action is not allowed. This incident
 will be reported." that means your Slack account is not allowed to
 change feature flags or you do not [have access](#access).
 
-### Enabling feature for preproduction testing
+### Enabling a feature for preproduction testing
 
 As a first step in a feature rollout, you should enable the feature on <https://staging.gitlab.com>
 and <https://dev.gitlab.org>.
 
-For example, to enable a feature for 25% of all users, run the following in
-Slack:
-
-```shell
-/chatops run feature set new_navigation_bar 25 --dev
-/chatops run feature set new_navigation_bar 25 --staging
-```
-
 These two environments have different scopes.
 `dev.gitlab.org` is a production CE environment that has internal GitLab Inc.
 traffic and is used for some development and other related work.
@@ -89,13 +55,65 @@ a (very) rough estimate of how your feature will look/behave on GitLab.com.
 Both of these instances are connected to Sentry so make sure you check the projects
 there for any exceptions while testing your feature after enabling the feature flag.
 
-Once you are confident enough that these environments are in a good state with your
-feature enabled, you can roll out the change to GitLab.com.
+For these preproduction environments, the commands should be run in a
+Slack channel for the stage the feature is relevant to. For example, use the
+`#s_monitor` channel for features developed by the Monitor stage, Health
+group.
+
+To enable a feature for 25% of all users, run the following in Slack:
+
+```shell
+/chatops run feature set new_navigation_bar 25 --dev
+/chatops run feature set new_navigation_bar 25 --staging
+```
 
 ### Enabling a feature for GitLab.com
 
-Similar to above, to enable a feature for 25% of all users, run the following in
-Slack:
+When a feature has successfully been
+[enabled on a preproduction](#enabling-a-feature-for-preproduction-testing)
+environment and verified as safe and working, you can roll out the
+change to GitLab.com (production).
+
+#### Communicate the change
+
+Some feature flag changes on GitLab.com should be communicated with
+parts of the company. The developer responsible needs to determine
+whether this is necessary and the appropriate level of communication.
+This depends on the feature and what sort of impact it might have.
+
+As a guideline:
+
+- For simple features that are low-risk, and easily rolled back, then
+  just proceed to [enabling the feature in `#production`](#process).
+- For features that will impact user experience consider notifying
+  `#support_gitlab-com` beforehand.
+- For features with significant downstream effects (e.g.: turning on/off
+  Elasticsearch indexing) consider coordinating with `#production`
+  beforehand.
+
+#### Process
+
+Before toggling any feature flag, check that there are no ongoing
+significant incidents on GitLab.com. You can do this by checking the
+`#production` and `#incident-management` Slack channels, or looking for
+[open incident issues](https://gitlab.com/gitlab-com/gl-infra/production/issues/?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=incident)
+(although check the dates and times).
+
+We do not want to introduce changes during an incident, as it can make
+diagnosis and resolution of the incident much harder to achieve, and
+also will largely invalidate your rollout process as you will be unable
+to assess whether the rollout was without problems or not.
+
+If there is any doubt, ask in `#production`.
+
+The following `/chatops` commands should be performed in the Slack
+`#production` channel.
+
+When you begin to enable the feature, please link to the relevant
+Feature Flag Rollout Issue within a Slack thread of the first `/chatops`
+command you make so people can understand the change if they need to.
+
+To enable a feature for 25% of all users, run the following in Slack:
 
 ```shell
 /chatops run feature set new_navigation_bar 25
@@ -150,6 +168,23 @@ NOTE: **Note:**
 **Percentage of time** rollout is not a good idea if what you want is to make sure a feature
 is always on or off to the users.
 
+### Feature flag change logging
+
+Any feature flag change that affects GitLab.com (production) will
+automatically be logged in an issue.
+
+The issue is created in the
+[gl-infra/feature-flag-log](https://gitlab.com/gitlab-com/gl-infra/feature-flag-log/issues?scope=all&utf8=%E2%9C%93&state=closed)
+project, and it will at minimum log the Slack handle of person enabling
+a feature flag, the time, and the name of the flag being changed.
+
+The issue is then also posted to GitLab's internal
+[Grafana dashboard](https://dashboards.gitlab.net/) as an annotation
+marker to make the change even more visible.
+
+Changes to the issue format can be submitted in the
+[Chatops project](https://gitlab.com/gitlab-com/chatops).
+
 ## Cleaning up
 
 Once the change is deemed stable, submit a new merge request to remove the