Commit 9393ff2f authored by Yorick Peterse's avatar Yorick Peterse

Document the need for feature flags

This adds a development guide explaining that we are going to use
feature flags more often, why, what the benefits are, and so on.

See https://gitlab.com/gitlab-org/gitlab-ce/issues/49619 for more
information.
parent 78b3eea7
...@@ -74,14 +74,31 @@ star, smile, etc.). Some good tips about code reviews can be found in our ...@@ -74,14 +74,31 @@ star, smile, etc.). Some good tips about code reviews can be found in our
## Feature freeze on the 7th for the release on the 22nd ## Feature freeze on the 7th for the release on the 22nd
After 7th at 23:59 (Pacific Time Zone) of each month, RC1 of the upcoming release (to be shipped on the 22nd) is created and deployed to GitLab.com and the stable branch for this release is frozen, which means master is no longer merged into it. After 7th at 23:59 (Pacific Time Zone) of each month, RC1 of the upcoming
Merge requests may still be merged into master during this period, release (to be shipped on the 22nd) is created and deployed to GitLab.com and
but they will go into the _next_ release, unless they are manually cherry-picked into the stable branch. the stable branch for this release is frozen, which means master is no longer
merged into it. Merge requests may still be merged into master during this
period, but they will go into the _next_ release, unless they are manually
cherry-picked into the stable branch.
By freezing the stable branches 2 weeks prior to a release, we reduce the risk of a last minute merge request potentially breaking things. By freezing the stable branches 2 weeks prior to a release, we reduce the risk
of a last minute merge request potentially breaking things.
Any release candidate that gets created after this date can become a final release, Any release candidate that gets created after this date can become a final
hence the name release candidate. release, hence the name release candidate.
### Feature flags
Merge requests that make changes hidden behind a feature flag, or remove an
existing feature flag because a feature is deemed stable, may be merged (and
picked into the stable branches) up to the 19th of the month. Such merge
requests should have the ~"feature flag" label assigned, and don't require a
corresponding exception request to be created.
While rare, release managers may decide to reject picking a change into a stable
branch, even when feature flags are used. This might be necessary if the changes
are deemed problematic, too invasive, or there simply isn't enough time to
properly test how the changes behave on GitLab.com.
### Between the 1st and the 7th ### Between the 1st and the 7th
...@@ -223,36 +240,36 @@ Check [this guide](https://gitlab.com/gitlab-org/release/docs/blob/master/genera ...@@ -223,36 +240,36 @@ Check [this guide](https://gitlab.com/gitlab-org/release/docs/blob/master/genera
A ~bug is a defect, error, failure which causes the system to behave incorrectly or prevents it from fulfilling the product requirements. A ~bug is a defect, error, failure which causes the system to behave incorrectly or prevents it from fulfilling the product requirements.
The level of impact of a ~bug can vary from blocking a whole functionality The level of impact of a ~bug can vary from blocking a whole functionality
or a feature usability bug. A bug should always be linked to a severity level. or a feature usability bug. A bug should always be linked to a severity level.
Refer to our [severity levels](../CONTRIBUTING.md#severity-labels) Refer to our [severity levels](../CONTRIBUTING.md#severity-labels)
Whether the bug is also a regression or not, the triage process should start as soon as possible. Whether the bug is also a regression or not, the triage process should start as soon as possible.
Ensure that the Engineering Manager and/or the Product Manager for the relative area is involved to prioritize the work as needed. Ensure that the Engineering Manager and/or the Product Manager for the relative area is involved to prioritize the work as needed.
### Regressions ### Regressions
A ~regression implies that a previously **verified working functionality** no longer works. A ~regression implies that a previously **verified working functionality** no longer works.
Regressions are a subset of bugs. We use the ~regression label to imply that the defect caused the functionality to regress. Regressions are a subset of bugs. We use the ~regression label to imply that the defect caused the functionality to regress.
The label tells us that something worked before and it needs extra attention from Engineering and Product Managers to schedule/reschedule. The label tells us that something worked before and it needs extra attention from Engineering and Product Managers to schedule/reschedule.
The regression label does not apply to ~bugs for new features for which functionality was **never verified as working**. The regression label does not apply to ~bugs for new features for which functionality was **never verified as working**.
These, by definition, are not regressions. These, by definition, are not regressions.
A regression should always have the `regression:xx.x` label on it to designate when it was introduced. A regression should always have the `regression:xx.x` label on it to designate when it was introduced.
Regressions should be considered high priority issues that should be solved as soon as possible, especially if they have severe impact on users. Regressions should be considered high priority issues that should be solved as soon as possible, especially if they have severe impact on users.
### Managing bugs ### Managing bugs
**Prioritization:** We give higher priority to regressions on features that worked in the last recent monthly release and the current release candidates. **Prioritization:** We give higher priority to regressions on features that worked in the last recent monthly release and the current release candidates.
The two scenarios below can [bypass the exception request in the release process](https://gitlab.com/gitlab-org/release/docs/blob/master/general/exception-request/process.md#after-the-7th), where the affected regression version matches the current monthly release version. The two scenarios below can [bypass the exception request in the release process](https://gitlab.com/gitlab-org/release/docs/blob/master/general/exception-request/process.md#after-the-7th), where the affected regression version matches the current monthly release version.
* A regression which worked in the **Last monthly release** * A regression which worked in the **Last monthly release**
* **Example:** In 11.0 we released a new `feature X` that is verified as working. Then in release 11.1 the feature no longer works, this is regression for 11.1. The issue should have the `regression:11.1` label. * **Example:** In 11.0 we released a new `feature X` that is verified as working. Then in release 11.1 the feature no longer works, this is regression for 11.1. The issue should have the `regression:11.1` label.
* *Note:* When we say `the last recent monthly release`, this can refer to either the version currently running on GitLab.com, or the most recent version available in the package repositories. * *Note:* When we say `the last recent monthly release`, this can refer to either the version currently running on GitLab.com, or the most recent version available in the package repositories.
* A regression which worked in the **Current release candidates** * A regression which worked in the **Current release candidates**
* **Example:** In 11.1-RC3 we shipped a new feature which has been verified as working. Then in 11.1-RC5 the feature no longer works, this is regression for 11.1. The issue should have the `regression:11.1` label. * **Example:** In 11.1-RC3 we shipped a new feature which has been verified as working. Then in 11.1-RC5 the feature no longer works, this is regression for 11.1. The issue should have the `regression:11.1` label.
* *Note:* Because GitLab.com runs release candidates of new releases, a regression can be reported in a release before its 'official' release date on the 22nd of the month. * *Note:* Because GitLab.com runs release candidates of new releases, a regression can be reported in a release before its 'official' release date on the 22nd of the month.
When a bug is found: When a bug is found:
1. Create an issue describing the problem in the most detailed way possible. 1. Create an issue describing the problem in the most detailed way possible.
...@@ -264,11 +281,11 @@ When a bug is found: ...@@ -264,11 +281,11 @@ When a bug is found:
The counterpart Product Manager is included to weigh-in on prioritization as needed. The counterpart Product Manager is included to weigh-in on prioritization as needed.
1. If the ~bug is **NOT** a regression: 1. If the ~bug is **NOT** a regression:
1. The Engineering Manager decides which milestone the bug will be fixed. The appropriate milestone is applied. 1. The Engineering Manager decides which milestone the bug will be fixed. The appropriate milestone is applied.
1. If the bug is a ~regression: 1. If the bug is a ~regression:
1. Determine the release that the regression affects and add the corresponding `regression:xx.x` label. 1. Determine the release that the regression affects and add the corresponding `regression:xx.x` label.
1. If the affected release version can't be determined, add the generic ~regression label for the time being. 1. If the affected release version can't be determined, add the generic ~regression label for the time being.
1. If the affected version `xx.x` in `regression:xx.x` is the **current release**, it's recommended to schedule the fix for the current milestone. 1. If the affected version `xx.x` in `regression:xx.x` is the **current release**, it's recommended to schedule the fix for the current milestone.
1. This falls under regressions which worked in the last release and the current RCs. More detailed explanations in the **Prioritization** section above. 1. This falls under regressions which worked in the last release and the current RCs. More detailed explanations in the **Prioritization** section above.
1. If the affected version `xx.x` in `regression:xx.x` is older than the **current release** 1. If the affected version `xx.x` in `regression:xx.x` is older than the **current release**
1. If the regression is an ~S1 severity, it's recommended to schedule the fix for the current milestone. We would like to fix the highest severity regression as soon as we can. 1. If the regression is an ~S1 severity, it's recommended to schedule the fix for the current milestone. We would like to fix the highest severity regression as soon as we can.
1. If the regression is an ~S2, ~S3 or ~S4 severity, the regression may be scheduled for later milestones at the discretion of the Engineering Manager and Product Manager. 1. If the regression is an ~S2, ~S3 or ~S4 severity, the regression may be scheduled for later milestones at the discretion of the Engineering Manager and Product Manager.
......
...@@ -65,13 +65,18 @@ In the rare case that you need the feature flag to be on automatically, use ...@@ -65,13 +65,18 @@ In the rare case that you need the feature flag to be on automatically, use
Feature.enabled?(:feature_flag, project, default_enabled: true) Feature.enabled?(:feature_flag, project, default_enabled: true)
``` ```
For more information about rolling out changes using feature flags, refer to the
[Rolling out changes using feature flags](rolling_out_changes_using_feature_flags.md)
guide.
### Specs ### Specs
In the test environment `Feature.enabled?` is stubbed to always respond to `true`, In the test environment `Feature.enabled?` is stubbed to always respond to `true`,
so we make sure behavior under feature flag doesn't go untested in some non-specific so we make sure behavior under feature flag doesn't go untested in some non-specific
contexts. contexts.
If you need to test the feature flag in a different state, you need to stub it with: Whenever a feature flag is present, make sure to test _both_ states of the
feature flag. You can stub a feature flag as follows:
```ruby ```ruby
stub_feature_flags(my_feature_flag: false) stub_feature_flags(my_feature_flag: false)
......
# Rolling out changes using feature flags
[Feature flags](feature_flags.md) can be used to gradually roll out changes, be
it a new feature, or a performance improvement. By using feature flags, we can
comfortably measure the impact of our changes, while still being able to easily
disable those changes, without having to revert an entire release.
## When to use feature flags
Starting with GitLab 11.4, developers are required to use feature flags for
non-trivial changes. Such changes include:
* New features (e.g. a new merge request widget, epics, etc).
* Complex performance improvements that may require additional testing in
production, such as rewriting complex queries.
* Invasive changes to the user interface, such as a new navigation bar or the
removal of a sidebar.
* Adding support for importing projects from a third-party service.
In all cases, those working on the changes can best decide if a feature flag is
necessary. For example, changing the color of a button doesn't need a feature
flag, while changing the navigation bar definitely needs one. In case you are
uncertain if a feature flag is necessary, simply ask about this in the merge
request, and those reviewing the changes will likely provide you with an answer.
When using a feature flag for UI elements, make sure to _also_ use a feature
flag for the underlying backend code, if there is any. This ensures there is
absolutely no way to use the feature until it is enabled.
## The cost of feature flags
When reading the above, one might be tempted to think this procedure is going to
add a lot of work. Fortunately, this is not the case, and we'll show why. For
this example we'll specify the cost of the work to do as a number, ranging from
0 to infinity. The greater the number, the more expensive the work is. The cost
does _not_ translate to time, it's just a way of measuring complexity of one
change relative to another.
Let's say we are building a new feature, and we have determined that the cost of
this is 10. We have also determined that the cost of adding a feature flag check
in a variety of places is 1. If we do not use feature flags, and our feature
works as intended, our total cost is 10. This however is the best case scenario.
Optimising for the best case scenario is guaranteed to lead to trouble, whereas
optimising for the worst case scenario is almost always better.
To illustrate this, let's say our feature causes an outage, and there's no
immediate way to resolve it. This means we'd have to take the following steps to
resolve the outage:
1. Revert the release.
1. Perform any cleanups that might be necessary, depending on the changes that
were made.
1. Revert the commit, ensuring the "master" branch remains stable. This is
especially necessary if solving the problem can take days or even weeks.
1. Pick the revert commit into the appropriate stable branches, ensuring we
don't block any future releases until the problem is resolved.
As history has shown, these steps are time consuming, complex, often involve
many developers, and worst of all: our users will have a bad experience using
GitLab.com until the problem is resolved.
Now let's say that all of this has an associated cost of 10. This means that in
the worst case scenario, which we should optimise for, our total cost is now 20.
If we had used a feature flag, things would have been very different. We don't
need to revert a release, and because feature flags are disabled by default we
don't need to revert and pick any Git commits. In fact, all we have to do is
disable the feature, and _maybe_ perform some cleanup. Let's say that the cost
of this is 1. In this case, our best case cost is 11: 10 to build the feature,
and 1 to add the feature flag. The worst case cost is now 12: 10 to build the
feature, 1 to add the feature flag, and 1 to disable it.
Here we can see that in the best case scenario the work necessary is only a tiny
bit more compared to not using a feature flag. Meanwhile, the process of
reverting our changes has been made significantly cheaper, to the point of being
trivial.
In other words, feature flags do not slow down the development process. Instead,
they speed up the process as managing incidents now becomes _much_ easier. Once
continuous deployments are easier to perform, the time to iterate on a feature
is reduced even further, as you no longer need to wait weeks before your changes
are available on GitLab.com.
## Rolling out changes
The procedure of using feature flags is straightforward, and similar to not
using them. You add the necessary tests (make sure to test both the on and off
states of your feature flag(s)), make sure they all pass, have the code
reviewed, etc. You then submit your merge request, and add the ~"feature flag"
label. This label is used to signal to release managers that your changes are
hidden behind a feature flag and that it is safe to pick the MR into a stable
branch, without the need for an exception request.
When the changes are deployed it is time to start rolling out the feature to our
users. The exact procedure of rolling out a change is unspecified, as this can
vary from change to change. However, in general we recommend rolling out changes
incrementally, instead of enabling them for everybody right away. We also
recommend you to _not_ enable a feature _before_ the code is being deployed.
This allows you to separate rolling out a feature from a deploy, making it
easier to measure the impact of both separately.
GitLab's feature library (using
[Flipper](https://github.com/jnunemaker/flipper), and covered in the [Feature
Flags](feature_flags.md) guide) supports rolling out changes to a percentage of
users. This in turn can be controlled using [GitLab
chatops](https://docs.gitlab.com/ee/ci/chatops/).
For example, to enable a feature for 25% of all users, run the following in
Slack:
```
/chatops run feature set new_navigation_bar 25
```
This will enable the feature for GitLab.com, with `new_navigation_bar` being the
name of the feature. We can also enable the feature for <https://dev.gitlab.org>
or <https://staging.gitlab.com>:
```
/chatops run feature set new_navigation_bar 25 --dev
/chatops run feature set new_navigation_bar 25 --staging
```
If you are not certain what percentages to use, simply use the following steps:
1. 25%
1. 50%
1. 75%
1. 100%
Between every step you'll want to wait a little while and monitor the
appropriate graphs on <https://dashboards.gitlab.net>. The exact time to wait
may differ. For some features a few minutes is enough, while for others you may
want to wait several hours or even days. This is entirely up to you, just make
sure it is clearly communicated to your team, and the Production team if you
anticipate any potential problems.
Once a change is deemed stable, submit a new merge request to remove the
feature flag. This ensures the change is available to all users and self-hosted
instances. Make sure to add the ~"feature flag" label to this merge request so
release managers are aware the changes are hidden behind a feature flag. If the
merge request has to be picked into a stable branch (e.g. after the 7th), make
sure to also add the appropriate "Pick into X" label (e.g. "Pick into 11.4").
One might be tempted to think this will delay the release of a feature by at
least one month (= one release). This is not the case. A feature flag does not
have to stick around for a specific amount of time (e.g. at least one release),
instead they should stick around until the feature is deemed stable. Stable
means it works on GitLab.com without causing any problems, such as outages. In
most cases this will translate to a feature (with a feature flag) being shipped
in RC1, followed by the feature flag being removed in RC2. This in turn means
the feature will be stable by the time we publish a stable package around the
22nd of the month.
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment