Merge branch 'jj-add-better-glex-docs' into 'master'

Split docs for GLEX and experimentation.rb

See merge request gitlab-org/gitlab!56052

doc/development/experiment_guide/experimentation.md

+---
+stage: Growth
+group: Activation
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+---
+
+# Create an A/B test with `Experimentation Module`
+
+## Implement the experiment
+
+1. Add the experiment to the `Gitlab::Experimentation::EXPERIMENTS` hash in
+   [`experimentation.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib%2Fgitlab%2Fexperimentation.rb):
+
+   ```ruby
+   EXPERIMENTS = {
+     other_experiment: {
+       #...
+     },
+     # Add your experiment here:
+     signup_flow: {
+       tracking_category: 'Growth::Activation::Experiment::SignUpFlow' # Used for providing the category when setting up tracking data
+     }
+   }.freeze
+   ```
+
+1. Use the experiment in the code.
+
+   Experiments can be performed on a `subject`. The provided `subject` should
+   respond to `to_global_id` or `to_s`.
+   The resulting string is bucketed and assigned to either the control or the
+   experimental group, so you must always provide the same `subject`
+   for an experiment to have the same experience.
+
+   1. Use this standard for the experiment in a controller:
+
+      - Experiment run for a user:
+
+        ```ruby
+        class ProjectController < ApplicationController
+          def show
+            # experiment_enabled?(:experiment_key) is also available in views and helpers
+            if experiment_enabled?(:signup_flow, subject: current_user)
+              # render the experiment
+            else
+              # render the original version
+            end
+          end
+        end
+        ```
+
+      - Experiment run for a namespace:
+
+        ```ruby
+        if experiment_enabled?(:signup_flow, subject: namespace)
+          # experiment code
+        else
+          # control code
+        end
+        ```
+
+      When no subject is given, it falls back to a cookie that gets set and is consistent until
+      the cookie gets deleted.
+
+      ```ruby
+      class RegistrationController < ApplicationController
+        def show
+          # falls back to a cookie
+          if experiment_enabled?(:signup_flow)
+            # render the experiment
+          else
+            # render the original version
+          end
+        end
+      end
+      ```
+
+   1. Make the experiment available to the frontend in a controller. This example
+      checks whether the experiment is enabled and pushes the result to the frontend:
+
+     ```ruby
+     before_action do
+       push_frontend_experiment(:signup_flow, subject: current_user)
+     end
+     ```
+
+     You can check the state of the feature flag in JavaScript:
+
+     ```javascript
+     import { isExperimentEnabled } from '~/experimentation';
+
+     if ( isExperimentEnabled('signupFlow') ) {
+       // ...
+     }
+     ```
+
+You can also run an experiment outside of the controller scope, such as in a worker:
+
+```ruby
+class SomeWorker
+  def perform
+    # Check if the experiment is active at all (the percentage_of_time_value > 0)
+    return unless Gitlab::Experimentation.active?(:experiment_key)
+
+    # Since we cannot access cookies in a worker, we need to bucket models
+    # based on a unique, unchanging attribute instead.
+    # It is therefore necessery to always provide the same subject.
+    if Gitlab::Experimentation.in_experiment_group?(:experiment_key, subject: user)
+      # execute experimental code
+    else
+      # execute control code
+    end
+  end
+end
+```
+
+## Implement tracking events
+
+To determine whether the experiment is a success or not, we must implement tracking events
+to acquire data for analyzing. We can send events to Snowplow via either the backend or frontend.
+Read the [product intelligence guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) for more details.
+
+### Track backend events
+
+The framework provides a helper method that is available in controllers:
+
+```ruby
+before_action do
+  track_experiment_event(:signup_flow, 'action', 'value', subject: current_user)
+end
+```
+
+To test it:
+
+```ruby
+context 'when the experiment is active and the user is in the experimental group' do
+  before do
+    stub_experiment(signup_flow: true)
+    stub_experiment_for_subject(signup_flow: true)
+  end
+
+  it 'tracks an event', :snowplow do
+    subject
+
+    expect_snowplow_event(
+      category: 'Growth::Activation::Experiment::SignUpFlow',
+      action: 'action',
+      value: 'value',
+      label: 'experimentation_subject_id',
+      property: 'experimental_group'
+    )
+  end
+end
+```
+
+### Track frontend events
+
+The framework provides a helper method that is available in controllers:
+
+```ruby
+before_action do
+  push_frontend_experiment(:signup_flow, subject: current_user)
+  frontend_experimentation_tracking_data(:signup_flow, 'action', 'value', subject: current_user)
+end
+```
+
+This pushes tracking data to `gon.experiments` and `gon.tracking_data`.
+
+```ruby
+expect(Gon.experiments['signupFlow']).to eq(true)
+
+expect(Gon.tracking_data).to eq(
+  {
+    category: 'Growth::Activation::Experiment::SignUpFlow',
+    action: 'action',
+    value: 'value',
+    label: 'experimentation_subject_id',
+    property: 'experimental_group'
+  }
+)
+```
+
+To track it:
+
+```javascript
+import { isExperimentEnabled } from '~/lib/utils/experimentation';
+import Tracking from '~/tracking';
+
+document.addEventListener('DOMContentLoaded', () => {
+  const signupFlowExperimentEnabled = isExperimentEnabled('signupFlow');
+
+  if (signupFlowExperimentEnabled && gon.tracking_data) {
+    const { category, action, ...data } = gon.tracking_data;
+
+    Tracking.event(category, action, data);
+  }
+}
+```
+
+To test it in Jest:
+
+```javascript
+import { withGonExperiment } from 'helpers/experimentation_helper';
+import Tracking from '~/tracking';
+
+describe('event tracking', () => {
+  describe('with tracking data', () => {
+    withGonExperiment('signupFlow');
+
+    beforeEach(() => {
+      jest.spyOn(Tracking, 'event').mockImplementation(() => {});
+
+      gon.tracking_data = {
+        category: 'Growth::Activation::Experiment::SignUpFlow',
+        action: 'action',
+        value: 'value',
+        label: 'experimentation_subject_id',
+        property: 'experimental_group'
+      };
+    });
+
+    it('should track data', () => {
+      performAction()
+
+      expect(Tracking.event).toHaveBeenCalledWith(
+        'Growth::Activation::Experiment::SignUpFlow',
+        'action',
+        {
+          value: 'value',
+          label: 'experimentation_subject_id',
+          property: 'experimental_group'
+        },
+      );
+    });
+  });
+});
+```
+
+## Record experiment user
+
+In addition to the anonymous tracking of events, we can also record which users
+have participated in which experiments, and whether they were given the control
+experience or the experimental experience.
+
+The `record_experiment_user` helper method is available to all controllers, and it
+enables you to record these experiment participants (the current user) and which
+experience they were given:
+
+```ruby
+before_action do
+  record_experiment_user(:signup_flow)
+end
+```
+
+Subsequent calls to this method for the same experiment and the same user have no
+effect unless the user is then enrolled into a different experience. This happens
+when we roll out the experimental experience to a greater percentage of users.
+
+This data is completely separate from the [events tracking data](#implement-tracking-events).
+They are not linked together in any way.
+
+### Add context
+
+You can add arbitrary context data in a hash which gets stored as part of the experiment
+user record. New calls to the `record_experiment_user` with newer contexts are merged
+deeply into the existing context.
+
+This data can then be used by data analytics dashboards.
+
+```ruby
+before_action do
+  record_experiment_user(:signup_flow, foo: 42, bar: { a: 22})
+  # context is { "foo" => 42, "bar" => { "a" => 22 }}
+end
+
+# Additional contexts for newer record calls are merged deeply
+record_experiment_user(:signup_flow, foo: 40, bar: { b: 2 }, thor: 3)
+# context becomes { "foo" => 40, "bar" => { "a" => 22, "b" => 2 }, "thor" => 3}
+```
+
+## Record experiment conversion event
+
+Along with the tracking of backend and frontend events and the
+[recording of experiment participants](#record-experiment-user), we can also record
+when a user performs the desired conversion event action. For example:
+
+- **Experimental experience:** Show an in-product nudge to test if the change causes more
+  people to sign up for trials.
+- **Conversion event:** The user starts a trial.
+
+The `record_experiment_conversion_event` helper method is available to all controllers.
+Use it to record the conversion event for the current user, regardless of whether
+the user is in the control or experimental group:
+
+```ruby
+before_action do
+  record_experiment_conversion_event(:signup_flow)
+end
+```
+
+Note that the use of this method requires that we have first
+[recorded the user](#record-experiment-user) as being part of the experiment.
+
+## Enable the experiment
+
+After all merge requests have been merged, use [ChatOps](../../ci/chatops/index.md) in the
+[appropriate channel](../feature_flags/controls.md#communicate-the-change) to start the experiment for 10% of the users.
+The feature flag should have the name of the experiment with the `_experiment_percentage` suffix appended.
+For visibility, share any commands run against production in the `#s_growth` channel:
+
+  ```shell
+  /chatops run feature set signup_flow_experiment_percentage 10
+  ```
+
+  If you notice issues with the experiment, you can disable the experiment by removing the feature flag:
+
+  ```shell
+  /chatops run feature delete signup_flow_experiment_percentage
+  ```
+
+## Add user to experiment group manually
+
+To force the application to add your current user into the experiment group,
+add a query string parameter to the path where the experiment runs. If you add the
+query string parameter, the experiment works only for this request, and doesn't work
+after following links or submitting forms.
+
+For example, to forcibly enable the `EXPERIMENT_KEY` experiment, add `force_experiment=EXPERIMENT_KEY`
+to the URL:
+
+```shell
+https://gitlab.com/<EXPERIMENT_ENTRY_URL>?force_experiment=<EXPERIMENT_KEY>
+```
+
+## Add user to experiment group with a cookie
+
+You can force the current user into the experiment group for `<EXPERIMENT_KEY>`
+during the browser session by using your browser's developer tools:
+
+```javascript
+document.cookie = "force_experiment=<EXPERIMENT_KEY>; path=/";
+```
+
+Use a comma to list more than one experiment to be forced:
+
+```javascript
+document.cookie = "force_experiment=<EXPERIMENT_KEY>,<ANOTHER_EXPERIMENT_KEY>; path=/";
+```
+
+To clear the experiments, unset the `force_experiment` cookie:
+
+```javascript
+document.cookie = "force_experiment=; path=/";
+```
+
+## Testing and test helpers
+
+### RSpec
+
+Use the following in RSpec to mock the experiment:
+
+```ruby
+context 'when the experiment is active' do
+  before do
+    stub_experiment(signup_flow: true)
+  end
+
+  context 'when the user is in the experimental group' do
+    before do
+      stub_experiment_for_subject(signup_flow: true)
+    end
+
+    it { is_expected.to do_experimental_thing }
+  end
+
+  context 'when the user is in the control group' do
+    before do
+      stub_experiment_for_subject(signup_flow: false)
+    end
+
+    it { is_expected.to do_control_thing }
+  end
+end
+```
+
+### Jest
+
+Use the following in Jest to mock the experiment:
+
+```javascript
+import { withGonExperiment } from 'helpers/experimentation_helper';
+
+describe('given experiment is enabled', () => {
+  withGonExperiment('signupFlow');
+
+  it('should do the experimental thing', () => {
+    expect(wrapper.find('.js-some-experiment-triggered-element')).toEqual(expect.any(Element));
+  });
+});
+```

doc/development/experiment_guide/gitlab_experiment.md

+---
+stage: Growth
+group: Adoption
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
+---
+
+# Implementing an A/B/n experiment using GLEX
+
+## Introduction
+
+`Gitlab::Experiment` (GLEX) is tightly coupled with the concepts provided by
+[Feature flags in development of GitLab](../feature_flags/index.md). Here, we refer
+to this layer as feature flags, and may also use the term Flipper, because we
+built our development and experiment feature flags atop it.
+
+You're strongly encouraged to read and understand the
+[Feature flags in development of GitLab](../feature_flags/index.md) portion of the
+documentation before considering running experiments. Experiments add additional
+concepts which may seem confusing or advanced without understanding the underpinnings
+of how GitLab uses feature flags in development. One concept: GLEX supports multivariate
+experiments, which are sometimes referred to as A/B/n tests.
+
+The [`gitlab-experiment` project](https://gitlab.com/gitlab-org/gitlab-experiment)
+exists in a separate repository, so it can be shared across any GitLab property that uses
+Ruby. You should feel comfortable reading the documentation on that project as well
+if you want to dig into more advanced topics.
+
+## Glossary of terms
+
+To ensure a shared language, you should understand these fundamental terms we use
+when communicating about experiments:
+
+- `experiment`: Any deviation of code paths we want to run at some times, but not others.
+- `context`: A consistent experience we provide in an experiment.
+- `control`: The default, or "original" code path.
+- `candidate`: Defines an experiment with only one code path.
+- `variant(s)`: Defines an experiment with multiple code paths.
+
+### How it works
+
+Use this decision tree diagram to understand how GLEX works. When an experiment runs,
+the following logic is executed to determine what variant should be provided,
+given how the experiment has been defined and using the provided context:
+
+```mermaid
+graph TD
+    GP[General Pool/Population] --> Enabled?
+    Running? -->|Yes| Cached?[Cached? / Pre-segmented?]
+    Running? -->|No| Excluded[Control / No Tracking]
+    Cached? -->|No| Excluded?
+    Cached? -->|Yes| Cached[Cached Value]
+    Excluded? -->|Yes / Cached| Excluded
+    Excluded? -->|No| Segmented?
+    Segmented? -->|Yes / Cached| VariantA
+    Segmented? -->|No| Included?[Experiment Group?]
+    Included? -->|Yes| Rollout
+    Included? -->|No| Control
+    Rollout -->|Cached| VariantA
+    Rollout -->|Cached| VariantB
+    Rollout -->|Cached| VariantC
+
+classDef included fill:#380d75,color:#ffffff,stroke:none
+classDef excluded fill:#fca121,stroke:none
+classDef cached fill:#2e2e2e,color:#ffffff,stroke:none
+classDef default fill:#fff,stroke:#6e49cb
+
+class VariantA,VariantB,VariantC included
+class Control,Excluded excluded
+class Cached cached
+```
+
+## Implement an experiment
+
+Start by generating a feature flag using the `bin/feature-flag` command as you
+normally would for a development feature flag, making sure to use `experiment` for
+the type. For the sake of documentation let's name our feature flag (and experiment)
+"pill_color".
+
+```shell
+bin/feature-flag pill_color -t experiment
+```
+
+After you generate the desired feature flag, you can immediately implement an
+experiment in code. An experiment implementation can be as simple as:
+
+```ruby
+experiment(:pill_color, actor: current_user) do |e|
+  e.use { 'control' }
+  e.try(:red) { 'red' }
+  e.try(:blue) { 'blue' }
+end
+```
+
+When this code executes, the experiment is run, a variant is assigned, and (if within a
+controller or view) a `window.gon.experiment.pillColor` object will be available in the
+client layer, with details like:
+
+- The assigned variant.
+- The context key for client tracking events.
+
+In addition, when an experiment runs, an event is tracked for
+the experiment `:assignment`. We cover more about events, tracking, and
+the client layer later.
+
+In local development, you can make the experiment active by using the feature flag
+interface. You can also target specific cases by providing the relevant experiment
+to the call to enable the feature flag:
+
+```ruby
+# Enable for everyone
+Feature.enable(:pill_color)
+
+# Get the `experiment` method -- already available in controllers, views, and mailers.
+include Gitlab::Experiment::Dsl
+# Enable for only the first user
+Feature.enable(:pill_color, experiment(:pill_color, actor: User.first))
+```
+
+To roll out your experiment feature flag on an environment, run
+the following command using ChatOps (which is covered in more depth in the
+[Feature flags in development of GitLab](../feature_flags/index.md) documentation).
+This command creates a scenario where half of everyone who encounters
+the experiment would be assigned the _control_, 25% would be assigned the _red_
+variant, and 25% would be assigned the _blue_ variant:
+
+```slack_slash_commands
+/chatops run feature set pill_color 50 --actors
+```
+
+For an even distribution in this example, change the command to set it to 66% instead
+of 50.
+
+NOTE:
+To immediately stop running an experiment, use the
+`/chatops run feature set pill_color false` command.
+
+<div class="panel panel-danger">
+**DANGER**
+{: .panel-heading}
+<div class="panel-body">
+
+We strongly recommend using the `--actors` flag when using the ChatOps commands,
+because anything else may give odd behaviors due to:
+
+- How the caching of variant assignment is handled.
+- How the default `percentage_of_time` is unpredictable and pseudo-random.
+
+</div>
+</div>
+
+We can also implement this experiment in a HAML file with HTML wrappings:
+
+```haml
+#cta-interface
+  - experiment(:pill_color, actor: current_user) do |e|
+    - e.use do
+      .pill-button control
+    - e.try(:red) do
+      .pill-button.red red
+    - e.try(:blue) do
+      .pill-button.blue blue
+```
+
+### The importance of context
+
+In our previous example experiment, our context (this is an important term) is a hash
+that's set to `{ actor: current_user }`. Context must be unique based on how you
+want to run your experiment, and should be understood at a lower level.
+
+It's expected, and recommended, that you use some of these
+contexts to simplify reporting:
+
+- `{ actor: current_user }`: Assigns a variant and is "sticky" to each user
+  (or "client" if `current_user` is nil) who enters the experiment.
+- `{ project: project }`: Assigns a variant and is "sticky" to the project currently
+  being viewed. If running your experiment is more useful when viewing a project,
+  rather than when a specific user is viewing any project, consider this approach.
+- `{ group: group }`: Similar to the project example, but applies to a wider
+  scope of projects and users.
+- `{ actor: current_user, project: project }`: Assigns a variant and is "sticky"
+  to the user who is viewing the given project. This creates a different variant
+  assignment possibility for every project that `current_user` views. Understand this
+  can create a large cache size if an experiment like this in a highly trafficked part
+  of the application.
+- `{ wday: Time.current.wday }`: Assigns a variant based on the current day of the
+  week. In this example, it would consistently assign one variant on Friday, and a
+  potentially different variant on Saturday.
+
+Context is critical to how you define and report on your experiment. It's usually
+the most important aspect of how you choose to implement your experiment, so consider
+it carefully, and discuss it with the wider team if needed. Also, take into account
+that the context you choose affects our cache size.
+
+After the above examples, we can state the general case: *given a specific
+and consistent context, we can provide a consistent experience and track events for
+that experience.* To dive a bit deeper into the implementation details: a context key
+is generated from the context that's provided. Use this context key to:
+
+- Determine the assigned variant.
+- Identify events tracked against that context key.
+
+We can think about this as the experience that we've rendered, which is both dictated
+and tracked by the context key. The context key is used to track the interaction and
+results of the experience we've rendered to that context key. These concepts are
+somewhat abstract and hard to understand initially, but this approach enables us to
+communicate about experiments as something that's wider than just user behavior.
+
+NOTE:
+Using `actor:` utilizes cookies if the `current_user` is nil. If you don't need
+cookies though - meaning that the exposed functionality would only be visible to
+signed in users - `{ user: current_user }` would be just as effective.
+
+WARNING:
+The caching of variant assignment is done by using this context, and so consider
+your impact on the cache size when defining your experiment. If you use
+`{ time: Time.current }` you would be inflating the cache size every time the
+experiment is run. Not only that, your experiment would not be "sticky" and events
+wouldn't be resolvable.
+
+### Advanced experimentation
+
+GLEX allows for two general implementation styles:
+
+1. The simple experiment style described previously.
+1. A more advanced style where an experiment class can be provided.
+
+The advanced style is handled by naming convention, and works similar to what you
+would expect in Rails.
+
+To generate a custom experiment class that can override the defaults in
+`ApplicationExperiment` (our base GLEX implementation), use the rails generator:
+
+```shell
+rails generate gitlab:experiment pill_color control red blue
+```
+
+This generates an experiment class in `app/experiments/pill_color_experiment.rb`
+with the variants (or _behaviors_) we've provided to the generator. Here's an example
+of how that class would look after migrating the previous example into it:
+
+```ruby
+class PillColorExperiment < ApplicationExperiment
+  def control_behavior
+    'control'
+  end
+
+  def red_behavior
+    'red'
+  end
+
+  def blue_behavior
+    'blue'
+  end
+end
+```
+
+We can now simplify where we run our experiment to the following call, instead of
+providing the block we were initially providing, by explicitly calling `run`:
+
+```ruby
+experiment(:pill_color, actor: current_user).run
+```
+
+The _behavior_ methods we defined in our experiment class represent the default
+implementation. You can still use the block syntax to override these _behavior_
+methods however, so the following would also be valid:
+
+```ruby
+experiment(:pill_color, actor: current_user) do |e|
+  e.use { '<strong>control</strong>' }
+end
+```
+
+NOTE:
+When passing a block to the `experiment` method, it is implicitly invoked as
+if `run` has been called.
+
+#### Segmentation rules
+
+You can use runtime segmentation rules to, for instance, segment contexts into a specific
+variant. The `segment` method is a callback (like `before_action`) and so allows providing
+a block or method name.
+
+In this example, any user named `'Richard'` would always be assigned the _red_
+variant, and any account older than 2 weeks old would be assigned the _blue_ variant:
+
+```ruby
+class PillColorExperiment < ApplicationExperiment
+  segment(variant: :red) { context.actor.first_name == 'Richard' }
+  segment :old_account?, variant: :blue
+
+  # ...behaviors
+
+  private
+
+  def old_account?
+    context.actor.created_at < 2.weeks.ago
+  end
+end
+```
+
+When an experiment runs, the segmentation rules are executed in the order they're
+defined. The first segmentation rule to produce a truthy result assigns the variant.
+
+In our example, any user named `'Richard'`, regardless of account age, will always
+be assigned the _red_ variant. If you want the opposite logic, flip the order.
+
+NOTE:
+Keep in mind when defining segmentation rules: after a truthy result, the remaining
+segmentation rules are skipped to achieve optimal performance.
+
+#### Exclusion rules
+
+Exclusion rules are similar to segmentation rules, but are intended to determine
+if a context should even be considered as something we should include in the experiment
+and track events toward. Exclusion means we don't care about the events in relation
+to the given context.
+
+These examples exclude all users named `'Richard'`, *and* any account
+older than 2 weeks old. Not only are they given the control behavior - which could
+be nothing - but no events are tracked in these cases as well.
+
+```ruby
+class PillColorExperiment < ApplicationExperiment
+  exclude :old_account?, ->{ context.actor.first_name == 'Richard' }
+
+  # ...behaviors
+
+  private
+
+  def old_account?
+    context.actor.created_at < 2.weeks.ago
+  end
+end
+```
+
+We can also do exclusion when we run the experiment. For instance,
+if we wanted to prevent the inclusion of non-administrators in an experiment, consider
+the following experiment. This type of logic enables us to do complex experiments
+while preventing us from passing things into our experiments, because
+we want to minimize passing things into our experiments:
+
+```ruby
+experiment(:pill_color, actor: current_user) do |e|
+  e.exclude! unless can?(current_user, :admin_project, project)
+end
+```
+
+You may also need to check exclusion in custom tracking logic by calling `should_track?`:
+
+```ruby
+class PillColorExperiment < ApplicationExperiment
+  # ...behaviors
+
+  def expensive_tracking_logic
+    return unless should_track?
+
+    track(:my_event, value: expensive_method_call)
+  end
+end
+```
+
+Exclusion rules aren't the best way to determine if an experiment is active. Override
+the `enabled?` method for a high-level way of determining if an experiment should
+run and track. Make the `enabled?` check as efficient as possible because it's the
+first early opt-out path an experiment can implement.
+
+### Tracking events
+
+One of the most important aspects of experiments is gathering data and reporting on
+it. GLEX provides an interface that allows tracking events across an experiment.
+You can implement it consistently if you provide the same context between
+calls to your experiment. If you do not yet understand context, you should read
+about contexts now.
+
+We can assume we run the experiment in one or a few places, but
+track events potentially in many places. The tracking call remains the same, with
+the arguments you would normally use when
+[tracking events using snowplow](../snowplow.md). The easiest example
+of tracking an event in Ruby would be:
+
+```ruby
+experiment(:pill_color, actor: current_user).track(:created)
+```
+
+When you run an experiment with any of these examples, an `:assigned` event
+is tracked automatically by default. All events that are tracked from an
+experiment have a special
+[experiment context](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_experiment/jsonschema/1-0-0)
+added to the event. This can be used - typically by the data team - to create a connection
+between the events on a given experiment.
+
+If our current user hasn't encountered the experiment yet (meaning where the experiment
+is run), and we track an event for them, they are assigned a variant and see
+that variant if they ever encountered the experiment later, when an `:assignment`
+event would be tracked at that time for them.
+
+NOTE:
+GitLab tries to be sensitive and respectful of our customers regarding tracking,
+so GLEX allows us to implement an experiment without ever tracking identifying
+IDs. It's not always possible, though, based on experiment reporting requirements.
+You may be asked from time to time to track a specific record ID in experiments.
+The approach is largely up to the PM and engineer creating the implementation.
+No recommendations are provided here at this time.
+
+## Test with RSpec
+
+This gem provides some RSpec helpers and custom matchers. These are in flux as of GitLab 13.10.
+
+First, require the RSpec support file to mix in some of the basics:
+
+```ruby
+require 'gitlab/experiment/rspec'
+```
+
+You still need to include matchers and other aspects, which happens
+automatically for files in `spec/experiments`, but for other files and specs
+you want to include it in, you can specify the `:experiment` type:
+
+```ruby
+it "tests", :experiment do
+end
+```
+
+### Stub helpers
+
+You can stub experiments using `stub_experiments`. Pass it a hash using experiment
+names as the keys, and the variants you want each to resolve to, as the values:
+
+```ruby
+# Ensures the experiments named `:example` & `:example2` are both
+# "enabled" and that each will resolve to the given variant
+# (`:my_variant` & `:control` respectively).
+stub_experiments(example: :my_variant, example2: :control)
+
+experiment(:example) do |e|
+  e.enabled? # => true
+  e.variant.name # => 'my_variant'
+end
+
+experiment(:example2) do |e|
+  e.enabled? # => true
+  e.variant.name # => 'control'
+end
+```
+
+### Exclusion and segmentation matchers
+
+You can also test the exclusion and segmentation matchers.
+
+```ruby
+class ExampleExperiment < ApplicationExperiment
+  exclude { context.actor.first_name == 'Richard' }
+  segment(variant: :candidate) { context.actor.username == 'jejacks0n' }
+end
+
+excluded = double(username: 'rdiggitty', first_name: 'Richard')
+segmented = double(username: 'jejacks0n', first_name: 'Jeremy')
+
+# exclude matcher
+expect(experiment(:example)).to exclude(actor: excluded)
+expect(experiment(:example)).not_to exclude(actor: segmented)
+
+# segment matcher
+expect(experiment(:example)).to segment(actor: segmented).into(:candidate)
+expect(experiment(:example)).not_to segment(actor: excluded)
+```
+
+### Tracking matcher
+
+Tracking events is a major aspect of experimentation. We try
+to provide a flexible way to ensure your tracking calls are covered.
+
+You can do this on the instance level or at an "any instance" level:
+
+```ruby
+subject = experiment(:example)
+
+expect(subject).to track(:my_event)
+
+subject.track(:my_event)
+```
+
+You can use the `on_any_instance` chain method to specify that it could happen on
+any instance of the experiment. This helps you if you're calling
+`experiment(:example).track` downstream:
+
+```ruby
+expect(experiment(:example)).to track(:my_event).on_any_instance
+
+experiment(:example).track(:my_event)
+```
+
+A full example of the methods you can chain onto the `track` matcher:
+
+```ruby
+expect(experiment(:example)).to track(:my_event, value: 1, property: '_property_')
+  .on_any_instance
+  .with_context(foo: :bar)
+  .for(:variant_name)
+
+experiment(:example, :variant_name, foo: :bar).track(:my_event, value: 1, property: '_property_')
+```
+
+## Experiments in the client layer
+
+This is in flux as of GitLab 13.10, and can't be documented just yet.
+
+Any experiment that's been run in the request lifecycle surfaces in `window.gon.experiment`,
+and matches [this schema](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_experiment/jsonschema/1-0-0)
+so you can use it when resolving some concepts around experimentation in the client layer.
+
+## Notes on feature flags
+
+NOTE:
+We use the terms "enabled" and "disabled" here, even though it's against our
+[documentation style guide recommendations](../documentation/styleguide/index.md#avoid-ableist-language)
+because these are the terms that the feature flag documentation uses.
+
+You may already be familiar with the concept of feature flags in GitLab, but using
+feature flags in experiments is a bit different. While in general terms, a feature flag
+is viewed as being either `on` or `off`, this isn't accurate for experiments.
+
+Generally, `off` means that when we ask if a feature flag is enabled, it will always
+return `false`, and `on` means that it will always return `true`. An interim state,
+considered `conditional`, also exists. GLEX takes advantage of this trinary state of
+feature flags. To understand this `conditional` aspect: consider that either of these
+settings puts a feature flag into this state:
+
+- Setting a `percentage_of_actors` of any percent greater than 0%.
+- Enabling it for a single user or group.
+
+Conditional means that it returns `true` in some situations, but not all situations.
+
+When a feature flag is disabled (meaning the state is `off`), the experiment is
+considered _inactive_. You can visualize this in the [decision tree diagram](#how-it-works)
+as reaching the first [Running?] node, and traversing the negative path.
+
+When a feature flag is rolled out to a `percentage_of_actors` or similar (meaning the
+state is `conditional`) the experiment is considered to be _running_
+where sometimes the control is assigned, and sometimes the candidate is assigned.
+We don't refer to this as being enabled, because that's a confusing and overloaded
+term here. In the experiment terms, our experiment is _running_, and the feature flag is
+`conditional`.
+
+When a feature flag is enabled (meaning the state is `on`), the candidate will always be
+assigned.
+
+We should try to be consistent with our terms, and so for experiments, we have an
+_inactive_ experiment until we set the feature flag to `conditional`. After which,
+our experiment is then considered _running_. If you choose to "enable" your feature flag,
+you should consider the experiment to be _resolved_, because everyone is assigned
+the candidate unless they've opted out of experimentation.
+
+As of GitLab 13.10, work is being done to improve this process and how we communicate
+about it.

doc/development/experiment_guide/index.md

@@ -36,396 +36,27 @@ and link to the issue that resolves the experiment. If the experiment is
 successful and becomes part of the product, any follow up issues should be
 addressed.
 
-## Implement an experiment
+## Implementing an experiment
 
-There are two options to conduct experiments:
+There are currently two options when implementing an experiment.
 
-1. [GitLab Experiment](https://gitlab.com/gitlab-org/gitlab-experiment/) is a gem included in GitLab.
-1. [`Experimentation Module`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib%2Fgitlab%2Fexperimentation.rb) is built in the GitLab codebase.
+One is built into GitLab directly and has been around for a while (this is called
+`Exerimentation Module`), and the other is provided by
+[`gitlab-experiment`](https://gitlab.com/gitlab-org/gitlab-experiment) and is referred
+to as `Gitlab::Experiment` -- GLEX for short.
 
-Both methods use [experiment](../feature_flags/development.md#experiment-type) feature flags.
+Both approaches use [experiment](../feature_flags/development.md#experiment-type)
+feature flags, and there is currently no strong suggestion to use one over the other.
 
-Historical Context: `Experimentation Module` was built iteratively with the needs that appeared while implementing Growth sub-department experiments. The `gitlab-experiment` gem was built with the learnings of the `Experimentation Module` and an easier to use API.
+| Feature              | `Experimentation Module` | GLEX |
+| -------------------- |------------------------- | ---- |
+| Record user grouping | Yes                      | No   |
+| Uses feature flags   | Yes                      | Yes  |
+| Multivariate (A/B/n) | No                       | Yes  |
 
-Currently both methods for running experiments are included in the codebase. The features are slightly different:
+- [Implementing an A/B experiment using `Experimentation Module`](experimentation.md)
+- [Implementing an A/B/n experiment using GLEX](gitlab_experiment.md)
 
-| Feature | `Experiment Module` | `gitlab-experiment` |
-| ------ | ------ | ------ |
-| Record user grouping | Yes | No (not natively) |
-| Uses feature flags | Yes | Yes |
-| Multivariate | No | Yes |
-
-However, there is currently no strong suggestion to use one over the other.
-
-### Experiments using `gitlab-experiment` **(FREE SAAS)**
-
-> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300383) in GitLab 13.7.
-> - It's [deployed behind a feature flag](../../user/feature_flags.md), disabled by default.
-> - It's enabled on GitLab.com.
-> - It is not yet intended for use in GitLab self-managed instances.
-
-You find out how to conduct experiments using `gitlab-experiment` in the [README](https://gitlab.com/gitlab-org/gitlab-experiment/-/blob/master/README.md).
-
-### Experiments using the `Experimentation Module`
-
-1. Add the experiment to the `Gitlab::Experimentation::EXPERIMENTS` hash in the [`Experimentation Module`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib%2Fgitlab%2Fexperimentation.rb):
-
-   ```ruby
-   EXPERIMENTS = {
-     other_experiment: {
-       #...
-     },
-     # Add your experiment here:
-     signup_flow: {
-       tracking_category: 'Growth::Activation::Experiment::SignUpFlow' # Used for providing the category when setting up tracking data
-     }
-   }.freeze
-   ```
-
-1. Use the experiment in the code.
-
-   Experiments can be performed on a `subject`. The `subject` that gets provided needs to respond to `to_global_id` or `to_s`.
-   The resulting string is bucketed and assigned to either the control or the experimental group. It's therefore necessary to always provide the same `subject` for an experiment to have the same experience.
-
-   - Use this standard for the experiment in a controller:
-
-     Experiment run for a user:
-
-     ```ruby
-     class ProjectController < ApplicationController
-       def show
-         # experiment_enabled?(:experiment_key) is also available in views and helpers
-         if experiment_enabled?(:signup_flow, subject: current_user)
-           # render the experiment
-         else
-           # render the original version
-         end
-       end
-     end
-     ```
-
-     or experiment run for a namespace:
-
-     ```ruby
-     if experiment_enabled?(:signup_flow, subject: namespace)
-       # experiment code
-     else
-       # control code
-     end
-     ```
-
-     When no subject is given, it falls back to a cookie that gets set and is consistent until
-     the cookie gets deleted.
-
-     ```ruby
-     class RegistrationController < ApplicationController
-       def show
-         # falls back to a cookie
-         if experiment_enabled?(:signup_flow)
-           # render the experiment
-         else
-           # render the original version
-         end
-       end
-     end
-     ```
-
-   - Make the experiment available to the frontend in a controller:
-
-     ```ruby
-     before_action do
-       push_frontend_experiment(:signup_flow, subject: current_user)
-     end
-     ```
-
-     The above checks whether the experiment is enabled and pushes the result to the frontend.
-
-     The Frontend helpers for this are no longer used in production.
-     [More details TBD](https://gitlab.com/gitlab-org/gitlab/-/issues/323934).
-
-   - It is also possible to run an experiment outside of the controller scope, for example in a worker:
-
-     ```ruby
-     class SomeWorker
-       def perform
-         # Check if the experiment is active at all (the percentage_of_time_value > 0)
-         return unless Gitlab::Experimentation.active?(:experiment_key)
-
-         # Since we cannot access cookies in a worker, we need to bucket models based on a unique, unchanging attribute instead.
-         # It is therefore necessery to always provide the same subject.
-         if Gitlab::Experimentation.in_experiment_group?(:experiment_key, subject: user)
-           # execute experimental code
-         else
-           # execute control code
-         end
-       end
-     end
-     ```
-
-#### Implement the tracking events
-
-To determine whether the experiment is a success or not, we must implement tracking events
-to acquire data for analyzing. We can send events to Snowplow via either the backend or frontend.
-Read the [product intelligence guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) for more details.
-
-##### Track backend events
-
-The framework provides the following helper method that is available in controllers:
-
-```ruby
-before_action do
-  track_experiment_event(:signup_flow, 'action', 'value', subject: current_user)
-end
-```
-
-Which can be tested as follows:
-
-```ruby
-context 'when the experiment is active and the user is in the experimental group' do
-  before do
-    stub_experiment(signup_flow: true)
-    stub_experiment_for_subject(signup_flow: true)
-  end
-
-  it 'tracks an event', :snowplow do
-    subject
-
-    expect_snowplow_event(
-      category: 'Growth::Activation::Experiment::SignUpFlow',
-      action: 'action',
-      value: 'value',
-      label: 'experimentation_subject_id',
-      property: 'experimental_group'
-    )
-  end
-end
-```
-
-##### Track frontend events
-
-The framework provides the following helper method that is available in controllers:
-
-```ruby
-before_action do
-  push_frontend_experiment(:signup_flow, subject: current_user)
-  frontend_experimentation_tracking_data(:signup_flow, 'action', 'value', subject: current_user)
-end
-```
-
-This pushes tracking data to `gon.experiments` and `gon.tracking_data`.
-
-```ruby
-expect(Gon.experiments['signupFlow']).to eq(true)
-
-expect(Gon.tracking_data).to eq(
-  {
-    category: 'Growth::Activation::Experiment::SignUpFlow',
-    action: 'action',
-    value: 'value',
-    label: 'experimentation_subject_id',
-    property: 'experimental_group'
-  }
-)
-```
-
-Which can then be used for tracking as follows:
-
-```javascript
-import Tracking from '~/tracking';
-
-document.addEventListener('DOMContentLoaded', () => {
-  const signupFlowExperimentEnabled = gon.experiments['signupFlow'];
-
-  if (signupFlowExperimentEnabled && gon.tracking_data) {
-    const { category, action, ...data } = gon.tracking_data;
-
-    Tracking.event(category, action, data);
-  }
-}
-```
-
-Which can be tested in Jest as follows:
-
-```javascript
-import { withGonExperiment } from 'helpers/experimentation_helper';
-import Tracking from '~/tracking';
-
-describe('event tracking', () => {
-  describe('with tracking data', () => {
-    withGonExperiment('signupFlow');
-
-    beforeEach(() => {
-      jest.spyOn(Tracking, 'event').mockImplementation(() => {});
-
-      gon.tracking_data = {
-        category: 'Growth::Activation::Experiment::SignUpFlow',
-        action: 'action',
-        value: 'value',
-        label: 'experimentation_subject_id',
-        property: 'experimental_group'
-      };
-    });
-
-    it('should track data', () => {
-      performAction()
-
-      expect(Tracking.event).toHaveBeenCalledWith(
-        'Growth::Activation::Experiment::SignUpFlow',
-        'action',
-        {
-          value: 'value',
-          label: 'experimentation_subject_id',
-          property: 'experimental_group'
-        },
-      );
-    });
-  });
-});
-```
-
-#### Record experiment user
-
-In addition to the anonymous tracking of events, we can also record which users have participated in which experiments and whether they were given the control experience or the experimental experience.
-
-The `record_experiment_user` helper method is available to all controllers, and it enables you to record these experiment participants (the current user) and which experience they were given:
-
-```ruby
-before_action do
-  record_experiment_user(:signup_flow)
-end
-```
-
-Subsequent calls to this method for the same experiment and the same user have no effect unless the user has gets enrolled into a different experience. This happens when we roll out the experimental experience to a greater percentage of users.
-
-Note that this data is completely separate from the [events tracking data](#implement-the-tracking-events). They are not linked together in any way.
-
-##### Add context
-
-You can add arbitrary context data in a hash which gets stored as part of the experiment user record. New calls to the `record_experiment_user` with newer contexts get merged deeply into the existing context.
-
-This data can then be used by data analytics dashboards.
-
-```ruby
-before_action do
-  record_experiment_user(:signup_flow, foo: 42, bar: { a: 22})
-  # context is { "foo" => 42, "bar" => { "a" => 22 }}
-end
-
-# Additional contexts for newer record calls are merged deeply
-record_experiment_user(:signup_flow, foo: 40, bar: { b: 2 }, thor: 3)
-# context becomes { "foo" => 40, "bar" => { "a" => 22, "b" => 2 }, "thor" => 3}
-```
-
-#### Record experiment conversion event
-
-Along with the tracking of backend and frontend events and the [recording of experiment participants](#record-experiment-user), we can also record when a user performs the desired conversion event action. For example:
-
-- **Experimental experience:** Show an in-product nudge to see if it causes more people to sign up for trials.
-- **Conversion event:** The user starts a trial.
-
-The `record_experiment_conversion_event` helper method is available to all controllers. It enables us to record the conversion event for the current user, regardless of whether they are in the control or experimental group:
-
-```ruby
-before_action do
-  record_experiment_conversion_event(:signup_flow)
-end
-```
-
-Note that the use of this method requires that we have first [recorded the user as being part of the experiment](#record-experiment-user).
-
-#### Enable the experiment
-
-After all merge requests have been merged, use [`chatops`](../../ci/chatops/index.md) in the
-[appropriate channel](../feature_flags/controls.md#communicate-the-change) to start the experiment for 10% of the users.
-The feature flag should have the name of the experiment with the `_experiment_percentage` suffix appended.
-For visibility, please also share any commands run against production in the `#s_growth` channel:
-
-  ```shell
-  /chatops run feature set signup_flow_experiment_percentage 10
-  ```
-
-  If you notice issues with the experiment, you can disable the experiment by removing the feature flag:
-
-  ```shell
-  /chatops run feature delete signup_flow_experiment_percentage
-  ```
-
-#### Manually force the current user to be in the experiment group
-
-You may force the application to put your current user in the experiment group. To do so
-add a query string parameter to the path where the experiment runs. If you do so,
-the experiment will work only for this request and won't work after following links or submitting forms.
-
-For example, to forcibly enable the `EXPERIMENT_KEY` experiment, add `force_experiment=EXPERIMENT_KEY`
-to the URL:
-
-```shell
-https://gitlab.com/<EXPERIMENT_ENTRY_URL>?force_experiment=<EXPERIMENT_KEY>
-```
-
-#### A cookie-based approach to force an experiment
-
-It's possible to force the current user to be in the experiment group for `<EXPERIMENT_KEY>`
-during the browser session by using your browser's developer tools:
-
-```javascript
-document.cookie = "force_experiment=<EXPERIMENT_KEY>; path=/";
-```
-
-Use a comma to list more than one experiment to be forced:
-
-```javascript
-document.cookie = "force_experiment=<EXPERIMENT_KEY>,<ANOTHER_EXPERIMENT_KEY>; path=/";
-```
-
-To clear the experiments, unset the `force_experiment` cookie:
-
-```javascript
-document.cookie = "force_experiment=; path=/";
-```
-
-#### Testing and test helpers
-
-##### RSpec
-
-Use the following in RSpec to mock the experiment:
-
-```ruby
-context 'when the experiment is active' do
-  before do
-    stub_experiment(signup_flow: true)
-  end
-
-  context 'when the user is in the experimental group' do
-    before do
-      stub_experiment_for_subject(signup_flow: true)
-    end
-
-    it { is_expected.to do_experimental_thing }
-  end
-
-  context 'when the user is in the control group' do
-    before do
-      stub_experiment_for_subject(signup_flow: false)
-    end
-
-    it { is_expected.to do_control_thing }
-  end
-end
-```
-
-##### Jest
-
-Use the following in Jest to mock the experiment:
-
-```javascript
-import { withGonExperiment } from 'helpers/experimentation_helper';
-
-describe('given experiment is enabled', () => {
-  withGonExperiment('signupFlow');
-
-  it('should do the experimental thing', () => {
-    expect(wrapper.find('.js-some-experiment-triggered-element')).toEqual(expect.any(Element));
-  });
-});
-```
+Historical Context: `Experimentation Module` was built iteratively with the needs that
+appeared while implementing Growth sub-department experiments, while GLEX was built
+with the learnings of the team and an easier to use API.