Add dev docs for Code Intelligence

The document describes the design behind Code Intelligence

doc/development/README.md

@@ -99,6 +99,7 @@ Complementary reads:
 - [Code comments](code_comments.md)
 - [Renaming features](renaming_features.md)
 - [Windows Development on GCP](windows.md)
+- [Code Intelligence](code_intelligence/index.md)
 
 ## Performance guides
 

doc/development/code_intelligence/index.md

+# Code Intelligence
+
+> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/1576) in GitLab 13.1.
+
+This document describes the design behind [Code Intelligence](../../user/project/code_intelligence.md).
+
+GitLab's built-in Code Intelligence is powered by
+[LSIF](https://lsif.dev) and comes down to generating an LSIF document for a
+project in a CI job, processing the data, uploading it as a CI artifact and
+displaying this information for the files in the project.
+
+Here is a sequence diagram for uploading an LSIF artifact:
+
+```mermaid
+sequenceDiagram
+    participant Runner
+    participant Workhorse
+    participant Rails
+    participant Object Storage
+
+    Runner->>+Workhorse: POST /v4/jobs/:id/artifacts
+    Workhorse->>+Rails: POST /:id/artifacts/authorize
+    Rails-->>-Workhorse: Respond with ProcessLsif header
+    Note right of Workhorse: Process LSIF file
+    Workhorse->>+Object Storage: Put file
+    Object Storage-->>-Workhorse: request results
+    Workhorse->>+Rails: POST /:id/artifacts
+    Rails-->>-Workhorse: request results
+    Workhorse-->>-Runner: request results
+```
+
+1. The CI/CD job generates a document in an LSIF format (usually `dump.lsif`) using [an
+   indexer](https://lsif.dev) for the language of a project. The format
+   [describes](https://github.com/sourcegraph/sourcegraph/blob/master/doc/user/code_intelligence/writing_an_indexer.md)
+   interactions between a method or function and its definition(s) or references. The
+   document is marked to be stored as an LSIF report artifact.
+
+1. After receiving a request for storing the artifact, Workhorse asks
+   GitLab Rails to authorize the upload.
+
+1. GitLab Rails validates whether the artifact can be uploaded and sends
+   `ProcessLsif: true` header if the lsif artifact can be processed.
+
+1. Workhorse reads the LSIF document line by line and generates code intelligence
+   data for each file in the project. The output is a zipped directory of JSON
+   files which imitates the structure of the project:
+
+   Project:
+
+   ```code
+   app
+     controllers
+       application_controller.rb
+     models
+       application.rb
+   ```
+
+   Generated data:
+
+   ```code
+   app
+     controllers
+       application_controller.rb.json
+     models
+       application.rb.json
+   ```
+
+1. The zipped directory is stored as a ZIP artifact. Workhorse replaces the
+   original LSIF document with a set of JSON files in the ZIP artifact and
+   generates metadata for it. The metadata makes it possible to view a single
+   file in a ZIP file without unpacking or loading the whole file. That allows us
+   to access code intelligence data for a single file.
+
+1. When a file is viewed in the GitLab application, frontend fetches code
+   intelligence data for the file directly from the object storage. The file
+   contains information about code units in the file. For example:
+
+   ```json
+   [
+       {
+        "definition_path": "cmd/check/main.go#L4",
+        "hover": [
+            {
+                "language": "go",
+                "tokens": [
+                    [
+                        {
+                            "class": "kn",
+                            "value": "package"
+                        },
+                        {
+                            "value": " "
+                        },
+                        {
+                            "class": "s",
+                            "value": "\"fmt\""
+                        }
+                    ]
+                ]
+            },
+            {
+                "value": "Package fmt implements formatted I/O with functions analogous to C's printf and scanf.  The format 'verbs' are derived from C's but are simpler. \n\n### hdr-PrintingPrinting\nThe verbs: \n\nGeneral: \n\n```\n%v\tthe value in a default format\n\twhen printing st..."
+            }
+        ],
+        "start_char": 2,
+        "start_line": 33
+      }
+      ...
+    ]
+   ```