Commit 23ab5439 authored by Stan Hu's avatar Stan Hu Committed by Marcia Ramos

Add docs on the inclusion of LFS files in archives

Part of https://gitlab.com/gitlab-org/gitlab/-/issues/15079
parent ea044e31
......@@ -112,10 +112,12 @@ Parameters:
## Get file archive
> Support for [including Git LFS blobs](../topics/git/lfs/index.md#lfs-objects-in-project-archives) was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15079) in GitLab 13.5.
Get an archive of the repository. This endpoint can be accessed without
authentication if the repository is publicly accessible.
This endpoint has a rate limit threshold of 5 requests per minute.
This endpoint has a rate limit threshold of 5 requests per minute for GitLab.com users.
```plaintext
GET /projects/:id/repository/archive[.format]
......
......@@ -10,3 +10,74 @@ and the slides on [Google Slides](https://docs.google.com/presentation/d/1E-aw6-
and in [PDF](https://gitlab.com/gitlab-org/create-stage/uploads/07a89257a140db067bdfb484aecd35e1/Git_LFS_Deep_Dive__Create_.pdf).
Everything covered in this deep dive was accurate as of GitLab 11.10, and while specific
details may have changed since then, it should still serve as a good introduction.
## Including LFS blobs in project archives
> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15079) in GitLab 13.5.
The following diagram illustrates how GitLab resolves LFS files for project archives:
```mermaid
sequenceDiagram
autonumber
Client->>+Workhorse: GET /group/project/-/archive/master.zip
Workhorse->>+Rails: GET /group/project/-/archive/master.zip
Rails->>+Workhorse: Gitlab-Workhorse-Send-Data git-archive
Workhorse->>Gitaly: SendArchiveRequest
Gitaly->>Git: git archive master
Git->>Smudge: OID 12345
Smudge->>+Workhorse: GET /internal/api/v4/lfs?oid=12345&gl_repository=project-1234
Workhorse->>+Rails: GET /internal/api/v4/lfs?oid=12345&gl_repository=project-1234
Rails->>+Workhorse: Gitlab-Workhorse-Send-Data send-url
Workhorse->>Smudge: <LFS data>
Smudge->>Git: <LFS data>
Git->>Gitaly: <streamed data>
Gitaly->>Workhorse: <streamed data>
Workhorse->>Client: master.zip
```
1. The user requests the project archive from the UI.
1. Workhorse forwards this request to Rails.
1. If the user is authorized to download the archive, Rails replies with
an HTTP header of `Gitlab-Workhorse-Send-Data` with a base64-encoded
JSON payload prefaced with `git-archive`. This payload includes the
`SendArchiveRequest` binary message, which is encoded again in base64.
1. Workhorse decodes the `Gitlab-Workhorse-Send-Data` payload. If the
archive already exists in the archive cache, Workhorse sends that
file. Otherwise, Workhorse sends the `SendArchiveRequest` to the
appropriate Gitaly server.
1. The Gitaly server will call `git archive <ref>` to begin generating
the Git archive on-the-fly. If the `include_lfs_blobs` flag is enabled,
Gitaly enables a custom LFS smudge filter via the `-c
filter.lfs.smudge=/path/to/gitaly-lfs-smudge` Git option.
1. When `git` identifies a possible LFS pointer using the
`.gitattributes` file, `git` calls `gitaly-lfs-smudge` and provides the
LFS pointer via the standard input. Gitaly provides `GL_PROJECT_PATH`
and `GL_INTERNAL_CONFIG` as environment variables to enable lookup of
the LFS object.
1. If a valid LFS pointer is decoded, `gitaly-lfs-smudge` makes an
internal API call to Workhorse to download the LFS object from GitLab.
1. Workhorse forwards this request to Rails. If the LFS object exists
and is associated with the project, Rails sends `ArchivePath` either
with a path where the LFS object resides (for local disk) or a
pre-signed URL (when object storage is enabled) via the
`Gitlab-Workhorse-Send-Data` HTTP header with a payload prefaced with
`send-url`.
1. Workhorse retrieves the file and send it to the `gitaly-lfs-smudge`
process, which writes the contents to the standard output.
1. `git` reads this output and sends it back to the Gitaly process.
1. Gitaly sends the data back to Rails.
1. The archive data is sent back to the client.
In step 7, the `gitaly-lfs-smudge` filter must talk to Workhorse, not to
Rails, or an invalid LFS blob will be saved. To support this, GitLab
13.5 [changed the default Omnibus configuration to have Gitaly talk to
the Workhorse](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4592)
instead of Rails.
One side effect of this change: the correlation ID of the original
request is not preserved for the internal API requests made by Gitaly
(or `gitaly-lfs-smudge`), such as the one made in step 8. The
correlation IDs for those API requests will be random values until [this
Workhorse issue](https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/309) is
resolved.
......@@ -42,7 +42,6 @@ Documentation for GitLab instance administrators is under [LFS administration do
credentials store is recommended
- Git LFS always assumes HTTPS so if you have GitLab server on HTTP you will have
to add the URL to Git configuration manually (see [troubleshooting](#troubleshooting))
- Files added using Git LFS are [not included in the archives created using "download zip" functionality](https://gitlab.com/gitlab-org/gitlab/-/issues/15079)
NOTE: **Note:**
With 8.12 GitLab added LFS support to SSH. The Git LFS communication
......@@ -112,6 +111,51 @@ To remove objects from LFS:
See the documentation on [File Locking](../../../user/project/file_lock.md).
## LFS objects in project archives
> - Support for including Git LFS blobs inside [project source downloads](../../../user/project/repository/index.md) was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15079) in GitLab 13.5.
> - It's [deployed behind a feature flag](../../../user/feature_flags.md), disabled by default.
> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-lfs-objects-in-project-archives). **(CORE ONLY)**
CAUTION: **Warning:**
This feature might not be available to you. Check the **version history** note above for details.
Prior to GitLab 13.5, [project source
downloads](../../../user/project/repository/index.md) would include Git
LFS pointers instead of the actual objects. For example, LFS pointers
look like the following:
```markdown
version https://git-lfs.github.com/spec/v1
oid sha256:3ea5dd307f195f449f0e08234183b82e92c3d5f4cff11c2a6bb014f9e0de12aa
size 177735
```
Starting with GitLab 13.5, these pointers are converted to the uploaded
LFS object if the `include_lfs_blobs_in_archive` feature flag is
enabled.
Technical details about how this works can be found in the [development documentation for LFS](../../../development/lfs.md#including-lfs-blobs-in-project-archives).
### Enable or disable LFS objects in project archives
_LFS objects in project archives_ is under development and not ready for production use. It is
deployed behind a feature flag that is **disabled by default**.
[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md)
can enable it.
To enable it:
```ruby
Feature.enable(:include_lfs_blobs_in_archive)
```
To disable it:
```ruby
Feature.disable(:include_lfs_blobs_in_archive)
```
## Troubleshooting
### error: Repository or object not found
......
......@@ -247,6 +247,7 @@ used for cloning your project. The button is only shown on macOS.
## Download Source Code
> Support for directory download was [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/24704) in GitLab 11.11.
> Support for [including Git LFS blobs](../../../topics/git/lfs#lfs-objects-in-project-archives) was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15079) in GitLab 13.5.
The source code stored in a repository can be downloaded from the UI.
By clicking the download icon, a dropdown will open with links to download the following:
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment