Skip to content

G
gitlab-ce
Commit fe8219d8 authored by Mike Lewis's avatar Mike Lewis
Browse files
Options

Merge branch 'docs-workflow-sunset-1' into 'master' 

Move 6 docs out of the workflow dir

See merge request gitlab-org/gitlab!18553
parents 9d81e97d 999b7cae
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 1180 additions and 1128 deletions
doc/administration/geo/replication/object_storage.md
View file @ fe8219d8
......@@ -30,7 +30,7 @@ To enable GitLab replication, you must:
checkbox.
For LFS, follow the documentation to
[set up LFS object storage](../../../workflow/lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage).
[set up LFS object storage](../../lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage).
For CI job artifacts, there is similar documentation to configure
[jobs artifact object storage](../../job_artifacts.md#using-object-storage)
......
doc/administration/git_annex.md 0 → 100644
View file @ fe8219d8
---
disqus_identifier: 'https://docs.gitlab.com/ee/workflow/git_annex.html'
---
# Git annex
> **Warning:** GitLab has [completely
removed][deprecate-annex-issue] in GitLab 9.0 (2017/03/22).
Read through the [migration guide from git-annex to Git LFS][guide].
The biggest limitation of Git, compared to some older centralized version
control systems, has been the maximum size of the repositories.
The general recommendation is to not have Git repositories larger than 1GB to
preserve performance. Although GitLab has no limit (some repositories in GitLab
are over 50GB!), we subscribe to the advice to keep repositories as small as
you can.
Not being able to version control large binaries is a big problem for many
larger organizations.
Videos, photos, audio, compiled binaries and many other types of files are too
large. As a workaround, people keep artwork-in-progress in a Dropbox folder and
only check in the final result. This results in using outdated files, not
having a complete history and increases the risk of losing work.
This problem is solved in GitLab Enterprise Edition by integrating the
[git-annex] application.
`git-annex` allows managing large binaries with Git without checking the
contents into Git.
You check-in only a symlink that contains the SHA-1 of the large binary. If you
need the large binary, you can sync it from the GitLab server over `rsync`, a
very fast file copying tool.
## GitLab git-annex Configuration
`git-annex` is disabled by default in GitLab. Below you will find the
configuration options required to enable it.
### Requirements
`git-annex` needs to be installed both on the server and the client side.
For Debian-like systems (e.g., Debian, Ubuntu) this can be achieved by running:
```
sudo apt-get update && sudo apt-get install git-annex
```
For RedHat-like systems (e.g., CentOS, RHEL) this can be achieved by running:
```
sudo yum install epel-release && sudo yum install git-annex
```
### Configuration for Omnibus packages
For Omnibus GitLab packages, only one configuration setting is needed.
The Omnibus package will internally set the correct options in all locations.
1. In `/etc/gitlab/gitlab.rb` add the following line:
```ruby
gitlab_shell['git_annex_enabled'] = true
```
1. Save the file and [reconfigure GitLab][] for the changes to take effect.
### Configuration for installations from source
There are 2 settings to enable git-annex on your GitLab server.
One is located in `config/gitlab.yml` of the GitLab repository and the other
one is located in `config.yml` of GitLab Shell.
1. In `config/gitlab.yml` add or edit the following lines:
```yaml
gitlab_shell:
git_annex_enabled: true
```
1. In `config.yml` of GitLab Shell add or edit the following lines:
```yaml
git_annex_enabled: true
```
1. Save the files and [restart GitLab][] for the changes to take effect.
## Using GitLab git-annex
> **Note:**
> Your Git remotes must be using the SSH protocol, not HTTP(S).
Here is an example workflow of uploading a very large file and then checking it
into your Git repository:
```bash
git clone git@example.com:group/project.git
git annex init 'My Laptop' # initialize the annex project and give an optional description
cp ~/tmp/debian.iso ./ # copy a large file into the current directory
git annex add debian.iso # add the large file to git annex
git commit -am "Add Debian iso" # commit the file metadata
git annex sync --content # sync the Git repo and large file to the GitLab server
```
The output should look like this:
```
commit
On branch master
Your branch is ahead of 'origin/master' by 1 commit.
(use "git push" to publish your local commits)
nothing to commit, working tree clean
ok
pull origin
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 5 (delta 2), reused 0 (delta 0)
Unpacking objects: 100% (5/5), done.
From example.com:group/project
497842b..5162f80 git-annex -> origin/git-annex
ok
(merging origin/git-annex into git-annex...)
(recording state in git...)
copy debian.iso (checking origin...) (to origin...)
SHA256E-s26214400--8092b3d482fb1b7a5cf28c43bc1425c8f2d380e86869c0686c49aa7b0f086ab2.iso
26,214,400 100% 638.88kB/s 0:00:40 (xfr#1, to-chk=0/1)
ok
pull origin
ok
(recording state in git...)
push origin
Counting objects: 15, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (13/13), done.
Writing objects: 100% (15/15), 1.64 KiB | 0 bytes/s, done.
Total 15 (delta 1), reused 0 (delta 0)
To example.com:group/project.git
* [new branch] git-annex -> synced/git-annex
* [new branch] master -> synced/master
ok
```
Your files can be found in the `master` branch, but you'll notice that there
are more branches created by the `annex sync` command.
Git Annex will also create a new directory at `.git/annex/` and will record the
tracked files in the `.git/config` file. The files you assign to be tracked
with `git-annex` will not affect the existing `.git/config` records. The files
are turned into symbolic links that point to data in `.git/annex/objects/`.
The `debian.iso` file in the example will contain the symbolic link:
```
.git/annex/objects/ZW/1k/SHA256E-s82701--6384039733b5035b559efd5a2e25a493ab6e09aabfd5162cc03f6f0ec238429d.png/SHA256E-s82701--6384039733b5035b559efd5a2e25a493ab6e09aabfd5162cc03f6f0ec238429d.iso
```
Use `git annex info` to retrieve the information about the local copy of your
repository.
---
Downloading a single large file is also very simple:
```bash
git clone git@gitlab.example.com:group/project.git
git annex sync # sync Git branches but not the large file
git annex get debian.iso # download the large file
```
To download all files:
```bash
git clone git@gitlab.example.com:group/project.git
git annex sync --content # sync Git branches and download all the large files
```
By using `git-annex` without GitLab, anyone that can access the server can also
access the files of all projects, but GitLab Annex ensures that you can only
access files of projects you have access to (developer, maintainer, or owner role).
## How it works
Internally GitLab uses [GitLab Shell] to handle SSH access and this was a great
integration point for `git-annex`.
There is a setting in GitLab Shell so you can disable GitLab Annex support
if you want to.
## Troubleshooting tips
Differences in version of `git-annex` on the GitLab server and on local machines
can cause `git-annex` to raise unpredicted warnings and errors.
Consult the [Annex upgrade page][annex-upgrade] for more information about
the differences between versions. You can find out which version is installed
on your server by navigating to <https://pkgs.org/download/git-annex> and
searching for your distribution.
Although there is no general guide for `git-annex` errors, there are a few tips
on how to go around the warnings.
### `git-annex-shell: Not a git-annex or gcrypt repository`
This warning can appear on the initial `git annex sync --content` and is caused
by differences in `git-annex-shell`. You can read more about it
[in this git-annex issue][issue].
One important thing to note is that despite the warning, the `sync` succeeds
and the files are pushed to the GitLab repository.
If you get hit by this, you can run the following command inside the repository
that the warning was raised:
```
git config remote.origin.annex-ignore false
```
Consecutive runs of `git annex sync --content` **should not** produce this
warning and the output should look like this:
```
commit ok
pull origin
ok
pull origin
ok
push origin
```
[annex-upgrade]: https://git-annex.branchable.com/upgrades/
[deprecate-annex-issue]: https://gitlab.com/gitlab-org/gitlab/issues/1648
[git-annex]: https://git-annex.branchable.com/ "git-annex website"
[gitlab shell]: https://gitlab.com/gitlab-org/gitlab-shell "GitLab Shell repository"
[guide]: lfs/migrate_from_git_annex_to_git_lfs.html
[issue]: https://git-annex.branchable.com/forum/Error_from_git-annex-shell_on_creation_of_gcrypt_special_remote/ "git-annex issue"
[reconfigure GitLab]: restart_gitlab.md#omnibus-gitlab-reconfigure
[restart GitLab]: restart_gitlab.md#installations-from-source
doc/administration/gitaly/index.md
View file @ fe8219d8
......@@ -551,7 +551,7 @@ a few things that you need to do:
to eliminate the need for a shared authorized_keys file.
1. Configure [object storage for job artifacts](../job_artifacts.md#using-object-storage)
including [incremental logging](../job_logs.md#new-incremental-logging-architecture).
1. Configure [object storage for LFS objects](../../workflow/lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage).
1. Configure [object storage for LFS objects](../lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage).
1. Configure [object storage for uploads](../uploads.md#using-object-storage-core-only).
NOTE: **Note:**
......
doc/administration/index.md
View file @ fe8219d8
......@@ -161,9 +161,10 @@ Learn how to install, configure, update, and maintain your GitLab instance.
## Git configuration options
- [Custom Git hooks](custom_hooks.md): Custom Git hooks (on the filesystem) for when webhooks aren't enough.
- [Git LFS configuration](../workflow/lfs/lfs_administration.md): Learn how to configure LFS for GitLab.
- [Git LFS configuration](lfs/lfs_administration.md): Learn how to configure LFS for GitLab.
- [Housekeeping](housekeeping.md): Keep your Git repositories tidy and fast.
- [Configuring Git Protocol v2](git_protocol.md): Git protocol version 2 support.
- [Manage large files with `git-annex` (Deprecated)](git_annex.md)
## Monitoring GitLab
......
doc/administration/lfs/lfs_administration.md 0 → 100644
View file @ fe8219d8
This diff is collapsed.
doc/administration/lfs/manage_large_binaries_with_git_lfs.md 0 → 100644
View file @ fe8219d8
---
disqus_identifier: 'https://docs.gitlab.com/ee/workflow/lfs/manage_large_binaries_with_git_lfs.html'
---
# Git LFS
Managing large files such as audio, video and graphics files has always been one
of the shortcomings of Git. The general recommendation is to not have Git repositories
larger than 1GB to preserve performance.
![Git LFS tracking status](img/lfs-icon.png)
An LFS icon is shown on files tracked by Git LFS to denote if a file is stored
as a blob or as an LFS pointer.
## How it works
Git LFS client talks with the GitLab server over HTTPS. It uses HTTP Basic Authentication
to authorize client requests. Once the request is authorized, Git LFS client receives
instructions from where to fetch or where to push the large file.
## GitLab server configuration
Documentation for GitLab instance administrators is under [LFS administration doc](lfs_administration.md).
## Requirements
- Git LFS is supported in GitLab starting with version 8.2
- Git LFS must be enabled under project settings
- [Git LFS client](https://git-lfs.github.com) version 1.0.1 and up
## Known limitations
- Git LFS v1 original API is not supported since it was deprecated early in LFS
development
- When SSH is set as a remote, Git LFS objects still go through HTTPS
- Any Git LFS request will ask for HTTPS credentials to be provided so a good Git
credentials store is recommended
- Git LFS always assumes HTTPS so if you have GitLab server on HTTP you will have
to add the URL to Git config manually (see [troubleshooting](#troubleshooting))
NOTE: **Note:**
With 8.12 GitLab added LFS support to SSH. The Git LFS communication
still goes over HTTP, but now the SSH client passes the correct credentials
to the Git LFS client, so no action is required by the user.
## Using Git LFS
Lets take a look at the workflow when you need to check large files into your Git
repository with Git LFS. For example, if you want to upload a very large file and
check it into your Git repository:
```bash
git clone git@gitlab.example.com:group/project.git
git lfs install # initialize the Git LFS project
git lfs track "*.iso" # select the file extensions that you want to treat as large files
```
Once a certain file extension is marked for tracking as a LFS object you can use
Git as usual without having to redo the command to track a file with the same extension:
```bash
cp ~/tmp/debian.iso ./ # copy a large file into the current directory
git add . # add the large file to the project
git commit -am "Added Debian iso" # commit the file meta data
git push origin master # sync the git repo and large file to the GitLab server
```
**Make sure** that `.gitattributes` is tracked by Git. Otherwise Git
LFS will not be working properly for people cloning the project:
```bash
git add .gitattributes
```
Cloning the repository works the same as before. Git automatically detects the
LFS-tracked files and clones them via HTTP. If you performed the `git clone`
command with a SSH URL, you have to enter your GitLab credentials for HTTP
authentication.
```bash
git clone git@gitlab.example.com:group/project.git
```
If you already cloned the repository and you want to get the latest LFS object
that are on the remote repository, eg. for a branch from origin:
```bash
git lfs fetch origin master
```
### Migrate an existing repo to Git LFS
Read the documentation on how to [migrate an existing Git repo with Git LFS](../../topics/git/migrate_to_git_lfs/index.md).
## File Locking
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/35856) in GitLab 10.5.
The first thing to do before using File Locking is to tell Git LFS which
kind of files are lockable. The following command will store PNG files
in LFS and flag them as lockable:
```bash
git lfs track "*.png" --lockable
```
After executing the above command a file named `.gitattributes` will be
created or updated with the following content:
```bash
*.png filter=lfs diff=lfs merge=lfs -text lockable
```
You can also register a file type as lockable without using LFS
(In order to be able to lock/unlock a file you need a remote server that implements the LFS File Locking API),
in order to do that you can edit the `.gitattributes` file manually:
```bash
*.pdf lockable
```
After a file type has been registered as lockable, Git LFS will make
them readonly on the file system automatically. This means you will
need to lock the file before editing it.
### Managing Locked Files
Once you're ready to edit your file you need to lock it first:
```bash
git lfs lock images/banner.png
Locked images/banner.png
```
This will register the file as locked in your name on the server:
```bash
git lfs locks
images/banner.png joe ID:123
```
Once you have pushed your changes, you can unlock the file so others can
also edit it:
```bash
git lfs unlock images/banner.png
```
You can also unlock by id:
```bash
git lfs unlock --id=123
```
If for some reason you need to unlock a file that was not locked by you,
you can use the `--force` flag as long as you have a `maintainer` access on
the project:
```bash
git lfs unlock --id=123 --force
```
## Troubleshooting
### error: Repository or object not found
There are a couple of reasons why this error can occur:
- You don't have permissions to access certain LFS object
Check if you have permissions to push to the project or fetch from the project.
- Project is not allowed to access the LFS object
LFS object you are trying to push to the project or fetch from the project is not
available to the project anymore. Probably the object was removed from the server.
- Local Git repository is using deprecated LFS API
### Invalid status for `<url>` : 501
Git LFS will log the failures into a log file.
To view this log file, while in project directory:
```bash
git lfs logs last
```
If the status `error 501` is shown, it is because:
- Git LFS is not enabled in project settings. Check your project settings and
enable Git LFS.
- Git LFS support is not enabled on the GitLab server. Check with your GitLab
administrator why Git LFS is not enabled on the server. See
[LFS administration documentation](lfs_administration.md) for instructions
on how to enable LFS support.
- Git LFS client version is not supported by GitLab server. Check your Git LFS
version with `git lfs version`. Check the Git config of the project for traces
of deprecated API with `git lfs -l`. If `batch = false` is set in the config,
remove the line and try to update your Git LFS client. Only version 1.0.1 and
newer are supported.
### getsockopt: connection refused
If you push a LFS object to a project and you receive an error similar to:
`Post <URL>/info/lfs/objects/batch: dial tcp IP: getsockopt: connection refused`,
the LFS client is trying to reach GitLab through HTTPS. However, your GitLab
instance is being served on HTTP.
This behaviour is caused by Git LFS using HTTPS connections by default when a
`lfsurl` is not set in the Git config.
To prevent this from happening, set the lfs url in project Git config:
```bash
git config --add lfs.url "http://gitlab.example.com/group/project.git/info/lfs"
```
### Credentials are always required when pushing an object
NOTE: **Note:**
With 8.12 GitLab added LFS support to SSH. The Git LFS communication
still goes over HTTP, but now the SSH client passes the correct credentials
to the Git LFS client, so no action is required by the user.
Given that Git LFS uses HTTP Basic Authentication to authenticate the user pushing
the LFS object on every push for every object, user HTTPS credentials are required.
By default, Git has support for remembering the credentials for each repository
you use. This is described in [Git credentials man pages](https://git-scm.com/docs/gitcredentials).
For example, you can tell Git to remember the password for a period of time in
which you expect to push the objects:
```bash
git config --global credential.helper 'cache --timeout=3600'
```
This will remember the credentials for an hour after which Git operations will
require re-authentication.
If you are using OS X you can use `osxkeychain` to store and encrypt your credentials.
For Windows, you can use `wincred` or Microsoft's [Git Credential Manager for Windows](https://github.com/Microsoft/Git-Credential-Manager-for-Windows/releases).
More details about various methods of storing the user credentials can be found
on [Git Credential Storage documentation](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage).
### LFS objects are missing on push
GitLab checks files to detect LFS pointers on push. If LFS pointers are detected, GitLab tries to verify that those files already exist in LFS on GitLab.
Verify that LFS in installed locally and consider a manual push with `git lfs push --all`.
If you are storing LFS files outside of GitLab you can disable LFS on the project by setting `lfs_enabled: false` with the [projects API](../../api/projects.md#edit-project).
### Hosting LFS objects externally
It is possible to host LFS objects externally by setting a custom LFS url with `git config -f .lfsconfig lfs.url https://example.com/<project>.git/info/lfs`.
You might choose to do this if you are using an appliance like a Sonatype Nexus to store LFS data. If you choose to use an external LFS store,
GitLab will not be able to verify LFS objects which means that pushes will fail if you have GitLab LFS support enabled.
To stop push failure, LFS support can be disabled in the [Project settings](../../user/project/settings/index.md). This means you will lose GitLab LFS value-adds (Verifying LFS objects, UI integration for LFS).
doc/administration/lfs/migrate_from_git_annex_to_git_lfs.md 0 → 100644
View file @ fe8219d8
# Migration guide from Git Annex to Git LFS
>**Note:**
Git Annex support [has been removed][issue-remove-annex] in GitLab Enterprise
Edition 9.0 (2017/03/22).
Both [Git Annex][] and [Git LFS][] are tools to manage large files in Git.
## History
Git Annex [was introduced in GitLab Enterprise Edition 7.8][post-3], at a time
where Git LFS didn't yet exist. A few months later, GitLab brought support for
Git LFS in [GitLab 8.2][post-2] and is available for both Community and
Enterprise editions.
## Differences between Git Annex and Git LFS
Some items below are general differences between the two protocols and some are
ones that GitLab developed.
- Git Annex works only through SSH, whereas Git LFS works both with SSH and HTTPS
(SSH support was added in GitLab 8.12).
- Annex files are stored in a sub-directory of the normal repositories, whereas
LFS files are stored outside of the repositories in a place you can define.
- Git Annex requires a more complex setup, but has much more options than Git
LFS. You can compare the commands each one offers by running `man git-annex`
and `man git-lfs`.
- Annex files cannot be browsed directly in GitLab's interface, whereas LFS
files can.
## Migration steps
>**Note:**
Since Git Annex files are stored in a sub-directory of the normal repositories
(`.git/annex/objects`) and LFS files are stored outside of the repositories,
they are not compatible as they are using a different scheme. Therefore, the
migration has to be done manually per repository.
There are basically two steps you need to take in order to migrate from Git
Annex to Git LFS.
### TL; DR
If you know what you are doing and want to skip the reading, this is what you
need to do (we assume you have [git-annex enabled](../git_annex.md#using-gitlab-git-annex) in your
repository and that you have made backups in case something goes wrong).
Fire up a terminal, navigate to your Git repository and:
1. Disable `git-annex`:
```bash
git annex sync --content
git annex direct
git annex uninit
git annex indirect
```
1. Enable `git-lfs`:
```
git lfs install
git lfs track <files>
git add .
git commit -m "commit message"
git push
```
### Disabling Git Annex in your repo
Before changing anything, make sure you have a backup of your repository first.
There are a couple of ways to do that, but you can simply clone it to another
local path and maybe push it to GitLab if you want a remote backup as well.
Here you'll find a guide on
[how to back up a **git-annex** repository to an external hard drive][bkp-ext-drive].
Since Annex files are stored as objects with symlinks and cannot be directly
modified, we need to first remove those symlinks.
NOTE: **Note:**
Make sure the you read about the [`direct` mode][annex-direct] as it contains
useful information that may fit in your use case. Note that `annex direct` is
deprecated in Git Annex version 6, so you may need to upgrade your repository
if the server also has Git Annex 6 installed. Read more in the
[Git Annex troubleshooting tips](../git_annex.md#troubleshooting-tips) section.
1. Backup your repository
```bash
cd repository
git annex sync --content
cd ..
git clone repository repository-backup
cd repository-backup
git annex get
cd ..
```
1. Use `annex direct`:
```bash
cd repository
git annex direct
```
The output should be similar to this:
```bash
commit
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working tree clean
ok
direct debian.iso ok
direct ok
```
1. Disable Git Annex with [`annex uninit`][uninit]:
```bash
git annex uninit
```
The output should be similar to this:
```bash
unannex debian.iso ok
Deleted branch git-annex (was 2534d2c).
```
This will `unannex` every file in the repository, leaving the original files.
1. Switch back to `indirect` mode:
```bash
git annex indirect
```
The output should be similar to this:
```bash
(merging origin/git-annex into git-annex...)
(recording state in git...)
commit (recording state in git...)
ok
(recording state in git...)
[master fac3194] commit before switching to indirect mode
1 file changed, 1 deletion(-)
delete mode 120000 alpine-virt-3.4.4-x86_64.iso
ok
indirect ok
ok
```
---
At this point, you have two options. Either add, commit and push the files
directly back to GitLab or switch to Git LFS. We will tackle the LFS switch in
the next section.
### Enabling Git LFS in your repo
Git LFS is enabled by default on all GitLab products (GitLab CE, GitLab EE,
GitLab.com), therefore, you don't need to do anything server-side.
1. First, make sure you have `git-lfs` installed locally:
```bash
git lfs help
```
If the terminal doesn't prompt you with a full response on `git-lfs` commands,
[install the Git LFS client][install-lfs] first.
1. Inside the repo, run the following command to initiate LFS:
```bash
git lfs install
```
1. Enable `git-lfs` for the group of files you want to track. You
can track specific files, all files containing the same extension, or an
entire directory:
```bash
git lfs track images/01.png # per file
git lfs track **/*.png # per extension
git lfs track images/ # per directory
```
Once you do that, run `git status` and you'll see `.gitattributes` added
to your repo. It collects all file patterns that you chose to track via
`git-lfs`.
1. Add the files, commit and push them to GitLab:
```bash
git add .
git commit -m "commit message"
git push
```
If your remote is set up with HTTP, you will be asked to enter your login
credentials. If you have [2FA enabled](../../user/profile/account/two_factor_authentication.md), make sure to use a
[personal access token](../../user/profile/account/two_factor_authentication.md#personal-access-tokens)
instead of your password.
## Removing the Git Annex branches
After the migration finishes successfully, you can remove all `git-annex`
related branches from your repository.
On GitLab, navigate to your project's **Repository ➔ Branches** and delete all
branches created by Git Annex: `git-annex`, and all under `synced/`.
![repository branches](img/git-annex-branches.png)
You can also do this on the command line with:
```bash
git branch -d synced/master
git branch -d synced/git-annex
git push origin :synced/master
git push origin :synced/git-annex
git push origin :git-annex
git remote prune origin
```
If there are still some Annex objects inside your repository (`.git/annex/`)
or references inside `.git/config`, run `annex uninit` again:
```bash
git annex uninit
```
## Further Reading
- (Blog Post) [Getting Started with Git FLS][post-1]
- (Blog Post) [Announcing LFS Support in GitLab][post-2]
- (Blog Post) [GitLab Annex Solves the Problem of Versioning Large Binaries with Git][post-3]
- (GitLab Docs) [Git Annex](../git_annex.md)
- (GitLab Docs) [Git LFS](manage_large_binaries_with_git_lfs.md)
[annex-direct]: https://git-annex.branchable.com/direct_mode/
[bkp-ext-drive]: https://www.thomas-krenn.com/en/wiki/Git-annex_Repository_on_an_External_Hard_Drive
[Git Annex]: http://git-annex.branchable.com/
[Git LFS]: https://git-lfs.github.com/
[install-lfs]: https://git-lfs.github.com/
[issue-remove-annex]: https://gitlab.com/gitlab-org/gitlab/issues/1648
[lfs-track]: https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/#tracking-files-with-lfs
[post-1]: https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/
[post-2]: https://about.gitlab.com/blog/2015/11/23/announcing-git-lfs-support-in-gitlab/
[post-3]: https://about.gitlab.com/blog/2015/02/17/gitlab-annex-solves-the-problem-of-versioning-large-binaries-with-git/
[uninit]: https://git-annex.branchable.com/git-annex-uninit/
doc/administration/repository_storage_types.md
View file @ fe8219d8
......@@ -183,7 +183,8 @@ CI Artifacts are S3 compatible since **9.4** (GitLab Premium), and available in
##### LFS Objects
LFS Objects implements a similar storage pattern using 2 chars, 2 level folders, following Git own implementation:
[LFS Objects in GitLab](lfs/manage_large_binaries_with_git_lfs.md) implement a similar
storage pattern using 2 chars, 2 level folders, following Git's own implementation:
```ruby
"shared/lfs-objects/#{oid[0..1}/#{oid[2..3]}/#{oid[4..-1]}"
......@@ -192,7 +193,7 @@ LFS Objects implements a similar storage pattern using 2 chars, 2 level folders,
"shared/lfs-objects/89/09/029eb962194cfb326259411b22ae3f4a814b5be4f80651735aeef9f3229c"
```
They are also S3 compatible since **10.0** (GitLab Premium), and available in GitLab Core since **10.7**.
LFS objects are also [S3 compatible](lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage).
[ce-2821]: https://gitlab.com/gitlab-com/infrastructure/issues/2821
[ce-28283]: https://gitlab.com/gitlab-org/gitlab-foss/issues/28283
......
doc/development/contributing/merge_request_workflow.md
View file @ fe8219d8
......@@ -36,7 +36,7 @@ include a regression test are merged quickly, while new features without proper
tests might be slower to receive feedback. The workflow to make a merge
request is as follows:
1. [Fork](../../workflow/forking_workflow.md#creating-a-fork) the project into
1. [Fork](../../user/project/repository/forking_workflow.md) the project into
your personal namespace (or group) on GitLab.com.
1. Create a feature branch in your fork (don't work off `master`).
1. Write [tests](../rake_tasks.md#run-tests) and code.
......
doc/development/git_object_deduplication.md
View file @ fe8219d8
# How Git object deduplication works in GitLab
When a GitLab user [forks a project](../workflow/forking_workflow.md),
When a GitLab user [forks a project](../user/project/repository/forking_workflow.md),
GitLab creates a new Project with an associated Git repository that is a
copy of the original project at the time of the fork. If a large project
gets forked often, this can lead to a quick increase in Git repository
......
doc/gitlab-basics/fork-project.md
View file @ fe8219d8
......@@ -8,4 +8,4 @@ A fork is a copy of an original repository that you put in another namespace
where you can experiment and apply changes that you can later decide whether or
not to share, without affecting the original project.
It takes just a few steps to [fork a project in GitLab](../workflow/forking_workflow.md#creating-a-fork).
It takes just a few steps to [fork a project in GitLab](../user/project/repository/forking_workflow.md#creating-a-fork).
doc/install/aws/index.md
View file @ fe8219d8
......@@ -539,7 +539,7 @@ which would otherwise take much space.
In particular, you can store in S3:
- [The Git LFS objects](../../workflow/lfs/lfs_administration.md#s3-for-omnibus-installations) ((Omnibus GitLab installations))
- [The Git LFS objects](../../administration/lfs/lfs_administration.md#s3-for-omnibus-installations) ((Omnibus GitLab installations))
- [The Container Registry images](../../administration/packages/container_registry.md#container-registry-storage-driver) (Omnibus GitLab installations)
- [The GitLab CI/CD job artifacts](../../administration/job_artifacts.md#using-object-storage) (Omnibus GitLab installations)
......
doc/intro/README.md
View file @ fe8219d8
......@@ -24,7 +24,7 @@ Create issues, labels, milestones, cast your vote, and review issues.
Create merge requests and review code.
- [Fork a project and contribute to it](../workflow/forking_workflow.md)
- [Fork a project and contribute to it](../user/project/repository/forking_workflow.md)
- [Create a new merge request](../gitlab-basics/add-merge-request.md)
- [Automatically close issues from merge requests](../user/project/issues/managing_issues.md#closing-issues-automatically)
- [Automatically merge when pipeline succeeds](../user/project/merge_requests/merge_when_pipeline_succeeds.md)
......
doc/topics/git/migrate_to_git_lfs/index.md
View file @ fe8219d8
......@@ -163,9 +163,9 @@ but commented out to help encourage others to add to it in the future. -->
## References
- [Getting Started with Git LFS](https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/)
- [Migrate from Git Annex to Git LFS](../../../workflow/lfs/migrate_from_git_annex_to_git_lfs.md)
- [GitLab's Git LFS user documentation](../../../workflow/lfs/manage_large_binaries_with_git_lfs.md)
- [GitLab's Git LFS administrator documentation](../../../workflow/lfs/lfs_administration.md)
- [Migrate from Git Annex to Git LFS](../../../administration/lfs/migrate_from_git_annex_to_git_lfs.md)
- [GitLab's Git LFS user documentation](../../../administration/lfs/manage_large_binaries_with_git_lfs.md)
- [GitLab's Git LFS administrator documentation](../../../administration/lfs/lfs_administration.md)
- Alternative method to [migrate an existing repo to Git LFS](https://github.com/git-lfs/git-lfs/wiki/Tutorial#migrating-existing-repository-data-to-lfs)
<!--
......
doc/user/admin_area/settings/account_and_limit_settings.md
View file @ fe8219d8
......@@ -32,7 +32,7 @@ For instance, consider the following workflow:
1. Your team develops apps which require large files to be stored in
the application repository.
1. Although you have enabled [Git LFS](../../../workflow/lfs/manage_large_binaries_with_git_lfs.md#git-lfs)
1. Although you have enabled [Git LFS](../../../administration/lfs/manage_large_binaries_with_git_lfs.md#git-lfs)
to your project, your storage has grown significantly.
1. Before you exceed available storage, you set up a limit of 10 GB
per repository.
......
doc/user/project/repository/file_finder.md 0 → 100644
View file @ fe8219d8
---
disqus_identifier: 'https://docs.gitlab.com/ee/workflow/file_finder.html'
---
# File finder
> [Introduced][gh-9889] in GitLab 8.4.
The file finder feature allows you to search for a file in a repository using the
GitLab UI.
You can find the **Find File** button when in the **Files** section of a
project.
![Find file button](img/file_finder_find_button.png)
For those who prefer to keep their fingers on the keyboard, there is a
[shortcut button](../../../workflow/shortcuts.md) as well, which you can invoke from _anywhere_
in a project.
Press `t` to launch the File search function when in **Issues**,
**Merge requests**, **Milestones**, even the project's settings.
Start typing what you are searching for and watch the magic happen. With the
up/down arrows, you go up and down the results, with `Esc` you close the search
and go back to **Files**.
## How it works
The File finder feature is powered by the [Fuzzy filter](https://github.com/jeancroy/fuzz-aldrin-plus) library.
It implements a fuzzy search with highlight, and tries to provide intuitive
results by recognizing patterns that people use while searching.
For example, consider the [GitLab CE repository][ce] and that we want to open
the `app/controllers/admin/deploy_keys_controller.rb` file.
Using fuzzy search, we start by typing letters that get us closer to the file.
**Protip:** To narrow down your search, include `/` in your search terms.
![Find file button](img/file_finder_find_file.png)
[gh-9889]: https://github.com/gitlabhq/gitlabhq/pull/9889 "File finder pull request"
[ce]: https://gitlab.com/gitlab-org/gitlab-foss/tree/master "GitLab CE repository"
doc/user/project/repository/forking_workflow.md 0 → 100644
View file @ fe8219d8
---
disqus_identifier: 'https://docs.gitlab.com/ee/workflow/forking_workflow.html'
---
# Project forking workflow
Forking a project to your own namespace is useful if you have no write
access to the project you want to contribute to. If you do have write
access or can request it, we recommend working together in the same
repository since it is simpler. See our [GitLab Flow](../../../workflow/gitlab_flow.md)
document more information about using branches to work together.
## Creating a fork
Forking a project is in most cases a two-step process.
1. Click on the fork button located located in between the star and clone buttons on the project's home page.
![Fork button](img/forking_workflow_fork_button.png)
1. Once you do that, you'll be presented with a screen where you can choose
the namespace to fork to. Only namespaces (groups and your own
namespace) where you have write access to, will be shown. Click on the
namespace to create your fork there.
![Choose namespace](img/forking_workflow_choose_namespace.png)
**Note:**
If the namespace you chose to fork the project to has another project with
the same path name, you will be presented with a warning that the forking
could not be completed. Try to resolve the error before repeating the forking
process.
![Path taken error](img/forking_workflow_path_taken_error.png)
After the forking is done, you can start working on the newly created
repository. There, you will have full [Owner](../../permissions.md)
access, so you can set it up as you please.
## Merging upstream
Once you are ready to send your code back to the main project, you need
to create a merge request. Choose your forked project's main branch as
the source and the original project's main branch as the destination and
create the [merge request](../merge_requests/index.md).
![Selecting branches](img/forking_workflow_branch_select.png)
You can then assign the merge request to someone to have them review
your changes. Upon pressing the 'Submit Merge Request' button, your
changes will be added to the repository and branch you're merging into.
![New merge request](img/forking_workflow_merge_request.png)
[gitlab flow]: https://about.gitlab.com/blog/2014/09/29/gitlab-flow/ "GitLab Flow blog post"
doc/user/project/repository/index.md
View file @ fe8219d8
......@@ -11,7 +11,8 @@ A repository is part of a [project](../index.md), which has a lot of other featu
## Create a repository
To create a new repository, all you need to do is
[create a new project](../../../gitlab-basics/create-project.md).
[create a new project](../../../gitlab-basics/create-project.md) or
[fork an existing project](forking_workflow.md).
Once you create a new project, you can add new files via UI
(read the section below) or via command line.
......@@ -55,7 +56,7 @@ To get started with the command line, please read through the
### Find files
Use GitLab's [file finder](../../../workflow/file_finder.md) to search for files in a repository.
Use GitLab's [file finder](file_finder.md) to search for files in a repository.
### Supported markup languages and extensions
......
doc/workflow/README.md
View file @ fe8219d8
......@@ -18,13 +18,13 @@ comments: false
- [Due date for issues](../user/project/issues/due_dates.md)
- [Issue Board](../user/project/issue_board.md)
- [Keyboard shortcuts](shortcuts.md)
- [File finder](file_finder.md)
- [File finder](../user/project/repository/file_finder.md)
- [File lock](../user/project/file_lock.md) **(PREMIUM)**
- [Labels](../user/project/labels.md)
- [Issue weight](issue_weight.md) **(STARTER)**
- [Notification emails](notifications.md)
- [Projects](../user/project/index.md)
- [Project forking workflow](forking_workflow.md)
- [Project forking workflow](../user/project/repository/forking_workflow.md)
- [Project users](../user/project/members/index.md)
- [Protected branches](../user/project/protected_branches.md)
- [Protected tags](../user/project/protected_tags.md)
......@@ -47,7 +47,6 @@ comments: false
- [Merge request approvals](../user/project/merge_requests/merge_request_approvals.md) **(STARTER)**
- [Repository mirroring](repository_mirroring.md) **(STARTER)**
- [Service Desk](../user/project/service_desk.md) **(PREMIUM)**
- [Manage large binaries with Git LFS](lfs/manage_large_binaries_with_git_lfs.md)
- [Importing from SVN, GitHub, Bitbucket, etc](importing/README.md)
- [Todos](todos.md)
- [Snippets](../user/snippets.md)
......
doc/workflow/file_finder.md
View file @ fe8219d8
# File finder
---
redirect_to: '../user/project/repository/file_finder.md'
---
> [Introduced][gh-9889] in GitLab 8.4.
The file finder feature allows you to quickly shortcut your way when you are
searching for a file in a repository using the GitLab UI.
You can find the **Find File** button when in the **Files** section of a
project.
![Find file button](img/file_finder_find_button.png)
For those who prefer to keep their fingers on the keyboard, there is a
[shortcut button](shortcuts.md) as well, which you can invoke from _anywhere_
in a project.
Press `t` to launch the File search function when in **Issues**,
**Merge requests**, **Milestones**, even the project's settings.
Start typing what you are searching for and watch the magic happen. With the
up/down arrows, you go up and down the results, with `Esc` you close the search
and go back to **Files**.
## How it works
The File finder feature is powered by the [Fuzzy filter](https://github.com/jeancroy/fuzz-aldrin-plus) library.
It implements a fuzzy search with highlight, and tries to provide intuitive
results by recognizing patterns that people use while searching.
For example, consider the [GitLab CE repository][ce] and that we want to open
the `app/controllers/admin/deploy_keys_controller.rb` file.
Using fuzzy search, we start by typing letters that get us closer to the file.
**Protip:** To narrow down your search, include `/` in your search terms.
![Find file button](img/file_finder_find_file.png)
[gh-9889]: https://github.com/gitlabhq/gitlabhq/pull/9889 "File finder pull request"
[ce]: https://gitlab.com/gitlab-org/gitlab-foss/tree/master "GitLab CE repository"
This document was moved to [another location](../user/project/repository/file_finder.md).
doc/workflow/forking_workflow.md
View file @ fe8219d8
# Project forking workflow
---
redirect_to: '../user/project/repository/forking_workflow.md'
---
Forking a project to your own namespace is useful if you have no write
access to the project you want to contribute to. If you do have write
access or can request it, we recommend working together in the same
repository since it is simpler. See our [GitLab Flow](gitlab_flow.md)
document more information about using branches to work together.
## Creating a fork
Forking a project is in most cases a two-step process.
1. Click on the fork button located located in between the star and clone buttons on the project's home page.
![Fork button](img/forking_workflow_fork_button.png)
1. Once you do that, you'll be presented with a screen where you can choose
the namespace to fork to. Only namespaces (groups and your own
namespace) where you have write access to, will be shown. Click on the
namespace to create your fork there.
![Choose namespace](img/forking_workflow_choose_namespace.png)
**Note:**
If the namespace you chose to fork the project to has another project with
the same path name, you will be presented with a warning that the forking
could not be completed. Try to resolve the error before repeating the forking
process.
![Path taken error](img/forking_workflow_path_taken_error.png)
After the forking is done, you can start working on the newly created
repository. There, you will have full [Owner](../user/permissions.md)
access, so you can set it up as you please.
## Merging upstream
Once you are ready to send your code back to the main project, you need
to create a merge request. Choose your forked project's main branch as
the source and the original project's main branch as the destination and
create the [merge request](merge_requests.md).
![Selecting branches](forking/branch_select.png)
You can then assign the merge request to someone to have them review
your changes. Upon pressing the 'Submit Merge Request' button, your
changes will be added to the repository and branch you're merging into.
![New merge request](forking/merge_request.png)
[gitlab flow]: https://about.gitlab.com/blog/2014/09/29/gitlab-flow/ "GitLab Flow blog post"
This document was moved to [another location](../user/project/repository/forking_workflow.md).
doc/workflow/git_annex.md
View file @ fe8219d8
# Git annex
> **Warning:** GitLab has [completely
removed][deprecate-annex-issue] in GitLab 9.0 (2017/03/22).
Read through the [migration guide from git-annex to Git LFS][guide].
The biggest limitation of Git, compared to some older centralized version
control systems, has been the maximum size of the repositories.
The general recommendation is to not have Git repositories larger than 1GB to
preserve performance. Although GitLab has no limit (some repositories in GitLab
are over 50GB!), we subscribe to the advice to keep repositories as small as
you can.
Not being able to version control large binaries is a big problem for many
larger organizations.
Videos, photos, audio, compiled binaries and many other types of files are too
large. As a workaround, people keep artwork-in-progress in a Dropbox folder and
only check in the final result. This results in using outdated files, not
having a complete history and increases the risk of losing work.
This problem is solved in GitLab Enterprise Edition by integrating the
[git-annex] application.
`git-annex` allows managing large binaries with Git without checking the
contents into Git.
You check-in only a symlink that contains the SHA-1 of the large binary. If you
need the large binary, you can sync it from the GitLab server over `rsync`, a
very fast file copying tool.
## GitLab git-annex Configuration
`git-annex` is disabled by default in GitLab. Below you will find the
configuration options required to enable it.
### Requirements
`git-annex` needs to be installed both on the server and the client side.
For Debian-like systems (e.g., Debian, Ubuntu) this can be achieved by running:
```
sudo apt-get update && sudo apt-get install git-annex
```
For RedHat-like systems (e.g., CentOS, RHEL) this can be achieved by running:
```
sudo yum install epel-release && sudo yum install git-annex
```
### Configuration for Omnibus packages
For Omnibus GitLab packages, only one configuration setting is needed.
The Omnibus package will internally set the correct options in all locations.
1. In `/etc/gitlab/gitlab.rb` add the following line:
```ruby
gitlab_shell['git_annex_enabled'] = true
```
1. Save the file and [reconfigure GitLab][] for the changes to take effect.
### Configuration for installations from source
There are 2 settings to enable git-annex on your GitLab server.
One is located in `config/gitlab.yml` of the GitLab repository and the other
one is located in `config.yml` of GitLab Shell.
1. In `config/gitlab.yml` add or edit the following lines:
```yaml
gitlab_shell:
git_annex_enabled: true
```
1. In `config.yml` of GitLab Shell add or edit the following lines:
```yaml
git_annex_enabled: true
```
1. Save the files and [restart GitLab][] for the changes to take effect.
## Using GitLab git-annex
> **Note:**
> Your Git remotes must be using the SSH protocol, not HTTP(S).
Here is an example workflow of uploading a very large file and then checking it
into your Git repository:
```bash
git clone git@example.com:group/project.git
git annex init 'My Laptop' # initialize the annex project and give an optional description
cp ~/tmp/debian.iso ./ # copy a large file into the current directory
git annex add debian.iso # add the large file to git annex
git commit -am "Add Debian iso" # commit the file metadata
git annex sync --content # sync the Git repo and large file to the GitLab server
```
The output should look like this:
```
commit
On branch master
Your branch is ahead of 'origin/master' by 1 commit.
(use "git push" to publish your local commits)
nothing to commit, working tree clean
ok
pull origin
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 5 (delta 2), reused 0 (delta 0)
Unpacking objects: 100% (5/5), done.
From example.com:group/project
497842b..5162f80 git-annex -> origin/git-annex
ok
(merging origin/git-annex into git-annex...)
(recording state in git...)
copy debian.iso (checking origin...) (to origin...)
SHA256E-s26214400--8092b3d482fb1b7a5cf28c43bc1425c8f2d380e86869c0686c49aa7b0f086ab2.iso
26,214,400 100% 638.88kB/s 0:00:40 (xfr#1, to-chk=0/1)
ok
pull origin
ok
(recording state in git...)
push origin
Counting objects: 15, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (13/13), done.
Writing objects: 100% (15/15), 1.64 KiB | 0 bytes/s, done.
Total 15 (delta 1), reused 0 (delta 0)
To example.com:group/project.git
* [new branch] git-annex -> synced/git-annex
* [new branch] master -> synced/master
ok
```
Your files can be found in the `master` branch, but you'll notice that there
are more branches created by the `annex sync` command.
Git Annex will also create a new directory at `.git/annex/` and will record the
tracked files in the `.git/config` file. The files you assign to be tracked
with `git-annex` will not affect the existing `.git/config` records. The files
are turned into symbolic links that point to data in `.git/annex/objects/`.
The `debian.iso` file in the example will contain the symbolic link:
```
.git/annex/objects/ZW/1k/SHA256E-s82701--6384039733b5035b559efd5a2e25a493ab6e09aabfd5162cc03f6f0ec238429d.png/SHA256E-s82701--6384039733b5035b559efd5a2e25a493ab6e09aabfd5162cc03f6f0ec238429d.iso
```
Use `git annex info` to retrieve the information about the local copy of your
repository.
---
redirect_to: '../administration/git_annex.md'
---
Downloading a single large file is also very simple:
```bash
git clone git@gitlab.example.com:group/project.git
git annex sync # sync Git branches but not the large file
git annex get debian.iso # download the large file
```
To download all files:
```bash
git clone git@gitlab.example.com:group/project.git
git annex sync --content # sync Git branches and download all the large files
```
By using `git-annex` without GitLab, anyone that can access the server can also
access the files of all projects, but GitLab Annex ensures that you can only
access files of projects you have access to (developer, maintainer, or owner role).
## How it works
Internally GitLab uses [GitLab Shell] to handle SSH access and this was a great
integration point for `git-annex`.
There is a setting in GitLab Shell so you can disable GitLab Annex support
if you want to.
## Troubleshooting tips
Differences in version of `git-annex` on the GitLab server and on local machines
can cause `git-annex` to raise unpredicted warnings and errors.
Consult the [Annex upgrade page][annex-upgrade] for more information about
the differences between versions. You can find out which version is installed
on your server by navigating to <https://pkgs.org/download/git-annex> and
searching for your distribution.
Although there is no general guide for `git-annex` errors, there are a few tips
on how to go around the warnings.
### `git-annex-shell: Not a git-annex or gcrypt repository`
This warning can appear on the initial `git annex sync --content` and is caused
by differences in `git-annex-shell`. You can read more about it
[in this git-annex issue][issue].
One important thing to note is that despite the warning, the `sync` succeeds
and the files are pushed to the GitLab repository.
If you get hit by this, you can run the following command inside the repository
that the warning was raised:
```
git config remote.origin.annex-ignore false
```
Consecutive runs of `git annex sync --content` **should not** produce this
warning and the output should look like this:
```
commit ok
pull origin
ok
pull origin
ok
push origin
```
[annex-upgrade]: https://git-annex.branchable.com/upgrades/
[deprecate-annex-issue]: https://gitlab.com/gitlab-org/gitlab/issues/1648
[git-annex]: https://git-annex.branchable.com/ "git-annex website"
[gitlab shell]: https://gitlab.com/gitlab-org/gitlab-shell "GitLab Shell repository"
[guide]: lfs/migrate_from_git_annex_to_git_lfs.html
[issue]: https://git-annex.branchable.com/forum/Error_from_git-annex-shell_on_creation_of_gcrypt_special_remote/ "git-annex issue"
[reconfigure GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure
[restart GitLab]: ../administration/restart_gitlab.md#installations-from-source
This document was moved to [another location](../administration/git_annex.md).
doc/workflow/git_lfs.md
View file @ fe8219d8
---
redirect_to: 'lfs/manage_large_binaries_with_git_lfs.md'
redirect_to: '../administration/lfs/manage_large_binaries_with_git_lfs.md'
---
This document was moved to [another location](lfs/manage_large_binaries_with_git_lfs.md).
This document was moved to [another location](../administration/lfs/manage_large_binaries_with_git_lfs.md).
doc/workflow/lfs/lfs_administration.md
View file @ fe8219d8
This diff is collapsed.
doc/workflow/lfs/manage_large_binaries_with_git_lfs.md
View file @ fe8219d8
# Git LFS
---
redirect_to: '../../administration/lfs/manage_large_binaries_with_git_lfs.md'
---
Managing large files such as audio, video and graphics files has always been one
of the shortcomings of Git. The general recommendation is to not have Git repositories
larger than 1GB to preserve performance.
![Git LFS tracking status](img/lfs-icon.png)
An LFS icon is shown on files tracked by Git LFS to denote if a file is stored
as a blob or as an LFS pointer.
## How it works
Git LFS client talks with the GitLab server over HTTPS. It uses HTTP Basic Authentication
to authorize client requests. Once the request is authorized, Git LFS client receives
instructions from where to fetch or where to push the large file.
## GitLab server configuration
Documentation for GitLab instance administrators is under [LFS administration doc](lfs_administration.md).
## Requirements
- Git LFS is supported in GitLab starting with version 8.2
- Git LFS must be enabled under project settings
- [Git LFS client](https://git-lfs.github.com) version 1.0.1 and up
## Known limitations
- Git LFS v1 original API is not supported since it was deprecated early in LFS
development
- When SSH is set as a remote, Git LFS objects still go through HTTPS
- Any Git LFS request will ask for HTTPS credentials to be provided so a good Git
credentials store is recommended
- Git LFS always assumes HTTPS so if you have GitLab server on HTTP you will have
to add the URL to Git config manually (see [troubleshooting](#troubleshooting))
NOTE: **Note:**
With 8.12 GitLab added LFS support to SSH. The Git LFS communication
still goes over HTTP, but now the SSH client passes the correct credentials
to the Git LFS client, so no action is required by the user.
## Using Git LFS
Lets take a look at the workflow when you need to check large files into your Git
repository with Git LFS. For example, if you want to upload a very large file and
check it into your Git repository:
```bash
git clone git@gitlab.example.com:group/project.git
git lfs install # initialize the Git LFS project
git lfs track "*.iso" # select the file extensions that you want to treat as large files
```
Once a certain file extension is marked for tracking as a LFS object you can use
Git as usual without having to redo the command to track a file with the same extension:
```bash
cp ~/tmp/debian.iso ./ # copy a large file into the current directory
git add . # add the large file to the project
git commit -am "Added Debian iso" # commit the file meta data
git push origin master # sync the git repo and large file to the GitLab server
```
**Make sure** that `.gitattributes` is tracked by Git. Otherwise Git
LFS will not be working properly for people cloning the project:
```bash
git add .gitattributes
```
Cloning the repository works the same as before. Git automatically detects the
LFS-tracked files and clones them via HTTP. If you performed the `git clone`
command with a SSH URL, you have to enter your GitLab credentials for HTTP
authentication.
```bash
git clone git@gitlab.example.com:group/project.git
```
If you already cloned the repository and you want to get the latest LFS object
that are on the remote repository, eg. for a branch from origin:
```bash
git lfs fetch origin master
```
### Migrate an existing repo to Git LFS
Read the documentation on how to [migrate an existing Git repo with Git LFS](../../topics/git/migrate_to_git_lfs/index.md).
## File Locking
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/35856) in GitLab 10.5.
The first thing to do before using File Locking is to tell Git LFS which
kind of files are lockable. The following command will store PNG files
in LFS and flag them as lockable:
```bash
git lfs track "*.png" --lockable
```
After executing the above command a file named `.gitattributes` will be
created or updated with the following content:
```bash
*.png filter=lfs diff=lfs merge=lfs -text lockable
```
You can also register a file type as lockable without using LFS
(In order to be able to lock/unlock a file you need a remote server that implements the LFS File Locking API),
in order to do that you can edit the `.gitattributes` file manually:
```bash
*.pdf lockable
```
After a file type has been registered as lockable, Git LFS will make
them readonly on the file system automatically. This means you will
need to lock the file before editing it.
### Managing Locked Files
Once you're ready to edit your file you need to lock it first:
```bash
git lfs lock images/banner.png
Locked images/banner.png
```
This will register the file as locked in your name on the server:
```bash
git lfs locks
images/banner.png joe ID:123
```
Once you have pushed your changes, you can unlock the file so others can
also edit it:
```bash
git lfs unlock images/banner.png
```
You can also unlock by id:
```bash
git lfs unlock --id=123
```
If for some reason you need to unlock a file that was not locked by you,
you can use the `--force` flag as long as you have a `maintainer` access on
the project:
```bash
git lfs unlock --id=123 --force
```
## Troubleshooting
### error: Repository or object not found
There are a couple of reasons why this error can occur:
- You don't have permissions to access certain LFS object
Check if you have permissions to push to the project or fetch from the project.
- Project is not allowed to access the LFS object
LFS object you are trying to push to the project or fetch from the project is not
available to the project anymore. Probably the object was removed from the server.
- Local Git repository is using deprecated LFS API
### Invalid status for `<url>` : 501
Git LFS will log the failures into a log file.
To view this log file, while in project directory:
```bash
git lfs logs last
```
If the status `error 501` is shown, it is because:
- Git LFS is not enabled in project settings. Check your project settings and
enable Git LFS.
- Git LFS support is not enabled on the GitLab server. Check with your GitLab
administrator why Git LFS is not enabled on the server. See
[LFS administration documentation](lfs_administration.md) for instructions
on how to enable LFS support.
- Git LFS client version is not supported by GitLab server. Check your Git LFS
version with `git lfs version`. Check the Git config of the project for traces
of deprecated API with `git lfs -l`. If `batch = false` is set in the config,
remove the line and try to update your Git LFS client. Only version 1.0.1 and
newer are supported.
### getsockopt: connection refused
If you push a LFS object to a project and you receive an error similar to:
`Post <URL>/info/lfs/objects/batch: dial tcp IP: getsockopt: connection refused`,
the LFS client is trying to reach GitLab through HTTPS. However, your GitLab
instance is being served on HTTP.
This behaviour is caused by Git LFS using HTTPS connections by default when a
`lfsurl` is not set in the Git config.
To prevent this from happening, set the lfs url in project Git config:
```bash
git config --add lfs.url "http://gitlab.example.com/group/project.git/info/lfs"
```
### Credentials are always required when pushing an object
NOTE: **Note:**
With 8.12 GitLab added LFS support to SSH. The Git LFS communication
still goes over HTTP, but now the SSH client passes the correct credentials
to the Git LFS client, so no action is required by the user.
Given that Git LFS uses HTTP Basic Authentication to authenticate the user pushing
the LFS object on every push for every object, user HTTPS credentials are required.
By default, Git has support for remembering the credentials for each repository
you use. This is described in [Git credentials man pages](https://git-scm.com/docs/gitcredentials).
For example, you can tell Git to remember the password for a period of time in
which you expect to push the objects:
```bash
git config --global credential.helper 'cache --timeout=3600'
```
This will remember the credentials for an hour after which Git operations will
require re-authentication.
If you are using OS X you can use `osxkeychain` to store and encrypt your credentials.
For Windows, you can use `wincred` or Microsoft's [Git Credential Manager for Windows](https://github.com/Microsoft/Git-Credential-Manager-for-Windows/releases).
More details about various methods of storing the user credentials can be found
on [Git Credential Storage documentation](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage).
### LFS objects are missing on push
GitLab checks files to detect LFS pointers on push. If LFS pointers are detected, GitLab tries to verify that those files already exist in LFS on GitLab.
Verify that LFS in installed locally and consider a manual push with `git lfs push --all`.
If you are storing LFS files outside of GitLab you can disable LFS on the project by setting `lfs_enabled: false` with the [projects API](../../api/projects.md#edit-project).
### Hosting LFS objects externally
It is possible to host LFS objects externally by setting a custom LFS url with `git config -f .lfsconfig lfs.url https://example.com/<project>.git/info/lfs`.
You might choose to do this if you are using an appliance like a Sonatype Nexus to store LFS data. If you choose to use an external LFS store,
GitLab will not be able to verify LFS objects which means that pushes will fail if you have GitLab LFS support enabled.
To stop push failure, LFS support can be disabled in the [Project settings](../../user/project/settings/index.md). This means you will lose GitLab LFS value-adds (Verifying LFS objects, UI integration for LFS).
This document was moved to [another location](../../administration/lfs/manage_large_binaries_with_git_lfs.md).
doc/workflow/lfs/migrate_from_git_annex_to_git_lfs.md
View file @ fe8219d8
# Migration guide from Git Annex to Git LFS
>**Note:**
Git Annex support [has been removed][issue-remove-annex] in GitLab Enterprise
Edition 9.0 (2017/03/22).
Both [Git Annex][] and [Git LFS][] are tools to manage large files in Git.
## History
Git Annex [was introduced in GitLab Enterprise Edition 7.8][post-3], at a time
where Git LFS didn't yet exist. A few months later, GitLab brought support for
Git LFS in [GitLab 8.2][post-2] and is available for both Community and
Enterprise editions.
## Differences between Git Annex and Git LFS
Some items below are general differences between the two protocols and some are
ones that GitLab developed.
- Git Annex works only through SSH, whereas Git LFS works both with SSH and HTTPS
(SSH support was added in GitLab 8.12).
- Annex files are stored in a sub-directory of the normal repositories, whereas
LFS files are stored outside of the repositories in a place you can define.
- Git Annex requires a more complex setup, but has much more options than Git
LFS. You can compare the commands each one offers by running `man git-annex`
and `man git-lfs`.
- Annex files cannot be browsed directly in GitLab's interface, whereas LFS
files can.
## Migration steps
>**Note:**
Since Git Annex files are stored in a sub-directory of the normal repositories
(`.git/annex/objects`) and LFS files are stored outside of the repositories,
they are not compatible as they are using a different scheme. Therefore, the
migration has to be done manually per repository.
There are basically two steps you need to take in order to migrate from Git
Annex to Git LFS.
### TL; DR
If you know what you are doing and want to skip the reading, this is what you
need to do (we assume you have [git-annex enabled](../git_annex.md#using-gitlab-git-annex) in your
repository and that you have made backups in case something goes wrong).
Fire up a terminal, navigate to your Git repository and:
1. Disable `git-annex`:
```bash
git annex sync --content
git annex direct
git annex uninit
git annex indirect
```
1. Enable `git-lfs`:
```
git lfs install
git lfs track <files>
git add .
git commit -m "commit message"
git push
```
### Disabling Git Annex in your repo
Before changing anything, make sure you have a backup of your repository first.
There are a couple of ways to do that, but you can simply clone it to another
local path and maybe push it to GitLab if you want a remote backup as well.
Here you'll find a guide on
[how to back up a **git-annex** repository to an external hard drive][bkp-ext-drive].
Since Annex files are stored as objects with symlinks and cannot be directly
modified, we need to first remove those symlinks.
NOTE: **Note:**
Make sure the you read about the [`direct` mode][annex-direct] as it contains
useful information that may fit in your use case. Note that `annex direct` is
deprecated in Git Annex version 6, so you may need to upgrade your repository
if the server also has Git Annex 6 installed. Read more in the
[Git Annex troubleshooting tips](../git_annex.md#troubleshooting-tips) section.
1. Backup your repository
```bash
cd repository
git annex sync --content
cd ..
git clone repository repository-backup
cd repository-backup
git annex get
cd ..
```
1. Use `annex direct`:
```bash
cd repository
git annex direct
```
The output should be similar to this:
```bash
commit
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working tree clean
ok
direct debian.iso ok
direct ok
```
1. Disable Git Annex with [`annex uninit`][uninit]:
```bash
git annex uninit
```
The output should be similar to this:
```bash
unannex debian.iso ok
Deleted branch git-annex (was 2534d2c).
```
This will `unannex` every file in the repository, leaving the original files.
1. Switch back to `indirect` mode:
```bash
git annex indirect
```
The output should be similar to this:
```bash
(merging origin/git-annex into git-annex...)
(recording state in git...)
commit (recording state in git...)
ok
(recording state in git...)
[master fac3194] commit before switching to indirect mode
1 file changed, 1 deletion(-)
delete mode 120000 alpine-virt-3.4.4-x86_64.iso
ok
indirect ok
ok
```
---
redirect_to: '../../administration/lfs/migrate_from_git_annex_to_git_lfs.md'
---
At this point, you have two options. Either add, commit and push the files
directly back to GitLab or switch to Git LFS. We will tackle the LFS switch in
the next section.
### Enabling Git LFS in your repo
Git LFS is enabled by default on all GitLab products (GitLab CE, GitLab EE,
GitLab.com), therefore, you don't need to do anything server-side.
1. First, make sure you have `git-lfs` installed locally:
```bash
git lfs help
```
If the terminal doesn't prompt you with a full response on `git-lfs` commands,
[install the Git LFS client][install-lfs] first.
1. Inside the repo, run the following command to initiate LFS:
```bash
git lfs install
```
1. Enable `git-lfs` for the group of files you want to track. You
can track specific files, all files containing the same extension, or an
entire directory:
```bash
git lfs track images/01.png # per file
git lfs track **/*.png # per extension
git lfs track images/ # per directory
```
Once you do that, run `git status` and you'll see `.gitattributes` added
to your repo. It collects all file patterns that you chose to track via
`git-lfs`.
1. Add the files, commit and push them to GitLab:
```bash
git add .
git commit -m "commit message"
git push
```
If your remote is set up with HTTP, you will be asked to enter your login
credentials. If you have [2FA enabled](../../user/profile/account/two_factor_authentication.md), make sure to use a
[personal access token](../../user/profile/account/two_factor_authentication.md#personal-access-tokens)
instead of your password.
## Removing the Git Annex branches
After the migration finishes successfully, you can remove all `git-annex`
related branches from your repository.
On GitLab, navigate to your project's **Repository ➔ Branches** and delete all
branches created by Git Annex: `git-annex`, and all under `synced/`.
![repository branches](images/git-annex-branches.png)
You can also do this on the command line with:
```bash
git branch -d synced/master
git branch -d synced/git-annex
git push origin :synced/master
git push origin :synced/git-annex
git push origin :git-annex
git remote prune origin
```
If there are still some Annex objects inside your repository (`.git/annex/`)
or references inside `.git/config`, run `annex uninit` again:
```bash
git annex uninit
```
## Further Reading
- (Blog Post) [Getting Started with Git FLS][post-1]
- (Blog Post) [Announcing LFS Support in GitLab][post-2]
- (Blog Post) [GitLab Annex Solves the Problem of Versioning Large Binaries with Git][post-3]
- (GitLab Docs) [Git Annex](../git_annex.md)
- (GitLab Docs) [Git LFS](manage_large_binaries_with_git_lfs.md)
[annex-direct]: https://git-annex.branchable.com/direct_mode/
[bkp-ext-drive]: https://www.thomas-krenn.com/en/wiki/Git-annex_Repository_on_an_External_Hard_Drive
[Git Annex]: http://git-annex.branchable.com/
[Git LFS]: https://git-lfs.github.com/
[install-lfs]: https://git-lfs.github.com/
[issue-remove-annex]: https://gitlab.com/gitlab-org/gitlab/issues/1648
[lfs-track]: https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/#tracking-files-with-lfs
[post-1]: https://about.gitlab.com/blog/2017/01/30/getting-started-with-git-lfs-tutorial/
[post-2]: https://about.gitlab.com/blog/2015/11/23/announcing-git-lfs-support-in-gitlab/
[post-3]: https://about.gitlab.com/blog/2015/02/17/gitlab-annex-solves-the-problem-of-versioning-large-binaries-with-git/
[uninit]: https://git-annex.branchable.com/git-annex-uninit/
This document was moved to [another location](../../administration/lfs/migrate_from_git_annex_to_git_lfs.md).
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7