Merge branch 'migrate-annex-lfs' into 'master'

Git-Annex to Git-LFS migration guide

Closes #1693

See merge request !1230

doc/university/README.md

@@ -130,6 +130,7 @@ The curriculum is composed of GitLab videos, screencasts, presentations, project
 1. [GitLab Flow Overview](https://about.gitlab.com/2014/09/29/gitlab-flow/)
 1. [Always Start with an Issue](https://about.gitlab.com/2016/03/03/start-with-an-issue/)
 1. [GitLab Flow Documentation](https://docs.gitlab.com/ee/workflow/gitlab_flow.html)
+1. [Git-Annex to Git-LFS Migration Guide](https://docs.gitlab.com/ee/workflow/lfs/migrate_from_git-annex_to_git-lfs.html)
 
 #### 2.5. GitLab Comparisons
 

doc/workflow/README.md

@@ -18,6 +18,7 @@
 - [Labels](../user/project/labels.md)
 - (EE) [Issue weight](issue_weight.md)
 - (EE) [Manage large binaries with git annex](git_annex.md)
+- [Git-Annex to Git-LFS Migration Guide](lfs/migrate_from_git_annex_to_git_lfs.html)
 - [Notification emails](notifications.md)
 - [Project Features](project_features.md)
 - [Project forking workflow](forking_workflow.md)

doc/workflow/git_annex.md

 # Git annex
 
+> **Warning:** GitLab is deprecating git-annex support
+from GitLab 8.17 (2017/02/22) on. It will be [completely
+removed][deprecate-annex-issue] in GitLab 9.0 (2017/03/22).
+Read through the [migration guide from git-annex to git-lfs][guide].
+
 The biggest limitation of Git, compared to some older centralized version
 control systems, has been the maximum size of the repositories.
 
@@ -56,7 +61,7 @@ The Omnibus package will internally set the correct options in all locations.
     gitlab_shell['git_annex_enabled'] = true
     ```
 
-1.  Save the file and [reconfigure GitLab] for the changes to take effect.
+1.  Save the file and [reconfigure GitLab][] for the changes to take effect.
 
 ### Configuration for installations from source
 
@@ -78,29 +83,90 @@ one is located in `config.yml` of gitlab-shell.
     git_annex_enabled: true
     ```
 
-1.  Save the files and [restart GitLab] for the changes to take effect.
+1.  Save the files and [restart GitLab][] for the changes to take effect.
 
 ## Using GitLab git-annex
 
-_**Important note:** Your Git remotes must be using the SSH protocol, not HTTP(S)._
+>
+**Note:**
+Your Git remotes must be using the SSH protocol, not HTTP(S).
 
 Here is an example workflow of uploading a very large file and then checking it
 into your Git repository:
 
 ```bash
-git clone git@gitlab.example.com:group/project.git
-git annex init 'My Laptop'       # initialize the annex project
+git clone git@example.com:group/project.git
+
+git annex init 'My Laptop'       # initialize the annex project and give an optional description
 cp ~/tmp/debian.iso ./           # copy a large file into the current directory
-git annex add .                  # add the large file to git annex
+git annex add debian.iso         # add the large file to git annex
 git commit -am "Add Debian iso"  # commit the file metadata
-git annex sync --content         # sync the git repo and large file to the GitLab server
+git annex sync --content         # sync the Git repo and large file to the GitLab server
+```
+
+The output should look like this:
+
+```
+commit
+On branch master
+Your branch is ahead of 'origin/master' by 1 commit.
+  (use "git push" to publish your local commits)
+nothing to commit, working tree clean
+ok
+pull origin
+remote: Counting objects: 5, done.
+remote: Compressing objects: 100% (4/4), done.
+remote: Total 5 (delta 2), reused 0 (delta 0)
+Unpacking objects: 100% (5/5), done.
+From example.com:group/project
+   497842b..5162f80  git-annex  -> origin/git-annex
+ok
+(merging origin/git-annex into git-annex...)
+(recording state in git...)
+copy debian.iso (checking origin...) (to origin...)
+SHA256E-s26214400--8092b3d482fb1b7a5cf28c43bc1425c8f2d380e86869c0686c49aa7b0f086ab2.iso
+     26,214,400 100%  638.88kB/s    0:00:40 (xfr#1, to-chk=0/1)
+ok
+pull origin
+ok
+(recording state in git...)
+push origin
+Counting objects: 15, done.
+Delta compression using up to 4 threads.
+Compressing objects: 100% (13/13), done.
+Writing objects: 100% (15/15), 1.64 KiB | 0 bytes/s, done.
+Total 15 (delta 1), reused 0 (delta 0)
+To example.com:group/project.git
+ * [new branch]      git-annex -> synced/git-annex
+ * [new branch]      master -> synced/master
+ok
 ```
 
+Your files can be found in the `master` branch, but you'll notice that there
+are more branches created by the `annex sync` command.
+
+Git Annex will also create a new directory at `.git/annex/` and will record the
+tracked files in the `.git/config` file. The files you assign to be tracked
+with `git-annex` will not affect the existing `.git/config` records. The files
+are turned into symbolic links that point to data in `.git/annex/objects/`.
+
+The `debian.iso` file in the example will contain the symbolic link:
+
+```
+.git/annex/objects/ZW/1k/SHA256E-s82701--6384039733b5035b559efd5a2e25a493ab6e09aabfd5162cc03f6f0ec238429d.png/SHA256E-s82701--6384039733b5035b559efd5a2e25a493ab6e09aabfd5162cc03f6f0ec238429d.iso
+```
+
+Use `git annex info` to retrieve the information about the local copy of your
+repository.
+
+---
+
 Downloading a single large file is also very simple:
 
 ```bash
 git clone git@gitlab.example.com:group/project.git
-git annex sync             # sync git branches but not the large file
+
+git annex sync             # sync Git branches but not the large file
 git annex get debian.iso   # download the large file
 ```
 
@@ -108,17 +174,13 @@ To download all files:
 
 ```bash
 git clone git@gitlab.example.com:group/project.git
-git annex sync --content  # sync git branches and download all the large files
-```
 
-You don't have to setup `git-annex` on a separate server or add annex remotes
-to the repository.
+git annex sync --content  # sync Git branches and download all the large files
+```
 
 By using `git-annex` without GitLab, anyone that can access the server can also
-access the files of all projects.
-
-GitLab annex ensures that you can only access files of projects you have access
-to (developer, master, or owner role).
+access the files of all projects, but GitLab Annex ensures that you can only
+access files of projects you have access to (developer, master, or owner role).
 
 ## How it works
 
@@ -127,13 +189,16 @@ integration point for `git-annex`.
 There is a setting in gitlab-shell so you can disable GitLab Annex support
 if you want to.
 
-_**Important note:** Your Git remotes must be using the SSH protocol, not HTTP(S)._
-
 ## Troubleshooting tips
 
 Differences in version of `git-annex` on the GitLab server and on local machines
 can cause `git-annex` to raise unpredicted warnings and errors.
 
+Consult the [Annex upgrade page][annex-upgrade] for more information about
+the differences between versions. You can find out which version is installed
+on your server by navigating to <https://pkgs.org/download/git-annex> and
+searching for your distribution.
+
 Although there is no general guide for `git-annex` errors, there are a few tips
 on how to go around the warnings.
 
@@ -165,8 +230,11 @@ ok
 push origin
 ```
 
+[annex-upgrade]: https://git-annex.branchable.com/upgrades/
+[deprecate-annex-issue]: https://gitlab.com/gitlab-org/gitlab-ee/issues/1648
+[git-annex]: https://git-annex.branchable.com/ "git-annex website"
 [gitlab shell]: https://gitlab.com/gitlab-org/gitlab-shell "GitLab Shell repository"
+[guide]: lfs/migrate_from_git_annex_to_git_lfs.html
 [issue]: https://git-annex.branchable.com/forum/Error_from_git-annex-shell_on_creation_of_gcrypt_special_remote/ "git-annex issue"
-[git-annex]: https://git-annex.branchable.com/ "git-annex website"
 [reconfigure GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure
 [restart GitLab]: ../administration/restart_gitlab.md#installations-from-source

doc/workflow/lfs/migrate_from_git_annex_to_git_lfs.md

+# Migration guide from Git Annex to Git LFS
+
+>**Note:**
+Git Annex support [will be dropped][issue-remove-annex] with GitLab Enterprise
+Edition 9.0 which is to be released on March 22nd 2017.
+
+Both [Git Annex][] and [Git LFS][] are tools to manage large files in Git.
+
+## History
+
+Git Annex [was introduced in GitLab Enterprise Edition 7.8][post-3], at a time
+where Git LFS didn't yet exist. A few months later, GitLab brought support for
+Git LFS in [GitLab 8.2][post-2] and is available for both Community and
+Enterprise editions.
+
+## Differences between Git Annex and Git LFS
+
+Some items below are general differences between the two protocols and some are
+ones that GitLab developed.
+
+- Git Annex works only through SSH, whereas Git LFS works both with SSH and HTTPS
+  (SSH support was added in GitLab 8.12).
+- Annex files are stored in a sub-directory of the normal repositories, whereas
+  LFS files are stored outside of the repositories in a place you can define.
+- Git Annex requires a more complex setup, but has much more options than Git
+  LFS. You can compare the commands each one offers by running `man git-annex`
+  and `man git-lfs`.
+- Annex files cannot be browsed directly in GitLab's interface, whereas LFS
+  files can.
+
+## Migration steps
+
+>**Note:**
+Since Git Annex files are stored in a sub-directory of the normal repositories
+(`.git/annex/objects`) and LFS files are stored outside of the repositories,
+they are not compatible as they are using a different scheme. Therefore, the
+migration has to be done manually per repository.
+
+There are basically two steps you need to take in order to migrate from Git
+Annex to Git LFS.
+
+### TL; DR
+
+If you know what you are doing and want to skip the reading, this is what you
+need to do (we assume you have [git-annex enabled][annex-gitlab-use] in your
+repository). Fire up a terminal, navigate to your Git repository and:
+
+1. Disable `git-annex`:
+
+    ```bash
+    git annex sync --content
+    git annex direct
+    git annex uninit
+    git annex indirect
+    ```
+
+1. Enable `git-lfs`:
+
+    ```
+    git lfs track <files>
+    git add .
+    git commit -m "commit message"
+    git push
+    ```
+
+### Disabling Git Annex in your repo
+
+Before changing anything, make sure you have a backup of your repository first.
+There are a couple of ways to do that, but you can simply clone it to another
+local path and maybe push it to GitLab if you want a remote backup as well.
+Here you'll find a guide on
+[how to back up a **git-annex** repository to an external hard drive][bkp-ext-drive].
+
+Since Annex files are stored as objects with symlinks and cannot be directly
+modified, we need to first remove those symlinks.
+
+>**Note:**
+Make sure the you read about the [`direct` mode][annex-direct] as it contains
+useful information that may fit in your use case. Note that `annex direct` is
+deprecated in Git Annex version 6, so you may need to upgrade your repository
+if the server also has Git Annex 6 installed. Read more in the
+[Git Annex troubleshooting tips][annex-tips] section.
+
+1. Use `annex direct`:
+
+    ```bash
+    git annex direct
+    ```
+
+    The output should be similar to this:
+
+    ```bash
+    commit
+    On branch master
+    Your branch is up-to-date with 'origin/master'.
+    nothing to commit, working tree clean
+    ok
+    direct debian.iso ok
+    direct  ok
+    ```
+
+1. Disable Git Annex with [`annex uninit`][uninit]:
+
+    ```bash
+    git annex uninit
+    ```
+
+    The output should be similar to this:
+
+    ```bash
+    unannex debian.iso ok
+    Deleted branch git-annex (was 2534d2c).
+    ```
+
+    This will `unannex` every file in the repository, leaving the original files.
+
+1. Switch back to `indirect` mode:
+
+    ```bash
+    git annex indirect
+    ```
+
+    The output should be similar to this:
+
+    ```bash
+    (merging origin/git-annex into git-annex...)
+    (recording state in git...)
+    commit  (recording state in git...)
+
+    ok
+    (recording state in git...)
+    [master fac3194] commit before switching to indirect mode
+     1 file changed, 1 deletion(-)
+     delete mode 120000 alpine-virt-3.4.4-x86_64.iso
+    ok
+    indirect  ok
+    ok
+    ```
+---
+
+At this point, you have two options. Either add, commit and push the files
+directly back to GitLab or switch to Git LFS. We will tackle the LFS switch in
+the next section.
+
+### Enabling Git LFS in your repo
+
+Git LFS is enabled by default on all GitLab products (GitLab CE, GitLab EE,
+GitLab.com), therefore, you don't need to do anything server-side.
+
+1. First, make sure you have `git-lfs` installed locally:
+
+    ```bash
+    git lfs help
+    ```
+
+    If the terminal doesn't prompt you with a full response on `git-lfs` commands,
+    [install the Git LFS client][install-lfs] first.
+
+1. Inside the repo, run the following command to initiate LFS:
+
+    ```bash
+    git lfs install
+    ```
+
+1. Enable `git-lfs` for the group of files you want to track. You
+   can track specific files, all files containing the same extension, or an
+   entire directory:
+
+    ```bash
+    git lfs track images/01.png   # per file
+    git lfs track **/*.png        # per extension
+    git lfs track images/         # per directory
+    ```
+
+    Once you do that, run `git status` and you'll see `.gitattributes` added
+    to your repo. It collects all file patterns that you chose to track via
+    `git-lfs`.
+
+1. Add the files, commit and push them to GitLab:
+
+    ```bash
+    git add .
+    git commit -m "commit message"
+    git push
+    ```
+
+    If your remote is set up with HTTP, you will be asked to enter your login
+    credentials. If you have [2FA enabled][2fa], make sure to use a
+    [personal access token][token] instead of your password.
+
+## Removing the Git Annex branches
+
+After the migration finishes successfully, you can remove all `git-annex`
+related branches from your repository.
+
+On GitLab, navigate to your project's **Repository ➔ Branches** and delete all
+branches created by Git Annex: `git-annex`, and all under `synced/`.
+
+![repository branches](images/git-annex-branches.png)
+
+If there are still some Annex objects inside your repository (`.git/annex/`)
+or references inside `.git/config`, run `annex uninit` again:
+
+```bash
+git annex uninit
+```
+
+## Further Reading
+
+- (Blog Post) [Getting Started with Git FLS][post-1]
+- (Blog Post) [Announcing LFS Support in GitLab][post-2]
+- (Blog Post) [GitLab Annex Solves the Problem of Versioning Large Binaries with Git][post-3]
+- (GitLab Docs) [Git Annex][doc-1]
+- (GitLab Docs) [Git LFS][doc-2]
+
+[2fa]: ../../user/profile/account/two_factor_authentication.md
+[token]: ../../user/profile/account/two_factor_authentication.html#personal-access-tokens
+[annex-tips]: ../git_annex.html#troubleshooting-tips
+[annex-direct]: https://git-annex.branchable.com/direct_mode/
+[annex-gitlab-use]: ../git_annex.md#using-gitlab-git-annex
+[annex-ee]: https://docs.gitlab.com/ee/workflow/git_annex.html
+[bkp-ext-drive]: https://www.thomas-krenn.com/en/wiki/Git-annex_Repository_on_an_External_Hard_Drive
+[doc-1]: https://docs.gitlab.com/ee/workflow/git_annex.html
+[doc-2]: https://docs.gitlab.com/ee/workflow/lfs/manage_large_binaries_with_git_lfs.html
+[Git Annex]: http://git-annex.branchable.com/
+[Git LFS]: https://git-lfs.github.com/
+[install-lfs]: https://git-lfs.github.com/
+[issue-remove-annex]: https://gitlab.com/gitlab-org/gitlab-ee/issues/1648
+[lfs-track]: https://about.gitlab.com/2017/01/30/getting-started-with-git-lfs-tutorial/#tracking-files-with-lfs
+[post-1]: https://about.gitlab.com/2017/01/30/getting-started-with-git-lfs-tutorial/
+[post-2]: https://about.gitlab.com/2015/11/23/announcing-git-lfs-support-in-gitlab/
+[post-3]: https://about.gitlab.com/2015/02/17/gitlab-annex-solves-the-problem-of-versioning-large-binaries-with-git/
+[uninit]: https://git-annex.branchable.com/git-annex-uninit/