git-backup issueshttps://lab.nexedi.com/kirr/git-backup/-/issues2020-03-09T18:23:44Zhttps://lab.nexedi.com/kirr/git-backup/-/issues/7Remove by hand files that were left by previous run2020-03-09T18:23:44ZivRemove by hand files that were left by previous runI just wanted to document this issue for new gitlab-backup users:
When `gitlab-backup restore -vupok -go SHA1` has been interrupted, the error message shows up when running again:
```
E: /srv/slapgrid/slappart6/srv/runner/instance/slappart0/var/backup/abignumberhere_gitlab_backup.tar already exists
```
Solution: Remove that old file.
Re-running may give another error:
```
cmd_restore: mkdir /srv/slapgrid/slappart6/srv/runner/instance/slappart0/var/repositories.abignumberhere: file exists
```
Solution: Remove the tar container (which was re-created) + this folder.I just wanted to document this issue for new gitlab-backup users:
When `gitlab-backup restore -vupok -go SHA1` has been interrupted, the error message shows up when running again:
```
E: /srv/slapgrid/slappart6/srv/runner/instance/slappart0/var/backup/abignumberhere_gitlab_backup.tar already exists
```
Solution: Remove that old file.
Re-running may give another error:
```
cmd_restore: mkdir /srv/slapgrid/slappart6/srv/runner/instance/slappart0/var/repositories.abignumberhere: file exists
```
Solution: Remove the tar container (which was re-created) + this folder.https://lab.nexedi.com/kirr/git-backup/-/issues/6`gitlab-backup` on gitlab instance with empty repo only keeps backup repo locked2020-03-09T18:23:44Ziv`gitlab-backup` on gitlab instance with empty repo only keeps backup repo lockedI am trying to use gitlab-backup to backup a set of repositories I created on a fresh gitlab instance. With my user ('root'), I created 3 git repo, but I didn't commited anything in any of them.
Running `gitlab-backup pull` gave:
* Dumping gitlab data (except repositories & db)
Dumping database ...
[SKIPPED]
[...]
Deleting old backups ... skipping
* Unpacking dump tarballs
* Dumping database
* git-backup pull everything
# creating root commit
# file gitlab/misc <- /srv/slapgrid/slappart18/srv/runner/instance/slappart0/backup-gitlab.git/gitlab-backup.kvR8EE/gitlab_backup/artifacts.tar.gz/.gitlab-backup-keep
# file gitlab/misc <- /srv/slapgrid/slappart18/srv/runner/instance/slappart0/backup-gitlab.git/gitlab-backup.kvR8EE/gitlab_backup/backup_information.yml
# file gitlab/misc <- /srv/slapgrid/slappart18/srv/runner/instance/slappart0/backup-gitlab.git/gitlab-backup.kvR8EE/gitlab_backup/builds.tar.gz/.gitlab-backup-keep
# file gitlab/misc <- /srv/slapgrid/slappart18/srv/runner/instance/slappart0/backup-gitlab.git/gitlab-backup.kvR8EE/gitlab_backup/db/database.pgdump/2841.dat/2841.dat.aa
[...]
# file gitlab/repo <- /srv/slapgrid/slappart18/srv/runner/instance/slappart0/var/repositories/root/project3.wiki.git/description
# file gitlab/repo <- /srv/slapgrid/slappart18/srv/runner/instance/slappart0/var/repositories/root/project3.wiki.git/hooks
# file gitlab/repo <- /srv/slapgrid/slappart18/srv/runner/instance/slappart0/var/repositories/root/project3.wiki.git/info/exclude
cmd_pull: raisef: refs/backup/20161103-0924/: strange for-each-ref entry ""
and `git log` in the backup repo:
commit 6333abdb5e39a004b1a1db21b35db1e5235ed46c
Author: iv <isabelle.vallet@nexedi.com>
Date: Thu Nov 3 09:24:41 2016 +0000
Initialize git-backup repository
After I commited something (and removed current locked backup repo), it went all fine. After that, I added an empty repo, and it went all fine as well.I am trying to use gitlab-backup to backup a set of repositories I created on a fresh gitlab instance. With my user ('root'), I created 3 git repo, but I didn't commited anything in any of them.
Running `gitlab-backup pull` gave:
* Dumping gitlab data (except repositories & db)
Dumping database ...
[SKIPPED]
[...]
Deleting old backups ... skipping
* Unpacking dump tarballs
* Dumping database
* git-backup pull everything
# creating root commit
# file gitlab/misc <- /srv/slapgrid/slappart18/srv/runner/instance/slappart0/backup-gitlab.git/gitlab-backup.kvR8EE/gitlab_backup/artifacts.tar.gz/.gitlab-backup-keep
# file gitlab/misc <- /srv/slapgrid/slappart18/srv/runner/instance/slappart0/backup-gitlab.git/gitlab-backup.kvR8EE/gitlab_backup/backup_information.yml
# file gitlab/misc <- /srv/slapgrid/slappart18/srv/runner/instance/slappart0/backup-gitlab.git/gitlab-backup.kvR8EE/gitlab_backup/builds.tar.gz/.gitlab-backup-keep
# file gitlab/misc <- /srv/slapgrid/slappart18/srv/runner/instance/slappart0/backup-gitlab.git/gitlab-backup.kvR8EE/gitlab_backup/db/database.pgdump/2841.dat/2841.dat.aa
[...]
# file gitlab/repo <- /srv/slapgrid/slappart18/srv/runner/instance/slappart0/var/repositories/root/project3.wiki.git/description
# file gitlab/repo <- /srv/slapgrid/slappart18/srv/runner/instance/slappart0/var/repositories/root/project3.wiki.git/hooks
# file gitlab/repo <- /srv/slapgrid/slappart18/srv/runner/instance/slappart0/var/repositories/root/project3.wiki.git/info/exclude
cmd_pull: raisef: refs/backup/20161103-0924/: strange for-each-ref entry ""
and `git log` in the backup repo:
commit 6333abdb5e39a004b1a1db21b35db1e5235ed46c
Author: iv <isabelle.vallet@nexedi.com>
Date: Thu Nov 3 09:24:41 2016 +0000
Initialize git-backup repository
After I commited something (and removed current locked backup repo), it went all fine. After that, I added an empty repo, and it went all fine as well.https://lab.nexedi.com/kirr/git-backup/-/issues/5Unable to import git2go when building git-backup2020-03-09T18:23:44ZivUnable to import git2go when building git-backupI wanted to use the latest version of git-backup, so I ran:
golang16/bin/go build -o git-backup *.go
but then, go complained about an import requiring a git repo to be cloned and built: github.com/libgit2/git2go. After some research, I did the following steps:
mkdir gopath
export GOPATH=$(pwd)/gopath
mkdir $GOPATH/{src,bin,pkg} # git-backup is cloned in src
export PATH="$PATH:$(pwd)/srv/runner/software/f7445a0b1019dad7cbae9b7eaeedb0b0/parts/pkgconfig/bin/" # pkgconfig is required by go get
[...]/go get "github.com/libgit2/git2go"
but this last one seems to be able to clone the repo in gopath/src/ but then fails:
$ ./srv/runner/software/f7445a0b1019dad7cbae9b7eaeedb0b0/parts/golang16/bin/go get "github.com/libgit2/git2go"
# pkg-config --cflags libgit2
Package libgit2 was not found in the pkg-config search path.
Perhaps you should add the directory containing `libgit2.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libgit2' found
pkg-config: exit status 1
I found on my system a file called libgit2.pc in `/srv/slapgrid/slappart18/srv/runner/software/f7445a0b1019dad7cbae9b7eaeedb0b0/parts/gitlab/vendor/bundle/ruby/2.1.0/gems/rugged-0.24.0/vendor/libgit2/build/` so I appended this path to PKG_CONFIG_PATH environment variable.
then, I get:
$ ./srv/runner/software/f7445a0b1019dad7cbae9b7eaeedb0b0/parts/golang16/bin/go get "github.com/libgit2/git2go"
# github.com/libgit2/git2go
gopath/src/github.com/libgit2/git2go/blame.go:4:18: fatal error: git2.h: No such file or directory
compilation terminated.
which is in `./srv/runner/software/f7445a0b1019dad7cbae9b7eaeedb0b0/parts/gitlab/vendor/bundle/ruby/2.1.0/gems/rugged-0.24.0/vendor/libgit2/include/git2.h`
I tried to add its parent directory to GOPATH but it didn't help much, so I don't know how I am supposed to do to be able to import git2go and build the new git-backup. Do you have any clue?I wanted to use the latest version of git-backup, so I ran:
golang16/bin/go build -o git-backup *.go
but then, go complained about an import requiring a git repo to be cloned and built: github.com/libgit2/git2go. After some research, I did the following steps:
mkdir gopath
export GOPATH=$(pwd)/gopath
mkdir $GOPATH/{src,bin,pkg} # git-backup is cloned in src
export PATH="$PATH:$(pwd)/srv/runner/software/f7445a0b1019dad7cbae9b7eaeedb0b0/parts/pkgconfig/bin/" # pkgconfig is required by go get
[...]/go get "github.com/libgit2/git2go"
but this last one seems to be able to clone the repo in gopath/src/ but then fails:
$ ./srv/runner/software/f7445a0b1019dad7cbae9b7eaeedb0b0/parts/golang16/bin/go get "github.com/libgit2/git2go"
# pkg-config --cflags libgit2
Package libgit2 was not found in the pkg-config search path.
Perhaps you should add the directory containing `libgit2.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libgit2' found
pkg-config: exit status 1
I found on my system a file called libgit2.pc in `/srv/slapgrid/slappart18/srv/runner/software/f7445a0b1019dad7cbae9b7eaeedb0b0/parts/gitlab/vendor/bundle/ruby/2.1.0/gems/rugged-0.24.0/vendor/libgit2/build/` so I appended this path to PKG_CONFIG_PATH environment variable.
then, I get:
$ ./srv/runner/software/f7445a0b1019dad7cbae9b7eaeedb0b0/parts/golang16/bin/go get "github.com/libgit2/git2go"
# github.com/libgit2/git2go
gopath/src/github.com/libgit2/git2go/blame.go:4:18: fatal error: git2.h: No such file or directory
compilation terminated.
which is in `./srv/runner/software/f7445a0b1019dad7cbae9b7eaeedb0b0/parts/gitlab/vendor/bundle/ruby/2.1.0/gems/rugged-0.24.0/vendor/libgit2/include/git2.h`
I tried to add its parent directory to GOPATH but it didn't help much, so I don't know how I am supposed to do to be able to import git2go and build the new git-backup. Do you have any clue?https://lab.nexedi.com/kirr/git-backup/-/issues/4build new version of git-backup and go language2020-03-09T18:23:44Zivbuild new version of git-backup and go languageAs I am not familiar with go language, I was not able to easily build git-backup. Thanks to @tc, we found out that all files are not linked, so we have to do:
$ go build *.go -o git-backup
BTW, we were also curious to know why [crypto/sha1](https://golang.org/pkg/crypto/sha1/) was not used. Is it because of assembly language in original code that may fail in case of custom CPU architectures?As I am not familiar with go language, I was not able to easily build git-backup. Thanks to @tc, we found out that all files are not linked, so we have to do:
$ go build *.go -o git-backup
BTW, we were also curious to know why [crypto/sha1](https://golang.org/pkg/crypto/sha1/) was not used. Is it because of assembly language in original code that may fail in case of custom CPU architectures?https://lab.nexedi.com/kirr/git-backup/-/issues/3Repo may be kept in locked state (after pulling failed).2020-03-09T18:23:44ZivRepo may be kept in locked state (after pulling failed).See [some](https://lab.nexedi.com/kirr/git-backup/issues/2#note_16361) [comments](https://lab.nexedi.com/kirr/git-backup/issues/2#note_16373) on issue 2.See [some](https://lab.nexedi.com/kirr/git-backup/issues/2#note_16361) [comments](https://lab.nexedi.com/kirr/git-backup/issues/2#note_16373) on issue 2.https://lab.nexedi.com/kirr/git-backup/-/issues/2Impossible to pull from dir with a non-bare repository2020-03-09T18:23:44ZivImpossible to pull from dir with a non-bare repositoryPulling/restoring a bare repository works fine for me (see commands in issue #1), but when I pull a non-bare repo, a problem occurs:
$ cd backup/
$ $ git-backup pull ../slapos/:slapos2
# file slapos2 <- ../slapos/.gitignore
# file slapos2 <- ../slapos/CHANGES.txt
# file slapos2 <- ../slapos/MANIFEST.in
# file slapos2 <- ../slapos/README.software.txt
# file slapos2 <- ../slapos/README.txt
# file slapos2 <- ../slapos/schema.json
# file slapos2 <- ../slapos/setup.py
# file slapos2 <- ../slapos/sla.schema.json
# git slapos2 <- ../slapos/.git
Traceback (most recent call last):
File "/usr/local/bin/git-backup", line 814, in <module>
main()
File "/usr/local/bin/git-backup", line 810, in main
cmd(argv[1:])
File "/usr/local/bin/git-backup", line 402, in cmd_pull
cmd_pull_(pullspecv)
File "/usr/local/bin/git-backup", line 465, in cmd_pull_
stderr=gitprogress())
File "/usr/local/bin/git-backup", line 119, in xgit
raise RuntimeError('git ' + ' '.join(argv) + '\n' + (stderr and stderr or '(failed)'))
RuntimeError: git -c fetch.fsckObjects=true fetch --no-tags ../slapos/.git refs/*:refs/backup/20160701-0820/slapos2/.git/*
fatal: Invalid refspec 'refs/*:refs/backup/20160701-0820/slapos2/.git/*'Pulling/restoring a bare repository works fine for me (see commands in issue #1), but when I pull a non-bare repo, a problem occurs:
$ cd backup/
$ $ git-backup pull ../slapos/:slapos2
# file slapos2 <- ../slapos/.gitignore
# file slapos2 <- ../slapos/CHANGES.txt
# file slapos2 <- ../slapos/MANIFEST.in
# file slapos2 <- ../slapos/README.software.txt
# file slapos2 <- ../slapos/README.txt
# file slapos2 <- ../slapos/schema.json
# file slapos2 <- ../slapos/setup.py
# file slapos2 <- ../slapos/sla.schema.json
# git slapos2 <- ../slapos/.git
Traceback (most recent call last):
File "/usr/local/bin/git-backup", line 814, in <module>
main()
File "/usr/local/bin/git-backup", line 810, in main
cmd(argv[1:])
File "/usr/local/bin/git-backup", line 402, in cmd_pull
cmd_pull_(pullspecv)
File "/usr/local/bin/git-backup", line 465, in cmd_pull_
stderr=gitprogress())
File "/usr/local/bin/git-backup", line 119, in xgit
raise RuntimeError('git ' + ' '.join(argv) + '\n' + (stderr and stderr or '(failed)'))
RuntimeError: git -c fetch.fsckObjects=true fetch --no-tags ../slapos/.git refs/*:refs/backup/20160701-0820/slapos2/.git/*
fatal: Invalid refspec 'refs/*:refs/backup/20160701-0820/slapos2/.git/*'https://lab.nexedi.com/kirr/git-backup/-/issues/1Empty dirs are not tracked (Restoring repo require their name to ends with "....2020-03-09T18:23:44ZivEmpty dirs are not tracked (Restoring repo require their name to ends with ".git")To reproduce the bug:
get some bare repo:
$ git clone --bare slapos slapos.git
backup-pull it:
$ mkdir backup && cd backup && git init
$ git-backup pull ../slapos.git/:slapos
when restoring:
$ git-backup restore HEAD slapos:../slapos.restored
$ cd .. && git clone slapos.restored slapos_bis
fatal: repository 'slapos.restored' does not exist
As @kirr explained me, it is due to git not tracking trees of empty directories:
$ls slapos.git/refs/ -Ra
slapos.git/refs/:
. .. heads tags
slapos.git/refs/heads:
. ..
slapos.git/refs/tags:
. ..
So some directory that are used by the restored repo are missing.
This is currently fixed by detecting if it is a git repo or not (it is also possible to backup usual folder/files) by [verifying its extension](https://lab.nexedi.com/kirr/git-backup/blob/a6cfe210f8bad8a816364220f148957cfdb75a41/git-backup#L679). So it works fine when restoring this way:
$ git-backup restore HEAD slapos:../slapos.restored.git
gives:
$ cd ../slapos.restored.git && git log
commit 5f5d5102cf2dd2b4a8d092e32dfc6b87fa3bbc12
Author: [...]To reproduce the bug:
get some bare repo:
$ git clone --bare slapos slapos.git
backup-pull it:
$ mkdir backup && cd backup && git init
$ git-backup pull ../slapos.git/:slapos
when restoring:
$ git-backup restore HEAD slapos:../slapos.restored
$ cd .. && git clone slapos.restored slapos_bis
fatal: repository 'slapos.restored' does not exist
As @kirr explained me, it is due to git not tracking trees of empty directories:
$ls slapos.git/refs/ -Ra
slapos.git/refs/:
. .. heads tags
slapos.git/refs/heads:
. ..
slapos.git/refs/tags:
. ..
So some directory that are used by the restored repo are missing.
This is currently fixed by detecting if it is a git repo or not (it is also possible to backup usual folder/files) by [verifying its extension](https://lab.nexedi.com/kirr/git-backup/blob/a6cfe210f8bad8a816364220f148957cfdb75a41/git-backup#L679). So it works fine when restoring this way:
$ git-backup restore HEAD slapos:../slapos.restored.git
gives:
$ cd ../slapos.restored.git && git log
commit 5f5d5102cf2dd2b4a8d092e32dfc6b87fa3bbc12
Author: [...]