slaprunner: fix argument list too long in exporter script when there is too many files

/reviewed-on nexedi/slapos!415

software/slaprunner/buildout.hash.cfg

@@ -30,7 +30,7 @@ md5sum = 7a879739afe55320ee96409bcc8a52ab
 
 [template-runner-export-script]
 filename = template/runner-export.sh.jinja2
-md5sum = 98ce179badc6af5979a64a7c3d0a2ceb
+md5sum = 5877e70b2bd5cfe06aff793125f65d6a
 
 [instance-runner-export]
 filename = instance-runner-export.cfg.in

software/slaprunner/template/runner-export.sh.jinja2

@@ -113,16 +113,18 @@ trap remove_tmp_files EXIT
 
 # Getting files from runner backup directory, as instance backup files may be
 # explicitely excluded from the backup, using the srv/exporter.exclude
+backup_directory_path="$tmp_directory/backup_files.txt"
 cd {{ directory['backup'] }}
-backup_directory_path=$(find . -path "./runner/instance/slappart*/srv/backup/*" -type f)
+find . -path "./runner/instance/slappart*/srv/backup/*" -type f -print0 > $backup_directory_path
 
 # If no backup found, it's over
-if [ -z "$backup_directory_path" ]; then
+if [ ! -s "$backup_directory_path" ]; then
   exit 0
 fi
 
 sleep 5
-sha256sum $backup_directory_path | sort -k 66 > "$tmp_backup_sum"
+cat $backup_directory_path | xargs -0 sha256sum | sort -k 66 > "$tmp_backup_sum"
+rm $backup_directory_path
 egrep "instance/slappart.*/srv/backup/" "$backup_directory/backup.signature" > "$tmp_filtered_signature"
 
 # If the diff fails, then the notifier will restart this script

[Julien Muchembled]

https://en.wikipedia.org/wiki/Cat_(Unix)#Useless_use_of_cat

[Alain Takoudjou]

thanks, if I understood well the command will become :

<$backup_directory_path | xargs -0 sha256sum | sort -k 66 > "$tmp_backup_sum"

[Julien Muchembled]

No. Either

<$backup_directory_path xargs -0 sha256sum | sort -k 66 > "$tmp_backup_sum"

or

xargs -0 sha256sum <$backup_directory_path | sort -k 66 > "$tmp_backup_sum"

BTW, sleep 5 deserves a comment if it's actually useful. Otherwise, I'd do without a temporary file:

find -path "./runner/instance/slappart*/srv/backup/*" -type f -print0 |
xargs -r -0 sha256sum | sort -k 66 > "$tmp_backup_sum"

Note the -r option to xargs. And stop if $tmp_backup_sum is empty.

[Alain Takoudjou]

Yes, this is good:

find -path "./runner/instance/slappart*/srv/backup/*" -type f -print0 |
xargs -r -0 sha256sum | sort -k 66 > "$tmp_backup_sum"

not sure why there is sleep 5 but I think it can be removed. Maybe @Nicolas can confirm.

[Julien Muchembled]

It was added in 9854c583 and I guess it's there to let enough time to an eventual backup process to do changes.

We would keep the sleep, but it looks fine to move it before the find.

[Nicolas Wavrant]

@jm is right, the "sleep 5" is used to increase the probability that files in use change, in order to restart the export process.