gendiff-ndx0: Script to show manual Nexedi-related changes inside dir

Inspired by gendiff from RPM. Defaults to .nxd0 suffix but suffix can be
set explicitly. See details in script header how to use & make local
changes so gendiff-nxd0 works.

bin/gendiff-nxd0

+#!/bin/sh
+# gendiff-nxd0 <dir>  - show manual Nexedi-related changes inside dir
+# (in vim do `:set patchmode=.nxd0` before making local changes for gendiff-nxd0 to work)
+
+suffix=.nxd0
+
+die() {
+	echo "$@" 1>&2
+	exit 1
+}
+
+usage() {
+	die "Usage: gendiff-nxd0 <dir> [extension=.nxd0]"
+}
+
+test $# -lt 1 && usage
+dir="$1"
+shift
+
+case $# in
+	0)
+		;;
+
+	1)
+		suffix="$1"
+		;;
+
+	*)
+		usage
+esac
+
+find "$dir" -name "*$suffix" | sort | \
+while read f; do
+	diff -u "$f" "${f%$suffix}"
+done

[Kirill Smelkov]

@nexedi, when making local changes on an instance it is better to be able to later extract them automatically (to apply to git tree, etc). For this I propose we use the convention to mark original files with .nxd0 extension and then use gendiff-nxd0 <dir> from this patch to see all such changes inside <dir>.

Making original copies with .ndx0 extension is easily done e.g. inside vim by :set patchmode=.nxd0.

[Julien Muchembled]

The fact that people end up putting raw diffs in slapos.git (e.g. Tyagov/slapos@a9e55d71a0f0036090964d95218249eb9629d657) gives a bad opinion about this. When the patched software is versioned with git, the proper way is to use git-format-patch.

[Kirill Smelkov]

gendiff-nxd0 does not replace git format-patch and I agree that putting patches cleanly should be better done with all headers. However not everything that needs to be tweaked on target is under git. And practicing unified way of making local after-install changes (we all try to avoid them, but practice shows it does not work 100% of the time) makes it later trivial to oversee and collect those locally made changes and put them back to git.

The procedure is thus:

• obtain manual changes from work site via gendiff-nxd0,
• apply it manyally to a git checkout via patch or git apply,
• amend as needed and eventually make a commit via git commit putting all interesting information into commit message.

then either we push the result somewhere or, in case we need to maintain the change as patch file, do git format-patch -1 and put the resulting patch file into proper place.

[Kirill Smelkov]

@nexedi, just now me, @klaus and @alain.takoudjou spent several hours and some nerves finding our why WWM production service became non-working after we rerun instantiation in the webrunner. We were observing that even though instance parameters for ZODB were correct and pointing to correct NEO cluster, actual zope.confs became having different data after the reinstantiation.

In the end it turned out slapos.recipe.template:jinja2 code was hand-modified by someone this way:

#update = install
   def update(self):
     pass

which was silently preventing parameters update to be propagated.

The situation was even more confusing because before the reinstantion, zope.confs were all correct and corresponding to the parameter in the webrunner. Since we did not changed those parameters before recent reinstantiation, we were expecting the operation to be a kind of noop. That turned out to be not true.

This was further traced to the fact, that even though slappart0 was receiving correct parameters from slapproxy, to zope's slapparts it was passing different set of ZODB parameters. That was likely because of above slapos.recipe.template:jinja2 change, because even though on slappart0 .installed.cfg contained the same set of parameters as in webrunner, its inner file slappart0/parts/dynamic-template-erp5/instance-erp5.cfg had different data. The mtime of slappart0/parts/dynamic-template-erp5/instance-erp5.cfg was Dec 6 2017 and, even though it should, it was not changing on redeploy.

That's how we digged through and 1) found manual edits to slapos.recipe.template, and that's how it became clear that 2) previously zope.conf(s) in instances were manually edited too (because else they should have all contain the same set of ZODB parameters as in slappart0/parts/dynamic-template-erp5/instance-erp5.cfg).

Editing in place is bad, however editing in place without being able to extract and see all edits done in an automatic way with 1 command like gendiff-nxd0 creates a lot of confusion, time waste and future looses of the edits.

Please, if for some reason you cannot avoid hand edits and need to go that way, please make sure to mark it appropriately so that other people can easily observe and find it. I still think that the best way here is as outlined in 2ac16a8f (comment 55309).

Kirill