pyprog macro, added in 0ee52376 (Generalize how nxdtest python script is
generated into pyprog recipe macro) works by generating buildout code at
runtime in text form. This way it should be careful when substituting strings,
because if those strings contain \n, then intended control flow might become broken.

For example when using pyprog with eggs = ${eggs:eggs} from stack/erp5/ ,
buildout breaks because erp5 defines eggs as multiline list:

    INFO     self.buildout.parse("""
    INFO   File "/srv/slapgrid/slappart47/srv/runner/software/7f1663e8148f227ce3c6a38fc52796e2/eggs/zc.buildout-2.7.1+slapos020-py3.9.egg/zc/buildout/buildout.py", line 1352, in parse
    INFO     sections = zc.buildout.configparser.parse(
    INFO   File "/srv/slapgrid/slappart47/srv/runner/software/7f1663e8148f227ce3c6a38fc52796e2/eggs/zc.buildout-2.7.1+slapos020-py3.9.egg/zc/buildout/configparser.py", line 241, in parse
    INFO     raise e
    INFO zc.buildout.configparser.ParsingError: File contains parsing errors:
    INFO         [line 18]: 'python-barcode\n'
    INFO         [line 19]: 'SOAPpy-py3\n'
    INFO         [line 20]: 'suds-py3\n'
    INFO         [line 21]: 'neoppod[admin, ctl, master]\n'
    INFO         [line 22]: 'cython-zstd\n'
    INFO         [line 23]: 'msgpack\n'
    INFO         [line 24]: 'mysqlclient\n'
    INFO         [line 25]: 'PyMySQL\n'
    INFO         [line 26]: 'ZODB\n'
    INFO         [line 27]: 'zodbtools\n'
    INFO         [line 28]: 'psutil\n'
         ...

-> Fix it via indenting eggs list like we already do with pyinit code.

/cc @kazuhiko, @jerome

Fluentbit Tail input documentation[1] says that by default maximum buffer
size is 32K which turned out to be too small in practice because we hit
a situation where enb.xlog started to have lines with ~ 34K and so
fluentbit ingestion stopped to work with the following error in
fluentbit log:

    [2024/10/23 20:30:23] [error] [input:tail:tail.0] file=/srv/slapgrid/slappart19/srv/monitor/public/enb.xlog requires a larger buffer size, lines are too long. Skipping file.

-> Fix that by increasing max buffer size to 1M which seems to be high
   enough at least for now.

Maybe it will make sense to configure this as unlimited, but I'm not
sure if going as unlimited is universally a good idea.

[1] https://docs.fluentbit.io/manual/pipeline/inputs/tail

/cc @lu.xu, @jhuge, @tomo
/reviewed-by @paul.graydon
/reviewed-on nexedi/slapos!1672

As discussed on nexedi/slapos@bb841a7b (comment 219278)
when using storage-path and passwd option, the storage file could not
be updated to the new format because of AttributeError _needs_migration.

This changes to no longer try to detect if the storage needs migration,
but just compare the expected content of the storage file during install
and overwrite the file if it is different.

This new approach also fix a behavior that re-running buildout with
storage-path option and a different passwd option did not update the
storage file. Now it is also updated.

( this also fixes a potential encoding problem on py2 )

fix a SyntaxWarning on py3.9

See merge request nexedi/slapos!1664

This test is using two connection one with a client to subscribe to a
topic and wait for message and another one with publish.single to
publish to the topic.
The test was failing from time to time because the publish might have
happened after the client was subscribed.

Refactor the test to use `loop` on the client to have more control
and be able to wait for the client to be subscribed using the
`on_subscribe` callback.

The test is also factorized, instead of having the same test twice for
IPv4 and IPv6, we pass the host as parameter.

See merge request nexedi/slapos!1665

from repozo doc:

> If a full backup is created, remove any prior full or incremental
> backup files (and associated metadata files) from the repository
> directory.

This solves a problem that after a pack some old repozo files were left
around, with this option they are automatically removed.

Products.TIDStorage was not ported to python3 and is not installed on
software-py3.cfg but the backup crontab expects tidstorage to be
present - as a result, it was silently failing to produce backups.

This brings minimal support to repozo backups on python3, without
Products.TIDStorage interraction and also extends software release test
to have a simple test checking that backups are produced and can be
restored.

Split the instances in two:
 - "default" instance is grafana, loki (for logs) and influxdb (for
metrics)
 - "agent" instance is telegraf collecting metrics and logs and sending
 it the the "default" instance.

Next steps will be that the agent becomes not used, instead the
slapos instances will be able to push metrics or logs directly, probably
using fluentbit and sending to either loki/influxdb or wendelin.

boot-image-url-select is used instead of default image being downloaded by the
software release.

If nothing is selected, the default boot-image-url-select is used, but not if
other way to obatin boot image is enabled.

The vm-img uses ISOs internally and let it handle them, thus qemu-kvm can be
just a component providing qemu machine.

Also nbd2-host and nbd2-port are asserted.

Such status file is not created (apparently never was)

GDAL is a component used for a particular client and without a
corresponding software release, so its compilation was not being tested
in SlapOS software release tests.

This commit adds GDAL to the list of tested software, so that it is not
broken by accident.

See merge request nexedi/slapos!1586.

Bumped both zc.buildout and zc.recipe.egg to keep them in sync.

See merge request nexedi/slapos!1586.

The upgrade is done to version 3.2.3, as this is the last version which
still supports Python 2.

This fixes the following compilation bug:

```c
gdalwmsdataset.cpp:383:37: error: ISO C++ forbids comparison between
pointer and integer [-fpermissive]
	if(pszHttpZeroBlockCodes == '\0') {
```

Jasper is no longer a dependency, and openjpeg is used instead.

See merge request nexedi/slapos!1586.

OpenJPEG provides a JPEG 2000 codec for use in GDAL.
This was previously provided by Jasper, but support for Jasper is
removed in recent GDAL versions.

See merge request nexedi/slapos!1586.

Most recent versions of GDAL require this functionality (and it does
not hurt in any case).

See merge request nexedi/slapos!1586.

Renamed proj4 to proj, to use current naming of the project.
Upgrades PROJ to the last available version (9.4.0) as a previous
step to upgrade GDAL, which depends on it.

See merge request nexedi/slapos!1586.

Upgrades GEOS to the last available version (3.12.2) as a previous
step to upgrade GDAL, which depends on it.

See merge request !1586.

We are using ZODB6 in ERP5 py3

needed for ZODB >= 5

THis is a fixup of 4df39d83.

Also copy generated binary to buildout root's bin folder.