component/systemd: Version up

Switch to pure meson build system and adapt configuration for SlapOS
requirements.

Support multiarch env by simply linking from lib64.

Version 249.6 incorporates fix from https://bugs.gentoo.org/824026 for:
  meson.build: ERROR: Object <[BooleanHolder] holds [bool]: False> of
  type bool does not support the +

component/systemd/buildout.cfg

@@ -9,16 +9,77 @@ extends =
   ../pkgconfig/buildout.cfg
   ../util-linux/buildout.cfg
   ../xz-utils/buildout.cfg
+  ../ninja/buildout.cfg
+  ../meson/buildout.cfg
+
+[systemd-markupsafe-download]
+recipe = slapos.recipe.build:download
+shared = true
+url = https://files.pythonhosted.org/packages/4d/de/32d741db316d8fdb7680822dd37001ef7a448255de9699ab4bfcbdf4172b/${:filename}
+filename = MarkupSafe-1.0.tar.gz
+md5sum = 2fcedc9284d50e577b5192e8e3578355
+
+[systemd-jinja2-download]
+recipe = slapos.recipe.build:download
+shared = true
+url = https://files.pythonhosted.org/packages/4f/e7/65300e6b32e69768ded990494809106f87da1d436418d5f1367ed3966fd7/${:filename}
+filename = Jinja2-2.11.3.tar.gz
+md5sum = 231dc00d34afb2672c497713fa9cdaaa
+
+[systemd-python]
+recipe = slapos.recipe.build
+shared = true
+markupsafe = ${systemd-markupsafe-download:target}
+jinja2 = ${systemd-jinja2-download:target}
+
+install =
+  import os
+  call([options['python'], '-m', 'venv', '--clear', location])
+  pip = os.path.join(location, 'bin', 'pip')
+  call([pip, 'install', '--no-index', options['markupsafe'], options['jinja2']])
+  call([pip, 'uninstall', '-y', 'pip', 'setuptools'])
+  # selftest
+  python = os.path.join(location, 'bin', 'python3')
+  call([python, '-c', 'import jinja2'])
+python = ${python3:executable}
+
+[systemd-python:python3]
+python = ${buildout:executable}
 
 [systemd]
-recipe = slapos.recipe.cmmi
+recipe = slapos.recipe.build
+shared = true
 # XXX This version requires Linux kernel >= 3.7.
-url = https://www.freedesktop.org/software/systemd/systemd-221.tar.xz
-md5sum = b4d5a253841cf28a98b7ec99c45e3716
-location = ${buildout:parts-directory}/${:_buildout_section_name_}
-make-options = rootprefix=${:location} SYSTEM_SYSVINIT_PATH=${:location}/etc/init.d
+url = https://github.com/systemd/systemd-stable/archive/v249.6.tar.gz
+md5sum = eb4a1923009abdb4e25c4802f4921a18
+install =
+  env = self.environ
+  extract_dir = self.extract(self.download(options['url'], options['md5sum']))
+  workdir = guessworkdir(extract_dir)
+  call([
+    'meson', 'setup', 'build',
+    '-Dprefix=%s' % (location,),
+    '-Drootprefix=%s' % (location,),
+    '-Dsysconfdir=%s/etc' % (location,),
+    '-Dsysvinit-path=%s/etc/init.d' % (location,),
+    '-Dsysvrcnd-path=%s/etc/rc.d' % (location,),
+    '-Dlocalstatedir=%s/var' % (location,),
+    '-Dcreate-log-dirs=False',
+    '-Dhwdb=False',
+  ], cwd=workdir, env=env)
+  call(['meson', 'setup', 'build',], cwd=workdir, env=env)
+  call(['meson', 'compile', '-C', 'build/'], cwd=workdir, env=env)
+  call(['meson', 'install', '-C', 'build/'], cwd=workdir, env=env)
+  import os
+  # On some systems multiarch is detected, and then libraries go to
+  # x86_64-linux-gnu, and as there was no way to control this behaviour
+  # during meson setup build phase, simply link it to lib64, where other
+  # parts are looking for libraries
+  if os.path.exists(os.path.join(location, 'x86_64-linux-gnu')):
+    call(['ln', '-s', 'x86_64-linux-gnu', 'lib64'], cwd=location)
+
 environment =
-  PATH=${coreutils:location}/bin:${gettext:location}/bin:${gperf:location}/bin:${intltool:location}/bin:${perl-XML-Parser:perl-PATH}:${pkgconfig:location}/bin:${xz-utils:location}/bin:%(PATH)s
+  PATH=${systemd-python:location}/bin:${meson:location}/bin:${ninja:location}/bin:${coreutils:location}/bin:${gettext:location}/bin:${gperf:location}/bin:${intltool:location}/bin:${perl-XML-Parser:perl-PATH}:${pkgconfig:location}/bin:${xz-utils:location}/bin:%(PATH)s
   CPPFLAGS=-I${libcap:location}/include -I${util-linux:location}/include
   LDFLAGS=-L${libcap:location}/lib -Wl,-rpath=${libcap:location}/lib -L${util-linux:location}/lib -Wl,-rpath=${util-linux:location}/lib
   PKG_CONFIG_PATH=${util-linux:location}/lib/pkgconfig

[Julien Muchembled]

Why --no-index and separate sections to download Jinja2 & MarkupSafe ? The problem is that 0895cef8 is wrong because the condition does not apply on the Python version that's used to install systemd. I mean that if [python3] default to 3.8+ and buildout runs Python 2, it's the recent MarkupSafe that must be used. Currently, the NEO-STRESS test suite fails because of that.

I don't really want to commit:

--- a/component/systemd/buildout.cfg
+++ b/component/systemd/buildout.cfg
@@ -19,7 +19,11 @@ url = https://files.pythonhosted.org/packages/bf/10/ff66fea6d1788c458663a84d8878
 filename = MarkupSafe-2.0.1.tar.gz
 md5sum = 892e0fefa3c488387e5cc0cad2daa523
 
-[systemd-markupsafe-download:sys.version_info < (3,8)]
+# WARNING: [systemd-markupsafe-download] is only used by systemd-python and
+#          when buildout runs with Python 2, MarkupSafe is installed with
+#          [python3] which default to Python 3.8+. Even if Python 2 can only
+#          install version 1.0, that's the other version we must use.
+[systemd-markupsafe-download:python3 and sys.version_info < (3,8)]
 url = https://files.pythonhosted.org/packages/4d/de/32d741db316d8fdb7680822dd37001ef7a448255de9699ab4bfcbdf4172b/${:filename}
 filename = MarkupSafe-1.0.tar.gz
 md5sum = 2fcedc9284d50e577b5192e8e3578355

If not perfect, better keep things simple.

If we could just write MarkupSafe=2.0.1 & MarkupSafe=1.0 (without everything else that appears in the download sections), it should be easy to have everything inside [systemd-python] and then choose depending on the actual version of python (${systemd-python:python} would be the name of the python section, from which we'd get the executable and the version).

Maybe we could use buildout API to download dependencies.

/cc @tleymone

[Julien Muchembled]

I suggest 8cc2f9d7. I didn't spend any time to see if we could use buildout API to download dependencies.

[Julien Muchembled]

For example:

Installing systemd-python.
Ignoring MarkupSafe: markers 'python_version < "3.8"' don't match your environment
Collecting Jinja2==2.11.3
  Using cached Jinja2-2.11.3-py2.py3-none-any.whl (125 kB)
Collecting MarkupSafe==2.0.1
  Using cached MarkupSafe-2.0.1-cp38-cp38-manylinux2010_x86_64.whl (30 kB)
Installing collected packages: MarkupSafe, Jinja2
Successfully installed Jinja2-2.11.3 MarkupSafe-2.0.1
WARNING: You are using pip version 20.2.3; however, version 22.2.1 is available.
You should consider upgrading via the '/srv/slapgrid/slappart93/srv/runner/shared/systemd-python/3ab3200afe9411d4dde2d59f4a090de6/bin/python3 -m pip install --upgrade pip' command.
Found existing installation: pip 20.2.3
Uninstalling pip-20.2.3:
  Successfully uninstalled pip-20.2.3
Found existing installation: setuptools 49.2.1
Uninstalling setuptools-49.2.1:
  Successfully uninstalled setuptools-49.2.1

[Julien Muchembled]

However, I start to think we do too much for Python 2. I still need qemu with Python 2 for NEO but no ceph. On old OS, it currently works because ceph is skipped but on Debian 10, [librbd] fails to build due to missing Python 3.

What about disabling ceph buildout runs with Python 2 and then stopping to use venvs in systemd/ceph ?

[Łukasz Nowak]

What about disabling ceph buildout runs with Python 2 and then stopping to use venvs in systemd/ceph ?

+1 for this then, as ceph is already disabled conditionally in b0601ca6 so the condition can be simply extended there.