1. 10 Aug, 2021 5 commits
    • Łukasz Nowak's avatar
    • Eric Zheng's avatar
      add headless chromium software release · 224b207e
      Eric Zheng authored
      This also adds a headless-chromium component and enables the http_sub
      module in the Nginx component. Some basic tests are included. See merge
      request nexedi/slapos!1014 for more details and discussion.
      224b207e
    • Eric Zheng's avatar
      software/headless-chromium: create headless Chromium SR · 907c97f0
      Eric Zheng authored
      This change includes a rewrite of the previous
      component/headless-chromium, which used an outdated version of Chromium.
      It differs from component/chromium in that:
      
      - It only provides a headless shell, rather than the full Chromium
        browser (though this can be changed).
      - It compiles from source rather than downloading a binary build.
      
      There is also a new software release software/headless-chromium which
      makes use of the updated component/headless-chromium to expose a remote
      debugging port.
      
      See also nexedi/slapos!1014.
      907c97f0
    • Kirill Smelkov's avatar
      wendelin.core: Fix test build · c8c3071f
      Kirill Smelkov authored
      Wendelin.core2[test] depends on scipy[1]. Before slapos.buildout
      2.7.1+slapos013 the build was succeeding because scipy was, likely, installed
      via binary wheel. But after that buildout upgrade the build fails:
      
          WARNING: The easy_install command is deprecated and will be removed in a future version.
          /tmp/easy_install-vYdhd1/scipy-0.19.0/setup.py:323: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
            warnings.warn("Unrecognized setuptools command, proceeding with "
          Traceback (most recent call last):
            File "<string>", line 10, in <module>
            File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/command/easy_install.py", line 2321, in main
              **kw
            File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/__init__.py", line 162, in setup
              return distutils.core.setup(**attrs)
            File "/srv/slapgrid/slappart91/srv/runner/shared/python2.7/93d57ff089fd75f374514794469a0538/lib/python2.7/distutils/core.py", line 151, in setup
              dist.run_commands()
            File "/srv/slapgrid/slappart91/srv/runner/shared/python2.7/93d57ff089fd75f374514794469a0538/lib/python2.7/distutils/dist.py", line 953, in run_commands
              self.run_command(cmd)
            File "/srv/slapgrid/slappart91/srv/runner/shared/python2.7/93d57ff089fd75f374514794469a0538/lib/python2.7/distutils/dist.py", line 972, in run_command
              cmd_obj.run()
            File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/command/easy_install.py", line 424, in run
              self.easy_install(spec, not self.no_deps)
            File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/command/easy_install.py", line 666, in easy_install
              return self.install_item(None, spec, tmpdir, deps, True)
            File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/command/easy_install.py", line 711, in install_item
              dists = self.install_eggs(spec, download, tmpdir)
            File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/command/easy_install.py", line 896, in install_eggs
              return self.build_and_install(setup_script, setup_base)
            File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/command/easy_install.py", line 1164, in build_and_install
              self.run_setup(setup_script, setup_base, args)
            File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/command/easy_install.py", line 1150, in run_setup
              run_setup(setup_script, args)
            File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/sandbox.py", line 253, in run_setup
              raise
            File "/srv/slapgrid/slappart91/srv/runner/shared/python2.7/93d57ff089fd75f374514794469a0538/lib/python2.7/contextlib.py", line 35, in __exit__
              self.gen.throw(type, value, traceback)
            File "<string>", line 7, in setup_context
            File "/srv/slapgrid/slappart91/srv/runner/shared/python2.7/93d57ff089fd75f374514794469a0538/lib/python2.7/contextlib.py", line 35, in __exit__
              self.gen.throw(type, value, traceback)
            File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/sandbox.py", line 195, in setup_context
              yield
            File "/srv/slapgrid/slappart91/srv/runner/shared/python2.7/93d57ff089fd75f374514794469a0538/lib/python2.7/contextlib.py", line 35, in __exit__
              self.gen.throw(type, value, traceback)
            File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/sandbox.py", line 166, in save_modules
              saved_exc.resume()
            File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/sandbox.py", line 141, in resume
              six.reraise(type, exc, self._tb)
            File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/sandbox.py", line 154, in save_modules
              yield saved
            File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/sandbox.py", line 195, in setup_context
              yield
            File "<string>", line 7, in setup_context
            File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/sandbox.py", line 250, in run_setup
              _execfile(setup_script, ns)
            File "/srv/slapgrid/slappart91/srv/runner/software/3335682bae677c2d474f9244e578f64b/eggs/setuptools-44.1.1-py2.7.egg/setuptools/sandbox.py", line 45, in _execfile
              exec(code, globals, locals)
            File "/tmp/easy_install-vYdhd1/scipy-0.19.0/setup.py", line 416, in <module>
            File "/tmp/easy_install-vYdhd1/scipy-0.19.0/setup.py", line 396, in setup_package
          ImportError: No module named numpy.distutils.core
          An error occurred when trying to install scipy 0.19.0. Look above this message for any errors that were output by easy_install.
          While:
            Installing wendelin.core-python.
            Base installation request: 'wendelin.core[test]', 'pygolang[pyx.build]', 'neoppod[tests]', 'ZEO[test]'
              Requirement of ZEO[test]: random2
              Requirement of ZEO[test]: manuel
              Requirement of ZEO[test]: zope.testing
              Requirement of ZEO[test]: zope.interface
              Requirement of ZEO[test]: zdaemon
              Requirement of ZEO[test]: ZConfig
              Requirement of ZEO[test]: zc.lockfile
              Requirement of ZEO[test]: persistent>=4.1.0
              Requirement of ZEO[test]: transaction<2.0.0
              Requirement of ZEO[test]: six
              Requirement of ZEO[test]: ZODB<4.999,>=4.2.0b1
              Requirement of neoppod[tests]==1.12.0: neoppod[admin,client,ctl,master,storage-importer,storage-mysqldb,storage-sqlite]
              Requirement of neoppod[tests]==1.12.0: psutil>=2
              Requirement of neoppod[tests]==1.12.0: zope.testing
              Requirement of neoppod[tests]==1.12.0: coverage
              Requirement of neoppod[tests]==1.12.0: python-dateutil
              Requirement of pygolang[pyx.build]==0.0.8: wheel
              Requirement of pygolang[pyx.build]==0.0.8: setuptools_dso>=1.7
              Requirement of pygolang[pyx.build]==0.0.8: setuptools
              Requirement of pygolang[pyx.build]==0.0.8: cython
              Requirement of pygolang[pyx.build]==0.0.8: Importing
              Requirement of pygolang[pyx.build]==0.0.8: decorator
              Requirement of pygolang[pyx.build]==0.0.8: six
              Requirement of pygolang[pyx.build]==0.0.8: gevent
              Requirement of wendelin.core[test]: scipy
              Requirement of wendelin.core[test]: pytest
              Requirement of wendelin.core[test]: psutil
              Requirement of wendelin.core[test]: six
              Requirement of wendelin.core[test]: pygolang>=0.0.8
              Requirement of wendelin.core[test]: zodbtools>=0.0.0.dev8
              Requirement of wendelin.core[test]: ZODB>=4
              Requirement of wendelin.core[test]: numpy
              Requirement of manuel: six
              Requirement of manuel: setuptools
              Requirement of zope.testing: zope.interface
              Requirement of zope.testing: zope.exceptions
              Requirement of zope.testing: setuptools
              Requirement of zope.interface: setuptools
              Requirement of zdaemon: setuptools
              Requirement of zdaemon: ZConfig
              Requirement of zc.lockfile: setuptools
              Requirement of persistent>=4.1.0: cffi
              Requirement of persistent>=4.1.0: zope.interface
              Requirement of transaction<2.0.0: zope.interface
              Requirement of ZODB<4.999,>=4.2.0b1: zodbpickle>=0.6.0
              Requirement of ZODB<4.999,>=4.2.0b1: zope.interface
              Requirement of ZODB<4.999,>=4.2.0b1: zc.lockfile
              Requirement of ZODB<4.999,>=4.2.0b1: six
              Requirement of ZODB<4.999,>=4.2.0b1: transaction>=1.5.0
              Requirement of ZODB<4.999,>=4.2.0b1: ZConfig
              Requirement of ZODB<4.999,>=4.2.0b1: BTrees>=4.2.0
              Requirement of ZODB<4.999,>=4.2.0b1: persistent>=4.2.0
              Requirement of neoppod[admin,client,ctl,master,storage-importer,storage-mysqldb,storage-sqlite]: ZODB3>=3.10dev
              Requirement of neoppod[admin,client,ctl,master,storage-importer,storage-mysqldb,storage-sqlite]: mysqlclient
              Requirement of neoppod[admin,client,ctl,master,storage-importer,storage-mysqldb,storage-sqlite]: setproctitle
              Requirement of neoppod[admin,client,ctl,master,storage-importer,storage-mysqldb,storage-sqlite]: msgpack>=0.5.6
              Requirement of neoppod[admin,client,ctl,master,storage-importer,storage-mysqldb,storage-sqlite]: ZODB3>=3.10dev
              Requirement of neoppod[admin,client,ctl,master,storage-importer,storage-mysqldb,storage-sqlite]: python-dateutil
              Requirement of python-dateutil: six>=1.5
              Requirement of gevent: greenlet>=0.4.17
              Requirement of gevent: setuptools
              Requirement of gevent: zope.interface
              Requirement of gevent: zope.event
            Getting distribution for 'scipy==0.19.0'.
          Error: Couldn't install: scipy 0.19.0
      
      -> Fix it by explicitly installing SciPy via our component/scipy/ which lists numpy in its setup_eggs.
      
      [1] https://lab.nexedi.com/kirr/wendelin.core/blob/5e647b59/setup.py#L384-386
      c8c3071f
    • Xavier Thompson's avatar
      slapos-sr-testing: Fix default parameters handling · a4f7af38
      Xavier Thompson authored
      If `only-sr` is not provided, all the SR tests should run.
      
      Fixes merge request !1011
      a4f7af38
  2. 09 Aug, 2021 1 commit
  3. 06 Aug, 2021 2 commits
    • Jérome Perrin's avatar
      software/repman: retry requests to prevent flaky test · 4ef735c6
      Jérome Perrin authored
      Test was sometimes failing with:
      
      ======================================================================
      FAIL: test_http_get (test.TestRepman)
      ----------------------------------------------------------------------
      Traceback (most recent call last):
        File "slapos-repository/software/repman/test/test.py", line 81, in test_http_get
          self.assertTrue(cluster['isProvision'])
      AssertionError: False is not true
      
      It can take a little time before everything is ready and we don't have a
      promise for this, because it's really application level.
      4ef735c6
    • Jérome Perrin's avatar
      Minor fixes to schemas · 926d2097
      Jérome Perrin authored
      See merge request nexedi/slapos!1031
      926d2097
  4. 05 Aug, 2021 4 commits
  5. 02 Aug, 2021 9 commits
  6. 30 Jul, 2021 1 commit
    • Vincent Pelletier's avatar
      stack/erp5: Make replicating mariadb write to their own binlogs. · 2b1d5b16
      Vincent Pelletier authored
      This change has no effect on stand-alone setups (which is what is
      automatically deployed).
      On setups configured to replicate from another database, it allows them to
      takeover the role of the primary server, which allows rebuilding a
      redundant setup. On such setup, the cost of this option is the extra I/O
      load of binlogs being written to (in addition to the relay log and
      database), and the corresponding disk space use.
      2b1d5b16
  7. 29 Jul, 2021 3 commits
    • Łukasz Nowak's avatar
      41af80a5
    • Thomas Gambier's avatar
    • Kirill Smelkov's avatar
      Fix build for zodbtools/pygolang/nxdtest/... tests · 3d6db516
      Kirill Smelkov authored
      Commit 7d1ea024 (version up: zc.buildout 2.7.1+slapos013) switched buildout to
      not use egg wheels, even for wheels that are source-only without any binaries.
      As the result some eggs stopped to install because they setup_require other
      eggs and neither buildout handles setup_requires automatically, nor there is
      explicit setup_require declaration in corresponding components.
      
      An example of such egg is python-dateutil: it setup_requires setuptools-scm,
      and without switching python-dateutil to zc.recipe.egg:custom with
      setup_requires the build fails.
      
      The way 7d1ea024 used to workaround this is to mark all such eggs with :whl
      suffix marking them that "it is ok to use wheel for this particular egg". The
      complete list of eggs for which such markings was done in 7d1ea024 is:
      
          ipykernel
          plantuml
          mock
          PasteScript
          backports.functools-lru-cache
          cliff
          stevedore
          CacheControl
          pytest-runner
          jsonschema
          lockfile
          python-dateutil
      
      However this list is not really complete because after 7d1ea024 the build
      started to fail for e.g. zodbtools test:
      
              Traceback (most recent call last):
                File "<string>", line 10, in <module>
                File "/srv/slapgrid/slappart16/srv/testnode/dfp/soft/c670f74c8de08964f63c4b9f7d48c3d1/eggs/setuptools-44.1.1-py2.7.egg/setuptools/command/easy_install.py", line 2321, in main
                ...
                File "/srv/slapgrid/slappart16/srv/testnode/dfp/soft/c670f74c8de08964f63c4b9f7d48c3d1/eggs/setuptools-44.1.1-py2.7.egg/pkg_resources/__init__.py", line 786, in resolve
                  raise DistributionNotFound(req, requirers)
              pkg_resources.DistributionNotFound: The 'setuptools-scm' distribution was not found and is required by the application
              An error occurred when trying to install pytest 4.6.11. Look above this message for any errors that were output by easy_install.
              While:
                Installing zodbtools-python.
                Base installation request: 'zodbtools[test]'
                  Requirement of zodbtools[test]: mock
                  Requirement of zodbtools[test]: pytz
                  Requirement of zodbtools[test]: freezegun
                  Requirement of zodbtools[test]: pytest
                  Requirement of zodbtools[test]: dateparser
                  Requirement of zodbtools[test]: six
                  Requirement of zodbtools[test]: pygolang>=0.0.0.dev6
                  Requirement of zodbtools[test]: zope.interface
                  Requirement of zodbtools[test]: zodburi
                  Requirement of zodbtools[test]: ZODB
                  Requirement of mock: funcsigs>=1
                  Requirement of mock: six
                  Requirement of freezegun: python-dateutil>=2.7
                Getting distribution for 'pytest==4.6.11'.
              Error: Couldn't install: pytest 4.6.11
      
      and similarly for pygolang and nxdtest tests.
      
      I would be more comfortable if the fix would be to teach buildout to either
      handle setup_requires automatically, or to automatically install wheels if the
      wheel is source-only. However, given the way - that 7d1ea024 started to use
      to handle such situations - let's fix this via explicitly marking pytest, and
      other related eggs that setup_require something and are source-only, as :whl.
      
      NOTE we did not notice the problem earlier because on testnodes, after the
      switch to new buildout, pytest egg was already installed and incremental
      rebuild did not need to reinstall pytest egg. So most of pygolang and zodbtools
      test results remained in "ok" state, and only zodbtools-kirr, that was recently
      reconfigured or changed testnode, indicated this issue. That's why the problem
      remained unnoticed for a long time.
      
      /reviewed-by @jerome
      /reviewed-on nexedi/slapos!1026
      3d6db516
  8. 28 Jul, 2021 2 commits
  9. 27 Jul, 2021 2 commits
  10. 26 Jul, 2021 2 commits
  11. 23 Jul, 2021 2 commits
  12. 20 Jul, 2021 2 commits
  13. 16 Jul, 2021 2 commits
    • Lisa Casino's avatar
      html5as: update with the new recipe · e6668258
      Lisa Casino authored
      e6668258
    • Kirill Smelkov's avatar
      v↑ tempstorage(ZODB4) 2.12.2 -> 3-nxd · 610bad24
      Kirill Smelkov authored
      In 6e696faa "v↑ tempstorage 2.12.2 -> 5.2(ZODB5) 3-nxd(ZODB4-wc2)" we upgraded
      tempstorage for ZODB5 and ZODB4-wc2, but were careful and left tempstorage,
      that was being used with ZODB4, intact. However @tomo suggests that we should
      also upgrade tempstorage for ZODB4 because of the loadBefore fixes and not to
      further grow divergence in between ZODB4 and ZODB4-wc2:
      
          I understand that keeping the same version of tempstorage for ZODB 4 is to
          be on the safe side but I'd rather change it to tempstorage4-wc2 anyway. I mean
          we have a bugfix but we are afraid to apply it. Also, The less difference
          between ZODB 4 and ZODB 4-wc2 the better (everything will be more tested this way).
          ( nexedi/slapos!1009 (comment 137985) )
      
      -> So let's do that.
      610bad24
  14. 15 Jul, 2021 3 commits
    • Jérome Perrin's avatar
      software/erp5: fix tmpdir in caucase partition · d96b6895
      Jérome Perrin authored
      This was introduced recently, in 0aecb41a (software: Use local tmp for
      caucase, 2021-05-18). This is harmless, but uses a weird-looking directory
      d96b6895
    • Jérome Perrin's avatar
      component/tesseract: fix permissions of downloaded traineddata · 7df9bc95
      Jérome Perrin authored
      With the default umask of running slapos node, the default umask create files
      that are only readable by user, not by group:
      
          slapuser10@COMP-1234:~$ ls -al /opt/slapgrid/bf0ccebdbd76d491bcbe16b7998529b2/parts/tesseract/share/tessdata/
          total 125296
          drwxr-x--- 4 slapsoft slapsoft 4096 Jul 13 05:57 .
          drwxr-x--- 3 slapsoft slapsoft 4096 Jul 13 05:57 ..
          -rw------- 1 slapsoft slapsoft 44366093 Jul 13 05:57 chi_sim.traineddata
          drwxr-x--- 2 slapsoft slapsoft 4096 Jul 13 05:57 configs
          -rw------- 1 slapsoft slapsoft 23466654 Jul 13 05:57 eng.traineddata
          -rw------- 1 slapsoft slapsoft 14213351 Jul 13 05:57 fra.traineddata
          -rw------- 1 slapsoft slapsoft 35659159 Jul 13 05:57 jpn.traineddata
          -rw------- 1 slapsoft slapsoft 10562874 Jul 13 05:57 osd.traineddata
          -rw-r--r-- 1 slapsoft slapsoft 572 Jul 13 05:57 pdf.ttf
          drwxr-x--- 2 slapsoft slapsoft 4096 Jul 13 05:57 tessconfigs
      
      but when using multiple users, partitions users are members of slapsoft group,
      so these files needs to be group readable.
      7df9bc95
    • Thomas Gambier's avatar