1. 20 Oct, 2020 6 commits
    • Kirill Smelkov's avatar
      *: Factor-out NumPy version into component/numpy/ · eacc0038
      Kirill Smelkov authored
      Move `numpy=1.16.4` from all over the place into component/numpy.
      Don't move if a different numpy version is used, or it looks like a
      software cares to use exactly particular version.
      Downgrade pygolang/test.cfg from numpy=1.16.6 to numpy=1.16.4 and use
      common numpy component version - using numpy=1.16.6 is not required for
      pygolang testing and so this downgrade is acceptable. It will be better
      to upgrade NumPy to latest in component/numpy/ as a future separate step.
      
      See previous patch where it was decided and explained that version for
      component <X> lives in component/X/.
      eacc0038
    • Kirill Smelkov's avatar
      stack/nxdtest · 0dd3e119
      Kirill Smelkov authored
      Currently Nexedi takes mostly a copy-paste approach when it comes to
      organizing bin/runTestSuite and an instance to run this inside testnode:
      
          kirr@deco:~/src/wendelin/slapos/slapos-master$ git ls-files |grep runTestSuite
          software/build-rina/runTestSuite.in
          software/buildout-testing/runTestSuite.in
          software/cython-test/runTestSuite.in
          software/erp5testnode/testsuite/deploy-test/runTestSuite.py
          software/jstestnode/runTestSuite.in
          software/neoppod/runTestSuite.in
          software/neoppod/stress-testing/runTestSuite.in
          software/unstable/pyodide/runTestSuite.in
      
      and adding new tests to be run this way will only increase the entropy.
      
      Let's stop this, at least for new tests, and use a uniform approach without copying:
      
      1) I've created nxdtest tool[1] which allows a project to specify in
         declarative way how to test itself: given such declaration nxdtest
         takes on itself the responsibility to communicate with testnode
         master, to drive test commands spawning etc, and leaves to the
         project only semantic part that is specific to the particular
         project.
      
         Nxdtest is based on my go/neo/t/nxd/runTestSuite[2,3] and on the code
         I've seen elsewhere scattered in other runTestSuite.in files.
      
      2) Let's add stack/nxdtest.cfg to SlapOS that provides easy way for a
         software or component to organize testing of itself.
      
      This patch adds stack/nxdtest and migrates software/neotest to use it
      (because stack/nxdtest is based on it and this way it helps to track the
      changes).
      
      In the following patches we will add testing support for pygolang,
      zodbtools and wendelin.core .
      
      [1] https://lab.nexedi.com/kirr/nxdtest/blob/master/nxdtest
      [2] kirr/neo@51b18490
      [3] kirr/neo@f67c147d
      0dd3e119
    • Kirill Smelkov's avatar
      neotest: v↑ everything *.go · ff633c08
      Kirill Smelkov authored
      Update lab.nexedi.com/kirr/neo.git and lab.nexedi.com/kirr/go123 to latest.
      Update all other go packages.
      
      The update puls in the following new dependencies:
      
          github.com/shamaton/msgpack
          github.com/tinylib/msgp
            github.com/philhofer/fwd
            github.com/ttacon/chalk
            golang.org/x/mod
            golang.org/x/xerrors
      
          github.com/davecgh/go-spew
          github.com/pmezard/go-difflib
          gopkg.in/yaml.v3
      ff633c08
    • Kirill Smelkov's avatar
    • Kirill Smelkov's avatar
      6b9c863b
    • Kirill Smelkov's avatar
      ZEO: Factor it to separate component · 83618cd6
      Kirill Smelkov authored
      We already patch ZEO4 with TCP_NODELAY patch (see 5cf4cf1f "ERP5:
      enable TCP_NODELAY for ZEO") and we will need to backport more patches
      to ZEO4 branch for wendelin.core 2 to work correctly.
      
      It's not only software/neoppod which uses ZEO, and it is not convenient for
      all other software-releases to inherit from neoppod to use correct
      version and build of ZEO egg. For this reason factor out details of ZEO
      egg building into component/ZEO and let users use ${ZEO:egg} where ZEO
      is needed. This way ZEO will be correctly installed for all users.
      
      This patch should be a non-functional change. We switch to
      nexedi/ZEO@5114f909 revision which corresponds to ZEO 4.3.1 +
      TCP_NODELAY.patch
      
      Adding other patches to ZEO4 needed by wendelin.core 2 will be done as a
      separate step.
      83618cd6
  2. 24 Apr, 2020 1 commit
  3. 22 Mar, 2020 1 commit
  4. 08 Nov, 2019 1 commit
  5. 13 Aug, 2019 1 commit
  6. 07 Aug, 2019 1 commit
  7. 05 Feb, 2019 3 commits
  8. 21 Jan, 2019 1 commit
  9. 29 Oct, 2018 1 commit
    • Kirill Smelkov's avatar
      golang: Don't require users to put \ into gowork.install · 754acb1b
      Kirill Smelkov authored
      Previously it was required to put \ into multi-entry gowork.install,
      because the way it was processed was to directly pass it into bash
      command, and if \n were there the command was cut by bash and the
      trailing was considered as another command.
      
      We can avoid that by explicitly processing \n in gowork.install and
      translate it to just ' '. This helps people because now gowork.install
      becomes just a regular multiline buildout entry, similar e.g. to
      gowork.cpkgpath et all.
      
      Here is one example where it helps:
      
      jerome/slapos@95c45fe0 (comment 68408)
      
      /reviewed-by @jerome
      /reviewed-on nexedi/slapos!430
      754acb1b
  10. 17 Jul, 2018 1 commit
  11. 12 Jul, 2018 1 commit
  12. 03 Jul, 2018 1 commit
    • Kirill Smelkov's avatar
      neotest: Fix comment in runTestSuite wrapper · d3475887
      Kirill Smelkov authored
      Commit c6c919dc (Remove all occurrences of the obsolete
      'parameters-extra' option) removed
      
      	parameters-extra = true
      
      because after 5c980caf (Drop the 'parameters-extra' option and always
      forward extra parameters) "$@" is always implicitly added to argv.
      
      Clarify the comment that was explaining what `parameters-extra = true`
      was doing, since it was not updated during parameters-extra removal.
      d3475887
  13. 27 Mar, 2018 1 commit
    • Kirill Smelkov's avatar
      neotest: v↑ everything · fb5d114a
      Kirill Smelkov authored
      Update verion of lab.nexed.com/kirr/neo.git used and all of its dependencies.
      
      NEO/go now uses 2 Cgo packages:
      
      - github.com/gwenn/gosqlite, and
      - github.com/DataDog/czlib
      
      and github.com/soheilhy/cmux to multiplex NEO & HTTP traffic on the same port.
      
      Otherwise it is straighforward update which transitively pulled some more
      dependencies.
      fb5d114a
  14. 16 Mar, 2018 2 commits
  15. 05 Feb, 2018 3 commits
    • Kirill Smelkov's avatar
      neotest: Recent util-linux is also needed · 5685a461
      Kirill Smelkov authored
      Similarly to coreutils we need recent util-linux because if e.g. a testnode has
      old version of it its lsblk does not work to what neotest expects, e.g.:
      
      	lsblk: unknown column: REV,SIZE
      5685a461
    • Kirill Smelkov's avatar
      neotest: Recent coreutils are needed to run · c5d0d9fc
      Kirill Smelkov authored
      For example we need realpath and if base os provides only older coreutils there
      is no such utility and neotest fails:
      
      /srv/slapgrid/slappart8/srv/testnode/bct/soft/55158b6a5b027aa40954c3f974d5b3e9/go.work/src/lab.nexedi.com/kirr/neo/go/neo/t/neotest: line 589: realpath: command not found
      
      Fix it by building & hooking into $PATH coreutils by ourselves.
      c5d0d9fc
    • Kirill Smelkov's avatar
      neotest: Actually add instance and teach it to run `neotest bench-local` under... · 41597528
      Kirill Smelkov authored
      neotest: Actually add instance and teach it to run `neotest bench-local` under Nexedi testing infrastructure
      
      Commit 31fe231f (neotest: Draft software-release to run NEO/go & friends
      tests/benchmarks under webrunner) added "neotest" software release, but
      there it added only software supplying - not instantiation - this way
      allowing only manual play with neotest.
      
      Recently @rafael approached me saying that one of my webrunner is red on
      monitoring because "neotest" software inside fails to instantiate, and
      suggested to create a simple instance with bin/runTestSuite inside, so
      that this way neotest could be added to a testnode and run on regular
      basis by our testing infrastructure.
      
      So here it goes - simple instance with only bin/runTestSuite which tries
      to adhere to testnode protocol.
      
      For the actual runTestSuite program I followed the same approach as for
      bin/neotest itself - the main program is not processed via any templates
      and is used as it was pristinely downloaded, and all buildout
      integration is done via first sourcing $SR/neotest-env.sh .
      So bin/runTestSuite inside instance follows this approach.
      
      erp5.util is updated from 0.4.49 to latest 0.4.50 because there were
      backward-incompatible changes in 0.4.50 release (2 @luke: which btw will
      render neoppod and build-rina runTestSuite(s) broken when they try to
      upgrade) and for me it would be a waste of time to later go through
      updating for 0.4.49 -> 0.4.50 API change, so I used the latest release
      right from start.
      
      /cc @Tyagov
      /see also kirr/neo@526491d3
      41597528
  16. 25 Oct, 2017 1 commit
    • Kirill Smelkov's avatar
      neotest: Draft software-release to run NEO/go & friends tests/benchmarks under webrunner · 31fe231f
      Kirill Smelkov authored
      We provide neotest SR which builds current NEO/go together with the
      tools which are needed to run the benchmarks. Go part uses
      just-introduced in previous commit gowork infrastructure.
      
      The SR does not yet provide automated service to run the tests
      automatically periodically and to upload the results to ERP5 (all marked
      as TODO).
      
      However with present state it provides all the infrastructure for people
      to try to run NEO tests in their webrunner: a `neotest` program is
      installed in buildout top-level bin/ . One should use it to run the
      tests.
      
      For example:
      
      	slapuser14@vifibcloud-onlinenet-hosting-004:~/srv/runner/t$ ../software/4db21ec948dce895b38ba93c1def3ab1/bin/neotest
      	Neotest is a tool to functionally test and benchmark NEO.
      
      	Usage:
      
      	        neotest command [arguments]
      
      	The commands are:
      
      	        bench-local     run benchmarks when client and server are both on the same localhost
      	        bench-cluster   run benchmarks when server is local and client is on another node
      
      	        run-client      run client benchmarks against separate server
      	        bench-disk      benchmark local disk (already part of bench-{local,cluster})
      	        bench-cpu       benchmark local cpu  (already part of bench-{local,cluster})
      
      	        deploy          deploy NEO & needed software for tests to remote host
      	        deploy-local    deploy NEO & needed software for tests locally
      
      	        info            print information about a node
      	        info-local      print information about local deployment
      
      	Additional utility commands:
      
      	        cpustat         run a command and print CPU-related statistics
      
      	slapuser14@vifibcloud-onlinenet-hosting-004:~/srv/runner/t$ ../software/4db21ec948dce895b38ba93c1def3ab1/bin/neotest info-local
      	# Thu, 19 Oct 2017 18:13:35 +0300
      	# slapuser14@sd-112617.dedibox.fr (2001:67c:1254:e:8c::1 (+ 20·ipv6) 163.172.70.8 (+ 20·ipv4))
      	# Linux vifibcloud-onlinenet-hosting-004 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux
      	# cpu: Intel(R) Xeon(R) CPU D-1531 @ 2.20GHz
      	# cpu[0-11]: freq: intel_pstate/powersave [.80GHz - 2.70GHz]
      	# cpu[0-11]: idle: acpi_idle/menu: POLL(0μs) C1(1μs) C2(41μs)
      	# cpu: WARNING: frequency not fixed - benchmark timings won't be stable
      	# cpu: WARNING: C-state exit-latency is max 41μs - up to that can add to networked and IPC request-reply latency
      	# md1 (raid0) -> sda3  sdb3
      	# sda: SAMSUNG MZ7LN256  rev 100Q 238.5G
      	# sdb: SAMSUNG MZ7LN256  rev 100Q 238.5G
      	# eth0: Intel Corporation I350 Gigabit Network Connection rev 01
      	# eth0: features: rx tx sg tso !ufo gso gro !lro rxvlan txvlan !ntuple rxhash ...
      	# eth0: coalesce: rxc: 3μs/0f/0μs-irq/0f-irq,  txc: 0μs/0f/0μs-irq/0f-irq
      	# eth0: up, speed=1000, mtu=1500, txqlen=1000, !gro_flush_timeout
      	# eth1: Intel Corporation I350 Gigabit Network Connection rev 01
      	# eth1: features: rx tx sg tso !ufo gso gro !lro rxvlan txvlan !ntuple rxhash ...
      	# eth1: coalesce: rxc: 3μs/0f/0μs-irq/0f-irq,  txc: 0μs/0f/0μs-irq/0f-irq
      	# eth1: down, speed=?, mtu=1500, txqlen=1000, !gro_flush_timeout
      	# Python 2.7.14
      	# go version go1.9.1 linux/amd64
      	# sqlite 3.19.0 (py mod 2.6.0)
      	# mysqld  Ver 10.1.28-MariaDB for Linux on x86_64 (MariaDB Server)
      	# neo             : v1.8-1326-g4d0cd894
      	# zodb            : 4.4.5
      	# zeo             : 4.3.1
      	# mysqlclient     : 1.3.12
      	# wendelin.core   : v0.11-4-g38fbc83
      
      slapuser14@vifibcloud-onlinenet-hosting-004:~/srv/runner/t$ time ../software/4db21ec948dce895b38ba93c1def3ab1/bin/neotest bench-local 2>&1 |tee 1.txt
      ...
      *** FileStorage
      Benchmarkvifibcloud-onlinenet-hosting-004/fs1/zhash.py 1 24.4 µs/object # crc32:1552c530  oid=0..2127  nread=8534126  t=0.052s  # POLL·0 C1·49 C2·853
      Benchmarkvifibcloud-onlinenet-hosting-004/fs1/zhash.py 1 24.6 µs/object # crc32:1552c530  oid=0..2127  nread=8534126  t=0.052s  # POLL·0 C1·63 C2·894
      Benchmarkvifibcloud-onlinenet-hosting-004/fs1/zhash.py 1 20.7 µs/object # crc32:1552c530  oid=0..2127  nread=8534126  t=0.044s  # POLL·0 C1·39 C2·803
      Benchmarkvifibcloud-onlinenet-hosting-004/fs1/zhash.py 1 20.8 µs/object # crc32:1552c530  oid=0..2127  nread=8534126  t=0.044s  # POLL·1 C1·18 C2·629
      Benchmarkvifibcloud-onlinenet-hosting-004/fs1/zhash.py 1 20.7 µs/object # crc32:1552c530  oid=0..2127  nread=8534126  t=0.044s  # POLL·0 C1·12 C2·669
      ...
      31fe231f