neotest: Draft software-release to run NEO/go & friends tests/benchmarks under webrunner
We provide neotest SR which builds current NEO/go together with the tools which are needed to run the benchmarks.
The SR does not yet provide automated service to run the tests automatically periodically and to upload the results to ERP5 (all marked as TODO).
However with present state it provides all the infrastructure for people
to try to run NEO tests in their webrunner: a neotest program is
installed in buildout top-level bin/ . One should use it to run the
tests.
An example run output could be seen here:
https://lab.nexedi.com/snippets/258
With benchstat (https://godoc.org/golang.org/x/perf/cmd/benchstat) it can be summarized or compared to another run. Here is e.g. summarization:
name pystone/s
vifibcloud-onlinenet-hosting-004/pystone 156k ± 6%
name µs/op
vifibcloud-onlinenet-hosting-004/sha1/py/1024B 1.71 ± 6%
vifibcloud-onlinenet-hosting-004/sha1/go/1024B 2.26 ±73%
vifibcloud-onlinenet-hosting-004/sha1/py/4096B 6.39 ± 4%
vifibcloud-onlinenet-hosting-004/sha1/go/4096B 5.66 ± 2%
name us/op
vifibcloud-onlinenet-hosting-004/disk/randread/direct/4K-min 75.7 ± 4%
vifibcloud-onlinenet-hosting-004/disk/randread/direct/4K-avg 95.2 ± 1%
vifibcloud-onlinenet-hosting-004/disk/randread/pagecache/4K-avg 1.84 ± 0%
name time/op
vifibcloud-onlinenet-hosting-004/disk/randread/pagecache/4K-min 288ns ± 5%
vifibcloud-onlinenet-hosting-004/disk/randread/pagecache/4K-avg 876ns ± 9%
name µs/object
dataset:wczblk1-8
vifibcloud-onlinenet-hosting-004/fs1/zhash.py 22.2 ±11%
vifibcloud-onlinenet-hosting-004/fs1/zhash.py-P16 40.6 ±19%
vifibcloud-onlinenet-hosting-004/fs1/zhash.go 4.00 ±40%
vifibcloud-onlinenet-hosting-004/fs1/zhash.go+prefetch128 5.58 ±24%
vifibcloud-onlinenet-hosting-004/fs1/zhash.go-P16 3.68 ±88%
vifibcloud-onlinenet-hosting-004/zeo/zhash.py 626 ±40%
vifibcloud-onlinenet-hosting-004/zeo/zhash.py-P16 2.17k ± 3%
vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.py 823 ±17%
vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.py-P16 2.04k ± 3%
vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.go 666 ± 9%
vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.go+prefetch128 138 ±10%
vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.go-P16 1.96k ± 0%
vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.py 1.02k ±71%
vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.py-P16 3.09k ± 1%
vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.go 602 ±29%
vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.go+prefetch128 213 ±14%
vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.go-P16 4.66k ± 1%
vifibcloud-onlinenet-hosting-004/neo/go/zhash.py 326 ±32%
vifibcloud-onlinenet-hosting-004/neo/go/zhash.py-P16 554 ± 7%
vifibcloud-onlinenet-hosting-004/neo/go/zhash.go 39.6 ± 7%
vifibcloud-onlinenet-hosting-004/neo/go/zhash.go+prefetch128 24.3 ±18%
vifibcloud-onlinenet-hosting-004/neo/go/zhash.go-P16 213 ± 6%
vifibcloud-onlinenet-hosting-004/neo/go(!sha1)/zhash.go 42.8 ±13%
vifibcloud-onlinenet-hosting-004/neo/go(!sha1)/zhash.go+prefetch128 24.5 ±19%
vifibcloud-onlinenet-hosting-004/neo/go(!sha1)/zhash.go-P16 70.2 ±62%
dataset:prod1-1024
vifibcloud-onlinenet-hosting-004/fs1/zhash.py 18.8 ±22%
vifibcloud-onlinenet-hosting-004/fs1/zhash.py-P16 33.6 ±21%
vifibcloud-onlinenet-hosting-004/fs1/zhash.go 3.46 ±25%
vifibcloud-onlinenet-hosting-004/fs1/zhash.go+prefetch128 5.36 ±25%
vifibcloud-onlinenet-hosting-004/fs1/zhash.go-P16 2.56 ±45%
vifibcloud-onlinenet-hosting-004/zeo/zhash.py 617 ±23%
vifibcloud-onlinenet-hosting-004/zeo/zhash.py-P16 2.19k ± 1%
vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.py 717 ±46%
vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.py-P16 2.02k ± 1%
vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.go 297 ±14%
vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.go+prefetch128 130 ±10%
vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.go-P16 1.86k ± 7%
vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.py 680 ±61%
vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.py-P16 4.76k ± 1%
vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.go 269 ±13%
vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.go+prefetch128 178 ± 8%
vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.go-P16 3.03k ± 1%
vifibcloud-onlinenet-hosting-004/neo/go/zhash.py 227 ±10%
vifibcloud-onlinenet-hosting-004/neo/go/zhash.py-P16 654 ± 5%
vifibcloud-onlinenet-hosting-004/neo/go/zhash.go 32.4 ± 8%
vifibcloud-onlinenet-hosting-004/neo/go/zhash.go+prefetch128 15.5 ±12%
vifibcloud-onlinenet-hosting-004/neo/go/zhash.go-P16 261 ±12%
vifibcloud-onlinenet-hosting-004/neo/go(!sha1)/zhash.go 24.2 ± 1%
vifibcloud-onlinenet-hosting-004/neo/go(!sha1)/zhash.go+prefetch128 13.9 ±25%
vifibcloud-onlinenet-hosting-004/neo/go(!sha1)/zhash.go-P16 270 ±10%The infrastructure to build Go projects under SlapOS is also introduced along the way (was applied separetely as 1b540151).
Please see details in the individual commit messages.
/cc @nexedi
P.S. the results are noisy becuase under regular webrunner I do not have root and cannot run e.g. cpupower frequency-set -g performance.