- 04 Nov, 2017 1 commit
-
-
Julien Muchembled authored
-
- 03 Nov, 2017 4 commits
-
-
Julien Muchembled authored
See nexedi/erp5@5169f193
-
eteri authored
Support for HTTPS added. @rafael @kirr @gabriel /reviewed-on nexedi/slapos!249
-
Kazuhiko Shiozaki authored
-
Kazuhiko Shiozaki authored
-
- 02 Nov, 2017 1 commit
-
-
Rafael Monnerat authored
-
- 30 Oct, 2017 2 commits
-
-
Romain Courteaud authored
-
Jérome Perrin authored
We are preparing a new buildout version slapos.buildout!12 but until that we can force the default index in profiles /reviewed-on !248
-
- 27 Oct, 2017 1 commit
-
-
Eteri authored
Version up for golang from go1.9.1 to go1.9.2 @rafael @kirr @alain.takoudjou [ kirr: from https://golang.org/doc/devel/release.html#go1.9.minor go1.9.2 (released 2017/10/25) includes fixes to the compiler, linker, runtime, documentation, go command, and the crypto/x509, database/sql, log, and net/smtp packages. It includes a fix to a bug introduced in Go 1.9.1 that broke go get of non-Git repositories under certain conditions. See the Go 1.9.2 milestone on our issue tracker for details. @gabriel says they tested go1.9.2 on Caddy ] /reviewed-on !247
-
- 26 Oct, 2017 1 commit
-
-
Kazuhiko Shiozaki authored
-
- 25 Oct, 2017 5 commits
-
-
Kirill Smelkov authored
9b5d8262 (v↑ zodbtools (0.0.0.dev4)) added zodburi and in turn ZEO dependency to neoppod/software-common.cfg because zodburi currently non-optionally requires ZEO. However I forgot to notice that if we always pin ZEO to 4.x it will require `transaction < 2` and this will break neoppod/software-zodb5.cfg which requires ZODB 5.x and in turn `transaction >= 2`: Installing neoppod. Version and requirements information containing transaction: [versions] constraint on transaction: 2.1.2 Requirement of ZODB==5.3.0: transaction>=2.0.3 Requirement of transaction>=2.0.3: zope.interface Requirement of ZEO: transaction<2.0.0 While: Installing neoppod. Error: The requirement ('transaction<2.0.0') is not allowed by your [versions] constraint (2.1.2) We already have ZODB = 4.x in software-common.cfg and software-zodb5.cfg overrides it to ZODB = 5.x, but ZEO was left not-overwritten and this way there becomes conflicting requirement on transaction. Fix it by also overriding ZEO in software-zodb5.cfg /reported-and-reviewed-by @jm /reported-on nexedi/slapos!242 (comment 49754)
-
Eteri authored
Caddy initial version with gowork.cfg @rafael @kirr /reviewed-on nexedi/slapos!244
-
Alain Takoudjou authored
[golang14] fix TestParseInLocation test: Install package from latest changes on upstream go14 branch https://github.com/golang/go/commit/4d5426a570c2820c5894a61b52e3dc147e4e7925 golang14 is still failing to build on some server, the commit above fix the problem on test TestParseInLocation. This is why we install golang14 from upstream go14 branch. [ kirr: see also relateddiscussion on nexedi/slapos!133 ] /reviewed-by @kirr /reviewed-on nexedi/slapos!245
-
Kirill Smelkov authored
We provide neotest SR which builds current NEO/go together with the tools which are needed to run the benchmarks. The SR does not yet provide automated service to run the tests automatically periodically and to upload the results to ERP5 (all marked as TODO). However with present state it provides all the infrastructure for people to try to run NEO tests in their webrunner: a `neotest` program is installed in buildout top-level bin/ . One should use it to run the tests. An example run output could be seen here: https://lab.nexedi.com/snippets/258 With benchstat (https://godoc.org/golang.org/x/perf/cmd/benchstat) it can be summarized or compared to another run. Here is e.g. summarization: ``` name pystone/s vifibcloud-onlinenet-hosting-004/pystone 156k ± 6% name µs/op vifibcloud-onlinenet-hosting-004/sha1/py/1024B 1.71 ± 6% vifibcloud-onlinenet-hosting-004/sha1/go/1024B 2.26 ±73% vifibcloud-onlinenet-hosting-004/sha1/py/4096B 6.39 ± 4% vifibcloud-onlinenet-hosting-004/sha1/go/4096B 5.66 ± 2% name us/op vifibcloud-onlinenet-hosting-004/disk/randread/direct/4K-min 75.7 ± 4% vifibcloud-onlinenet-hosting-004/disk/randread/direct/4K-avg 95.2 ± 1% vifibcloud-onlinenet-hosting-004/disk/randread/pagecache/4K-avg 1.84 ± 0% name time/op vifibcloud-onlinenet-hosting-004/disk/randread/pagecache/4K-min 288ns ± 5% vifibcloud-onlinenet-hosting-004/disk/randread/pagecache/4K-avg 876ns ± 9% name µs/object dataset:wczblk1-8 vifibcloud-onlinenet-hosting-004/fs1/zhash.py 22.2 ±11% vifibcloud-onlinenet-hosting-004/fs1/zhash.py-P16 40.6 ±19% vifibcloud-onlinenet-hosting-004/fs1/zhash.go 4.00 ±40% vifibcloud-onlinenet-hosting-004/fs1/zhash.go+prefetch128 5.58 ±24% vifibcloud-onlinenet-hosting-004/fs1/zhash.go-P16 3.68 ±88% vifibcloud-onlinenet-hosting-004/zeo/zhash.py 626 ±40% vifibcloud-onlinenet-hosting-004/zeo/zhash.py-P16 2.17k ± 3% vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.py 823 ±17% vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.py-P16 2.04k ± 3% vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.go 666 ± 9% vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.go+prefetch128 138 ±10% vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.go-P16 1.96k ± 0% vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.py 1.02k ±71% vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.py-P16 3.09k ± 1% vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.go 602 ±29% vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.go+prefetch128 213 ±14% vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.go-P16 4.66k ± 1% vifibcloud-onlinenet-hosting-004/neo/go/zhash.py 326 ±32% vifibcloud-onlinenet-hosting-004/neo/go/zhash.py-P16 554 ± 7% vifibcloud-onlinenet-hosting-004/neo/go/zhash.go 39.6 ± 7% vifibcloud-onlinenet-hosting-004/neo/go/zhash.go+prefetch128 24.3 ±18% vifibcloud-onlinenet-hosting-004/neo/go/zhash.go-P16 213 ± 6% vifibcloud-onlinenet-hosting-004/neo/go(!sha1)/zhash.go 42.8 ±13% vifibcloud-onlinenet-hosting-004/neo/go(!sha1)/zhash.go+prefetch128 24.5 ±19% vifibcloud-onlinenet-hosting-004/neo/go(!sha1)/zhash.go-P16 70.2 ±62% dataset:prod1-1024 vifibcloud-onlinenet-hosting-004/fs1/zhash.py 18.8 ±22% vifibcloud-onlinenet-hosting-004/fs1/zhash.py-P16 33.6 ±21% vifibcloud-onlinenet-hosting-004/fs1/zhash.go 3.46 ±25% vifibcloud-onlinenet-hosting-004/fs1/zhash.go+prefetch128 5.36 ±25% vifibcloud-onlinenet-hosting-004/fs1/zhash.go-P16 2.56 ±45% vifibcloud-onlinenet-hosting-004/zeo/zhash.py 617 ±23% vifibcloud-onlinenet-hosting-004/zeo/zhash.py-P16 2.19k ± 1% vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.py 717 ±46% vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.py-P16 2.02k ± 1% vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.go 297 ±14% vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.go+prefetch128 130 ±10% vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.go-P16 1.86k ± 7% vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.py 680 ±61% vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.py-P16 4.76k ± 1% vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.go 269 ±13% vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.go+prefetch128 178 ± 8% vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.go-P16 3.03k ± 1% vifibcloud-onlinenet-hosting-004/neo/go/zhash.py 227 ±10% vifibcloud-onlinenet-hosting-004/neo/go/zhash.py-P16 654 ± 5% vifibcloud-onlinenet-hosting-004/neo/go/zhash.go 32.4 ± 8% vifibcloud-onlinenet-hosting-004/neo/go/zhash.go+prefetch128 15.5 ±12% vifibcloud-onlinenet-hosting-004/neo/go/zhash.go-P16 261 ±12% vifibcloud-onlinenet-hosting-004/neo/go(!sha1)/zhash.go 24.2 ± 1% vifibcloud-onlinenet-hosting-004/neo/go(!sha1)/zhash.go+prefetch128 13.9 ±25% vifibcloud-onlinenet-hosting-004/neo/go(!sha1)/zhash.go-P16 270 ±10% ``` The infrastructure to build Go projects under SlapOS is also introduced along the way (was applied separetely as nexedi/slapos@1b540151). Please see details in the individual commit messages. /cc @nexedi P.S. the results are noisy becuase under regular webrunner I do not have root and cannot run e.g. `cpupower frequency-set -g performance`. /reviewed-on nexedi/slapos!242
-
Kirill Smelkov authored
We provide neotest SR which builds current NEO/go together with the tools which are needed to run the benchmarks. Go part uses just-introduced in previous commit gowork infrastructure. The SR does not yet provide automated service to run the tests automatically periodically and to upload the results to ERP5 (all marked as TODO). However with present state it provides all the infrastructure for people to try to run NEO tests in their webrunner: a `neotest` program is installed in buildout top-level bin/ . One should use it to run the tests. For example: slapuser14@vifibcloud-onlinenet-hosting-004:~/srv/runner/t$ ../software/4db21ec948dce895b38ba93c1def3ab1/bin/neotest Neotest is a tool to functionally test and benchmark NEO. Usage: neotest command [arguments] The commands are: bench-local run benchmarks when client and server are both on the same localhost bench-cluster run benchmarks when server is local and client is on another node run-client run client benchmarks against separate server bench-disk benchmark local disk (already part of bench-{local,cluster}) bench-cpu benchmark local cpu (already part of bench-{local,cluster}) deploy deploy NEO & needed software for tests to remote host deploy-local deploy NEO & needed software for tests locally info print information about a node info-local print information about local deployment Additional utility commands: cpustat run a command and print CPU-related statistics slapuser14@vifibcloud-onlinenet-hosting-004:~/srv/runner/t$ ../software/4db21ec948dce895b38ba93c1def3ab1/bin/neotest info-local # Thu, 19 Oct 2017 18:13:35 +0300 # slapuser14@sd-112617.dedibox.fr (2001:67c:1254:e:8c::1 (+ 20·ipv6) 163.172.70.8 (+ 20·ipv4)) # Linux vifibcloud-onlinenet-hosting-004 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux # cpu: Intel(R) Xeon(R) CPU D-1531 @ 2.20GHz # cpu[0-11]: freq: intel_pstate/powersave [.80GHz - 2.70GHz] # cpu[0-11]: idle: acpi_idle/menu: POLL(0μs) C1(1μs) C2(41μs) # cpu: WARNING: frequency not fixed - benchmark timings won't be stable # cpu: WARNING: C-state exit-latency is max 41μs - up to that can add to networked and IPC request-reply latency # md1 (raid0) -> sda3 sdb3 # sda: SAMSUNG MZ7LN256 rev 100Q 238.5G # sdb: SAMSUNG MZ7LN256 rev 100Q 238.5G # eth0: Intel Corporation I350 Gigabit Network Connection rev 01 # eth0: features: rx tx sg tso !ufo gso gro !lro rxvlan txvlan !ntuple rxhash ... # eth0: coalesce: rxc: 3μs/0f/0μs-irq/0f-irq, txc: 0μs/0f/0μs-irq/0f-irq # eth0: up, speed=1000, mtu=1500, txqlen=1000, !gro_flush_timeout # eth1: Intel Corporation I350 Gigabit Network Connection rev 01 # eth1: features: rx tx sg tso !ufo gso gro !lro rxvlan txvlan !ntuple rxhash ... # eth1: coalesce: rxc: 3μs/0f/0μs-irq/0f-irq, txc: 0μs/0f/0μs-irq/0f-irq # eth1: down, speed=?, mtu=1500, txqlen=1000, !gro_flush_timeout # Python 2.7.14 # go version go1.9.1 linux/amd64 # sqlite 3.19.0 (py mod 2.6.0) # mysqld Ver 10.1.28-MariaDB for Linux on x86_64 (MariaDB Server) # neo : v1.8-1326-g4d0cd894 # zodb : 4.4.5 # zeo : 4.3.1 # mysqlclient : 1.3.12 # wendelin.core : v0.11-4-g38fbc83 slapuser14@vifibcloud-onlinenet-hosting-004:~/srv/runner/t$ time ../software/4db21ec948dce895b38ba93c1def3ab1/bin/neotest bench-local 2>&1 |tee 1.txt ... *** FileStorage Benchmarkvifibcloud-onlinenet-hosting-004/fs1/zhash.py 1 24.4 µs/object # crc32:1552c530 oid=0..2127 nread=8534126 t=0.052s # POLL·0 C1·49 C2·853 Benchmarkvifibcloud-onlinenet-hosting-004/fs1/zhash.py 1 24.6 µs/object # crc32:1552c530 oid=0..2127 nread=8534126 t=0.052s # POLL·0 C1·63 C2·894 Benchmarkvifibcloud-onlinenet-hosting-004/fs1/zhash.py 1 20.7 µs/object # crc32:1552c530 oid=0..2127 nread=8534126 t=0.044s # POLL·0 C1·39 C2·803 Benchmarkvifibcloud-onlinenet-hosting-004/fs1/zhash.py 1 20.8 µs/object # crc32:1552c530 oid=0..2127 nread=8534126 t=0.044s # POLL·1 C1·18 C2·629 Benchmarkvifibcloud-onlinenet-hosting-004/fs1/zhash.py 1 20.7 µs/object # crc32:1552c530 oid=0..2127 nread=8534126 t=0.044s # POLL·0 C1·12 C2·669 ...
-
- 24 Oct, 2017 8 commits
-
-
Hardik Juneja authored
-
Hardik Juneja authored
-
Boxiang Sun authored
-
Eteri authored
-
Eteri authored
-
Eteri authored
-
Yusei Tahara authored
-
Yusei Tahara authored
-
- 23 Oct, 2017 17 commits
-
-
eteri authored
-
eteri authored
-
eteri authored
-
eteri authored
-
eteri authored
-
eteri authored
-
eteri authored
-
Gabriel Monnerat authored
-
Gabriel Monnerat authored
-
Gabriel Monnerat authored
WIP
-
Kirill Smelkov authored
lmbench is a bit outdated tool to do OS performance analysis. The upcoming neotest SR will use lat_tcp from this packages to measure TCP round-trip-time latency. We use a bit patched version with fixes to lat_tcp for errors not to go unnoticed and also lat_tcp.go added to see how using Go compares to plain lat_tcp.c for latencies. In the upcoming neotest SR both lat_tcp C & Go versions will be used to automatically benchmark TCP round-trip-times over network link in between 2 machines.
-
Kirill Smelkov authored
This tool can be used to measure disk IO latency for various workloads (random read, sequential read, direct or cached, write, ...). We use a bit patched version shows not only avg latency but also its distribution. (the patches are currently dirty/not very robust and not yet sent to upstream). In the upcoming neotest SR ioping will be used to automatically benchmark underlying disk characteristics.
-
Kirill Smelkov authored
ethtool can be used to adjust NIC settings, such as e.g. interrupt coalescing and various others. It usually requires root to do so. However regular users can query NIC settings via ethtool just ok. This way upcoming neotest SR will use it to query/display NIC setting which are essential to get good networked latency and automatically produce warnings if it is not.
-
Kirill Smelkov authored
Starting from this version zodbtools requires zodburi to open ZODB storages by URL (see nexedi/zodbtools@db3b84ec and nexedi/zodbtools@82b06413) zodburi in turn requires ZEO (maybe we might patch it to only optionally depend on it), so we move ZEO requirement from stack/ERP5 to the place where it is actually first needed.
-
Kirill Smelkov authored
In Go world development workflow is organized around so-called workspace where multiple packages can be installed / worked on etc. The following page describes Go workspaces: https://golang.org/doc/code.html A new [gowork] section and infrastructure around it is introduced. Quoting code: # gowork is a top-level section representing workspace # # users should add `install` field to [gowork] to describe packages they want to # be installed (+ automatically their dependencies are installed too). e.g. # # [gowork] # install = # lab.nexedi.com/kirr/neo/go/... \ # github.com/pkg/profile \ # golang.org/x/perf/cmd/benchstat The way it all works inside is: - gowork organizes to create go.work/ directory in buildout root - inside it creates env.sh which when sources in shell adjusts all paths so that appropriate go compiler is in path, GOPATH is also set appropriately, etc - in other words everything needs for using/working on this workspace is setup in the environment. - in actual user software profile the list of Go projects which needs to be installed by SlapOS has to be listed. The [gowork] machinery takes care about git-cloning these projects and `go install`s them after. - by SlapOS design builds have to be reproducible and so every component state/version has to be fixed. A convenient helper (gowork-snapshot) to get snapshot of git go packages installed with their exact revisions is added. This should be used to automatically (re-)generate list of go git repositories & their pinning for a workspace. For a new Go project that needs to be slaposified the workflow should be: - first get a project into working state via any convenient way (`go get`, etc.) - generate list of go packages & their pinning with gowork-snapshot - extend golang/buildout.cfg and in project software profile add [gowork] install = your-top-level-packages as it was outlined above. P.S. [golang] is removed because 1. it is not used anywhere in slapos.git tree, and 2. the canonical way to work on go projects from now on will be to use [gowork]. There latest go version is preconfigured, but users can override it to some particular other version in their projects as they need. /reviewed-on nexedi/slapos!242 /also-needed-for nexedi/slapos!243
-
Rafael Monnerat authored
This reverts merge request !243
-