1. 30 Oct, 2017 2 commits
  2. 27 Oct, 2017 1 commit
    • Eteri's avatar
      version up: golang · f8dfe078
      Eteri authored
      Version up for golang from go1.9.1 to go1.9.2
      
      @rafael @kirr @alain.takoudjou
      
      [ kirr: from https://golang.org/doc/devel/release.html#go1.9.minor
      
      go1.9.2 (released 2017/10/25) includes fixes to the compiler, linker, runtime, documentation, go command, and the crypto/x509, database/sql, log, and net/smtp packages. It includes a fix to a bug introduced in Go 1.9.1 that broke go get of non-Git repositories under certain conditions. See the Go 1.9.2 milestone on our issue tracker for details.
      
      @gabriel says they tested go1.9.2 on Caddy ]
      
      /reviewed-on !247
      f8dfe078
  3. 26 Oct, 2017 1 commit
  4. 25 Oct, 2017 5 commits
    • Kirill Smelkov's avatar
      fixup! v↑ zodbtools (0.0.0.dev4) · 6fc10621
      Kirill Smelkov authored
      9b5d8262 (v↑ zodbtools (0.0.0.dev4)) added zodburi and in turn ZEO
      dependency to neoppod/software-common.cfg because zodburi currently
      non-optionally requires ZEO.
      
      However I forgot to notice that if we always pin ZEO to 4.x it will
      require `transaction < 2` and this will break neoppod/software-zodb5.cfg
      which requires ZODB 5.x and in turn `transaction >= 2`:
      
      	Installing neoppod.
      	Version and requirements information containing transaction:
      	  [versions] constraint on transaction: 2.1.2
      	  Requirement of ZODB==5.3.0: transaction>=2.0.3
      	  Requirement of transaction>=2.0.3: zope.interface
      	  Requirement of ZEO: transaction<2.0.0
      	While:
      	  Installing neoppod.
      	Error: The requirement ('transaction<2.0.0') is not allowed by your [versions] constraint (2.1.2)
      
      We already have ZODB = 4.x in software-common.cfg and software-zodb5.cfg
      overrides it to ZODB = 5.x, but ZEO was left not-overwritten and this
      way there becomes conflicting requirement on transaction.
      
      Fix it by also overriding ZEO in software-zodb5.cfg
      
      /reported-and-reviewed-by @jm
      /reported-on !242 (comment 49754)
      6fc10621
    • Eteri's avatar
      Caddy v0.1 · 914f4dbf
      Eteri authored
      Caddy initial version with gowork.cfg
      
      @rafael @kirr 
      
      /reviewed-on !244
      914f4dbf
    • Alain Takoudjou's avatar
      [golang14] fix TestParseInLocation test: Install package from latest changes... · 1daeadb7
      Alain Takoudjou authored
      [golang14] fix TestParseInLocation test: Install package from latest changes on upstream go14 branch
      
      https://github.com/golang/go/commit/4d5426a570c2820c5894a61b52e3dc147e4e7925
      
      golang14 is still failing to build on some server, the commit above fix the
      problem on test TestParseInLocation. This is why we install golang14 from upstream go14 branch.
      
      [ kirr: see also relateddiscussion on !133 ]
      
      /reviewed-by @kirr
      /reviewed-on !245
      1daeadb7
    • Kirill Smelkov's avatar
      neotest: Draft software-release to run NEO/go & friends tests/benchmarks under webrunner · fd43ba62
      Kirill Smelkov authored
      We provide neotest SR which builds current NEO/go together with the
      tools which are needed to run the benchmarks.
      
      The SR does not yet provide automated service to run the tests
      automatically periodically and to upload the results to ERP5 (all marked
      as TODO).
      
      However with present state it provides all the infrastructure for people
      to try to run NEO tests in their webrunner: a `neotest` program is
      installed in buildout top-level bin/ . One should use it to run the
      tests.
      
      An example run output could be seen here:
      
      https://lab.nexedi.com/snippets/258
      
      With benchstat (https://godoc.org/golang.org/x/perf/cmd/benchstat) it can be summarized or compared to another run. Here is e.g. summarization:
      
      ```
      name                                                                 pystone/s
      vifibcloud-onlinenet-hosting-004/pystone                              156k ± 6%
      
      name                                                                 µs/op
      vifibcloud-onlinenet-hosting-004/sha1/py/1024B                        1.71 ± 6%
      vifibcloud-onlinenet-hosting-004/sha1/go/1024B                        2.26 ±73%
      vifibcloud-onlinenet-hosting-004/sha1/py/4096B                        6.39 ± 4%
      vifibcloud-onlinenet-hosting-004/sha1/go/4096B                        5.66 ± 2%
      
      name                                                                 us/op
      vifibcloud-onlinenet-hosting-004/disk/randread/direct/4K-min          75.7 ± 4%
      vifibcloud-onlinenet-hosting-004/disk/randread/direct/4K-avg          95.2 ± 1%
      vifibcloud-onlinenet-hosting-004/disk/randread/pagecache/4K-avg       1.84 ± 0%
      
      name                                                                 time/op
      vifibcloud-onlinenet-hosting-004/disk/randread/pagecache/4K-min      288ns ± 5%
      vifibcloud-onlinenet-hosting-004/disk/randread/pagecache/4K-avg      876ns ± 9%
      
      name                                                                 µs/object
      dataset:wczblk1-8
      vifibcloud-onlinenet-hosting-004/fs1/zhash.py                         22.2 ±11%
      vifibcloud-onlinenet-hosting-004/fs1/zhash.py-P16                     40.6 ±19%
      vifibcloud-onlinenet-hosting-004/fs1/zhash.go                         4.00 ±40%
      vifibcloud-onlinenet-hosting-004/fs1/zhash.go+prefetch128             5.58 ±24%
      vifibcloud-onlinenet-hosting-004/fs1/zhash.go-P16                     3.68 ±88%
      vifibcloud-onlinenet-hosting-004/zeo/zhash.py                          626 ±40%
      vifibcloud-onlinenet-hosting-004/zeo/zhash.py-P16                    2.17k ± 3%
      vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.py                823 ±17%
      vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.py-P16          2.04k ± 3%
      vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.go                666 ± 9%
      vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.go+prefetch128    138 ±10%
      vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.go-P16          1.96k ± 0%
      vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.py                 1.02k ±71%
      vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.py-P16             3.09k ± 1%
      vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.go                   602 ±29%
      vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.go+prefetch128       213 ±14%
      vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.go-P16             4.66k ± 1%
      vifibcloud-onlinenet-hosting-004/neo/go/zhash.py                       326 ±32%
      vifibcloud-onlinenet-hosting-004/neo/go/zhash.py-P16                   554 ± 7%
      vifibcloud-onlinenet-hosting-004/neo/go/zhash.go                      39.6 ± 7%
      vifibcloud-onlinenet-hosting-004/neo/go/zhash.go+prefetch128          24.3 ±18%
      vifibcloud-onlinenet-hosting-004/neo/go/zhash.go-P16                   213 ± 6%
      vifibcloud-onlinenet-hosting-004/neo/go(!sha1)/zhash.go               42.8 ±13%
      vifibcloud-onlinenet-hosting-004/neo/go(!sha1)/zhash.go+prefetch128   24.5 ±19%
      vifibcloud-onlinenet-hosting-004/neo/go(!sha1)/zhash.go-P16           70.2 ±62%
      dataset:prod1-1024
      vifibcloud-onlinenet-hosting-004/fs1/zhash.py                         18.8 ±22%
      vifibcloud-onlinenet-hosting-004/fs1/zhash.py-P16                     33.6 ±21%
      vifibcloud-onlinenet-hosting-004/fs1/zhash.go                         3.46 ±25%
      vifibcloud-onlinenet-hosting-004/fs1/zhash.go+prefetch128             5.36 ±25%
      vifibcloud-onlinenet-hosting-004/fs1/zhash.go-P16                     2.56 ±45%
      vifibcloud-onlinenet-hosting-004/zeo/zhash.py                          617 ±23%
      vifibcloud-onlinenet-hosting-004/zeo/zhash.py-P16                    2.19k ± 1%
      vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.py                717 ±46%
      vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.py-P16          2.02k ± 1%
      vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.go                297 ±14%
      vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.go+prefetch128    130 ±10%
      vifibcloud-onlinenet-hosting-004/neo/py/sqlite/zhash.go-P16          1.86k ± 7%
      vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.py                   680 ±61%
      vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.py-P16             4.76k ± 1%
      vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.go                   269 ±13%
      vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.go+prefetch128       178 ± 8%
      vifibcloud-onlinenet-hosting-004/neo/py/sql/zhash.go-P16             3.03k ± 1%
      vifibcloud-onlinenet-hosting-004/neo/go/zhash.py                       227 ±10%
      vifibcloud-onlinenet-hosting-004/neo/go/zhash.py-P16                   654 ± 5%
      vifibcloud-onlinenet-hosting-004/neo/go/zhash.go                      32.4 ± 8%
      vifibcloud-onlinenet-hosting-004/neo/go/zhash.go+prefetch128          15.5 ±12%
      vifibcloud-onlinenet-hosting-004/neo/go/zhash.go-P16                   261 ±12%
      vifibcloud-onlinenet-hosting-004/neo/go(!sha1)/zhash.go               24.2 ± 1%
      vifibcloud-onlinenet-hosting-004/neo/go(!sha1)/zhash.go+prefetch128   13.9 ±25%
      vifibcloud-onlinenet-hosting-004/neo/go(!sha1)/zhash.go-P16            270 ±10%
      ```
      
      The infrastructure to build Go projects under SlapOS is also introduced along the way
      (was applied separetely as 1b540151).
      
      Please see details in the individual commit messages. 
      
      /cc @nexedi 
      
      P.S. the results are noisy becuase under regular webrunner I do not have root and cannot run e.g. `cpupower frequency-set -g performance`.
      
      /reviewed-on !242
      fd43ba62
    • Kirill Smelkov's avatar
      neotest: Draft software-release to run NEO/go & friends tests/benchmarks under webrunner · 31fe231f
      Kirill Smelkov authored
      We provide neotest SR which builds current NEO/go together with the
      tools which are needed to run the benchmarks. Go part uses
      just-introduced in previous commit gowork infrastructure.
      
      The SR does not yet provide automated service to run the tests
      automatically periodically and to upload the results to ERP5 (all marked
      as TODO).
      
      However with present state it provides all the infrastructure for people
      to try to run NEO tests in their webrunner: a `neotest` program is
      installed in buildout top-level bin/ . One should use it to run the
      tests.
      
      For example:
      
      	slapuser14@vifibcloud-onlinenet-hosting-004:~/srv/runner/t$ ../software/4db21ec948dce895b38ba93c1def3ab1/bin/neotest
      	Neotest is a tool to functionally test and benchmark NEO.
      
      	Usage:
      
      	        neotest command [arguments]
      
      	The commands are:
      
      	        bench-local     run benchmarks when client and server are both on the same localhost
      	        bench-cluster   run benchmarks when server is local and client is on another node
      
      	        run-client      run client benchmarks against separate server
      	        bench-disk      benchmark local disk (already part of bench-{local,cluster})
      	        bench-cpu       benchmark local cpu  (already part of bench-{local,cluster})
      
      	        deploy          deploy NEO & needed software for tests to remote host
      	        deploy-local    deploy NEO & needed software for tests locally
      
      	        info            print information about a node
      	        info-local      print information about local deployment
      
      	Additional utility commands:
      
      	        cpustat         run a command and print CPU-related statistics
      
      	slapuser14@vifibcloud-onlinenet-hosting-004:~/srv/runner/t$ ../software/4db21ec948dce895b38ba93c1def3ab1/bin/neotest info-local
      	# Thu, 19 Oct 2017 18:13:35 +0300
      	# slapuser14@sd-112617.dedibox.fr (2001:67c:1254:e:8c::1 (+ 20·ipv6) 163.172.70.8 (+ 20·ipv4))
      	# Linux vifibcloud-onlinenet-hosting-004 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux
      	# cpu: Intel(R) Xeon(R) CPU D-1531 @ 2.20GHz
      	# cpu[0-11]: freq: intel_pstate/powersave [.80GHz - 2.70GHz]
      	# cpu[0-11]: idle: acpi_idle/menu: POLL(0μs) C1(1μs) C2(41μs)
      	# cpu: WARNING: frequency not fixed - benchmark timings won't be stable
      	# cpu: WARNING: C-state exit-latency is max 41μs - up to that can add to networked and IPC request-reply latency
      	# md1 (raid0) -> sda3  sdb3
      	# sda: SAMSUNG MZ7LN256  rev 100Q 238.5G
      	# sdb: SAMSUNG MZ7LN256  rev 100Q 238.5G
      	# eth0: Intel Corporation I350 Gigabit Network Connection rev 01
      	# eth0: features: rx tx sg tso !ufo gso gro !lro rxvlan txvlan !ntuple rxhash ...
      	# eth0: coalesce: rxc: 3μs/0f/0μs-irq/0f-irq,  txc: 0μs/0f/0μs-irq/0f-irq
      	# eth0: up, speed=1000, mtu=1500, txqlen=1000, !gro_flush_timeout
      	# eth1: Intel Corporation I350 Gigabit Network Connection rev 01
      	# eth1: features: rx tx sg tso !ufo gso gro !lro rxvlan txvlan !ntuple rxhash ...
      	# eth1: coalesce: rxc: 3μs/0f/0μs-irq/0f-irq,  txc: 0μs/0f/0μs-irq/0f-irq
      	# eth1: down, speed=?, mtu=1500, txqlen=1000, !gro_flush_timeout
      	# Python 2.7.14
      	# go version go1.9.1 linux/amd64
      	# sqlite 3.19.0 (py mod 2.6.0)
      	# mysqld  Ver 10.1.28-MariaDB for Linux on x86_64 (MariaDB Server)
      	# neo             : v1.8-1326-g4d0cd894
      	# zodb            : 4.4.5
      	# zeo             : 4.3.1
      	# mysqlclient     : 1.3.12
      	# wendelin.core   : v0.11-4-g38fbc83
      
      slapuser14@vifibcloud-onlinenet-hosting-004:~/srv/runner/t$ time ../software/4db21ec948dce895b38ba93c1def3ab1/bin/neotest bench-local 2>&1 |tee 1.txt
      ...
      *** FileStorage
      Benchmarkvifibcloud-onlinenet-hosting-004/fs1/zhash.py 1 24.4 µs/object # crc32:1552c530  oid=0..2127  nread=8534126  t=0.052s  # POLL·0 C1·49 C2·853
      Benchmarkvifibcloud-onlinenet-hosting-004/fs1/zhash.py 1 24.6 µs/object # crc32:1552c530  oid=0..2127  nread=8534126  t=0.052s  # POLL·0 C1·63 C2·894
      Benchmarkvifibcloud-onlinenet-hosting-004/fs1/zhash.py 1 20.7 µs/object # crc32:1552c530  oid=0..2127  nread=8534126  t=0.044s  # POLL·0 C1·39 C2·803
      Benchmarkvifibcloud-onlinenet-hosting-004/fs1/zhash.py 1 20.8 µs/object # crc32:1552c530  oid=0..2127  nread=8534126  t=0.044s  # POLL·1 C1·18 C2·629
      Benchmarkvifibcloud-onlinenet-hosting-004/fs1/zhash.py 1 20.7 µs/object # crc32:1552c530  oid=0..2127  nread=8534126  t=0.044s  # POLL·0 C1·12 C2·669
      ...
      31fe231f
  5. 24 Oct, 2017 8 commits
  6. 23 Oct, 2017 18 commits
    • eteri's avatar
      9e215760
    • eteri's avatar
      586f076c
    • eteri's avatar
      caddy: remove extra empty lines from software.cfg · 68b4afdf
      eteri authored
      68b4afdf
    • eteri's avatar
      caddy: add port number in connection information · d45a5921
      eteri authored
      d45a5921
    • eteri's avatar
      caddy: add dummy index.html file · 7026461b
      eteri authored
      7026461b
    • eteri's avatar
      caddy: improve configuration · 05a90458
      eteri authored
      05a90458
    • eteri's avatar
      caddy: improve caddy configuration · e25c9fe2
      eteri authored
      e25c9fe2
    • Gabriel Monnerat's avatar
      caddy: install caddy dependencies · 6a196a11
      Gabriel Monnerat authored
      6a196a11
    • Gabriel Monnerat's avatar
      caddy: clean up · 6d7df7d6
      Gabriel Monnerat authored
      6d7df7d6
    • Gabriel Monnerat's avatar
      caddy: compile golang 1.9 and install Caddy · efe54803
      Gabriel Monnerat authored
      WIP
      efe54803
    • Kirill Smelkov's avatar
      lmbench: New component to measure many performance characteristics · 88f70453
      Kirill Smelkov authored
      lmbench is a bit outdated tool to do OS performance analysis. The
      upcoming neotest SR will use lat_tcp from this packages to measure TCP
      round-trip-time latency.
      
      We use a bit patched version with fixes to lat_tcp for errors not to go
      unnoticed and also lat_tcp.go added to see how using Go compares to
      plain lat_tcp.c for latencies.
      
      In the upcoming neotest SR both lat_tcp C & Go versions will be used to
      automatically benchmark TCP round-trip-times over network link in between
      2 machines.
      88f70453
    • Kirill Smelkov's avatar
      ioping: New component to measure disk IO latency · 35a58508
      Kirill Smelkov authored
      This tool can be used to measure disk IO latency for various workloads
      (random read, sequential read, direct or cached, write, ...).
      
      We use a bit patched version shows not only avg latency but also its
      distribution. (the patches are currently dirty/not very robust and not
      yet sent to upstream).
      
      In the upcoming neotest SR ioping will be used to automatically
      benchmark underlying disk characteristics.
      35a58508
    • Kirill Smelkov's avatar
      ethtool: New component to query and control NICs driver and hardware settings · a79c0856
      Kirill Smelkov authored
      ethtool can be used to adjust NIC settings, such as e.g. interrupt
      coalescing and various others. It usually requires root to do so.
      
      However regular users can query NIC settings via ethtool just ok.
      
      This way upcoming neotest SR will use it to query/display NIC setting
      which are essential to get good networked latency and automatically
      produce warnings if it is not.
      a79c0856
    • Kirill Smelkov's avatar
      v↑ zodbtools (0.0.0.dev4) · 9b5d8262
      Kirill Smelkov authored
      Starting from this version zodbtools requires zodburi to open ZODB
      storages by URL (see zodbtools@db3b84ec
      and zodbtools@82b06413)
      
      zodburi in turn requires ZEO (maybe we might patch it to only optionally
      depend on it), so we move ZEO requirement from stack/ERP5 to the place
      where it is actually first needed.
      9b5d8262
    • Kirill Smelkov's avatar
      golang: Infrastructure to build Go workspaces / projects · 1b540151
      Kirill Smelkov authored
      In Go world development workflow is organized around so-called workspace
      where multiple packages can be installed / worked on etc. The following
      page describes Go workspaces:
      
      https://golang.org/doc/code.html
      
      A new [gowork] section and infrastructure around it is introduced.
      Quoting code:
      
          # gowork is a top-level section representing workspace
          #
          # users should add `install` field to [gowork] to describe packages they want to
          # be installed (+ automatically their dependencies are installed too). e.g.
          #
          #   [gowork]
          #   install =
          #       lab.nexedi.com/kirr/neo/go/...  \
          #       github.com/pkg/profile          \
          #       golang.org/x/perf/cmd/benchstat
      
      The way it all works inside is:
      
      - gowork organizes to create go.work/ directory in buildout root
      - inside it creates env.sh which when sources in shell adjusts all paths
        so that appropriate go compiler is in path, GOPATH is also set
        appropriately, etc - in other words everything needs for using/working
        on this workspace is setup in the environment.
      - in actual user software profile the list of Go projects which needs to
        be installed by SlapOS has to be listed. The [gowork] machinery takes
        care about git-cloning these projects and `go install`s them after.
      - by SlapOS design builds have to be reproducible and so every component
        state/version has to be fixed.
      
        A convenient helper (gowork-snapshot) to get snapshot of git go
        packages installed with their exact revisions is added. This should be
        used to automatically (re-)generate list of go git repositories &
        their pinning for a workspace.
      
      For a new Go project that needs to be slaposified the workflow should be:
      
      - first get a project into working state via any convenient way (`go get`, etc.)
      - generate list of go packages & their pinning with gowork-snapshot
      - extend golang/buildout.cfg and in project software profile add
      
      	[gowork]
        	install = your-top-level-packages
      
        as it was outlined above.
      
      P.S.
      
      [golang] is removed because
      
      1. it is not used anywhere in slapos.git tree, and
      2. the canonical way to work on go projects from now on will be to use
         [gowork]. There latest go version is preconfigured, but users can
         override it to some particular other version in their projects as they
         need.
      
      /reviewed-on	 !242
      /also-needed-for !243
      1b540151
    • Rafael Monnerat's avatar
      Revert "Caddy 0.1" · 4e24a566
      Rafael Monnerat authored
      This reverts merge request !243
      4e24a566
    • Eteri's avatar
      Caddy 0.1 · 471bcbee
      Eteri authored
      The first proof of concept  of Caddy in Slapos. 
      
      For now, everything is inside software/caddy but in the next merge request we will move to stack. 
      
      @rafael @romain 
      
      
      /reviewed-on !243
      471bcbee
    • eteri's avatar
      8121bc33
  7. 20 Oct, 2017 5 commits