• Kirill Smelkov's avatar
    go/neo/t/neotest: ZODB benchmarks · 3f578560
    Kirill Smelkov authored
    Add to neotest zbench-local and zbench-cluster commands that perform
    ZODB benchmarks on FileStorage, ZEO and NEO with Python and Go clients
    either locally, or with a server and client running on 2 different nodes.
    
    There are 2 client programs: tzodb.py and tzodb.go which for now compute
    hash of whole latest objects stream in a ZODB database. On server side
    neotest is taught to launch ZEO and various NEO clusters and to execute
    client load on them.
    
    Two test datasets are used: wczblk1-8 - the dataset with wendelin.core ZBlk1
    objects covering 8M array, and prod1-1024 - synthethic dataset that tries to
    represent regular ERP5 instance. Both datasets are very small and so we can
    assume they reside completely in server disk cache while running benchmarks.
    Benchmark timings will thus give pure storage software processing latency, as
    pagecache hit time is on par, or less, to 1µs.
    
    Example output:
    
    	x/src/lab.nexedi.com/kirr/neo/go/neo/t$ ./neotest zbench-local
    	dataset:	wczblk1-8
    	node:
    	cluster:	deco
    
    	*** generating fs1 data...
    	I: RAM:  7.47GB
    	I: WORK: 0.01GB
    	gen signal t=0...1.05e+06  float64  (= 0.01GB)
    	gen signal blk [0:1048576]  (100.0%)
    	VIRT: 297 MB	RSS: 48MB
    
    	*** generating sqlite data...
    	I: RAM:  7.47GB
    	I: WORK: 0.01GB
    	gen signal t=0...1.05e+06  float64  (= 0.01GB)
    	gen signal blk [0:1048576]  (100.0%)
    	VIRT: 386 MB	RSS: 58MB
    	2018-07-10 19:57:35.7065 ERROR     NEO        [           app: 91] primary master is down
    	Cluster state changed
    
    	*** generating sql data...
    	2018-07-10 19:57:35 140115116649600 [Note] /usr/sbin/mysqld (mysqld 10.1.29-MariaDB-6+b1) starting as process 27574 ...
    	2018-07-10 19:57:39 140205509999744 [Note] /usr/sbin/mysqld (mysqld 10.1.29-MariaDB-6+b1) starting as process 27603 ...
    	2018-07-10 19:57:42 139692109810816 [Note] /usr/sbin/mysqld (mysqld 10.1.29-MariaDB-6+b1) starting as process 27633 ...
    	2018-07-10 19:57:45 139759221546112 [Note] mysqld (mysqld 10.1.29-MariaDB-6+b1) starting as process 27662 ...
    	I: RAM:  7.47GB
    	I: WORK: 0.01GB
    	gen signal t=0...1.05e+06  float64  (= 0.01GB)
    	gen signal blk [0:1048576]  (100.0%)
    	VIRT: 387 MB	RSS: 59MB
    	2018-07-10 19:57:48.2565 ERROR     NEO        [           app: 91] primary master is down
    	Cluster state changed
    
    	*** FileStorage
    	Benchmarkfs1-zhash.py 2127 16.3 µs/object	# crc32:14640593  nread=8540363  t=0.035s	# POLL·2 C1·73 C1E·38 C3·12 C6·36 C7s·0 C8·112 C9·0 C10·62
    	Benchmarkfs1-zhash.py 2127 16.6 µs/object	# crc32:14640593  nread=8540363  t=0.035s	# POLL·0 C1·113 C1E·21 C3·16 C6·56 C7s·0 C8·136 C9·0 C10·41
    	Benchmarkfs1-zhash.py 2127 15.9 µs/object	# crc32:14640593  nread=8540363  t=0.034s	# POLL·0 C1·71 C1E·36 C3·22 C6·50 C7s·0 C8·167 C9·0 C10·47
    	Benchmarkfs1-zhash.py 2127 15.9 µs/object	# crc32:14640593  nread=8540363  t=0.034s	# POLL·0 C1·77 C1E·32 C3·11 C6·55 C7s·0 C8·184 C9·0 C10·31
    	Benchmarkfs1-zhash.py 2127 16.0 µs/object	# crc32:14640593  nread=8540363  t=0.034s	# POLL·0 C1·78 C1E·15 C3·12 C6·51 C7s·0 C8·140 C9·0 C10·44
    
    	# 16 clients in parallel
    	Benchmarkfs1-zhash.py·P16 2127 129.0 µs/object	# crc32:14640593  nread=8540363  t=0.274s
    	Benchmarkfs1-zhash.py·P16 2127 132.6 µs/object	# crc32:14640593  nread=8540363  t=0.282s
    	Benchmarkfs1-zhash.py·P16 2127 135.0 µs/object	# crc32:14640593  nread=8540363  t=0.287s
    	Benchmarkfs1-zhash.py·P16 2127 135.3 µs/object	# crc32:14640593  nread=8540363  t=0.288s
    	Benchmarkfs1-zhash.py·P16 2127 136.6 µs/object	# crc32:14640593  nread=8540363  t=0.291s
    	Benchmarkfs1-zhash.py·P16 2127 122.8 µs/object	# crc32:14640593  nread=8540363  t=0.261s
    	Benchmarkfs1-zhash.py·P16 2127 130.9 µs/object	# crc32:14640593  nread=8540363  t=0.279s
    	Benchmarkfs1-zhash.py·P16 2127 126.4 µs/object	# crc32:14640593  nread=8540363  t=0.269s
    	Benchmarkfs1-zhash.py·P16 2127 125.8 µs/object	# crc32:14640593  nread=8540363  t=0.268s
    	Benchmarkfs1-zhash.py·P16 2127 108.3 µs/object	# crc32:14640593  nread=8540363  t=0.230s
    	Benchmarkfs1-zhash.py·P16 2127 131.0 µs/object	# crc32:14640593  nread=8540363  t=0.279s
    	Benchmarkfs1-zhash.py·P16 2127 124.1 µs/object	# crc32:14640593  nread=8540363  t=0.264s
    	Benchmarkfs1-zhash.py·P16 2127 129.3 µs/object	# crc32:14640593  nread=8540363  t=0.275s
    	Benchmarkfs1-zhash.py·P16 2127 125.0 µs/object	# crc32:14640593  nread=8540363  t=0.266s
    	Benchmarkfs1-zhash.py·P16 2127 131.5 µs/object	# crc32:14640593  nread=8540363  t=0.280s
    	Benchmarkfs1-zhash.py·P16 2127 131.4 µs/object	# crc32:14640593  nread=8540363  t=0.280s
    	# POLL·0 C1·4 C1E·13 C3·11 C6·79 C7s·0 C8·14 C9·0 C10·0
    
    	...
    
    And its summary via benchstat:
    
    	x/src/lab.nexedi.com/kirr/neo/go/neo/t$ benchstat -split node,cluster,dataset x.log
    	name                                 time/object
    	cluster:deco dataset:wczblk1-8
    	fs1-zhash.py                         16.1µs ± 3%
    	fs1-zhash.py·P16                      130µs ± 5%
    	fs1-zhash.go                         3.00µs ±10%
    	fs1-zhash.go+prefetch128             3.40µs ±18%
    	fs1-zhash.go·P16                     10.2µs ±71%
    	zeo/py/fs1-zhash.py                   336µs ± 3%
    	zeo/py/fs1-zhash.py·P16              3.22ms ± 6%
    	zeo/py/fs1-zhash.go                   112µs ± 2%
    	zeo/py/fs1-zhash.go+prefetch128      60.9µs ± 1%
    	zeo/py/fs1-zhash.go·P16              1.07ms ± 5%
    	neo/py(!log)/sqlite·P1-zhash.py       291µs ± 2%
    	neo/py(!log)/sqlite·P1-zhash.py·P16  2.86ms ± 1%
    	neo/py(!log)/sql·P1-zhash.py          318µs ± 4%
    	neo/py(!log)/sql·P1-zhash.py·P16     3.99ms ± 0%
    	cluster:deco dataset:prod1-1024
    	fs1-zhash.py                         12.3µs ± 1%
    	fs1-zhash.py·P16                      106µs ±10%
    	fs1-zhash.go                         2.56µs ±10%
    	fs1-zhash.go+prefetch128             2.68µs ± 8%
    	fs1-zhash.go·P16                     9.48µs ±43%
    	zeo/py/fs1-zhash.py                   319µs ± 3%
    	zeo/py/fs1-zhash.py·P16              3.13ms ± 3%
    	zeo/py/fs1-zhash.go                   101µs ± 5%
    	zeo/py/fs1-zhash.go+prefetch128      56.9µs ± 1%
    	zeo/py/fs1-zhash.go·P16              1.19ms ± 4%
    	neo/py(!log)/sqlite·P1-zhash.py       281µs ± 3%
    	neo/py(!log)/sqlite·P1-zhash.py·P16  2.80ms ± 1%
    	neo/py(!log)/sql·P1-zhash.py          316µs ± 1%
    	neo/py(!log)/sql·P1-zhash.py·P16     3.91ms ± 1%
    
    Since there is no NEO/go support yet, corresponding neotest parts are merged,
    but commented-out with appropriate remark.
    
    Parallel access is simulated with spawning many OS processes for now.
    This will change in the nearby followup patch to zwrk.
    
    Results of ZODB benchmarking were discussed in
    
    	http://navytux.spb.ru/~kirr/neo.html#performance-tests		, and
    	http://navytux.spb.ru/~kirr/neo.html#results-and-discussion
    
    Some draft history related to this patch:
    
    	lab.nexedi.com/kirr/neo/commit/e0d875bc	X neotest: Teach it to benchmark NEO with storage partitioned to several nodes
    	lab.nexedi.com/kirr/neo/commit/590f0a46	X neo/py uses n(replica) as n(real-replica) - 1
    	lab.nexedi.com/kirr/neo/commit/b655da26	X save time not benchmarking things we do not show
    	lab.nexedi.com/kirr/neo/commit/f834f40d	X zhash: Show N(obj) read, not 1, in place of N(iter)
    	lab.nexedi.com/kirr/neo/commit/a16e8d52	X teach golang to access ZEO
    	lab.nexedi.com/kirr/neo/commit/b9827725	X switch to using no compression, because this way it is more fair for comparing storage latencies
    	lab.nexedi.com/kirr/neo/commit/c0067335	X neotest: Don't depend on killall
    	lab.nexedi.com/kirr/neo/commit/2bcd6ebb	X neotest: add zbench-local & zbench-cluster subcomands
    	lab.nexedi.com/kirr/neo/commit/fb165ad9	X neotest: Also benchmark NEO/py with logging disabled
    	lab.nexedi.com/kirr/neo/commit/2118ba38	X neotest: Help mysqlk_install_db find its basedir under SlapOS
    	lab.nexedi.com/kirr/neo/commit/80eaa05e	X zgenprod1 tool
    	lab.nexedi.com/kirr/neo/commit/eb0e516f	X check hash result and error if mismatch (zhash.* part); neotest part pending
    	lab.nexedi.com/kirr/neo/commit/046370db	X benchify rest of bench-cluster
    	lab.nexedi.com/kirr/neo/commit/2d13818e	X bench-local + zhash: Add output in std bench format
    	lab.nexedi.com/kirr/neo/commit/1d692a3b	X add NEO/go with SHA1 disabled (both Sgo and Cgo to regular benchmarks)
    3f578560
neotest 39 KB