Commit 01916f09 authored by Kirill Smelkov's avatar Kirill Smelkov

X Draft demo that reading data through wcfs works

and gives exactly the same data as non-wcfs Wendelin.core:

	---- 8< ----
	(neo) (z-dev) (g.env) kirr@deco:~/src/wendelin/wendelin.core$ free -h
	              total        used        free      shared  buff/cache   available
	Mem:          7,5Gi       931Mi       613Mi       194Mi       6,0Gi       6,1Gi
	Swap:            0B          0B          0B

	(neo) (z-dev) (g.env) kirr@deco:~/src/wendelin/wendelin.core$ time ./demo/demo_zbigarray.py gen 1.fs
	I: RAM:  7.47GB
	I: WORK: 14.94GB
	gen signal t=0...2.00e+09  float64  (= 14.94GB)
	gen signal blk [0:4194304]  (0.2%)
	gen signal blk [4194304:8388608]  (0.4%)
	gen signal blk [8388608:12582912]  (0.6%)
	gen signal blk [12582912:16777216]  (0.8%)
	gen signal blk [16777216:20971520]  (1.0%)
	gen signal blk [20971520:25165824]  (1.3%)
	...
	gen signal blk [1988100096:1992294400]  (99.4%)
	gen signal blk [1992294400:1996488704]  (99.6%)
	gen signal blk [1996488704:2000683008]  (99.8%)
	gen signal blk [2000683008:2004649984]  (100.0%)
	VIRT: 457 MB    RSS: 259MB

	real    7m51,814s
	user    2m19,001s
	sys     0m42,615s

	(neo) (z-dev) (g.env) kirr@deco:~/src/wendelin/wendelin.core$ time ./demo/demo_zbigarray.py read 1.fs
	I: RAM:  7.47GB
	sig: 2004649984 float64 (= 14.94GB)
	<sig>:  2.3794727102747662e-08
	S(sig): 47.70009930580747
	VIRT: 245 MB    RSS: 49MB

	real    2m36,006s
	user    0m14,773s
	sys     0m59,467s

	(neo) (z-dev) (g.env) kirr@deco:~/src/wendelin/wendelin.core$ time WENDELIN_CORE_VIRTMEM=r:wcfs+w:uvmm ./demo/demo_zbigarray.py read 1.fs
	I: RAM:  7.47GB
	sig: 2004649984 float64 (= 14.94GB)
	wcfs: 2019/02/22 21:33:48 zodb: FIXME: open file:///home/kirr/src/wendelin/wendelin.core/1.fs: cache is not ready for invalidations -> NoCache forced
	db.openx 03cdd855e0d73622 nopool=true   ; δtail (03cdd855e0d73622, 03cdd855e0d73622]
	db.openx 03cdd855e0d73622 nopool=false  ; δtail (03cdd855e0d73622, 03cdd855e0d73622]
	W0222 21:35:20.282163    6697 misc.go:84] /: lookup "lib": invalid argument: not @rev
	W0222 21:35:20.334896    6697 misc.go:84] /: lookup "libX11.so": invalid argument: not @rev
	W0222 21:35:20.340128    6697 misc.go:84] /: lookup "libX11.so.so": invalid argument: not @rev
	W0222 21:35:20.342492    6697 misc.go:84] /: lookup "libX11.so.la": invalid argument: not @rev
	<sig>:  2.3794727102747662e-08
	S(sig): 47.70009930580747
	VIRT: 371 MB    RSS: 37MB

	real    6m8,611s
	user    0m10,167s
	sys     0m21,964s
	---- 8< ----

Wcfs was not yet optimized at all. Offhand `perf top` was showing lots of time
is spent in garbage collector, but maybe something to also debug on FUSE-level latencies.
In any way FileStorage case is not very representative and as for ZEO/NEO case
there are network requests to be made and they start to dominate the latency to
access one page/object.
parent 13ee0416
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment