X Draft demo that reading data through wcfs works
and gives exactly the same data as non-wcfs Wendelin.core: ---- 8< ---- (neo) (z-dev) (g.env) kirr@deco:~/src/wendelin/wendelin.core$ free -h total used free shared buff/cache available Mem: 7,5Gi 931Mi 613Mi 194Mi 6,0Gi 6,1Gi Swap: 0B 0B 0B (neo) (z-dev) (g.env) kirr@deco:~/src/wendelin/wendelin.core$ time ./demo/demo_zbigarray.py gen 1.fs I: RAM: 7.47GB I: WORK: 14.94GB gen signal t=0...2.00e+09 float64 (= 14.94GB) gen signal blk [0:4194304] (0.2%) gen signal blk [4194304:8388608] (0.4%) gen signal blk [8388608:12582912] (0.6%) gen signal blk [12582912:16777216] (0.8%) gen signal blk [16777216:20971520] (1.0%) gen signal blk [20971520:25165824] (1.3%) ... gen signal blk [1988100096:1992294400] (99.4%) gen signal blk [1992294400:1996488704] (99.6%) gen signal blk [1996488704:2000683008] (99.8%) gen signal blk [2000683008:2004649984] (100.0%) VIRT: 457 MB RSS: 259MB real 7m51,814s user 2m19,001s sys 0m42,615s (neo) (z-dev) (g.env) kirr@deco:~/src/wendelin/wendelin.core$ time ./demo/demo_zbigarray.py read 1.fs I: RAM: 7.47GB sig: 2004649984 float64 (= 14.94GB) <sig>: 2.3794727102747662e-08 S(sig): 47.70009930580747 VIRT: 245 MB RSS: 49MB real 2m36,006s user 0m14,773s sys 0m59,467s (neo) (z-dev) (g.env) kirr@deco:~/src/wendelin/wendelin.core$ time WENDELIN_CORE_VIRTMEM=r:wcfs+w:uvmm ./demo/demo_zbigarray.py read 1.fs I: RAM: 7.47GB sig: 2004649984 float64 (= 14.94GB) wcfs: 2019/02/22 21:33:48 zodb: FIXME: open file:///home/kirr/src/wendelin/wendelin.core/1.fs: cache is not ready for invalidations -> NoCache forced db.openx 03cdd855e0d73622 nopool=true ; δtail (03cdd855e0d73622, 03cdd855e0d73622] db.openx 03cdd855e0d73622 nopool=false ; δtail (03cdd855e0d73622, 03cdd855e0d73622] W0222 21:35:20.282163 6697 misc.go:84] /: lookup "lib": invalid argument: not @rev W0222 21:35:20.334896 6697 misc.go:84] /: lookup "libX11.so": invalid argument: not @rev W0222 21:35:20.340128 6697 misc.go:84] /: lookup "libX11.so.so": invalid argument: not @rev W0222 21:35:20.342492 6697 misc.go:84] /: lookup "libX11.so.la": invalid argument: not @rev <sig>: 2.3794727102747662e-08 S(sig): 47.70009930580747 VIRT: 371 MB RSS: 37MB real 6m8,611s user 0m10,167s sys 0m21,964s ---- 8< ---- Wcfs was not yet optimized at all. Offhand `perf top` was showing lots of time is spent in garbage collector, but maybe something to also debug on FUSE-level latencies. In any way FileStorage case is not very representative and as for ZEO/NEO case there are network requests to be made and they start to dominate the latency to access one page/object.
Showing