1. 23 Sep, 2015 2 commits
    • Kirill Smelkov's avatar
      lib/mem: Allow memcpy destination to be bigger than source · 520cbc6f
      Kirill Smelkov authored
      i.e. it is ok to copy smaller data into larger buffer.
      520cbc6f
    • Kirill Smelkov's avatar
      lib/mem: Allow memcpy & friends to work on arbitrary-length buffer · a35106c2
      Kirill Smelkov authored
      - not only multiple of 8. We can do it by using uint8 typed arrays, and
      it does not hurt performance:
      
      In [1]: from wendelin.lib.mem import bzero, memset, memcpy
      In [2]: A = bytearray(2*1024*1024)
      In [3]: B = bytearray(2*1024*1024)
      
              memcpy(B, A)    bzero(A)        memset(A, 0xff)
      
      old:    718 µs          227 µs / 1116   228 µs / 1055 (*)
      new:    718 µs          176 µs / 1080   175 µs / 1048
      
          (*) the second number comes from e.g.
      
              In [8]: timeit bzero(A)
              The slowest run took 4.63 times longer than the fastest.
              This could mean that an intermediate result is being cached
              10000 loops, best of 3: 228 µs per loop
      
              so the second number is more realistic and says performance
              stays aproximately the same and only slightly improves.
      a35106c2
  2. 06 Aug, 2015 4 commits
  3. 26 Jun, 2015 1 commit
    • Kirill Smelkov's avatar
      tests: Allow to test with ZEO & NEO ZODB storages · 7fc4ec66
      Kirill Smelkov authored
      Previously we were always testing with DBs backed up by FileStorage. Now
      we provide a way to run the testsuite with user selected storage
      backend:
      
          $ WENDELIN_CORE_TEST_DB="<fs>"   make test.py     # test with temporary db with FileStorage
          $ WENDELIN_CORE_TEST_DB="<zeo>"  make test.py     # ----------//---------- with ZEO
          $ WENDELIN_CORE_TEST_DB="<neo>"  make test.py     # ----------//---------- with NEO
      
          $ WENDELIN_CORE_TEST_DB=neo://db@master  make test.py     # test with externally provided DB
      
      Default is still to run tests with FileStorage.
      
      /cc @jm
      7fc4ec66
  4. 25 Jun, 2015 2 commits
  5. 02 Jun, 2015 1 commit
    • Kirill Smelkov's avatar
      *: It is not safe to use multiply.reduce() - it overflows · 73926487
      Kirill Smelkov authored
      e.g.
      
          In [1]: multiply.reduce((1<<30, 1<<30, 1<<30))
          Out[1]: 0
      
      instead of
      
          In [2]: (1<<30) * (1<<30) * (1<<30)
          Out[2]: 1237940039285380274899124224
      
          In [3]: 1<<90
          Out[3]: 1237940039285380274899124224
      
      also multiply.reduce returns int64, instead of python int:
      
          In [4]: type( multiply.reduce([1,2,3]) )
          Out[4]: numpy.int64
      
      which also leads to overflow-related problems if we further compute with
      this value and other integers and results exceeds int64 - it becomes
      float:
      
          In [5]: idx0_stop = 18446744073709551615
      
          In [6]: stride0   = numpy.int64(1)
      
          In [7]: byte0_stop = idx0_stop * stride0
      
          In [8]: byte0_stop
          Out[8]: 1.8446744073709552e+19
      
      and then it becomes a real problem for BigArray.__getitem__()
      
          wendelin.core/bigarray/__init__.py:326: RuntimeWarning: overflow encountered in long_scalars
            page0_min  = min(byte0_start, byte0_stop+byte0_stride) // pagesize # TODO -> fileh.pagesize
      
      and then
      
          >           vma0 = self._fileh.mmap(page0_min, page0_max-page0_min+1)
          E           TypeError: integer argument expected, got float
      
      ~~~~
      
      So just avoid multiple.reduce() and do our own mul() properly the same
      way sum() is builtin into python, and we avoid overflow-related
      problems.
      73926487
  6. 03 Apr, 2015 4 commits
    • Kirill Smelkov's avatar
      bigfile: Basic benchmarks · bb9d8bf1
      Kirill Smelkov authored
          - for virtual memory subsytem
          - for ZBigFiles
      
      They are not currently great, e.g. for virtmem we have in-kernel
      overhead of page clearing - in perf profiles, for bigfile_mmap compared
      to file_read kernel's clear_page_c raises significantly.
      
      That is the worker for clearing page memory and we currently cannot
      avoid that - any memory obtained from kernel (MAP_ANONYMOUS, mmap(file)
      with hole, etc...) comes pre-initialized to zeros to userspace.
      
      This can be seen in the benchmarks as well: file_readbig differs from
      file_read in only that the latter uses 1 small buffer and the first
      allocates large memory (cleared by kernel + python does the memset).
      
          bigfile/tests/bench_virtmem.py@125::bench_file_mmap_adler32     0.47  (0.86 0.49 0.47)
          bigfile/tests/bench_virtmem.py@126::bench_file_read_adler32     0.69  (1.11 0.71 0.69)
          bigfile/tests/bench_virtmem.py@127::bench_file_readbig_adler32  1.41  (1.70 1.42 1.41)
          bigfile/tests/bench_virtmem.py@128::bench_bigfile_mmap_adler32  1.42  (1.45 1.42 1.51)
      
          bigfile/tests/bench_virtmem.py@130::bench_file_mmap_md5         1.52  (1.91 1.54 1.52)
          bigfile/tests/bench_virtmem.py@131::bench_file_read_md5         1.73  (2.10 1.75 1.73)
          bigfile/tests/bench_virtmem.py@132::bench_file_readbig_md5      2.44  (2.73 2.46 2.44)
          bigfile/tests/bench_virtmem.py@133::bench_bigfile_mmap_md5      2.40  (2.48 2.40 2.53)
      
      There is MAP_UNINITIALIZED which works only for non-mmu targets and only
      if explicitly allowed when configuring kernel (off by default).
      
      There were patches to disable that pages zeroing, as it gives
      significant speedup for people's workloads, e.g. [1,2] but all of them
      did not got merged for security reasons.
      
      [1] http://marc.info/?t=132691315900001&r=1&w=2
      [2] http://thread.gmane.org/gmane.linux.kernel/548926
      
      ~~~~
      
      For ZBigFile - it is the storage who is dominating in profiles.
      bb9d8bf1
    • Kirill Smelkov's avatar
      lib/mem: Python utilities to zero, set & copy memory · 699b1375
      Kirill Smelkov authored
      Like C bzero / memset & memcopy - but work on python buffers. We
      leverage NumPy for doing actual work, and this way NumPy becomes a
      depenency.
      
      Having NumPy as a dependency is ok - we'll for sure need it later as we
      are trying to build out-of-core ndarrays.
      699b1375
    • Kirill Smelkov's avatar
      lib/utils: Small C utilities we'll use · 3e5e78cd
      Kirill Smelkov authored
      Like taking an exact integer log2, upcasting pointers for C-style
      inheritance done in a Plan9 way, and wrappers to functions which should
      never fail.
      3e5e78cd
    • Kirill Smelkov's avatar
      lib/bug: Small utilities to say there is a warning, or a todo, or a bug on condition · db28e53f
      Kirill Smelkov authored
      Modelled by ones used in Linux kernel.
      db28e53f