1. 08 Jul, 2019 1 commit
  2. 04 Jul, 2018 1 commit
    • Kirill Smelkov's avatar
      py.bench: Move it to -> pygolang · 318efce0
      Kirill Smelkov authored
      So that it can be available to everyone and in particular B & friends to
      be available from introduced importable golang.testing package. The move
      itself:
      
      	kirr/pygolang@9bf03d9c
      
      While moving the code was restructured / improved a bit and py.bench
      interface reworked to mimic `go test -bench` in defaults.
      318efce0
  3. 12 Dec, 2017 1 commit
    • Kirill Smelkov's avatar
      virtmem: Benchmarks for pagefault handling · 3cfc2728
      Kirill Smelkov authored
      Benchmark the time it takes for virtmem to handle pagefault with noop
      loadblk for loadblk both implemented in C and in Python.
      
      On my computer it is:
      
      	name          µs/op
      	PagefaultC    269 ± 0%
      	pagefault_py  291 ± 0%
      
      Quite a big time in other words.
      
      It turned out to be mostly spent in fallocate'ing pages on tmpfs from
      /dev/shm. Part of the above 269 µs/op is taken by freeing (reclaiming)
      pages back when benchmarking work size exceed /dev/shm size, and part to
      allocating.
      
      If I limit the work size (via npage in benchmem.c) to be less than whole
      /dev/shm it starts to be ~ 170 µs/op and with additional tracing it
      shows as something like this:
      
          	.. on_pagefault_start   0.954 µs
          	.. vma_on_pagefault_pre 0.954 µs
          	.. ramh_alloc_page_pre  0.954 µs
          	.. ramh_alloc_page      169.992 µs
          	.. vma_on_pagefault     172.853 µs
          	.. vma_on_pagefault_pre 172.853 µs
          	.. vma_on_pagefault     174.046 µs
          	.. on_pagefault_end     174.046 µs
          	.. whole:               171.900 µs
      
      so almost all time is spent in ramh_alloc_page which is doing the fallocate:
      
      	https://lab.nexedi.com/nexedi/wendelin.core/blob/f11386a4/bigfile/ram_shmfs.c#L125
      
      Simple benchmark[1] confirmed it is indeed the case for fallocate(tmpfs) to be
      relatively slow[2] (and that for recent kernels it regressed somewhat
      compared to Linux 3.16). Profile flamegraph for that benchmark[3] shows
      internal loading of shmem_fallocate which for 1 hardware page is not
      that too slow (e.g. <1µs) but when a request comes for a region
      internally performs it page by page and so accumulates that ~ 170µs for 2M.
      
      I've tried to briefly rerun the benchmark with huge pages activated on /dev/shm via
      
      	mount /dev/shm -o huge=always,remount
      
      as both regular user and as root but it was executing several times
      slower. Probably something to investigate more later.
      
      [1] https://lab.nexedi.com/kirr/misc/blob/4f84a06e/tmpfs/t_fallocate.c
      [2] https://lab.nexedi.com/kirr/misc/blob/4f84a06e/tmpfs/1.txt
      [3] https://lab.nexedi.com/kirr/misc/raw/4f84a06e/tmpfs/fallocate-2M-nohuge.svg
      3cfc2728
  4. 24 Oct, 2017 1 commit
    • Kirill Smelkov's avatar
      Relicense to GPLv3+ with wide exception for all Free Software / Open Source... · f11386a4
      Kirill Smelkov authored
      Relicense to GPLv3+ with wide exception for all Free Software / Open Source projects + Business options.
      
      Nexedi stack is licensed under Free Software licenses with various exceptions
      that cover three business cases:
      
      - Free Software
      - Proprietary Software
      - Rebranding
      
      As long as one intends to develop Free Software based on Nexedi stack, no
      license cost is involved. Developing proprietary software based on Nexedi stack
      may require a proprietary exception license. Rebranding Nexedi stack is
      prohibited unless rebranding license is acquired.
      
      Through this licensing approach, Nexedi expects to encourage Free Software
      development without restrictions and at the same time create a framework for
      proprietary software to contribute to the long term sustainability of the
      Nexedi stack.
      
      Please see https://www.nexedi.com/licensing for details, rationale and options.
      f11386a4
  5. 06 Aug, 2015 1 commit
    • Kirill Smelkov's avatar
      bigfile/virtmem: Big Virtmem lock · d53271b9
      Kirill Smelkov authored
      At present several threads running can corrupt internal virtmem
      datastructures (e.g. ram->lru_list, fileh->pagemap, etc).
      
      This can happen even if we have zope instances only with 1 worker thread
      - because there are other "system" thread, and python garbage collection
      can trigger at any thread, so if a virtmem object, e.g. VMA or FileH was
      there sitting at GC queue to be collected, their collection, and thus
      e.g. vma_unmap() and fileh_close() will be called from
      different-from-worker thread.
      
      Because of that virtmem just has to be aware of threads not to allow
      internal datastructure corruption.
      
      On the other hand, the idea of introducing userspace virtual memory
      manager turned out to be not so good from performance and complexity
      point of view, and thus the plan is to try to move it back into the
      kernel. This way it does not make sense to do a well-optimised locking
      implementation for userspace version.
      
      So we do just a simple single "protect-all" big lock for virtmem.
      
      Of a particular note is interaction with Python's GIL - any long-lived
      lock has to be taken with GIL released, because else it can deadlock:
      
          t1  t2
      
          G
          V   G
         !G   V
          G
      
      so we introduce helpers to make sure the GIL is not taken, and to retake
      it back if we were holding it initially.
      
      Those helpers (py_gil_ensure_unlocked / py_gil_retake_if_waslocked) are
      symmetrical opposites to what Python provides to make sure the GIL is
      locked (via PyGILState_Ensure / PyGILState_Release).
      
      Otherwise, the patch is more-or-less straightforward application for
      one-big-lock to protect everything idea.
      d53271b9
  6. 25 May, 2015 1 commit
    • Kirill Smelkov's avatar
      tests: Don't forget to depend on ccan's config.h · dcdb0302
      Kirill Smelkov authored
      Else, in a clean repo the tests do not build from scratch:
      
          $ make test
          x86_64-linux-gnu-gcc -pthread -g -Wall -D_GNU_SOURCE -std=gnu99 -fplan9-extensions -Wno-declaration-after-statement -Wno-error=declaration-after-statement  -Iinclude -I3rdparty/ccan -I3rdparty/include   bigfile/tests/test_virtmem.c lib/bug.c lib/utils.c 3rdparty/ccan/ccan/tap/tap.c  -o bigfile/tests/test_virtmem.t
          In file included from include/wendelin/list.h:11:0,
                           from include/wendelin/bigfile/virtmem.h:33,
                           from bigfile/tests/../virtmem.c:23,
                           from bigfile/tests/test_virtmem.c:21:
          3rdparty/ccan/ccan/array_size/array_size.h:4:20: fatal error: config.h: No such file or directory
           #include "config.h"
                              ^
          compilation terminated.
          3rdparty/ccan/ccan/tap/tap.c:26:20: fatal error: config.h: No such file or directory
           #include "config.h"
                              ^
          compilation terminated.
          Makefile:99: recipe for target 'bigfile/tests/test_virtmem.t' failed
          make: *** [bigfile/tests/test_virtmem.t] Error 1
      dcdb0302
  7. 15 Apr, 2015 1 commit
    • Kirill Smelkov's avatar
      Fix build with Python that enables -Werror=declaration-after-statement · e0b25398
      Kirill Smelkov authored
      In C99 declaration after statement is ok, and we explicitly compile with -std=gnu99.
      
      Python >= 3.4 however adds -Werror=declaration-after-statement even for extension
      modules irregardless of their compilation flags:
      
        https://bugs.python.org/issue21121
      
      and the build fails this way:
      
          building 'wendelin.bigfile._bigfile' extension
          creating build/temp.linux-x86_64-3.4
          creating build/temp.linux-x86_64-3.4/bigfile
          creating build/temp.linux-x86_64-3.4/lib
          gcc -pthread -Wno-unused-result -Werror=declaration-after-statement -DNDEBUG -fmessage-length=0 -grecord-gcc-switches -O2 -Wall
      -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -DOPENSSL_LOAD_CONF -fPIC -D_GNU_SOURCE
      -I./include -I./3rdparty/ccan -I./3rdparty/include -I/usr/include/python3.4m -c bigfile/_bigfile.c -o build/temp.linux-x86_64-3.4/bigfile/_bigfile.o
      -std=gnu99 -fplan9-extensions -fvisibility=hidden
          bigfile/_bigfile.c: In function ‘pyfile_new’:
          bigfile/_bigfile.c:679:5: error: ISO C90 forbids mixed declarations and code [-Werror=declaration-after-statement]
               static char *kw_list[] = {"blksize", NULL};
               ^
          cc1: some warnings being treated as errors
          error: command 'gcc' failed with exit status 1
      
      Ensure we turn off this warning and error because we already rely on compiling in C99 mode.
      Reported-and-tested-by: 's avatarIvan Tyagov <ivan@tyagov.com>
      e0b25398
  8. 03 Apr, 2015 6 commits
    • Kirill Smelkov's avatar
      Prepare to benchmark python code with py.test · bcab1246
      Kirill Smelkov authored
      For this, a small wrapper over py.test is developed (to discover/collect
      functions to benchmar, etc) and then it runs such functions several
      times in a boxed enveronment.
      
      Benchmarks should be named bench_*.py
      bcab1246
    • Kirill Smelkov's avatar
      Prepare to test python code with py.test · 8ba39a60
      Kirill Smelkov authored
      We'll use py.test for unit-testing of Python part.
      8ba39a60
    • Kirill Smelkov's avatar
      bigfile/virtmem: Userspace Virtual Memory Manager · 9a293c2d
      Kirill Smelkov authored
      Does similar things to what kernel does - users can mmap file parts into
      address space and access them read/write. The manager will be getting
      invoked by hardware/OS kernel for cases when there is no page loaded for
      read, or when a previousle read-only page is being written to.
      
      Additionally to features provided in kernel, it support to be used to
      store back changes in transactional way (see fileh_dirty_writeout()) and
      potentially use huge pages for mappings (though this is currently TODO)
      9a293c2d
    • Kirill Smelkov's avatar
      Low-level pagefault handler · 6f7d4d64
      Kirill Smelkov authored
      We hook into SIGSEGV and handle read/write pagefaults this way.
      
      In this patch there goes stub code that only detects faults and
      determines (in arch specific way) whether fault was for read or write
      and there is a TODO to pass that information to higher level.
      
      It also comes with tests to detect we still crash if we access something
      incorrectly, so people could have coredumps and investigate them.
      6f7d4d64
    • Kirill Smelkov's avatar
      Prepare to unit-test C stuff · cd5646af
      Kirill Smelkov authored
      We'll be testing C code in a similiar-to-python way - keep test_*.c
      files nearby and compile/executing them as needed.
      
      Tests are run several times in several ways - as plainly compiled and
      also with additional error checkers:
      
       - AddressSanitizer
       - ThreadSanitizer
       - and similiar checkers from Valgrind
      cd5646af
    • Kirill Smelkov's avatar
      Basic setup.py / Makefile to build/install/sdist stuff + bigfile.so skeleton · 5755a6b3
      Kirill Smelkov authored
      It is an early project decision to use gnu99 & Plan9 C extensions, to
      simplify C code.
      
      So far we only build stub wendelin/bigfile/_bigfile.so .
      
      Makefile is introduced because there will be targets which are easier to
      handle at make level. For end users no make knowledge is required -
      usual `python setup.py build|install|...` work, redirecting to make
      where necessary.
      
      As was promised (e870781d "Top-level in-tree import redirector")
      setup.py contains install-time hooks to handle in-tree wendelin.py and
      install it as a module namespace.
      
      For sdist, we just use `git ls-files` info if we are in a checkout.
      5755a6b3