An error occurred fetching the project authors.
- 28 Oct, 2021 1 commit
-
-
Kirill Smelkov authored
Add initial stub for WCFS program and tests. WCFS functionality will be added step-by-step in follow-up commits. Some preliminary history: 0ae88a32 X .nxdtest: Verify Go bits with GOMAXPROCS=1,2,`nproc` 23528eb4 X wcfs: make it to use go modules for dependencies
-
- 25 Oct, 2021 3 commits
-
-
Kirill Smelkov authored
Upcoming libwcfs (C++ part of WCFS client) will need to use virtmem code and link to libvirtmem.
-
Kirill Smelkov authored
Do what we can do without gdb and then tail to regular segmentation fault. With core file gdb can still be used, but it is handy if we already can get traceback of the crash into the log automatically. TODO better use https://github.com/ianlancetaylor/libbacktrace because backtrace_symbols often does not provide symbolic information. We do not do this now because libbacktrace is not always automatically installed.
-
Kirill Smelkov authored
This makes sure that those programs are always built afresh instead being stuck at outdated build. This is needed because corresponding test .c file includes many other .c files and we don't implement dependency tracking.
-
- 08 Jul, 2019 1 commit
-
-
Kirill Smelkov authored
For failing case compiler-rt support was added in 2014 - 5 years ago (see links in removed code).
-
- 04 Jul, 2018 1 commit
-
-
Kirill Smelkov authored
So that it can be available to everyone and in particular B & friends to be available from introduced importable golang.testing package. The move itself: pygolang@9bf03d9c While moving the code was restructured / improved a bit and py.bench interface reworked to mimic `go test -bench` in defaults.
-
- 12 Dec, 2017 1 commit
-
-
Kirill Smelkov authored
Benchmark the time it takes for virtmem to handle pagefault with noop loadblk for loadblk both implemented in C and in Python. On my computer it is: name µs/op PagefaultC 269 ± 0% pagefault_py 291 ± 0% Quite a big time in other words. It turned out to be mostly spent in fallocate'ing pages on tmpfs from /dev/shm. Part of the above 269 µs/op is taken by freeing (reclaiming) pages back when benchmarking work size exceed /dev/shm size, and part to allocating. If I limit the work size (via npage in benchmem.c) to be less than whole /dev/shm it starts to be ~ 170 µs/op and with additional tracing it shows as something like this: .. on_pagefault_start 0.954 µs .. vma_on_pagefault_pre 0.954 µs .. ramh_alloc_page_pre 0.954 µs .. ramh_alloc_page 169.992 µs .. vma_on_pagefault 172.853 µs .. vma_on_pagefault_pre 172.853 µs .. vma_on_pagefault 174.046 µs .. on_pagefault_end 174.046 µs .. whole: 171.900 µs so almost all time is spent in ramh_alloc_page which is doing the fallocate: https://lab.nexedi.com/nexedi/wendelin.core/blob/f11386a4/bigfile/ram_shmfs.c#L125 Simple benchmark[1] confirmed it is indeed the case for fallocate(tmpfs) to be relatively slow[2] (and that for recent kernels it regressed somewhat compared to Linux 3.16). Profile flamegraph for that benchmark[3] shows internal loading of shmem_fallocate which for 1 hardware page is not that too slow (e.g. <1µs) but when a request comes for a region internally performs it page by page and so accumulates that ~ 170µs for 2M. I've tried to briefly rerun the benchmark with huge pages activated on /dev/shm via mount /dev/shm -o huge=always,remount as both regular user and as root but it was executing several times slower. Probably something to investigate more later. [1] https://lab.nexedi.com/kirr/misc/blob/4f84a06e/tmpfs/t_fallocate.c [2] https://lab.nexedi.com/kirr/misc/blob/4f84a06e/tmpfs/1.txt [3] https://lab.nexedi.com/kirr/misc/raw/4f84a06e/tmpfs/fallocate-2M-nohuge.svg
-
- 24 Oct, 2017 1 commit
-
-
Kirill Smelkov authored
Relicense to GPLv3+ with wide exception for all Free Software / Open Source projects + Business options. Nexedi stack is licensed under Free Software licenses with various exceptions that cover three business cases: - Free Software - Proprietary Software - Rebranding As long as one intends to develop Free Software based on Nexedi stack, no license cost is involved. Developing proprietary software based on Nexedi stack may require a proprietary exception license. Rebranding Nexedi stack is prohibited unless rebranding license is acquired. Through this licensing approach, Nexedi expects to encourage Free Software development without restrictions and at the same time create a framework for proprietary software to contribute to the long term sustainability of the Nexedi stack. Please see https://www.nexedi.com/licensing for details, rationale and options.
-
- 06 Aug, 2015 1 commit
-
-
Kirill Smelkov authored
At present several threads running can corrupt internal virtmem datastructures (e.g. ram->lru_list, fileh->pagemap, etc). This can happen even if we have zope instances only with 1 worker thread - because there are other "system" thread, and python garbage collection can trigger at any thread, so if a virtmem object, e.g. VMA or FileH was there sitting at GC queue to be collected, their collection, and thus e.g. vma_unmap() and fileh_close() will be called from different-from-worker thread. Because of that virtmem just has to be aware of threads not to allow internal datastructure corruption. On the other hand, the idea of introducing userspace virtual memory manager turned out to be not so good from performance and complexity point of view, and thus the plan is to try to move it back into the kernel. This way it does not make sense to do a well-optimised locking implementation for userspace version. So we do just a simple single "protect-all" big lock for virtmem. Of a particular note is interaction with Python's GIL - any long-lived lock has to be taken with GIL released, because else it can deadlock: t1 t2 G V G !G V G so we introduce helpers to make sure the GIL is not taken, and to retake it back if we were holding it initially. Those helpers (py_gil_ensure_unlocked / py_gil_retake_if_waslocked) are symmetrical opposites to what Python provides to make sure the GIL is locked (via PyGILState_Ensure / PyGILState_Release). Otherwise, the patch is more-or-less straightforward application for one-big-lock to protect everything idea.
-
- 25 May, 2015 1 commit
-
-
Kirill Smelkov authored
Else, in a clean repo the tests do not build from scratch: $ make test x86_64-linux-gnu-gcc -pthread -g -Wall -D_GNU_SOURCE -std=gnu99 -fplan9-extensions -Wno-declaration-after-statement -Wno-error=declaration-after-statement -Iinclude -I3rdparty/ccan -I3rdparty/include bigfile/tests/test_virtmem.c lib/bug.c lib/utils.c 3rdparty/ccan/ccan/tap/tap.c -o bigfile/tests/test_virtmem.t In file included from include/wendelin/list.h:11:0, from include/wendelin/bigfile/virtmem.h:33, from bigfile/tests/../virtmem.c:23, from bigfile/tests/test_virtmem.c:21: 3rdparty/ccan/ccan/array_size/array_size.h:4:20: fatal error: config.h: No such file or directory #include "config.h" ^ compilation terminated. 3rdparty/ccan/ccan/tap/tap.c:26:20: fatal error: config.h: No such file or directory #include "config.h" ^ compilation terminated. Makefile:99: recipe for target 'bigfile/tests/test_virtmem.t' failed make: *** [bigfile/tests/test_virtmem.t] Error 1
-
- 15 Apr, 2015 1 commit
-
-
Kirill Smelkov authored
In C99 declaration after statement is ok, and we explicitly compile with -std=gnu99. Python >= 3.4 however adds -Werror=declaration-after-statement even for extension modules irregardless of their compilation flags: https://bugs.python.org/issue21121 and the build fails this way: building 'wendelin.bigfile._bigfile' extension creating build/temp.linux-x86_64-3.4 creating build/temp.linux-x86_64-3.4/bigfile creating build/temp.linux-x86_64-3.4/lib gcc -pthread -Wno-unused-result -Werror=declaration-after-statement -DNDEBUG -fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -DOPENSSL_LOAD_CONF -fPIC -D_GNU_SOURCE -I./include -I./3rdparty/ccan -I./3rdparty/include -I/usr/include/python3.4m -c bigfile/_bigfile.c -o build/temp.linux-x86_64-3.4/bigfile/_bigfile.o -std=gnu99 -fplan9-extensions -fvisibility=hidden bigfile/_bigfile.c: In function ‘pyfile_new’: bigfile/_bigfile.c:679:5: error: ISO C90 forbids mixed declarations and code [-Werror=declaration-after-statement] static char *kw_list[] = {"blksize", NULL}; ^ cc1: some warnings being treated as errors error: command 'gcc' failed with exit status 1 Ensure we turn off this warning and error because we already rely on compiling in C99 mode. Reported-and-tested-by:
Ivan Tyagov <ivan@tyagov.com>
-
- 03 Apr, 2015 6 commits
-
-
Kirill Smelkov authored
For this, a small wrapper over py.test is developed (to discover/collect functions to benchmar, etc) and then it runs such functions several times in a boxed enveronment. Benchmarks should be named bench_*.py
-
Kirill Smelkov authored
We'll use py.test for unit-testing of Python part.
-
Kirill Smelkov authored
Does similar things to what kernel does - users can mmap file parts into address space and access them read/write. The manager will be getting invoked by hardware/OS kernel for cases when there is no page loaded for read, or when a previousle read-only page is being written to. Additionally to features provided in kernel, it support to be used to store back changes in transactional way (see fileh_dirty_writeout()) and potentially use huge pages for mappings (though this is currently TODO)
-
Kirill Smelkov authored
We hook into SIGSEGV and handle read/write pagefaults this way. In this patch there goes stub code that only detects faults and determines (in arch specific way) whether fault was for read or write and there is a TODO to pass that information to higher level. It also comes with tests to detect we still crash if we access something incorrectly, so people could have coredumps and investigate them.
-
Kirill Smelkov authored
We'll be testing C code in a similiar-to-python way - keep test_*.c files nearby and compile/executing them as needed. Tests are run several times in several ways - as plainly compiled and also with additional error checkers: - AddressSanitizer - ThreadSanitizer - and similiar checkers from Valgrind
-
Kirill Smelkov authored
It is an early project decision to use gnu99 & Plan9 C extensions, to simplify C code. So far we only build stub wendelin/bigfile/_bigfile.so . Makefile is introduced because there will be targets which are easier to handle at make level. For end users no make knowledge is required - usual `python setup.py build|install|...` work, redirecting to make where necessary. As was promised (e870781d "Top-level in-tree import redirector") setup.py contains install-time hooks to handle in-tree wendelin.py and install it as a module namespace. For sdist, we just use `git ls-files` info if we are in a checkout.
-