- 06 Aug, 2015 1 commit
-
-
Kirill Smelkov authored
At present several threads running can corrupt internal virtmem datastructures (e.g. ram->lru_list, fileh->pagemap, etc). This can happen even if we have zope instances only with 1 worker thread - because there are other "system" thread, and python garbage collection can trigger at any thread, so if a virtmem object, e.g. VMA or FileH was there sitting at GC queue to be collected, their collection, and thus e.g. vma_unmap() and fileh_close() will be called from different-from-worker thread. Because of that virtmem just has to be aware of threads not to allow internal datastructure corruption. On the other hand, the idea of introducing userspace virtual memory manager turned out to be not so good from performance and complexity point of view, and thus the plan is to try to move it back into the kernel. This way it does not make sense to do a well-optimised locking implementation for userspace version. So we do just a simple single "protect-all" big lock for virtmem. Of a particular note is interaction with Python's GIL - any long-lived lock has to be taken with GIL released, because else it can deadlock: t1 t2 G V G !G V G so we introduce helpers to make sure the GIL is not taken, and to retake it back if we were holding it initially. Those helpers (py_gil_ensure_unlocked / py_gil_retake_if_waslocked) are symmetrical opposites to what Python provides to make sure the GIL is locked (via PyGILState_Ensure / PyGILState_Release). Otherwise, the patch is more-or-less straightforward application for one-big-lock to protect everything idea.
-
- 25 May, 2015 1 commit
-
-
Kirill Smelkov authored
Else, in a clean repo the tests do not build from scratch: $ make test x86_64-linux-gnu-gcc -pthread -g -Wall -D_GNU_SOURCE -std=gnu99 -fplan9-extensions -Wno-declaration-after-statement -Wno-error=declaration-after-statement -Iinclude -I3rdparty/ccan -I3rdparty/include bigfile/tests/test_virtmem.c lib/bug.c lib/utils.c 3rdparty/ccan/ccan/tap/tap.c -o bigfile/tests/test_virtmem.t In file included from include/wendelin/list.h:11:0, from include/wendelin/bigfile/virtmem.h:33, from bigfile/tests/../virtmem.c:23, from bigfile/tests/test_virtmem.c:21: 3rdparty/ccan/ccan/array_size/array_size.h:4:20: fatal error: config.h: No such file or directory #include "config.h" ^ compilation terminated. 3rdparty/ccan/ccan/tap/tap.c:26:20: fatal error: config.h: No such file or directory #include "config.h" ^ compilation terminated. Makefile:99: recipe for target 'bigfile/tests/test_virtmem.t' failed make: *** [bigfile/tests/test_virtmem.t] Error 1
-
- 15 Apr, 2015 1 commit
-
-
Kirill Smelkov authored
In C99 declaration after statement is ok, and we explicitly compile with -std=gnu99. Python >= 3.4 however adds -Werror=declaration-after-statement even for extension modules irregardless of their compilation flags: https://bugs.python.org/issue21121 and the build fails this way: building 'wendelin.bigfile._bigfile' extension creating build/temp.linux-x86_64-3.4 creating build/temp.linux-x86_64-3.4/bigfile creating build/temp.linux-x86_64-3.4/lib gcc -pthread -Wno-unused-result -Werror=declaration-after-statement -DNDEBUG -fmessage-length=0 -grecord-gcc-switches -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -DOPENSSL_LOAD_CONF -fPIC -D_GNU_SOURCE -I./include -I./3rdparty/ccan -I./3rdparty/include -I/usr/include/python3.4m -c bigfile/_bigfile.c -o build/temp.linux-x86_64-3.4/bigfile/_bigfile.o -std=gnu99 -fplan9-extensions -fvisibility=hidden bigfile/_bigfile.c: In function ‘pyfile_new’: bigfile/_bigfile.c:679:5: error: ISO C90 forbids mixed declarations and code [-Werror=declaration-after-statement] static char *kw_list[] = {"blksize", NULL}; ^ cc1: some warnings being treated as errors error: command 'gcc' failed with exit status 1 Ensure we turn off this warning and error because we already rely on compiling in C99 mode. Reported-and-tested-by:
Ivan Tyagov <ivan@tyagov.com>
-
- 03 Apr, 2015 6 commits
-
-
Kirill Smelkov authored
For this, a small wrapper over py.test is developed (to discover/collect functions to benchmar, etc) and then it runs such functions several times in a boxed enveronment. Benchmarks should be named bench_*.py
-
Kirill Smelkov authored
We'll use py.test for unit-testing of Python part.
-
Kirill Smelkov authored
Does similar things to what kernel does - users can mmap file parts into address space and access them read/write. The manager will be getting invoked by hardware/OS kernel for cases when there is no page loaded for read, or when a previousle read-only page is being written to. Additionally to features provided in kernel, it support to be used to store back changes in transactional way (see fileh_dirty_writeout()) and potentially use huge pages for mappings (though this is currently TODO)
-
Kirill Smelkov authored
We hook into SIGSEGV and handle read/write pagefaults this way. In this patch there goes stub code that only detects faults and determines (in arch specific way) whether fault was for read or write and there is a TODO to pass that information to higher level. It also comes with tests to detect we still crash if we access something incorrectly, so people could have coredumps and investigate them.
-
Kirill Smelkov authored
We'll be testing C code in a similiar-to-python way - keep test_*.c files nearby and compile/executing them as needed. Tests are run several times in several ways - as plainly compiled and also with additional error checkers: - AddressSanitizer - ThreadSanitizer - and similiar checkers from Valgrind
-
Kirill Smelkov authored
It is an early project decision to use gnu99 & Plan9 C extensions, to simplify C code. So far we only build stub wendelin/bigfile/_bigfile.so . Makefile is introduced because there will be targets which are easier to handle at make level. For end users no make knowledge is required - usual `python setup.py build|install|...` work, redirecting to make where necessary. As was promised (e870781d "Top-level in-tree import redirector") setup.py contains install-time hooks to handle in-tree wendelin.py and install it as a module namespace. For sdist, we just use `git ls-files` info if we are in a checkout.
-