- 29 Sep, 2019 1 commit
-
-
Kirill Smelkov authored
-
- 18 Sep, 2019 1 commit
-
-
Kirill Smelkov authored
-
- 15 Jul, 2019 1 commit
-
-
Kirill Smelkov authored
There was an XXX of whether fileh_open should be a BigFile method or global function. However if it would be a global function it will need to anyway accept file parameter to indicate which file is opened, and that in turn suggests that it should be a file method. Remove XXX.
-
- 12 Jul, 2019 3 commits
-
-
Kirill Smelkov authored
try/finally was used in a couple of places to save/restore default ZBlk format setting. Move the restore part close to save with the help of defer.
-
Kirill Smelkov authored
For tests this makes sure that if one test fails, it won't make following tests fail just because the next test will fail trying to lock test database. For regular code (demo_zbigarray.py) this is also a good thing to do - to always close the database irregardless of whether an exception was raised before program reached end of main. Pygolang becomes regular - not test only - dependency. Being regular dependency is currently required only by demo_zbigarray.py, but it will be also used in upcoming wcfs, so adding pygolang into wendelin.core dependencies aligns with the plan. dbclose now uses defer almost everywhere - there are still few places in tests, where one test function is opening/closing test database multiple times - those were not (yet ?) converted.
-
Kirill Smelkov authored
Instead of raises(Exception, 'code') do with raises(Exception): code This removes lots of warnings, similar to below example: bigfile/tests/test_basic.py::test_basic /home/kirr/src/wendelin/wendelin.core/bigfile/tests/test_basic.py:79: PytestDeprecationWarning: raises(..., 'code(as_a_string)') is deprecated, use the context manager form or use `exec()` directly See https://docs.pytest.org/en/latest/deprecations.html#raises-warns-exec raises(ROAttributeError, "f.blksize = 1") # RO attribute
-
- 11 Jul, 2019 3 commits
-
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-> into CHECK_MRU and CHECK_DIRTY. Besides improving signal/noise ratio in tests, it also gives more details by printing full lists when things are found to be different from expected.
-
- 10 Jul, 2019 2 commits
-
-
Kirill Smelkov authored
The undef defined in that function were placed in next function going after it which does not define any macro.
-
Kirill Smelkov authored
-
- 09 Jul, 2019 2 commits
-
-
Kirill Smelkov authored
It was already said there that allocated page is not associated with fileh. However the code is doing more - it does not add the page to ram->lru_list and (obviously) to fileh->dirty_pages lists. Document that explicitly.
-
Kirill Smelkov authored
It should release resources associated with RAM. Make it call .ram_close from RAM ops. Add corresponding .ram_close to ram_shmfs. This fixes SHMFS_RAM->prefix leak.
-
- 08 Jul, 2019 6 commits
-
-
Kirill Smelkov authored
test_ram is low-level test that tests RAM pages allocation/mmapping. As allocated pages are not integrated with virtmem (not added to any file mapping and RAM->lru_list) the Page structs have to be explicitly freed. Fixes e.g. Direct leak of 80 byte(s) in 1 object(s) allocated from: #0 0x7ff29af46518 in calloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe9518) #1 0x56131dc22289 in zalloc include/wendelin/utils.h:67 #2 0x56131dc225d6 in ramh_alloc_page bigfile/tests/../ram.c:41 #3 0x56131dc2a19e in main bigfile/tests/test_ram.c:130 #4 0x7ff29ac9f09a in __libc_start_main ../csu/libc-start.c:308
-
Kirill Smelkov authored
Else on-heap allocated RAM object is leaked. Fixes e.g. the following error on ASAN: Direct leak of 56 byte(s) in 1 object(s) allocated from: #0 0x7fc9ef390518 in calloc (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xe9518) #1 0x555ca792f309 in zalloc include/wendelin/utils.h:67 #2 0x555ca7935f9a in ram_limited_new bigfile/tests/../../t/t_utils.c:35 #3 0x555ca793a0ba in test_file_access_synthetic bigfile/tests/test_virtmem.c:292 #4 0x555ca7967bc4 in main bigfile/tests/test_virtmem.c:1121 #5 0x7fc9ef0e909a in __libc_start_main ../csu/libc-start.c:308
-
Kirill Smelkov authored
Only one place that was using ram_new was missing to call ram_close in the end.
-
Kirill Smelkov authored
For failing case compiler-rt support was added in 2014 - 5 years ago (see links in removed code).
-
Kirill Smelkov authored
-> Use .[test] to refer to them. https://stackoverflow.com/a/41398850/9456786
-
Kirill Smelkov authored
Just an update to latest CCAN - there is actually no changes to modules that we use (tap, array_size, minmax, bitmap, build_assert)
-
- 18 Jun, 2019 4 commits
-
-
Kirill Smelkov authored
-
Kirill Smelkov authored
Let's keep on test coverage for last 2 stable Python releases.
-
Kirill Smelkov authored
Starting from Python 3.7 the place to keep exception state was changed: https://github.com/python/cpython/commit/ae3087c638 NOTE ZEO4 does not wok with Python3.7, because ZEO4 uses "async" for a variable and that breaks because starting from Python3.7 "async" became a keyword. After the fix wendelin.core tests pass under all python2.7, python3.6 and python3.7.
-
Kirill Smelkov authored
Dependency added here: https://github.com/zopefoundation/ZODB/commit/e0bc8bd567 If we don't provide mock, e.g. py27-ZODB5-*-zeo-* breaks: def setup_module(): global testdb > testdb = getTestDB() bigarray/tests/test_arrayzodb.py:38: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ .tox/py27-ZODB5-zblk1-zeo-numpy115/lib/python2.7/site-packages/wendelin/lib/testing.py:342: in getTestDB testdb = testdb_factory(testdb_uri) .tox/py27-ZODB5-zblk1-zeo-numpy115/lib/python2.7/site-packages/wendelin/lib/testing.py:245: in __init__ from ZEO.tests import forker .tox/py27-ZODB5-zblk1-zeo-numpy115/lib/python2.7/site-packages/ZEO/tests/forker.py:29: in <module> import ZODB.tests.util _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ """ from ZODB.MappingStorage import DB import atexit import os import persistent import re import tempfile import time import transaction import unittest import warnings import ZODB.utils from ZODB.Connection import TransactionMetaData import zope.testing.setupstack from zope.testing import renormalizing try: from unittest import mock except ImportError: > import mock E ImportError: No module named mock .tox/py27-ZODB5-zblk1-zeo-numpy115/lib/python2.7/site-packages/ZODB/tests/util.py:35: ImportError
-
- 23 May, 2019 2 commits
-
-
Kirill Smelkov authored
-
Kirill Smelkov authored
This continues c7c01ce4 (bigfile/zodb: ZODB.Connection can migrate between threads on close/open and we have to care): Until now we were retrieving zconn.transaction_manager on _ZBigFileH init, and further using that transaction manager for every connection reopen. However that is not correct because on every reopen connection can be given new transaction manager. We were not practically hitting the bug because until recently ZODB was, by default, using the same ThreadTransactionManager manager instance as Connection.transaction_manager for all connections, and not doing all steps needed to keep _ZBigFileH.transaction_manager in sync to Connection was forgiven - a particular transaction manager that was used was TransactionManager instance implicitly associated with current thread by global threading.Local transaction.manager . However starting from ZODB 5.5.0 Connection code was changed to remember as .transaction_manager the particular TransactionManager instance without any threading.Local games: https://github.com/zopefoundation/ZODB/commit/b6ac40f153 https://github.com/zopefoundation/ZODB/issues/208 https://github.com/zopefoundation/ZODB/pull/226 Given that we were not syncing properly that broke wendelin.core tests, for example: bigfile/tests/test_filezodb.py::test_bigfile_filezodb_vs_conn_migration Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/home/kirr/src/wendelin/wendelin.core/bigfile/tests/test_filezodb.py", line 401, in T11 transaction.commit() # should be nothing File "/home/kirr/src/wendelin/venv/z-dev/local/lib/python2.7/site-packages/transaction/_manager.py", line 252, in commit return self.manager.commit() File "/home/kirr/src/wendelin/venv/z-dev/local/lib/python2.7/site-packages/transaction/_manager.py", line 131, in commit return self.get().commit() File "/home/kirr/src/wendelin/venv/z-dev/local/lib/python2.7/site-packages/transaction/_transaction.py", line 298, in commit self._synchronizers.map(lambda s: s.beforeCompletion(self)) File "/home/kirr/src/wendelin/venv/z-dev/local/lib/python2.7/site-packages/transaction/weakset.py", line 61, in map f(elt) File "/home/kirr/src/wendelin/venv/z-dev/local/lib/python2.7/site-packages/transaction/_transaction.py", line 298, in <lambda> self._synchronizers.map(lambda s: s.beforeCompletion(self)) File "/home/kirr/src/wendelin/wendelin.core/bigfile/file_zodb.py", line 671, in beforeCompletion assert txn is zconn.transaction_manager.get() AssertionError What is happening here is that one thread used the connection and ZBigFile/_ZBigFileH associated with it, then the connection was closed and released to DB pool. Then the connection was reopened but by another thread and thus with different TransactionManager instance and oops - _ZBigFileH.transaction_manager is different because it is TransactionManager instance that was used by the first thread. Fix it by resyncing _ZBigFileH.transaction_manager on every connection reopen. No new test as existing tests already cover the problem when run with ZODB >= 5.5.0 .
-
- 03 May, 2019 1 commit
-
-
Kirill Smelkov authored
This continues 6ab95220 (t/qemu-runlinux: Issue terminal resize before running program) and fully initializes terminal before spawning user application. This has practical effect to restore line wrapping for xterm, as kernel, initially assuming it has "linux" type terminal, somehow messes xterm settings: before the patch lines that were wider than terminal width were not wrapped and characters in the last position were printed over each other. After the patch printed lines are automatically wrapped and test output is not lost. Tput hint found here: https://unix.stackexchange.com/questions/105958
-
- 17 Mar, 2019 1 commit
-
-
Kirill Smelkov authored
0: ERROR+ on boot/run 1: INFO+ on run 2: INFO+ on boot/run 3: DEBUG+ on boot/run It is convenient not to see large kernel boot log on every test run. "1" (single -v) is also convenient: one can skip the boot log but still see details of what is going on when test workload is run. -vv and -vvv are there to see full picture.
-
- 27 Feb, 2019 3 commits
-
-
Kirill Smelkov authored
bpf is needed for tools like bpftrace. fusectl is needed to observe things like /sys/fs/fuse/connections/X/waiting.
-
Kirill Smelkov authored
Else program inside sees default terminal settings to be 80x25, while after the patch it sees correct settings. Still something is not completely right with terminal settings, as e.g. vsplit is not working correctly in vim (vertical ruler is not straight).
-
Kirill Smelkov authored
Graphics mode runs in another window with its own terminal emulation, so propagating e.g. TERM=xterm to where it is emulated as TERM=linux is not correct.
-
- 13 Feb, 2019 1 commit
-
-
Kirill Smelkov authored
Continuing 76d8f76d (Script to run compiled linux kernel with root fs mounted from host) update the script to run/debug linux inside QEMU: - teach it to run specified program + args, instead of hardcoded /bin/sh; - before tailing to user program, builtin init mounts /proc, /sys, ... inside - previously it was /proc, /sys from host seen on those mountpoints and it was very misleading - e.g. ps was showing processes from host, not inside, etc. - builtin init also cares to run specified program with the same current directory that was current on host, and environments such as $HOME, $PATH, $TERM, ... are also propagated. - allow to optionally run QEMU with graphics, instead of stdout only; - increase available RAM from 128M to 512M (with 128M running py.test inside is failing with fork: not enough memory). This updated version was useful to debug WCFS(*) & FUSE issues by running kirr@deco:~/src/wendelin/wendelin.core/wcfs$ ../t/qemu-runlinux ~/src/linux/linux/arch/x86_64/boot/bzImage py.test -vsx -k test_wcfs See https://marc.info/?l=linux-fsdevel&m=155000277921155&w=2 for details. (*) WCFS is still being draft and worked on t branch.
-
- 11 Jan, 2019 1 commit
-
-
Kirill Smelkov authored
On a testing instance we started to see segfaults in pyvma_dealloc() with inside calls to vma_unmap but with NULL pyvma->fileh. That was strange, becuse before calling vma_unmap(), the code explicitly checks whether pyvma->fileh is !NULL. That was, as it turned out, due to pyvma_dealloc being called twice at the same time from two python threads. Here is how that was possible: T1 decrefs pyvma and finds its reference count drops to zero. It calls pyvma_dealloc. From there vma_unmap() is called, which calls virt_lock() and that releases GIL first. Another thread T2 was waiting for GIL, it acquires it, does some work at python level and somehow triggers GC. Since PyVMA supports cyclic GC, it was on GC list and thus GC calls dealloc for the same vma again. Here is how it looks in the backtraces: T1: #0 0x00007f6aefc57827 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0, futex_word=0x1e011d0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205 #1 do_futex_wait (sem=sem@entry=0x1e011d0, abstime=0x0) at sem_waitcommon.c:111 #2 0x00007f6aefc578d4 in __new_sem_wait_slow (sem=0x1e011d0, abstime=0x0) at sem_waitcommon.c:181 #3 0x00007f6aefc5797a in __new_sem_wait (sem=<optimized out>) at sem_wait.c:29 #4 0x00000000004ffbc4 in PyThread_acquire_lock () #5 0x00000000004dbe8a in PyEval_RestoreThread () #6 0x00007f6ac6d3b8fc in py_gil_retake_if_waslocked (arg=0x4f18f00) at bigfile/_bigfile.c:1048 #7 0x00007f6ac6d3dcfc in virt_gil_retake_if_waslocked (gilstate=0x4f18f00) at bigfile/virtmem.c:78 #8 0x00007f6ac6d3dd30 in virt_lock () at bigfile/virtmem.c:92 #9 0x00007f6ac6d3e724 in vma_unmap (vma=0x7f6a7e0c4100) at bigfile/virtmem.c:271 #10 0x00007f6ac6d3a0bc in pyvma_dealloc (pyvma0=0x7f6a7e0c40e0) at bigfile/_bigfile.c:284 ... #13 0x00000000004d76b0 in PyEval_EvalFrameEx () T2: #5 0x00007f6ac6d3a081 in pyvma_dealloc (pyvma0=0x7f6a7e0c40e0) at bigfile/_bigfile.c:276 #6 0x0000000000500450 in ?? () #7 0x00000000004ffd82 in _PyObject_GC_New () #8 0x0000000000485392 in PyList_New () #9 0x00000000004d3bff in PyEval_EvalFrameEx () T2 does the work of vma_unmap and clears C-level vma. Then, when T1 wakes up and returns to vma_unmap, it sees vma->file and all other fields cleared -> oops segfault. Fix it by removing pyvma from GC list before going to do actual destruction. This way if a concurrent GC triggers, it won't see the vma object on its list, and thus won't have a chance to invoke its destructor the second time. The bug was introduced in 450ad804 (bigarray: ArrayRef support for BigArray) when PyVMA was changed to be cyclic-GC aware. However at that time, even Python documentation itself was not saying PyObject_GC_UnTrack is needed, as it was added only in 2.7.15 after finding that many types in CPython itself are vulnerable to similar segfaults: https://github.com/python/cpython/commit/4cde4bdcc86 https://bugs.python.org/issue31095 It is pity, that CPython took the approach to force all type authors to care to invoke PyObject_GC_UnTrack explicitly, instead of doing that automatically in Python runtime before calling tp_dealloc. /cc @Tyagov, @klaus /reviewed-on !11
-
- 29 Oct, 2018 2 commits
-
-
Kirill Smelkov authored
Structured creates view of the array interpreting its minor axis as fully covered by a dtype. It is similar to arr.view(dtype) + corresponding reshape, but does not have limitations of ndarray.view(). For example: In [1]: a = np.arange(3*3, dtype=np.int32).reshape((3,3)) In [2]: a Out[2]: array([[0, 1, 2], [3, 4, 5], [6, 7, 8]], dtype=int32) In [3]: b = a[:2,:2] In [4]: b Out[4]: array([[0, 1], [3, 4]], dtype=int32) In [5]: dtxy = np.dtype([('x', np.int32), ('y', np.int32)]) In [6]: dtxy Out[6]: dtype([('x', '<i4'), ('y', '<i4')]) In [7]: b.view(dtxy) --------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-66-af98529aa150> in <module>() ----> 1 b.view(dtxy) ValueError: To change to a dtype of a different size, the array must be C-contiguous In [8]: structured(b, dtxy) Out[8]: array([(0, 1), (3, 4)], dtype=[('x', '<i4'), ('y', '<i4')]) Structured always creates view and never copies data. Here is original context where separately playing with .shape and .dtype was not enough, since it was creating array copy and OOM'ing the machine: klaus/wendelin@cbe4938b
-
Kirill Smelkov authored
We are going to use this code in another place, so move this out to dommon place as a preparatory step first. On a related note: Since ArrayRef is generic and quite independent from BigArray (it only supports it, but equally it supports just other - e.g. plain arrays), the proper place for it might be also to be lib/xnumpy.py . We might get to this topic a bit later.
-
- 12 Oct, 2018 2 commits
-
-
Kirill Smelkov authored
RAMArray is compatible to ZBigArray in API and semantic, but stores its data in RAM only. It is useful in situations where ZBigArray compatible data type is needed, but the amount of data is small and the data itself is needed only temporarily - e.g. in a simulation. Please see details in individual patches. Original merge request by @klaus (!8). /cc @Tyagov /reviewed-on !9
-
Kirill Smelkov authored
RAMArray is compatible to ZBigArray in API and semantic, but stores its data in RAM only. It is useful in situations where ZBigArray compatible data type is needed, but the amount of data is small and the data itself is needed only temporarily - e.g. in a simulation. Implementation is based on mmapping temporary files from /dev/shm/... and passing them as file handles, similarly to how ZBigArray works, to BigArray. We don't use just numpy.ndarray because of append - for ZBigArray append works in O(1), but more importantly it does not copy data. This way mmapings previously created for ZBigArray views, continue to correctly alias array data. If we would be using ndarray directly, since ndarray.resize copies data, that property would not be preserved. Original patch by Klaus Wölfel <klaus@nexedi.com> (nexedi/wendelin.core!8)
-
- 11 Oct, 2018 1 commit
-
-
Kirill Smelkov authored
bigarray/tests: Factor out a way to spcify on which BigFile/BigFileH an array is tested into fixture parameter Currently we have only one BigFile and its BigFileH handle. However in the next patch, for RAMArray, we'll be adding handles for opened RAM files, and it would be good to test whole BigArray functionality on data served by those handles too. Prepare for this and first factor out into testbig fixture the way to open such handles.
-
- 04 Jul, 2018 1 commit
-
-
Kirill Smelkov authored
So that it can be available to everyone and in particular B & friends to be available from introduced importable golang.testing package. The move itself: kirr/pygolang@9bf03d9c While moving the code was restructured / improved a bit and py.bench interface reworked to mimic `go test -bench` in defaults.
-
- 26 Jun, 2018 2 commits
-
-
Kirill Smelkov authored
-
Kirill Smelkov authored
-