Mutex lock/unlock should not fail if mutex was correctly initialized/used.

We factored out SIGSEGV block/restore from fileh_dirty_writeout() to all
functions in cb7a7055 (bigfile/virtmem: Block/restore SIGSEGV in
non-pagefault-handling function). The restoration however just sets
whole thread sigmask.

It could be possible that between block/restore calls procmask for other
signals could be changed, and this way - setting procmask directly - we
will overwrite them.

So be careful, and when restoring SIGSEGV mask, touch mask bit for only
that signal.

( we need xsigismember helper to get this done, which is also introduced
  in this patch )

The mistake was there from the beginning - from 3e5e78cd (lib/utils:
Small C utilities we'll use).

We'll need this for function which return error not in errno - e.g.
pthread_sigmask().

Previously we were always testing with DBs backed up by FileStorage. Now
we provide a way to run the testsuite with user selected storage
backend:

    $ WENDELIN_CORE_TEST_DB="<fs>"   make test.py     # test with temporary db with FileStorage
    $ WENDELIN_CORE_TEST_DB="<zeo>"  make test.py     # ----------//---------- with ZEO
    $ WENDELIN_CORE_TEST_DB="<neo>"  make test.py     # ----------//---------- with NEO

    $ WENDELIN_CORE_TEST_DB=neo://db@master  make test.py     # test with externally provided DB

Default is still to run tests with FileStorage.

/cc @jm

Done via manual hacky way for now. The clean solution would be to reuse
e.g. repoze.zodbconn[1] or zodburi[2] and teach them to support NEO.

But for now we can't -- those eggs depend on ZODB, and we still use
ZODB3 for maintaining compatibility with both ZODB3.10 and ZODB4.

/cc @jm

[1] https://pypi.python.org/pypi/repoze.zodbconn
[2] https://pypi.python.org/pypi/zodburi

Factor out those routines to open a ZODB database to common place.

The reason for doing so is that we'll soon teach dbopen to automatically
recognize several protocols, e.g. neo:// and zeo:// and this way,
clients who use dbopen() could automatically access storages besides
FileStorage.

e.g.

    In [1]: multiply.reduce((1<<30, 1<<30, 1<<30))
    Out[1]: 0

instead of

    In [2]: (1<<30) * (1<<30) * (1<<30)
    Out[2]: 1237940039285380274899124224

    In [3]: 1<<90
    Out[3]: 1237940039285380274899124224

also multiply.reduce returns int64, instead of python int:

    In [4]: type( multiply.reduce([1,2,3]) )
    Out[4]: numpy.int64

which also leads to overflow-related problems if we further compute with
this value and other integers and results exceeds int64 - it becomes
float:

    In [5]: idx0_stop = 18446744073709551615

    In [6]: stride0   = numpy.int64(1)

    In [7]: byte0_stop = idx0_stop * stride0

    In [8]: byte0_stop
    Out[8]: 1.8446744073709552e+19

and then it becomes a real problem for BigArray.__getitem__()

    wendelin.core/bigarray/__init__.py:326: RuntimeWarning: overflow encountered in long_scalars
      page0_min  = min(byte0_start, byte0_stop+byte0_stride) // pagesize # TODO -> fileh.pagesize

and then

    >           vma0 = self._fileh.mmap(page0_min, page0_max-page0_min+1)
    E           TypeError: integer argument expected, got float

~~~~

So just avoid multiple.reduce() and do our own mul() properly the same
way sum() is builtin into python, and we avoid overflow-related
problems.

    - for virtual memory subsytem
    - for ZBigFiles

They are not currently great, e.g. for virtmem we have in-kernel
overhead of page clearing - in perf profiles, for bigfile_mmap compared
to file_read kernel's clear_page_c raises significantly.

That is the worker for clearing page memory and we currently cannot
avoid that - any memory obtained from kernel (MAP_ANONYMOUS, mmap(file)
with hole, etc...) comes pre-initialized to zeros to userspace.

This can be seen in the benchmarks as well: file_readbig differs from
file_read in only that the latter uses 1 small buffer and the first
allocates large memory (cleared by kernel + python does the memset).

    bigfile/tests/bench_virtmem.py@125::bench_file_mmap_adler32     0.47  (0.86 0.49 0.47)
    bigfile/tests/bench_virtmem.py@126::bench_file_read_adler32     0.69  (1.11 0.71 0.69)
    bigfile/tests/bench_virtmem.py@127::bench_file_readbig_adler32  1.41  (1.70 1.42 1.41)
    bigfile/tests/bench_virtmem.py@128::bench_bigfile_mmap_adler32  1.42  (1.45 1.42 1.51)

    bigfile/tests/bench_virtmem.py@130::bench_file_mmap_md5         1.52  (1.91 1.54 1.52)
    bigfile/tests/bench_virtmem.py@131::bench_file_read_md5         1.73  (2.10 1.75 1.73)
    bigfile/tests/bench_virtmem.py@132::bench_file_readbig_md5      2.44  (2.73 2.46 2.44)
    bigfile/tests/bench_virtmem.py@133::bench_bigfile_mmap_md5      2.40  (2.48 2.40 2.53)

There is MAP_UNINITIALIZED which works only for non-mmu targets and only
if explicitly allowed when configuring kernel (off by default).

There were patches to disable that pages zeroing, as it gives
significant speedup for people's workloads, e.g. [1,2] but all of them
did not got merged for security reasons.

[1] http://marc.info/?t=132691315900001&r=1&w=2
[2] http://thread.gmane.org/gmane.linux.kernel/548926

~~~~

For ZBigFile - it is the storage who is dominating in profiles.

Like C bzero / memset & memcopy - but work on python buffers. We
leverage NumPy for doing actual work, and this way NumPy becomes a
depenency.

Having NumPy as a dependency is ok - we'll for sure need it later as we
are trying to build out-of-core ndarrays.

Like taking an exact integer log2, upcasting pointers for C-style
inheritance done in a Plan9 way, and wrappers to functions which should
never fail.

Modelled by ones used in Linux kernel.