Commit c4c25753 authored by Kirill Smelkov's avatar Kirill Smelkov

.

parent d4b93b03
...@@ -26,9 +26,9 @@ This way only regular ZODB objects - not blobs - are used, and whenever file ...@@ -26,9 +26,9 @@ This way only regular ZODB objects - not blobs - are used, and whenever file
data is changed, δ(ZODB) is proportional to δ(data). data is changed, δ(ZODB) is proportional to δ(data).
Being BigFile ZBigFile can be memory-mapped. Created mappings provide lazy Being BigFile ZBigFile can be memory-mapped. Created mappings provide lazy
on-access block loading and dirtying. This way ZBigFile larger than RAM can be on-read block loading and on-write dirtying. This way ZBigFile larger than RAM
accessed transparently as if it was a regular array in RAM. Changes made to can be accessed transparently as if it was a regular array in RAM. Changes made
ZBigFile data will be either saved or discarded depending on current to ZBigFile data will be either saved or discarded depending on current
transaction completion - commit or abort. The amount of ZBigFile changes in one transaction completion - commit or abort. The amount of ZBigFile changes in one
transaction is limited by available RAM. transaction is limited by available RAM.
...@@ -41,7 +41,12 @@ ZBigFile does not weaken ZODB ACID properties, in particular: ...@@ -41,7 +41,12 @@ ZBigFile does not weaken ZODB ACID properties, in particular:
API for clients API for clients
--------------- ---------------
.fileh_open() -> open new BigFileH-like object which can be mmaped API for clients is ZBigFile class and its .fileh_open() method:
.fileh_open() -> opens new BigFileH-like object which can be mmaped
The primary user of ZBigFile is ZBigArray (see bigarray/__init__.py and
bigarray/array_zodb.py), but ZBigFile itself can be used directly too.
Operating mode Operating mode
...@@ -51,9 +56,9 @@ Two operating modes are provided: "local-cache" and "shared-cache". ...@@ -51,9 +56,9 @@ Two operating modes are provided: "local-cache" and "shared-cache".
Local-cache is the mode wendelin.core was originally implemented with in 2014. Local-cache is the mode wendelin.core was originally implemented with in 2014.
In this mode ZBigFile data is loaded from ZODB directly via current ZODB connection. In this mode ZBigFile data is loaded from ZODB directly via current ZODB connection.
It was relatively straight-forward to implement, but cached file data becomes It was relatively straight-forward to implement, but cached file data become
duplicated in between ZODB connections of current process and in between duplicated in between ZODB connections of current process and in between
several ZODB processes. several client processes that use ZODB.
In shared-cache mode file's data is accessed through special filesystem for In shared-cache mode file's data is accessed through special filesystem for
which data cache is centrally maintained by OS kernel. This mode was added in which data cache is centrally maintained by OS kernel. This mode was added in
...@@ -64,16 +69,24 @@ see wcfs/wcfs.go which describes the filesystem and shared-cache mode in detail. ...@@ -64,16 +69,24 @@ see wcfs/wcfs.go which describes the filesystem and shared-cache mode in detail.
The mode of operation can be selected via environment variable:: The mode of operation can be selected via environment variable::
$WENDELIN_CORE_VIRTMEM $WENDELIN_CORE_VIRTMEM
rw:uvmm local-cache (i.e. !wcfs) (default) rw:uvmm local-cache (i.e. !wcfs) (default)
r:wcfs+w:uvmm shared-cache (i.e. wcfs) r:wcfs+w:uvmm shared-cache (i.e. wcfs)
Data format Data format
----------- -----------
$WENDELIN_CORE_ZBLK_FMT Due to weakness of current ZODB storage servers, wendelin.core cannot provide
ZBlk0 at the same time both fast reads and small database size growth on small data
ZBlk1 changes. "Small" here means something like 1-10000 bytes as larger changes
become comparable to 2M block size and are handled efficiently out of the box.
Until the problem is fixed on ZODB server side, users have to explicitly
indicate via environment variable that their workload is "small changes" if
they prefer to prioritize database size over access speed::
$WENDELIN_CORE_ZBLK_FMT
ZBlk0 fast reads (default)
ZBlk1 small changes
To represent BigFile as ZODB objects, each file block is represented separately To represent BigFile as ZODB objects, each file block is represented separately
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment