Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
W
wendelin.core
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
Analytics
Analytics
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Commits
Issue Boards
Open sidebar
Joshua
wendelin.core
Commits
c4c25753
Commit
c4c25753
authored
Dec 06, 2019
by
Kirill Smelkov
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
.
parent
d4b93b03
Changes
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
24 additions
and
11 deletions
+24
-11
bigfile/file_zodb.py
bigfile/file_zodb.py
+24
-11
No files found.
bigfile/file_zodb.py
View file @
c4c25753
...
@@ -26,9 +26,9 @@ This way only regular ZODB objects - not blobs - are used, and whenever file
...
@@ -26,9 +26,9 @@ This way only regular ZODB objects - not blobs - are used, and whenever file
data is changed, δ(ZODB) is proportional to δ(data).
data is changed, δ(ZODB) is proportional to δ(data).
Being BigFile ZBigFile can be memory-mapped. Created mappings provide lazy
Being BigFile ZBigFile can be memory-mapped. Created mappings provide lazy
on-
access block loading and dirtying. This way ZBigFile larger than RAM can be
on-
read block loading and on-write dirtying. This way ZBigFile larger than RAM
accessed transparently as if it was a regular array in RAM. Changes made to
can be accessed transparently as if it was a regular array in RAM. Changes made
ZBigFile data will be either saved or discarded depending on current
to
ZBigFile data will be either saved or discarded depending on current
transaction completion - commit or abort. The amount of ZBigFile changes in one
transaction completion - commit or abort. The amount of ZBigFile changes in one
transaction is limited by available RAM.
transaction is limited by available RAM.
...
@@ -41,7 +41,12 @@ ZBigFile does not weaken ZODB ACID properties, in particular:
...
@@ -41,7 +41,12 @@ ZBigFile does not weaken ZODB ACID properties, in particular:
API for clients
API for clients
---------------
---------------
.fileh_open() -> open new BigFileH-like object which can be mmaped
API for clients is ZBigFile class and its .fileh_open() method:
.fileh_open() -> opens new BigFileH-like object which can be mmaped
The primary user of ZBigFile is ZBigArray (see bigarray/__init__.py and
bigarray/array_zodb.py), but ZBigFile itself can be used directly too.
Operating mode
Operating mode
...
@@ -51,9 +56,9 @@ Two operating modes are provided: "local-cache" and "shared-cache".
...
@@ -51,9 +56,9 @@ Two operating modes are provided: "local-cache" and "shared-cache".
Local-cache is the mode wendelin.core was originally implemented with in 2014.
Local-cache is the mode wendelin.core was originally implemented with in 2014.
In this mode ZBigFile data is loaded from ZODB directly via current ZODB connection.
In this mode ZBigFile data is loaded from ZODB directly via current ZODB connection.
It was relatively straight-forward to implement, but cached file data become
s
It was relatively straight-forward to implement, but cached file data become
duplicated in between ZODB connections of current process and in between
duplicated in between ZODB connections of current process and in between
several
ZODB processes
.
several
client processes that use ZODB
.
In shared-cache mode file's data is accessed through special filesystem for
In shared-cache mode file's data is accessed through special filesystem for
which data cache is centrally maintained by OS kernel. This mode was added in
which data cache is centrally maintained by OS kernel. This mode was added in
...
@@ -71,9 +76,17 @@ The mode of operation can be selected via environment variable::
...
@@ -71,9 +76,17 @@ The mode of operation can be selected via environment variable::
Data format
Data format
-----------
-----------
$WENDELIN_CORE_ZBLK_FMT
Due to weakness of current ZODB storage servers, wendelin.core cannot provide
ZBlk0
at the same time both fast reads and small database size growth on small data
ZBlk1
changes. "Small" here means something like 1-10000 bytes as larger changes
become comparable to 2M block size and are handled efficiently out of the box.
Until the problem is fixed on ZODB server side, users have to explicitly
indicate via environment variable that their workload is "small changes" if
they prefer to prioritize database size over access speed::
$WENDELIN_CORE_ZBLK_FMT
ZBlk0 fast reads (default)
ZBlk1 small changes
To represent BigFile as ZODB objects, each file block is represented separately
To represent BigFile as ZODB objects, each file block is represented separately
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment