Commit f696ce92 authored by Kirill Smelkov's avatar Kirill Smelkov

bigfile/zodb: Do not waste DB space storing trailing zeros for a ZBlk

A lot of times data in blocks come shorter than block size and the rest
of the memory page is zeros (because it was pre-filled zeros by OS when
page was allocated).

Do a simple heuristic and trim those trailing zeros and not store them
into DB.

With this change size of DB created by running bigfile and bigarray
tests changes as following:

            bigfile     bigarray

    old     145M         35M
    new      24K          6M

Trimming trailing zeros is currently done with str.rstrip('\0') which
creates a copy. When/if needed this could be optimized to work in-place.
parent e0b00bda
...@@ -79,7 +79,9 @@ class ZBlk(Persistent): ...@@ -79,7 +79,9 @@ class ZBlk(Persistent):
# client requests us to set blkdata to be later saved to DB # client requests us to set blkdata to be later saved to DB
# (DB <- ) ._v_blkdata <- memory-page # (DB <- ) ._v_blkdata <- memory-page
def setblkdata(self, buf): def setblkdata(self, buf):
self._v_blkdata = bytes(buf) # FIXME does memcpy blkdata = bytes(buf) # FIXME does memcpy
# trim trailing \0
self._v_blkdata = blkdata.rstrip(b'\0') # FIXME copy
# DB (through pickle) requests us to emit state to save # DB (through pickle) requests us to emit state to save
...@@ -238,9 +240,10 @@ class ZBigFile(LivePersistent): ...@@ -238,9 +240,10 @@ class ZBigFile(LivePersistent):
# TODO use specialized unpickler and load data directly to blk. # TODO use specialized unpickler and load data directly to blk.
# also in DB better store directly {#blk -> #dataref} without ZBlk overhead # also in DB better store directly {#blk -> #dataref} without ZBlk overhead
blkdata = zblk.loadblkdata() blkdata = zblk.loadblkdata()
assert len(blkdata) == self._v_file.blksize assert len(blkdata) <= self._v_file.blksize
zblk.bindzfile(self, blk) zblk.bindzfile(self, blk)
memcpy(buf, blkdata) # FIXME memcpy memcpy(buf, blkdata) # FIXME memcpy
#bzero(buftail) # not needed - buf comes pre-cleared from OS
# store data dirty page -> ZODB obj # store data dirty page -> ZODB obj
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment