Commit 1647d24a authored by Kirill Smelkov's avatar Kirill Smelkov

bigfile/zodb: Apply auto format as default only in WCFS mode

This semantically reverts 99f262dd (bigfile/zodb: Make auto format the
default) for wendelin.core-1 mode because in non-WCFS mode there are
known problems with data corruption on BTree topology changes(*) and
auto mode actually does change those topologies with first setting
ZBigFile[blk] -> ZBlk1 and then updating the same block to point to
ZBlk0 object.

Avoid pressuring those problems and use auto as default only in WCFS
mode that should handle invalidations with all those BTree topology
changes well.

The patch is based on suggestion by Levin Zimmermann: nexedi/wendelin.core!20 (comment 212405)

We have to move _default_use_wcfs because now it is invoked at module
import time and needs to be already defined at the time of the call.

(*) see nexedi/wendelin.core@8c32c9f6 for details.
parent 07087ec8
...@@ -91,10 +91,10 @@ on change pattern and works relatively good regarding both access speed and ...@@ -91,10 +91,10 @@ on change pattern and works relatively good regarding both access speed and
database size for append-like workloads:: database size for append-like workloads::
$WENDELIN_CORE_ZBLK_FMT $WENDELIN_CORE_ZBLK_FMT
ZBlk0 fast reads ZBlk0 fast reads (default in !wcfs mode)
ZBlk1 small changes ZBlk1 small changes
auto (default) heuristically use either ZBlk0 or ZBlk1 auto heuristically use either ZBlk0 or ZBlk1 (default in wcfs mode)
depending on change pattern depending on change pattern
Description of block formats follow: Description of block formats follow:
...@@ -491,8 +491,19 @@ ZBlk_fmt_registry = { ...@@ -491,8 +491,19 @@ ZBlk_fmt_registry = {
'auto': _ZBlk_auto, 'auto': _ZBlk_auto,
} }
# _default_use_wcfs returns whether default virtmem setting is to use wcfs or not.
def _default_use_wcfs():
virtmem = os.environ.get("WENDELIN_CORE_VIRTMEM", "rw:uvmm") # unset -> !wcfs
virtmem = virtmem.lower()
return {"r:wcfs+w:uvmm": True, "rw:uvmm": False}[virtmem]
# format for updated blocks # format for updated blocks
ZBlk_fmt_write = os.environ.get('WENDELIN_CORE_ZBLK_FMT', 'auto') ZBlk_fmt_write = os.environ.get('WENDELIN_CORE_ZBLK_FMT',
# auto is used only with wcfs because !wcfs mode does
# not handle BTree topology well leading to data corruption
# https://lab.nexedi.com/nexedi/wendelin.core/commit/8c32c9f6
'auto' if _default_use_wcfs() else
'ZBlk0')
if ZBlk_fmt_write not in ZBlk_fmt_registry: if ZBlk_fmt_write not in ZBlk_fmt_registry:
raise RuntimeError('E: Unknown ZBlk format %r' % ZBlk_fmt_write) raise RuntimeError('E: Unknown ZBlk format %r' % ZBlk_fmt_write)
...@@ -678,21 +689,13 @@ class ZBigFile(LivePersistent): ...@@ -678,21 +689,13 @@ class ZBigFile(LivePersistent):
# - not set -> behave according to global default # - not set -> behave according to global default
def fileh_open(self, _use_wcfs=None): def fileh_open(self, _use_wcfs=None):
if _use_wcfs is None: if _use_wcfs is None:
_use_wcfs = self._default_use_wcfs() _use_wcfs = _default_use_wcfs()
fileh = _ZBigFileH(self, _use_wcfs) fileh = _ZBigFileH(self, _use_wcfs)
self._v_filehset.add(fileh) self._v_filehset.add(fileh)
return fileh return fileh
# _default_use_wcfs returns whether default virtmem setting is to use wcfs or not.
@staticmethod
def _default_use_wcfs():
virtmem = os.environ.get("WENDELIN_CORE_VIRTMEM", "rw:uvmm") # unset -> !wcfs
virtmem = virtmem.lower()
return {"r:wcfs+w:uvmm": True, "rw:uvmm": False}[virtmem]
# BigFileH wrapper that also acts as DataManager proxying changes ZODB <- virtmem # BigFileH wrapper that also acts as DataManager proxying changes ZODB <- virtmem
# at two-phase-commit (TPC), and ZODB -> virtmem on objects invalidation. # at two-phase-commit (TPC), and ZODB -> virtmem on objects invalidation.
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment