Provide integration with virtmem, so that WCFS Mapping can be associated
and managed under virtmem VMA. In other words provide support so that WCFS can
be used as ZBigFile backend in "mmap overlay" mode (see fae045cc "bigfile/virtmem:
Introduce "mmap overlay" mode" for description of mmap-overlay mode).
We'll need this functionality for ZBigFile + WCFS client integration.
Virtmem integration will be tested via running whole wendelin.core functional
testsuite in wcfs-mode after the next patch.
Quoting added description:
---- 8< ----
Integration with wendelin.core virtmem layer
This client package can be used standalone, but additionally provides
integration with wendelin.core userspace virtual memory manager: when a
Mapping is created, it can be associated as serving base layer for a
particular virtmem VMA via FileH.mmap(vma=...). In that case, since virtmem
itself adds another layer of dirty pages over read-only base provided by
│RW│ │RW│ ← virtmem VMA dirty pages
VMA base = X@at view provided by Mapping:
─── ───── ────────────────────────── ───── /head/bigfile/X
the Mapping will interact with virtmem layer to coordinate
updates to mapping virtual memory.
How it works
Wcfs client integrates with virtmem layer to support virtmem handle
dirtying pages of read-only base-layer that wcfs client provides via
isolated Mapping. For wcfs-backed bigfiles every virtmem VMA is interlinked
VMA -> BigFileH -> ZBigFile -----> Z
Mapping -> FileH -> wcfs server --> DB
When a page is write-accessed, virtmem mmaps in a page of RAM in place of
accessed virtual memory, copies base-layer content provided by Mapping into
there, and marks that page as read-write.
Upon receiving pin message, the pinner consults virtmem, whether
corresponding page was already dirtied in virtmem's BigFileH (call to
__fileh_page_isdirty), and if it was, the pinner does not remmap Mapping
part to wcfs/@revX/f and just leaves dirty page in its place, remembering
pin information in fileh._pinned.
Once dirty pages are no longer needed (either after discard/abort or
writeout/commit), virtmem asks wcfs client to remmap corresponding regions
of Mapping in its place again via calls to Mapping.remmap_blk for previously
The scheme outlined above does not need to split Mapping upon dirtying an
See bigfile_ops interface (wendelin/bigfile/file.h) that explains base-layer
and overlaying from virtmem point of view. For wcfs this interface is
provided by small wcfs client wrapper in bigfile/file_zodb.cpp.
(+) see bigfile_ops interface (wendelin/bigfile/file.h) that gives virtmem
point of view on layering.
Some preliminary history:
f330bd2f X wcfs/client: Overview += interaction with virtmem layer