Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
W
wendelin.core
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Analytics
Analytics
CI / CD
Repository
Value Stream
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
Levin Zimmermann
wendelin.core
Commits
49f826b1
Commit
49f826b1
authored
Nov 16, 2021
by
Kirill Smelkov
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
wendelin.core v2.0.alpha1
parent
ff8bd199
Changes
3
Show whitespace changes
Inline
Side-by-side
Showing
3 changed files
with
60 additions
and
2 deletions
+60
-2
CHANGELOG.rst
CHANGELOG.rst
+54
-0
README.rst
README.rst
+5
-1
setup.py
setup.py
+1
-1
No files found.
CHANGELOG.rst
View file @
49f826b1
Wendelin.core change history
============================
2.0.alpha1 (2021-11-16)
-----------------------
This is a major pre-release that reduces wendelin.core RAM consumption
dramatically:
The project switches to be mainly using kernel virtual memory manager.
Bigfiles are now primarily accessed with plain OS-level mmap to files from
synthetic WCFS filesystem. This makes bigfile's cache (now it is the kernel's
pagecache) to be shared in between several processes.
In addition a custom coherency protocol is provided, which allows clients,
that want to receive isolation guarantee ("I" from ACID), to still use the shared
cache and at the same time get bigfile view isolated from other's changes.
By default wendelin.core python client continues to provide full ACID semantics as
before.
In addition to being significantly more efficient, WCFS also fixes
data-corruption bugs that were discovered__ in how Wendelin.core 1 handles
invalidations on BTree topology change.
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/8c32c9f6
Please see wcfs.go__ for description of the new filesystem.
__ https://lab.nexedi.com/nexedi/wendelin.core/blob/master/wcfs/wcfs.go
Major steps: 1__, 2__, 3__, 4__, 5__, 6__, 7__, 8__, 9__, 10__, 11__, 12__,
13__, 14__, 15__, 16__, 17__, 18__, 19__, 20__, 21__.
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/2c152d41?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/e3f2ee2d?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/0e829874?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/a8595565?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/b87edcfe?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/1f2cd49d?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/27df5a3b?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/80153aa5?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/2ab4be93?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/f980471f?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/4430de41?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/6f0cdaff?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/10f7153a?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/fae045cc?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/23362204?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/ceadfcc7?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/1dba3a9a?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/1f866c00?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/e11edc70?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/986cf86e?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/c5e18c74?expanded=1
0.13 (2019-06-18)
-----------------
...
...
README.rst
View file @
49f826b1
...
...
@@ -81,13 +81,17 @@ limitations and things that need to be improved:
Thus
- we are currently working on improved wendelin.core design and implementation,
which
will use kernel virtual memory manager (instead of
one implemented__ in__
which
uses kernel virtual memory manager (complemented by
one implemented__ in__
userspace__) with arrays backend presented to kernel via FUSE as virtual
filesystem implemented in Go.
As of 2021 November `this filesystem`__ reached its alpha state and is staged
to be tried for real.
__ https://lab.nexedi.com/nexedi/wendelin.core/blob/master/include/wendelin/bigfile/virtmem.h
__ https://lab.nexedi.com/nexedi/wendelin.core/blob/master/bigfile/virtmem.c
__ https://lab.nexedi.com/nexedi/wendelin.core/blob/master/bigfile/pagefault.c
__ https://lab.nexedi.com/nexedi/wendelin.core/blob/master/wcfs/wcfs.go
In parallel we will also:
...
...
setup.py
View file @
49f826b1
...
...
@@ -290,7 +290,7 @@ libwcfs_h = [
setup
(
name
=
'wendelin.core'
,
version
=
'
0.13
'
,
version
=
'
2.0.alpha1
'
,
description
=
'Out-of-core NumPy arrays'
,
long_description
=
'%s
\
n
----
\
n
\
n
%s'
%
(
readfile
(
'README.rst'
),
readfile
(
'CHANGELOG.rst'
)),
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment