CHANGELOG.rst 11.1 KB
Newer Older
Kirill Smelkov's avatar
Kirill Smelkov committed
1 2 3
Wendelin.core change history
============================

4 5 6 7 8
- `2.0.alpha2`__ (2022-01-27)  Fix several crashes discovered by first on-field usage.

  __ https://lab.nexedi.com/nexedi/wendelin.core/compare/49f826b1...a36cdcc3


9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62
2.0.alpha1 (2021-11-16)
-----------------------

This is a major pre-release that reduces wendelin.core RAM consumption
dramatically:

The project switches to be mainly using kernel virtual memory manager.
Bigfiles are now primarily accessed with plain OS-level mmap to files from
synthetic WCFS filesystem. This makes bigfile's cache (now it is the kernel's
pagecache) to be shared in between several processes.

In addition a custom coherency protocol is provided, which allows clients,
that want to receive isolation guarantee ("I" from ACID), to still use the shared
cache and at the same time get bigfile view isolated from other's changes.

By default wendelin.core python client continues to provide full ACID semantics as
before.

In addition to being significantly more efficient, WCFS also fixes
data-corruption bugs that were discovered__ in how Wendelin.core 1 handles
invalidations on BTree topology change.

__ https://lab.nexedi.com/nexedi/wendelin.core/commit/8c32c9f6

Please see wcfs.go__ for description of the new filesystem.

__ https://lab.nexedi.com/nexedi/wendelin.core/blob/master/wcfs/wcfs.go

Major steps: 1__, 2__, 3__, 4__, 5__, 6__, 7__, 8__, 9__, 10__, 11__, 12__,
13__, 14__, 15__, 16__, 17__, 18__, 19__, 20__, 21__.

__ https://lab.nexedi.com/nexedi/wendelin.core/commit/2c152d41?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/e3f2ee2d?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/0e829874?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/a8595565?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/b87edcfe?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/1f2cd49d?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/27df5a3b?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/80153aa5?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/2ab4be93?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/f980471f?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/4430de41?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/6f0cdaff?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/10f7153a?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/fae045cc?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/23362204?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/ceadfcc7?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/1dba3a9a?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/1f866c00?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/e11edc70?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/986cf86e?expanded=1
__ https://lab.nexedi.com/nexedi/wendelin.core/commit/c5e18c74?expanded=1


Kirill Smelkov's avatar
Kirill Smelkov committed
63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116
0.13 (2019-06-18)
-----------------

- Add support for Python 3.7 (commit__).

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/bca5f79e6f

- Add `RAMArray` class which is compatible to `ZBigArray` in API and semantic,
  but stores its data in RAM only (`commit 1`__, 2__).

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/7365979b9d
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/fc9b69d8e1

- Add `lib.xnumpy.structured` - utility to create structured view of an array (`commit 1`__, 2__).

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/6a5dfefaf8
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/32ca80e2d5

- Fix logic to keep `ZBigFileH` in sync with ZODB connection (commit__).

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/d9d6adec1b

- Fix crash on PyVMA deallocation (commit__).

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/d97641d2ba

- Move `py.bench` to pygolang__ so that it can be used not only in
  Wendelin.core (commit__).

  __ https://pypi.org/project/pygolang/
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/318efce0bf

- Enhance `t/qemu-runlinux` - utility that is used to work on combined
  kernel/user-space workloads (`commit 1`__, 2__, 3__, 4__, 5__, 6__).
  This was in particular useful to develop Linux kernel fixes that are needed
  for Wendelin.core 2.0 (`kernel commit 1`__, 2__, 3__, 4__, 5__, 6__, 7__).

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/fe541453f8
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/ccca055cfe
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/6ab952207e
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/a568d6d999
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/208aca62ae
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/89fb89929a

  __ https://git.kernel.org/linus/ad2ba64dd489
  __ https://git.kernel.org/linus/10dce8af3422
  __ https://git.kernel.org/linus/bbd84f33652f
  __ https://git.kernel.org/linus/c5bf68fe0c86
  __ https://git.kernel.org/linus/438ab720c675
  __ https://git.kernel.org/linus/7640682e67b3
  __ https://git.kernel.org/linus/d4b13963f217

- Various bugfixes.

Kirill Smelkov's avatar
Kirill Smelkov committed
117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171
0.12 (2018-04-16)
-----------------

- Update licensing to be in line with whole Nexedi stack (`commit`__). Please
  see https://www.nexedi.com/licensing for details, rationale and options.

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/f11386a4

- Add `ArrayRef` utility to find out for a NumPy array its top-level root
  parent and how to recreate the array as some view of the root;
  this builds the foundation for e.g. sending arrays as references without copy
  in CMFActivity joblib backend
  (`commit 1`__, 2__, 3__).

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/e9d61a89
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/d53371b6
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/450ad804


- Don't crash if during `loadblk()` garbage collection was run twice at tricky
  times (`commit 1`__, 2__).

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/4228d8b6
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/3804cc39

- Don't crash on writeout if previously `storeblk()` resulted in error
  (`commit`__).

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/87bf4908



- Fix `py.bench` and rework it to produce output in Go benchmarking format
  (`commit 1`__, 2__, 3__, 4__, 5__); add benchmarks for handling pagefaults
  (`commit`__).

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/51f252d4
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/074ce24d
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/ed13c3f9
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/fc08766d
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/5a1ed45a
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/3cfc2728

- Use zodbtools/zodburi, if available, to open database by URL
  (`commit`__).

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/f785ac07

- Start to make sure it works with ZODB5 too (`commit 1`__, 2__).

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/808b59b7
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/0dbf3c44

- Various bugfixes.

Kirill Smelkov's avatar
Kirill Smelkov committed
172 173 174 175 176 177 178
0.11 (2017-03-28)
-----------------

- Switch back to using ZBlk0 format by default (`commit`__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/0b68f178

Kirill Smelkov's avatar
Kirill Smelkov committed
179 180 181 182 183 184 185
0.10 (2017-03-16)
-----------------

- Tell the world `dtype=object` is not supported (`commit`__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/e44bd761

Kirill Smelkov's avatar
Kirill Smelkov committed
186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202
0.9 (2017-01-17)
----------------

- Avoid deadlocks via doing `storeblk()` calls with virtmem lock released
  (`commit 1`__, 2__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/8bb7f2f2
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/fb4bfb32

- Don't crash if in `loadblk()` implementation an exception is internally
  raised & caught
  (`commit 1`__, 2__, 3__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/9aa6a5d7
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/61b18a40
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/024c246c

Kirill Smelkov's avatar
Kirill Smelkov committed
203 204 205 206 207 208 209
0.8 (2016-09-28)
----------------

- Do not leak memory when loading data in ZBlk1 format (`commit`__).

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/542917d1

Kirill Smelkov's avatar
Kirill Smelkov committed
210 211 212 213 214 215 216 217 218 219 220 221 222 223
0.7 (2016-07-14)
------------------

- Add support for Python 3.5 (`commit 1`__, 2__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/20115391
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/e6beab19

- Fix bug in pagemap code which could lead to crashes and other issues (`commit`__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/ee9bcd00

- Various bugfixes

Kirill Smelkov's avatar
Kirill Smelkov committed
224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240
0.6 (2016-06-13)
----------------

- Add support for FORTRAN ordering (`commit 1`__, 2__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/ab9ca2df
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/2ca0f076


- Avoid deadlocks via doing `loadblk()` calls with virtmem lock released
  (`commit 1`__, 2__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/f49c11a3
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/0231a65d

- Various bugfixes

Kirill Smelkov's avatar
Kirill Smelkov committed
241 242 243 244 245 246 247 248 249 250 251 252 253
0.5 (2015-10-02)
----------------

- Introduce another storage format, which is optimized for small changes, and
  make it the default.
  (`commit 1`__, 2__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/13c0c17c
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/9ae42085

- Various bugfixes and documentation improvements


Kirill Smelkov's avatar
Kirill Smelkov committed
254 255 256
0.4 (2015-08-19)
----------------

257
- Add support for O(δ) in-place BigArray.append() (commit__)
Kirill Smelkov's avatar
Kirill Smelkov committed
258

259 260 261 262 263
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/1245acc9

- Implement proper multithreading support (commit__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/d53271b9
Kirill Smelkov's avatar
Kirill Smelkov committed
264 265

- Implement proper RAM pages invalidation when backing ZODB objects are changed
266 267 268 269
  from outside (`commit 1`__, 2__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/cb779c7b
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/92bfd03e
Kirill Smelkov's avatar
Kirill Smelkov committed
270 271

- Fix all kind of failures that could happen when ZODB connection changes
272 273 274 275
  worker thread in-between handling requests (`commit 1`__, 2__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/c7c01ce4
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/64d1f40b
Kirill Smelkov's avatar
Kirill Smelkov committed
276 277

- Tox tests now cover usage with FileStorage, ZEO and NEO ZODB storages
278 279 280 281
  (`commit 1`__, 2__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/010eeb35
  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/7fc4ec66
Kirill Smelkov's avatar
Kirill Smelkov committed
282 283 284 285 286

- Various bugfixes



Kirill Smelkov's avatar
Kirill Smelkov committed
287 288 289 290 291 292 293 294 295 296 297 298
0.3 (2015-06-12)
----------------

- Add support for automatic BigArray -> ndarray conversion, so that e.g. the
  following::

    A = BigArray(...)
    numpy.mean(A)       # passing BigArray to plain NumPy function

  either succeeds, or raises MemoryError if not enough address space is
  available to cover whole A. (current limitation is ~ 127TB on linux/amd64)

299 300 301
  (commit__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/00db08d6
Kirill Smelkov's avatar
Kirill Smelkov committed
302 303 304 305

- Various bugfixes (build-fixes, crashes, overflows, etc)


Kirill Smelkov's avatar
Kirill Smelkov committed
306 307 308
0.2 (2015-05-25)
----------------

309 310 311
- Add support for O(1) in-place BigArray.resize() (commit__)

  __ https://lab.nexedi.com/nexedi/wendelin.core/commit/ca064f75
Kirill Smelkov's avatar
Kirill Smelkov committed
312 313 314 315 316 317 318 319

- Various build bugfixes (older systems, non-std python, etc)


0.1 (2015-04-03)
----------------

- Initial release