-
Kirill Smelkov authored
A lot of times data in blocks come shorter than block size and the rest of the memory page is zeros (because it was pre-filled zeros by OS when page was allocated). Do a simple heuristic and trim those trailing zeros and not store them into DB. With this change size of DB created by running bigfile and bigarray tests changes as following: bigfile bigarray old 145M 35M new 24K 6M Trimming trailing zeros is currently done with str.rstrip('\0') which creates a copy. When/if needed this could be optimized to work in-place.
f696ce92