Demo program that shows how to work with ZBigArrays bigger than RAM in size
This shows how to first generate such arrays (in steps, as every transaction change should fit in memory), and then gather data from whole array using C/Fortran/etc code. It shows how to compute mean via NumPy's ndarray.mean() It also shows that e.g. ndarray.var() wants to create temporaries in size of original ndarray and that would fail, because it does not fit into RAM. ndarray.var() should not need to create such temporaries in principle - all it has to do is to first compute mean, and then compute sum (Xi - <X>)^2 in a loop. <X> is scalar, Xi - is just access to original array. ~~~~ So this also show NumPy can be incrementally improved to avoid creating such temporaries, and then it will work.
Showing with 123 additions and 0 deletions