• Kirill Smelkov's avatar
    Demo program that shows how to work with ZBigArrays bigger than RAM in size · 1ee72371
    Kirill Smelkov authored
    This shows how to first generate such arrays (in steps, as every
    transaction change should fit in memory), and then gather data from
    whole array using C/Fortran/etc code.
    It shows how to compute mean via NumPy's ndarray.mean()
    It also shows that e.g. ndarray.var() wants to create temporaries in
    size of original ndarray and that would fail, because it does not fit
    into RAM.
    ndarray.var() should not need to create such temporaries in principle -
    all it has to do is to first compute mean, and then compute
        sum (Xi - <X>)^2
    in a loop.
    <X> is scalar, Xi - is just access to original array.
    So this also show NumPy can be incrementally improved to avoid creating
    such temporaries, and then it will work.
__init__.py 0 Bytes