• Kirill Smelkov's avatar
    bigarray: Teach it how to automatically convert to ndarray (if enough address space is available) · 00db08d6
    Kirill Smelkov authored
    BigArrays can be big - up to 2^64 bytes, and thus in general it is not
    possible to represent whole BigArray as ndarray view, because address
    space is usually smaller on 64bit architectures.
    
    However users often try to pass BigArrays to numpy functions as-is, and
    numpy finds a way to convert, or start converting, BigArray to ndarray -
    via detecting it as a sequence, and extracting elements one-by-one.
    Which is slooooow.
    
    Because of the above, we provide users a well-defined service:
    - if virtual address space is available - we succeed at creating ndarray
      view for whole BigArray, without delay and copying.
    - if not - we report properly the error and give hint how BigArrays have
      to be processed in chunks.
    
    Verifying that big BigArrays cannot be converted to ndarray also tests
    for behaviour and issues fixed in last 5 patches.
    
    /cc @Tyagov
    /cc @klaus
    00db08d6
__init__.py 14 KB