• Jim Fulton's avatar
    Avoid seeking on import (#188) · 2115c90a
    Jim Fulton authored
    * Don't seek on imports
    
    (other than the possible seek for custom importers)
    
    We were seeking to handle blob markers. This has two major drawbacks:
    
    1. It wasn't possible to use a non-seekable file.  A use case for
       export/import is to copy database data.  An intermediate file, and
       associated I/O, could be avoided using a pipe, but pipes aren't
       seekable.
    
    2. Seeks cause file-buffer data to be discarded, making IO far more
       expensive.
    
    We didn't really need blob markers, because the preceeding blob data
    records serve as markers.  (Now we're stuck with them for backward
    compatibility.)
    
    * Make cp's buffer size larger and configurable.
    
    * Use the storage temprary directory when importing blobs
    
    To avoid an extra copy.
    
    Also, allow the copy (cp) buffer sie to be overridden on export.
    
    (I see no obvious way to plumb it on import. :( )
    
    * Oops, need to use a binary literal (Python 3)
    
    * Respond to PR comments
    2115c90a
ExportImport.py 6.82 KB