• Levin Zimmermann's avatar
    ZBigFile: Add ZBlk format option 'h' (heuristic) · 850703f8
    Levin Zimmermann authored
    There are two formats to save data with a ZBigFile: ZBlk0 and ZBlk1.
    They differ by adjusting the ratio between access-time and growing
    disk-space, where ZBlk1 is better regarding to disk space, while ZBlk0
    has a better access-time. Wendelin.core users may not always know yet or
    care which format fits better for their data. In this case it may be
    easier for users to just let the program automatically select the ZBlk
    format. With this patch and the new 'h' (for heuristic) option of the
    'ZBlk' argument of ZBigFile, this is now possible. The 'h' option isn't
    really a new ZBlk format in itself, but it just tries to automatically
    select the best ZBlk format option according to the characteristics
    of the changes that the user applies to the ZBigFile.
    
    With this patch comes a test (bigfile/tests/test-zblk-fmt) that creates
    benchmarks for different combinations and zblk formats. The test aims
    at checking that the 'heuristic' format behaves mostly as good as the
    explicitly set formats:
    
    Use only a very small change size, so that heuristic always uses ZBlk1
    ---------------------------------------------
    ---------------------------------------------
    Set change_size_set to 20
    Set change_count to 500
    Set arrsize to 1000000
    
    Run tests with format h:
    
    	ZODB storage size: 20.388751 MB
    	Access time: 3.83012294769
    
    Run tests with format ZBlk0:
    
    	ZODB storage size: 1064.636095 MB
    	Access time: 3.63488578796
    
    Run tests with format ZBlk1:
    
    	ZODB storage size: 18.59421 MB
    	Access time: 3.93918204308
    
    ---------------------------------------------
    ---------------------------------------------
    
    Use only a very big change size, so that heuristic always uses ZBlk0
    ---------------------------------------------
    ---------------------------------------------
    Set change_size_set to 200000
    Set change_count to 500
    Set arrsize to 1000000
    
    Run tests with format h:
    
    	ZODB storage size: 2113.32534 MB
    	Access time: 3.79592084885
    
    Run tests with format ZBlk0:
    
    	ZODB storage size: 2113.254473 MB
    	Access time: 3.76431703568
    
    Run tests with format ZBlk1:
    
    	ZODB storage size: 1651.236315 MB
    	Access time: 4.11528992653
    
    ---------------------------------------------
    ---------------------------------------------
    
    Mix between change size so that heuristic switches between ZBlk0 and ZBlk1
    ---------------------------------------------
    ---------------------------------------------
    Set change_size_set to 20,200000
    Set change_count to 500
    Set arrsize to 1000000
    
    Run tests with format h:
    
    	ZODB storage size: 820.17736 MB
    	Access time: 3.85217094421
    
    Run tests with format ZBlk0:
    
    	ZODB storage size: 1576.361791 MB
    	Access time: 3.65322995186
    
    Run tests with format ZBlk1:
    
    	ZODB storage size: 815.323463 MB
    	Access time: 3.96401691437
    
    ---------------------------------------------
    ---------------------------------------------
    850703f8
zblk.go 14.5 KB