• Jason Madden's avatar
    Rework bench_spawn to be perf based for more reliable numbers · 3c3e7450
    Jason Madden authored
    I get this on 3.6.4:
    
    .....................
    eventlet spawn: Mean +- std dev: 12.2 us +- 1.2 us
    .....................
    eventlet sleep: Mean +- std dev: 16.3 us +- 0.8 us
    .....................
    gevent spawn: Mean +- std dev: 13.1 us +- 1.1 us
    .....................
    gevent sleep: Mean +- std dev: 10.4 us +- 0.8 us
    .....................
    WARNING: the benchmark result may be unstable
    * the standard deviation (2.73 us) is 17% of the mean (16.1 us)
    
    Try to rerun the benchmark with more runs, values and/or loops.
    Run 'python -m perf system tune' command to reduce the system jitter.
    Use perf stats, perf dump and perf hist to analyze results.
    Use --quiet option to hide these warnings.
    
    geventpool spawn: Mean +- std dev: 16.1 us +- 2.7 us
    .....................
    geventpool sleep: Mean +- std dev: 11.2 us +- 0.5 us
    .....................
    geventraw spawn: Mean +- std dev: 4.95 us +- 0.42 us
    .....................
    geventraw sleep: Mean +- std dev: 7.34 us +- 0.28 us
    .....................
    none spawn: Mean +- std dev: 1.98 us +- 0.05 us
    .....................
    geventpool join: Mean +- std dev: 6.59 us +- 0.25 us
    .....................
    WARNING: the benchmark result may be unstable
    * the standard deviation (1.28 us) is 10% of the mean (12.7 us)
    
    Try to rerun the benchmark with more runs, values and/or loops.
    Run 'python -m perf system tune' command to reduce the system jitter.
    Use perf stats, perf dump and perf hist to analyze results.
    Use --quiet option to hide these warnings.
    
    eventlet spawn kwarg: Mean +- std dev: 12.7 us +- 1.3 us
    .....................
    gevent spawn kwarg: Mean +- std dev: 14.6 us +- 1.2 us
    .....................
    WARNING: the benchmark result may be unstable
    * the standard deviation (2.81 us) is 17% of the mean (17.0 us)
    
    Try to rerun the benchmark with more runs, values and/or loops.
    Run 'python -m perf system tune' command to reduce the system jitter.
    Use perf stats, perf dump and perf hist to analyze results.
    Use --quiet option to hide these warnings.
    
    geventpool spawn kwarg: Mean +- std dev: 17.0 us +- 2.8 us
    .....................
    geventraw spawn kwarg: Mean +- std dev: 6.11 us +- 0.45 us
    .....................
    none spawn kwarg: Mean +- std dev: 2.22 us +- 0.07 us
    
    And this on 2.7.14:
    
    .....................
    WARNING: the benchmark result may be unstable
    * the standard deviation (2.10 us) is 11% of the mean (18.4 us)
    
    Try to rerun the benchmark with more runs, values and/or loops.
    Run 'python -m perf system tune' command to reduce the system jitter.
    Use perf stats, perf dump and perf hist to analyze results.
    Use --quiet option to hide these warnings.
    
    eventlet spawn: Mean +- std dev: 18.4 us +- 2.1 us
    .....................
    eventlet sleep: Mean +- std dev: 23.1 us +- 0.8 us
    .....................
    WARNING: the benchmark result may be unstable
    * the standard deviation (4.39 us) is 25% of the mean (17.3 us)
    
    Try to rerun the benchmark with more runs, values and/or loops.
    Run 'python -m perf system tune' command to reduce the system jitter.
    Use perf stats, perf dump and perf hist to analyze results.
    Use --quiet option to hide these warnings.
    
    gevent spawn: Mean +- std dev: 17.3 us +- 4.4 us
    .....................
    gevent sleep: Mean +- std dev: 10.3 us +- 0.5 us
    .....................
    WARNING: the benchmark result may be unstable
    * the standard deviation (3.92 us) is 16% of the mean (24.7 us)
    
    Try to rerun the benchmark with more runs, values and/or loops.
    Run 'python -m perf system tune' command to reduce the system jitter.
    Use perf stats, perf dump and perf hist to analyze results.
    Use --quiet option to hide these warnings.
    
    geventpool spawn: Mean +- std dev: 24.7 us +- 3.9 us
    .....................
    geventpool sleep: Mean +- std dev: 13.5 us +- 0.9 us
    .....................
    geventraw spawn: Mean +- std dev: 6.91 us +- 0.49 us
    .....................
    geventraw sleep: Mean +- std dev: 8.95 us +- 0.30 us
    .....................
    none spawn: Mean +- std dev: 2.21 us +- 0.04 us
    .....................
    geventpool join: Mean +- std dev: 7.93 us +- 0.28 us
    .....................
    eventlet spawn kwarg: Mean +- std dev: 17.4 us +- 1.3 us
    .....................
    WARNING: the benchmark result may be unstable
    * the standard deviation (4.11 us) is 27% of the mean (15.1 us)
    * the maximum (24.2 us) is 60% greater than the mean (15.1 us)
    
    Try to rerun the benchmark with more runs, values and/or loops.
    Run 'python -m perf system tune' command to reduce the system jitter.
    Use perf stats, perf dump and perf hist to analyze results.
    Use --quiet option to hide these warnings.
    
    gevent spawn kwarg: Mean +- std dev: 15.1 us +- 4.1 us
    .....................
    WARNING: the benchmark result may be unstable
    * the standard deviation (4.74 us) is 18% of the mean (26.8 us)
    
    Try to rerun the benchmark with more runs, values and/or loops.
    Run 'python -m perf system tune' command to reduce the system jitter.
    Use perf stats, perf dump and perf hist to analyze results.
    Use --quiet option to hide these warnings.
    
    geventpool spawn kwarg: Mean +- std dev: 26.8 us +- 4.7 us
    .....................
    WARNING: the benchmark result may be unstable
    * the standard deviation (959 ns) is 12% of the mean (8.00 us)
    
    Try to rerun the benchmark with more runs, values and/or loops.
    Run 'python -m perf system tune' command to reduce the system jitter.
    Use perf stats, perf dump and perf hist to analyze results.
    Use --quiet option to hide these warnings.
    
    geventraw spawn kwarg: Mean +- std dev: 8.00 us +- 0.96 us
    .....................
    none spawn kwarg: Mean +- std dev: 2.48 us +- 0.06 us
    
    Partial PyPy results:
    
    .........
    WARNING: the benchmark result may be unstable
    * the standard deviation (5.77 us) is 18% of the mean (32.5 us)
    * the maximum (52.2 us) is 61% greater than the mean (32.5 us)
    
    Try to rerun the benchmark with more runs, values and/or loops.
    Run 'python -m perf system tune' command to reduce the system jitter.
    Use perf stats, perf dump and perf hist to analyze results.
    Use --quiet option to hide these warnings.
    
    eventlet spawn: Mean +- std dev: 32.5 us +- 5.8 us
    .........
    eventlet sleep: Mean +- std dev: 39.9 us +- 2.4 us
    .........
    WARNING: the benchmark result may be unstable
    * the standard deviation (8.90 us) is 43% of the mean (20.6 us)
    * the minimum (8.50 us) is 59% smaller than the mean (20.6 us)
    * the maximum (41.5 us) is 102% greater than the mean (20.6 us)
    
    Try to rerun the benchmark with more runs, values and/or loops.
    Run 'python -m perf system tune' command to reduce the system jitter.
    Use perf stats, perf dump and perf hist to analyze results.
    Use --quiet option to hide these warnings.
    
    gevent spawn: Mean +- std dev: 20.6 us +- 8.9 us
    .........
    gevent sleep: Mean +- std dev: 4.20 us +- 0.21 us
    .........
    WARNING: the benchmark result may be unstable
    * the standard deviation (8.52 us) is 50% of the mean (17.2 us)
    * the minimum (7.74 us) is 55% smaller than the mean (17.2 us)
    * the maximum (58.1 us) is 238% greater than the mean (17.2 us)
    
    Try to rerun the benchmark with more runs, values and/or loops.
    Run 'python -m perf system tune' command to reduce the system jitter.
    Use perf stats, perf dump and perf hist to analyze results.
    Use --quiet option to hide these warnings.
    
    geventpool spawn: Mean +- std dev: 17.2 us +- 8.5 us
    .........
    WARNING: the benchmark result may be unstable
    * the standard deviation (968 ns) is 18% of the mean (5.26 us)
    * the maximum (10.5 us) is 99% greater than the mean (5.26 us)
    
    Try to rerun the benchmark with more runs, values and/or loops.
    Run 'python -m perf system tune' command to reduce the system jitter.
    Use perf stats, perf dump and perf hist to analyze results.
    Use --quiet option to hide these warnings.
    
    geventpool sleep: Mean +- std dev: 5.26 us +- 0.97 us
    ..........
    WARNING: the benchmark result may be unstable
    * the standard deviation (1.45 us) is 52% of the mean (2.80 us)
    * the maximum (5.50 us) is 96% greater than the mean (2.80 us)
    
    Try to rerun the benchmark with more runs, values and/or loops.
    Run 'python -m perf system tune' command to reduce the system jitter.
    Use perf stats, perf dump and perf hist to analyze results.
    Use --quiet option to hide these warnings.
    
    geventraw spawn: Mean +- std dev: 2.80 us +- 1.45 us
    .........
    geventraw sleep: Mean +- std dev: 4.24 us +- 0.27 us
    .........
    none spawn: Mean +- std dev: 1.10 us +- 0.06 us
    3c3e7450
bench_spawn.py 5.1 KB