• Jason Madden's avatar
    Use /dev/fd|/proc/self/fd to get open FDs to close in Popen · 461d6966
    Jason Madden authored
    If those aren't available, use the old brute-force approach. This is
    closer to what CPython does in its C implementation, and is much
    faster.
    
    We don't have to worry about the async signal safe stuff the C code
    does because, guess what, we're running Python code here already
    anyway, so much of it could wind up doing something that's not
    actually safe anyway. Oh well.
    
    Since we depend on Python 3.4 and above now, we can rely on the
    CLOEXEC flag being set by default and not have to manually check
    everything.
    
    This speeds up 2.7 (close_fds defaults to *false* there, so the
    default case doesn't change):
    
    | Benchmark                 | 27_bench_subprocess | 27_bench_subprocess_dirfd     |
    +---------------------------+---------------------+-------------------------------+
    | spawn native no close_fds | 1.81 ms             | 1.79 ms: 1.01x faster (-1%)   |
    | spawn gevent no close_fds | 2.11 ms             | 2.20 ms: 1.04x slower (+4%)   |
    | spawn native close_fds    | 31.0 ms             | 30.2 ms: 1.03x faster (-3%)   |
    | spawn gevent close_fds    | 31.6 ms             | 2.56 ms: 12.31x faster (-92%) |
    
    And it really speeds up 3.7 (close_fds defaults to *true* there, so
    the default case is much faster, and the non-default case is even better):
    
    | Benchmark                 | 37_bench_subprocess | 37_bench_subprocess_dirfd     |
    +---------------------------+---------------------+-------------------------------+
    | spawn native no close_fds | 1.34 ms             | 1.27 ms: 1.06x faster (-6%)   |
    | spawn gevent no close_fds | 117 ms              | 3.05 ms: 38.27x faster (-97%) |
    | spawn native close_fds    | 1.36 ms             | 1.30 ms: 1.04x faster (-4%)   |
    | spawn gevent close_fds    | 32.5 ms             | 3.34 ms: 9.75x faster (-90%)  |
    
    Fixes #1172
    461d6966
To find the state of this project's repository at the time of any of these versions, check out the tags.
CHANGES.rst 24.1 KB