• Kirill Smelkov's avatar
    dropbear: Don't waste transfer time in favour of small-memory machines defaults · 605e564b
    Kirill Smelkov authored
    I recently discovered that dropbear, compared to openssh over same link
    with same host and user keys, transfers files several times slower -
    sometimes up to an order of magnitude slower. This equally applies to
    both fast LAN IPv4 networks (tried on 1Gbps) and relatively slow "over
    re6stnet" networks with geographically separated client and server.
    
    The problem turned out to be dropbear has very small receive window size
    default:
    
        https://github.com/mkj/dropbear/blob/DROPBEAR_0.53.1/options.h#L258
    
        ---- 8< ----
        /* Window size limits. These tend to be a trade-off between memory
           usage and network performance: */
        /* Size of the network receive window. This amount of memory is allocated
           as a per-channel receive buffer. Increasing this value can make a
           significant difference to network performance. 24kB was empirically
           chosen for a 100mbit ethernet network. The value can be altered at
           runtime with the -W argument. */
        #ifndef DEFAULT_RECV_WINDOW
        #define DEFAULT_RECV_WINDOW 24576
        #endif
        ---- 8< ----
    
    this is maybe appropriate for embedded systems, but small receive window
    affects networking throughput a lot: e.g. I've tried to run several
    file downloading benchmarks and changing receive window size from
    default 24K to 1M (this is maximum according to dropbear documentation) on
    client side made the downloading several times faster and comparable to
    downloading speed with using openssh as client.
    
    Dropbear has -W option for setting receive window size at runtime, so we could
    go through all places which use dropbear and add appropriate `-W ...` there.
    But I think fixing dropbear itself to have more sane defaults for
    not-very-memory-constrained devices (= devices we run slapos on) is preferable
    - this way all services will automatically benefit from transfer speedup
    without taking action on their side with only doing recompilation/redeployment.
    
    Besides changing only recv window size at runtime breaks compatibility with
    openssh: if we only do `-W 1M` on server and try to upload data with openssh as
    client, dropbear complains
    
        [3302] Apr 17 23:10:06 Exit (slapuser2): Bad packet size 32777
    
    and connection terminates. Thus RECV_MAX_PAYLOAD_LEN increase is also
    required, which cannot be done via option at runtime:
    
        https://github.com/mkj/dropbear/blob/DROPBEAR_0.53.1/options.h#L268
    
        ---- 8< ----
        /* Maximum size of a received SSH data packet - this _MUST_ be >= 32768
           in order to interoperate with other implementations */
        #ifndef RECV_MAX_PAYLOAD_LEN
        #define RECV_MAX_PAYLOAD_LEN 32768
        #endif
        ---- 8< ----
    
    So let's increase DEFAULT_RECV_WINDOW to 1M and RECV_MAX_PAYLOAD_LEN
    appropriately (experimentally found that at 512K the complain goes
    away).
    
    Below is rough benchmark before and after the change (timings are not
    very precise, because machines are otherwise busy and network
    fluctuates, but I think they are reasonable in showing we have at least
    several times improvement):
    
                                            before  after
    
        download 1M (re6stnet, slow link)   ~12s    ~4s
        upload   1M (re6stnet, slow link)   ~12s    ~4s
    
        download 100M (1Gbps LAN, IPv4)     ~12s    ~4s
        upload   100M (1Gbps LAN, IPv4)     ~12s    ~4s
    
            legend:
                client=dbclient
                server=dropbear
    
    NOTE the original problem equally applies to latest dropbear master, so it is
        not because we are running old 0.53.1 ( from 2011 ! )
    
    NOTE wrt transfer speed openssh is still faster - not an order of
        magnitude, but somewhat.  Also openssh uses much less CPU compared to
        dropbear.
    
    /cc @Tyagov, @klaus
    
    @nexedi, why at all we use dropbear in the first place instead of openssh?
    
    /reviewed-by @rafael
    /reviewed-on nexedi/slapos!68
    605e564b
buildout.cfg 1.77 KB