• Huy Nguyen's avatar
    net/dcb: Add dcbnl buffer attribute · e549f6f9
    Huy Nguyen authored
    In this patch, we add dcbnl buffer attribute to allow user
    change the NIC's buffer configuration such as priority
    to buffer mapping and buffer size of individual buffer.
    
    This attribute combined with pfc attribute allows advanced user to
    fine tune the qos setting for specific priority queue. For example,
    user can give dedicated buffer for one or more priorities or user
    can give large buffer to certain priorities.
    
    The dcb buffer configuration will be controlled by lldptool.
    lldptool -T -i eth2 -V BUFFER prio 0,2,5,7,1,2,3,6
      maps priorities 0,1,2,3,4,5,6,7 to receive buffer 0,2,5,7,1,2,3,6
    lldptool -T -i eth2 -V BUFFER size 87296,87296,0,87296,0,0,0,0
      sets receive buffer size for buffer 0,1,2,3,4,5,6,7 respectively
    
    After discussion on mailing list with Jakub, Jiri, Ido and John, we agreed to
    choose dcbnl over devlink interface since this feature is intended to set
    port attributes which are governed by the netdev instance of that port, where
    devlink API is more suitable for global ASIC configurations.
    
    We present an use case scenario where dcbnl buffer attribute configured
    by advance user helps reduce the latency of messages of different sizes.
    
    Scenarios description:
    On ConnectX-5, we run latency sensitive traffic with
    small/medium message sizes ranging from 64B to 256KB and bandwidth sensitive
    traffic with large messages sizes 512KB and 1MB. We group small, medium,
    and large message sizes to their own pfc enables priorities as follow.
      Priorities 1 & 2 (64B, 256B and 1KB)
      Priorities 3 & 4 (4KB, 8KB, 16KB, 64KB, 128KB and 256KB)
      Priorities 5 & 6 (512KB and 1MB)
    
    By default, ConnectX-5 maps all pfc enabled priorities to a single
    lossless fixed buffer size of 50% of total available buffer space. The
    other 50% is assigned to lossy buffer. Using dcbnl buffer attribute,
    we create three equal size lossless buffers. Each buffer has 25% of total
    available buffer space. Thus, the lossy buffer size reduces to 25%. Priority
    to lossless  buffer mappings are set as follow.
      Priorities 1 & 2 on lossless buffer #1
      Priorities 3 & 4 on lossless buffer #2
      Priorities 5 & 6 on lossless buffer #3
    
    We observe improvements in latency for small and medium message sizes
    as follows. Please note that the large message sizes bandwidth performance is
    reduced but the total bandwidth remains the same.
      256B message size (42 % latency reduction)
      4K message size (21% latency reduction)
      64K message size (16% latency reduction)
    
    CC: Ido Schimmel <idosch@idosch.org>
    CC: Jakub Kicinski <jakub.kicinski@netronome.com>
    CC: Jiri Pirko <jiri@resnulli.us>
    CC: Or Gerlitz <gerlitz.or@gmail.com>
    CC: Parav Pandit <parav@mellanox.com>
    CC: Aron Silverton <aron.silverton@oracle.com>
    Signed-off-by: default avatarHuy Nguyen <huyn@mellanox.com>
    Reviewed-by: default avatarParav Pandit <parav@mellanox.com>
    Signed-off-by: default avatarSaeed Mahameed <saeedm@mellanox.com>
    e549f6f9
dcbnl.c 49.4 KB