• Chuck Lever's avatar
    RDMA/core: Trace points for diagnosing completion queue issues · 3e5901cb
    Chuck Lever authored
    Sample trace events:
    
       kworker/u29:0-300   [007]   120.042217: cq_alloc:             cq.id=4 nr_cqe=161 comp_vector=2 poll_ctx=WORKQUEUE
              <idle>-0     [002]   120.056292: cq_schedule:          cq.id=4
        kworker/2:1H-482   [002]   120.056402: cq_process:           cq.id=4 wake-up took 109 [us] from interrupt
        kworker/2:1H-482   [002]   120.056407: cq_poll:              cq.id=4 requested 16, returned 1
              <idle>-0     [002]   120.067503: cq_schedule:          cq.id=4
        kworker/2:1H-482   [002]   120.067537: cq_process:           cq.id=4 wake-up took 34 [us] from interrupt
        kworker/2:1H-482   [002]   120.067541: cq_poll:              cq.id=4 requested 16, returned 1
              <idle>-0     [002]   120.067657: cq_schedule:          cq.id=4
        kworker/2:1H-482   [002]   120.067672: cq_process:           cq.id=4 wake-up took 15 [us] from interrupt
        kworker/2:1H-482   [002]   120.067674: cq_poll:              cq.id=4 requested 16, returned 1
    
     ...
    
             systemd-1     [002]   122.392653: cq_schedule:          cq.id=4
        kworker/2:1H-482   [002]   122.392688: cq_process:           cq.id=4 wake-up took 35 [us] from interrupt
        kworker/2:1H-482   [002]   122.392693: cq_poll:              cq.id=4 requested 16, returned 16
        kworker/2:1H-482   [002]   122.392836: cq_poll:              cq.id=4 requested 16, returned 16
        kworker/2:1H-482   [002]   122.392970: cq_poll:              cq.id=4 requested 16, returned 16
        kworker/2:1H-482   [002]   122.393083: cq_poll:              cq.id=4 requested 16, returned 16
        kworker/2:1H-482   [002]   122.393195: cq_poll:              cq.id=4 requested 16, returned 3
    
    Several features to note in this output:
     - The WCE count and context type are reported at allocation time
     - The CPU and kworker for each CQ is evident
     - The CQ's restracker ID is tagged on each trace event
     - CQ poll scheduling latency is measured
     - Details about how often single completions occur versus multiple
       completions are evident
     - The cost of the ULP's completion handler is recorded
    
    Link: https://lore.kernel.org/r/20191218201815.30584.3481.stgit@manet.1015granger.netSigned-off-by: default avatarChuck Lever <chuck.lever@oracle.com>
    Reviewed-by: default avatarParav Pandit <parav@mellanox.com>
    Signed-off-by: default avatarJason Gunthorpe <jgg@mellanox.com>
    3e5901cb
trace.c 311 Bytes