• Bart Van Assche's avatar
    block: Improve performance for BLK_MQ_F_BLOCKING drivers · 65a558f6
    Bart Van Assche authored
    blk_mq_run_queue() runs the queue asynchronously if BLK_MQ_F_BLOCKING
    has been set. This is suboptimal since running the queue asynchronously
    is slower than running the queue synchronously. This patch modifies
    blk_mq_run_queue() as follows if BLK_MQ_F_BLOCKING has been set:
    - Run the queue synchronously if it is allowed to sleep.
    - Run the queue asynchronously if it is not allowed to sleep.
    Additionally, blk_mq_run_hw_queue(hctx, false) calls are modified into
    blk_mq_run_hw_queue(hctx, hctx->flags & BLK_MQ_F_BLOCKING) if the caller
    may be invoked from atomic context.
    
    The following caller chains have been reviewed:
    
    blk_mq_run_hw_queue(hctx, false)
      blk_mq_get_tag()      /* may sleep, hence the functions it calls may also sleep */
      blk_execute_rq()             /* may sleep */
      blk_mq_run_hw_queues(q, async=false)
        blk_freeze_queue_start()   /* may sleep */
        blk_mq_requeue_work()      /* may sleep */
        scsi_kick_queue()
          scsi_requeue_run_queue() /* may sleep */
          scsi_run_host_queues()
            scsi_ioctl_reset()     /* may sleep */
      blk_mq_insert_requests(hctx, ctx, list, run_queue_async=false)
        blk_mq_dispatch_plug_list(plug, from_sched=false)
          blk_mq_flush_plug_list(plug, from_schedule=false)
            __blk_flush_plug(plug, from_schedule=false)
    	blk_add_rq_to_plug()
    	  blk_mq_submit_bio()  /* may sleep if REQ_NOWAIT has not been set */
      blk_mq_plug_issue_direct()
        blk_mq_flush_plug_list()   /* see above */
      blk_mq_dispatch_plug_list(plug, from_sched=false)
        blk_mq_flush_plug_list()   /* see above */
      blk_mq_try_issue_directly()
        blk_mq_submit_bio()        /* may sleep if REQ_NOWAIT has not been set */
      blk_mq_try_issue_list_directly(hctx, list)
        blk_mq_insert_requests() /* see above */
    
    Cc: Christoph Hellwig <hch@lst.de>
    Cc: Ming Lei <ming.lei@redhat.com>
    Signed-off-by: default avatarBart Van Assche <bvanassche@acm.org>
    Link: https://lore.kernel.org/r/20230721172731.955724-4-bvanassche@acm.orgSigned-off-by: default avatarJens Axboe <axboe@kernel.dk>
    65a558f6
scsi_lib.c 87.4 KB