Skip to content

L
linux
Switch branch/tag
  1. 28 May, 2014 7 commits
  3. 27 May, 2014 5 commits
  5. 26 May, 2014 2 commits
  7. 23 May, 2014 2 commits
  9. 22 May, 2014 1 commit
    • Jens Axboe's avatar
      blk-mq: split make request handler for multi and single queue · 07068d5b
      Jens Axboe authored 
      We want slightly different behavior from them:

- On single queue devices, we currently use the per-process plug
  for deferred IO and for merging.

- On multi queue devices, we don't use the per-process plug, but
  we want to go straight to hardware for SYNC IO.

Split blk_mq_make_request() into a blk_sq_make_request() for single
queue devices, and retain blk_mq_make_request() for multi queue
devices. Then we don't need multiple checks for q->nr_hw_queues
in the request mapping.
Signed-off-by: default avatarJens Axboe <axboe@fb.com>
      07068d5b
  11. 21 May, 2014 2 commits
  13. 20 May, 2014 5 commits
  15. 19 May, 2014 5 commits
  17. 13 May, 2014 1 commit
    • Jens Axboe's avatar
      blk-mq: improve support for shared tags maps · 0d2602ca
      Jens Axboe authored 
      This adds support for active queue tracking, meaning that the
blk-mq tagging maintains a count of active users of a tag set.
This allows us to maintain a notion of fairness between users,
so that we can distribute the tag depth evenly without starving
some users while allowing others to try unfair deep queues.

If sharing of a tag set is detected, each hardware queue will
track the depth of its own queue. And if this exceeds the total
depth divided by the number of active queues, the user is actively
throttled down.

The active queue count is done lazily to avoid bouncing that data
between submitter and completer. Each hardware queue gets marked
active when it allocates its first tag, and gets marked inactive
when 1) the last tag is cleared, and 2) the queue timeout grace
period has passed.
Signed-off-by: default avatarJens Axboe <axboe@fb.com>
      0d2602ca
  19. 10 May, 2014 5 commits
  21. 09 May, 2014 5 commits
    • Jens Axboe's avatar
      block: only calculate part_in_flight() once · 7276d02e
      Jens Axboe authored 
      We first check if we have inflight IO, then retrieve that
same number again. Usually this isn't that costly since the
chance of having the data dirtied in between is small, but
there's no reason for calling part_in_flight() twice.
Signed-off-by: default avatarJens Axboe <axboe@fb.com>
      7276d02e
    • Jens Axboe's avatar
      blk-mq: fix race in IO start accounting · cf4b50af
      Jens Axboe authored 
      Commit c6d600c6 opened up a small race where we could attempt to
account IO completion on a request, racing with IO start accounting.
Fix this up by ensuring that we've accounted for IO start before
inserting the request.
Signed-off-by: default avatarJens Axboe <axboe@fb.com>
      cf4b50af
    • Jens Axboe's avatar
      blk-mq: use sparser tag layout for lower queue depth · 59d13bf5
      Jens Axboe authored 
      For best performance, spreading tags over multiple cachelines
makes the tagging more efficient on multicore systems. But since
we have 8 * sizeof(unsigned long) tags per cacheline, we don't
always get a nice spread.

Attempt to spread the tags over at least 4 cachelines, using fewer
number of bits per unsigned long if we have to. This improves
tagging performance in setups with 32-128 tags. For higher depths,
the spread is the same as before (BITS_PER_LONG tags per cacheline).
Signed-off-by: default avatarJens Axboe <axboe@fb.com>
      59d13bf5
    • Jens Axboe's avatar
      blk-mq: implement new and more efficient tagging scheme · 4bb659b1
      Jens Axboe authored 
      blk-mq currently uses percpu_ida for tag allocation. But that only
works well if the ratio between tag space and number of CPUs is
sufficiently high. For most devices and systems, that is not the
case. The end result if that we either only utilize the tag space
partially, or we end up attempting to fully exhaust it and run
into lots of lock contention with stealing between CPUs. This is
not optimal.

This new tagging scheme is a hybrid bitmap allocator. It uses
two tricks to both be SMP friendly and allow full exhaustion
of the space:

1) We cache the last allocated (or freed) tag on a per blk-mq
   software context basis. This allows us to limit the space
   we have to search. The key element here is not caching it
   in the shared tag structure, otherwise we end up dirtying
   more shared cache lines on each allocate/free operation.

2) The tag space is split into cache line sized groups, and
   each context will start off randomly in that space. Even up
   to full utilization of the space, this divides the tag users
   efficiently into cache line groups, avoiding dirtying the same
   one both between allocators and between allocator and freeer.

This scheme shows drastically better behaviour, both on small
tag spaces but on large ones as well. It has been tested extensively
to show better performance for all the cases blk-mq cares about.
Signed-off-by: default avatarJens Axboe <axboe@fb.com>
      4bb659b1
    • Christoph Hellwig's avatar
      blk-mq: initialize struct request fields individually · af76e555
      Christoph Hellwig authored 
      This allows us to avoid a non-atomic memset over ->atomic_flags as well
as killing lots of duplicate initializations.
Signed-off-by: default avatarChristoph Hellwig <hch@lst.de>
Signed-off-by: default avatarJens Axboe <axboe@fb.com>
      af76e555
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7