• Waiman Long's avatar
    locking/rwsem: Add reader-owned state to the owner field · 19c5d690
    Waiman Long authored
    Currently, it is not possible to determine for sure if a reader
    owns a rwsem by looking at the content of the rwsem data structure.
    This patch adds a new state RWSEM_READER_OWNED to the owner field
    to indicate that readers currently own the lock. This enables us to
    address the following 2 issues in the rwsem optimistic spinning code:
    
     1) rwsem_can_spin_on_owner() will disallow optimistic spinning if
        the owner field is NULL which can mean either the readers own
        the lock or the owning writer hasn't set the owner field yet.
        In the latter case, we miss the chance to do optimistic spinning.
    
     2) While a writer is waiting in the OSQ and a reader takes the lock,
        the writer will continue to spin when out of the OSQ in the main
        rwsem_optimistic_spin() loop as the owner field is NULL wasting
        CPU cycles if some of readers are sleeping.
    
    Adding the new state will allow optimistic spinning to go forward as
    long as the owner field is not RWSEM_READER_OWNED and the owner is
    running, if set, but stop immediately when that state has been reached.
    
    On a 4-socket Haswell machine running on a 4.6-rc1 based kernel, the
    fio test with multithreaded randrw and randwrite tests on the same
    file on a XFS partition on top of a NVDIMM were run, the aggregated
    bandwidths before and after the patch were as follows:
    
      Test      BW before patch     BW after patch  % change
      ----      ---------------     --------------  --------
      randrw         988 MB/s          1192 MB/s      +21%
      randwrite     1513 MB/s          1623 MB/s      +7.3%
    
    The perf profile of the rwsem_down_write_failed() function in randrw
    before and after the patch were:
    
       19.95%  5.88%  fio  [kernel.vmlinux]  [k] rwsem_down_write_failed
       14.20%  1.52%  fio  [kernel.vmlinux]  [k] rwsem_down_write_failed
    
    The actual CPU cycles spend in rwsem_down_write_failed() dropped from
    5.88% to 1.52% after the patch.
    
    The xfstests was also run and no regression was observed.
    Signed-off-by: default avatarWaiman Long <Waiman.Long@hpe.com>
    Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
    Acked-by: default avatarJason Low <jason.low2@hp.com>
    Acked-by: default avatarDavidlohr Bueso <dave@stgolabs.net>
    Cc: Andrew Morton <akpm@linux-foundation.org>
    Cc: Dave Chinner <david@fromorbit.com>
    Cc: Douglas Hatch <doug.hatch@hpe.com>
    Cc: Linus Torvalds <torvalds@linux-foundation.org>
    Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
    Cc: Peter Hurley <peter@hurleysoftware.com>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Cc: Scott J Norton <scott.norton@hpe.com>
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Link: http://lkml.kernel.org/r/1463534783-38814-2-git-send-email-Waiman.Long@hpe.comSigned-off-by: default avatarIngo Molnar <mingo@kernel.org>
    19c5d690
rwsem.c 3.82 KB