1. 10 Jan, 2011 1 commit
  2. 21 Dec, 2010 2 commits
    • Dave Chinner's avatar
      xfs: convert grant head manipulations to lockless algorithm · d0eb2f38
      Dave Chinner authored
      The only thing that the grant lock remains to protect is the grant head
      manipulations when adding or removing space from the log. These calculations
      are already based on atomic variables, so we can already update them safely
      without locks. However, the grant head manpulations require atomic multi-step
      calculations to be executed, which the algorithms currently don't allow.
      
      To make these multi-step calculations atomic, convert the algorithms to
      compare-and-exchange loops on the atomic variables. That is, we sample the old
      value, perform the calculation and use atomic64_cmpxchg() to attempt to update
      the head with the new value. If the head has not changed since we sampled it,
      it will succeed and we are done. Otherwise, we rerun the calculation again from
      a new sample of the head.
      
      This allows us to remove the grant lock from around all the grant head space
      manipulations, and that effectively removes the grant lock from the log
      completely. Hence we can remove the grant lock completely from the log at this
      point.
      Signed-off-by: default avatarDave Chinner <dchinner@redhat.com>
      Reviewed-by: default avatarChristoph Hellwig <hch@lst.de>
      d0eb2f38
    • Dave Chinner's avatar
      xfs: introduce new locks for the log grant ticket wait queues · 3f16b985
      Dave Chinner authored
      The log grant ticket wait queues are currently protected by the log
      grant lock.  However, the queues are functionally independent from
      each other, and operations on them only require serialisation
      against other queue operations now that all of the other log
      variables they use are atomic values.
      
      Hence, we can make them independent of the grant lock by introducing
      new locks just to protect the lists operations. because the lists
      are independent, we can use a lock per list and ensure that reserve
      and write head queuing do not contend.
      
      To ensure forced shutdowns work correctly in conjunction with the
      new fast paths, ensure that we check whether the log has been shut
      down in the grant functions once we hold the relevant spin locks but
      before we go to sleep. This is needed to co-ordinate correctly with
      the wakeups that are issued on the ticket queues so we don't leave
      any processes sleeping on the queues during a shutdown.
      Signed-off-by: default avatarDave Chinner <dchinner@redhat.com>
      Reviewed-by: default avatarChristoph Hellwig <hch@lst.de>
      3f16b985
  3. 03 Dec, 2010 1 commit
  4. 21 Dec, 2010 1 commit
    • Dave Chinner's avatar
      xfs: convert l_tail_lsn to an atomic variable. · 1c3cb9ec
      Dave Chinner authored
      log->l_tail_lsn is currently protected by the log grant lock. The
      lock is only needed for serialising readers against writers, so we
      don't really need the lock if we make the l_tail_lsn variable an
      atomic. Converting the l_tail_lsn variable to an atomic64_t means we
      can start to peel back the grant lock from various operations.
      
      Also, provide functions to safely crack an atomic LSN variable into
      it's component pieces and to recombined the components into an
      atomic variable. Use them where appropriate.
      
      This also removes the need for explicitly holding a spinlock to read
      the l_tail_lsn on 32 bit platforms.
      Signed-off-by: default avatarDave Chinner <dchinner@redhat.com>
      
      1c3cb9ec
  5. 03 Dec, 2010 1 commit
    • Dave Chinner's avatar
      xfs: convert l_last_sync_lsn to an atomic variable · 84f3c683
      Dave Chinner authored
      log->l_last_sync_lsn is updated in only one critical spot - log
      buffer Io completion - and is protected by the grant lock here. This
      requires the grant lock to be taken for every log buffer IO
      completion. Converting the l_last_sync_lsn variable to an atomic64_t
      means that we do not need to take the grant lock in log buffer IO
      completion to update it.
      
      This also removes the need for explicitly holding a spinlock to read
      the l_last_sync_lsn on 32 bit platforms.
      Signed-off-by: default avatarDave Chinner <dchinner@redhat.com>
      Reviewed-by: default avatarChristoph Hellwig <hch@lst.de>
      84f3c683
  6. 21 Dec, 2010 6 commits
  7. 20 Dec, 2010 3 commits
  8. 03 Dec, 2010 1 commit
    • Dave Chinner's avatar
      xfs: consume iodone callback items on buffers as they are processed · c90821a2
      Dave Chinner authored
      To allow buffer iodone callbacks to consume multiple items off the
      callback list, first we need to convert the xfs_buf_do_callbacks()
      to consume items and always pull the next item from the head of the
      list.
      
      The means the item list walk is never dependent on knowing the
      next item on the list and hence allows callbacks to remove items
      from the list as well. This allows callbacks to do bulk operations
      by scanning the list for identical callbacks, consuming them all
      and then processing them in bulk, negating the need for multiple
      callbacks of that type.
      Signed-off-by: default avatarDave Chinner <dchinner@redhat.com>
      Reviewed-by: default avatarChristoph Hellwig <hch@lst.de>
      c90821a2
  9. 17 Dec, 2010 1 commit
    • Dave Chinner's avatar
      xfs: reduce the number of AIL push wakeups · e677d0f9
      Dave Chinner authored
      The xfaild often tries to rest to wait for congestion to pass of for
      IO to complete, but is regularly woken in tail-pushing situations.
      In severe cases, the xfsaild is getting woken tens of thousands of
      times a second. Reduce the number needless wakeups by only waking
      the xfsaild if the new target is larger than the old one. Further
      make short sleeps uninterruptible as they occur when the xfsaild has
      decided it needs to back off to allow some IO to complete and being
      woken early is counter-productive.
      Signed-off-by: default avatarDave Chinner <dchinner@redhat.com>
      Reviewed-by: default avatarChristoph Hellwig <hch@lst.de>
      e677d0f9
  10. 20 Dec, 2010 1 commit
    • Dave Chinner's avatar
      xfs: bulk AIL insertion during transaction commit · 0e57f6a3
      Dave Chinner authored
      When inserting items into the AIL from the transaction committed
      callbacks, we take the AIL lock for every single item that is to be
      inserted. For a CIL checkpoint commit, this can be tens of thousands
      of individual inserts, yet almost all of the items will be inserted
      at the same point in the AIL because they have the same index.
      
      To reduce the overhead and contention on the AIL lock for such
      operations, introduce a "bulk insert" operation which allows a list
      of log items with the same LSN to be inserted in a single operation
      via a list splice. To do this, we need to pre-sort the log items
      being committed into a temporary list for insertion.
      
      The complexity is that not every log item will end up with the same
      LSN, and not every item is actually inserted into the AIL. Items
      that don't match the commit LSN will be inserted and unpinned as per
      the current one-at-a-time method (relatively rare), while items that
      are not to be inserted will be unpinned and freed immediately. Items
      that are to be inserted at the given commit lsn are placed in a
      temporary array and inserted into the AIL in bulk each time the
      array fills up.
      
      As a result of this, we trade off AIL hold time for a significant
      reduction in traffic. lock_stat output shows that the worst case
      hold time is unchanged, but contention from AIL inserts drops by an
      order of magnitude and the number of lock traversal decreases
      significantly.
      Signed-off-by: default avatarDave Chinner <dchinner@redhat.com>
      Reviewed-by: default avatarChristoph Hellwig <hch@lst.de>
      0e57f6a3
  11. 03 Dec, 2010 1 commit
    • Dave Chinner's avatar
      xfs: clean up xfs_ail_delete() · eb3efa12
      Dave Chinner authored
      xfs_ail_delete() has a needlessly complex interface. It returns the log item
      that was passed in for deletion (which the callers then assert is identical to
      the one passed in), and callers of xfs_ail_delete() still need to invalidate
      current traversal cursors.
      
      Make xfs_ail_delete() return void, move the cursor invalidation inside it, and
      clean up the callers just to use the log item pointer they passed in.
      
      While cleaning up, remove the messy and unnecessary "/* ARGUSED */" comments
      around all these functions.
      Signed-off-by: default avatarDave Chinner <dchinner@redhat.com>
      Reviewed-by: default avatarChristoph Hellwig <hch@lst.de>
      eb3efa12
  12. 20 Dec, 2010 2 commits
    • Dave Chinner's avatar
      xfs: Pull EFI/EFD handling out from under the AIL lock · b199c8a4
      Dave Chinner authored
      EFI/EFD interactions are protected from races by the AIL lock. They
      are the only type of log items that require the the AIL lock to
      serialise internal state, so they need to be separated from the AIL
      lock before we can do bulk insert operations on the AIL.
      
      To acheive this, convert the counter of the number of extents in the
      EFI to an atomic so it can be safely manipulated by EFD processing
      without locks. Also, convert the EFI state flag manipulations to use
      atomic bit operations so no locks are needed to record state
      changes. Finally, use the state bits to determine when it is safe to
      free the EFI and clean up the code to do this neatly.
      Signed-off-by: default avatarDave Chinner <dchinner@redhat.com>
      Reviewed-by: default avatarChristoph Hellwig <hch@lst.de>
      b199c8a4
    • Dave Chinner's avatar
      xfs: fix EFI transaction cancellation. · 9c5f8414
      Dave Chinner authored
      XFS_EFI_CANCELED has not been set in the code base since
      xfs_efi_cancel() was removed back in 2006 by commit
      065d312e ("[XFS] Remove unused
      iop_abort log item operation), and even then xfs_efi_cancel() was
      never called. I haven't tracked it back further than that (beyond
      git history), but it indicates that the handling of EFIs in
      cancelled transactions has been broken for a long time.
      
      Basically, when we get an IOP_UNPIN(lip, 1); call from
      xfs_trans_uncommit() (i.e. remove == 1), if we don't free the log
      item descriptor we leak it. Fix the behviour to be correct and kill
      the XFS_EFI_CANCELED flag.
      Signed-off-by: default avatarDave Chinner <dchinner@redhat.com>
      Reviewed-by: default avatarChristoph Hellwig <hch@lst.de>
      9c5f8414
  13. 02 Dec, 2010 2 commits
    • Dave Chinner's avatar
      xfs: connect up buffer reclaim priority hooks · 821eb21d
      Dave Chinner authored
      Now that the buffer reclaim infrastructure can handle different reclaim
      priorities for different types of buffers, reconnect the hooks in the
      XFS code that has been sitting dormant since it was ported to Linux. This
      should finally give use reclaim prioritisation that is on a par with the
      functionality that Irix provided XFS 15 years ago.
      Signed-off-by: default avatarDave Chinner <dchinner@redhat.com>
      Reviewed-by: default avatarChristoph Hellwig <hch@lst.de>
      821eb21d
    • Dave Chinner's avatar
      xfs: add a lru to the XFS buffer cache · 430cbeb8
      Dave Chinner authored
      Introduce a per-buftarg LRU for memory reclaim to operate on. This
      is the last piece we need to put in place so that we can fully
      control the buffer lifecycle. This allows XFS to be responsibile for
      maintaining the working set of buffers under memory pressure instead
      of relying on the VM reclaim not to take pages we need out from
      underneath us.
      
      The implementation introduces a b_lru_ref counter into the buffer.
      This is currently set to 1 whenever the buffer is referenced and so is used to
      determine if the buffer should be added to the LRU or not when freed.
      Effectively it allows lazy LRU initialisation of the buffer so we do not need
      to touch the LRU list and locks in xfs_buf_find().
      
      Instead, when the buffer is being released and we drop the last
      reference to it, we check the b_lru_ref count and if it is none zero
      we re-add the buffer reference and add the inode to the LRU. The
      b_lru_ref counter is decremented by the shrinker, and whenever the
      shrinker comes across a buffer with a zero b_lru_ref counter, if
      released the LRU reference on the buffer. In the absence of a lookup
      race, this will result in the buffer being freed.
      
      This counting mechanism is used instead of a reference flag so that
      it is simple to re-introduce buffer-type specific reclaim reference
      counts to prioritise reclaim more effectively. We still have all
      those hooks in the XFS code, so this will provide the infrastructure
      to re-implement that functionality.
      Signed-off-by: default avatarDave Chinner <dchinner@redhat.com>
      Reviewed-by: default avatarChristoph Hellwig <hch@lst.de>
      430cbeb8
  14. 30 Nov, 2010 1 commit
  15. 16 Dec, 2010 1 commit
  16. 17 Dec, 2010 1 commit
    • Dave Chinner's avatar
      xfs: convert inode cache lookups to use RCU locking · 1a3e8f3d
      Dave Chinner authored
      With delayed logging greatly increasing the sustained parallelism of inode
      operations, the inode cache locking is showing significant read vs write
      contention when inode reclaim runs at the same time as lookups. There is
      also a lot more write lock acquistions than there are read locks (4:1 ratio)
      so the read locking is not really buying us much in the way of parallelism.
      
      To avoid the read vs write contention, change the cache to use RCU locking on
      the read side. To avoid needing to RCU free every single inode, use the built
      in slab RCU freeing mechanism. This requires us to be able to detect lookups of
      freed inodes, so enѕure that ever freed inode has an inode number of zero and
      the XFS_IRECLAIM flag set. We already check the XFS_IRECLAIM flag in cache hit
      lookup path, but also add a check for a zero inode number as well.
      
      We canthen convert all the read locking lockups to use RCU read side locking
      and hence remove all read side locking.
      Signed-off-by: default avatarDave Chinner <dchinner@redhat.com>
      Reviewed-by: default avatarAlex Elder <aelder@sgi.com>
      1a3e8f3d
  17. 16 Dec, 2010 1 commit
    • Dave Chinner's avatar
      xfs: rcu free inodes · d95b7aaf
      Dave Chinner authored
      Introduce RCU freeing of XFS inodes so that we can convert lookup
      traversals to use rcu_read_lock() protection. This patch only
      introduces the RCU freeing to minimise the potential conflicts with
      mainline if this is merged into mainline via a VFS patchset. It
      abuses the i_dentry list for the RCU callback structure because the
      VFS patches make this a union so it is safe to use like this and
      simplifies and merge issues.
      
      This patch uses basic RCU freeing rather than SLAB_DESTROY_BY_RCU.
      The later lookup patches need the same "found free inode" protection
      regardless of the RCU freeing method used, so once again the RCU
      freeing method can be dealt with apprpriately at merge time without
      affecting any other code.
      Signed-off-by: default avatarDave Chinner <dchinner@redhat.com>
      Reviewed-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
      d95b7aaf
  18. 23 Dec, 2010 1 commit
    • Dave Chinner's avatar
      xfs: don't truncate prealloc from frequently accessed inodes · 6e857567
      Dave Chinner authored
      A long standing problem for streaming writeѕ through the NFS server
      has been that the NFS server opens and closes file descriptors on an
      inode for every write. The result of this behaviour is that the
      ->release() function is called on every close and that results in
      XFS truncating speculative preallocation beyond the EOF.  This has
      an adverse effect on file layout when multiple files are being
      written at the same time - they interleave their extents and can
      result in severe fragmentation.
      
      To avoid this problem, keep track of ->release calls made on a dirty
      inode. For most cases, an inode is only going to be opened once for
      writing and then closed again during it's lifetime in cache. Hence
      if there are multiple ->release calls when the inode is dirty, there
      is a good chance that the inode is being accessed by the NFS server.
      Hence set a flag the first time ->release is called while there are
      delalloc blocks still outstanding on the inode.
      
      If this flag is set when ->release is next called, then do no
      truncate away the speculative preallocation - leave it there so that
      subsequent writes do not need to reallocate the delalloc space. This
      will prevent interleaving of extents of different inodes written
      concurrently to the same AG.
      
      If we get this wrong, it is not a big deal as we truncate
      speculative allocation beyond EOF anyway in xfs_inactive() when the
      inode is thrown out of the cache.
      Signed-off-by: default avatarDave Chinner <dchinner@redhat.com>
      Reviewed-by: default avatarChristoph Hellwig <hch@lst.de>
      6e857567
  19. 04 Jan, 2011 1 commit
    • Dave Chinner's avatar
      xfs: dynamic speculative EOF preallocation · 055388a3
      Dave Chinner authored
      Currently the size of the speculative preallocation during delayed
      allocation is fixed by either the allocsize mount option of a
      default size. We are seeing a lot of cases where we need to
      recommend using the allocsize mount option to prevent fragmentation
      when buffered writes land in the same AG.
      
      Rather than using a fixed preallocation size by default (up to 64k),
      make it dynamic by basing it on the current inode size. That way the
      EOF preallocation will increase as the file size increases.  Hence
      for streaming writes we are much more likely to get large
      preallocations exactly when we need it to reduce fragementation.
      
      For default settings, the size of the initial extents is determined
      by the number of parallel writers and the amount of memory in the
      machine. For 4GB RAM and 4 concurrent 32GB file writes:
      
      EXT: FILE-OFFSET           BLOCK-RANGE          AG AG-OFFSET                 TOTAL
         0: [0..1048575]:         1048672..2097247      0 (1048672..2097247)      1048576
         1: [1048576..2097151]:   5242976..6291551      0 (5242976..6291551)      1048576
         2: [2097152..4194303]:   12583008..14680159    0 (12583008..14680159)    2097152
         3: [4194304..8388607]:   25165920..29360223    0 (25165920..29360223)    4194304
         4: [8388608..16777215]:  58720352..67108959    0 (58720352..67108959)    8388608
         5: [16777216..33554423]: 117440584..134217791  0 (117440584..134217791) 16777208
         6: [33554424..50331511]: 184549056..201326143  0 (184549056..201326143) 16777088
         7: [50331512..67108599]: 251657408..268434495  0 (251657408..268434495) 16777088
      
      and for 16 concurrent 16GB file writes:
      
       EXT: FILE-OFFSET           BLOCK-RANGE          AG AG-OFFSET                 TOTAL
         0: [0..262143]:          2490472..2752615      0 (2490472..2752615)       262144
         1: [262144..524287]:     6291560..6553703      0 (6291560..6553703)       262144
         2: [524288..1048575]:    13631592..14155879    0 (13631592..14155879)     524288
         3: [1048576..2097151]:   30408808..31457383    0 (30408808..31457383)    1048576
         4: [2097152..4194303]:   52428904..54526055    0 (52428904..54526055)    2097152
         5: [4194304..8388607]:   104857704..109052007  0 (104857704..109052007)  4194304
         6: [8388608..16777215]:  209715304..218103911  0 (209715304..218103911)  8388608
         7: [16777216..33554423]: 452984848..469762055  0 (452984848..469762055) 16777208
      
      Because it is hard to take back specualtive preallocation, cases
      where there are large slow growing log files on a nearly full
      filesystem may cause premature ENOSPC. Hence as the filesystem nears
      full, the maximum dynamic prealloc size іs reduced according to this
      table (based on 4k block size):
      
      freespace       max prealloc size
        >5%             full extent (8GB)
        4-5%             2GB (8GB >> 2)
        3-4%             1GB (8GB >> 3)
        2-3%           512MB (8GB >> 4)
        1-2%           256MB (8GB >> 5)
        <1%            128MB (8GB >> 6)
      
      This should reduce the amount of space held in speculative
      preallocation for such cases.
      
      The allocsize mount option turns off the dynamic behaviour and fixes
      the prealloc size to whatever the mount option specifies. i.e. the
      behaviour is unchanged.
      Signed-off-by: default avatarDave Chinner <dchinner@redhat.com>
      055388a3
  20. 23 Dec, 2010 2 commits
    • Dave Chinner's avatar
      xfs: use KM_NOFS for allocations during attribute list operations · 622d8149
      Dave Chinner authored
      When listing attributes, we are doiing memory allocations under the
      inode ilock using only KM_SLEEP. This allows memory allocation to
      recurse back into the filesystem and do writeback, which may the
      ilock we already hold on the current inode. THis will deadlock.
      Hence use KM_NOFS for such allocations outside of transaction
      context to ensure that reclaim recursion does not occur.
      Reported-by: default avatarNick Piggin <npiggin@gmail.com>
      Signed-off-by: default avatarDave Chinner <dchinner@redhat.com>
      Reviewed-by: default avatarChristoph Hellwig <hch@lst.de>
      622d8149
    • Dave Chinner's avatar
      xfs: provide a inode iolock lockdep class · dcfcf205
      Dave Chinner authored
      The XFS iolock needs to be re-initialised to a new lock class before
      it enters reclaim to prevent lockdep false positives. Unfortunately,
      this is not sufficient protection as inodes in the XFS_IRECLAIMABLE
      state can be recycled and not re-initialised before being reused.
      
      We need to re-initialise the lock state when transfering out of
      XFS_IRECLAIMABLE state to XFS_INEW, but we need to keep the same
      class as if the inode was just allocated. Hence we need a specific
      lockdep class variable for the iolock so that both initialisations
      use the same class.
      
      While there, add a specific class for inodes in the reclaim state so
      that it is easy to tell from lockdep reports what state the inode
      was in that generated the report.
      Signed-off-by: default avatarDave Chinner <dchinner@redhat.com>
      Reviewed-by: default avatarChristoph Hellwig <hch@lst.de>
      dcfcf205
  21. 16 Dec, 2010 9 commits