- 18 Jul, 2024 1 commit
-
-
Benjamin Coddington authored
We've observed NFS clients with sync tasks sleeping in __rpc_execute waiting on RPC_TASK_QUEUED that have not responded to a wake-up from rpc_make_runnable(). I suspect this problem usually goes unnoticed, because on a busy client the task will eventually be re-awoken by another task completion or xprt event. However, if the state manager is draining the slot table, a sync task missing a wake-up can result in a hung client. We've been able to prove that the waker in rpc_make_runnable() successfully calls wake_up_bit() (ie- there's no race to tk_runstate), but the wake_up_bit() call fails to wake the waiter. I suspect the waker is missing the load of the bit's wait_queue_head, so waitqueue_active() is false. There are some very helpful comments about this problem above wake_up_bit(), prepare_to_wait(), and waitqueue_active(). Fix this by inserting smp_mb__after_atomic() before the wake_up_bit(), which pairs with prepare_to_wait() calling set_current_state(). Signed-off-by:
Benjamin Coddington <bcodding@redhat.com> Reviewed-by:
Jeff Layton <jlayton@kernel.org> Signed-off-by:
Anna Schumaker <Anna.Schumaker@Netapp.com>
-
- 17 Jul, 2024 2 commits
-
-
Christoph Hellwig authored
nfs_read_folio is a bit hard to follow because it mixes highlevel logic with the actual data read. Split the latter into a helper and update the comments to be more accurate. Signed-off-by:
Christoph Hellwig <hch@lst.de> Signed-off-by:
-
Christoph Hellwig authored
nfs_folio_length is unsafe to use without having the folio locked and a check for a NULL ->f_mapping that protects against truncations and can lead to kernel crashes. E.g. when running xfstests generic/065 with all nfs trace points enabled. Follow the model of the XFS trace points and pass in an explіcit offset and length. This has the additional benefit that these values can be more accurate as some of the users touch partial folio ranges. Fixes: eb5654b3 ("NFS: Enable tracing of nfs_invalidate_folio() and nfs_launder_folio()") Reported-by:
Chuck Lever <chuck.lever@oracle.com> Signed-off-by:
-
- 12 Jul, 2024 2 commits
-
-
Christoph Hellwig authored
nfs_update_folio has code to extend a write to the entire page under certain conditions. With the support for large folios this now suddenly extents to the variable sized and potentially much larger folio. Add code to limit the extension to the page boundaries of the start and end of the write, which matches the historic expecation and the code comments. Fixes: b73fe2dd6cd5 ("nfs: add support for large folios") Signed-off-by:Sagi Grimberg <sagi@grimberg.me> Reviewed-by:
-
Christoph Hellwig authored
Look for the udev generated persistent device name for NVMe devices in addition to the SCSI ones and the Redhat-specific device mapper name. This is the client side implementation of RFC 9561 "Using the Parallel NFS (pNFS) SCSI Layout to Access Non-Volatile Memory Express (NVMe) Storage Devices". Note that the udev rules for nvme are a bit of a mess and udev will only create a link for the uuid if the NVMe namespace has one, and not the NGUID. As the current RFCs don't support UUID based identifications this means the layout can't be used on such namespaces out of the box. A small tweak to the udev rules can work around it, and as the real fix I will submit a draft to the IETF NFSv4 working group to support UUID-based identifiers for SCSI and NVMe. Signed-off-by:
-
- 10 Jul, 2024 2 commits
-
-
Christoph Hellwig authored
The nfs_page_length is not used anywhere, remove it. Signed-off-by:
-
Christoph Hellwig authored
max_deviceinfo_size is not set anywhere, remove it. Signed-off-by:
-
- 08 Jul, 2024 33 commits
-
-
Christoph Hellwig authored
When NFS requests are split into sub-requests, nfs_inode_remove_request calls nfs_page_group_sync_on_bit to set PG_REMOVE on this sub-request and only completes the head requests once PG_REMOVE is set on all requests. This means that when nfs_lock_and_join_requests sees a PG_REMOVE bit, I/O on the request is in progress and has partially completed. If such a request is returned to nfs_try_to_update_request, it could be extended with the newly dirtied region and I/O for the combined range will be re-scheduled, leading to extra I/O. Change the logic to instead restart the search for a request when any PG_REMOVE bit is set, as the completion handler will remove the request as soon as it can take the page group lock. This not only avoid extending the I/O but also does the right thing for the callers that want to cancel or flush the request. Signed-off-by:
-
Christoph Hellwig authored
nfs_wait_on_request is now only used in write.c. Move it there and mark it static. Signed-off-by:
-
Christoph Hellwig authored
Fold nfs_page_group_lock_subrequests into nfs_lock_and_join_requests to prepare for future changes to this code, and move the helpers to write.c as well. Signed-off-by:
-
Christoph Hellwig authored
Fold nfs_folio_find_and_lock_request into the only caller to prepare for changes to this code. Signed-off-by:
-
Christoph Hellwig authored
nfs_folio_find_and_lock_request and the nfs_page_group_lock_head helper called by it spend quite some effort to deal with head vs subrequests. But given that only the head request can be stashed in the folio private data, non of that is required. Fold the locking logic from nfs_page_group_lock_head into nfs_folio_find_and_lock_request and simplify the result based on the invariant that we always find the head request in the folio private data. Signed-off-by:
-
Christoph Hellwig authored
nfs_folio_private_request is a trivial wrapper around, which itself has fallen out of favor and has been replaced with plain ->private dereferences in recent folio conversions. Do the same for nfs. Signed-off-by:
-
Christoph Hellwig authored
Remove the code testing folio_test_swapcache either explicitly or implicitly in pagemap.h headers, as is now handled using the direct I/O path and not the buffered I/O path that these helpers are located in. Signed-off-by:
-
Olga Kornievskaia authored
Previously in order to mark the communication with the DS server, we tried to use NFS_CS_DS in cl_flags. However, this flag would only be saved for the DS server and in case where DS equals MDS, the client would not find a matching nfs_client in nfs_match_client that represents the MDS (but is also a DS). Instead, don't rely on the NFS_CS_DS but instead use NFS_CS_PNFS. Fixes: 379e4adf ("NFSv4.1: fixup use EXCHGID4_FLAG_USE_PNFS_DS for DS server") Signed-off-by:
Olga Kornievskaia <kolga@netapp.com> Signed-off-by:
-
Jan Kara authored
Commit 6df25e58 ("nfs: remove reliance on bdi congestion") introduced NFS-private solution for limiting number of writes outstanding against a particular server. Unlike previous bdi congestion this algorithm actually works and limits number of outstanding writeback pages to nfs_congestion_kb which scales with amount of client's memory and is capped at 256 MB. As a result some workloads such as random buffered writes over NFS got slower (from ~170 MB/s to ~126 MB/s). The fio command to reproduce is: fio --direct=0 --ioengine=sync --thread --invalidate=1 --group_reporting=1 --runtime=300 --fallocate=posix --ramp_time=10 --new_group --rw=randwrite --size=64256m --numjobs=4 --bs=4k --fsync_on_close=1 --end_fsync=1 This happens because the client sends ~256 MB worth of dirty pages to the server and any further background writeback request is ignored until the number of writeback pages gets below the threshold of 192 MB. By the time this happens and clients decides to trigger another round of writeback, the server often has no pages to write and the disk is idle. To fix this problem and make the client react faster to eased congestion of the server by blocking waiting for congestion to resolve instead of aborting writeback. This improves the random 4k buffered write throughput to 184 MB/s. Reviewed-by:
Jan Kara <jack@suse.cz> Signed-off-by:
-
Jan Kara authored
Atomic types should better be initialized with atomic_long_set() instead of relying on zeroing done by kzalloc(). Clean this up. Reviewed-by:
-
Jan Kara authored
nfss->writeback is updated only when we are ending page writeback and at that moment we also clear nfss->write_congested. So there's no point in rechecking congestion state in nfs_commit_release_pages(). Drop the pointless check. Reviewed-by:
-
Chuck Lever authored
An administrator cannot take action on these messages, but the reported errors might be helpful for troubleshooting. Transition them to trace points so these events appear in the trace log and can be easily lined up with other traced NFS client operations. Examples: append_writer-6147 [000] 80.247393: bl_pr_key_reg: dev=8,0 (sda) key=0x6675bfcf59112e98 append_writer-6147 [000] 80.247842: bl_pr_key_unreg: dev=8,0 (sda) key=0x6675bfcf59112e98 umount.nfs4-6172 [002] 84.950409: bl_pr_key_unreg_err: dev=8,0 (sda) key=0x6675bfcf59112e98 status=RESERVATION_CONFLICT Reviewed-by: -
Chuck Lever authored
Since commit f931d837 ("nfs/blocklayout: refactor block device opening"), an error is reported when no multi-path device is found. But this isn't a fatal error if the subsequent device open is successful. On systems without multi-path devices, this message always appears whether there is a problem or not. Instead, generate less system journal noise by reporting an error only when both open attempts fail. The new error message is more actionable since it indicates that there is a real configuration issue to be addressed. Reviewed-by:
-
Chuck Lever authored
During generic/069 runs with pNFS SCSI layouts, the NFS client emits the following in the system journal: kernel: pNFS: failed to open device /dev/disk/by-id/dm-uuid-mpath-0x6001405e3366f045b7949eb8e4540b51 (-2) kernel: pNFS: using block device sdb (reservation key 0x666b60901e7b26b3) kernel: pNFS: failed to open device /dev/disk/by-id/dm-uuid-mpath-0x6001405e3366f045b7949eb8e4540b51 (-2) kernel: pNFS: using block device sdb (reservation key 0x666b60901e7b26b3) kernel: sd 6:0:0:1: reservation conflict kernel: sd 6:0:0:1: [sdb] tag#16 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s kernel: sd 6:0:0:1: [sdb] tag#16 CDB: Write(10) 2a 00 00 00 00 50 00 00 08 00 kernel: reservation conflict error, dev sdb, sector 80 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 2 kernel: sd 6:0:0:1: reservation conflict kernel: sd 6:0:0:1: reservation conflict kernel: sd 6:0:0:1: [sdb] tag#18 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s kernel: sd 6:0:0:1: [sdb] tag#17 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s kernel: sd 6:0:0:1: [sdb] tag#18 CDB: Write(10) 2a 00 00 00 00 60 00 00 08 00 kernel: sd 6:0:0:1: [sdb] tag#17 CDB: Write(10) 2a 00 00 00 00 58 00 00 08 00 kernel: reservation conflict error, dev sdb, sector 96 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0 kernel: reservation conflict error, dev sdb, sector 88 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 0 systemd[1]: fstests-generic-069.scope: Deactivated successfully. systemd[1]: fstests-generic-069.scope: Consumed 5.092s CPU time. systemd[1]: media-test.mount: Deactivated successfully. systemd[1]: media-scratch.mount: Deactivated successfully. kernel: sd 6:0:0:1: reservation conflict kernel: failed to unregister PR key. This appears to be due to a race. bl_alloc_lseg() calls this: 561 static struct nfs4_deviceid_node * 562 bl_find_get_deviceid(struct nfs_server *server, 563 const struct nfs4_deviceid *id, const struct cred *cred, 564 gfp_t gfp_mask) 565 { 566 struct nfs4_deviceid_node *node; 567 unsigned long start, end; 568 569 retry: 570 node = nfs4_find_get_deviceid(server, id, cred, gfp_mask); 571 if (!node) 572 return ERR_PTR(-ENODEV); nfs4_find_get_deviceid() does a lookup without the spin lock first. If it can't find a matching deviceid, it creates a new device_info (which calls bl_alloc_deviceid_node, and that registers the device's PR key). Then it takes the nfs4_deviceid_lock and looks up the deviceid again. If it finds it this time, bl_find_get_deviceid() frees the spare (new) device_info, which unregisters the PR key for the same device. Any subsequent I/O from this client on that device gets EBADE. The umount later unregisters the device's PR key again. To prevent this problem, register the PR key after the deviceid_node lookup. Signed-off-by: -
Trond Myklebust authored
Some pNFS implementations, such as flexible files, want the client to send the layout stats and layout errors that may have incurred while the metadata server was booting. To do so, the client sends a layoutreturn with an all-zero stateid while the server is in grace during reboot recovery. Signed-off-by:
Trond Myklebust <trond.myklebust@hammerspace.com> Reviewed-by:
-
Trond Myklebust authored
The layout will be automatically unhashed on final release of the reference count. Signed-off-by:
-
Trond Myklebust authored
Replace the boolean in nfs4_proc_layoutreturn() with a set of flags that will allow us to craft a version that is appropriate for reboot recovery. Signed-off-by:
-
Trond Myklebust authored
If the layout return failed due to a timeout or reboot, then leave the layout segments on the list so that the layout return gets replayed later. The exception would be if we're freeing the inode. Signed-off-by:
-
Trond Myklebust authored
If the server reboots, then handle it by deferring the layout return. Signed-off-by:
-
Trond Myklebust authored
If the layoutreturn-on-close fails due to an RPC layer problem, such as a timeout, then we want to retry at a later time. Add a helper function to allow this. Signed-off-by:
-
Trond Myklebust authored
Add a flag PNFS_LAYOUT_FILE_BULK_RETURN, that will attempt to return all the layouts in a pnfs_layout_destroy_byfsid/pnfs_layout_destroy_byclid call, instead of just invalidating them. Signed-off-by:
-
Trond Myklebust authored
Change the bool argument to a flag so that we can add different modes for doing bulk destroy of a layout. In particular, we will want the ability to schedule return of all the layouts associated with a given NFS server when it reboots. Signed-off-by:
-
Trond Myklebust authored
Ensure that we encode the actual stateid, and not any metadata. Signed-off-by:
-
Trond Myklebust authored
Signed-off-by:
-
Trond Myklebust authored
pnfs_layout_free_bulk_destroy_list() already checks for whether the list is empty or not. Signed-off-by:
-
Trond Myklebust authored
When we set the new share access modes for CLOSE in nfs4_close_prepare(). we should only set a mode of NFS4_SHARE_ACCESS_READ, NFS4_SHARE_ACCESS_WRITE or NFS4_SHARE_ACCESS_BOTH. Currently, we may also be passing in the NFSv4.1 share modes for controlling delegation requests in OPEN, which is wrong. Signed-off-by:
-
Lance Shelton authored
Add a callback to return the delegation in order to allow generic NFS code to return the delegation when appropriate. Signed-off-by:
Trond Myklebust <trond.myklebust@primarydata.com> Signed-off-by:
Lance Shelton <lance.shelton@hammerspace.com> Reviewed-by:
-
Trond Myklebust authored
Turn on the optimisation to allow the client to request that the server not return the open stateid when it returns a delegation. Signed-off-by:
-
Trond Myklebust authored
If the server returns a delegation stateid only, then don't try to set an open stateid. Signed-off-by:
-
Trond Myklebust authored
If the server supports the NFSv4.2 protocol extension to optimise away returning a stateid when it returns a delegation, then we cache that information in another capability flag. Signed-off-by:
-
Trond Myklebust authored
Query the server for the OPEN arguments that it supports so that we can figure out which extensions we can use. Signed-off-by:
-
Trond Myklebust authored
If the timestamps and size are delegated to the client, then it is authoritative w.r.t. their values, so we should not be requesting those values from the server. Note that this allows us to optimise away most GETATTR calls if the only changes to the attributes are the result of read() or write(). Signed-off-by:
-
Trond Myklebust authored
nfs_setattr calls nfs_update_inode() directly, so we have to reset the m/ctime there. Signed-off-by:
-
