Commit b381fbbc authored by Mateusz Guzik's avatar Mateusz Guzik Committed by Christian Brauner

vfs: elide smp_mb in iversion handling in the common case

According to bpftrace on these routines most calls result in cmpxchg,
which already provides the same guarantee.

In inode_maybe_inc_iversion elision is possible because even if the
wrong value was read due to now missing smp_mb fence, the issue is going
to correct itself after cmpxchg. If it appears cmpxchg wont be issued,
the fence + reload are there bringing back previous behavior.
Signed-off-by: default avatarMateusz Guzik <mjguzik@gmail.com>
Link: https://lore.kernel.org/r/20240815083310.3865-1-mjguzik@gmail.comReviewed-by: default avatarJeff Layton <jlayton@kernel.org>
Signed-off-by: default avatarChristian Brauner <brauner@kernel.org>
parent 433f9d76
...@@ -2003,13 +2003,19 @@ bool inode_maybe_inc_iversion(struct inode *inode, bool force) ...@@ -2003,13 +2003,19 @@ bool inode_maybe_inc_iversion(struct inode *inode, bool force)
* information, but the legacy inode_inc_iversion code used a spinlock * information, but the legacy inode_inc_iversion code used a spinlock
* to serialize increments. * to serialize increments.
* *
* Here, we add full memory barriers to ensure that any de-facto * We add a full memory barrier to ensure that any de facto ordering
* ordering with other info is preserved. * with other state is preserved (either implicitly coming from cmpxchg
* or explicitly from smp_mb if we don't know upfront if we will execute
* the former).
* *
* This barrier pairs with the barrier in inode_query_iversion() * These barriers pair with inode_query_iversion().
*/ */
smp_mb();
cur = inode_peek_iversion_raw(inode); cur = inode_peek_iversion_raw(inode);
if (!force && !(cur & I_VERSION_QUERIED)) {
smp_mb();
cur = inode_peek_iversion_raw(inode);
}
do { do {
/* If flag is clear then we needn't do anything */ /* If flag is clear then we needn't do anything */
if (!force && !(cur & I_VERSION_QUERIED)) if (!force && !(cur & I_VERSION_QUERIED))
...@@ -2038,20 +2044,22 @@ EXPORT_SYMBOL(inode_maybe_inc_iversion); ...@@ -2038,20 +2044,22 @@ EXPORT_SYMBOL(inode_maybe_inc_iversion);
u64 inode_query_iversion(struct inode *inode) u64 inode_query_iversion(struct inode *inode)
{ {
u64 cur, new; u64 cur, new;
bool fenced = false;
/*
* Memory barriers (implicit in cmpxchg, explicit in smp_mb) pair with
* inode_maybe_inc_iversion(), see that routine for more details.
*/
cur = inode_peek_iversion_raw(inode); cur = inode_peek_iversion_raw(inode);
do { do {
/* If flag is already set, then no need to swap */ /* If flag is already set, then no need to swap */
if (cur & I_VERSION_QUERIED) { if (cur & I_VERSION_QUERIED) {
/* if (!fenced)
* This barrier (and the implicit barrier in the smp_mb();
* cmpxchg below) pairs with the barrier in
* inode_maybe_inc_iversion().
*/
smp_mb();
break; break;
} }
fenced = true;
new = cur | I_VERSION_QUERIED; new = cur | I_VERSION_QUERIED;
} while (!atomic64_try_cmpxchg(&inode->i_version, &cur, new)); } while (!atomic64_try_cmpxchg(&inode->i_version, &cur, new));
return cur >> I_VERSION_QUERIED_SHIFT; return cur >> I_VERSION_QUERIED_SHIFT;
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment