Skip to content

L
linux
Commit 0e4cd9f2 authored by Catalin Marinas's avatar Catalin Marinas
Browse files
Options

Merge branch 'for-next/read-barrier-depends' into for-next/core 

* for-next/read-barrier-depends:
  : Allow architectures to override __READ_ONCE()
  arm64: Reduce the number of header files pulled into vmlinux.lds.S
  compiler.h: Move compiletime_assert() macros into compiler_types.h
  checkpatch: Remove checks relating to [smp_]read_barrier_depends()
  include/linux: Remove smp_read_barrier_depends() from comments
  tools/memory-model: Remove smp_read_barrier_depends() from informal doc
  Documentation/barriers/kokr: Remove references to [smp_]read_barrier_depends()
  Documentation/barriers: Remove references to [smp_]read_barrier_depends()
  locking/barriers: Remove definitions for [smp_]read_barrier_depends()
  alpha: Replace smp_read_barrier_depends() usage with smp_[r]mb()
  vhost: Remove redundant use of read_barrier_depends() barrier
  asm/rwonce: Don't pull <asm/barrier.h> into 'asm-generic/rwonce.h'
  asm/rwonce: Remove smp_read_barrier_depends() invocation
  alpha: Override READ_ONCE() with barriered implementation
  asm/rwonce: Allow __READ_ONCE to be overridden by the architecture
  compiler.h: Split {READ,WRITE}_ONCE definitions out into rwonce.h
  tools: bpf: Use local copy of headers including uapi/linux/filter.h
parents 18aa3bd5 5f1f7f6c
Hide whitespace changes
Inline Side-by-side
Showing with 324 additions and 547 deletions
Documentation/RCU/Design/Requirements/Requirements.rst
View file @ 0e4cd9f2
......@@ -463,7 +463,7 @@ again without disrupting RCU readers.
This guarantee was only partially premeditated. DYNIX/ptx used an
explicit memory barrier for publication, but had nothing resembling
``rcu_dereference()`` for subscription, nor did it have anything
resembling the ``smp_read_barrier_depends()`` that was later subsumed
resembling the dependency-ordering barrier that was later subsumed
into ``rcu_dereference()`` and later still into ``READ_ONCE()``. The
need for these operations made itself known quite suddenly at a
late-1990s meeting with the DEC Alpha architects, back in the days when
......
Documentation/memory-barriers.txt
View file @ 0e4cd9f2
......@@ -553,12 +553,12 @@ There are certain things that the Linux kernel memory barriers do not guarantee:
DATA DEPENDENCY BARRIERS (HISTORICAL)
-------------------------------------
As of v4.15 of the Linux kernel, an smp_read_barrier_depends() was
added to READ_ONCE(), which means that about the only people who
need to pay attention to this section are those working on DEC Alpha
architecture-specific code and those working on READ_ONCE() itself.
For those who need it, and for those who are interested in the history,
here is the story of data-dependency barriers.
As of v4.15 of the Linux kernel, an smp_mb() was added to READ_ONCE() for
DEC Alpha, which means that about the only people who need to pay attention
to this section are those working on DEC Alpha architecture-specific code
and those working on READ_ONCE() itself. For those who need it, and for
those who are interested in the history, here is the story of
data-dependency barriers.
The usage requirements of data dependency barriers are a little subtle, and
it's not always obvious that they're needed. To illustrate, consider the
......@@ -2708,144 +2708,6 @@ the properties of the memory window through which devices are accessed and/or
the use of any special device communication instructions the CPU may have.
CACHE COHERENCY
---------------
Life isn't quite as simple as it may appear above, however: for while the
caches are expected to be coherent, there's no guarantee that that coherency
will be ordered. This means that while changes made on one CPU will
eventually become visible on all CPUs, there's no guarantee that they will
become apparent in the same order on those other CPUs.
Consider dealing with a system that has a pair of CPUs (1 & 2), each of which
has a pair of parallel data caches (CPU 1 has A/B, and CPU 2 has C/D):
:
: +--------+
: +---------+ | |
+--------+ : +--->| Cache A |<------->| |
| | : | +---------+ | |
| CPU 1 |<---+ | |
| | : | +---------+ | |
+--------+ : +--->| Cache B |<------->| |
: +---------+ | |
: | Memory |
: +---------+ | System |
+--------+ : +--->| Cache C |<------->| |
| | : | +---------+ | |
| CPU 2 |<---+ | |
| | : | +---------+ | |
+--------+ : +--->| Cache D |<------->| |
: +---------+ | |
: +--------+
:
Imagine the system has the following properties:
(*) an odd-numbered cache line may be in cache A, cache C or it may still be
resident in memory;
(*) an even-numbered cache line may be in cache B, cache D or it may still be
resident in memory;
(*) while the CPU core is interrogating one cache, the other cache may be
making use of the bus to access the rest of the system - perhaps to
displace a dirty cacheline or to do a speculative load;
(*) each cache has a queue of operations that need to be applied to that cache
to maintain coherency with the rest of the system;
(*) the coherency queue is not flushed by normal loads to lines already
present in the cache, even though the contents of the queue may
potentially affect those loads.
Imagine, then, that two writes are made on the first CPU, with a write barrier
between them to guarantee that they will appear to reach that CPU's caches in
the requisite order:
CPU 1 CPU 2 COMMENT
=============== =============== =======================================
u == 0, v == 1 and p == &u, q == &u
v = 2;
smp_wmb(); Make sure change to v is visible before
change to p
<A:modify v=2> v is now in cache A exclusively
p = &v;
<B:modify p=&v> p is now in cache B exclusively
The write memory barrier forces the other CPUs in the system to perceive that
the local CPU's caches have apparently been updated in the correct order. But
now imagine that the second CPU wants to read those values:
CPU 1 CPU 2 COMMENT
=============== =============== =======================================
...
q = p;
x = *q;
The above pair of reads may then fail to happen in the expected order, as the
cacheline holding p may get updated in one of the second CPU's caches while
the update to the cacheline holding v is delayed in the other of the second
CPU's caches by some other cache event:
CPU 1 CPU 2 COMMENT
=============== =============== =======================================
u == 0, v == 1 and p == &u, q == &u
v = 2;
smp_wmb();
<A:modify v=2> <C:busy>
<C:queue v=2>
p = &v; q = p;
<D:request p>
<B:modify p=&v> <D:commit p=&v>
<D:read p>
x = *q;
<C:read *q> Reads from v before v updated in cache
<C:unbusy>
<C:commit v=2>
Basically, while both cachelines will be updated on CPU 2 eventually, there's
no guarantee that, without intervention, the order of update will be the same
as that committed on CPU 1.
To intervene, we need to interpolate a data dependency barrier or a read
barrier between the loads (which as of v4.15 is supplied unconditionally
by the READ_ONCE() macro). This will force the cache to commit its
coherency queue before processing any further requests:
CPU 1 CPU 2 COMMENT
=============== =============== =======================================
u == 0, v == 1 and p == &u, q == &u
v = 2;
smp_wmb();
<A:modify v=2> <C:busy>
<C:queue v=2>
p = &v; q = p;
<D:request p>
<B:modify p=&v> <D:commit p=&v>
<D:read p>
smp_read_barrier_depends()
<C:unbusy>
<C:commit v=2>
x = *q;
<C:read *q> Reads from v after v updated in cache
This sort of problem can be encountered on DEC Alpha processors as they have a
split cache that improves performance by making better use of the data bus.
While most CPUs do imply a data dependency barrier on the read when a memory
access depends on a read, not all do, so it may not be relied on.
Other CPUs may also have split caches, but must coordinate between the various
cachelets for normal memory accesses. The semantics of the Alpha removes the
need for hardware coordination in the absence of memory barriers, which
permitted Alpha to sport higher CPU clock rates back in the day. However,
please note that (again, as of v4.15) smp_read_barrier_depends() should not
be used except in Alpha arch-specific code and within the READ_ONCE() macro.
CACHE COHERENCY VS DMA
----------------------
......@@ -3009,10 +2871,8 @@ caches with the memory coherence system, thus making it seem like pointer
changes vs new data occur in the right order.
The Alpha defines the Linux kernel's memory model, although as of v4.15
the Linux kernel's addition of smp_read_barrier_depends() to READ_ONCE()
greatly reduced Alpha's impact on the memory model.
See the subsection on "Cache Coherency" above.
the Linux kernel's addition of smp_mb() to READ_ONCE() on Alpha greatly
reduced its impact on the memory model.
VIRTUAL MACHINE GUESTS
......
Documentation/translations/ko_KR/memory-barriers.txt
View file @ 0e4cd9f2
......@@ -577,7 +577,7 @@ ACQUIRE 는 해당 오퍼레이션의 로드 부분에만 적용되고 RELEASE
데이터 의존성 배리어 (역사적)
-----------------------------
리눅스 커널 v4.15 기준으로, smp_read_barrier_depends() 가 READ_ONCE() 에
리눅스 커널 v4.15 기준으로, smp_mb() 가 DEC Alpha 용 READ_ONCE() 코드에
추가되었는데, 이는 이 섹션에 주의를 기울여야 하는 사람들은 DEC Alpha 아키텍쳐
전용 코드를 만드는 사람들과 READ_ONCE() 자체를 만드는 사람들 뿐임을 의미합니다.
그런 분들을 위해, 그리고 역사에 관심 있는 분들을 위해, 여기 데이터 의존성
......@@ -2664,144 +2664,6 @@ CPU 코어는 프로그램의 인과성이 유지된다고만 여겨진다면
수도 있습니다.
캐시 일관성
-----------
하지만 삶은 앞에서 이야기한 것처럼 단순하지 않습니다: 캐시들은 일관적일 것으로
기대되지만, 그 일관성이 순서에도 적용될 거라는 보장은 없습니다. 한 CPU 에서
만들어진 변경 사항은 최종적으로는 시스템의 모든 CPU 에게 보여지게 되지만, 다른
CPU 들에게도 같은 순서로 보이게 될 거라는 보장은 없다는 뜻입니다.
두개의 CPU (1 & 2) 가 달려 있고, 각 CPU 에 두개의 데이터 캐시(CPU 1 은 A/B 를,
CPU 2 는 C/D 를 갖습니다)가 병렬로 연결되어 있는 시스템을 다룬다고 생각해
봅시다:
:
: +--------+
: +---------+ | |
+--------+ : +--->| Cache A |<------->| |
| | : | +---------+ | |
| CPU 1 |<---+ | |
| | : | +---------+ | |
+--------+ : +--->| Cache B |<------->| |
: +---------+ | |
: | Memory |
: +---------+ | System |
+--------+ : +--->| Cache C |<------->| |
| | : | +---------+ | |
| CPU 2 |<---+ | |
| | : | +---------+ | |
+--------+ : +--->| Cache D |<------->| |
: +---------+ | |
: +--------+
:
이 시스템이 다음과 같은 특성을 갖는다 생각해 봅시다:
(*) 홀수번 캐시라인은 캐시 A, 캐시 C 또는 메모리에 위치할 수 있음;
(*) 짝수번 캐시라인은 캐시 B, 캐시 D 또는 메모리에 위치할 수 있음;
(*) CPU 코어가 한개의 캐시에 접근하는 동안, 다른 캐시는 - 더티 캐시라인을
메모리에 내리거나 추측성 로드를 하거나 하기 위해 - 시스템의 다른 부분에
액세스 하기 위해 버스를 사용할 수 있음;
(*) 각 캐시는 시스템의 나머지 부분들과 일관성을 맞추기 위해 해당 캐시에
적용되어야 할 오퍼레이션들의 큐를 가짐;
(*) 이 일관성 큐는 캐시에 이미 존재하는 라인에 가해지는 평범한 로드에 의해서는
비워지지 않는데, 큐의 오퍼레이션들이 이 로드의 결과에 영향을 끼칠 수 있다
할지라도 그러함.
이제, 첫번째 CPU 에서 두개의 쓰기 오퍼레이션을 만드는데, 해당 CPU 의 캐시에
요청된 순서로 오퍼레이션이 도달됨을 보장하기 위해 두 오퍼레이션 사이에 쓰기
배리어를 사용하는 상황을 상상해 봅시다:
CPU 1 CPU 2 COMMENT
=============== =============== =======================================
u == 0, v == 1 and p == &u, q == &u
v = 2;
smp_wmb(); v 의 변경이 p 의 변경 전에 보일 것을
분명히 함
<A:modify v=2> v 는 이제 캐시 A 에 독점적으로 존재함
p = &v;
<B:modify p=&v> p 는 이제 캐시 B 에 독점적으로 존재함
여기서의 쓰기 메모리 배리어는 CPU 1 의 캐시가 올바른 순서로 업데이트 된 것으로
시스템의 다른 CPU 들이 인지하게 만듭니다. 하지만, 이제 두번째 CPU 가 그 값들을
읽으려 하는 상황을 생각해 봅시다:
CPU 1 CPU 2 COMMENT
=============== =============== =======================================
...
q = p;
x = *q;
위의 두개의 읽기 오퍼레이션은 예상된 순서로 일어나지 못할 수 있는데, 두번째 CPU
의 한 캐시에 다른 캐시 이벤트가 발생해 v 를 담고 있는 캐시라인의 해당 캐시에의
업데이트가 지연되는 사이, p 를 담고 있는 캐시라인은 두번째 CPU 의 다른 캐시에
업데이트 되어버렸을 수 있기 때문입니다.
CPU 1 CPU 2 COMMENT
=============== =============== =======================================
u == 0, v == 1 and p == &u, q == &u
v = 2;
smp_wmb();
<A:modify v=2> <C:busy>
<C:queue v=2>
p = &v; q = p;
<D:request p>
<B:modify p=&v> <D:commit p=&v>
<D:read p>
x = *q;
<C:read *q> 캐시에 업데이트 되기 전의 v 를 읽음
<C:unbusy>
<C:commit v=2>
기본적으로, 두개의 캐시라인 모두 CPU 2 에 최종적으로는 업데이트 될 것이지만,
별도의 개입 없이는, 업데이트의 순서가 CPU 1 에서 만들어진 순서와 동일할
것이라는 보장이 없습니다.
여기에 개입하기 위해선, 데이터 의존성 배리어나 읽기 배리어를 로드 오퍼레이션들
사이에 넣어야 합니다 (v4.15 부터는 READ_ONCE() 매크로에 의해 무조건적으로
그렇게 됩니다). 이렇게 함으로써 캐시가 다음 요청을 처리하기 전에 일관성 큐를
처리하도록 강제하게 됩니다.
CPU 1 CPU 2 COMMENT
=============== =============== =======================================
u == 0, v == 1 and p == &u, q == &u
v = 2;
smp_wmb();
<A:modify v=2> <C:busy>
<C:queue v=2>
p = &v; q = p;
<D:request p>
<B:modify p=&v> <D:commit p=&v>
<D:read p>
smp_read_barrier_depends()
<C:unbusy>
<C:commit v=2>
x = *q;
<C:read *q> 캐시에 업데이트 된 v 를 읽음
이런 부류의 문제는 DEC Alpha 계열 프로세서들에서 발견될 수 있는데, 이들은
데이터 버스를 좀 더 잘 사용해 성능을 개선할 수 있는, 분할된 캐시를 가지고 있기
때문입니다. 대부분의 CPU 는 하나의 읽기 오퍼레이션의 메모리 액세스가 다른 읽기
오퍼레이션에 의존적이라면 데이터 의존성 배리어를 내포시킵니다만, 모두가 그런건
아니기 때문에 이점에 의존해선 안됩니다.
다른 CPU 들도 분할된 캐시를 가지고 있을 수 있지만, 그런 CPU 들은 평범한 메모리
액세스를 위해서도 이 분할된 캐시들 사이의 조정을 해야만 합니다. Alpha 는 가장
약한 메모리 순서 시맨틱 (semantic) 을 선택함으로써 메모리 배리어가 명시적으로
사용되지 않았을 때에는 그런 조정이 필요하지 않게 했으며, 이는 Alpha 가 당시에
더 높은 CPU 클락 속도를 가질 수 있게 했습니다. 하지만, (다시 말하건대, v4.15
이후부터는) Alpha 아키텍쳐 전용 코드와 READ_ONCE() 매크로 내부에서를 제외하고는
smp_read_barrier_depends() 가 사용되지 않아야 함을 알아두시기 바랍니다.
캐시 일관성 VS DMA
------------------
......@@ -2962,10 +2824,8 @@ Alpha CPU 의 일부 버전은 분할된 데이터 캐시를 가지고 있어서
데이터의 발견을 올바른 순서로 일어나게 하기 때문입니다.
리눅스 커널의 메모리 배리어 모델은 Alpha 에 기초해서 정의되었습니다만, v4.15
부터는 리눅스 커널이 READ_ONCE() 내에 smp_read_barrier_depends() 를 추가해서
Alpha 의 메모리 모델로의 영향력이 크게 줄어들긴 했습니다.
위의 "캐시 일관성" 서브섹션을 참고하세요.
부터는 Alpha 용 READ_ONCE() 코드 내에 smp_mb() 가 추가되어서 메모리 모델로의
Alpha 의 영향력이 크게 줄어들었습니다.
가상 머신 게스트
......
arch/alpha/include/asm/atomic.h
View file @ 0e4cd9f2
......@@ -16,10 +16,10 @@
/*
* To ensure dependency ordering is preserved for the _relaxed and
* _release atomics, an smp_read_barrier_depends() is unconditionally
* inserted into the _relaxed variants, which are used to build the
* barriered versions. Avoid redundant back-to-back fences in the
* _acquire and _fence versions.
* _release atomics, an smp_mb() is unconditionally inserted into the
* _relaxed variants, which are used to build the barriered versions.
* Avoid redundant back-to-back fences in the _acquire and _fence
* versions.
*/
#define __atomic_acquire_fence()
#define __atomic_post_full_fence()
......@@ -70,7 +70,7 @@ static inline int atomic_##op##_return_relaxed(int i, atomic_t *v) \
".previous" \
:"=&r" (temp), "=m" (v->counter), "=&r" (result) \
:"Ir" (i), "m" (v->counter) : "memory"); \
smp_read_barrier_depends(); \
smp_mb(); \
return result; \
}
......@@ -88,7 +88,7 @@ static inline int atomic_fetch_##op##_relaxed(int i, atomic_t *v) \
".previous" \
:"=&r" (temp), "=m" (v->counter), "=&r" (result) \
:"Ir" (i), "m" (v->counter) : "memory"); \
smp_read_barrier_depends(); \
smp_mb(); \
return result; \
}
......@@ -123,7 +123,7 @@ static __inline__ s64 atomic64_##op##_return_relaxed(s64 i, atomic64_t * v) \
".previous" \
:"=&r" (temp), "=m" (v->counter), "=&r" (result) \
:"Ir" (i), "m" (v->counter) : "memory"); \
smp_read_barrier_depends(); \
smp_mb(); \
return result; \
}
......@@ -141,7 +141,7 @@ static __inline__ s64 atomic64_fetch_##op##_relaxed(s64 i, atomic64_t * v) \
".previous" \
:"=&r" (temp), "=m" (v->counter), "=&r" (result) \
:"Ir" (i), "m" (v->counter) : "memory"); \
smp_read_barrier_depends(); \
smp_mb(); \
return result; \
}
......
arch/alpha/include/asm/barrier.h
View file @ 0e4cd9f2
......@@ -2,64 +2,15 @@
#ifndef __BARRIER_H
#define __BARRIER_H
#include <asm/compiler.h>
#define mb() __asm__ __volatile__("mb": : :"memory")
#define rmb() __asm__ __volatile__("mb": : :"memory")
#define wmb() __asm__ __volatile__("wmb": : :"memory")
/**
* read_barrier_depends - Flush all pending reads that subsequents reads
* depend on.
*
* No data-dependent reads from memory-like regions are ever reordered
* over this barrier. All reads preceding this primitive are guaranteed
* to access memory (but not necessarily other CPUs' caches) before any
* reads following this primitive that depend on the data return by
* any of the preceding reads. This primitive is much lighter weight than
* rmb() on most CPUs, and is never heavier weight than is
* rmb().
*
* These ordering constraints are respected by both the local CPU
* and the compiler.
*
* Ordering is not guaranteed by anything other than these primitives,
* not even by data dependencies. See the documentation for
* memory_barrier() for examples and URLs to more information.
*
* For example, the following code would force ordering (the initial
* value of "a" is zero, "b" is one, and "p" is "&a"):
*
* <programlisting>
* CPU 0 CPU 1
*
* b = 2;
* memory_barrier();
* p = &b; q = p;
* read_barrier_depends();
* d = *q;
* </programlisting>
*
* because the read of "*q" depends on the read of "p" and these
* two reads are separated by a read_barrier_depends(). However,
* the following code, with the same initial values for "a" and "b":
*
* <programlisting>
* CPU 0 CPU 1
*
* a = 2;
* memory_barrier();
* b = 3; y = b;
* read_barrier_depends();
* x = a;
* </programlisting>
*
* does not enforce ordering, since there is no data dependency between
* the read of "a" and the read of "b". Therefore, on some CPUs, such
* as Alpha, "y" could be set to 3 and "x" to 0. Use rmb()
* in cases like this where there are no data dependencies.
*/
#define read_barrier_depends() __asm__ __volatile__("mb": : :"memory")
#define __smp_load_acquire(p) \
({ \
compiletime_assert_atomic_type(*p); \
__READ_ONCE(*p); \
})
#ifdef CONFIG_SMP
#define __ASM_SMP_MB "\tmb\n"
......
arch/alpha/include/asm/pgtable.h
View file @ 0e4cd9f2
......@@ -277,9 +277,9 @@ extern inline pte_t pte_mkdirty(pte_t pte) { pte_val(pte) |= __DIRTY_BITS; retur
extern inline pte_t pte_mkyoung(pte_t pte) { pte_val(pte) |= __ACCESS_BITS; return pte; }
/*
* The smp_read_barrier_depends() in the following functions are required to
* order the load of *dir (the pointer in the top level page table) with any
* subsequent load of the returned pmd_t *ret (ret is data dependent on *dir).
* The smp_rmb() in the following functions are required to order the load of
* *dir (the pointer in the top level page table) with any subsequent load of
* the returned pmd_t *ret (ret is data dependent on *dir).
*
* If this ordering is not enforced, the CPU might load an older value of
* *ret, which may be uninitialized data. See mm/memory.c:__pte_alloc for
......@@ -293,7 +293,7 @@ extern inline pte_t pte_mkyoung(pte_t pte) { pte_val(pte) |= __ACCESS_BITS; retu
extern inline pmd_t * pmd_offset(pud_t * dir, unsigned long address)
{
pmd_t *ret = (pmd_t *) pud_page_vaddr(*dir) + ((address >> PMD_SHIFT) & (PTRS_PER_PAGE - 1));
smp_read_barrier_depends(); /* see above */
smp_rmb(); /* see above */
return ret;
}
#define pmd_offset pmd_offset
......@@ -303,7 +303,7 @@ extern inline pte_t * pte_offset_kernel(pmd_t * dir, unsigned long address)
{
pte_t *ret = (pte_t *) pmd_page_vaddr(*dir)
+ ((address >> PAGE_SHIFT) & (PTRS_PER_PAGE - 1));
smp_read_barrier_depends(); /* see above */
smp_rmb(); /* see above */
return ret;
}
#define pte_offset_kernel pte_offset_kernel
......
arch/alpha/include/asm/rwonce.h 0 → 100644
View file @ 0e4cd9f2
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Copyright (C) 2019 Google LLC.
*/
#ifndef __ASM_RWONCE_H
#define __ASM_RWONCE_H
#ifdef CONFIG_SMP
#include <asm/barrier.h>
/*
* Alpha is apparently daft enough to reorder address-dependent loads
* on some CPU implementations. Knock some common sense into it with
* a memory barrier in READ_ONCE().
*
* For the curious, more information about this unusual reordering is
* available in chapter 15 of the "perfbook":
*
* https://kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook.html
*
*/
#define __READ_ONCE(x) \
({ \
__unqual_scalar_typeof(x) __x = \
(*(volatile typeof(__x) *)(&(x))); \
mb(); \
(typeof(x))__x; \
})
#endif /* CONFIG_SMP */
#include <asm-generic/rwonce.h>
#endif /* __ASM_RWONCE_H */
arch/arm/include/asm/vdso/gettimeofday.h
View file @ 0e4cd9f2
......@@ -7,6 +7,7 @@
#ifndef __ASSEMBLY__
#include <asm/barrier.h>
#include <asm/errno.h>
#include <asm/unistd.h>
#include <asm/vdso/cp15.h>
......
arch/arm64/include/asm/kernel-pgtable.h
View file @ 0e4cd9f2
......@@ -8,7 +8,7 @@
#ifndef __ASM_KERNEL_PGTABLE_H
#define __ASM_KERNEL_PGTABLE_H
#include <linux/pgtable.h>
#include <asm/pgtable-hwdef.h>
#include <asm/sparsemem.h>
/*
......
arch/arm64/include/asm/memory.h
View file @ 0e4cd9f2
......@@ -10,11 +10,8 @@
#ifndef __ASM_MEMORY_H
#define __ASM_MEMORY_H
#include <linux/compiler.h>
#include <linux/const.h>
#include <linux/sizes.h>
#include <linux/types.h>
#include <asm/bug.h>
#include <asm/page-def.h>
/*
......@@ -157,11 +154,15 @@
#endif
#ifndef __ASSEMBLY__
extern u64 vabits_actual;
#define PAGE_END (_PAGE_END(vabits_actual))
#include <linux/bitops.h>
#include <linux/compiler.h>
#include <linux/mmdebug.h>
#include <linux/types.h>
#include <asm/bug.h>
extern u64 vabits_actual;
#define PAGE_END (_PAGE_END(vabits_actual))
extern s64 physvirt_offset;
extern s64 memstart_addr;
......
arch/arm64/include/asm/uaccess.h
View file @ 0e4cd9f2
......@@ -19,6 +19,7 @@
#include <linux/string.h>
#include <asm/cpufeature.h>
#include <asm/mmu.h>
#include <asm/ptrace.h>
#include <asm/memory.h>
#include <asm/extable.h>
......
arch/arm64/include/asm/vdso/compat_gettimeofday.h
View file @ 0e4cd9f2
......@@ -7,6 +7,7 @@
#ifndef __ASSEMBLY__
#include <asm/barrier.h>
#include <asm/unistd.h>
#include <asm/errno.h>
......
arch/arm64/include/asm/vdso/gettimeofday.h
View file @ 0e4cd9f2
......@@ -7,6 +7,7 @@
#ifndef __ASSEMBLY__
#include <asm/barrier.h>
#include <asm/unistd.h>
#define VDSO_HAS_CLOCK_GETRES 1
......
arch/arm64/kernel/entry.S
View file @ 0e4cd9f2
......@@ -15,6 +15,7 @@
#include <asm/assembler.h>
#include <asm/asm-offsets.h>
#include <asm/asm_pointer_auth.h>
#include <asm/bug.h>
#include <asm/cpufeature.h>
#include <asm/errno.h>
#include <asm/esr.h>
......
arch/arm64/kernel/vmlinux.lds.S
View file @ 0e4cd9f2
......@@ -10,7 +10,6 @@
#include <asm-generic/vmlinux.lds.h>
#include <asm/cache.h>
#include <asm/kernel-pgtable.h>
#include <asm/thread_info.h>
#include <asm/memory.h>
#include <asm/page.h>
......
arch/arm64/kvm/hyp-init.S
View file @ 0e4cd9f2
......@@ -6,6 +6,7 @@
#include <linux/linkage.h>
#include <asm/alternative.h>
#include <asm/assembler.h>
#include <asm/kvm_arm.h>
#include <asm/kvm_mmu.h>
......
arch/riscv/include/asm/vdso/gettimeofday.h
View file @ 0e4cd9f2
......@@ -4,6 +4,7 @@
#ifndef __ASSEMBLY__
#include <asm/barrier.h>
#include <asm/unistd.h>
#include <asm/csr.h>
#include <uapi/linux/time.h>
......
drivers/vhost/vhost.c
View file @ 0e4cd9f2
......@@ -2092,11 +2092,6 @@ static int get_indirect(struct vhost_virtqueue *vq,
return ret;
}
iov_iter_init(&from, READ, vq->indirect, ret, len);
/* We will use the result as an address to read from, so most
* architectures only need a compiler barrier here. */
read_barrier_depends();
count = len / sizeof desc;
/* Buffers are chained via a 16 bit next field, so
* we can have at most 2^16 of these. */
......
include/asm-generic/Kbuild
View file @ 0e4cd9f2
......@@ -45,6 +45,7 @@ mandatory-y += pci.h
mandatory-y += percpu.h
mandatory-y += pgalloc.h
mandatory-y += preempt.h
mandatory-y += rwonce.h
mandatory-y += sections.h
mandatory-y += serial.h
mandatory-y += shmparam.h
......
include/asm-generic/barrier.h
View file @ 0e4cd9f2
......@@ -13,7 +13,7 @@
#ifndef __ASSEMBLY__
#include <linux/compiler.h>
#include <asm/rwonce.h>
#ifndef nop
#define nop() asm volatile ("nop")
......@@ -46,10 +46,6 @@
#define dma_wmb() wmb()
#endif
#ifndef read_barrier_depends
#define read_barrier_depends() do { } while (0)
#endif
#ifndef __smp_mb
#define __smp_mb() mb()
#endif
......@@ -62,10 +58,6 @@
#define __smp_wmb() wmb()
#endif
#ifndef __smp_read_barrier_depends
#define __smp_read_barrier_depends() read_barrier_depends()
#endif
#ifdef CONFIG_SMP
#ifndef smp_mb
......@@ -80,10 +72,6 @@
#define smp_wmb() __smp_wmb()
#endif
#ifndef smp_read_barrier_depends
#define smp_read_barrier_depends() __smp_read_barrier_depends()
#endif
#else /* !CONFIG_SMP */
#ifndef smp_mb
......@@ -98,10 +86,6 @@
#define smp_wmb() barrier()
#endif
#ifndef smp_read_barrier_depends
#define smp_read_barrier_depends() do { } while (0)
#endif
#endif /* CONFIG_SMP */
#ifndef __smp_store_mb
......@@ -196,7 +180,6 @@ do { \
#define virt_mb() __smp_mb()
#define virt_rmb() __smp_rmb()
#define virt_wmb() __smp_wmb()
#define virt_read_barrier_depends() __smp_read_barrier_depends()
#define virt_store_mb(var, value) __smp_store_mb(var, value)
#define virt_mb__before_atomic() __smp_mb__before_atomic()
#define virt_mb__after_atomic() __smp_mb__after_atomic()
......
include/asm-generic/rwonce.h 0 → 100644
View file @ 0e4cd9f2
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Prevent the compiler from merging or refetching reads or writes. The
* compiler is also forbidden from reordering successive instances of
* READ_ONCE and WRITE_ONCE, but only when the compiler is aware of some
* particular ordering. One way to make the compiler aware of ordering is to
* put the two invocations of READ_ONCE or WRITE_ONCE in different C
* statements.
*
* These two macros will also work on aggregate data types like structs or
* unions.
*
* Their two major use cases are: (1) Mediating communication between
* process-level code and irq/NMI handlers, all running on the same CPU,
* and (2) Ensuring that the compiler does not fold, spindle, or otherwise
* mutilate accesses that either do not require ordering or that interact
* with an explicit memory barrier or atomic instruction that provides the
* required ordering.
*/
#ifndef __ASM_GENERIC_RWONCE_H
#define __ASM_GENERIC_RWONCE_H
#ifndef __ASSEMBLY__
#include <linux/compiler_types.h>
#include <linux/kasan-checks.h>
#include <linux/kcsan-checks.h>
/*
* Yes, this permits 64-bit accesses on 32-bit architectures. These will
* actually be atomic in some cases (namely Armv7 + LPAE), but for others we
* rely on the access being split into 2x32-bit accesses for a 32-bit quantity
* (e.g. a virtual address) and a strong prevailing wind.
*/
#define compiletime_assert_rwonce_type(t) \
compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
"Unsupported access size for {READ,WRITE}_ONCE().")
/*
* Use __READ_ONCE() instead of READ_ONCE() if you do not require any
* atomicity. Note that this may result in tears!
*/
#ifndef __READ_ONCE
#define __READ_ONCE(x) (*(const volatile __unqual_scalar_typeof(x) *)&(x))
#endif
#define READ_ONCE(x) \
({ \
compiletime_assert_rwonce_type(x); \
__READ_ONCE(x); \
})
#define __WRITE_ONCE(x, val) \
do { \
*(volatile typeof(x) *)&(x) = (val); \
} while (0)
#define WRITE_ONCE(x, val) \
do { \
compiletime_assert_rwonce_type(x); \
__WRITE_ONCE(x, val); \
} while (0)
static __no_sanitize_or_inline
unsigned long __read_once_word_nocheck(const void *addr)
{
return __READ_ONCE(*(unsigned long *)addr);
}
/*
* Use READ_ONCE_NOCHECK() instead of READ_ONCE() if you need to load a
* word from memory atomically but without telling KASAN/KCSAN. This is
* usually used by unwinding code when walking the stack of a running process.
*/
#define READ_ONCE_NOCHECK(x) \
({ \
compiletime_assert(sizeof(x) == sizeof(unsigned long), \
"Unsupported access size for READ_ONCE_NOCHECK()."); \
(typeof(x))__read_once_word_nocheck(&(x)); \
})
static __no_kasan_or_inline
unsigned long read_word_at_a_time(const void *addr)
{
kasan_check_read(addr, 1);
return *(unsigned long *)addr;
}
#endif /* __ASSEMBLY__ */
#endif /* __ASM_GENERIC_RWONCE_H */
include/linux/compiler.h
View file @ 0e4cd9f2
......@@ -230,28 +230,6 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
# define __UNIQUE_ID(prefix) __PASTE(__PASTE(__UNIQUE_ID_, prefix), __LINE__)
#endif
/*
* Prevent the compiler from merging or refetching reads or writes. The
* compiler is also forbidden from reordering successive instances of
* READ_ONCE and WRITE_ONCE, but only when the compiler is aware of some
* particular ordering. One way to make the compiler aware of ordering is to
* put the two invocations of READ_ONCE or WRITE_ONCE in different C
* statements.
*
* These two macros will also work on aggregate data types like structs or
* unions.
*
* Their two major use cases are: (1) Mediating communication between
* process-level code and irq/NMI handlers, all running on the same CPU,
* and (2) Ensuring that the compiler does not fold, spindle, or otherwise
* mutilate accesses that either do not require ordering or that interact
* with an explicit memory barrier or atomic instruction that provides the
* required ordering.
*/
#include <asm/barrier.h>
#include <linux/kasan-checks.h>
#include <linux/kcsan-checks.h>
/**
* data_race - mark an expression as containing intentional data races
*
......@@ -272,65 +250,6 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
__v; \
})
/*
* Use __READ_ONCE() instead of READ_ONCE() if you do not require any
* atomicity or dependency ordering guarantees. Note that this may result
* in tears!
*/
#define __READ_ONCE(x) (*(const volatile __unqual_scalar_typeof(x) *)&(x))
#define __READ_ONCE_SCALAR(x) \
({ \
__unqual_scalar_typeof(x) __x = __READ_ONCE(x); \
smp_read_barrier_depends(); \
(typeof(x))__x; \
})
#define READ_ONCE(x) \
({ \
compiletime_assert_rwonce_type(x); \
__READ_ONCE_SCALAR(x); \
})
#define __WRITE_ONCE(x, val) \
do { \
*(volatile typeof(x) *)&(x) = (val); \
} while (0)
#define WRITE_ONCE(x, val) \
do { \
compiletime_assert_rwonce_type(x); \
__WRITE_ONCE(x, val); \
} while (0)
static __no_sanitize_or_inline
unsigned long __read_once_word_nocheck(const void *addr)
{
return __READ_ONCE(*(unsigned long *)addr);
}
/*
* Use READ_ONCE_NOCHECK() instead of READ_ONCE() if you need to load a
* word from memory atomically but without telling KASAN/KCSAN. This is
* usually used by unwinding code when walking the stack of a running process.
*/
#define READ_ONCE_NOCHECK(x) \
({ \
unsigned long __x; \
compiletime_assert(sizeof(x) == sizeof(__x), \
"Unsupported access size for READ_ONCE_NOCHECK()."); \
__x = __read_once_word_nocheck(&(x)); \
smp_read_barrier_depends(); \
(typeof(x))__x; \
})
static __no_kasan_or_inline
unsigned long read_word_at_a_time(const void *addr)
{
kasan_check_read(addr, 1);
return *(unsigned long *)addr;
}
#endif /* __KERNEL__ */
/*
......@@ -354,57 +273,6 @@ static inline void *offset_to_ptr(const int *off)
#endif /* __ASSEMBLY__ */
/* Compile time object size, -1 for unknown */
#ifndef __compiletime_object_size
# define __compiletime_object_size(obj) -1
#endif
#ifndef __compiletime_warning
# define __compiletime_warning(message)
#endif
#ifndef __compiletime_error
# define __compiletime_error(message)
#endif
#ifdef __OPTIMIZE__
# define __compiletime_assert(condition, msg, prefix, suffix) \
do { \
extern void prefix ## suffix(void) __compiletime_error(msg); \
if (!(condition)) \
prefix ## suffix(); \
} while (0)
#else
# define __compiletime_assert(condition, msg, prefix, suffix) do { } while (0)
#endif
#define _compiletime_assert(condition, msg, prefix, suffix) \
__compiletime_assert(condition, msg, prefix, suffix)
/**
* compiletime_assert - break build and emit msg if condition is false
* @condition: a compile-time constant condition to check
* @msg: a message to emit if condition is false
*
* In tradition of POSIX assert, this macro will break the build if the
* supplied condition is *false*, emitting the supplied error message if the
* compiler has support to do so.
*/
#define compiletime_assert(condition, msg) \
_compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
#define compiletime_assert_atomic_type(t) \
compiletime_assert(__native_word(t), \
"Need native word sized stores/loads for atomicity.")
/*
* Yes, this permits 64-bit accesses on 32-bit architectures. These will
* actually be atomic in some cases (namely Armv7 + LPAE), but for others we
* rely on the access being split into 2x32-bit accesses for a 32-bit quantity
* (e.g. a virtual address) and a strong prevailing wind.
*/
#define compiletime_assert_rwonce_type(t) \
compiletime_assert(__native_word(t) || sizeof(t) == sizeof(long long), \
"Unsupported access size for {READ,WRITE}_ONCE().")
/* &a[0] degrades to a pointer: a different type from an array */
#define __must_be_array(a) BUILD_BUG_ON_ZERO(__same_type((a), &(a)[0]))
......@@ -414,4 +282,6 @@ static inline void *offset_to_ptr(const int *off)
*/
#define prevent_tail_call_optimization() mb()
#include <asm/rwonce.h>
#endif /* __LINUX_COMPILER_H */
include/linux/compiler_types.h
View file @ 0e4cd9f2
......@@ -300,6 +300,47 @@ struct ftrace_likely_data {
(sizeof(t) == sizeof(char) || sizeof(t) == sizeof(short) || \
sizeof(t) == sizeof(int) || sizeof(t) == sizeof(long))
/* Compile time object size, -1 for unknown */
#ifndef __compiletime_object_size
# define __compiletime_object_size(obj) -1
#endif
#ifndef __compiletime_warning
# define __compiletime_warning(message)
#endif
#ifndef __compiletime_error
# define __compiletime_error(message)
#endif
#ifdef __OPTIMIZE__
# define __compiletime_assert(condition, msg, prefix, suffix) \
do { \
extern void prefix ## suffix(void) __compiletime_error(msg); \
if (!(condition)) \
prefix ## suffix(); \
} while (0)
#else
# define __compiletime_assert(condition, msg, prefix, suffix) do { } while (0)
#endif
#define _compiletime_assert(condition, msg, prefix, suffix) \
__compiletime_assert(condition, msg, prefix, suffix)
/**
* compiletime_assert - break build and emit msg if condition is false
* @condition: a compile-time constant condition to check
* @msg: a message to emit if condition is false
*
* In tradition of POSIX assert, this macro will break the build if the
* supplied condition is *false*, emitting the supplied error message if the
* compiler has support to do so.
*/
#define compiletime_assert(condition, msg) \
_compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
#define compiletime_assert_atomic_type(t) \
compiletime_assert(__native_word(t), \
"Need native word sized stores/loads for atomicity.")
/* Helpers for emitting diagnostics in pragmas. */
#ifndef __diag
#define __diag(string)
......
include/linux/nospec.h
View file @ 0e4cd9f2
......@@ -5,6 +5,8 @@
#ifndef _LINUX_NOSPEC_H
#define _LINUX_NOSPEC_H
#include <linux/compiler.h>
#include <asm/barrier.h>
struct task_struct;
......
include/linux/percpu-refcount.h
View file @ 0e4cd9f2
......@@ -155,7 +155,7 @@ static inline bool __ref_is_percpu(struct percpu_ref *ref,
* between contaminating the pointer value, meaning that
* READ_ONCE() is required when fetching it.
*
* The smp_read_barrier_depends() implied by READ_ONCE() pairs
* The dependency ordering from the READ_ONCE() pairs
* with smp_store_release() in __percpu_ref_switch_to_percpu().
*/
percpu_ptr = READ_ONCE(ref->percpu_count_ptr);
......
include/linux/ptr_ring.h
View file @ 0e4cd9f2
......@@ -107,7 +107,7 @@ static inline int __ptr_ring_produce(struct ptr_ring *r, void *ptr)
return -ENOSPC;
/* Make sure the pointer we are storing points to a valid data. */
/* Pairs with smp_read_barrier_depends in __ptr_ring_consume. */
/* Pairs with the dependency ordering in __ptr_ring_consume. */
smp_wmb();
WRITE_ONCE(r->queue[r->producer++], ptr);
......
mm/memory.c
View file @ 0e4cd9f2
......@@ -437,7 +437,7 @@ int __pte_alloc(struct mm_struct *mm, pmd_t *pmd)
* of a chain of data-dependent loads, meaning most CPUs (alpha
* being the notable exception) will already guarantee loads are
* seen in-order. See the alpha page table accessors for the
* smp_read_barrier_depends() barriers in page table walking code.
* smp_rmb() barriers in page table walking code.
*/
smp_wmb(); /* Could be smp_wmb__xxx(before|after)_spin_lock */
......
scripts/checkpatch.pl
View file @ 0e4cd9f2
......@@ -5903,8 +5903,7 @@ sub process {
my $barriers = qr{
mb|
rmb|
wmb|
read_barrier_depends
wmb
}x;
my $barrier_stems = qr{
mb__before_atomic|
......@@ -5953,12 +5952,6 @@ sub process {
}
}
# check for smp_read_barrier_depends and read_barrier_depends
if (!$file && $line =~ /\b(smp_|)read_barrier_depends\s*\(/) {
WARN("READ_BARRIER_DEPENDS",
"$1read_barrier_depends should only be used in READ_ONCE or DEC Alpha code\n" . $herecurr);
}
# check of hardware specific defines
if ($line =~ m@^.\s*\#\s*if.*\b(__i386__|__powerpc64__|__sun__|__s390x__)\b@ && $realfile !~ m@include/asm-@) {
CHK("ARCH_DEFINES",
......
tools/bpf/Makefile
View file @ 0e4cd9f2
......@@ -9,7 +9,8 @@ MAKE = make
INSTALL ?= install
CFLAGS += -Wall -O2
CFLAGS += -D__EXPORTED_HEADERS__ -I$(srctree)/include/uapi -I$(srctree)/include
CFLAGS += -D__EXPORTED_HEADERS__ -I$(srctree)/tools/include/uapi \
-I$(srctree)/tools/include
# This will work when bpf is built in tools env. where srctree
# isn't set and when invoked from selftests build, where srctree
......
tools/include/uapi/linux/filter.h 0 → 100644
View file @ 0e4cd9f2
/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
/*
* Linux Socket Filter Data Structures
*/
#ifndef __LINUX_FILTER_H__
#define __LINUX_FILTER_H__
#include <linux/types.h>
#include <linux/bpf_common.h>
/*
* Current version of the filter code architecture.
*/
#define BPF_MAJOR_VERSION 1
#define BPF_MINOR_VERSION 1
/*
* Try and keep these values and structures similar to BSD, especially
* the BPF code definitions which need to match so you can share filters
*/
struct sock_filter { /* Filter block */
__u16 code; /* Actual filter code */
__u8 jt; /* Jump true */
__u8 jf; /* Jump false */
__u32 k; /* Generic multiuse field */
};
struct sock_fprog { /* Required for SO_ATTACH_FILTER. */
unsigned short len; /* Number of filter blocks */
struct sock_filter *filter;
};
/* ret - BPF_K and BPF_X also apply */
#define BPF_RVAL(code) ((code) & 0x18)
#define BPF_A 0x10
/* misc */
#define BPF_MISCOP(code) ((code) & 0xf8)
#define BPF_TAX 0x00
#define BPF_TXA 0x80
/*
* Macros for filter block array initializers.
*/
#ifndef BPF_STMT
#define BPF_STMT(code, k) { (unsigned short)(code), 0, 0, k }
#endif
#ifndef BPF_JUMP
#define BPF_JUMP(code, k, jt, jf) { (unsigned short)(code), jt, jf, k }
#endif
/*
* Number of scratch memory words for: BPF_ST and BPF_STX
*/
#define BPF_MEMWORDS 16
/* RATIONALE. Negative offsets are invalid in BPF.
We use them to reference ancillary data.
Unlike introduction new instructions, it does not break
existing compilers/optimizers.
*/
#define SKF_AD_OFF (-0x1000)
#define SKF_AD_PROTOCOL 0
#define SKF_AD_PKTTYPE 4
#define SKF_AD_IFINDEX 8
#define SKF_AD_NLATTR 12
#define SKF_AD_NLATTR_NEST 16
#define SKF_AD_MARK 20
#define SKF_AD_QUEUE 24
#define SKF_AD_HATYPE 28
#define SKF_AD_RXHASH 32
#define SKF_AD_CPU 36
#define SKF_AD_ALU_XOR_X 40
#define SKF_AD_VLAN_TAG 44
#define SKF_AD_VLAN_TAG_PRESENT 48
#define SKF_AD_PAY_OFFSET 52
#define SKF_AD_RANDOM 56
#define SKF_AD_VLAN_TPID 60
#define SKF_AD_MAX 64
#define SKF_NET_OFF (-0x100000)
#define SKF_LL_OFF (-0x200000)
#define BPF_NET_OFF SKF_NET_OFF
#define BPF_LL_OFF SKF_LL_OFF
#endif /* __LINUX_FILTER_H__ */
tools/memory-model/Documentation/explanation.txt
View file @ 0e4cd9f2
......@@ -1122,12 +1122,10 @@ maintain at least the appearance of FIFO order.
In practice, this difficulty is solved by inserting a special fence
between P1's two loads when the kernel is compiled for the Alpha
architecture. In fact, as of version 4.15, the kernel automatically
adds this fence (called smp_read_barrier_depends() and defined as
nothing at all on non-Alpha builds) after every READ_ONCE() and atomic
load. The effect of the fence is to cause the CPU not to execute any
po-later instructions until after the local cache has finished
processing all the stores it has already received. Thus, if the code
was changed to:
adds this fence after every READ_ONCE() and atomic load on Alpha. The
effect of the fence is to cause the CPU not to execute any po-later
instructions until after the local cache has finished processing all
the stores it has already received. Thus, if the code was changed to:
P1()
{
......@@ -1146,14 +1144,14 @@ READ_ONCE() or another synchronization primitive rather than accessed
directly.
The LKMM requires that smp_rmb(), acquire fences, and strong fences
share this property with smp_read_barrier_depends(): They do not allow
the CPU to execute any po-later instructions (or po-later loads in the
case of smp_rmb()) until all outstanding stores have been processed by
the local cache. In the case of a strong fence, the CPU first has to
wait for all of its po-earlier stores to propagate to every other CPU
in the system; then it has to wait for the local cache to process all
the stores received as of that time -- not just the stores received
when the strong fence began.
share this property: They do not allow the CPU to execute any po-later
instructions (or po-later loads in the case of smp_rmb()) until all
outstanding stores have been processed by the local cache. In the
case of a strong fence, the CPU first has to wait for all of its
po-earlier stores to propagate to every other CPU in the system; then
it has to wait for the local cache to process all the stores received
as of that time -- not just the stores received when the strong fence
began.
And of course, none of this matters for any architecture other than
Alpha.
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment
GitLab Nexedi Edition | About GitLab | About Nexedi | 沪ICP备2021021310号-2 | 沪ICP备2021021310号-7