Commit b69e2ef2 authored by Keith Busch's avatar Keith Busch Committed by Christoph Hellwig

nvme-pci: dma read memory barrier for completions

Control dependencies do not guarantee load order across the condition,
allowing a CPU to predict and speculate memory reads.

Commit 324b494c inlined verifying a new completion with its
handling. At least one architecture was observed to access the contents
out of order, resulting in the driver using stale data for the
completion.

Add a dma read barrier before reading the completion queue entry and
after the condition its contents depend on to ensure the read order is
determinsitic.
Reported-by: default avatarJohn Garry <john.garry@huawei.com>
Suggested-by: default avatarWill Deacon <will@kernel.org>
Signed-off-by: default avatarKeith Busch <kbusch@kernel.org>
Tested-by: default avatarJohn Garry <john.garry@huawei.com>
Acked-by: default avatarWill Deacon <will@kernel.org>
Reviewed-by: default avatarSagi Grimberg <sagi@grimberg.me>
Signed-off-by: default avatarChristoph Hellwig <hch@lst.de>
parent 59c7c3ca
...@@ -989,6 +989,11 @@ static inline int nvme_process_cq(struct nvme_queue *nvmeq) ...@@ -989,6 +989,11 @@ static inline int nvme_process_cq(struct nvme_queue *nvmeq)
while (nvme_cqe_pending(nvmeq)) { while (nvme_cqe_pending(nvmeq)) {
found++; found++;
/*
* load-load control dependency between phase and the rest of
* the cqe requires a full read memory barrier
*/
dma_rmb();
nvme_handle_cqe(nvmeq, nvmeq->cq_head); nvme_handle_cqe(nvmeq, nvmeq->cq_head);
nvme_update_cq_head(nvmeq); nvme_update_cq_head(nvmeq);
} }
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment