linux/drivers/nvme/host
Michal Wnukowski f1ed3df20d nvme-pci: add a memory barrier to nvme_dbbuf_update_and_check_event
In many architectures loads may be reordered with older stores to
different locations.  In the nvme driver the following two operations
could be reordered:

 - Write shadow doorbell (dbbuf_db) into memory.
 - Read EventIdx (dbbuf_ei) from memory.

This can result in a potential race condition between driver and VM host
processing requests (if given virtual NVMe controller has a support for
shadow doorbell).  If that occurs, then the NVMe controller may decide to
wait for MMIO doorbell from guest operating system, and guest driver may
decide not to issue MMIO doorbell on any of subsequent commands.

This issue is purely timing-dependent one, so there is no easy way to
reproduce it. Currently the easiest known approach is to run "Oracle IO
Numbers" (orion) that is shipped with Oracle DB:

orion -run advanced -num_large 0 -size_small 8 -type rand -simulate \
	concat -write 40 -duration 120 -matrix row -testname nvme_test

Where nvme_test is a .lun file that contains a list of NVMe block
devices to run test against. Limiting number of vCPUs assigned to given
VM instance seems to increase chances for this bug to occur. On test
environment with VM that got 4 NVMe drives and 1 vCPU assigned the
virtual NVMe controller hang could be observed within 10-20 minutes.
That correspond to about 400-500k IO operations processed (or about
100GB of IO read/writes).

Orion tool was used as a validation and set to run in a loop for 36
hours (equivalent of pushing 550M IO operations). No issues were
observed. That suggest that the patch fixes the issue.

Fixes: f9f38e3338 ("nvme: improve performance for virtual NVMe devices")
Signed-off-by: Michal Wnukowski <wnukowski@google.com>
Reviewed-by: Keith Busch <keith.busch@intel.com>
Reviewed-by: Sagi Grimberg <sagi@grimberg.me>
[hch: updated changelog and comment a bit]
Signed-off-by: Christoph Hellwig <hch@lst.de>
2018-08-28 08:40:42 +02:00
..
core.c nvme: set gendisk read only based on nsattr 2018-08-08 11:55:49 +02:00
fabrics.c for-4.19/block-20180812 2018-08-14 10:23:25 -07:00
fabrics.h nvme: if_ready checks to fail io to deleting controller 2018-07-24 13:44:40 +02:00
fault_inject.c nvme: Add fault injection feature 2018-03-26 08:53:43 -06:00
fc.c for-4.19/block-20180812 2018-08-14 10:23:25 -07:00
Kconfig IB: Revert "remove redundant INFINIBAND kconfig dependencies" 2018-05-28 10:40:16 -06:00
lightnvm.c lightnvm: remove minor version check for 2.0 2018-08-05 19:36:09 -06:00
Makefile nvme: Add fault injection feature 2018-03-26 08:53:43 -06:00
multipath.c nvme: fixup crash on failed discovery 2018-08-07 14:40:27 +02:00
nvme.h Merge branch 'nvme-4.19' of git://git.infradead.org/nvme into for-4.19/block2 2018-08-05 19:34:09 -06:00
pci.c nvme-pci: add a memory barrier to nvme_dbbuf_update_and_check_event 2018-08-28 08:40:42 +02:00
rdma.c Merge branch 'linus/master' into rdma.git for-next 2018-08-16 14:21:29 -06:00
trace.c nvme: add disk name to trace events 2018-07-24 15:55:48 +02:00
trace.h nvme: add disk name to trace events 2018-07-24 15:55:48 +02:00