mbox series

[SRU,J:linux-bluefield,v6,0/6] Add VFIO P2P support

Message ID 20240923223522.1203691-1-witu@nvidia.com
Headers show
Series Add VFIO P2P support | expand

Message

William Tu Sept. 23, 2024, 10:35 p.m. UTC 
BugLink: https://bugs.launchpad.net/bugs/2077887

The series adds support for VFIO P2P feature for NVMe target offload.
We took the patch (not upstreamed) below with some fix to make it work.
https://patchwork.kernel.org/project/linux-rdma/cover/0-v2-472615b3877e+28f7-vfio_dma_buf_jgg@nvidia.com/

* Kernel Config
Need to enable
CONFIG_PCI_P2PDMA (enable p2p dma transaction)
CONFIG_DMABUF_MOVE_NOTIFY (no need to pin down the memory)

The feature doesn't work with IOMMU, so either we disable it
for the system, "iommu.passthrough=1 iommu=off" in kernel boot
parameter, or conditionally disable it for VFIO.
CONFIG_VFIO_NOIOMMU (optional)
- You should enable CONFIG_VFIO_NOIOMMU in the kernel config if your
vfio module does not have vfio enable_unsafe_noiommu_mode in the parameter list.

* /etc/modprobe.d/vfio.conf
$ cat /etc/modprobe.d/vfio.conf
# The kernel hungs when 0000:06:00.0 (Non-Volatile memory controller: KIOXIA
# Corporation Device 0001) is bound to the vfio-pci module, and the following
# error is printed:
#
# vfio-pci 0000:06:00.0: Unable to change power state from D3hot to D0, device inaccessible
#
# The configuration below is a workarround for the issue.
options vfio-pci disable_idle_d3=1

# Allow using VFIO without IOMMU. This is only required when SMMU is disabled.
options vfio enable_unsafe_noiommu_mode=1

* Setup and Test
1. For ease of testing, we take a BF3 and boot the OS into mmc instead
of NVMe drive. So leaving NVMe drive free to use.

2. use a test program to verify its correctness.
The program create ibv rdma program, open nvme drive, create dma vfio
buf from nvme, and let the rdma program read (DMA P2P) the version
number of the nvme drive.
https://gitlab.com/Mellanox/spdk_team/vfio-dmabuf-test

* UAPI changes
see "vfio/pci: Allow MMIO regions to be exported through
    dma-buf" for detail

v6: 
- remove comments, fix the vfio_pci_set_power_state code

v5:
- add back the missing power management code, suggested by Bartlomiej

v4:
- fix Makefile

v1->v2:
- introduce new ioctl uAPI for vfio dma-buf

v2->v3:
- squash the ioctl uapi change into
  "vfio/pci: Allow MMIO regions to be exported through dma-buf"


Jason Gunthorpe (4):
  UBUNTU: SAUCE: dma-buf: Add dma_buf_try_get()
  UBUNTU: SAUCE: vfio: Add vfio_device_get()
  UBUNTU: SAUCE: vfio_pci: Do not open code pci_try_reset_function()
  UBUNTU: SAUCE: vfio/pci: Allow MMIO regions to be exported through
    dma-buf

Sergey Gorenko (1):
  UBUNTU: SAUCE: vfio/pci: Fix p2p address

William Tu (1):
  UBUNTU: [Config] bluefield: add config for VFIO P2P

 debian.bluefield/config/annotations |   2 +
 drivers/vfio/pci/Makefile           |   1 +
 drivers/vfio/pci/dma_buf.c          | 265 ++++++++++++++++++++++++++++
 drivers/vfio/pci/vfio_pci_config.c  |  24 +--
 drivers/vfio/pci/vfio_pci_core.c    |  72 +++++---
 drivers/vfio/pci/vfio_pci_priv.h    |  28 +++
 include/linux/dma-buf.h             |  13 ++
 include/linux/vfio.h                |   5 +
 include/linux/vfio_pci_core.h       |   1 +
 include/uapi/linux/vfio.h           |  17 ++
 10 files changed, 392 insertions(+), 36 deletions(-)
 create mode 100644 drivers/vfio/pci/dma_buf.c
 create mode 100644 drivers/vfio/pci/vfio_pci_priv.h

Comments

Bartlomiej Zolnierkiewicz Sept. 24, 2024, 1:51 p.m. UTC | #1 
On Tue, Sep 24, 2024 at 12:36 AM William Tu <witu@nvidia.com> wrote:
>
> BugLink: https://bugs.launchpad.net/bugs/2077887
>
> The series adds support for VFIO P2P feature for NVMe target offload.
> We took the patch (not upstreamed) below with some fix to make it work.
> https://patchwork.kernel.org/project/linux-rdma/cover/0-v2-472615b3877e+28f7-vfio_dma_buf_jgg@nvidia.com/
>
> * Kernel Config
> Need to enable
> CONFIG_PCI_P2PDMA (enable p2p dma transaction)
> CONFIG_DMABUF_MOVE_NOTIFY (no need to pin down the memory)
>
> The feature doesn't work with IOMMU, so either we disable it
> for the system, "iommu.passthrough=1 iommu=off" in kernel boot
> parameter, or conditionally disable it for VFIO.
> CONFIG_VFIO_NOIOMMU (optional)
> - You should enable CONFIG_VFIO_NOIOMMU in the kernel config if your
> vfio module does not have vfio enable_unsafe_noiommu_mode in the parameter list.
>
> * /etc/modprobe.d/vfio.conf
> $ cat /etc/modprobe.d/vfio.conf
> # The kernel hungs when 0000:06:00.0 (Non-Volatile memory controller: KIOXIA
> # Corporation Device 0001) is bound to the vfio-pci module, and the following
> # error is printed:
> #
> # vfio-pci 0000:06:00.0: Unable to change power state from D3hot to D0, device inaccessible
> #
> # The configuration below is a workarround for the issue.
> options vfio-pci disable_idle_d3=1
>
> # Allow using VFIO without IOMMU. This is only required when SMMU is disabled.
> options vfio enable_unsafe_noiommu_mode=1
>
> * Setup and Test
> 1. For ease of testing, we take a BF3 and boot the OS into mmc instead
> of NVMe drive. So leaving NVMe drive free to use.
>
> 2. use a test program to verify its correctness.
> The program create ibv rdma program, open nvme drive, create dma vfio
> buf from nvme, and let the rdma program read (DMA P2P) the version
> number of the nvme drive.
> https://gitlab.com/Mellanox/spdk_team/vfio-dmabuf-test
>
> * UAPI changes
> see "vfio/pci: Allow MMIO regions to be exported through
>     dma-buf" for detail
>
> v6:
> - remove comments, fix the vfio_pci_set_power_state code

Sorry but the fix is incorrect. vfio_pci_dma_buf_move(cur, true) call
inside the loop was removed and the vfio_pci_set_power_state(cur,
PCI_D0) call was added inside the loop. The code needs to have both
calls inside the loop to be correct.

--
Best regards,
Bartlomiej

> v5:
> - add back the missing power management code, suggested by Bartlomiej
>
> v4:
> - fix Makefile
>
> v1->v2:
> - introduce new ioctl uAPI for vfio dma-buf
>
> v2->v3:
> - squash the ioctl uapi change into
>   "vfio/pci: Allow MMIO regions to be exported through dma-buf"
>
>
> Jason Gunthorpe (4):
>   UBUNTU: SAUCE: dma-buf: Add dma_buf_try_get()
>   UBUNTU: SAUCE: vfio: Add vfio_device_get()
>   UBUNTU: SAUCE: vfio_pci: Do not open code pci_try_reset_function()
>   UBUNTU: SAUCE: vfio/pci: Allow MMIO regions to be exported through
>     dma-buf
>
> Sergey Gorenko (1):
>   UBUNTU: SAUCE: vfio/pci: Fix p2p address
>
> William Tu (1):
>   UBUNTU: [Config] bluefield: add config for VFIO P2P
>
>  debian.bluefield/config/annotations |   2 +
>  drivers/vfio/pci/Makefile           |   1 +
>  drivers/vfio/pci/dma_buf.c          | 265 ++++++++++++++++++++++++++++
>  drivers/vfio/pci/vfio_pci_config.c  |  24 +--
>  drivers/vfio/pci/vfio_pci_core.c    |  72 +++++---
>  drivers/vfio/pci/vfio_pci_priv.h    |  28 +++
>  include/linux/dma-buf.h             |  13 ++
>  include/linux/vfio.h                |   5 +
>  include/linux/vfio_pci_core.h       |   1 +
>  include/uapi/linux/vfio.h           |  17 ++
>  10 files changed, 392 insertions(+), 36 deletions(-)
>  create mode 100644 drivers/vfio/pci/dma_buf.c
>  create mode 100644 drivers/vfio/pci/vfio_pci_priv.h
>
> --
> 2.34.1
>
>
> --
> kernel-team mailing list
> kernel-team@lists.ubuntu.com
> https://lists.ubuntu.com/mailman/listinfo/kernel-team
William Tu Sept. 24, 2024, 1:59 p.m. UTC | #2 
Hi Bartlomiej,

Thanks for your review. I will fix it.
William

From: Bartlomiej Zolnierkiewicz <bartlomiej.zolnierkiewicz@canonical.com>
Date: Tuesday, September 24, 2024 at 6:52 AM
To: William Tu <witu@nvidia.com>
Cc: Ubuntu Kernel Team <kernel-team@lists.ubuntu.com>, Vladimir Sokolovsky <vlad@nvidia.com>, Bodong Wang <bodong@nvidia.com>, Sergey Gorenko <sergeygo@nvidia.com>, Jason Gunthorpe <jgg@nvidia.com>, Ziv Waksman <zwaksman@nvidia.com>
Subject: NACK/Cmnt: [SRU][J:linux-bluefield][PATCH v6 0/6] Add VFIO P2P support
External email: Use caution opening links or attachments


On Tue, Sep 24, 2024 at 12:36 AM William Tu <witu@nvidia.com> wrote:
>
> BugLink: https://bugs.launchpad.net/bugs/2077887
>
> The series adds support for VFIO P2P feature for NVMe target offload.
> We took the patch (not upstreamed) below with some fix to make it work.
> https://patchwork.kernel.org/project/linux-rdma/cover/0-v2-472615b3877e+28f7-vfio_dma_buf_jgg@nvidia.com/
>
> * Kernel Config
> Need to enable
> CONFIG_PCI_P2PDMA (enable p2p dma transaction)
> CONFIG_DMABUF_MOVE_NOTIFY (no need to pin down the memory)
>
> The feature doesn't work with IOMMU, so either we disable it
> for the system, "iommu.passthrough=1 iommu=off" in kernel boot
> parameter, or conditionally disable it for VFIO.
> CONFIG_VFIO_NOIOMMU (optional)
> - You should enable CONFIG_VFIO_NOIOMMU in the kernel config if your
> vfio module does not have vfio enable_unsafe_noiommu_mode in the parameter list.
>
> * /etc/modprobe.d/vfio.conf
> $ cat /etc/modprobe.d/vfio.conf
> # The kernel hungs when 0000:06:00.0 (Non-Volatile memory controller: KIOXIA
> # Corporation Device 0001) is bound to the vfio-pci module, and the following
> # error is printed:
> #
> # vfio-pci 0000:06:00.0: Unable to change power state from D3hot to D0, device inaccessible
> #
> # The configuration below is a workarround for the issue.
> options vfio-pci disable_idle_d3=1
>
> # Allow using VFIO without IOMMU. This is only required when SMMU is disabled.
> options vfio enable_unsafe_noiommu_mode=1
>
> * Setup and Test
> 1. For ease of testing, we take a BF3 and boot the OS into mmc instead
> of NVMe drive. So leaving NVMe drive free to use.
>
> 2. use a test program to verify its correctness.
> The program create ibv rdma program, open nvme drive, create dma vfio
> buf from nvme, and let the rdma program read (DMA P2P) the version
> number of the nvme drive.
> https://gitlab.com/Mellanox/spdk_team/vfio-dmabuf-test
>
> * UAPI changes
> see "vfio/pci: Allow MMIO regions to be exported through
>     dma-buf" for detail
>
> v6:
> - remove comments, fix the vfio_pci_set_power_state code

Sorry but the fix is incorrect. vfio_pci_dma_buf_move(cur, true) call
inside the loop was removed and the vfio_pci_set_power_state(cur,
PCI_D0) call was added inside the loop. The code needs to have both
calls inside the loop to be correct.

--
Best regards,
Bartlomiej

> v5:
> - add back the missing power management code, suggested by Bartlomiej
>
> v4:
> - fix Makefile
>
> v1->v2:
> - introduce new ioctl uAPI for vfio dma-buf
>
> v2->v3:
> - squash the ioctl uapi change into
>   "vfio/pci: Allow MMIO regions to be exported through dma-buf"
>
>
> Jason Gunthorpe (4):
>   UBUNTU: SAUCE: dma-buf: Add dma_buf_try_get()
>   UBUNTU: SAUCE: vfio: Add vfio_device_get()
>   UBUNTU: SAUCE: vfio_pci: Do not open code pci_try_reset_function()
>   UBUNTU: SAUCE: vfio/pci: Allow MMIO regions to be exported through
>     dma-buf
>
> Sergey Gorenko (1):
>   UBUNTU: SAUCE: vfio/pci: Fix p2p address
>
> William Tu (1):
>   UBUNTU: [Config] bluefield: add config for VFIO P2P
>
>  debian.bluefield/config/annotations |   2 +
>  drivers/vfio/pci/Makefile           |   1 +
>  drivers/vfio/pci/dma_buf.c          | 265 ++++++++++++++++++++++++++++
>  drivers/vfio/pci/vfio_pci_config.c  |  24 +--
>  drivers/vfio/pci/vfio_pci_core.c    |  72 +++++---
>  drivers/vfio/pci/vfio_pci_priv.h    |  28 +++
>  include/linux/dma-buf.h             |  13 ++
>  include/linux/vfio.h                |   5 +
>  include/linux/vfio_pci_core.h       |   1 +
>  include/uapi/linux/vfio.h           |  17 ++
>  10 files changed, 392 insertions(+), 36 deletions(-)
>  create mode 100644 drivers/vfio/pci/dma_buf.c
>  create mode 100644 drivers/vfio/pci/vfio_pci_priv.h
>
> --
> 2.34.1
>
>
> --
> kernel-team mailing list
> kernel-team@lists.ubuntu.com
> https://lists.ubuntu.com/mailman/listinfo/kernel-team