| Message ID | 20231102071302.1818071-1-zhenzhong.duan@intel.com |
|---|---|
| Headers | show |
| Series | vfio: Adopt iommufd | expand |
On 11/2/23 08:12, Zhenzhong Duan wrote: > Hi, > > Thanks all for giving guides and comments on previous series, here is > the v4 of pure iommufd support part. > > Based on Cédric's suggestion, this series includes an effort to remove > spapr code from container.c, now all spapr functions are moved to spapr.c > or spapr_pci_vfio.c, but there are still a few trival check on > VFIO_SPAPR_TCE_*_IOMMU which I am not sure if deserved to introduce many > callbacks and duplicate code just to remove them. Some functions are moved > to spapr.c instead of spapr_pci_vfio.c to avoid compile issue because > spapr_pci_vfio.c is arch specific, or else we need to introduce stub > functions to those spapr functions moved. > > > PATCH 1-5: Move spapr functions to spapr*.c PATCH 1-5 applied to vfio-next. Thanks, C.
On 11/2/23 08:12, Zhenzhong Duan wrote: > Hi, > > Thanks all for giving guides and comments on previous series, here is > the v4 of pure iommufd support part. > > Based on Cédric's suggestion, this series includes an effort to remove > spapr code from container.c, now all spapr functions are moved to spapr.c > or spapr_pci_vfio.c, but there are still a few trival check on > VFIO_SPAPR_TCE_*_IOMMU which I am not sure if deserved to introduce many > callbacks and duplicate code just to remove them. Some functions are moved > to spapr.c instead of spapr_pci_vfio.c to avoid compile issue because > spapr_pci_vfio.c is arch specific, or else we need to introduce stub > functions to those spapr functions moved. > > > PATCH 1-5: Move spapr functions to spapr*.c > PATCH 6-20: Abstract out base container > PATCH 21-24: Introduce sparpr container and its specific interface PATCH 6-24 applied to vfio-next : https://github.com/legoater/qemu/commits/vfio-next (with a global s/fucntional/functional/) I also pushed the remaining patches on : https://github.com/legoater/qemu/commits/vfio-8.2 with a slight rework of the IOMMUFD configuration, now done per platform. The VFIO frontend and the 'iommufd' object are only available on x86_64, arm, s390x. Thanks, C. > PATCH 25: Add --enable/--disable-iommufd config support > PATCH 26: Introduce iommufd object > PATCH 27-33: add IOMMUFD container and cdev support > PATCH 34-39: fd passing for IOMMUFD object and cdev > PATCH 40: make VFIOContainerBase parameter const > PATCH 41: Compile out for PPC > > > We have done wide test with different combinations, e.g: > - PCI device were tested > - FD passing and hot reset with some trick. > - device hotplug test with legacy and iommufd backends > - with or without vIOMMU for legacy and iommufd backends > - divices linked to different iommufds > - VFIO migration with a E800 net card(no dirty sync support) passthrough > - platform, ccw and ap were only compile-tested due to environment limit > > > Given some iommufd kernel limitations, the iommufd backend is > not yet fully on par with the legacy backend w.r.t. features like: > - p2p mappings (you will see related error traces) > - dirty page sync > - and etc. > > > qemu code: https://github.com/yiliu1765/qemu/commits/zhenzhong/iommufd_cdev_v4 > Based on vfio-next, commit id: f686924775 > > -------------------------------------------------------------------------- > > Below are some background and graph about the design: > > With the introduction of iommufd, the Linux kernel provides a generic > interface for userspace drivers to propagate their DMA mappings to kernel > for assigned devices. This series does the porting of the VFIO devices > onto the /dev/iommu uapi and let it coexist with the legacy implementation. > > At QEMU level, interactions with the /dev/iommu are abstracted by a new > iommufd object (compiled in with the CONFIG_IOMMUFD option). > > Any QEMU device (e.g. vfio device) wishing to use /dev/iommu must be > linked with an iommufd object. In this series, the vfio-pci device is > granted with such capability (other VFIO devices are not yet ready): > > It gets a new optional parameter named iommufd which allows to pass > an iommufd object: > > -object iommufd,id=iommufd0 > -device vfio-pci,host=0000:02:00.0,iommufd=iommufd0 > > Note the /dev/iommu and vfio cdev can be externally opened by a > management layer. In such a case the fd is passed: > > -object iommufd,id=iommufd0,fd=22 > -device vfio-pci,iommufd=iommufd0,fd=23 > > If the fd parameter is not passed, the fd is opened by QEMU. > See https://www.mail-archive.com/qemu-devel@nongnu.org/msg937155.html > for detailed discuss on this requirement. > > If no iommufd option is passed to the vfio-pci device, iommufd is not > used and the end-user gets the behavior based on the legacy vfio iommu > interfaces: > > -device vfio-pci,host=0000:02:00.0 > > While the legacy kernel interface is group-centric, the new iommufd > interface is device-centric, relying on device fd and iommufd. > > To support both interfaces in the QEMU VFIO device we reworked the vfio > container abstraction so that the generic VFIO code can use either > backend. > > The VFIOContainer object becomes a base object derived into > a) the legacy VFIO container and > b) the new iommufd based container. > > The base object implements generic code such as code related to > memory_listener and address space management whereas the derived > objects implement callbacks specific to either BE, legacy and > iommufd. Indeed each backend has its own way to setup secure context > and dma management interface. The below diagram shows how it looks > like with both BEs. > > VFIO AddressSpace/Memory > +-------+ +----------+ +-----+ +-----+ > | pci | | platform | | ap | | ccw | > +---+---+ +----+-----+ +--+--+ +--+--+ +----------------------+ > | | | | | AddressSpace | > | | | | +------------+---------+ > +---V-----------V-----------V--------V----+ / > | VFIOAddressSpace | <------------+ > | | | MemoryListener > | VFIOContainer list | > +-------+----------------------------+----+ > | | > | | > +-------V------+ +--------V----------+ > | iommufd | | vfio legacy | > | container | | container | > +-------+------+ +--------+----------+ > | | > | /dev/iommu | /dev/vfio/vfio > | /dev/vfio/devices/vfioX | /dev/vfio/$group_id > Userspace | | > ============+============================+=========================== > Kernel | device fd | > +---------------+ | group/container fd > | (BIND_IOMMUFD | | (SET_CONTAINER/SET_IOMMU) > | ATTACH_IOAS) | | device fd > | | | > | +-------V------------V-----------------+ > iommufd | | vfio | > (map/unmap | +---------+--------------------+-------+ > ioas_copy) | | | map/unmap > | | | > +------V------+ +-----V------+ +------V--------+ > | iommfd core | | device | | vfio iommu | > +-------------+ +------------+ +---------------+ > > [Secure Context setup] > - iommufd BE: uses device fd and iommufd to setup secure context > (bind_iommufd, attach_ioas) > - vfio legacy BE: uses group fd and container fd to setup secure context > (set_container, set_iommu) > [Device access] > - iommufd BE: device fd is opened through /dev/vfio/devices/vfioX > - vfio legacy BE: device fd is retrieved from group fd ioctl > [DMA Mapping flow] > 1. VFIOAddressSpace receives MemoryRegion add/del via MemoryListener > 2. VFIO populates DMA map/unmap via the container BEs > *) iommufd BE: uses iommufd > *) vfio legacy BE: uses container fd > > > Changelog: > v4: > - add CONFIG_IOMMUFD check for IOMMUFDProperties (Markus) > - add doc for default case without fd (Markus) > - Fix build issue reported by Markus and Cédric > - Simply use SPDX identifier in new file (Cédric) > - make vfio_container_init/destroy helper a seperate patch (Cédric) > - make vrdl_list movement a seperate patch (Cédric) > - add const for some callback parameters (Cédric) > - add g_assert in VFIOIOMMUOps callback (Cédric) > - introduce pci_hot_reset callback (Cédric) > - remove VFIOIOMMUSpaprOps (Cédric) > - initialize g_autofree to NULL (Cédric) > - adjust func name prefix and trace event in iommufd.c (Cédric) > - add RB > > v3: > - Rename base container as VFIOContainerBase and legacy container as container (Cédric) > - Drop VFIO_IOMMU_BACKEND_OPS class and use struct instead (Cédric) > - Cleanup container.c by introducing spapr backend and move spapr code out (Cédric) > - Introduce vfio_iommu_spapr_ops (Cédric) > - Add doc of iommufd in qom.json and have iommufd member sorted (Markus) > - patch19 and patch21 are splitted to two parts to facilitate review > > v2: > - patch "vfio: Add base container" in v1 is split into patch1-15 per Cédric > - add fd passing to platform/ap/ccw vfio device > - add (uintptr_t) cast in iommufd_backend_map_dma() per Cédric > - rename char_dev.h to chardev_open.h for same naming scheme per Daniel > - add full copyright per Daniel and Jason > > > Note changelog below are from full IOMMUFD series: > > v1: > - Alloc hwpt instead of using auto hwpt > - elaborate iommufd code per Nicolin > - consolidate two patches and drop as.c > - typo error fix and function rename > > rfcv4: > - rebase on top of v8.0.3 > - Add one patch from Yi which is about vfio device add in kvm > - Remove IOAS_COPY optimization and focus on functions in this patchset > - Fix wrong name issue reported and fix suggested by Matthew > - Fix compilation issue reported and fix sugggsted by Nicolin > - Use query_dirty_bitmap callback to replace get_dirty_bitmap for better > granularity > - Add dev_iter_next() callback to avoid adding so many callback > at container scope, add VFIODevice.hwpt to support that > - Restore all functions back to common from container whenever possible, > mainly migration and reset related functions > - Add --enable/disable-iommufd config option, enabled by default in linux > - Remove VFIODevice.hwpt_next as it's redundant with VFIODevice.next > - Adapt new VFIO_DEVICE_PCI_HOT_RESET uAPI for IOMMUFD backed device > - vfio_kvm_device_add/del_group call vfio_kvm_device_add/del_fd to remove > redundant code > - Add FD passing support for vfio device backed by IOMMUFD > - Fix hot unplug resource leak issue in vfio_legacy_detach_device() > - Fix FD leak in vfio_get_devicefd() > > rfcv3: > - rebase on top of v7.2.0 > - Fix the compilation with CONFIG_IOMMUFD unset by using true classes for > VFIO backends > - Fix use after free in error path, reported by Alister > - Split common.c in several steps to ease the review > > rfcv2: > - remove the first three patches of rfcv1 > - add open cdev helper suggested by Jason > - remove the QOMification of the VFIOContainer and simply use standard ops > (David) > - add "-object iommufd" suggested by Alex > > Thanks > Zhenzhong > > Eric Auger (11): > vfio/container: Switch to dma_map|unmap API > vfio/common: Move giommu_list in base container > vfio/container: Move space field to base container > vfio/container: Switch to IOMMU BE > set_dirty_page_tracking/query_dirty_bitmap API > vfio/container: Convert functions to base container > vfio/container: Move pgsizes and dma_max_mappings to base container > vfio/container: Move listener to base container > vfio/container: Move dirty_pgsizes and max_dirty_bitmap_size to base > container > vfio/container: Implement attach/detach_device > backends/iommufd: Introduce the iommufd object > vfio/pci: Allow the selection of a given iommu backend > > Yi Liu (2): > util/char_dev: Add open_cdev() > vfio/iommufd: Implement the iommufd backend > > Zhenzhong Duan (28): > vfio/container: Move IBM EEH related functions into spapr_pci_vfio.c > vfio/container: Move vfio_container_add/del_section_window into > spapr.c > vfio/container: Move spapr specific init/deinit into spapr.c > vfio/spapr: Make vfio_spapr_create/remove_window static > vfio/common: Move vfio_host_win_add/del into spapr.c > vfio: Introduce base object for VFIOContainer and targeted interface > vfio/container: Introduce a empty VFIOIOMMUOps > vfio/common: Introduce vfio_container_init/destroy helper > vfio/container: Move per container device list in base container > vfio/container: Move vrdl_list to base container > vfio/container: Move iova_ranges to base container > vfio/spapr: Introduce spapr backend and target interface > vfio/spapr: switch to spapr IOMMU BE add/del_section_window > vfio/spapr: Move prereg_listener into spapr container > vfio/spapr: Move hostwin_list into spapr container > Add iommufd configure option > vfio/iommufd: Relax assert check for iommufd backend > vfio/iommufd: Add support for iova_ranges > vfio/pci: Extract out a helper vfio_pci_get_pci_hot_reset_info > vfio/pci: Introduce a vfio pci hot reset interface > vfio/iommufd: Enable pci hot reset through iommufd cdev interface > vfio/pci: Make vfio cdev pre-openable by passing a file handle > vfio: Allow the selection of a given iommu backend for platform ap and > ccw > vfio/platform: Make vfio cdev pre-openable by passing a file handle > vfio/ap: Make vfio cdev pre-openable by passing a file handle > vfio/ccw: Make vfio cdev pre-openable by passing a file handle > vfio: Make VFIOContainerBase poiner parameter const in VFIOIOMMUOps > callbacks > vfio: Compile out iommufd for PPC target > > MAINTAINERS | 13 + > meson.build | 6 + > qapi/qom.json | 22 + > hw/vfio/pci.h | 6 + > include/hw/vfio/vfio-common.h | 118 ++--- > include/hw/vfio/vfio-container-base.h | 121 +++++ > include/hw/vfio/vfio-platform.h | 1 + > include/hw/vfio/vfio.h | 7 - > include/qemu/chardev_open.h | 16 + > include/sysemu/iommufd.h | 46 ++ > backends/iommufd-stub.c | 59 +++ > backends/iommufd.c | 257 ++++++++++ > hw/ppc/spapr_pci_vfio.c | 100 +++- > hw/vfio/ap.c | 38 +- > hw/vfio/ccw.c | 40 +- > hw/vfio/common.c | 330 ++++++------ > hw/vfio/container-base.c | 101 ++++ > hw/vfio/container.c | 443 ++++------------ > hw/vfio/helpers.c | 34 +- > hw/vfio/iommufd.c | 697 ++++++++++++++++++++++++++ > hw/vfio/pci.c | 112 +++-- > hw/vfio/platform.c | 45 +- > hw/vfio/spapr.c | 338 ++++++++++++- > util/chardev_open.c | 81 +++ > backends/Kconfig | 4 + > backends/meson.build | 5 + > backends/trace-events | 12 + > hw/vfio/meson.build | 4 + > hw/vfio/trace-events | 18 +- > meson_options.txt | 2 + > qemu-options.hx | 13 + > scripts/meson-buildoptions.sh | 3 + > util/meson.build | 1 + > 33 files changed, 2403 insertions(+), 690 deletions(-) > create mode 100644 include/hw/vfio/vfio-container-base.h > delete mode 100644 include/hw/vfio/vfio.h > create mode 100644 include/qemu/chardev_open.h > create mode 100644 include/sysemu/iommufd.h > create mode 100644 backends/iommufd-stub.c > create mode 100644 backends/iommufd.c > create mode 100644 hw/vfio/container-base.c > create mode 100644 hw/vfio/iommufd.c > create mode 100644 util/chardev_open.c >
On 11/7/23 1:28 PM, Cédric Le Goater wrote: > On 11/2/23 08:12, Zhenzhong Duan wrote: >> Hi, >> >> Thanks all for giving guides and comments on previous series, here is >> the v4 of pure iommufd support part. >> >> Based on Cédric's suggestion, this series includes an effort to remove >> spapr code from container.c, now all spapr functions are moved to spapr.c >> or spapr_pci_vfio.c, but there are still a few trival check on >> VFIO_SPAPR_TCE_*_IOMMU which I am not sure if deserved to introduce many >> callbacks and duplicate code just to remove them. Some functions are moved >> to spapr.c instead of spapr_pci_vfio.c to avoid compile issue because >> spapr_pci_vfio.c is arch specific, or else we need to introduce stub >> functions to those spapr functions moved. >> >> >> PATCH 1-5: Move spapr functions to spapr*.c >> PATCH 6-20: Abstract out base container >> PATCH 21-24: Introduce sparpr container and its specific interface > > PATCH 6-24 applied to vfio-next : > > https://github.com/legoater/qemu/commits/vfio-next > > (with a global s/fucntional/functional/) > > > I also pushed the remaining patches on : > > https://github.com/legoater/qemu/commits/vfio-8.2 > > with a slight rework of the IOMMUFD configuration, now done per platform. > The VFIO frontend and the 'iommufd' object are only available on x86_64, > arm, s390x. FYI, I first tried this vfio-8.2 branch on s390x but wasn't actually able to use the iommufd backend (was getting errors like Property 'vfio-pci.iommufd' not found) so I think something isn't actually enabling IOMMUFD as expected with your change... Instead I tested on s390x using vfio-next + patches 25-41 of this series on top. Legacy backend regression testing worked fine for vfio-pci, vfio-ap and vfio-ccw. Using iommufd backend for vfio-pci on s390 exposes an s390-only issue related to accounting of vfio DMA limit (code in hw/s390x/s390-pci-vfio.c assumes VFIODevice.group is never null, but that's no longer true when we use the iommufd backend with cdev). We don't even need to track this when using the iommufd backend -- With that issue bypassed, vfio-pci testing on s390x looks good so far. I'll send a separate fix for that. Using the iommufd backend for vfio-ccw and vfio-ap did not work, see response to patch 28. Thanks, Matt
>-----Original Message----- >From: Matthew Rosato <mjrosato@linux.ibm.com> >Sent: Wednesday, November 8, 2023 11:27 AM >Subject: Re: [PATCH v4 00/41] vfio: Adopt iommufd > >On 11/7/23 1:28 PM, Cédric Le Goater wrote: >> On 11/2/23 08:12, Zhenzhong Duan wrote: >>> Hi, >>> >>> Thanks all for giving guides and comments on previous series, here is >>> the v4 of pure iommufd support part. >>> >>> Based on Cédric's suggestion, this series includes an effort to remove >>> spapr code from container.c, now all spapr functions are moved to spapr.c >>> or spapr_pci_vfio.c, but there are still a few trival check on >>> VFIO_SPAPR_TCE_*_IOMMU which I am not sure if deserved to introduce >many >>> callbacks and duplicate code just to remove them. Some functions are moved >>> to spapr.c instead of spapr_pci_vfio.c to avoid compile issue because >>> spapr_pci_vfio.c is arch specific, or else we need to introduce stub >>> functions to those spapr functions moved. >>> >>> >>> PATCH 1-5: Move spapr functions to spapr*.c >>> PATCH 6-20: Abstract out base container >>> PATCH 21-24: Introduce sparpr container and its specific interface >> >> PATCH 6-24 applied to vfio-next : >> >> https://github.com/legoater/qemu/commits/vfio-next >> >> (with a global s/fucntional/functional/) >> >> >> I also pushed the remaining patches on : >> >> https://github.com/legoater/qemu/commits/vfio-8.2 >> >> with a slight rework of the IOMMUFD configuration, now done per platform. >> The VFIO frontend and the 'iommufd' object are only available on x86_64, >> arm, s390x. Thanks Cédric. > >FYI, I first tried this vfio-8.2 branch on s390x but wasn't actually able to use the >iommufd backend (was getting errors like Property 'vfio-pci.iommufd' not found) >so I think something isn't actually enabling IOMMUFD as expected with your >change... It looks CONFIG_IOMMUFD is recognized by Kconfig sub-system but not received by compiler. I'm still digging how to pass CONFIG_IOMMUFD to compiler. > >Instead I tested on s390x using vfio-next + patches 25-41 of this series on top. > >Legacy backend regression testing worked fine for vfio-pci, vfio-ap and vfio-ccw. > >Using iommufd backend for vfio-pci on s390 exposes an s390-only issue related >to accounting of vfio DMA limit (code in hw/s390x/s390-pci-vfio.c assumes >VFIODevice.group is never null, but that's no longer true when we use the >iommufd backend with cdev). We don't even need to track this when using the >iommufd backend -- With that issue bypassed, vfio-pci testing on s390x looks >good so far. I'll send a separate fix for that. Thanks for fixing that. BRs. Zhenzhong > >Using the iommufd backend for vfio-ccw and vfio-ap did not work, see response >to patch 28. > >Thanks, >Matt
>-----Original Message----- >From: Duan, Zhenzhong >Sent: Wednesday, November 8, 2023 4:38 PM >Subject: RE: [PATCH v4 00/41] vfio: Adopt iommufd > > > >>-----Original Message----- >>From: Matthew Rosato <mjrosato@linux.ibm.com> >>Sent: Wednesday, November 8, 2023 11:27 AM >>Subject: Re: [PATCH v4 00/41] vfio: Adopt iommufd >> >>On 11/7/23 1:28 PM, Cédric Le Goater wrote: >>> On 11/2/23 08:12, Zhenzhong Duan wrote: >>>> Hi, >>>> >>>> Thanks all for giving guides and comments on previous series, here is >>>> the v4 of pure iommufd support part. >>>> >>>> Based on Cédric's suggestion, this series includes an effort to remove >>>> spapr code from container.c, now all spapr functions are moved to spapr.c >>>> or spapr_pci_vfio.c, but there are still a few trival check on >>>> VFIO_SPAPR_TCE_*_IOMMU which I am not sure if deserved to introduce >>many >>>> callbacks and duplicate code just to remove them. Some functions are moved >>>> to spapr.c instead of spapr_pci_vfio.c to avoid compile issue because >>>> spapr_pci_vfio.c is arch specific, or else we need to introduce stub >>>> functions to those spapr functions moved. >>>> >>>> >>>> PATCH 1-5: Move spapr functions to spapr*.c >>>> PATCH 6-20: Abstract out base container >>>> PATCH 21-24: Introduce sparpr container and its specific interface >>> >>> PATCH 6-24 applied to vfio-next : >>> >>> https://github.com/legoater/qemu/commits/vfio-next >>> >>> (with a global s/fucntional/functional/) >>> >>> >>> I also pushed the remaining patches on : >>> >>> https://github.com/legoater/qemu/commits/vfio-8.2 >>> >>> with a slight rework of the IOMMUFD configuration, now done per platform. >>> The VFIO frontend and the 'iommufd' object are only available on x86_64, >>> arm, s390x. > >Thanks Cédric. > >> >>FYI, I first tried this vfio-8.2 branch on s390x but wasn't actually able to use the >>iommufd backend (was getting errors like Property 'vfio-pci.iommufd' not found) >>so I think something isn't actually enabling IOMMUFD as expected with your >>change... > >It looks CONFIG_IOMMUFD is recognized by Kconfig sub-system but not received >by compiler. I'm still digging how to pass CONFIG_IOMMUFD to compiler. Need below change to pass CONFIG_IOMMUFD to compiler. diff --git a/hw/vfio/ap.c b/hw/vfio/ap.c index 0a810f8b88..2a3263b51f 100644 --- a/hw/vfio/ap.c +++ b/hw/vfio/ap.c @@ -30,6 +30,7 @@ #include "exec/address-spaces.h" #include "qom/object.h" #include "monitor/monitor.h" +#include CONFIG_DEVICES #define TYPE_VFIO_AP_DEVICE "vfio-ap" diff --git a/hw/vfio/ccw.c b/hw/vfio/ccw.c index a674bd8d6d..08101ad445 100644 --- a/hw/vfio/ccw.c +++ b/hw/vfio/ccw.c @@ -31,6 +31,7 @@ #include "qemu/main-loop.h" #include "qemu/module.h" #include "monitor/monitor.h" +#include CONFIG_DEVICES struct VFIOCCWDevice { S390CCWDevice cdev; diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index d8f658ea47..3121b5f985 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -44,6 +44,7 @@ #include "migration/qemu-file.h" #include "sysemu/iommufd.h" #include "monitor/monitor.h" +#include CONFIG_DEVICES #define TYPE_VFIO_PCI_NOHOTPLUG "vfio-pci-nohotplug" Thanks Zhenzhong
On 11/8/23 04:26, Matthew Rosato wrote: > On 11/7/23 1:28 PM, Cédric Le Goater wrote: >> On 11/2/23 08:12, Zhenzhong Duan wrote: >>> Hi, >>> >>> Thanks all for giving guides and comments on previous series, here is >>> the v4 of pure iommufd support part. >>> >>> Based on Cédric's suggestion, this series includes an effort to remove >>> spapr code from container.c, now all spapr functions are moved to spapr.c >>> or spapr_pci_vfio.c, but there are still a few trival check on >>> VFIO_SPAPR_TCE_*_IOMMU which I am not sure if deserved to introduce many >>> callbacks and duplicate code just to remove them. Some functions are moved >>> to spapr.c instead of spapr_pci_vfio.c to avoid compile issue because >>> spapr_pci_vfio.c is arch specific, or else we need to introduce stub >>> functions to those spapr functions moved. >>> >>> >>> PATCH 1-5: Move spapr functions to spapr*.c >>> PATCH 6-20: Abstract out base container >>> PATCH 21-24: Introduce sparpr container and its specific interface >> >> PATCH 6-24 applied to vfio-next : >> >> https://github.com/legoater/qemu/commits/vfio-next >> >> (with a global s/fucntional/functional/) >> >> >> I also pushed the remaining patches on : >> >> https://github.com/legoater/qemu/commits/vfio-8.2 >> >> with a slight rework of the IOMMUFD configuration, now done per platform. >> The VFIO frontend and the 'iommufd' object are only available on x86_64, >> arm, s390x. > > FYI, I first tried this vfio-8.2 branch on s390x but wasn't actually able to use the iommufd backend (was getting errors like Property 'vfio-pci.iommufd' not found) so I think something isn't actually enabling IOMMUFD as expected with your change... yes. The previous method used to enable the IOMMUFD device with a ./configure script option was exposing the CONFIG_IOMMUFD define globally. The current method using the Kconfig files requires an extra : #include CONFIG_DEVICES in each file using CONFIG_IOMMUFD. I didn't see it because when compiled natively on x86_64 the CONFIG_IOMMUFD define is included for some (magic) reason. It is not the case on other arches, ppc64, aarch64, s390x. I did the update and repushed vfio-8.2. Should work now. > > Instead I tested on s390x using vfio-next + patches 25-41 of this series on top. > > Legacy backend regression testing worked fine for vfio-pci, vfio-ap and vfio-ccw. ok. Good. This means that vfio-next is in good shape. > Using iommufd backend for vfio-pci on s390 exposes an s390-only issue related to accounting of vfio DMA limit (code in hw/s390x/s390-pci-vfio.c assumes VFIODevice.group is never null, but that's no longer true when we use the iommufd backend with cdev). We don't even need to track this when using the iommufd backend -- With that issue bypassed, vfio-pci testing on s390x looks good so far. I'll send a separate fix for that. Thanks, C. > Using the iommufd backend for vfio-ccw and vfio-ap did not work, see response to patch 28. > > Thanks, > Matt >
>>> FYI, I first tried this vfio-8.2 branch on s390x but wasn't actually able to use the >>> iommufd backend (was getting errors like Property 'vfio-pci.iommufd' not found) >>> so I think something isn't actually enabling IOMMUFD as expected with your >>> change... >> >> It looks CONFIG_IOMMUFD is recognized by Kconfig sub-system but not received >> by compiler. I'm still digging how to pass CONFIG_IOMMUFD to compiler. > > Need below change to pass CONFIG_IOMMUFD to compiler. > > diff --git a/hw/vfio/ap.c b/hw/vfio/ap.c > index 0a810f8b88..2a3263b51f 100644 > --- a/hw/vfio/ap.c > +++ b/hw/vfio/ap.c > @@ -30,6 +30,7 @@ > #include "exec/address-spaces.h" > #include "qom/object.h" > #include "monitor/monitor.h" > +#include CONFIG_DEVICES > > #define TYPE_VFIO_AP_DEVICE "vfio-ap" > > diff --git a/hw/vfio/ccw.c b/hw/vfio/ccw.c > index a674bd8d6d..08101ad445 100644 > --- a/hw/vfio/ccw.c > +++ b/hw/vfio/ccw.c > @@ -31,6 +31,7 @@ > #include "qemu/main-loop.h" > #include "qemu/module.h" > #include "monitor/monitor.h" > +#include CONFIG_DEVICES > > struct VFIOCCWDevice { > S390CCWDevice cdev; > diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c > index d8f658ea47..3121b5f985 100644 > --- a/hw/vfio/pci.c > +++ b/hw/vfio/pci.c > @@ -44,6 +44,7 @@ > #include "migration/qemu-file.h" > #include "sysemu/iommufd.h" > #include "monitor/monitor.h" > +#include CONFIG_DEVICES > > #define TYPE_VFIO_PCI_NOHOTPLUG "vfio-pci-nohotplug" yep. I pushed forced vfio-8.2 with these changes. Thanks, C.
Hi, Thanks all for giving guides and comments on previous series, here is the v4 of pure iommufd support part. Based on Cédric's suggestion, this series includes an effort to remove spapr code from container.c, now all spapr functions are moved to spapr.c or spapr_pci_vfio.c, but there are still a few trival check on VFIO_SPAPR_TCE_*_IOMMU which I am not sure if deserved to introduce many callbacks and duplicate code just to remove them. Some functions are moved to spapr.c instead of spapr_pci_vfio.c to avoid compile issue because spapr_pci_vfio.c is arch specific, or else we need to introduce stub functions to those spapr functions moved. PATCH 1-5: Move spapr functions to spapr*.c PATCH 6-20: Abstract out base container PATCH 21-24: Introduce sparpr container and its specific interface PATCH 25: Add --enable/--disable-iommufd config support PATCH 26: Introduce iommufd object PATCH 27-33: add IOMMUFD container and cdev support PATCH 34-39: fd passing for IOMMUFD object and cdev PATCH 40: make VFIOContainerBase parameter const PATCH 41: Compile out for PPC We have done wide test with different combinations, e.g: - PCI device were tested - FD passing and hot reset with some trick. - device hotplug test with legacy and iommufd backends - with or without vIOMMU for legacy and iommufd backends - divices linked to different iommufds - VFIO migration with a E800 net card(no dirty sync support) passthrough - platform, ccw and ap were only compile-tested due to environment limit Given some iommufd kernel limitations, the iommufd backend is not yet fully on par with the legacy backend w.r.t. features like: - p2p mappings (you will see related error traces) - dirty page sync - and etc. qemu code: https://github.com/yiliu1765/qemu/commits/zhenzhong/iommufd_cdev_v4 Based on vfio-next, commit id: f686924775 -------------------------------------------------------------------------- Below are some background and graph about the design: With the introduction of iommufd, the Linux kernel provides a generic interface for userspace drivers to propagate their DMA mappings to kernel for assigned devices. This series does the porting of the VFIO devices onto the /dev/iommu uapi and let it coexist with the legacy implementation. At QEMU level, interactions with the /dev/iommu are abstracted by a new iommufd object (compiled in with the CONFIG_IOMMUFD option). Any QEMU device (e.g. vfio device) wishing to use /dev/iommu must be linked with an iommufd object. In this series, the vfio-pci device is granted with such capability (other VFIO devices are not yet ready): It gets a new optional parameter named iommufd which allows to pass an iommufd object: -object iommufd,id=iommufd0 -device vfio-pci,host=0000:02:00.0,iommufd=iommufd0 Note the /dev/iommu and vfio cdev can be externally opened by a management layer. In such a case the fd is passed: -object iommufd,id=iommufd0,fd=22 -device vfio-pci,iommufd=iommufd0,fd=23 If the fd parameter is not passed, the fd is opened by QEMU. See https://www.mail-archive.com/qemu-devel@nongnu.org/msg937155.html for detailed discuss on this requirement. If no iommufd option is passed to the vfio-pci device, iommufd is not used and the end-user gets the behavior based on the legacy vfio iommu interfaces: -device vfio-pci,host=0000:02:00.0 While the legacy kernel interface is group-centric, the new iommufd interface is device-centric, relying on device fd and iommufd. To support both interfaces in the QEMU VFIO device we reworked the vfio container abstraction so that the generic VFIO code can use either backend. The VFIOContainer object becomes a base object derived into a) the legacy VFIO container and b) the new iommufd based container. The base object implements generic code such as code related to memory_listener and address space management whereas the derived objects implement callbacks specific to either BE, legacy and iommufd. Indeed each backend has its own way to setup secure context and dma management interface. The below diagram shows how it looks like with both BEs. VFIO AddressSpace/Memory +-------+ +----------+ +-----+ +-----+ | pci | | platform | | ap | | ccw | +---+---+ +----+-----+ +--+--+ +--+--+ +----------------------+ | | | | | AddressSpace | | | | | +------------+---------+ +---V-----------V-----------V--------V----+ / | VFIOAddressSpace | <------------+ | | | MemoryListener | VFIOContainer list | +-------+----------------------------+----+ | | | | +-------V------+ +--------V----------+ | iommufd | | vfio legacy | | container | | container | +-------+------+ +--------+----------+ | | | /dev/iommu | /dev/vfio/vfio | /dev/vfio/devices/vfioX | /dev/vfio/$group_id Userspace | | ============+============================+=========================== Kernel | device fd | +---------------+ | group/container fd | (BIND_IOMMUFD | | (SET_CONTAINER/SET_IOMMU) | ATTACH_IOAS) | | device fd | | | | +-------V------------V-----------------+ iommufd | | vfio | (map/unmap | +---------+--------------------+-------+ ioas_copy) | | | map/unmap | | | +------V------+ +-----V------+ +------V--------+ | iommfd core | | device | | vfio iommu | +-------------+ +------------+ +---------------+ [Secure Context setup] - iommufd BE: uses device fd and iommufd to setup secure context (bind_iommufd, attach_ioas) - vfio legacy BE: uses group fd and container fd to setup secure context (set_container, set_iommu) [Device access] - iommufd BE: device fd is opened through /dev/vfio/devices/vfioX - vfio legacy BE: device fd is retrieved from group fd ioctl [DMA Mapping flow] 1. VFIOAddressSpace receives MemoryRegion add/del via MemoryListener 2. VFIO populates DMA map/unmap via the container BEs *) iommufd BE: uses iommufd *) vfio legacy BE: uses container fd Changelog: v4: - add CONFIG_IOMMUFD check for IOMMUFDProperties (Markus) - add doc for default case without fd (Markus) - Fix build issue reported by Markus and Cédric - Simply use SPDX identifier in new file (Cédric) - make vfio_container_init/destroy helper a seperate patch (Cédric) - make vrdl_list movement a seperate patch (Cédric) - add const for some callback parameters (Cédric) - add g_assert in VFIOIOMMUOps callback (Cédric) - introduce pci_hot_reset callback (Cédric) - remove VFIOIOMMUSpaprOps (Cédric) - initialize g_autofree to NULL (Cédric) - adjust func name prefix and trace event in iommufd.c (Cédric) - add RB v3: - Rename base container as VFIOContainerBase and legacy container as container (Cédric) - Drop VFIO_IOMMU_BACKEND_OPS class and use struct instead (Cédric) - Cleanup container.c by introducing spapr backend and move spapr code out (Cédric) - Introduce vfio_iommu_spapr_ops (Cédric) - Add doc of iommufd in qom.json and have iommufd member sorted (Markus) - patch19 and patch21 are splitted to two parts to facilitate review v2: - patch "vfio: Add base container" in v1 is split into patch1-15 per Cédric - add fd passing to platform/ap/ccw vfio device - add (uintptr_t) cast in iommufd_backend_map_dma() per Cédric - rename char_dev.h to chardev_open.h for same naming scheme per Daniel - add full copyright per Daniel and Jason Note changelog below are from full IOMMUFD series: v1: - Alloc hwpt instead of using auto hwpt - elaborate iommufd code per Nicolin - consolidate two patches and drop as.c - typo error fix and function rename rfcv4: - rebase on top of v8.0.3 - Add one patch from Yi which is about vfio device add in kvm - Remove IOAS_COPY optimization and focus on functions in this patchset - Fix wrong name issue reported and fix suggested by Matthew - Fix compilation issue reported and fix sugggsted by Nicolin - Use query_dirty_bitmap callback to replace get_dirty_bitmap for better granularity - Add dev_iter_next() callback to avoid adding so many callback at container scope, add VFIODevice.hwpt to support that - Restore all functions back to common from container whenever possible, mainly migration and reset related functions - Add --enable/disable-iommufd config option, enabled by default in linux - Remove VFIODevice.hwpt_next as it's redundant with VFIODevice.next - Adapt new VFIO_DEVICE_PCI_HOT_RESET uAPI for IOMMUFD backed device - vfio_kvm_device_add/del_group call vfio_kvm_device_add/del_fd to remove redundant code - Add FD passing support for vfio device backed by IOMMUFD - Fix hot unplug resource leak issue in vfio_legacy_detach_device() - Fix FD leak in vfio_get_devicefd() rfcv3: - rebase on top of v7.2.0 - Fix the compilation with CONFIG_IOMMUFD unset by using true classes for VFIO backends - Fix use after free in error path, reported by Alister - Split common.c in several steps to ease the review rfcv2: - remove the first three patches of rfcv1 - add open cdev helper suggested by Jason - remove the QOMification of the VFIOContainer and simply use standard ops (David) - add "-object iommufd" suggested by Alex Thanks Zhenzhong Eric Auger (11): vfio/container: Switch to dma_map|unmap API vfio/common: Move giommu_list in base container vfio/container: Move space field to base container vfio/container: Switch to IOMMU BE set_dirty_page_tracking/query_dirty_bitmap API vfio/container: Convert functions to base container vfio/container: Move pgsizes and dma_max_mappings to base container vfio/container: Move listener to base container vfio/container: Move dirty_pgsizes and max_dirty_bitmap_size to base container vfio/container: Implement attach/detach_device backends/iommufd: Introduce the iommufd object vfio/pci: Allow the selection of a given iommu backend Yi Liu (2): util/char_dev: Add open_cdev() vfio/iommufd: Implement the iommufd backend Zhenzhong Duan (28): vfio/container: Move IBM EEH related functions into spapr_pci_vfio.c vfio/container: Move vfio_container_add/del_section_window into spapr.c vfio/container: Move spapr specific init/deinit into spapr.c vfio/spapr: Make vfio_spapr_create/remove_window static vfio/common: Move vfio_host_win_add/del into spapr.c vfio: Introduce base object for VFIOContainer and targeted interface vfio/container: Introduce a empty VFIOIOMMUOps vfio/common: Introduce vfio_container_init/destroy helper vfio/container: Move per container device list in base container vfio/container: Move vrdl_list to base container vfio/container: Move iova_ranges to base container vfio/spapr: Introduce spapr backend and target interface vfio/spapr: switch to spapr IOMMU BE add/del_section_window vfio/spapr: Move prereg_listener into spapr container vfio/spapr: Move hostwin_list into spapr container Add iommufd configure option vfio/iommufd: Relax assert check for iommufd backend vfio/iommufd: Add support for iova_ranges vfio/pci: Extract out a helper vfio_pci_get_pci_hot_reset_info vfio/pci: Introduce a vfio pci hot reset interface vfio/iommufd: Enable pci hot reset through iommufd cdev interface vfio/pci: Make vfio cdev pre-openable by passing a file handle vfio: Allow the selection of a given iommu backend for platform ap and ccw vfio/platform: Make vfio cdev pre-openable by passing a file handle vfio/ap: Make vfio cdev pre-openable by passing a file handle vfio/ccw: Make vfio cdev pre-openable by passing a file handle vfio: Make VFIOContainerBase poiner parameter const in VFIOIOMMUOps callbacks vfio: Compile out iommufd for PPC target MAINTAINERS | 13 + meson.build | 6 + qapi/qom.json | 22 + hw/vfio/pci.h | 6 + include/hw/vfio/vfio-common.h | 118 ++--- include/hw/vfio/vfio-container-base.h | 121 +++++ include/hw/vfio/vfio-platform.h | 1 + include/hw/vfio/vfio.h | 7 - include/qemu/chardev_open.h | 16 + include/sysemu/iommufd.h | 46 ++ backends/iommufd-stub.c | 59 +++ backends/iommufd.c | 257 ++++++++++ hw/ppc/spapr_pci_vfio.c | 100 +++- hw/vfio/ap.c | 38 +- hw/vfio/ccw.c | 40 +- hw/vfio/common.c | 330 ++++++------ hw/vfio/container-base.c | 101 ++++ hw/vfio/container.c | 443 ++++------------ hw/vfio/helpers.c | 34 +- hw/vfio/iommufd.c | 697 ++++++++++++++++++++++++++ hw/vfio/pci.c | 112 +++-- hw/vfio/platform.c | 45 +- hw/vfio/spapr.c | 338 ++++++++++++- util/chardev_open.c | 81 +++ backends/Kconfig | 4 + backends/meson.build | 5 + backends/trace-events | 12 + hw/vfio/meson.build | 4 + hw/vfio/trace-events | 18 +- meson_options.txt | 2 + qemu-options.hx | 13 + scripts/meson-buildoptions.sh | 3 + util/meson.build | 1 + 33 files changed, 2403 insertions(+), 690 deletions(-) create mode 100644 include/hw/vfio/vfio-container-base.h delete mode 100644 include/hw/vfio/vfio.h create mode 100644 include/qemu/chardev_open.h create mode 100644 include/sysemu/iommufd.h create mode 100644 backends/iommufd-stub.c create mode 100644 backends/iommufd.c create mode 100644 hw/vfio/container-base.c create mode 100644 hw/vfio/iommufd.c create mode 100644 util/chardev_open.c