Message ID | 20230918094507.409050-1-jing2.liu@intel.com |
---|---|
Headers | show |
Series | Support dynamic MSI-X allocation | expand |
On Mon, 18 Sep 2023 05:45:03 -0400 Jing Liu <jing2.liu@intel.com> wrote: > Changes since v1: > - v1: https://www.mail-archive.com/qemu-devel@nongnu.org/msg982842.html > - Revise Qemu to QEMU. (Cédric) > - Add g_free when failure of getting MSI-X irq info. (Cédric) > - Apply Cédric's Reviewed-by. (Cédric) > - Use g_autofree to automatically release. (Cédric) > - Remove the failure message in vfio_enable_msix_no_vec(). (Cédric) > > Changes since RFC v1: > - RFC v1: https://www.mail-archive.com/qemu-devel@nongnu.org/msg978637.html > - Revise the comments. (Alex) > - Report error of getting irq info and remove the trace of failure > case. (Alex, Cédric) > - Only store dynamic allocation flag as a bool type and test > accordingly. (Alex) > - Move dynamic allocation detection to vfio_msix_early_setup(). (Alex) > - Change the condition logic in vfio_msix_vector_do_use() that moving > the defer_kvm_irq_routing test out and create a common place to update > nr_vectors. (Alex) > - Consolidate the way of MSI-X enabling during device initialization and > interrupt restoring that uses fd = -1 trick. Create a function doing > that. (Alex) > > Before kernel v6.5, dynamic allocation of MSI-X interrupts was not > supported. QEMU therefore when allocating a new interrupt, should first > release all previously allocated interrupts (including disable of MSI-X) > and re-allocate all interrupts that includes the new one. > > The kernel series [1] adds the support of dynamic MSI-X allocation to > vfio-pci and uses the existing flag VFIO_IRQ_INFO_NORESIZE to guide user > space, that when dynamic MSI-X is supported the flag is cleared. > > This series makes the behavior for VFIO PCI devices when dynamic MSI-X > allocation is supported. When guest unmasks an interrupt, QEMU can > directly allocate an interrupt on host for this and has nothing to do > with the previously allocated ones. Therefore, host only allocates > interrupts for those unmasked (enabled) interrupts inside guest when > dynamic MSI-X allocation is supported by device. > > When guests enable MSI-X with all of the vectors masked, QEMU need match > the state to enable MSI-X with no vector enabled. During migration > restore, QEMU also need enable MSI-X first in dynamic allocation mode, > to avoid the guest unused vectors being allocated on host. To > consolidate them, we use vector 0 with an invalid fd to get MSI-X > enabled and create a common function for this. This is cleaner than > setting userspace triggering and immediately release. > > Any feedback is appreciated. > > Jing > > [1] https://lwn.net/Articles/931679/ > > Jing Liu (4): > vfio/pci: detect the support of dynamic MSI-X allocation > vfio/pci: enable vector on dynamic MSI-X allocation > vfio/pci: use an invalid fd to enable MSI-X > vfio/pci: enable MSI-X in interrupt restoring on dynamic allocation > > hw/vfio/pci.c | 121 +++++++++++++++++++++++++++++++++---------- > hw/vfio/pci.h | 1 + > hw/vfio/trace-events | 2 +- > 3 files changed, 96 insertions(+), 28 deletions(-) > Some minor comments on 2/ but otherwise: Reviewed-by: Alex Williamson <alex.williamson@redhat.com>
Hi Alex, > On Sat, 9/23/2023 4:57AM, Alex Williamson <alex.williamson@redhat.com> wrote: > > On Mon, 18 Sep 2023 05:45:03 -0400 > Jing Liu <jing2.liu@intel.com> wrote: > > > Changes since v1: > > - v1: > > https://www.mail-archive.com/qemu-devel@nongnu.org/msg982842.html > > - Revise Qemu to QEMU. (Cédric) > > - Add g_free when failure of getting MSI-X irq info. (Cédric) > > - Apply Cédric's Reviewed-by. (Cédric) > > - Use g_autofree to automatically release. (Cédric) > > - Remove the failure message in vfio_enable_msix_no_vec(). (Cédric) > > > > Changes since RFC v1: > > - RFC v1: > > https://www.mail-archive.com/qemu-devel@nongnu.org/msg978637.html > > - Revise the comments. (Alex) > > - Report error of getting irq info and remove the trace of failure > > case. (Alex, Cédric) > > - Only store dynamic allocation flag as a bool type and test > > accordingly. (Alex) > > - Move dynamic allocation detection to vfio_msix_early_setup(). (Alex) > > - Change the condition logic in vfio_msix_vector_do_use() that moving > > the defer_kvm_irq_routing test out and create a common place to update > > nr_vectors. (Alex) > > - Consolidate the way of MSI-X enabling during device initialization and > > interrupt restoring that uses fd = -1 trick. Create a function doing > > that. (Alex) > > > > Before kernel v6.5, dynamic allocation of MSI-X interrupts was not > > supported. QEMU therefore when allocating a new interrupt, should > > first release all previously allocated interrupts (including disable > > of MSI-X) and re-allocate all interrupts that includes the new one. > > > > The kernel series [1] adds the support of dynamic MSI-X allocation to > > vfio-pci and uses the existing flag VFIO_IRQ_INFO_NORESIZE to guide > > user space, that when dynamic MSI-X is supported the flag is cleared. > > > > This series makes the behavior for VFIO PCI devices when dynamic MSI-X > > allocation is supported. When guest unmasks an interrupt, QEMU can > > directly allocate an interrupt on host for this and has nothing to do > > with the previously allocated ones. Therefore, host only allocates > > interrupts for those unmasked (enabled) interrupts inside guest when > > dynamic MSI-X allocation is supported by device. > > > > When guests enable MSI-X with all of the vectors masked, QEMU need > > match the state to enable MSI-X with no vector enabled. During > > migration restore, QEMU also need enable MSI-X first in dynamic > > allocation mode, to avoid the guest unused vectors being allocated on > > host. To consolidate them, we use vector 0 with an invalid fd to get > > MSI-X enabled and create a common function for this. This is cleaner > > than setting userspace triggering and immediately release. > > > > Any feedback is appreciated. > > > > Jing > > > > [1] https://lwn.net/Articles/931679/ > > > > Jing Liu (4): > > vfio/pci: detect the support of dynamic MSI-X allocation > > vfio/pci: enable vector on dynamic MSI-X allocation > > vfio/pci: use an invalid fd to enable MSI-X > > vfio/pci: enable MSI-X in interrupt restoring on dynamic allocation > > > > hw/vfio/pci.c | 121 +++++++++++++++++++++++++++++++++---------- > > hw/vfio/pci.h | 1 + > > hw/vfio/trace-events | 2 +- > > 3 files changed, 96 insertions(+), 28 deletions(-) > > > > Some minor comments on 2/ but otherwise: > > Reviewed-by: Alex Williamson <alex.williamson@redhat.com> Thank you very much for the feedback. Will apply on v3 with fix for 2/4. Jing