From patchwork Tue Aug 22 07:29:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jing Liu X-Patchwork-Id: 1823957 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=R0teADzb; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4RVLbh2z2Rz1yNm for ; Tue, 22 Aug 2023 17:30:24 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qYLpy-0005bp-JG; Tue, 22 Aug 2023 03:30:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qYLpv-0005aT-Kr for qemu-devel@nongnu.org; Tue, 22 Aug 2023 03:29:59 -0400 Received: from mgamail.intel.com ([134.134.136.31]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qYLpt-0006Xv-3e for qemu-devel@nongnu.org; Tue, 22 Aug 2023 03:29:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1692689397; x=1724225397; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=9Rrhag1iF1R36pfHA2u5a0Dq0T7osuYFAC6JG8f89y8=; b=R0teADzbgPU1IJHc6aRrkKugBCZRxp+xWMQ2yigCt8xEF1pjgjcxUN5c AoBgT9OheDU4A0s8S63hQSD6H2z1jpA0I0+bzr67++QUe4Z715Noxfsyu CVLEcIfFHR2delBz3R1eINTeCy8KXfvaF1XPO4sgqz/zWuCQFtSHlRjDM yKFZ0uw7fdbkXzCjky1nCIN6gqXoU1c15TnyM29qkKrQQ1vCjSzVk8Zqp UgdGh/IfNEPQ81OlkZS0RZbYNYoCKBvqH9VgCNvnZp69L9EchaV12Vsvq jOweB7jD4PkpGcBPzSJkmUu3KzdPOTG8RH/2YQVgqPFJR/rFX8YwCWaB/ g==; X-IronPort-AV: E=McAfee;i="6600,9927,10809"; a="437721786" X-IronPort-AV: E=Sophos;i="6.01,192,1684825200"; d="scan'208";a="437721786" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Aug 2023 00:29:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10809"; a="850502019" X-IronPort-AV: E=Sophos;i="6.01,192,1684825200"; d="scan'208";a="850502019" Received: from vmmteam.bj.intel.com ([10.240.192.110]) by fmsmga002.fm.intel.com with ESMTP; 22 Aug 2023 00:29:38 -0700 From: Jing Liu To: qemu-devel@nongnu.org Cc: alex.williamson@redhat.com, clg@redhat.com, pbonzini@redhat.com, kevin.tian@intel.com, reinette.chatre@intel.com, jing2.liu@intel.com, jing2.liu@linux.intel.com Subject: [PATCH v1 1/4] vfio/pci: detect the support of dynamic MSI-X allocation Date: Tue, 22 Aug 2023 03:29:24 -0400 Message-Id: <20230822072927.224803-2-jing2.liu@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20230822072927.224803-1-jing2.liu@intel.com> References: <20230822072927.224803-1-jing2.liu@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=134.134.136.31; envelope-from=jing2.liu@intel.com; helo=mgamail.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Kernel provides the guidance of dynamic MSI-X allocation support of passthrough device, by clearing the VFIO_IRQ_INFO_NORESIZE flag to guide user space. Fetch the flags from host to determine if dynamic MSI-X allocation is supported. Originally-by: Reinette Chatre Signed-off-by: Jing Liu Reviewed-by: Cédric Le Goater --- Changes since RFC v1: - Filter the dynamic MSI-X allocation flag and store as a bool type. (Alex) - Move the detection to vfio_msix_early_setup(). (Alex) - Report error of getting irq info and remove the trace of failure case. (Alex, Cédric) --- hw/vfio/pci.c | 15 +++++++++++++-- hw/vfio/pci.h | 1 + hw/vfio/trace-events | 2 +- 3 files changed, 15 insertions(+), 3 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index a205c6b1130f..8a3b34f3c196 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -1493,7 +1493,9 @@ static void vfio_msix_early_setup(VFIOPCIDevice *vdev, Error **errp) uint8_t pos; uint16_t ctrl; uint32_t table, pba; - int fd = vdev->vbasedev.fd; + int ret, fd = vdev->vbasedev.fd; + struct vfio_irq_info irq_info = { .argsz = sizeof(irq_info), + .index = VFIO_PCI_MSIX_IRQ_INDEX }; VFIOMSIXInfo *msix; pos = pci_find_capability(&vdev->pdev, PCI_CAP_ID_MSIX); @@ -1530,6 +1532,14 @@ static void vfio_msix_early_setup(VFIOPCIDevice *vdev, Error **errp) msix->pba_offset = pba & ~PCI_MSIX_FLAGS_BIRMASK; msix->entries = (ctrl & PCI_MSIX_FLAGS_QSIZE) + 1; + ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info); + if (ret < 0) { + error_setg_errno(errp, -ret, "failed to get MSI-X irq info"); + return; + } + + msix->noresize = !!(irq_info.flags & VFIO_IRQ_INFO_NORESIZE); + /* * Test the size of the pba_offset variable and catch if it extends outside * of the specified BAR. If it is the case, we need to apply a hardware @@ -1562,7 +1572,8 @@ static void vfio_msix_early_setup(VFIOPCIDevice *vdev, Error **errp) } trace_vfio_msix_early_setup(vdev->vbasedev.name, pos, msix->table_bar, - msix->table_offset, msix->entries); + msix->table_offset, msix->entries, + msix->noresize); vdev->msix = msix; vfio_pci_fixup_msix_region(vdev); diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index a2771b9ff3cc..0717574d79e9 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -113,6 +113,7 @@ typedef struct VFIOMSIXInfo { uint32_t table_offset; uint32_t pba_offset; unsigned long *pending; + bool noresize; } VFIOMSIXInfo; #define TYPE_VFIO_PCI "vfio-pci" diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index ee7509e68e4f..6de5d9ba8e46 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -27,7 +27,7 @@ vfio_vga_read(uint64_t addr, int size, uint64_t data) " (0x%"PRIx64", %d) = 0x%" vfio_pci_read_config(const char *name, int addr, int len, int val) " (%s, @0x%x, len=0x%x) 0x%x" vfio_pci_write_config(const char *name, int addr, int val, int len) " (%s, @0x%x, 0x%x, len=0x%x)" vfio_msi_setup(const char *name, int pos) "%s PCI MSI CAP @0x%x" -vfio_msix_early_setup(const char *name, int pos, int table_bar, int offset, int entries) "%s PCI MSI-X CAP @0x%x, BAR %d, offset 0x%x, entries %d" +vfio_msix_early_setup(const char *name, int pos, int table_bar, int offset, int entries, bool noresize) "%s PCI MSI-X CAP @0x%x, BAR %d, offset 0x%x, entries %d, noresize %d" vfio_check_pcie_flr(const char *name) "%s Supports FLR via PCIe cap" vfio_check_pm_reset(const char *name) "%s Supports PM reset" vfio_check_af_flr(const char *name) "%s Supports FLR via AF cap" From patchwork Tue Aug 22 07:29:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jing Liu X-Patchwork-Id: 1823958 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=cCA77phk; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4RVLbm6Xvcz1yNm for ; Tue, 22 Aug 2023 17:30:28 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qYLpv-0005aO-4h; Tue, 22 Aug 2023 03:29:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qYLpt-0005ZB-4H for qemu-devel@nongnu.org; Tue, 22 Aug 2023 03:29:57 -0400 Received: from mgamail.intel.com ([134.134.136.31]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qYLpq-0006XO-Bj for qemu-devel@nongnu.org; Tue, 22 Aug 2023 03:29:56 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1692689394; x=1724225394; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=PVRuLt02WMM11u89aqiq5phWxZsW3LYwM6qdkVNQNnU=; b=cCA77phkewA3H/gB+xfbsQLZdPGdNNIRQhgDlNfMO5qj8aWuW/6wg+bf Q6Df/yGCQT8DQRD+Hvk1yryxiDachx3qvpNO6tS+1+S/0yi8QTifIBhY1 q2kMDGIwCfbAYarTuLNDsGaxoUbeiVK46Mo+xyhicVDR2sEK4w+o4wuNY EcRJW87PC04P3uyraGWrd8DcX0xnkNyxt9rhd6AElZE/K/TBb9aevfdQF dK/6nPMUNhXdzGF+8foazE3ZtEsLAC+u8/3+e5SY7sD0q/BiBzQ0yljCA cvh9xVgA1/emBlNDtlPNwmfLdkNt2FZXWzLW9GXqv9K169lGHwNjoFGXX Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10809"; a="437721789" X-IronPort-AV: E=Sophos;i="6.01,192,1684825200"; d="scan'208";a="437721789" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Aug 2023 00:29:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10809"; a="850502042" X-IronPort-AV: E=Sophos;i="6.01,192,1684825200"; d="scan'208";a="850502042" Received: from vmmteam.bj.intel.com ([10.240.192.110]) by fmsmga002.fm.intel.com with ESMTP; 22 Aug 2023 00:29:40 -0700 From: Jing Liu To: qemu-devel@nongnu.org Cc: alex.williamson@redhat.com, clg@redhat.com, pbonzini@redhat.com, kevin.tian@intel.com, reinette.chatre@intel.com, jing2.liu@intel.com, jing2.liu@linux.intel.com Subject: [PATCH v1 2/4] vfio/pci: enable vector on dynamic MSI-X allocation Date: Tue, 22 Aug 2023 03:29:25 -0400 Message-Id: <20230822072927.224803-3-jing2.liu@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20230822072927.224803-1-jing2.liu@intel.com> References: <20230822072927.224803-1-jing2.liu@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=134.134.136.31; envelope-from=jing2.liu@intel.com; helo=mgamail.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org The vector_use callback is used to enable vector that is unmasked in guest. The kernel used to only support static MSI-X allocation. When allocating a new interrupt using "static MSI-X allocation" kernels, Qemu first disables all previously allocated vectors and then re-allocates all including the new one. The nr_vectors of VFIOPCIDevice indicates that all vectors from 0 to nr_vectors are allocated (and may be enabled), which is used to to loop all the possibly used vectors When, e.g., disabling MSI-X interrupts. Extend the vector_use function to support dynamic MSI-X allocation when host supports the capability. Qemu therefore can individually allocate and enable a new interrupt without affecting others or causing interrupts lost during runtime. Utilize nr_vectors to calculate the upper bound of enabled vectors in dynamic MSI-X allocation mode since looping all msix_entries_nr is not efficient and unnecessary. Signed-off-by: Jing Liu Tested-by: Reinette Chatre --- Changes since RFC v1: - Test vdev->msix->noresize to identify the allocation mode. (Alex) - Move defer_kvm_irq_routing test out and update nr_vectors in a common place before vfio_enable_vectors(). (Alex) - Revise the comments. (Alex) --- hw/vfio/pci.c | 44 +++++++++++++++++++++++++++----------------- 1 file changed, 27 insertions(+), 17 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 8a3b34f3c196..31f36d68bb19 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -470,6 +470,7 @@ static int vfio_msix_vector_do_use(PCIDevice *pdev, unsigned int nr, VFIOPCIDevice *vdev = VFIO_PCI(pdev); VFIOMSIVector *vector; int ret; + int old_nr_vecs = vdev->nr_vectors; trace_vfio_msix_vector_do_use(vdev->vbasedev.name, nr); @@ -512,33 +513,42 @@ static int vfio_msix_vector_do_use(PCIDevice *pdev, unsigned int nr, } /* - * We don't want to have the host allocate all possible MSI vectors - * for a device if they're not in use, so we shutdown and incrementally - * increase them as needed. + * When dynamic allocation is not supported, we don't want to have the + * host allocate all possible MSI vectors for a device if they're not + * in use, so we shutdown and incrementally increase them as needed. + * nr_vectors represents the total number of vectors allocated. + * + * When dynamic allocation is supported, let the host only allocate + * and enable a vector when it is in use in guest. nr_vectors represents + * the upper bound of vectors being enabled (but not all of the ranges + * is allocated or enabled). */ if (vdev->nr_vectors < nr + 1) { vdev->nr_vectors = nr + 1; - if (!vdev->defer_kvm_irq_routing) { + } + + if (!vdev->defer_kvm_irq_routing) { + if (vdev->msix->noresize && (old_nr_vecs < nr + 1)) { vfio_disable_irqindex(&vdev->vbasedev, VFIO_PCI_MSIX_IRQ_INDEX); ret = vfio_enable_vectors(vdev, true); if (ret) { error_report("vfio: failed to enable vectors, %d", ret); } - } - } else { - Error *err = NULL; - int32_t fd; - - if (vector->virq >= 0) { - fd = event_notifier_get_fd(&vector->kvm_interrupt); } else { - fd = event_notifier_get_fd(&vector->interrupt); - } + Error *err = NULL; + int32_t fd; - if (vfio_set_irq_signaling(&vdev->vbasedev, - VFIO_PCI_MSIX_IRQ_INDEX, nr, - VFIO_IRQ_SET_ACTION_TRIGGER, fd, &err)) { - error_reportf_err(err, VFIO_MSG_PREFIX, vdev->vbasedev.name); + if (vector->virq >= 0) { + fd = event_notifier_get_fd(&vector->kvm_interrupt); + } else { + fd = event_notifier_get_fd(&vector->interrupt); + } + + if (vfio_set_irq_signaling(&vdev->vbasedev, + VFIO_PCI_MSIX_IRQ_INDEX, nr, + VFIO_IRQ_SET_ACTION_TRIGGER, fd, &err)) { + error_reportf_err(err, VFIO_MSG_PREFIX, vdev->vbasedev.name); + } } } From patchwork Tue Aug 22 07:29:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jing Liu X-Patchwork-Id: 1823956 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=eHKRmZoa; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4RVLbf0rYhz1yNm for ; Tue, 22 Aug 2023 17:30:22 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qYLpy-0005bs-JM; Tue, 22 Aug 2023 03:30:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qYLpw-0005av-AL for qemu-devel@nongnu.org; Tue, 22 Aug 2023 03:30:00 -0400 Received: from mgamail.intel.com ([134.134.136.31]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qYLpt-0006XO-EH for qemu-devel@nongnu.org; Tue, 22 Aug 2023 03:29:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1692689397; x=1724225397; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=8iAnTg/4sqWStxyxOqja9WHMf+YUwpcUVO/YKALjC4I=; b=eHKRmZoaePB6PLSim2VuZmzmVVgQfz02Gw0PdQaxk1uSKdx9nY9sFr5E gw0U/maqp2D+5aRFdk8mSKqFs4eG0O1Iv4YYxD/Ct+AgWcJuG96DsND1l Dw6pj0IMZXnn/qgY5f/XNbVmXhlsiHPx5HDtLGXL8z4EnOYtj+6/MKqFr MA5xMeC0lqFBjVQ3+ZO74iX8uly8UJ53b5Pc84iUGKq++NXJKcDukh5rN y5zr+3CVYvccQoTgdJvMeajij0aK0igREBqiT5zBAscPdxFRY6Iv+QIjU zC0b0rEKjU2iTgjj1O38xyqKq0UOKytQsJ8hHreBN/EW/grBlwOko941N w==; X-IronPort-AV: E=McAfee;i="6600,9927,10809"; a="437721817" X-IronPort-AV: E=Sophos;i="6.01,192,1684825200"; d="scan'208";a="437721817" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Aug 2023 00:29:45 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10809"; a="850502098" X-IronPort-AV: E=Sophos;i="6.01,192,1684825200"; d="scan'208";a="850502098" Received: from vmmteam.bj.intel.com ([10.240.192.110]) by fmsmga002.fm.intel.com with ESMTP; 22 Aug 2023 00:29:43 -0700 From: Jing Liu To: qemu-devel@nongnu.org Cc: alex.williamson@redhat.com, clg@redhat.com, pbonzini@redhat.com, kevin.tian@intel.com, reinette.chatre@intel.com, jing2.liu@intel.com, jing2.liu@linux.intel.com Subject: [PATCH v1 3/4] vfio/pci: use an invalid fd to enable MSI-X Date: Tue, 22 Aug 2023 03:29:26 -0400 Message-Id: <20230822072927.224803-4-jing2.liu@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20230822072927.224803-1-jing2.liu@intel.com> References: <20230822072927.224803-1-jing2.liu@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=134.134.136.31; envelope-from=jing2.liu@intel.com; helo=mgamail.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Guests typically enable MSI-X with all of the vectors masked in the MSI-X vector table. To match the guest state of device, Qemu enables MSI-X by enabling vector 0 with userspace triggering and immediately release. However the release function actually does not release it due to already using userspace mode. It is no need to enable triggering on host and rely on the mask bit to avoid spurious interrupts. Use an invalid fd (i.e. fd = -1) is enough to get MSI-X enabled. After dynamic MSI-X allocation is supported, the interrupt restoring also need use such way to enable MSI-X, therefore, create a function for that. Suggested-by: Alex Williamson Signed-off-by: Jing Liu --- Changes since RFC v1: - A new patch. Use an invalid fd to get MSI-X enabled instead of using userspace triggering. (Alex) --- hw/vfio/pci.c | 50 ++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 42 insertions(+), 8 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 31f36d68bb19..e24c21241a0c 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -369,6 +369,39 @@ static void vfio_msi_interrupt(void *opaque) notify(&vdev->pdev, nr); } +/* + * Get MSI-X enabled, but no vector enabled, by setting vector 0 with an invalid + * fd to kernel. + */ +static int vfio_enable_msix_no_vec(VFIOPCIDevice *vdev) +{ + struct vfio_irq_set *irq_set; + int ret = 0, argsz; + int32_t *fd; + + argsz = sizeof(*irq_set) + sizeof(*fd); + + irq_set = g_malloc0(argsz); + irq_set->argsz = argsz; + irq_set->flags = VFIO_IRQ_SET_DATA_EVENTFD | + VFIO_IRQ_SET_ACTION_TRIGGER; + irq_set->index = VFIO_PCI_MSIX_IRQ_INDEX; + irq_set->start = 0; + irq_set->count = 1; + fd = (int32_t *)&irq_set->data; + *fd = -1; + + ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_SET_IRQS, irq_set); + if (ret) { + error_report("vfio: failed to enable MSI-X with vector 0 trick, %d", + ret); + } + + g_free(irq_set); + + return ret; +} + static int vfio_enable_vectors(VFIOPCIDevice *vdev, bool msix) { struct vfio_irq_set *irq_set; @@ -618,6 +651,8 @@ static void vfio_commit_kvm_msi_virq_batch(VFIOPCIDevice *vdev) static void vfio_msix_enable(VFIOPCIDevice *vdev) { + int ret; + vfio_disable_interrupts(vdev); vdev->msi_vectors = g_new0(VFIOMSIVector, vdev->msix->entries); @@ -640,8 +675,6 @@ static void vfio_msix_enable(VFIOPCIDevice *vdev) vfio_commit_kvm_msi_virq_batch(vdev); if (vdev->nr_vectors) { - int ret; - ret = vfio_enable_vectors(vdev, true); if (ret) { error_report("vfio: failed to enable vectors, %d", ret); @@ -655,13 +688,14 @@ static void vfio_msix_enable(VFIOPCIDevice *vdev) * MSI-X capability, but leaves the vector table masked. We therefore * can't rely on a vector_use callback (from request_irq() in the guest) * to switch the physical device into MSI-X mode because that may come a - * long time after pci_enable_msix(). This code enables vector 0 with - * triggering to userspace, then immediately release the vector, leaving - * the physical device with no vectors enabled, but MSI-X enabled, just - * like the guest view. + * long time after pci_enable_msix(). This code sets vector 0 with an + * invalid fd to make the physical device MSI-X enabled, but with no + * vectors enabled, just like the guest view. */ - vfio_msix_vector_do_use(&vdev->pdev, 0, NULL, NULL); - vfio_msix_vector_release(&vdev->pdev, 0); + ret = vfio_enable_msix_no_vec(vdev); + if (ret) { + error_report("vfio: failed to enable MSI-X, %d", ret); + } } trace_vfio_msix_enable(vdev->vbasedev.name); From patchwork Tue Aug 22 07:29:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jing Liu X-Patchwork-Id: 1823959 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=kvMxvlXU; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4RVLcT3vp6z1yNm for ; Tue, 22 Aug 2023 17:31:05 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qYLqD-0005jq-2j; Tue, 22 Aug 2023 03:30:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qYLpz-0005cd-Ox for qemu-devel@nongnu.org; Tue, 22 Aug 2023 03:30:05 -0400 Received: from mgamail.intel.com ([134.134.136.31]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qYLpw-0006Wr-DL for qemu-devel@nongnu.org; Tue, 22 Aug 2023 03:30:03 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1692689400; x=1724225400; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=5GkLkoCqRrhPFbnkLhCZrkLgHIZAwtVy/bSmP5P7hpw=; b=kvMxvlXU2LwqTZmF73izpr7jym3VlZ4EfxxkDEZ4RnAry4xZa9n+KDWl WpZlsqlapsKganjMN+eNGn8ZAaROmu/McUfk98dcXbjTreROhOOfl7OFr pjTbG9+N9IHdvoPTgYl3eiy59sin6lUtLMp8dyoxZEMk1CcMwzR688d92 8fPrD/rBg6tKGN//0UVvPML0bkk3PTE6goTCG4/L6setY8W34AsPDTrVw 5EKVDczvMQ/ffM2c3+nx3l9o+dMuHcg03qzOwMvuBiw7Q2LelbAVlrG8N uHX7FMyLHAZXZd/CxUxWzRuyAMicfuGqO4PuT5kkzLUCfiLtNFEC8z6fU w==; X-IronPort-AV: E=McAfee;i="6600,9927,10809"; a="437721823" X-IronPort-AV: E=Sophos;i="6.01,192,1684825200"; d="scan'208";a="437721823" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 22 Aug 2023 00:29:48 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10809"; a="850502124" X-IronPort-AV: E=Sophos;i="6.01,192,1684825200"; d="scan'208";a="850502124" Received: from vmmteam.bj.intel.com ([10.240.192.110]) by fmsmga002.fm.intel.com with ESMTP; 22 Aug 2023 00:29:45 -0700 From: Jing Liu To: qemu-devel@nongnu.org Cc: alex.williamson@redhat.com, clg@redhat.com, pbonzini@redhat.com, kevin.tian@intel.com, reinette.chatre@intel.com, jing2.liu@intel.com, jing2.liu@linux.intel.com Subject: [PATCH v1 4/4] vfio/pci: enable MSI-X in interrupt restoring on dynamic allocation Date: Tue, 22 Aug 2023 03:29:27 -0400 Message-Id: <20230822072927.224803-5-jing2.liu@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20230822072927.224803-1-jing2.liu@intel.com> References: <20230822072927.224803-1-jing2.liu@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=134.134.136.31; envelope-from=jing2.liu@intel.com; helo=mgamail.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org During migration restoring, vfio_enable_vectors() is called to restore enabling MSI-X interrupts for assigned devices. It sets the range from 0 to nr_vectors to kernel to enable MSI-X and the vectors unmasked in guest. During the MSI-X enabling, all the vectors within the range are allocated according to the VFIO_DEVICE_SET_IRQS ioctl. When dynamic MSI-X allocation is supported, we only want the guest unmasked vectors being allocated and enabled. Use vector 0 with an invalid fd to get MSI-X enabled, after that, all the vectors can be allocated in need. Signed-off-by: Jing Liu --- Changes since RFC v1: - Revise the comments. (Alex) - Call the new helper function in previous patch to enable MSI-X. (Alex) --- hw/vfio/pci.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index e24c21241a0c..3c37d036e727 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -408,6 +408,23 @@ static int vfio_enable_vectors(VFIOPCIDevice *vdev, bool msix) int ret = 0, i, argsz; int32_t *fds; + /* + * If dynamic MSI-X allocation is supported, the vectors to be allocated + * and enabled can be scattered. Before kernel enabling MSI-X, setting + * nr_vectors causes all these vectors to be allocated on host. + * + * To keep allocation as needed, use vector 0 with an invalid fd to get + * MSI-X enabled first, then set vectors with a potentially sparse set of + * eventfds to enable interrupts only when enabled in guest. + */ + if (msix && !vdev->msix->noresize) { + ret = vfio_enable_msix_no_vec(vdev); + + if (ret) { + return ret; + } + } + argsz = sizeof(*irq_set) + (vdev->nr_vectors * sizeof(*fds)); irq_set = g_malloc0(argsz);