From patchwork Mon Jul 3 07:15:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Zhenzhong Duan X-Patchwork-Id: 1802532 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=H9P8xIdG; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4QvcyG2S4Lz20b1 for ; Mon, 3 Jul 2023 17:29:58 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qGDys-0006CQ-VF; Mon, 03 Jul 2023 03:28:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qGDyq-0006Bx-L1 for qemu-devel@nongnu.org; Mon, 03 Jul 2023 03:28:16 -0400 Received: from mga12.intel.com ([192.55.52.136]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qGDyo-0003we-Qt for qemu-devel@nongnu.org; Mon, 03 Jul 2023 03:28:16 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1688369294; x=1719905294; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=3Tzb73hY6oDdkY1FyoMeSKY04AeNZkybsLZ2x4i3oAI=; b=H9P8xIdGCjhbWhXA5sXVWdf98B8EbOVY9zjij+gdnYzF8vmt/jbXzSah CYa/huAqrEi0dCahdIbNwiPqC+J3e9V09WqkRw11xiYqFIIgoLR+5E5DW tKy4wvil03NyEP07lJsyiI2SWbGFAd5fBnOZwqyvUwPUkLUiimImH+cja PyP05khwzzno+oU3OCTzrTVmyK0Ni1FRxvxab4oQFMtC+CXDWyic32UVg Dx7k8r0sEPIJ6F/rCUqWZRGHjMxD11RlNVW14RvOz7uM0OFr6tZO2IKX8 TCYWUvnbG77IEJmarbJ7t8whLU+zHqiDXDoW2+Iy7t6tEdgs2PwcNhyvk w==; X-IronPort-AV: E=McAfee;i="6600,9927,10759"; a="342405633" X-IronPort-AV: E=Sophos;i="6.01,177,1684825200"; d="scan'208";a="342405633" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Jul 2023 00:28:12 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10759"; a="748058763" X-IronPort-AV: E=Sophos;i="6.01,177,1684825200"; d="scan'208";a="748058763" Received: from duan-server-s2600bt.bj.intel.com ([10.240.192.147]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Jul 2023 00:28:10 -0700 From: Zhenzhong Duan To: qemu-devel@nongnu.org Cc: alex.williamson@redhat.com, clg@redhat.com, joao.m.martins@oracle.com, avihaih@nvidia.com, chao.p.peng@intel.com Subject: [PATCH v6 3/7] vfio/pci: Disable INTx in vfio_realize error path Date: Mon, 3 Jul 2023 15:15:06 +0800 Message-Id: <20230703071510.160712-2-zhenzhong.duan@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230703071510.160712-1-zhenzhong.duan@intel.com> References: <20230703071510.160712-1-zhenzhong.duan@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.136; envelope-from=zhenzhong.duan@intel.com; helo=mga12.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org When vfio realize fails, INTx isn't disabled if it has been enabled. This may confuse host side with unhandled interrupt report. Fixes: c5478fea27ac ("vfio/pci: Respond to KVM irqchip change notifier") Signed-off-by: Zhenzhong Duan Reviewed-by: Joao Martins Reviewed-by: Cédric Le Goater --- hw/vfio/pci.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index ab6645ba60af..31c4ab250fbe 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3220,6 +3220,9 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) return; out_deregister: + if (vdev->interrupt == VFIO_INT_INTx) { + vfio_intx_disable(vdev); + } pci_device_set_intx_routing_notifier(&vdev->pdev, NULL); if (vdev->irqchip_change_notifier.notify) { kvm_irqchip_remove_change_notifier(&vdev->irqchip_change_notifier); From patchwork Mon Jul 3 07:15:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Zhenzhong Duan X-Patchwork-Id: 1802527 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=iL9aIGY3; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4QvcxS1RyYz20bn for ; Mon, 3 Jul 2023 17:29:15 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qGDyv-0006DA-I5; Mon, 03 Jul 2023 03:28:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qGDys-0006CI-5h for qemu-devel@nongnu.org; Mon, 03 Jul 2023 03:28:18 -0400 Received: from mga12.intel.com ([192.55.52.136]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qGDyq-0003wp-7U for qemu-devel@nongnu.org; Mon, 03 Jul 2023 03:28:17 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1688369296; x=1719905296; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Addli9nRybH8vvZHhKuJjk8c+EBwFtcNEwWt3YWXIsg=; b=iL9aIGY3KEM09EwuBzJpsbAlPQ4ZcrFlK9NLyIbvH32GUSHhfNKlmyGt OSSPUXGbFdHSKwwwA4y8jSWKS9PDCN5re+jOn+Re2F54ygMJ+yMoOozO+ tBsLNapUU8M4pKndJdNzUSXk2lLKh/0D8NNr6wg8V2v25x4JMePmldful A/fUN3E36SUzrcAlfUVsXOQTQ7XINaX97Qvq2z/FrqUO2x0mdFqjYkcYd LxCchcM7ENXrnD5MI+5fou7lTbbFJd/oBxQ2QVv2WyF1dbRBse6PaBtoZ tA7Zq6qUmM9yiKPZ72HoNln6lhjKUk7kPbHFRILpRMCRx0ljhBVl0r+lk g==; X-IronPort-AV: E=McAfee;i="6600,9927,10759"; a="342405640" X-IronPort-AV: E=Sophos;i="6.01,177,1684825200"; d="scan'208";a="342405640" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Jul 2023 00:28:14 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10759"; a="748058770" X-IronPort-AV: E=Sophos;i="6.01,177,1684825200"; d="scan'208";a="748058770" Received: from duan-server-s2600bt.bj.intel.com ([10.240.192.147]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Jul 2023 00:28:12 -0700 From: Zhenzhong Duan To: qemu-devel@nongnu.org Cc: alex.williamson@redhat.com, clg@redhat.com, joao.m.martins@oracle.com, avihaih@nvidia.com, chao.p.peng@intel.com Subject: [PATCH v6 4/7] vfio/migration: Change vIOMMU blocker from global to per device Date: Mon, 3 Jul 2023 15:15:07 +0800 Message-Id: <20230703071510.160712-3-zhenzhong.duan@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230703071510.160712-1-zhenzhong.duan@intel.com> References: <20230703071510.160712-1-zhenzhong.duan@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.136; envelope-from=zhenzhong.duan@intel.com; helo=mga12.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Contrary to multiple device blocker which needs to consider already-attached devices to unblock/block dynamically, the vIOMMU migration blocker is a device specific config. Meaning it only needs to know whether the device is bypassing or not the vIOMMU (via machine property, or per pxb-pcie::bypass_iommu), and does not need the state of currently present devices. For this reason, the vIOMMU global migration blocker can be consolidated into the per-device migration blocker, allowing us to remove some unnecessary code. This change also makes vfio_mig_active() more accurate as it doesn't check for global blocker. Signed-off-by: Zhenzhong Duan Reviewed-by: Joao Martins Reviewed-by: Cédric Le Goater --- hw/vfio/common.c | 51 ++--------------------------------- hw/vfio/migration.c | 7 ++--- hw/vfio/pci.c | 1 - include/hw/vfio/vfio-common.h | 3 +-- 4 files changed, 7 insertions(+), 55 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 77e2ee0e5c6e..9aac21abb76e 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -362,7 +362,6 @@ bool vfio_mig_active(void) } static Error *multiple_devices_migration_blocker; -static Error *giommu_migration_blocker; static unsigned int vfio_migratable_device_num(void) { @@ -420,55 +419,9 @@ void vfio_unblock_multiple_devices_migration(void) multiple_devices_migration_blocker = NULL; } -static bool vfio_viommu_preset(void) +bool vfio_viommu_preset(VFIODevice *vbasedev) { - VFIOAddressSpace *space; - - QLIST_FOREACH(space, &vfio_address_spaces, list) { - if (space->as != &address_space_memory) { - return true; - } - } - - return false; -} - -int vfio_block_giommu_migration(VFIODevice *vbasedev, Error **errp) -{ - int ret; - - if (giommu_migration_blocker || - !vfio_viommu_preset()) { - return 0; - } - - if (vbasedev->enable_migration == ON_OFF_AUTO_ON) { - error_setg(errp, - "Migration is currently not supported with vIOMMU enabled"); - return -EINVAL; - } - - error_setg(&giommu_migration_blocker, - "Migration is currently not supported with vIOMMU enabled"); - ret = migrate_add_blocker(giommu_migration_blocker, errp); - if (ret < 0) { - error_free(giommu_migration_blocker); - giommu_migration_blocker = NULL; - } - - return ret; -} - -void vfio_migration_finalize(void) -{ - if (!giommu_migration_blocker || - vfio_viommu_preset()) { - return; - } - - migrate_del_blocker(giommu_migration_blocker); - error_free(giommu_migration_blocker); - giommu_migration_blocker = NULL; + return vbasedev->group->container->space->as != &address_space_memory; } static void vfio_set_migration_error(int err) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 1db7d52ab2c1..e6e5e85f7580 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -878,9 +878,10 @@ int vfio_migration_realize(VFIODevice *vbasedev, Error **errp) return ret; } - ret = vfio_block_giommu_migration(vbasedev, errp); - if (ret) { - return ret; + if (vfio_viommu_preset(vbasedev)) { + error_setg(&err, "%s: Migration is currently not supported " + "with vIOMMU enabled", vbasedev->name); + return vfio_block_migration(vbasedev, err, errp); } trace_vfio_migration_realize(vbasedev->name); diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 31c4ab250fbe..c2cf7454ece6 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3255,7 +3255,6 @@ static void vfio_instance_finalize(Object *obj) */ vfio_put_device(vdev); vfio_put_group(group); - vfio_migration_finalize(); } static void vfio_exitfn(PCIDevice *pdev) diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 93429b9abba0..45167c8a8a54 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -227,7 +227,7 @@ extern VFIOGroupList vfio_group_list; bool vfio_mig_active(void); int vfio_block_multiple_devices_migration(VFIODevice *vbasedev, Error **errp); void vfio_unblock_multiple_devices_migration(void); -int vfio_block_giommu_migration(VFIODevice *vbasedev, Error **errp); +bool vfio_viommu_preset(VFIODevice *vbasedev); int64_t vfio_mig_bytes_transferred(void); void vfio_reset_bytes_transferred(void); @@ -254,6 +254,5 @@ int vfio_spapr_remove_window(VFIOContainer *container, int vfio_migration_realize(VFIODevice *vbasedev, Error **errp); void vfio_migration_exit(VFIODevice *vbasedev); -void vfio_migration_finalize(void); #endif /* HW_VFIO_VFIO_COMMON_H */ From patchwork Mon Jul 3 07:15:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Zhenzhong Duan X-Patchwork-Id: 1802529 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=NAeSqX51; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Qvcxz4Hwvz20b1 for ; Mon, 3 Jul 2023 17:29:43 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qGDyw-0006DZ-HS; Mon, 03 Jul 2023 03:28:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qGDyu-0006Cn-4b for qemu-devel@nongnu.org; Mon, 03 Jul 2023 03:28:20 -0400 Received: from mga12.intel.com ([192.55.52.136]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qGDys-0003xB-AT for qemu-devel@nongnu.org; Mon, 03 Jul 2023 03:28:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1688369298; x=1719905298; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=E9Sj3zLFXvdZ19TDrlDnntBtvlFXthWHSMWRElCtyzI=; b=NAeSqX51nG36HJfjpY0DrySiwVZFn/HQVzmctNU5tv21v8kA1E4/l/4Q XxhGmNUeYu16ZpO6cMyU+mI9OG2Exq1GL3Zk0yI8O4yKRuF5NI63M2/Fx 7y5BDNnMVkF6SYvgddEW/Rlz5AnUuPhf4oUH7VOLEUIJ6jiv8HdSfaIxR kcjTw/apLSu5uCExgAdHB4q8Ao4+sbgpZRMpFHgfhR/66gBzDRnMItxCc IUPmS/xGNdmnML+xq0ps+Pdiu8aCpXxwcSmCLDrX6t07Rn7bWAb9CvjF5 fIYYv5bFlBa6Hf1bgDyMiTNy23tO7yr3BXABYik4kk6cNsLbbcxHysoP6 A==; X-IronPort-AV: E=McAfee;i="6600,9927,10759"; a="342405652" X-IronPort-AV: E=Sophos;i="6.01,177,1684825200"; d="scan'208";a="342405652" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Jul 2023 00:28:16 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10759"; a="748058776" X-IronPort-AV: E=Sophos;i="6.01,177,1684825200"; d="scan'208";a="748058776" Received: from duan-server-s2600bt.bj.intel.com ([10.240.192.147]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Jul 2023 00:28:14 -0700 From: Zhenzhong Duan To: qemu-devel@nongnu.org Cc: alex.williamson@redhat.com, clg@redhat.com, joao.m.martins@oracle.com, avihaih@nvidia.com, chao.p.peng@intel.com Subject: [PATCH v6 5/7] vfio/migration: Free resources when vfio_migration_realize fails Date: Mon, 3 Jul 2023 15:15:08 +0800 Message-Id: <20230703071510.160712-4-zhenzhong.duan@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230703071510.160712-1-zhenzhong.duan@intel.com> References: <20230703071510.160712-1-zhenzhong.duan@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.136; envelope-from=zhenzhong.duan@intel.com; helo=mga12.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org When vfio_realize() succeeds, hot unplug will call vfio_exitfn() to free resources allocated in vfio_realize(); when vfio_realize() fails, vfio_exitfn() is never called and we need to free resources in vfio_realize(). In the case that vfio_migration_realize() fails, e.g: with -only-migratable & enable-migration=off, we see below: (qemu) device_add vfio-pci,host=81:11.1,id=vfio1,bus=root1,enable-migration=off 0000:81:11.1: Migration disabled Error: disallowing migration blocker (--only-migratable) for: 0000:81:11.1: Migration is disabled for VFIO device If we hotplug again we should see same log as above, but we see: (qemu) device_add vfio-pci,host=81:11.1,id=vfio1,bus=root1,enable-migration=off Error: vfio 0000:81:11.1: device is already attached That's because some references to VFIO device isn't released. For resources allocated in vfio_migration_realize(), free them by jumping to out_deinit path with calling a new function vfio_migration_deinit(). For resources allocated in vfio_realize(), free them by jumping to de-register path in vfio_realize(). Signed-off-by: Zhenzhong Duan Reviewed-by: Joao Martins Reviewed-by: Cédric Le Goater --- hw/vfio/migration.c | 33 +++++++++++++++++++++++---------- hw/vfio/pci.c | 1 + 2 files changed, 24 insertions(+), 10 deletions(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index e6e5e85f7580..e3954570c853 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -802,6 +802,17 @@ static int vfio_migration_init(VFIODevice *vbasedev) return 0; } +static void vfio_migration_deinit(VFIODevice *vbasedev) +{ + VFIOMigration *migration = vbasedev->migration; + + remove_migration_state_change_notifier(&migration->migration_state); + qemu_del_vm_change_state_handler(migration->vm_state); + unregister_savevm(VMSTATE_IF(vbasedev->dev), "vfio", vbasedev); + vfio_migration_free(vbasedev); + vfio_unblock_multiple_devices_migration(); +} + static int vfio_block_migration(VFIODevice *vbasedev, Error *err, Error **errp) { int ret; @@ -866,7 +877,7 @@ int vfio_migration_realize(VFIODevice *vbasedev, Error **errp) error_setg(&err, "%s: VFIO device doesn't support device dirty tracking", vbasedev->name); - return vfio_block_migration(vbasedev, err, errp); + goto add_blocker; } warn_report("%s: VFIO device doesn't support device dirty tracking", @@ -875,29 +886,31 @@ int vfio_migration_realize(VFIODevice *vbasedev, Error **errp) ret = vfio_block_multiple_devices_migration(vbasedev, errp); if (ret) { - return ret; + goto out_deinit; } if (vfio_viommu_preset(vbasedev)) { error_setg(&err, "%s: Migration is currently not supported " "with vIOMMU enabled", vbasedev->name); - return vfio_block_migration(vbasedev, err, errp); + goto add_blocker; } trace_vfio_migration_realize(vbasedev->name); return 0; + +add_blocker: + ret = vfio_block_migration(vbasedev, err, errp); +out_deinit: + if (ret) { + vfio_migration_deinit(vbasedev); + } + return ret; } void vfio_migration_exit(VFIODevice *vbasedev) { if (vbasedev->migration) { - VFIOMigration *migration = vbasedev->migration; - - remove_migration_state_change_notifier(&migration->migration_state); - qemu_del_vm_change_state_handler(migration->vm_state); - unregister_savevm(VMSTATE_IF(vbasedev->dev), "vfio", vbasedev); - vfio_migration_free(vbasedev); - vfio_unblock_multiple_devices_migration(); + vfio_migration_deinit(vbasedev); } if (vbasedev->migration_blocker) { diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index c2cf7454ece6..9154dd929d07 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3210,6 +3210,7 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) ret = vfio_migration_realize(vbasedev, errp); if (ret) { error_report("%s: Migration disabled", vbasedev->name); + goto out_deregister; } } From patchwork Mon Jul 3 07:15:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Zhenzhong Duan X-Patchwork-Id: 1802528 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=n4cH2awW; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4QvcxS1NJfz20b1 for ; Mon, 3 Jul 2023 17:29:15 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qGDyw-0006DY-H9; Mon, 03 Jul 2023 03:28:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qGDyv-0006DL-Hx for qemu-devel@nongnu.org; Mon, 03 Jul 2023 03:28:21 -0400 Received: from mga12.intel.com ([192.55.52.136]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qGDyt-0003wp-Vu for qemu-devel@nongnu.org; Mon, 03 Jul 2023 03:28:21 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1688369300; x=1719905300; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=TmL+HEBZmg//YXAT1wthhau6V4CBxfpWlzpevCqqaD4=; b=n4cH2awWIRqnF2/zZXIlcjshaiWRtLKFjKh4k7n9JrhZ9JDdeqQHyGnE U2NVxHndHHylK3bvfUibJyTilPGeDBAx7ZbsJhxiZ11r7N6Jg8aI2ovyF 0tt4UegDmBD4cQ3IXxx9jVXEkIY1bm4RZLZtY3U0tJ/h0IfLqQObM2MCF FAbu2CSpEW+FcSvuLGk0Q2AQOe+lFtgXUFd+Tpzwg5wFlYyQYE1NQ5rHY efh/xIFbINjdcH8+N8qhtXWgjEmHjAmySPQ8yUs6HNDxH9QXX9/u/qgat ECpElgXRUNeRCrw0cAzjBnAqyFK01d0YX9sZ9o4/ZEZnAmVDcC4vbpMBX g==; X-IronPort-AV: E=McAfee;i="6600,9927,10759"; a="342405665" X-IronPort-AV: E=Sophos;i="6.01,177,1684825200"; d="scan'208";a="342405665" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Jul 2023 00:28:19 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10759"; a="748058780" X-IronPort-AV: E=Sophos;i="6.01,177,1684825200"; d="scan'208";a="748058780" Received: from duan-server-s2600bt.bj.intel.com ([10.240.192.147]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Jul 2023 00:28:16 -0700 From: Zhenzhong Duan To: qemu-devel@nongnu.org Cc: alex.williamson@redhat.com, clg@redhat.com, joao.m.martins@oracle.com, avihaih@nvidia.com, chao.p.peng@intel.com Subject: [PATCH v6 6/7] vfio/migration: Remove print of "Migration disabled" Date: Mon, 3 Jul 2023 15:15:09 +0800 Message-Id: <20230703071510.160712-5-zhenzhong.duan@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230703071510.160712-1-zhenzhong.duan@intel.com> References: <20230703071510.160712-1-zhenzhong.duan@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.136; envelope-from=zhenzhong.duan@intel.com; helo=mga12.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Property enable_migration supports [on/off/auto]. In ON mode, error pointer is passed to errp and logged. In OFF mode, we doesn't need to log "Migration disabled" as it's intentional. In AUTO mode, we should only ever see errors or warnings if the device supports migration and an error or incompatibility occurs while further probing or configuring it. Lack of support for migration shoundn't generate an error or warning. Signed-off-by: Zhenzhong Duan Reviewed-by: Joao Martins Reviewed-by: Cédric Le Goater --- hw/vfio/pci.c | 1 - 1 file changed, 1 deletion(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 9154dd929d07..eefd4ec330d9 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3209,7 +3209,6 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) if (!pdev->failover_pair_id) { ret = vfio_migration_realize(vbasedev, errp); if (ret) { - error_report("%s: Migration disabled", vbasedev->name); goto out_deregister; } } From patchwork Mon Jul 3 07:15:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Zhenzhong Duan X-Patchwork-Id: 1802531 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=aOf+2vsI; dkim-atps=neutral Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4QvcyC1Cwfz20b1 for ; Mon, 3 Jul 2023 17:29:55 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qGDz1-0006Ef-E6; Mon, 03 Jul 2023 03:28:27 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qGDyz-0006EI-Ma for qemu-devel@nongnu.org; Mon, 03 Jul 2023 03:28:25 -0400 Received: from mga12.intel.com ([192.55.52.136]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qGDyw-0003yC-UQ for qemu-devel@nongnu.org; Mon, 03 Jul 2023 03:28:25 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1688369302; x=1719905302; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=mzmiumRbkMiqShyNMR0IULmYkTFgTraEzbJQDh209tg=; b=aOf+2vsI5UILEoWWpITVh38XOm1BIV2TuHeiD6GJywp2kRSw2yg2XI5+ OvD6T6zLhwVnYqMDIC7a6s+vZKoZKi25/OIvirmx/2T08lqXLVhbrHpD4 ExR2QSVmiGaMw8Ni3quY1E8gwo9oNLxuFZ1Ps3MAjsVfpyupIKO0baXQb LhF4AAdgLEnwTPmDKhcHDN1clqIDAU2Jmrg14Uedc4C7YT1qQXrFPYLbQ Jiv4N8fW8tQ4C3gf6joWYpyOIYnrV7WqbQj4UBUpyn06FmxICO1GH5dFG kDHGORJ9NYuGE2I1YgMJvCMPqxNPZISp0cx4r/fJmdyia6bRMAnFlmjUS w==; X-IronPort-AV: E=McAfee;i="6600,9927,10759"; a="342405675" X-IronPort-AV: E=Sophos;i="6.01,177,1684825200"; d="scan'208";a="342405675" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Jul 2023 00:28:21 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10759"; a="748058786" X-IronPort-AV: E=Sophos;i="6.01,177,1684825200"; d="scan'208";a="748058786" Received: from duan-server-s2600bt.bj.intel.com ([10.240.192.147]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Jul 2023 00:28:19 -0700 From: Zhenzhong Duan To: qemu-devel@nongnu.org Cc: alex.williamson@redhat.com, clg@redhat.com, joao.m.martins@oracle.com, avihaih@nvidia.com, chao.p.peng@intel.com Subject: [PATCH v6 7/7] vfio/migration: Return bool type for vfio_migration_realize() Date: Mon, 3 Jul 2023 15:15:10 +0800 Message-Id: <20230703071510.160712-6-zhenzhong.duan@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230703071510.160712-1-zhenzhong.duan@intel.com> References: <20230703071510.160712-1-zhenzhong.duan@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.136; envelope-from=zhenzhong.duan@intel.com; helo=mga12.intel.com X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Make vfio_migration_realize() adhere to the convention of other realize() callbacks(like qdev_realize) by returning bool instead of int. Suggested-by: Cédric Le Goater Suggested-by: Joao Martins Signed-off-by: Zhenzhong Duan Reviewed-by: Cédric Le Goater Reviewed-by: Joao Martins --- hw/vfio/migration.c | 15 ++++++++++----- hw/vfio/pci.c | 3 +-- include/hw/vfio/vfio-common.h | 2 +- 3 files changed, 12 insertions(+), 8 deletions(-) diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index e3954570c853..2674f4bc472d 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -846,7 +846,12 @@ void vfio_reset_bytes_transferred(void) bytes_transferred = 0; } -int vfio_migration_realize(VFIODevice *vbasedev, Error **errp) +/* + * Return true when either migration initialized or blocker registered. + * Currently only return false when adding blocker fails which will + * de-register vfio device. + */ +bool vfio_migration_realize(VFIODevice *vbasedev, Error **errp) { Error *err = NULL; int ret; @@ -854,7 +859,7 @@ int vfio_migration_realize(VFIODevice *vbasedev, Error **errp) if (vbasedev->enable_migration == ON_OFF_AUTO_OFF) { error_setg(&err, "%s: Migration is disabled for VFIO device", vbasedev->name); - return vfio_block_migration(vbasedev, err, errp); + return !vfio_block_migration(vbasedev, err, errp); } ret = vfio_migration_init(vbasedev); @@ -869,7 +874,7 @@ int vfio_migration_realize(VFIODevice *vbasedev, Error **errp) vbasedev->name, ret, strerror(-ret)); } - return vfio_block_migration(vbasedev, err, errp); + return !vfio_block_migration(vbasedev, err, errp); } if (!vbasedev->dirty_pages_supported) { @@ -896,7 +901,7 @@ int vfio_migration_realize(VFIODevice *vbasedev, Error **errp) } trace_vfio_migration_realize(vbasedev->name); - return 0; + return true; add_blocker: ret = vfio_block_migration(vbasedev, err, errp); @@ -904,7 +909,7 @@ out_deinit: if (ret) { vfio_migration_deinit(vbasedev); } - return ret; + return !ret; } void vfio_migration_exit(VFIODevice *vbasedev) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index eefd4ec330d9..68dd99283620 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3207,8 +3207,7 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) } if (!pdev->failover_pair_id) { - ret = vfio_migration_realize(vbasedev, errp); - if (ret) { + if (!vfio_migration_realize(vbasedev, errp)) { goto out_deregister; } } diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 45167c8a8a54..da43d273524e 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -252,7 +252,7 @@ int vfio_spapr_create_window(VFIOContainer *container, int vfio_spapr_remove_window(VFIOContainer *container, hwaddr offset_within_address_space); -int vfio_migration_realize(VFIODevice *vbasedev, Error **errp); +bool vfio_migration_realize(VFIODevice *vbasedev, Error **errp); void vfio_migration_exit(VFIODevice *vbasedev); #endif /* HW_VFIO_VFIO_COMMON_H */