From patchwork Fri Jul 12 14:02:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 1959891 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=oracle.com header.i=@oracle.com header.a=rsa-sha256 header.s=corp-2023-11-20 header.b=QugoPQa4; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WLCyN74Whz1xpd for ; Sat, 13 Jul 2024 00:04:28 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sSGqy-00011F-Dt; Fri, 12 Jul 2024 10:02:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sSGqu-0000qP-5Z for qemu-devel@nongnu.org; Fri, 12 Jul 2024 10:02:24 -0400 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sSGqs-0005v8-BE for qemu-devel@nongnu.org; Fri, 12 Jul 2024 10:02:23 -0400 Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 46CDIhH0028172; Fri, 12 Jul 2024 14:02:17 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h= from:to:cc:subject:date:message-id:in-reply-to:references; s= corp-2023-11-20; bh=7LVpahXMk4/9lxYHPPx8Bi3WW1Xzr0usvH6SGIEWUdg=; b= QugoPQa4DibkyUQNIX8rMK5Bnj0cWXk/cH19FyzovW9P3kBPle1wOlWj+UTm0NYA uNJg5Yy78vVVEFLMPJMuHBDKmNMuSLmSU9HY6edrBfadup7J7Erv+n0pnoS2wjXV sf2KNQCKvySetkHIo5uETx27RrVVpZKXKzx+O99Chiej9ADuHTsUfPmBviZQkVLY sV4CTjIYlPvM2mW5XydRHwDaXRQ8jaNyc1y+yywT2sJPo73U7kbypB1ZEdssujKd sgTK8J7FNOPQ055GB60lTVr52rtts8n5raXsWLWCOMLO0ZA3233oja1a1UZ1KIF2 34YpyjaO3OSKHo3mRPr20Q== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 406wkyc4f0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Jul 2024 14:02:17 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 46CC5fAD010941; Fri, 12 Jul 2024 14:02:15 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 409vv74jsn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Jul 2024 14:02:15 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 46CE2DPo020438; Fri, 12 Jul 2024 14:02:14 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 409vv74jq7-2; Fri, 12 Jul 2024 14:02:14 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: "Michael S. Tsirkin" , Jason Wang , Philippe Mathieu-Daude , Eugenio Perez Martin , Peter Xu , Fabiano Rosas , Si-Wei Liu , Steve Sistare Subject: [RFC V1 1/7] migration: cpr_needed_for_reuse Date: Fri, 12 Jul 2024 07:02:05 -0700 Message-Id: <1720792931-456433-2-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1720792931-456433-1-git-send-email-steven.sistare@oracle.com> References: <1720792931-456433-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-07-12_10,2024-07-11_01,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 malwarescore=0 phishscore=0 spamscore=0 mlxscore=0 suspectscore=0 mlxlogscore=943 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2406180000 definitions=main-2407120094 X-Proofpoint-ORIG-GUID: K_EZ-MEHU4kdPD1xI367mEADj5pGTSec X-Proofpoint-GUID: K_EZ-MEHU4kdPD1xI367mEADj5pGTSec Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Define a vmstate "needed" helper. This will be moved to the preceding patch series "Live update: cpr-exec" because it is needed by multiple devices. Signed-off-by: Steve Sistare --- include/migration/cpr.h | 1 + migration/cpr.c | 5 +++++ 2 files changed, 6 insertions(+) diff --git a/include/migration/cpr.h b/include/migration/cpr.h index c6c60f87bc..8d20d3ec49 100644 --- a/include/migration/cpr.h +++ b/include/migration/cpr.h @@ -24,6 +24,7 @@ void cpr_resave_fd(const char *name, int id, int fd); int cpr_state_save(Error **errp); int cpr_state_load(Error **errp); +bool cpr_needed_for_reuse(void *opaque); QEMUFile *cpr_exec_output(Error **errp); QEMUFile *cpr_exec_input(Error **errp); diff --git a/migration/cpr.c b/migration/cpr.c index f756c1552d..843241c073 100644 --- a/migration/cpr.c +++ b/migration/cpr.c @@ -236,3 +236,8 @@ int cpr_state_load(Error **errp) return ret; } +bool cpr_needed_for_reuse(void *opaque) +{ + MigMode mode = migrate_mode(); + return mode == MIG_MODE_CPR_EXEC; +} From patchwork Fri Jul 12 14:02:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 1959887 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=oracle.com header.i=@oracle.com header.a=rsa-sha256 header.s=corp-2023-11-20 header.b=Vt6fxaB4; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WLCxF6Wc9z1xpd for ; Sat, 13 Jul 2024 00:03:29 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sSGrC-0001vG-89; Fri, 12 Jul 2024 10:02:42 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sSGr7-0001dx-14 for qemu-devel@nongnu.org; Fri, 12 Jul 2024 10:02:38 -0400 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sSGr4-0005vG-LB for qemu-devel@nongnu.org; Fri, 12 Jul 2024 10:02:36 -0400 Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 46CDIjM8028175; Fri, 12 Jul 2024 14:02:18 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h= from:to:cc:subject:date:message-id:in-reply-to:references; s= corp-2023-11-20; bh=oCT40AtDEKvFtYLo+FbOlAkqvMJe0KAt51MjZe2n1ug=; b= Vt6fxaB4wDudYnpPAclRlLdjZl2ZM61ClnJ2+0DMLYnD/F/1NYHHpWAwPrBuSo02 RhFazoF852vpZUSQlChX3QR55YjPxNV4Lqz+0KrVtErgMa81oLq5akyWP2R6E9Kx 1RboHZpFHxKNRbc+Ybu17HJI593oinvPs1Jqfg6ufZmmmZfPJYKPe1RxCnpk17aF V2CdM7Mqj9y54dV/vqoZ/soYfHNWB6Twy5lxQi7Oksc/U16e1SZHtEMoI1ercSZL ti75BQm3TOUZuz7EUvJVqG0SoYCF0LGR+5J52rgUhL7d40XDh3K0NnXVuKfhMFPh Qu6M9yZSVGMyX5n6cmZaPw== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 406wkyc4f5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Jul 2024 14:02:17 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 46CDmDek010418; Fri, 12 Jul 2024 14:02:16 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 409vv74jtp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Jul 2024 14:02:16 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 46CE2DPq020438; Fri, 12 Jul 2024 14:02:16 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 409vv74jq7-3; Fri, 12 Jul 2024 14:02:15 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: "Michael S. Tsirkin" , Jason Wang , Philippe Mathieu-Daude , Eugenio Perez Martin , Peter Xu , Fabiano Rosas , Si-Wei Liu , Steve Sistare Subject: [RFC V1 2/7] migration: skip dirty memory tracking for cpr Date: Fri, 12 Jul 2024 07:02:06 -0700 Message-Id: <1720792931-456433-3-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1720792931-456433-1-git-send-email-steven.sistare@oracle.com> References: <1720792931-456433-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-07-12_10,2024-07-11_01,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 malwarescore=0 phishscore=0 spamscore=0 mlxscore=0 suspectscore=0 mlxlogscore=712 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2406180000 definitions=main-2407120094 X-Proofpoint-ORIG-GUID: GiN25Z3wON-RkORBrRD22UTy7JGVmIxU X-Proofpoint-GUID: GiN25Z3wON-RkORBrRD22UTy7JGVmIxU Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org CPR preserves memory in place, so there is no need to track dirty memory. By skipping it, CPR can support devices that do not support tracking. Signed-off-by: Steve Sistare --- system/memory.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/system/memory.c b/system/memory.c index b7548bf112..aef584e638 100644 --- a/system/memory.c +++ b/system/memory.c @@ -27,6 +27,7 @@ #include "exec/memory-internal.h" #include "exec/ram_addr.h" +#include "migration/misc.h" #include "sysemu/kvm.h" #include "sysemu/runstate.h" #include "sysemu/tcg.h" @@ -2947,6 +2948,11 @@ bool memory_global_dirty_log_start(unsigned int flags, Error **errp) assert(flags && !(flags & (~GLOBAL_DIRTY_MASK))); + /* CPR preserves memory in place, so no need to track dirty memory */ + if (migrate_mode() != MIG_MODE_NORMAL) { + return true; + } + if (vmstate_change) { /* If there is postponed stop(), operate on it first */ postponed_stop_flags &= ~flags; @@ -3021,6 +3027,11 @@ static void memory_vm_change_state_handler(void *opaque, bool running, void memory_global_dirty_log_stop(unsigned int flags) { + /* CPR preserves memory in place, so no need to track dirty memory */ + if (migrate_mode() != MIG_MODE_NORMAL) { + return; + } + if (!runstate_is_running()) { /* Postpone the dirty log stop, e.g., to when VM starts again */ if (vmstate_change) { From patchwork Fri Jul 12 14:02:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 1959889 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=oracle.com header.i=@oracle.com header.a=rsa-sha256 header.s=corp-2023-11-20 header.b=iWTLo1sz; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WLCxs5Nf8z1xpd for ; Sat, 13 Jul 2024 00:04:01 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sSGrF-000293-JH; Fri, 12 Jul 2024 10:02:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sSGrE-00023F-1S for qemu-devel@nongnu.org; Fri, 12 Jul 2024 10:02:44 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sSGrC-0005vD-5P for qemu-devel@nongnu.org; Fri, 12 Jul 2024 10:02:43 -0400 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 46CDIN2Y025663; Fri, 12 Jul 2024 14:02:18 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h= from:to:cc:subject:date:message-id:in-reply-to:references; s= corp-2023-11-20; bh=dTbJwlJt4IPxvPC5MtqUjG7iInlfl4OVlAOz98Jk/dg=; b= iWTLo1sz4YI7aTl5St1+X1mpFFFsAs5fXVJqH3FPKzubm8ubXTHam0qc7K+UUYqa 5ldYgcSSOr6NhzSiSUlHcdrsIVb1LdRtdEd+iHmEV4LeiUPTs14B3enS48eRg5Oz 4DrV4YmsCTzOVbZ/Tdov3spWYfi6AkqSC9bwWNmCAvH7zZcUykFGEbKA0//4qwS+ 2rN+kj13bhwuA1+esdlM5zcTLzIoM2nQkRiwt5RHCHx2eukpO+06Ih3jEbjw4RPB cT5PCrv44WCL+v61+0FQiTNcYF37ry886ota4H+ZwNETOAt9KCfShLsB03PzYWj3 7s0An1MpacIWzp7pHC13kg== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 406wknuxvw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Jul 2024 14:02:18 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 46CDmDel010418; Fri, 12 Jul 2024 14:02:17 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 409vv74jv2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Jul 2024 14:02:17 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 46CE2DPs020438; Fri, 12 Jul 2024 14:02:17 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 409vv74jq7-4; Fri, 12 Jul 2024 14:02:17 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: "Michael S. Tsirkin" , Jason Wang , Philippe Mathieu-Daude , Eugenio Perez Martin , Peter Xu , Fabiano Rosas , Si-Wei Liu , Steve Sistare Subject: [RFC V1 3/7] vdpa/cpr: preserve device fd Date: Fri, 12 Jul 2024 07:02:07 -0700 Message-Id: <1720792931-456433-4-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1720792931-456433-1-git-send-email-steven.sistare@oracle.com> References: <1720792931-456433-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-07-12_10,2024-07-11_01,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 malwarescore=0 phishscore=0 spamscore=0 mlxscore=0 suspectscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2406180000 definitions=main-2407120094 X-Proofpoint-GUID: pHxkJgmHEbAxjEwcPvRl3fzpaqj2DqAx X-Proofpoint-ORIG-GUID: pHxkJgmHEbAxjEwcPvRl3fzpaqj2DqAx Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Save the vdpa device fd in CPR state when it is created, and fetch the fd from that state after CPR. Remember that the fd was reused, for subsequent patches. Signed-off-by: Steve Sistare --- include/hw/virtio/vhost-vdpa.h | 3 +++ net/vhost-vdpa.c | 24 ++++++++++++++++++------ 2 files changed, 21 insertions(+), 6 deletions(-) diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h index 0a9575b469..427458cfed 100644 --- a/include/hw/virtio/vhost-vdpa.h +++ b/include/hw/virtio/vhost-vdpa.h @@ -54,6 +54,9 @@ typedef struct vhost_vdpa_shared { /* Vdpa must send shadow addresses as IOTLB key for data queues, not GPA */ bool shadow_data; + /* Device descriptor is being reused after CPR restart */ + bool reused; + /* SVQ switching is in progress, or already completed? */ SVQTransitionState svq_switching; } VhostVDPAShared; diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index daa38428c5..e6010e8900 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -12,6 +12,7 @@ #include "qemu/osdep.h" #include "clients.h" #include "hw/virtio/virtio-net.h" +#include "migration/cpr.h" #include "net/vhost_net.h" #include "net/vhost-vdpa.h" #include "hw/virtio/vhost-vdpa.h" @@ -240,8 +241,10 @@ static void vhost_vdpa_cleanup(NetClientState *nc) if (s->vhost_vdpa.index != 0) { return; } + cpr_delete_fd(nc->name, 0); qemu_close(s->vhost_vdpa.shared->device_fd); g_free(s->vhost_vdpa.shared); + s->vhost_vdpa.shared = NULL; } /** Dummy SetSteeringEBPF to support RSS for vhost-vdpa backend */ @@ -1675,6 +1678,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer, int nvqs, bool is_datapath, bool svq, + bool reused, struct vhost_vdpa_iova_range iova_range, uint64_t features, VhostVDPAShared *shared, @@ -1712,6 +1716,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState *peer, &s->vhost_vdpa.migration_blocker); s->vhost_vdpa.shared = g_new0(VhostVDPAShared, 1); s->vhost_vdpa.shared->device_fd = vdpa_device_fd; + s->vhost_vdpa.shared->reused = reused; s->vhost_vdpa.shared->iova_range = iova_range; s->vhost_vdpa.shared->shadow_data = svq; } else if (!is_datapath) { @@ -1793,6 +1798,7 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name, struct vhost_vdpa_iova_range iova_range; NetClientState *nc; int queue_pairs, r, i = 0, has_cvq = 0; + bool reused; assert(netdev->type == NET_CLIENT_DRIVER_VHOST_VDPA); opts = &netdev->u.vhost_vdpa; @@ -1808,13 +1814,17 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name, return -1; } - if (opts->vhostdev) { + vdpa_device_fd = cpr_find_fd(name, 0); + reused = (vdpa_device_fd != -1); + + if (opts->vhostdev && vdpa_device_fd == -1) { vdpa_device_fd = qemu_open(opts->vhostdev, O_RDWR, errp); if (vdpa_device_fd == -1) { return -errno; } - } else { - /* has_vhostfd */ + cpr_save_fd(name, 0, vdpa_device_fd); + + } else if (opts->vhostfd) { vdpa_device_fd = monitor_fd_param(monitor_cur(), opts->vhostfd, errp); if (vdpa_device_fd == -1) { error_prepend(errp, "vhost-vdpa: unable to parse vhostfd: "); @@ -1855,7 +1865,8 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name, } ncs[i] = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name, vdpa_device_fd, i, 2, true, opts->x_svq, - iova_range, features, shared, errp); + reused, iova_range, features, shared, + errp); if (!ncs[i]) goto err; } @@ -1866,8 +1877,8 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char *name, nc = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name, vdpa_device_fd, i, 1, false, - opts->x_svq, iova_range, features, shared, - errp); + opts->x_svq, reused, iova_range, features, + shared, errp); if (!nc) goto err; } @@ -1882,6 +1893,7 @@ err: } qemu_close(vdpa_device_fd); + cpr_delete_fd(name, 0); return -1; } From patchwork Fri Jul 12 14:02:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 1959892 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=oracle.com header.i=@oracle.com header.a=rsa-sha256 header.s=corp-2023-11-20 header.b=atijf7pA; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WLCyf5tD2z1xpd for ; Sat, 13 Jul 2024 00:04:42 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sSGrA-0001mJ-BA; Fri, 12 Jul 2024 10:02:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sSGr2-0001NU-Oe for qemu-devel@nongnu.org; Fri, 12 Jul 2024 10:02:34 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sSGr1-0005vQ-5V for qemu-devel@nongnu.org; Fri, 12 Jul 2024 10:02:32 -0400 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 46CDIMrN023701; Fri, 12 Jul 2024 14:02:20 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h= from:to:cc:subject:date:message-id:in-reply-to:references; s= corp-2023-11-20; bh=S9Iux9B0t0Qwr10Zwlv9f/aWAepqp3WhMb9It8eH5qo=; b= atijf7pARGfwIVEaO8ZsSHzNWTf4maTJx2XWw/SFMHJGESsgFyL6Z/jG2Ebihtpg bAGOO0jkiD35Vfi86fgI+DBENt7MfnBC09XIhFUdjljojcNMN7R81QIaUYgpGpu+ 0Ty5M/1yAYyP6xekUW6pDWcpCfWQVQ1yPEHBRy/c4BFaNKtDlMRHeWZ/X8SgClW3 qprUKWKxlOGhhetDqGbEJ4HFQaJuvPoP+baY2JHQgyoAmhh/QJlQZdLR6666sB/X lNokS91rJMr2ZGZ7+Dh0wBWluSrDXo0XUksni+SG9Ulke/2Aq/kXnax+eucRstfO KCoagVLjeS/pQ61hlAJFzA== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 407emt37uk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Jul 2024 14:02:19 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 46CCwLpM010887; Fri, 12 Jul 2024 14:02:19 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 409vv74jw4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Jul 2024 14:02:19 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 46CE2DPu020438; Fri, 12 Jul 2024 14:02:18 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 409vv74jq7-5; Fri, 12 Jul 2024 14:02:18 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: "Michael S. Tsirkin" , Jason Wang , Philippe Mathieu-Daude , Eugenio Perez Martin , Peter Xu , Fabiano Rosas , Si-Wei Liu , Steve Sistare Subject: [RFC V1 4/7] vdpa/cpr: kernel interfaces Date: Fri, 12 Jul 2024 07:02:08 -0700 Message-Id: <1720792931-456433-5-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1720792931-456433-1-git-send-email-steven.sistare@oracle.com> References: <1720792931-456433-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-07-12_10,2024-07-11_01,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 malwarescore=0 phishscore=0 spamscore=0 mlxscore=0 suspectscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2406180000 definitions=main-2407120094 X-Proofpoint-GUID: a0OpHpbWmSD__iZJ8Vqiereq-MCnGA6Z X-Proofpoint-ORIG-GUID: a0OpHpbWmSD__iZJ8Vqiereq-MCnGA6Z Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Add the proposed vdpa kernel interfaces for CPR: VHOST_NEW_OWNER: new ioctl VHOST_BACKEND_F_NEW_OWNER: new capability VHOST_IOTLB_REMAP: new iotlb message VHOST_BACKEND_F_IOTLB_REMAP: new capability Signed-off-by: Steve Sistare --- include/standard-headers/linux/vhost_types.h | 7 +++++++ linux-headers/linux/vhost.h | 9 +++++++++ 2 files changed, 16 insertions(+) diff --git a/include/standard-headers/linux/vhost_types.h b/include/standard-headers/linux/vhost_types.h index fd54044936..fa605ee1da 100644 --- a/include/standard-headers/linux/vhost_types.h +++ b/include/standard-headers/linux/vhost_types.h @@ -87,6 +87,7 @@ struct vhost_iotlb_msg { */ #define VHOST_IOTLB_BATCH_BEGIN 5 #define VHOST_IOTLB_BATCH_END 6 +#define VHOST_IOTLB_REMAP 7 uint8_t type; }; @@ -193,4 +194,10 @@ struct vhost_vdpa_iova_range { /* IOTLB don't flush memory mapping across device reset */ #define VHOST_BACKEND_F_IOTLB_PERSIST 0x8 +/* Supports VHOST_NEW_OWNER */ +#define VHOST_BACKEND_F_NEW_OWNER 0x9 + +/* Supports VHOST_IOTLB_REMAP */ +#define VHOST_BACKEND_F_IOTLB_REMAP 0xa + #endif diff --git a/linux-headers/linux/vhost.h b/linux-headers/linux/vhost.h index b95dd84eef..6c008c956a 100644 --- a/linux-headers/linux/vhost.h +++ b/linux-headers/linux/vhost.h @@ -123,6 +123,15 @@ #define VHOST_SET_BACKEND_FEATURES _IOW(VHOST_VIRTIO, 0x25, __u64) #define VHOST_GET_BACKEND_FEATURES _IOR(VHOST_VIRTIO, 0x26, __u64) +/* Set current process as the new owner of this file descriptor. The fd must + * already be owned, via a prior call to VHOST_SET_OWNER. The pinned memory + * count is transferred from the previous to the new owner. + * Errors: + * EINVAL: not owned + * EBUSY: caller is already the owner + * ENOMEM: RLIMIT_MEMLOCK exceeded + */ +#define VHOST_NEW_OWNER _IO(VHOST_VIRTIO, 0x27) /* VHOST_NET specific defines */ /* Attach virtio net ring to a raw socket, or tap device. From patchwork Fri Jul 12 14:02:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 1959885 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=oracle.com header.i=@oracle.com header.a=rsa-sha256 header.s=corp-2023-11-20 header.b=IpHgFPM/; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WLCx323ZCz1xpd for ; Sat, 13 Jul 2024 00:03:19 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sSGqw-0000xU-M9; Fri, 12 Jul 2024 10:02:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sSGqu-0000ra-EZ for qemu-devel@nongnu.org; Fri, 12 Jul 2024 10:02:24 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sSGqs-0005va-KU for qemu-devel@nongnu.org; Fri, 12 Jul 2024 10:02:24 -0400 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 46CDILPG023667; Fri, 12 Jul 2024 14:02:21 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h= from:to:cc:subject:date:message-id:in-reply-to:references; s= corp-2023-11-20; bh=GNCx5dMy+PRyGVvp4jQJTfzVCcno1ZP9aVCm4n9/+/o=; b= IpHgFPM/faqX6xcrG8wBngp+z4ZmvBS/oHRS2AImo8xKlqXGW6vWPULIZxqfMvmy REegDku6ej/3Ba9UvuPZdTDAmfIZttcIqZGLXmd7+T12vmflWLxHTOWe9W9dskSW OlJjrYlvvr/8biWyhIDnheF41SVIbTT5KNtzYAfQSPD+u+zKPKgOazMJjzNYGz/q vr2R7HktJNJkFt+txgpLoSgC34CznTIVlkBhmyZkuj82Z/MkAMTMCccGK9eKIy2B 4DsOrL6sMFrLUzbRcIR37zTdQrcQiQh4KAbvNr77X9q5MI3GZYJwW8EQlr4RD/sz 0IQfWMZSSmj/NOVV9smqFA== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 407emt37un-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Jul 2024 14:02:20 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 46CCDrL7010884; Fri, 12 Jul 2024 14:02:20 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 409vv74jxm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Jul 2024 14:02:20 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 46CE2DPw020438; Fri, 12 Jul 2024 14:02:19 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 409vv74jq7-6; Fri, 12 Jul 2024 14:02:19 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: "Michael S. Tsirkin" , Jason Wang , Philippe Mathieu-Daude , Eugenio Perez Martin , Peter Xu , Fabiano Rosas , Si-Wei Liu , Steve Sistare Subject: [RFC V1 5/7] vdpa/cpr: use VHOST_NEW_OWNER Date: Fri, 12 Jul 2024 07:02:09 -0700 Message-Id: <1720792931-456433-6-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1720792931-456433-1-git-send-email-steven.sistare@oracle.com> References: <1720792931-456433-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-07-12_10,2024-07-11_01,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 malwarescore=0 phishscore=0 spamscore=0 mlxscore=0 suspectscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2406180000 definitions=main-2407120094 X-Proofpoint-GUID: iNwx8GbM7VUm9GG7-ZReGUduC8NUMrsi X-Proofpoint-ORIG-GUID: iNwx8GbM7VUm9GG7-ZReGUduC8NUMrsi Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Block CPR if the kernel does not support VHOST_NEW_OWNER. After CPR, call VHOST_NEW_OWNER in new QEMU. Signed-off-by: Steve Sistare --- hw/virtio/trace-events | 1 + hw/virtio/vhost-vdpa.c | 24 ++++++++++++++++++++++-- include/hw/virtio/vhost.h | 1 + 3 files changed, 24 insertions(+), 2 deletions(-) diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index 3cf84e04a7..990c61be79 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -64,6 +64,7 @@ vhost_vdpa_set_vring_kick(void *dev, unsigned int index, int fd) "dev: %p index: vhost_vdpa_set_vring_call(void *dev, unsigned int index, int fd) "dev: %p index: %u fd: %d" vhost_vdpa_get_features(void *dev, uint64_t features) "dev: %p features: 0x%"PRIx64 vhost_vdpa_set_owner(void *dev) "dev: %p" +vhost_vdpa_new_owner(void *dev) "dev: %p" vhost_vdpa_vq_get_addr(void *dev, void *vq, uint64_t desc_user_addr, uint64_t avail_user_addr, uint64_t used_user_addr) "dev: %p vq: %p desc_user_addr: 0x%"PRIx64" avail_user_addr: 0x%"PRIx64" used_user_addr: 0x%"PRIx64 vhost_vdpa_get_iova_range(void *dev, uint64_t first, uint64_t last) "dev: %p first: 0x%"PRIx64" last: 0x%"PRIx64 vhost_vdpa_set_config_call(void *dev, int fd)"dev: %p fd: %d" diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 3cdaa12ed5..9e3f414ac2 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -769,6 +769,7 @@ static int vhost_vdpa_cleanup(struct vhost_dev *dev) vhost_vdpa_svq_cleanup(dev); dev->opaque = NULL; + migrate_del_blocker(&dev->cpr_blocker); return 0; } @@ -848,13 +849,13 @@ static int vhost_vdpa_set_backend_cap(struct vhost_dev *dev) uint64_t f = 0x1ULL << VHOST_BACKEND_F_IOTLB_MSG_V2 | 0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH | 0x1ULL << VHOST_BACKEND_F_IOTLB_ASID | - 0x1ULL << VHOST_BACKEND_F_SUSPEND; + 0x1ULL << VHOST_BACKEND_F_SUSPEND | + 0x1ULL << VHOST_BACKEND_F_NEW_OWNER; int r; if (vhost_vdpa_call(dev, VHOST_GET_BACKEND_FEATURES, &features)) { return -EFAULT; } - features &= f; if (vhost_vdpa_first_dev(dev)) { @@ -1360,6 +1361,18 @@ static int vhost_vdpa_dev_start(struct vhost_dev *dev, bool started) } if (started) { + /* + * Register a blocker the first time device is started (when we know + * its capabilities). + */ + if (!dev->cpr_blocker && + !(dev->backend_cap & BIT_ULL(VHOST_BACKEND_F_NEW_OWNER))) { + error_setg(&dev->cpr_blocker, "vhost-vdpa: device does not support " + "VHOST_BACKEND_F_NEW_OWNER"); + migrate_add_blocker_modes(&dev->cpr_blocker, &error_abort, + MIG_MODE_CPR_EXEC, -1); + } + if (vhost_dev_has_iommu(dev) && (v->shadow_vqs_enabled)) { error_report("SVQ can not work while IOMMU enable, please disable" "IOMMU and try again"); @@ -1518,10 +1531,17 @@ static int vhost_vdpa_get_features(struct vhost_dev *dev, static int vhost_vdpa_set_owner(struct vhost_dev *dev) { + struct vhost_vdpa *v = dev->opaque; + if (!vhost_vdpa_first_dev(dev)) { return 0; } + if (v->shared->reused) { + trace_vhost_vdpa_new_owner(dev); + return vhost_vdpa_call(dev, VHOST_NEW_OWNER, NULL); + } + trace_vhost_vdpa_set_owner(dev); return vhost_vdpa_call(dev, VHOST_SET_OWNER, NULL); } diff --git a/include/hw/virtio/vhost.h b/include/hw/virtio/vhost.h index d75faf46e9..3f1b802f85 100644 --- a/include/hw/virtio/vhost.h +++ b/include/hw/virtio/vhost.h @@ -133,6 +133,7 @@ struct vhost_dev { QLIST_HEAD(, vhost_iommu) iommu_list; IOMMUNotifier n; const VhostDevConfigOps *config_ops; + Error *cpr_blocker; }; extern const VhostOps kernel_ops; From patchwork Fri Jul 12 14:02:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 1959886 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=oracle.com header.i=@oracle.com header.a=rsa-sha256 header.s=corp-2023-11-20 header.b=RMFhAFae; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WLCx33RKjz1xqy for ; Sat, 13 Jul 2024 00:03:19 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sSGr2-0001Gc-7W; Fri, 12 Jul 2024 10:02:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sSGqv-0000wR-KU for qemu-devel@nongnu.org; Fri, 12 Jul 2024 10:02:26 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sSGqt-0005vk-Po for qemu-devel@nongnu.org; Fri, 12 Jul 2024 10:02:25 -0400 Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 46CDILap014304; Fri, 12 Jul 2024 14:02:22 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h= from:to:cc:subject:date:message-id:in-reply-to:references; s= corp-2023-11-20; bh=uprx7b42zMa86CF2fZVDrRfXE9oASXgLY0ct8yWJsOE=; b= RMFhAFae6duz1UKldOqsGkuO7PgVlsjUeQSj81BzgnjurmOigdbA994LU2z/v5QK cToVAf69MiokohKv3U5K/TizjmbmK+C3YgO1wX/yAIb/qkLsurxISwDvh7Hon410 TTmXSNExZihafWiAy6qxFheb/CXQ6VCV7lhw3z+pRc/y6MVhz3Cq4Z8///O7+Jyb AVHpgX/wb7RbwAf9hK59DrYMDDoqwGvIex4Rl5FbhLvrqJPrOudXEsV7kYaTzogA 1kZD64xF6DEn5ep5qF2mqoao++lH38Fi/Gc+1ZBqo0CURG23o7FK34S6rAn8+6vW 8uI7ohE4qrAIXqo4QhGTJg== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 406wybv2ea-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Jul 2024 14:02:22 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 46CDNu4H010919; Fri, 12 Jul 2024 14:02:21 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 409vv74jye-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Jul 2024 14:02:21 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 46CE2DQ0020438; Fri, 12 Jul 2024 14:02:21 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 409vv74jq7-7; Fri, 12 Jul 2024 14:02:20 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: "Michael S. Tsirkin" , Jason Wang , Philippe Mathieu-Daude , Eugenio Perez Martin , Peter Xu , Fabiano Rosas , Si-Wei Liu , Steve Sistare Subject: [RFC V1 6/7] vdpa/cpr: pass shadow parameter to dma functions Date: Fri, 12 Jul 2024 07:02:10 -0700 Message-Id: <1720792931-456433-7-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1720792931-456433-1-git-send-email-steven.sistare@oracle.com> References: <1720792931-456433-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-07-12_10,2024-07-11_01,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 malwarescore=0 phishscore=0 spamscore=0 mlxscore=0 suspectscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2406180000 definitions=main-2407120094 X-Proofpoint-GUID: DOAldhxAyZQzGSlr4M7dtTBq0BIqyCuI X-Proofpoint-ORIG-GUID: DOAldhxAyZQzGSlr4M7dtTBq0BIqyCuI Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Pass a parameter to the dma mapping functions that indicates if the memory backs rings or buffers for svq's. No functional change. Signed-off-by: Steve Sistare --- hw/virtio/vhost-vdpa.c | 19 ++++++++++--------- include/hw/virtio/vhost-vdpa.h | 4 ++-- net/vhost-vdpa.c | 5 +++-- 3 files changed, 15 insertions(+), 13 deletions(-) diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 9e3f414ac2..d9ebc396b7 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -92,7 +92,7 @@ static bool vhost_vdpa_listener_skipped_section(MemoryRegionSection *section, * This is not an ABI break since it is set to 0 by the initializer anyway. */ int vhost_vdpa_dma_map(VhostVDPAShared *s, uint32_t asid, hwaddr iova, - hwaddr size, void *vaddr, bool readonly) + hwaddr size, void *vaddr, bool readonly, bool shadow) { struct vhost_msg_v2 msg = {}; int fd = s->device_fd; @@ -124,7 +124,7 @@ int vhost_vdpa_dma_map(VhostVDPAShared *s, uint32_t asid, hwaddr iova, * This is not an ABI break since it is set to 0 by the initializer anyway. */ int vhost_vdpa_dma_unmap(VhostVDPAShared *s, uint32_t asid, hwaddr iova, - hwaddr size) + hwaddr size, bool shadow) { struct vhost_msg_v2 msg = {}; int fd = s->device_fd; @@ -234,7 +234,7 @@ static void vhost_vdpa_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) return; } ret = vhost_vdpa_dma_map(s, VHOST_VDPA_GUEST_PA_ASID, iova, - iotlb->addr_mask + 1, vaddr, read_only); + iotlb->addr_mask + 1, vaddr, read_only, false); if (ret) { error_report("vhost_vdpa_dma_map(%p, 0x%" HWADDR_PRIx ", " "0x%" HWADDR_PRIx ", %p) = %d (%m)", @@ -242,7 +242,7 @@ static void vhost_vdpa_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) } } else { ret = vhost_vdpa_dma_unmap(s, VHOST_VDPA_GUEST_PA_ASID, iova, - iotlb->addr_mask + 1); + iotlb->addr_mask + 1, false); if (ret) { error_report("vhost_vdpa_dma_unmap(%p, 0x%" HWADDR_PRIx ", " "0x%" HWADDR_PRIx ") = %d (%m)", @@ -376,7 +376,8 @@ static void vhost_vdpa_listener_region_add(MemoryListener *listener, vhost_vdpa_iotlb_batch_begin_once(s); ret = vhost_vdpa_dma_map(s, VHOST_VDPA_GUEST_PA_ASID, iova, - int128_get64(llsize), vaddr, section->readonly); + int128_get64(llsize), vaddr, section->readonly, + false); if (ret) { error_report("vhost vdpa map fail!"); goto fail_map; @@ -463,7 +464,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener, if (int128_eq(llsize, int128_2_64())) { llsize = int128_rshift(llsize, 1); ret = vhost_vdpa_dma_unmap(s, VHOST_VDPA_GUEST_PA_ASID, iova, - int128_get64(llsize)); + int128_get64(llsize), false); if (ret) { error_report("vhost_vdpa_dma_unmap(%p, 0x%" HWADDR_PRIx ", " @@ -473,7 +474,7 @@ static void vhost_vdpa_listener_region_del(MemoryListener *listener, iova += int128_get64(llsize); } ret = vhost_vdpa_dma_unmap(s, VHOST_VDPA_GUEST_PA_ASID, iova, - int128_get64(llsize)); + int128_get64(llsize), false); if (ret) { error_report("vhost_vdpa_dma_unmap(%p, 0x%" HWADDR_PRIx ", " @@ -1116,7 +1117,7 @@ static void vhost_vdpa_svq_unmap_ring(struct vhost_vdpa *v, hwaddr addr) size = ROUND_UP(result->size, qemu_real_host_page_size()); r = vhost_vdpa_dma_unmap(v->shared, v->address_space_id, result->iova, - size); + size, true); if (unlikely(r < 0)) { error_report("Unable to unmap SVQ vring: %s (%d)", g_strerror(-r), -r); return; @@ -1159,7 +1160,7 @@ static bool vhost_vdpa_svq_map_ring(struct vhost_vdpa *v, DMAMap *needle, r = vhost_vdpa_dma_map(v->shared, v->address_space_id, needle->iova, needle->size + 1, (void *)(uintptr_t)needle->translated_addr, - needle->perm == IOMMU_RO); + needle->perm == IOMMU_RO, true); if (unlikely(r != 0)) { error_setg_errno(errp, -r, "Cannot map region to device"); vhost_iova_tree_remove(v->shared->iova_tree, *needle); diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h index 427458cfed..aac6ad439c 100644 --- a/include/hw/virtio/vhost-vdpa.h +++ b/include/hw/virtio/vhost-vdpa.h @@ -82,9 +82,9 @@ int vhost_vdpa_get_iova_range(int fd, struct vhost_vdpa_iova_range *iova_range); int vhost_vdpa_set_vring_ready(struct vhost_vdpa *v, unsigned idx); int vhost_vdpa_dma_map(VhostVDPAShared *s, uint32_t asid, hwaddr iova, - hwaddr size, void *vaddr, bool readonly); + hwaddr size, void *vaddr, bool readonly, bool shadow); int vhost_vdpa_dma_unmap(VhostVDPAShared *s, uint32_t asid, hwaddr iova, - hwaddr size); + hwaddr size, bool shadow); typedef struct vdpa_iommu { VhostVDPAShared *dev_shared; diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c index e6010e8900..e3e861cfcc 100644 --- a/net/vhost-vdpa.c +++ b/net/vhost-vdpa.c @@ -499,7 +499,7 @@ static void vhost_vdpa_cvq_unmap_buf(struct vhost_vdpa *v, void *addr) } r = vhost_vdpa_dma_unmap(v->shared, v->address_space_id, map->iova, - map->size + 1); + map->size + 1, true); if (unlikely(r != 0)) { error_report("Device cannot unmap: %s(%d)", g_strerror(r), r); } @@ -524,7 +524,8 @@ static int vhost_vdpa_cvq_map_buf(struct vhost_vdpa *v, void *buf, size_t size, } r = vhost_vdpa_dma_map(v->shared, v->address_space_id, map.iova, - vhost_vdpa_net_cvq_cmd_page_len(), buf, !write); + vhost_vdpa_net_cvq_cmd_page_len(), buf, !write, + true); if (unlikely(r < 0)) { goto dma_map_err; } From patchwork Fri Jul 12 14:02:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 1959893 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=oracle.com header.i=@oracle.com header.a=rsa-sha256 header.s=corp-2023-11-20 header.b=JKcBlsmd; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WLCyg68QLz1xpd for ; Sat, 13 Jul 2024 00:04:43 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sSGrA-0001kW-Bp; Fri, 12 Jul 2024 10:02:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sSGr0-0001G6-PW for qemu-devel@nongnu.org; Fri, 12 Jul 2024 10:02:30 -0400 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sSGqx-0005wV-Q7 for qemu-devel@nongnu.org; Fri, 12 Jul 2024 10:02:30 -0400 Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 46CDIfXZ023475; Fri, 12 Jul 2024 14:02:24 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h= from:to:cc:subject:date:message-id:in-reply-to:references; s= corp-2023-11-20; bh=EU/zcw9pA4bSEmPOvY/bmmNwuOR1JofunRmYAaSvopg=; b= JKcBlsmdxvDhr2WoA0IYtiPTaVBycNF6W72guM+IxuzLsneU4ndO1C4/tLLP3X2a k0uSvQW1kZGiABDmKN50/7JJc2eP17jRPIpfujUR9r4bLlgccivVANBA2XkbMfib gKYlzJMOCW4c9uULZInTZNNQIX8ID0GMl+ZA7Ndi8dhQ7/9uaDD2cWYw40q6en96 vrQRrQEi2vKGq2CWk2E6M4TvQOE66Z6GxS5Dy/DzPI09nHtW933gRd3QwPR9jdZN egUVR3iWf4o7mPv2/MdZT85NSW81BjZvZ4FzSWKHovD1OgslGig05cTPABUkNtHL JOgjEDWB/gLBIjUVxOnLNw== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 406wgq41v9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Jul 2024 14:02:24 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 46CDci7r010950; Fri, 12 Jul 2024 14:02:22 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 409vv74k0h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 12 Jul 2024 14:02:22 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 46CE2DQ2020438; Fri, 12 Jul 2024 14:02:22 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 409vv74jq7-8; Fri, 12 Jul 2024 14:02:22 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: "Michael S. Tsirkin" , Jason Wang , Philippe Mathieu-Daude , Eugenio Perez Martin , Peter Xu , Fabiano Rosas , Si-Wei Liu , Steve Sistare Subject: [RFC V1 7/7] vdpa/cpr: preserve dma mappings Date: Fri, 12 Jul 2024 07:02:11 -0700 Message-Id: <1720792931-456433-8-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1720792931-456433-1-git-send-email-steven.sistare@oracle.com> References: <1720792931-456433-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-07-12_10,2024-07-11_01,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 malwarescore=0 phishscore=0 spamscore=0 mlxscore=0 suspectscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2406180000 definitions=main-2407120094 X-Proofpoint-ORIG-GUID: 7XVnE1vpO3qVh9XibFAKGwKVYDS_vpQL X-Proofpoint-GUID: 7XVnE1vpO3qVh9XibFAKGwKVYDS_vpQL Received-SPF: pass client-ip=205.220.165.32; envelope-from=steven.sistare@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Preserve dma mappings during CPR restart by suppressing dma_map and dma_unmap calls. For devices with capability VHOST_BACKEND_F_IOTLB_REMAP, convert dma_map calls to VHOST_IOTLB_REMAP to set the new userland VA for the existing mapping. However, map and unmap shadow vq buffers normally. Their pages are not locked in memory, and they are re-created after CPR. Signed-off-by: Steve Sistare --- hw/virtio/trace-events | 4 ++-- hw/virtio/vhost-vdpa.c | 30 +++++++++++++++++++++++++----- scripts/tracetool/__init__.py | 2 +- 3 files changed, 28 insertions(+), 8 deletions(-) diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events index 990c61be79..30d7f5ec69 100644 --- a/hw/virtio/trace-events +++ b/hw/virtio/trace-events @@ -31,8 +31,8 @@ vhost_user_create_notifier(int idx, void *n) "idx:%d n:%p" # vhost-vdpa.c vhost_vdpa_skipped_memory_section(int is_ram, int is_iommu, int is_protected, int is_ram_device, uint64_t first, uint64_t last, int page_mask) "is_ram=%d, is_iommu=%d, is_protected=%d, is_ram_device=%d iova_min=0x%"PRIx64" iova_last=0x%"PRIx64" page_mask=0x%x" -vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type) "vdpa_shared:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8 -vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint8_t type) "vdpa_shared:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8 +vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type, bool shadow, const char *override) "vdpa_shared:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8" shadow: %d %s" +vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint32_t asid, uint64_t iova, uint64_t size, uint8_t type, bool shadow, const char *override) "vdpa_shared:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8" shadow: %d %s" vhost_vdpa_listener_begin_batch(void *v, int fd, uint32_t msg_type, uint8_t type) "vdpa_shared:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8 vhost_vdpa_listener_commit(void *v, int fd, uint32_t msg_type, uint8_t type) "vdpa_shared:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8 vhost_vdpa_listener_region_add_unaligned(void *v, const char *name, uint64_t offset_as, uint64_t offset_page) "vdpa_shared: %p region %s offset_within_address_space %"PRIu64" offset_within_region %"PRIu64 diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index d9ebc396b7..3ee809abfe 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -22,6 +22,8 @@ #include "hw/virtio/vhost-vdpa.h" #include "exec/address-spaces.h" #include "migration/blocker.h" +#include "migration/cpr.h" +#include "migration/options.h" #include "qemu/cutils.h" #include "qemu/main-loop.h" #include "trace.h" @@ -97,18 +99,29 @@ int vhost_vdpa_dma_map(VhostVDPAShared *s, uint32_t asid, hwaddr iova, struct vhost_msg_v2 msg = {}; int fd = s->device_fd; int ret = 0; + bool remap = false, suppress = false; + + if (migrate_mode() == MIG_MODE_CPR_EXEC && !shadow) { + remap = !!(s->backend_cap & BIT_ULL(VHOST_BACKEND_F_IOTLB_REMAP)); + suppress = !remap; + } msg.type = VHOST_IOTLB_MSG_V2; msg.asid = asid; msg.iotlb.iova = iova; msg.iotlb.size = size; msg.iotlb.uaddr = (uint64_t)(uintptr_t)vaddr; - msg.iotlb.perm = readonly ? VHOST_ACCESS_RO : VHOST_ACCESS_RW; - msg.iotlb.type = VHOST_IOTLB_UPDATE; + msg.iotlb.perm = remap ? 0 : readonly ? VHOST_ACCESS_RO : VHOST_ACCESS_RW; + msg.iotlb.type = remap ? VHOST_IOTLB_REMAP : VHOST_IOTLB_UPDATE; trace_vhost_vdpa_dma_map(s, fd, msg.type, msg.asid, msg.iotlb.iova, msg.iotlb.size, msg.iotlb.uaddr, msg.iotlb.perm, - msg.iotlb.type); + msg.iotlb.type, shadow, + remap ? "(remap)" : suppress ? "(suppress)" : ""); + + if (suppress) { + return 0; + } if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) { error_report("failed to write, fd=%d, errno=%d (%s)", @@ -129,6 +142,7 @@ int vhost_vdpa_dma_unmap(VhostVDPAShared *s, uint32_t asid, hwaddr iova, struct vhost_msg_v2 msg = {}; int fd = s->device_fd; int ret = 0; + bool suppress = migrate_mode() == MIG_MODE_CPR_EXEC && !shadow; msg.type = VHOST_IOTLB_MSG_V2; msg.asid = asid; @@ -137,7 +151,12 @@ int vhost_vdpa_dma_unmap(VhostVDPAShared *s, uint32_t asid, hwaddr iova, msg.iotlb.type = VHOST_IOTLB_INVALIDATE; trace_vhost_vdpa_dma_unmap(s, fd, msg.type, msg.asid, msg.iotlb.iova, - msg.iotlb.size, msg.iotlb.type); + msg.iotlb.size, msg.iotlb.type, shadow, + suppress ? "(suppressed)" : ""); + + if (suppress) { + return 0; + } if (write(fd, &msg, sizeof(msg)) != sizeof(msg)) { error_report("failed to write, fd=%d, errno=%d (%s)", @@ -851,7 +870,8 @@ static int vhost_vdpa_set_backend_cap(struct vhost_dev *dev) 0x1ULL << VHOST_BACKEND_F_IOTLB_BATCH | 0x1ULL << VHOST_BACKEND_F_IOTLB_ASID | 0x1ULL << VHOST_BACKEND_F_SUSPEND | - 0x1ULL << VHOST_BACKEND_F_NEW_OWNER; + 0x1ULL << VHOST_BACKEND_F_NEW_OWNER | + 0x1ULL << VHOST_BACKEND_F_IOTLB_REMAP; int r; if (vhost_vdpa_call(dev, VHOST_GET_BACKEND_FEATURES, &features)) { diff --git a/scripts/tracetool/__init__.py b/scripts/tracetool/__init__.py index bc03238c0f..bfb181cb81 100644 --- a/scripts/tracetool/__init__.py +++ b/scripts/tracetool/__init__.py @@ -253,7 +253,7 @@ def __init__(self, name, props, fmt, args, lineno, filename, orig=None, self.event_trans = event_trans self.event_exec = event_exec - if len(args) > 10: + if len(args) > 11: raise ValueError("Event '%s' has more than maximum permitted " "argument count" % name)