From patchwork Mon Dec 27 10:58:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Haren Myneni X-Patchwork-Id: 1573347 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=MTai1mZq; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4JMvm42jMGz9t1C for ; Mon, 27 Dec 2021 21:58:28 +1100 (AEDT) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4JMvm402VJz2ynK for ; Mon, 27 Dec 2021 21:58:28 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=MTai1mZq; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=haren@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=MTai1mZq; dkim-atps=neutral Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4JMvln3jn8z2xWd for ; Mon, 27 Dec 2021 21:58:13 +1100 (AEDT) Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 1BRAgFU7028115; Mon, 27 Dec 2021 10:58:09 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=message-id : subject : from : to : cc : date : in-reply-to : references : content-type : mime-version : content-transfer-encoding; s=pp1; bh=k/Wr0fgYXRtSSEw6HPcmWAXmplQt4q3EUjqoFfFGUbQ=; b=MTai1mZq3Le5P8TlmRMJFakL9+GIZnLj3SpI9RJSxwPB30uTeMtoRNpbe7iUT2o5thfT +++Vrc7w4mR/ABoRPiKOnu0mW8bkTW1VEXBNcv8WkaO6J4ioqUT3UYBMIwq5CA99ljhm 2jZHcp2gzazBDOtWYuE8EtXSpifDOyFWsG22DQuBxnZIGLoH6Pf0RzXqtVxtZxltMiju pkxjnvLXmxIvCj7A6i/2XpIcjW8FdEc4Od+BVU2WW0MpPtySGNm6JVnOM40xeC0t46H8 d4k7YNMXxzhCMxNGA1ZxaabY7gAM0eS/fsKSed5z88d4NIkN64kxg68EO0jlxeol7DVQ IA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3d7by8078d-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 27 Dec 2021 10:58:09 +0000 Received: from m0098399.ppops.net (m0098399.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 1BRAnQOp016425; Mon, 27 Dec 2021 10:58:08 GMT Received: from ppma03dal.us.ibm.com (b.bd.3ea9.ip4.static.sl-reverse.com [169.62.189.11]) by mx0a-001b2d01.pphosted.com with ESMTP id 3d7by80788-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 27 Dec 2021 10:58:08 +0000 Received: from pps.filterd (ppma03dal.us.ibm.com [127.0.0.1]) by ppma03dal.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 1BRAExqg013669; Mon, 27 Dec 2021 10:58:08 GMT Received: from b03cxnp08025.gho.boulder.ibm.com (b03cxnp08025.gho.boulder.ibm.com [9.17.130.17]) by ppma03dal.us.ibm.com with ESMTP id 3d5tx9p1cw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 27 Dec 2021 10:58:08 +0000 Received: from b03ledav006.gho.boulder.ibm.com (b03ledav006.gho.boulder.ibm.com [9.17.130.237]) by b03cxnp08025.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 1BRAw5A132768368 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 27 Dec 2021 10:58:05 GMT Received: from b03ledav006.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5486CC6069; Mon, 27 Dec 2021 10:58:05 +0000 (GMT) Received: from b03ledav006.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4D8BFC6057; Mon, 27 Dec 2021 10:58:04 +0000 (GMT) Received: from localhost.localdomain (unknown [9.160.90.83]) by b03ledav006.gho.boulder.ibm.com (Postfix) with ESMTP; Mon, 27 Dec 2021 10:58:04 +0000 (GMT) Message-ID: <796dcf1c66fdbef0c7230bc64975a5eec6513d29.camel@linux.ibm.com> Subject: [RFC PATCH 1/3] powerpc/pseries/vas: Modify reconfig open/close functions for migration From: Haren Myneni To: mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, nathanl@linux.ibm.com Date: Mon, 27 Dec 2021 02:58:02 -0800 In-Reply-To: References: User-Agent: Evolution 3.36.5 (3.36.5-2.fc32) MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: eKayUYXyrtSAaNt-AVnVPwLxprx4kDs5 X-Proofpoint-GUID: chteohAJi0Z4ATncnIO2ZhOQI3gri5PO X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.790,Hydra:6.0.425,FMLib:17.11.62.513 definitions=2021-12-27_02,2021-12-24_01,2021-12-02_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 clxscore=1015 spamscore=0 adultscore=0 malwarescore=0 lowpriorityscore=0 phishscore=0 bulkscore=0 priorityscore=1501 suspectscore=0 mlxscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000 definitions=main-2112270052 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" VAS is a hardware engine stays on the chip. So when the partition migrates, all VAS windows on the source system have to be closed and reopen them on the destination after migration. This patch make changes to the current reconfig_open/close_windows functions to support migration: - Sets the window status to VAS_WIN_MIGRATE_CLOSE when closes and reopen windows with the same status during resume. - Continue to close all windows even if deallocate HCALL failed (should not happen) since no way to stop migration with the current LPM implementation. Signed-off-by: Haren Myneni --- arch/powerpc/include/asm/vas.h | 2 + arch/powerpc/platforms/pseries/vas.c | 63 ++++++++++++++++++++++------ 2 files changed, 52 insertions(+), 13 deletions(-) diff --git a/arch/powerpc/include/asm/vas.h b/arch/powerpc/include/asm/vas.h index 72d1df038b4b..8b28cd7aaedc 100644 --- a/arch/powerpc/include/asm/vas.h +++ b/arch/powerpc/include/asm/vas.h @@ -41,6 +41,8 @@ #define VAS_WIN_NO_CRED_CLOSE 0x4 /* Linux specific status when */ /* window is closed due to lost */ /* credit */ +#define VAS_WIN_MIGRATE_CLOSE 0x5 /* Linux status when window is */ + /* closed due to migration */ /* * Get/Set bit fields */ diff --git a/arch/powerpc/platforms/pseries/vas.c b/arch/powerpc/platforms/pseries/vas.c index 169f0cccb166..b9d1c0bac624 100644 --- a/arch/powerpc/platforms/pseries/vas.c +++ b/arch/powerpc/platforms/pseries/vas.c @@ -449,11 +449,12 @@ static int vas_deallocate_window(struct vas_window *vwin) mutex_lock(&vas_pseries_mutex); /* * VAS window is already closed in the hypervisor when - * lost the credit. So just remove the entry from - * the list, remove task references and free vas_window + * lost the credit or with migration. So just remove the entry + * from the list, remove task references and free vas_window * struct. */ - if (win->vas_win.status != VAS_WIN_NO_CRED_CLOSE) { + if ((win->vas_win.status != VAS_WIN_NO_CRED_CLOSE) && + (win->vas_win.status != VAS_WIN_MIGRATE_CLOSE)) { rc = deallocate_free_window(win); if (rc) { mutex_unlock(&vas_pseries_mutex); @@ -570,12 +571,14 @@ static int get_vas_capabilities(u8 feat, enum vas_cop_feat_type type, * by setting the remapping to new paste address if the window is * active. */ -static int reconfig_open_windows(struct vas_caps *vcaps, int creds) +static int reconfig_open_windows(struct vas_caps *vcaps, int creds, + bool migrate) { long domain[PLPAR_HCALL9_BUFSIZE] = {VAS_DEFAULT_DOMAIN_ID}; struct vas_cop_feat_caps *caps = &vcaps->caps; struct pseries_vas_window *win = NULL, *tmp; int rc, mv_ents = 0; + int status; /* * Nothing to do if there are no closed windows. @@ -594,8 +597,10 @@ static int reconfig_open_windows(struct vas_caps *vcaps, int creds) * (dedicated). If 1 core is added, this LPAR can have 20 more * credits. It means the kernel can reopen 20 windows. So move * 20 entries in the VAS windows lost and reopen next 20 windows. + * For partition migration, reopen all windows that are closed + * during resume. */ - if (vcaps->close_wins > creds) + if ((vcaps->close_wins > creds) && !migrate) mv_ents = vcaps->close_wins - creds; list_for_each_entry_safe(win, tmp, &vcaps->list, win_list) { @@ -605,11 +610,20 @@ static int reconfig_open_windows(struct vas_caps *vcaps, int creds) mv_ents--; } + /* + * Open windows if they are closed only with migration or + * DLPAR (lost credit) before. + */ + if (migrate) + status = VAS_WIN_MIGRATE_CLOSE; + else + status = VAS_WIN_NO_CRED_CLOSE; + list_for_each_entry_safe_from(win, tmp, &vcaps->list, win_list) { /* * Nothing to do on this window if it is active. */ - if (win->vas_win.status != VAS_WIN_NO_CRED_CLOSE) + if (win->vas_win.status != status) continue; rc = allocate_setup_window(win, (u64 *)&domain[0], @@ -652,17 +666,26 @@ static int reconfig_open_windows(struct vas_caps *vcaps, int creds) * the user space to fall back to SW compression and manage with the * existing windows. */ -static int reconfig_close_windows(struct vas_caps *vcap, int excess_creds) +static int reconfig_close_windows(struct vas_caps *vcap, int excess_creds, + bool migrate) { struct pseries_vas_window *win, *tmp; struct vas_user_win_ref *task_ref; struct vm_area_struct *vma; - int rc = 0; + int rc = 0, status; + + if (migrate) + status = VAS_WIN_MIGRATE_CLOSE; + else + status = VAS_WIN_NO_CRED_CLOSE; list_for_each_entry_safe(win, tmp, &vcap->list, win_list) { /* * This window is already closed due to lost credit * before. Go for next window. + * For migration, nothing to do since this window + * closed for DLPAR and will be reopened even on + * the destination system with other DLPAR operation. */ if (win->vas_win.status == VAS_WIN_NO_CRED_CLOSE) continue; @@ -674,7 +697,7 @@ static int reconfig_close_windows(struct vas_caps *vcap, int excess_creds) * Number of available credits are reduced, So select * and close windows. */ - win->vas_win.status = VAS_WIN_NO_CRED_CLOSE; + win->vas_win.status = status; mmap_write_lock(task_ref->mm); /* @@ -697,12 +720,24 @@ static int reconfig_close_windows(struct vas_caps *vcap, int excess_creds) * later when the process issued with close(FD). */ rc = deallocate_free_window(win); - if (rc) + /* + * This failure is from the hypervisor. + * No way to stop migration for these failures. + * So ignore error and continue closing other windows. + */ + if (rc && !migrate) return rc; vcap->close_wins++; - if (!--excess_creds) + /* + * For migration, do not depend on lpar_creds in case if + * mismatch with the hypervisor value (should not happen). + * So close all active windows in the list and will be + * reopened windows based on the new lpar_creds on the + * destination system during resume. + */ + if (!migrate && !--excess_creds) break; } @@ -757,7 +792,8 @@ int vas_reconfig_capabilties(u8 type) * target, reopen windows if they are closed due to * the previous DLPAR (core removal). */ - rc = reconfig_open_windows(vcaps, new_creds - lpar_creds); + rc = reconfig_open_windows(vcaps, new_creds - lpar_creds, + false); } else { /* * # active windows is more than new LPAR available @@ -765,7 +801,8 @@ int vas_reconfig_capabilties(u8 type) */ active_wins = vcaps->num_wins - vcaps->close_wins; if (active_wins > new_creds) - rc = reconfig_close_windows(vcaps, active_wins - new_creds); + rc = reconfig_close_windows(vcaps, active_wins - new_creds, + false); } out: From patchwork Mon Dec 27 10:59:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Haren Myneni X-Patchwork-Id: 1573348 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=UBR96HOF; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4JMvnh6lfQz9t1C for ; Mon, 27 Dec 2021 21:59:52 +1100 (AEDT) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4JMvnh4mMRz3bWk for ; Mon, 27 Dec 2021 21:59:52 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=UBR96HOF; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=haren@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=UBR96HOF; dkim-atps=neutral Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4JMvnP4R3lz2x9T for ; Mon, 27 Dec 2021 21:59:37 +1100 (AEDT) Received: from pps.filterd (m0098416.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 1BR9BjVi017410; Mon, 27 Dec 2021 10:59:31 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=message-id : subject : from : to : cc : date : in-reply-to : references : content-type : mime-version : content-transfer-encoding; s=pp1; bh=M4vyWnZTevfr+j9xbdYCjSvZQyyzXBjhKEGqEbiw9lI=; b=UBR96HOFyNbGizNNM7O+OVsfHYeVLWhFe31mUjFDGyVrAFX0Oya4irnkD1t5udjotyBT /eratAHE+IGY9JvEYDA3PGlfv3FLnBHKbrVRf+3zTYC3wEV1DuvJaP+BIbSrVJPbGQcx 1pUHYfZ31jIkXG947cC5jj8/WWdjeUboInIRmh1/nbqA67UFQQHdnQGfpAy+TBXCXO4g s6C0LbY32WgbrEWISYpo/clV8Cryj4lVKEk1f9xXoP6SYOEozx1MdVDwpxlurYBnPdO9 4PCdbujWqOVcQNJ4MOzemJGGy88/RdBz5HxgUH2l2hfHu0HVNaP+ZnIciUNRrQzfmOsO 3g== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 3d7amx9ku0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 27 Dec 2021 10:59:31 +0000 Received: from m0098416.ppops.net (m0098416.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 1BRAgww2032196; Mon, 27 Dec 2021 10:59:31 GMT Received: from ppma04dal.us.ibm.com (7a.29.35a9.ip4.static.sl-reverse.com [169.53.41.122]) by mx0b-001b2d01.pphosted.com with ESMTP id 3d7amx9ktn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 27 Dec 2021 10:59:30 +0000 Received: from pps.filterd (ppma04dal.us.ibm.com [127.0.0.1]) by ppma04dal.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 1BRAkPnM009421; Mon, 27 Dec 2021 10:59:30 GMT Received: from b01cxnp23032.gho.pok.ibm.com (b01cxnp23032.gho.pok.ibm.com [9.57.198.27]) by ppma04dal.us.ibm.com with ESMTP id 3d5tx9e0y4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 27 Dec 2021 10:59:30 +0000 Received: from b01ledav005.gho.pok.ibm.com (b01ledav005.gho.pok.ibm.com [9.57.199.110]) by b01cxnp23032.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 1BRAxQLs28311940 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 27 Dec 2021 10:59:26 GMT Received: from b01ledav005.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5E93AAE05C; Mon, 27 Dec 2021 10:59:26 +0000 (GMT) Received: from b01ledav005.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id ED206AE05F; Mon, 27 Dec 2021 10:59:24 +0000 (GMT) Received: from localhost.localdomain (unknown [9.160.90.83]) by b01ledav005.gho.pok.ibm.com (Postfix) with ESMTP; Mon, 27 Dec 2021 10:59:24 +0000 (GMT) Message-ID: <340462fe25b26377130b3204eca20e71ff0d80b0.camel@linux.ibm.com> Subject: [RFC PATCH 2/3] powerpc/pseries/vas: Add VAS suspend/resume notifier From: Haren Myneni To: mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, nathanl@linux.ibm.com Date: Mon, 27 Dec 2021 02:59:22 -0800 In-Reply-To: References: User-Agent: Evolution 3.36.5 (3.36.5-2.fc32) MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: XrX8rt-bvI6hvfI-qxS2MxNv6y6wHsf2 X-Proofpoint-GUID: YdKFje3EHX76mrmsyK49VO4RzYKq2_h9 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.790,Hydra:6.0.425,FMLib:17.11.62.513 definitions=2021-12-27_02,2021-12-24_01,2021-12-02_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 lowpriorityscore=0 bulkscore=0 phishscore=0 spamscore=0 mlxlogscore=999 suspectscore=0 priorityscore=1501 clxscore=1015 adultscore=0 malwarescore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000 definitions=main-2112270052 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Since VAS windows belong to the VAS hardware resource, the hypervisor expects the partition to close them on source partition and reopen them after the partition migrated on the destination machine. This suspend/resume notifier invokes suspend operation before and resume operation after migration. All active windows for both default and QoS types will be closed during suspend and reopen them during resume. During migration, the user space should expect paste instruction failure if issues copy/paste on these active windows. Signed-off-by: Haren Myneni --- arch/powerpc/platforms/pseries/vas.c | 97 +++++++++++++++++++++++++++- 1 file changed, 96 insertions(+), 1 deletion(-) diff --git a/arch/powerpc/platforms/pseries/vas.c b/arch/powerpc/platforms/pseries/vas.c index b9d1c0bac624..d2d7202a4f4e 100644 --- a/arch/powerpc/platforms/pseries/vas.c +++ b/arch/powerpc/platforms/pseries/vas.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include "vas.h" @@ -845,6 +846,98 @@ static struct notifier_block pseries_vas_nb = { .notifier_call = pseries_vas_notifier, }; +/* + * For LPM, all windows have to be closed on the source partition + * before migration and reopen them on the destination partition + * after migration. So closing windows during suspend and + * reopen them during resume. + */ +static int vas_migrate_windows(bool suspend) +{ + struct hv_vas_cop_feat_caps *hv_caps; + struct vas_cop_feat_caps *caps; + int lpar_creds, new_creds = 0; + struct vas_caps *vcaps; + int i, rc = 0; + + hv_caps = kmalloc(sizeof(*hv_caps), GFP_KERNEL); + if (!hv_caps) + return -ENOMEM; + + mutex_lock(&vas_pseries_mutex); + + for (i = 0; i < VAS_MAX_FEAT_TYPE; i++) { + vcaps = &vascaps[i]; + caps = &vcaps->caps; + lpar_creds = atomic_read(&caps->target_creds); + + rc = h_query_vas_capabilities(H_QUERY_VAS_CAPABILITIES, vcaps->feat, + (u64)virt_to_phys(hv_caps)); + if (!rc) { + new_creds = be16_to_cpu(hv_caps->target_lpar_creds); + /* + * Should not happen. But incase print messages, close all + * windows in the list during suspend and reopen windows + * based on new lpar_creds on the destination system. + */ + if (lpar_creds != new_creds) { + pr_err("%s: lpar creds: %d HV lpar creds: %d\n", + suspend ? "Suspend" : "Resume", lpar_creds, + new_creds); + pr_err("Used creds: %d, Active creds: %d\n", + atomic_read(&caps->used_creds), + vcaps->num_wins - vcaps->close_wins); + } + } else { + pr_err("%s: Get VAS capabilities failed with %d\n", + suspend ? "Suspend" : "Resume", rc); + /* + * We can not stop migration with the current lpm + * implementation. So continue closing all windows in the + * list (during suspend) and return without opending windows + * (during resume) if VAS capabilities HCALL failed. + */ + if (!suspend) + goto out; + } + + if (suspend) + rc = reconfig_close_windows(vcaps, vcaps->num_wins, true); + else { + atomic_set(&caps->target_creds, new_creds); + rc = reconfig_open_windows(vcaps, new_creds, true); + } + + /* + * Ignore errors during suspend and return for resume. + */ + if (rc && !suspend) + goto out; + } + +out: + mutex_unlock(&vas_pseries_mutex); + kfree(hv_caps); + return rc; +} + +static int vas_migration_handler(struct notifier_block *nb, + unsigned long action, void *data) +{ + if (action == PSERIES_RESUMING) + return vas_migrate_windows(false); + else + return vas_migrate_windows(true); + +} + +static struct pseries_suspend_handler vas_suspend_handler = { + .notifier_block = { + .notifier_call = vas_migration_handler, + }, +}; + + static int __init pseries_vas_init(void) { struct hv_vas_cop_feat_caps *hv_cop_caps; @@ -901,8 +994,10 @@ static int __init pseries_vas_init(void) } /* Processors can be added/removed only on LPAR */ - if (copypaste_feat && firmware_has_feature(FW_FEATURE_LPAR)) + if (copypaste_feat && firmware_has_feature(FW_FEATURE_LPAR)) { of_reconfig_notifier_register(&pseries_vas_nb); + pseries_register_suspend_handler(&vas_suspend_handler); + } pr_info("GZIP feature is available\n"); From patchwork Mon Dec 27 11:00:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Haren Myneni X-Patchwork-Id: 1573349 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=S4Epz44B; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4JMvpQ5tlsz9t1C for ; Mon, 27 Dec 2021 22:00:30 +1100 (AEDT) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4JMvpQ4Jw6z3c9Y for ; Mon, 27 Dec 2021 22:00:30 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=S4Epz44B; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=haren@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=S4Epz44B; dkim-atps=neutral Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4JMvp36GzWz3c65 for ; Mon, 27 Dec 2021 22:00:11 +1100 (AEDT) Received: from pps.filterd (m0187473.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.1.2/8.16.1.2) with SMTP id 1BRAfnwW029149; Mon, 27 Dec 2021 11:00:07 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=message-id : subject : from : to : cc : date : in-reply-to : references : content-type : mime-version : content-transfer-encoding; s=pp1; bh=FtuDLXOr6o0yKLtf7bVV0x2BIWOTK2wTt3NVrMo9Hss=; b=S4Epz44BcAx6mZkgNvVKBeuGhHzBhpQ+shq+Y/S98Sr21HpTXETKCoUZGYziZecZtnLF fCvbHbhoo7nSh019JKabEBIt2mbyJh3GnXV7lgMIHR1S2kjeyv0aQeYFpIuwkviTw0Lx SfN60IvpjJ32Azo5B+O5x0H45fOlQ6VU/S9f/tAB51+mK+ruHQ/tpHIpac91h0sCSVk4 hnSpA2xzWmBnqArb9CCwQnqNxP2Y83TNqw5pBcf8i5V8LV0YBljljVfQrIORSYO1Up/D E0CgAgh1Lu2sgXMyEOdssQs8p47gA7ujIkROEFNZWds/PBs6CaBchwUWPZAY7+tOXYID 0Q== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 3d7by6g879-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 27 Dec 2021 11:00:07 +0000 Received: from m0187473.ppops.net (m0187473.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.43/8.16.0.43) with SMTP id 1BRAiGk5034260; Mon, 27 Dec 2021 11:00:07 GMT Received: from ppma05wdc.us.ibm.com (1b.90.2fa9.ip4.static.sl-reverse.com [169.47.144.27]) by mx0a-001b2d01.pphosted.com with ESMTP id 3d7by6g86a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 27 Dec 2021 11:00:07 +0000 Received: from pps.filterd (ppma05wdc.us.ibm.com [127.0.0.1]) by ppma05wdc.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 1BRAE27l015737; Mon, 27 Dec 2021 11:00:06 GMT Received: from b03cxnp08025.gho.boulder.ibm.com (b03cxnp08025.gho.boulder.ibm.com [9.17.130.17]) by ppma05wdc.us.ibm.com with ESMTP id 3d5tx9r4tx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 27 Dec 2021 11:00:06 +0000 Received: from b03ledav002.gho.boulder.ibm.com (b03ledav002.gho.boulder.ibm.com [9.17.130.233]) by b03cxnp08025.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 1BRB03Q230277932 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 27 Dec 2021 11:00:03 GMT Received: from b03ledav002.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D8AD6136069; Mon, 27 Dec 2021 11:00:03 +0000 (GMT) Received: from b03ledav002.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E1EA313606E; Mon, 27 Dec 2021 11:00:02 +0000 (GMT) Received: from localhost.localdomain (unknown [9.160.90.83]) by b03ledav002.gho.boulder.ibm.com (Postfix) with ESMTP; Mon, 27 Dec 2021 11:00:02 +0000 (GMT) Message-ID: <92d87ead72556e946d7fa6775c509de8b6d11935.camel@linux.ibm.com> Subject: [RFC PATCH 3/3] powerpc/pseries/vas: Use migration_in_progress to disable DLPAR From: Haren Myneni To: mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, npiggin@gmail.com, nathanl@linux.ibm.com Date: Mon, 27 Dec 2021 03:00:01 -0800 In-Reply-To: References: User-Agent: Evolution 3.36.5 (3.36.5-2.fc32) MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: pamOVEMrN9X6m1YRHexC8nC5bFABXL0f X-Proofpoint-GUID: CHrGbM_bUzSoWNe8f5OSb1dkva5GCasQ X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.790,Hydra:6.0.425,FMLib:17.11.62.513 definitions=2021-12-27_02,2021-12-24_01,2021-12-02_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 clxscore=1015 mlxlogscore=999 priorityscore=1501 suspectscore=0 phishscore=0 impostorscore=0 mlxscore=0 spamscore=0 malwarescore=0 lowpriorityscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2110150000 definitions=main-2112270052 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Before migration starts, all secondary CPUs will be offline which can invoke VAS DLPAR event. So disable VAS DLPAR event with migration_in_progress flag during suspend and enable after resume operations. The current partition migration implementation does not freeze the user space and the user space can continue open VAS windows. So when migration_in_progress flag is enabled, VAS open window API returns -EBUSY. Signed-off-by: Haren Myneni --- arch/powerpc/platforms/pseries/vas.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/arch/powerpc/platforms/pseries/vas.c b/arch/powerpc/platforms/pseries/vas.c index d2d7202a4f4e..1a10c1904aaa 100644 --- a/arch/powerpc/platforms/pseries/vas.c +++ b/arch/powerpc/platforms/pseries/vas.c @@ -30,6 +30,7 @@ static bool copypaste_feat; static struct vas_caps vascaps[VAS_MAX_FEAT_TYPE]; static DEFINE_MUTEX(vas_pseries_mutex); +static bool migration_in_progress; static long hcall_return_busy_check(long rc) { @@ -349,8 +350,11 @@ static struct vas_window *vas_allocate_window(int vas_id, u64 flags, * same fault IRQ is not freed by the OS before. */ mutex_lock(&vas_pseries_mutex); - rc = allocate_setup_window(txwin, (u64 *)&domain[0], - cop_feat_caps->win_type); + if (migration_in_progress) + rc = -EBUSY; + else + rc = allocate_setup_window(txwin, (u64 *)&domain[0], + cop_feat_caps->win_type); mutex_unlock(&vas_pseries_mutex); if (rc) goto out; @@ -771,6 +775,9 @@ int vas_reconfig_capabilties(u8 type) return -ENOMEM; mutex_lock(&vas_pseries_mutex); + if (migration_in_progress) + goto out; + rc = h_query_vas_capabilities(H_QUERY_VAS_CAPABILITIES, vcaps->feat, (u64)virt_to_phys(hv_caps)); if (rc) @@ -866,6 +873,11 @@ static int vas_migrate_windows(bool suspend) mutex_lock(&vas_pseries_mutex); + if (suspend) + migration_in_progress = true; + else + migration_in_progress = false; + for (i = 0; i < VAS_MAX_FEAT_TYPE; i++) { vcaps = &vascaps[i]; caps = &vcaps->caps;