From patchwork Thu Jul 19 07:54:59 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vaibhav Jain X-Patchwork-Id: 946090 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41WRBv1Nprz9s3x for ; Thu, 19 Jul 2018 17:55:19 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=linux.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 41WRBv0BxpzDqCF for ; Thu, 19 Jul 2018 17:55:19 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=linux.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=linux.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=vaibhav@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=linux.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 41WRBn2wMZzDq6t for ; Thu, 19 Jul 2018 17:55:13 +1000 (AEST) Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w6J7o5bU092783 for ; Thu, 19 Jul 2018 03:55:11 -0400 Received: from e06smtp02.uk.ibm.com (e06smtp02.uk.ibm.com [195.75.94.98]) by mx0a-001b2d01.pphosted.com with ESMTP id 2kakgw7cne-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 19 Jul 2018 03:55:11 -0400 Received: from localhost by e06smtp02.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 19 Jul 2018 08:55:09 +0100 Received: from b06cxnps4075.portsmouth.uk.ibm.com (9.149.109.197) by e06smtp02.uk.ibm.com (192.168.101.132) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Thu, 19 Jul 2018 08:55:08 +0100 Received: from d06av25.portsmouth.uk.ibm.com (d06av25.portsmouth.uk.ibm.com [9.149.105.61]) by b06cxnps4075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w6J7t6hY35848276 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Thu, 19 Jul 2018 07:55:06 GMT Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6AF1611C05B; Thu, 19 Jul 2018 10:55:23 +0100 (BST) Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4534F11C04A; Thu, 19 Jul 2018 10:55:21 +0100 (BST) Received: from vajain21.in.ibm.com (unknown [9.109.223.189]) by d06av25.portsmouth.uk.ibm.com (Postfix) with ESMTP; Thu, 19 Jul 2018 10:55:21 +0100 (BST) From: Vaibhav Jain To: Frederic Barrat , Andrew Donnellan , skiboot@lists.ozlabs.org Date: Thu, 19 Jul 2018 13:24:59 +0530 X-Mailer: git-send-email 2.17.1 X-TM-AS-GCONF: 00 x-cbid: 18071907-0008-0000-0000-00000254F1EE X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18071907-0009-0000-0000-000021BB4BCC Message-Id: <20180719075500.17822-1-vaibhav@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-07-19_03:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1806210000 definitions=main-1807190085 Subject: [Skiboot] [PATCH 1/2] phb4: Reallocate PEC2 DMA-Read engines to improve GPU-Direct bandwidth X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Philippe Bergheaud , Christophe Lombard MIME-Version: 1.0 Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" We reallocate additional 16/8 DMA-Read engines allocated to stack0/1 on PEC2 respectively. This is needed to improve bandwidth available to the Mellanox CX5 adapter when trying to read GPU memory (GPU-Direct). If kernel cxl driver indicates a request to allocate maximum possible DMA read engines when calling enable_capi_mode() and card is attached to PEC2/stack0 slot then we assume its a Mellanox CX5 adapter. We then allocate additional 16/8 extra DMA read engines to stack0 and stack1 respectively on PEC2. This is done by populating the XPEC_PCI_PRDSTKOVR and XPEC_NEST_READ_STACK_OVERRIDE as suggested by the h/w team. Signed-off-by: Christophe Lombard Signed-off-by: Vaibhav Jain Reviewed-by: Andrew Donnellan --- hw/phb4.c | 40 +++++++++++++++++++++++++++++++++++++--- include/phb4-regs.h | 2 ++ 2 files changed, 39 insertions(+), 3 deletions(-) diff --git a/hw/phb4.c b/hw/phb4.c index 8ba3eb78..f2b92409 100644 --- a/hw/phb4.c +++ b/hw/phb4.c @@ -3806,7 +3806,7 @@ static void phb4_init_capp_regs(struct phb4 *p, uint32_t capp_eng) /* max PHB read buffers 0-47 */ reg = 0xFFFFFFFFFFFF0000; if (capp_eng & CAPP_MAX_DMA_READ_ENGINES) - reg = 0xFF00000000000000; + reg = 0xF000000000000000; xscom_write(p->chip_id, APC_FSM_READ_MASK + offset, reg); xscom_write(p->chip_id, XPT_FSM_RMM + offset, reg); } @@ -3814,7 +3814,7 @@ static void phb4_init_capp_regs(struct phb4 *p, uint32_t capp_eng) /* Set 30 Read machines for CAPP Minus 20-27 for DMA */ reg = 0xFFFFF00E00000000; if (capp_eng & CAPP_MAX_DMA_READ_ENGINES) - reg = 0xFF00000000000000; + reg = 0xF000000000000000; xscom_write(p->chip_id, APC_FSM_READ_MASK + offset, reg); xscom_write(p->chip_id, XPT_FSM_RMM + offset, reg); } @@ -3932,6 +3932,8 @@ static int64_t enable_capi_mode(struct phb4 *p, uint64_t pe_number, return OPAL_HARDWARE; } + stq_eng = 0x0000000000000000ULL; + dma_eng = 0x0000000000000000ULL; if (p->index == CAPP0_PHB_INDEX) { /* PBCQ is operating as a x16 stack * - The maximum number of engines give to CAPP will be @@ -3974,10 +3976,42 @@ static int64_t enable_capi_mode(struct phb4 *p, uint64_t pe_number, reg = 0x8000000000000000ULL; /* PEC works in CAPP Mode */ reg |= stq_eng; if (capp_eng & CAPP_MAX_DMA_READ_ENGINES) - dma_eng = 0x0000FF0000000000ULL; /* 16 CAPP Read machines */ + dma_eng = 0x0000F00000000000ULL; /* 4 CAPP Read machines */ reg |= dma_eng; xscom_write(p->chip_id, p->pe_xscom + XPEC_NEST_CAPP_CNTL, reg); + /* PEC2 has 3 ETU's + 16 pci lanes that can operate as x16, + * x8+x8 (bifurcated) or x8+x4+x4 (trifurcated) mode. When + * Mellanox CX5 card is attached to stack0 of this PEC, indicated by + * request to allocate CAPP_MAX_DMA_READ_ENGINES; we tweak the default + * dma-read engines allocations to maximize the DMA read performance + */ + if ((p->index == CAPP1_PHB_INDEX) && + (capp_eng & CAPP_MAX_DMA_READ_ENGINES)) { + + /* + * Allocate Additional 16/8 dma read engines to stack0/stack1 + * respectively. Read engines 0:31 are anyways always assigned + * to stack0. Also skip allocating DMA Read Engine-32 by + * enabling Bit[0] in XPEC_NEST_READ_STACK_OVERRIDE register. + * Enabling this bit seems cause a parity error reported in + * NFIR[1]-nonbar_pe. + */ + reg = 0x7fff80007F008000ULL; + + xscom_write(p->chip_id, p->pci_xscom + XPEC_PCI_PRDSTKOVR, reg); + xscom_write(p->chip_id, p->pe_xscom + + XPEC_NEST_READ_STACK_OVERRIDE, reg); + + /* Log this reallocation as it may impact dma performance of + * other slots connected to PEC2 + */ + PHBINF(p, "CAPP: Set %d dma-read engines for PEC2/stack-0\n", + 32 + __builtin_popcountll(reg & PPC_BITMASK(0, 31))); + PHBDBG(p, "CAPP: XPEC_NEST_READ_STACK_OVERRIDE: %016llx\n", + reg); + } + /* PCI to PB data movement ignores the PB init signal. */ xscom_write_mask(p->chip_id, p->pe_xscom + XPEC_NEST_PBCQ_HW_CONFIG, XPEC_NEST_PBCQ_HW_CONFIG_PBINIT, diff --git a/include/phb4-regs.h b/include/phb4-regs.h index d7b551f3..ef3cfa93 100644 --- a/include/phb4-regs.h +++ b/include/phb4-regs.h @@ -347,6 +347,7 @@ #define XPEC_NEST_PBCQ_HW_CONFIG_DIS_NODAL PPC_BIT(50) #define XPEC_NEST_PBCQ_HW_CONFIG_DIS_RNNN PPC_BIT(52) #define XPEC_NEST_CAPP_CNTL 0x7 +#define XPEC_NEST_READ_STACK_OVERRIDE 0x8 /* Nest base per-stack registers */ #define XPEC_NEST_STK_PCI_NFIR 0x0 @@ -381,6 +382,7 @@ /* PCI base registers */ #define XPEC_PCI_PBAIB_HW_CONFIG 0x0 #define XPEC_PCI_CAPP_SEC_BAR 0x1 +#define XPEC_PCI_PRDSTKOVR 0x2 /* PCI base per-stack registers */ #define XPEC_PCI_STK_PCI_FIR 0x0 From patchwork Thu Jul 19 07:55:00 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vaibhav Jain X-Patchwork-Id: 946091 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41WRCN5ZT7z9s1R for ; Thu, 19 Jul 2018 17:55:44 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=linux.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 41WRCN3lxhzDqDJ for ; Thu, 19 Jul 2018 17:55:44 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=linux.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=linux.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=vaibhav@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=linux.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 41WRBv3FknzDqCF for ; Thu, 19 Jul 2018 17:55:19 +1000 (AEST) Received: from pps.filterd (m0098413.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w6J7nBLt104860 for ; Thu, 19 Jul 2018 03:55:17 -0400 Received: from e06smtp07.uk.ibm.com (e06smtp07.uk.ibm.com [195.75.94.103]) by mx0b-001b2d01.pphosted.com with ESMTP id 2kap2qhggy-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 19 Jul 2018 03:55:16 -0400 Received: from localhost by e06smtp07.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 19 Jul 2018 08:55:15 +0100 Received: from b06cxnps4074.portsmouth.uk.ibm.com (9.149.109.196) by e06smtp07.uk.ibm.com (192.168.101.137) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Thu, 19 Jul 2018 08:55:13 +0100 Received: from d06av25.portsmouth.uk.ibm.com (d06av25.portsmouth.uk.ibm.com [9.149.105.61]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w6J7tCRb28573916 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Thu, 19 Jul 2018 07:55:12 GMT Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1CCEA11C069; Thu, 19 Jul 2018 10:55:29 +0100 (BST) Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 936E911C04C; Thu, 19 Jul 2018 10:55:26 +0100 (BST) Received: from vajain21.in.ibm.com (unknown [9.109.223.189]) by d06av25.portsmouth.uk.ibm.com (Postfix) with ESMTP; Thu, 19 Jul 2018 10:55:26 +0100 (BST) From: Vaibhav Jain To: Frederic Barrat , Andrew Donnellan , skiboot@lists.ozlabs.org Date: Thu, 19 Jul 2018 13:25:00 +0530 X-Mailer: git-send-email 2.17.1 In-Reply-To: <20180719075500.17822-1-vaibhav@linux.ibm.com> References: <20180719075500.17822-1-vaibhav@linux.ibm.com> X-TM-AS-GCONF: 00 x-cbid: 18071907-0028-0000-0000-000002DD71E8 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18071907-0029-0000-0000-000023953E71 Message-Id: <20180719075500.17822-2-vaibhav@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-07-19_03:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1806210000 definitions=main-1807190085 Subject: [Skiboot] [PATCH 2/2] doc: Add a man page for OPAL_PCI_SET_PHB_CAPI_MODE X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Philippe Bergheaud , Christophe Lombard MIME-Version: 1.0 Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" We add a man page describing the opal call OPAL_PCI_SET_PHB_CAPI_MODE used for activating/deactivating CAPP attached to a PEC for CAPI 1 & 2. Signed-off-by: Vaibhav Jain Reviewed-by: Andrew Donnellan --- .../opal-pci-set-phb-capi-mode-93.rst | 74 +++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 doc/opal-api/opal-pci-set-phb-capi-mode-93.rst diff --git a/doc/opal-api/opal-pci-set-phb-capi-mode-93.rst b/doc/opal-api/opal-pci-set-phb-capi-mode-93.rst new file mode 100644 index 00000000..c0c7bd8d --- /dev/null +++ b/doc/opal-api/opal-pci-set-phb-capi-mode-93.rst @@ -0,0 +1,74 @@ +OPAL_PCI_SET_PHB_CAPI_MODE +=========================== + +Switch the CAPP attached to the given PHB in one of the supported CAPI modes + +Parameters +---------- +``uint64_t phb_id`` + the ID of the PHB which identifies attached CAPP to perform mode switch on + +``uint64_t mode`` + A mode id as described below + +``pe_number`` + PE number for the initiating device + +Calling +------- + +Switch CAPP attached to the given PHB in one of the following supported modes: + +OPAL_PHB_CAPI_MODE_PCIE = 0 +OPAL_PHB_CAPI_MODE_CAPI = 1 +OPAL_PHB_CAPI_MODE_SNOOP_OFF = 2 +OPAL_PHB_CAPI_MODE_SNOOP_ON = 3 +OPAL_PHB_CAPI_MODE_DMA = 4 +OPAL_PHB_CAPI_MODE_DMA_TVT1 = 5 + +Modes `OPAL_PHB_CAPI_MODE_PCIE` and `OPAL_PHB_CAPI_MODE_CAPI` are used to +enable/disable CAPP attached to the PHB. + +Modes `OPAL_PHB_CAPI_MODE_SNOOP`_OFF and `OPAL_PHB_CAPI_MODE_SNOOP_ON` are +used to enable/disable CAPP snooping of Powerbus traffic for cache line +invalidates. + +Mode `OPAL_PHB_CAPI_MODE_DMA` and `OPAL_PHB_CAPI_MODE_DMA_TVT1` are used to +enable CAPP DMA mode. + +Presently Mode `OPAL_PHB_CAPI_MODE_DMA_TVT1` is exclusively used by the Mellanox +CX5 adapter. Requesting this mode will also indicate to opal that the card +requests maximum number of DMA read engines allocated to improve DMA read +performance at cost of reduced bandwidth available to other traffic including +CAPP-PSL transactions. + +Notes: +----- +* If PHB is in PEC2 then requesting mode `OPAL_PHB_CAPI_MODE_DMA_TVT1` will + allocate extra 16/8 dma read engines to the PHB depending on its stack + (stack 0/ stack 1). This is needed to improve the Direct-GPU DMA read + performance for the Mellanox CX5 card. +* Mode `OPAL_PHB_CAPI_MODE_PCIE` not yet supported on Power-9. +* Requesting mode `OPAL_PHB_CAPI_MODE_CAPI` on Power-9 will disable fast-reboot. +* Modes `OPAL_PHB_CAPI_MODE_DMA`, `OPAL_PHB_CAPI_MODE_SNOOP_OFF` are + not supported on Power-9 yet. + +Return Codes +------------ +OPAL_SUCCESS + Switch to the reqeuested capi mode performed successfully. + +OPAL_PARAMETER + The requested value of mode or phb_id parameter is not valid. + +OPAL_HARDWARE + An error occurred while switching the CAPP to requested mode. + +OPAL_UNSUPPORTED + Switching to requested capi mode is not possible at the moment + +OPAL_RESOURCE + CAPP ucode not available hence activating CAPP not supported. + +OPAL_BUSY + CAPP is presently in recovery-mode and mode switch cannot be performed.