From patchwork Tue Jul 9 15:05:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Barrat X-Patchwork-Id: 1129877 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45jlyX6yTwz9s8m for ; Wed, 10 Jul 2019 01:06:28 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=linux.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 45jlyX62jFzDqY6 for ; Wed, 10 Jul 2019 01:06:28 +1000 (AEST) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=linux.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=fbarrat@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=linux.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 45jlyB0XH5zDqT2 for ; Wed, 10 Jul 2019 01:06:09 +1000 (AEST) Received: from pps.filterd (m0098393.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x69EwDj8115496 for ; Tue, 9 Jul 2019 11:06:07 -0400 Received: from e06smtp03.uk.ibm.com (e06smtp03.uk.ibm.com [195.75.94.99]) by mx0a-001b2d01.pphosted.com with ESMTP id 2tmtvq7339-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 09 Jul 2019 11:06:05 -0400 Received: from localhost by e06smtp03.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 9 Jul 2019 16:06:02 +0100 Received: from b06cxnps4075.portsmouth.uk.ibm.com (9.149.109.197) by e06smtp03.uk.ibm.com (192.168.101.133) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 9 Jul 2019 16:06:00 +0100 Received: from d06av25.portsmouth.uk.ibm.com (d06av25.portsmouth.uk.ibm.com [9.149.105.61]) by b06cxnps4075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x69F5xVq38994098 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 9 Jul 2019 15:05:59 GMT Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1C2B911C052; Tue, 9 Jul 2019 15:05:59 +0000 (GMT) Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id CA67711C04A; Tue, 9 Jul 2019 15:05:58 +0000 (GMT) Received: from pic2.home (unknown [9.145.69.11]) by d06av25.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 9 Jul 2019 15:05:58 +0000 (GMT) From: Frederic Barrat To: skiboot@lists.ozlabs.org, andrew.donnellan@au1.ibm.com, clombard@linux.ibm.com Date: Tue, 9 Jul 2019 17:05:57 +0200 X-Mailer: git-send-email 2.21.0 MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19070915-0012-0000-0000-00000330B1F6 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19070915-0013-0000-0000-0000216A17AB Message-Id: <20190709150558.14957-1-fbarrat@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-07-09_06:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=834 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907090179 Subject: [Skiboot] [PATCH v2 1/2] hdata: Align SMP link definitions with current HDAT spec X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: hegdevasant@linux.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" All link usage values other than GPU were unused, so there's no functional change or backward compatibility issue. Signed-off-by: Frederic Barrat --- Changelog: v2: new patch for minor cleanup hdata/spira.c | 2 +- hdata/spira.h | 9 +++++---- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/hdata/spira.c b/hdata/spira.c index 6891a9c7..3d7e21cd 100644 --- a/hdata/spira.c +++ b/hdata/spira.c @@ -1370,7 +1370,7 @@ static void add_npu(struct dt_node *xscom, const struct HDIF_array_hdr *links, struct dt_node *node; /* only add a link node if this link is targeted at at device */ - if (be32_to_cpu(link->usage) != SMP_LINK_USE_DEVICE) + if (be32_to_cpu(link->usage) != SMP_LINK_USE_GPU) continue; /* diff --git a/hdata/spira.h b/hdata/spira.h index 09de4dad..563a98c2 100644 --- a/hdata/spira.h +++ b/hdata/spira.h @@ -1175,10 +1175,11 @@ struct sppcrd_smp_link { __be32 link_id; __be32 usage; #define SMP_LINK_USE_NONE 0 -#define SMP_LINK_USE_DEVICE 1 -#define SMP_LINK_USE_INTERPOSER 2 -#define SMP_LINK_USE_DRAWER 3 -#define SMP_LINK_USE_D2D 4 /* GPU to GPU */ +#define SMP_LINK_USE_GPU 1 +#define SMP_LINK_USE_OPENCAPI 2 +#define SMP_LINK_USE_INTERPOSER 3 +#define SMP_LINK_USE_DRAWER 4 +#define SMP_LINK_USE_D2D 5 /* GPU to GPU */ __be32 brick_id; __be32 lane_mask; From patchwork Tue Jul 9 15:05:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Barrat X-Patchwork-Id: 1129876 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 45jlyJ4cj2z9s8m for ; Wed, 10 Jul 2019 01:06:16 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=linux.ibm.com Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 45jlyJ2rdxzDqVC for ; Wed, 10 Jul 2019 01:06:16 +1000 (AEST) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=linux.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=fbarrat@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=linux.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 45jly75SvmzDqT2 for ; Wed, 10 Jul 2019 01:06:06 +1000 (AEST) Received: from pps.filterd (m0098413.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.27/8.16.0.27) with SMTP id x69EvlxV029975 for ; Tue, 9 Jul 2019 11:06:03 -0400 Received: from e06smtp01.uk.ibm.com (e06smtp01.uk.ibm.com [195.75.94.97]) by mx0b-001b2d01.pphosted.com with ESMTP id 2tmv1av2j1-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 09 Jul 2019 11:06:03 -0400 Received: from localhost by e06smtp01.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 9 Jul 2019 16:06:02 +0100 Received: from b06cxnps4075.portsmouth.uk.ibm.com (9.149.109.197) by e06smtp01.uk.ibm.com (192.168.101.131) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Tue, 9 Jul 2019 16:06:00 +0100 Received: from d06av25.portsmouth.uk.ibm.com (d06av25.portsmouth.uk.ibm.com [9.149.105.61]) by b06cxnps4075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id x69F5xMr39780468 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 9 Jul 2019 15:05:59 GMT Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6F19511C05B; Tue, 9 Jul 2019 15:05:59 +0000 (GMT) Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 29CA811C04A; Tue, 9 Jul 2019 15:05:59 +0000 (GMT) Received: from pic2.home (unknown [9.145.69.11]) by d06av25.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 9 Jul 2019 15:05:59 +0000 (GMT) From: Frederic Barrat To: skiboot@lists.ozlabs.org, andrew.donnellan@au1.ibm.com, clombard@linux.ibm.com Date: Tue, 9 Jul 2019 17:05:58 +0200 X-Mailer: git-send-email 2.21.0 In-Reply-To: <20190709150558.14957-1-fbarrat@linux.ibm.com> References: <20190709150558.14957-1-fbarrat@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 x-cbid: 19070915-4275-0000-0000-0000034B126C X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 19070915-4276-0000-0000-0000385B1357 Message-Id: <20190709150558.14957-2-fbarrat@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-07-09_06:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=2 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1810050000 definitions=main-1907090179 Subject: [Skiboot] [PATCH v2 2/2] npu2-opencapi: Add opencapi support on ZZ X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: hegdevasant@linux.ibm.com Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" This patch adds opencapi support on ZZ. It hard-codes the required device tree entries for the NPU and links. The alternative was to use HDAT, but it somehow proved too painful to do. The new device tree entries activate the npu2 init code on ZZ. On systems with no opencapi adapters, it should go unnoticed, as presence detection will skip link training. Signed-off-by: Frederic Barrat Reviewed-by: Oliver O'Halloran --- Desired for the branch that will be used for FW940 Changelog: v2: user proper dt APIs and explain problems with current HDAT data (Oliver) hdata/spira.c | 9 +++- platforms/ibm-fsp/zz.c | 96 +++++++++++++++++++++++++++++++++++++++++- 2 files changed, 103 insertions(+), 2 deletions(-) diff --git a/hdata/spira.c b/hdata/spira.c index 3d7e21cd..a23fd17c 100644 --- a/hdata/spira.c +++ b/hdata/spira.c @@ -1369,7 +1369,14 @@ static void add_npu(struct dt_node *xscom, const struct HDIF_array_hdr *links, uint64_t speed = 0, nvlink_speed = 0; struct dt_node *node; - /* only add a link node if this link is targeted at at device */ + /* + * Only add a link node if this link is targeted at a + * GPU device. + * + * If we ever activate it for an opencapi device, we + * should revisit the link definitions hard-coded + * on ZZ. + */ if (be32_to_cpu(link->usage) != SMP_LINK_USE_GPU) continue; diff --git a/platforms/ibm-fsp/zz.c b/platforms/ibm-fsp/zz.c index f44c618c..f657b38e 100644 --- a/platforms/ibm-fsp/zz.c +++ b/platforms/ibm-fsp/zz.c @@ -45,6 +45,97 @@ static const struct platform_ocapi zz_ocapi = { .odl_phy_swap = true, }; +#define NPU_BASE 0x5011000 +#define NPU_SIZE 0x2c +#define NPU_INDIRECT0 0x8000000009010c3f /* OB0 - no OB3 on ZZ */ + +static void create_link(struct dt_node *npu, int group, int index) +{ + struct dt_node *link; + uint32_t lane_mask; + + switch (index) { + case 2: + lane_mask = 0xf1e000; /* 0-3, 7-10 */ + break; + case 3: + lane_mask = 0x00078f; /* 13-16, 20-23 */ + break; + default: + assert(0); + } + + link = dt_new_addr(npu, "link", index); + dt_add_property_string(link, "compatible", "ibm,npu-link"); + dt_add_property_cells(link, "ibm,npu-link-index", index); + dt_add_property_u64s(link, "ibm,npu-phy", NPU_INDIRECT0); + dt_add_property_cells(link, "ibm,npu-lane-mask", lane_mask); + dt_add_property_cells(link, "ibm,npu-group-id", group); + dt_add_property_u64s(link, "ibm,link-speed", 25000000000ul); +} + +static void add_opencapi_dt_nodes(void) +{ + struct dt_node *npu, *xscom; + int npu_index = 0; + int phb_index = 7; + + /* + * In an ideal world, we should get all the NPU links + * information from HDAT. But after some effort, HDAT is still + * giving incorrect information for opencapi. As of this + * writing: + * 1. link usage is wrong for most FPGA cards (0xFFFF vs. 2) + * 2. the 24-bit lane mask is aligned differently than on + * other platforms (witherspoon) + * 3. connecting a link entry in HDAT to the real physical + * link will need extra work: + * - HDAT does presence detection and only lists links with + * an adapter, so we cannot use default ordering like on + * witherspoon + * - best option is probably the brick ID field (offset 8). + * It's coming straight from the MRW, but seems to match + * what we expect (2 or 3). Would need to be checked. + * + * To make things more fun, any change in the HDAT data needs + * to be coordinated with PHYP, which is using (some of) those + * fields. + * + * As a consequence: + * 1. the hdat parsing code in skiboot remains disabled (for + * opencapi) + * 2. we hard-code the NPU and links entries in the device + * tree. + * + * Getting the data from HDAT would have the advantage of + * providing the real link speed (20.0 vs. 25.78125 gbps), + * which is useful as there's one speed-dependent setting we + * need to do when initializing the NPU. Our hard coded + * definition assumes the higher speed and may need tuning in + * debug scenario using a lower link speed. + */ + dt_for_each_compatible(dt_root, xscom, "ibm,xscom") { + /* + * our hdat parsing code may create NPU nodes with no + * links, so let's make sure we start from a clean + * state + */ + npu = dt_find_by_name_addr(xscom, "npu", NPU_BASE); + if (npu) + dt_free(npu); + + npu = dt_new_addr(xscom, "npu", NPU_BASE); + dt_add_property_cells(npu, "reg", NPU_BASE, NPU_SIZE); + dt_add_property_strings(npu, "compatible", "ibm,power9-npu"); + dt_add_property_cells(npu, "ibm,npu-index", npu_index++); + dt_add_property_cells(npu, "ibm,phb-index", phb_index++); + dt_add_property_cells(npu, "ibm,npu-links", 2); + + create_link(npu, 1, 2); + create_link(npu, 2, 3); + } +} + static bool zz_probe(void) { /* FIXME: make this neater when the dust settles */ @@ -54,8 +145,11 @@ static bool zz_probe(void) dt_node_is_compatible(dt_root, "ibm,zz-2s4u") || dt_node_is_compatible(dt_root, "ibm,zz-1s4u+gen4") || dt_node_is_compatible(dt_root, "ibm,zz-2s2u+gen4") || - dt_node_is_compatible(dt_root, "ibm,zz-2s4u+gen4")) + dt_node_is_compatible(dt_root, "ibm,zz-2s4u+gen4")) { + + add_opencapi_dt_nodes(); return true; + } return false; }