From patchwork Thu Mar 1 21:45:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Reza Arbab X-Patchwork-Id: 880128 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zsmG85t5wz9s4v for ; Fri, 2 Mar 2018 08:46:08 +1100 (AEDT) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=linux.vnet.ibm.com Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zsmG84YlZzF1lQ for ; Fri, 2 Mar 2018 08:46:08 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=linux.vnet.ibm.com X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=none (mailfrom) smtp.mailfrom=linux.vnet.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=arbab@linux.vnet.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=linux.vnet.ibm.com Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zsmG042J1zF1k2 for ; Fri, 2 Mar 2018 08:46:00 +1100 (AEDT) Received: from pps.filterd (m0098417.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w21LhpMo018347 for ; Thu, 1 Mar 2018 16:45:57 -0500 Received: from e17.ny.us.ibm.com (e17.ny.us.ibm.com [129.33.205.207]) by mx0a-001b2d01.pphosted.com with ESMTP id 2gep8e8ry9-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Thu, 01 Mar 2018 16:45:57 -0500 Received: from localhost by e17.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 1 Mar 2018 16:45:56 -0500 Received: from b01cxnp22035.gho.pok.ibm.com (9.57.198.25) by e17.ny.us.ibm.com (146.89.104.204) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Thu, 1 Mar 2018 16:45:53 -0500 Received: from b01ledav002.gho.pok.ibm.com (b01ledav002.gho.pok.ibm.com [9.57.199.107]) by b01cxnp22035.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w21Ljrla49545392; Thu, 1 Mar 2018 21:45:53 GMT Received: from b01ledav002.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E10F6124037; Thu, 1 Mar 2018 17:48:07 -0500 (EST) Received: from arbab-laptop.localdomain (unknown [9.53.92.213]) by b01ledav002.gho.pok.ibm.com (Postfix) with ESMTP id C61FC12403F; Thu, 1 Mar 2018 17:48:07 -0500 (EST) Received: by arbab-laptop.localdomain (Postfix, from userid 152845) id A40F6460113; Thu, 1 Mar 2018 15:45:51 -0600 (CST) From: Reza Arbab To: skiboot@lists.ozlabs.org Date: Thu, 1 Mar 2018 15:45:51 -0600 X-Mailer: git-send-email 1.8.3.1 X-TM-AS-GCONF: 00 x-cbid: 18030121-0040-0000-0000-00000400BB2A X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00008609; HX=3.00000241; KW=3.00000007; PH=3.00000004; SC=3.00000254; SDB=6.00996966; UDB=6.00506891; IPR=6.00776284; MB=3.00019805; MTD=3.00000008; XFM=3.00000015; UTC=2018-03-01 21:45:54 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18030121-0041-0000-0000-00000801BBD7 Message-Id: <1519940751-9022-1-git-send-email-arbab@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2018-03-01_12:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=3 phishscore=0 bulkscore=0 spamscore=0 clxscore=1011 lowpriorityscore=0 impostorscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1709140000 definitions=main-1803010266 Subject: [Skiboot] [PATCH] npu2: Use unfiltered mode in XTS tables X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alistair Popple MIME-Version: 1.0 Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" The XTS_PID context table is limited to 256 possible pids/contexts. To relieve this limitation, make use of "unfiltered mode" instead. If an entry in the XTS_BDF table has the bit for unfiltered mode set, we can just use one context for that entire bdf/lpar, regardless of pid. Instead of of searching the XTS_PID table, the NMMU checkout request will simply use the entry indexed by lparshort id instead. Change opal_npu_init_context() to create these lparshort-indexed wildcard entries (0-15) instead of allocating one for each pid. Check that multiple calls for the same bdf all specify the same msr value. In opal_npu_destroy_context(), continue validating the bdf argument, ensuring that it actually maps to an lpar, but no longer remove anything from the XTS_PID table. If/when we start supporting virtualized GPUs, we might consider actually removing these wildcard entries by keeping a refcount, but keep things simple for now. Signed-off-by: Reza Arbab Acked-by: Alistair Popple --- hw/npu2.c | 73 ++++++++++++++++++++++------------------------------- include/npu2-regs.h | 2 ++ 2 files changed, 32 insertions(+), 43 deletions(-) diff --git a/hw/npu2.c b/hw/npu2.c index fad53c4..2a56ac6 100644 --- a/hw/npu2.c +++ b/hw/npu2.c @@ -2039,13 +2039,13 @@ static int npu_table_search(struct npu2 *p, uint64_t table_addr, int stride, * allocated. */ #define NPU2_VALID_ATS_MSR_BITS (MSR_DR | MSR_HV | MSR_PR | MSR_SF) -static int64_t opal_npu_init_context(uint64_t phb_id, int pasid, uint64_t msr, - uint64_t bdf) +static int64_t opal_npu_init_context(uint64_t phb_id, int pasid __unused, + uint64_t msr, uint64_t bdf) { struct phb *phb = pci_get_phb(phb_id); struct npu2 *p = phb_to_npu2(phb); - uint64_t xts_bdf, xts_bdf_pid = 0; - int id, lparshort; + uint64_t xts_bdf, old_xts_bdf_pid, xts_bdf_pid; + int id; if (!phb || phb->phb_type != phb_type_npu_v2) return OPAL_PARAMETER; @@ -2069,20 +2069,8 @@ static int64_t opal_npu_init_context(uint64_t phb_id, int pasid, uint64_t msr, goto out; } - lparshort = GETFIELD(NPU2_XTS_BDF_MAP_LPARSHORT, xts_bdf); - NPU2DBG(p, "Found LPARSHORT = 0x%x for BDF = 0x%03llx\n", lparshort, - bdf); - - /* - * Need to find a free context. - */ - id = npu_table_search(p, NPU2_XTS_PID_MAP, 0x20, NPU2_XTS_PID_MAP_SIZE, - &xts_bdf_pid, -1UL); - if (id < 0) { - NPU2ERR(p, "No XTS contexts available\n"); - id = OPAL_RESOURCE; - goto out; - } + id = GETFIELD(NPU2_XTS_BDF_MAP_LPARSHORT, xts_bdf); + NPU2DBG(p, "Found LPARSHORT = 0x%x for BDF = 0x%03llx\n", id, bdf); /* Enable this mapping for both real and virtual addresses */ xts_bdf_pid = SETFIELD(NPU2_XTS_PID_MAP_VALID_ATRGPA0, 0UL, 1); @@ -2090,8 +2078,7 @@ static int64_t opal_npu_init_context(uint64_t phb_id, int pasid, uint64_t msr, /* Enables TLBIE/MMIOSD forwarding for this entry */ xts_bdf_pid = SETFIELD(NPU2_XTS_PID_MAP_VALID_ATSD, xts_bdf_pid, 1); - xts_bdf_pid = SETFIELD(NPU2_XTS_PID_MAP_LPARSHORT, xts_bdf_pid, - lparshort); + xts_bdf_pid = SETFIELD(NPU2_XTS_PID_MAP_LPARSHORT, xts_bdf_pid, id); /* Set the relevant MSR bits */ xts_bdf_pid = SETFIELD(NPU2_XTS_PID_MAP_MSR_DR, xts_bdf_pid, @@ -2105,8 +2092,20 @@ static int64_t opal_npu_init_context(uint64_t phb_id, int pasid, uint64_t msr, * it here */ xts_bdf_pid = SETFIELD(NPU2_XTS_PID_MAP_MSR_SF, xts_bdf_pid, 1); - /* Finally set the PID/PASID */ - xts_bdf_pid = SETFIELD(NPU2_XTS_PID_MAP_PASID, xts_bdf_pid, pasid); + /* + * Throw an error if the wildcard entry for this bdf is already set + * with different msr bits. + */ + old_xts_bdf_pid = npu2_read(p, NPU2_XTS_PID_MAP + id*0x20); + if (old_xts_bdf_pid) { + if (GETFIELD(NPU2_XTS_PID_MAP_MSR, old_xts_bdf_pid) != + GETFIELD(NPU2_XTS_PID_MAP_MSR, xts_bdf_pid)) { + NPU2ERR(p, "%s: Unexpected MSR value\n", __func__); + id = OPAL_PARAMETER; + } + + goto out; + } /* Write the entry */ NPU2DBG(p, "XTS_PID_MAP[%03d] = 0x%08llx\n", id, xts_bdf_pid); @@ -2118,13 +2117,13 @@ out: } opal_call(OPAL_NPU_INIT_CONTEXT, opal_npu_init_context, 4); -static int opal_npu_destroy_context(uint64_t phb_id, uint64_t pid, uint64_t bdf) +static int opal_npu_destroy_context(uint64_t phb_id, uint64_t pid __unused, + uint64_t bdf) { struct phb *phb = pci_get_phb(phb_id); struct npu2 *p = phb_to_npu2(phb); - uint64_t xts_bdf, xts_bdf_pid; - uint64_t lparshort; - int id, rc = 0; + uint64_t xts_bdf; + int rc = 0; if (!phb || phb->phb_type != phb_type_npu_v2) return OPAL_PARAMETER; @@ -2137,26 +2136,13 @@ static int opal_npu_destroy_context(uint64_t phb_id, uint64_t pid, uint64_t bdf) &xts_bdf, NPU2_XTS_BDF_MAP_BDF) < 0) { NPU2ERR(p, "LPARID not associated with any GPU\n"); rc = OPAL_PARAMETER; - goto out; } - lparshort = GETFIELD(NPU2_XTS_BDF_MAP_LPARSHORT, xts_bdf); - NPU2DBG(p, "Found LPARSHORT = 0x%llx destroy context for BDF = 0x%03llx PID = 0x%llx\n", - lparshort, bdf, pid); - - /* Now find the entry in the bdf/pid table */ - xts_bdf_pid = SETFIELD(NPU2_XTS_PID_MAP_LPARSHORT, 0ul, lparshort); - xts_bdf_pid = SETFIELD(NPU2_XTS_PID_MAP_PASID, xts_bdf_pid, pid); - id = npu_table_search(p, NPU2_XTS_PID_MAP, 0x20, NPU2_XTS_PID_MAP_SIZE, &xts_bdf_pid, - NPU2_XTS_PID_MAP_LPARSHORT | NPU2_XTS_PID_MAP_PASID); - if (id < 0) { - rc = OPAL_PARAMETER; - goto out; - } + /* + * The bdf/pid table only contains wildcard entries, so we don't + * need to remove anything here. + */ - /* And zero the entry */ - npu2_write(p, NPU2_XTS_PID_MAP + id*0x20, 0); -out: unlock(&p->lock); return rc; } @@ -2208,6 +2194,7 @@ static int opal_npu_map_lpar(uint64_t phb_id, uint64_t bdf, uint64_t lparid, } xts_bdf_lpar = SETFIELD(NPU2_XTS_BDF_MAP_VALID, 0UL, 1); + xts_bdf_lpar = SETFIELD(NPU2_XTS_BDF_MAP_UNFILT, xts_bdf_lpar, 1); xts_bdf_lpar = SETFIELD(NPU2_XTS_BDF_MAP_BDF, xts_bdf_lpar, bdf); /* We only support radix for the moment */ diff --git a/include/npu2-regs.h b/include/npu2-regs.h index 73925f9..fd249f4 100644 --- a/include/npu2-regs.h +++ b/include/npu2-regs.h @@ -425,6 +425,7 @@ void npu2_write_mask(struct npu2 *p, uint64_t reg, uint64_t val, uint64_t mask); #define NPU2_XTS_MMIO_ATSD7_LPARID NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_XTS, 0x138) #define NPU2_XTS_BDF_MAP NPU2_REG_OFFSET(NPU2_STACK_MISC, NPU2_BLOCK_XTS, 0x4000) #define NPU2_XTS_BDF_MAP_VALID PPC_BIT(0) +#define NPU2_XTS_BDF_MAP_UNFILT PPC_BIT(1) #define NPU2_XTS_BDF_MAP_STACK PPC_BITMASK(4, 6) #define NPU2_XTS_BDF_MAP_BRICK PPC_BIT(7) #define NPU2_XTS_BDF_MAP_BDF PPC_BITMASK(16, 31) @@ -437,6 +438,7 @@ void npu2_write_mask(struct npu2 *p, uint64_t reg, uint64_t val, uint64_t mask); #define NPU2_XTS_PID_MAP_VALID_ATRGPA0 PPC_BIT(0) #define NPU2_XTS_PID_MAP_VALID_ATRGPA1 PPC_BIT(1) #define NPU2_XTS_PID_MAP_VALID_ATSD PPC_BIT(2) +#define NPU2_XTS_PID_MAP_MSR PPC_BITMASK(25,31) #define NPU2_XTS_PID_MAP_MSR_DR PPC_BIT(25) #define NPU2_XTS_PID_MAP_MSR_TA PPC_BIT(26) #define NPU2_XTS_PID_MAP_MSR_HV PPC_BIT(27)