From patchwork Tue Jun 6 14:59:45 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frederic Barrat X-Patchwork-Id: 771900 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3whvxb3W5mz9s7B for ; Wed, 7 Jun 2017 01:00:19 +1000 (AEST) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3whvxb2mwCzDqLR for ; Wed, 7 Jun 2017 01:00:19 +1000 (AEST) X-Original-To: skiboot@lists.ozlabs.org Delivered-To: skiboot@lists.ozlabs.org Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3whvx95Yh5zDqKq for ; Wed, 7 Jun 2017 00:59:57 +1000 (AEST) Received: from pps.filterd (m0098417.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.20/8.16.0.20) with SMTP id v56Ex1Vu067908 for ; Tue, 6 Jun 2017 10:59:53 -0400 Received: from e06smtp15.uk.ibm.com (e06smtp15.uk.ibm.com [195.75.94.111]) by mx0a-001b2d01.pphosted.com with ESMTP id 2awv1tj58f-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Tue, 06 Jun 2017 10:59:53 -0400 Received: from localhost by e06smtp15.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 6 Jun 2017 15:59:50 +0100 Received: from b06cxnps4076.portsmouth.uk.ibm.com (9.149.109.198) by e06smtp15.uk.ibm.com (192.168.101.145) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Tue, 6 Jun 2017 15:59:46 +0100 Received: from d06av25.portsmouth.uk.ibm.com (d06av25.portsmouth.uk.ibm.com [9.149.105.61]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id v56ExkVh27066420; Tue, 6 Jun 2017 14:59:46 GMT Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4C25611C04A; Tue, 6 Jun 2017 15:57:55 +0100 (BST) Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0D2F711C064; Tue, 6 Jun 2017 15:57:55 +0100 (BST) Received: from borneo.lab.toulouse-stg.fr.ibm.com (unknown [9.101.4.34]) by d06av25.portsmouth.uk.ibm.com (Postfix) with ESMTP; Tue, 6 Jun 2017 15:57:54 +0100 (BST) From: Frederic Barrat To: mikey@neuling.org, benh@kernel.crashing.org, stewart@linux.vnet.ibm.com, skiboot@lists.ozlabs.org Date: Tue, 6 Jun 2017 16:59:45 +0200 X-Mailer: git-send-email 2.11.0 X-TM-AS-GCONF: 00 x-cbid: 17060614-0020-0000-0000-00000380EFC4 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17060614-0021-0000-0000-000041FBC1B9 Message-Id: <20170606145945.26521-1-fbarrat@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2017-06-06_12:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1703280000 definitions=main-1706060254 Subject: [Skiboot] [PATCH v3] phb4: Activate shared PCI slot on witherspoon X-BeenThere: skiboot@lists.ozlabs.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Mailing list for skiboot development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: skiboot-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Skiboot" Witherspoon systems come with a 'shared' PCI slot: physically, it looks like a x16 slot, but it's actually two x8 slots connected to two PHBs of two different chips. Taking advantage of it requires some logic on the PCI adapter. Only the Mellanox CX5 adapter is known to support it at the time of this writing. This patch enables support for the shared slot on witherspoon if a x16 adapter is detected. Each x8 slot has a presence bit, so both bits need to be set for the activation to take place. Slot sharing is activated through a gpio. Note that there's no easy way to be sure that the card is indeed a shared-slot compatible PCI adapter and not a normal x16 card. Plugging a normal x16 adapter on the shared slot should be avoided on witherspoon, as the link won't train on the second slot, resulting in a timeout and a longer boot time. Only the first slot is usable and the x16 adapter will end up using only half the lines. If the PCI card plugged on the physical slot is only x8 (or less), then the presence bit of the second slot is not set, so this patch does nothing. The x8 (or less) adapter should work like on any other physical slot. Signed-off-by: Frederic Barrat --- Changelog: v3: fix in case there's only one chip left (Mikey) v2: address comment from Ben and Mikey core/pci.c | 3 ++ hw/phb4.c | 90 +++++++++++++++++++++++++++++++++++++++++- include/platform.h | 5 +++ include/skiboot.h | 1 + include/xscom-p9-regs.h | 3 ++ platforms/astbmc/witherspoon.c | 6 +++ 6 files changed, 107 insertions(+), 1 deletion(-) diff --git a/core/pci.c b/core/pci.c index 2d3c4f88..e5ec97e8 100644 --- a/core/pci.c +++ b/core/pci.c @@ -1646,6 +1646,9 @@ void pci_init_slots(void) */ time_wait_ms(20); + if (platform.pre_pci_fixup) + platform.pre_pci_fixup(); + prlog(PR_NOTICE, "PCI: Resetting PHBs...\n"); pci_do_jobs(pci_reset_phb); diff --git a/hw/phb4.c b/hw/phb4.c index b53dadfb..ee8e03f7 100644 --- a/hw/phb4.c +++ b/hw/phb4.c @@ -77,6 +77,94 @@ static void phb4_init_hw(struct phb4 *p, bool first_init); #define PHBLOGCFG(p, fmt, a...) do {} while (0) #endif +#define PHB4_PER_CHIP 6 /* Max 6 PHBs per chip on p9 */ +#define PHB4_SHARED_SLOT_IDX_WITHERSPOON 3 + +static int phb4_get_opal_id(unsigned int chip_id, unsigned int index) +{ + return chip_id * PHB4_PER_CHIP + index; +} + +static void phb4_activate_shared_slot_witherspoon(struct proc_chip *chip) +{ + uint64_t val; + + /* + * Shared slot activation is done by raising a GPIO line on the + * chip with the secondary slot. It will somehow activate the + * sideband signals between the slots. + * Need to wait 100us for stability. + */ + xscom_read(chip->id, P9_GPIO_DATA_OUT_ENABLE, &val); + val |= PPC_BIT(2); + xscom_write(chip->id, P9_GPIO_DATA_OUT_ENABLE, val); + + xscom_read(chip->id, P9_GPIO_DATA_OUT, &val); + val |= PPC_BIT(2); + xscom_write(chip->id, P9_GPIO_DATA_OUT, val); + time_wait_us(100); + prlog(PR_INFO, "Shared PCI slot activated\n"); +} + +void phb4_pre_pci_fixup_witherspoon(void) +{ + struct pci_slot *slot0, *slot1; + struct proc_chip *chip0, *chip1; + uint8_t p0 = 0, p1 = 0; + + /* + * Detect if a x16 card is present on the shared slot and + * do some extra configuration if it is. + * + * The shared slot, a.k.a "Slot 2" in the documentation, is + * connected to PEC2 phb index 3 on both chips. From skiboot, + * it looks like two x8 slots, each with its own presence bit. + * + * Here is the matrix of possibilities for the presence bits: + * + * slot0 presence slot1 presence + * 0 0 => no card + * 1 0 => x8 or less card detected + * 1 1 => x16 card detected + * 0 1 => invalid combination + * + * We only act if a x16 card is detected ('1 1' combination above). + * + * One issue is that we don't really know if it is a + * shared-slot-compatible card (such as Mellanox CX5) or + * a 'normal' x16 PCI card. We activate the shared slot in both cases, + * as it doesn't seem to hurt. + * + * If the card is a normal x16 PCI card, the link won't train on the + * second slot (nothing to do with the shared slot activation), the + * procedure will timeout, thus adding some delay to the boot time. + * Therefore the recommendation is that we shouldn't use a normal + * x16 card on the shared slot of a witherspoon. + * + * Plugging a x8 or less adapter on the shared slot should work + * like any other physical slot. + */ + chip0 = next_chip(NULL); + chip1 = next_chip(chip0); + if (!chip1 || next_chip(chip1)) { + prlog(PR_WARNING, + "Unexpected number of chips, skipping shared slot detection\n"); + return; + } + slot0 = pci_slot_find(phb4_get_opal_id(chip0->id, + PHB4_SHARED_SLOT_IDX_WITHERSPOON)); + slot1 = pci_slot_find(phb4_get_opal_id(chip1->id, + PHB4_SHARED_SLOT_IDX_WITHERSPOON)); + if (slot0 && slot1) { + if (slot0->ops.get_presence_state) + slot0->ops.get_presence_state(slot0, &p0); + if (slot1->ops.get_presence_state) + slot1->ops.get_presence_state(slot1, &p1); + if (p0 == 1 && p1 == 1) + phb4_activate_shared_slot_witherspoon(chip1); + } +} + /* Note: The "ASB" name is historical, practically this means access via * the XSCOM backdoor */ @@ -3290,7 +3378,7 @@ static void phb4_create(struct dt_node *np) /* We register the PHB before we initialize it so we * get a useful OPAL ID for it */ - pci_register_phb(&p->phb, p->chip_id * 6 + p->index); //6 PHBs per chip? + pci_register_phb(&p->phb, phb4_get_opal_id(p->chip_id, p->index)); /* Create slot structure */ slot = phb4_slot_create(&p->phb); diff --git a/include/platform.h b/include/platform.h index 91332042..4dcdb335 100644 --- a/include/platform.h +++ b/include/platform.h @@ -101,6 +101,11 @@ struct platform { void (*pci_setup_phb)(struct phb *phb, unsigned int index); /* + * This is called before resetting the PHBs (lift PERST) and + * probing the devices. The PHBs have already been initialized. + */ + void (*pre_pci_fixup)(void); + /* * Called during PCI scan for each device. For bridges, this is * called before its children are probed. This is called for * every device and for the PHB itself with a NULL pd though diff --git a/include/skiboot.h b/include/skiboot.h index 5c8b0c85..14953cab 100644 --- a/include/skiboot.h +++ b/include/skiboot.h @@ -218,6 +218,7 @@ extern int phb3_preload_capp_ucode(void); extern void phb3_preload_vpd(void); extern int phb4_preload_capp_ucode(void); extern void phb4_preload_vpd(void); +extern void phb4_pre_pci_fixup_witherspoon(void); extern void probe_npu(void); extern void probe_npu2(void); extern void uart_init(void); diff --git a/include/xscom-p9-regs.h b/include/xscom-p9-regs.h index 7da404ba..4738e812 100644 --- a/include/xscom-p9-regs.h +++ b/include/xscom-p9-regs.h @@ -18,4 +18,7 @@ #define P9X_EX_NCU_DARN_BAR 0x11011 #define P9X_EX_NCU_DARN_BAR_EN PPC_BIT(0) +#define P9_GPIO_DATA_OUT_ENABLE 0x00000000000B0054ull +#define P9_GPIO_DATA_OUT 0x00000000000B0051ull + #endif /* __XSCOM_P9_REGS_H__ */ diff --git a/platforms/astbmc/witherspoon.c b/platforms/astbmc/witherspoon.c index abaa7c9b..083e7cff 100644 --- a/platforms/astbmc/witherspoon.c +++ b/platforms/astbmc/witherspoon.c @@ -38,10 +38,16 @@ static bool witherspoon_probe(void) return true; } +static void witherspoon_pre_pci_fixup(void) +{ + phb4_pre_pci_fixup_witherspoon(); +} + DECLARE_PLATFORM(witherspoon_platform) = { .name = "Witherspoon", .probe = witherspoon_probe, .init = astbmc_init, + .pre_pci_fixup = witherspoon_pre_pci_fixup, .start_preload_resource = flash_start_preload_resource, .resource_loaded = flash_resource_loaded, .bmc = NULL, /* FIXME: Add openBMC */