From patchwork Mon May 18 18:24:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank Heimes X-Patchwork-Id: 1292715 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=canonical.com Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49QnVr0d73z9syB; Tue, 19 May 2020 04:25:07 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1jakRj-00051o-6C; Mon, 18 May 2020 18:25:03 +0000 Received: from youngberry.canonical.com ([91.189.89.112]) by huckleberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1jakRg-00050e-7S for kernel-team@lists.ubuntu.com; Mon, 18 May 2020 18:25:00 +0000 Received: from 2.general.fheimes.uk.vpn ([10.172.194.67] helo=T570.fritz.box) by youngberry.canonical.com with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1jakRf-0008Kv-T1 for kernel-team@lists.ubuntu.com; Mon, 18 May 2020 18:24:59 +0000 From: frank.heimes@canonical.com To: kernel-team@lists.ubuntu.com Subject: [SRU][F][PATCH 0/12] s390x/pci: enumerate pci functions per physical adapter (LP: 1874056) Date: Mon, 18 May 2020 20:24:26 +0200 Message-Id: <20200518182438.392269-1-frank.heimes@canonical.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" Buglink: https://bugs.launchpad.net/bugs/1874056 SRU Justification: [Impact] * On s390x the enumeration of PCI functions does not reflect which functions belongs to which physical adapter. * Layout of a PCI function address on Linux: 0000:00:00.0 ::. * On s390x, each function is presented as individual root complex today, e.g.: PCHID 0100 VF1 0000:00:00.0 PCHID 0100 VF23 0001:00:00.0 PCHID 0200 VF1 0002:00:00.0 OCHID 0100 VF17 0003:00:00.0 * On other platforms, the addresses correctly reflect the actual HW configuration. * Some device drivers (like mlx5, for Mellanox adapters) group functions of one physical adapter by checking which PCI functions have identical values for ::. * We need to use the same enumeration scheme to achieve this functionality on s390x. * In this case, the two physical functions of a Mellanox adapter need to get function number 0 and 1, and all virtual functions need to use the same : numbers with function/device numbers counting up. * Required result (example with 4 VFs per PF): PCHID 0100 PF 0 0000:00:00.0 PCHID 0100 PF 1 0000:00:00.1 PCHID 0100 PF 0 VF 0 0000:00:00.2 PCHID 0100 PF 0 VF 1 0000:00:00.3 PCHID 0100 PF 0 VF 2 0000:00:00.4 PCHID 0100 PF 0 VF 3 0000:00:00.5 PCHID 0100 PF 1 VF 0 0000:00:00.6 PCHID 0100 PF 1 VF 1 0000:00:00.7 PCHID 0100 PF 1 VF 2 0000:00:00.8 PCHID 0100 PF 1 VF 3 0000:00:00.9 PCHID 0200 PF 0 0001:00:00.0 [Fix] * Backport 1: https://launchpadlibrarian.net/479699471/0001-s390-pci-Improve-handling-of-unset-UID.patch * Backport 2: https://launchpadlibrarian.net/479699482/0002-s390-pci-embedding-hotplug_slot-in-zdev.patch * Backport 3: https://launchpadlibrarian.net/479699492/0003-s390-pci-Expose-new-port-attribute-for-PCIe-function.patch * Backport 4: https://launchpadlibrarian.net/479699497/0004-s390-pci-adaptation-of-iommu-to-multifunction.patch * Backport 5: https://launchpadlibrarian.net/479700706/0005-s390-pci-define-kernel-parameters-for-PCI-multifunct.patch * Backport 6: https://launchpadlibrarian.net/479700712/0006-s390-pci-define-RID-and-RID-available.patch * Backport 7: https://launchpadlibrarian.net/479700739/0007-s390-pci-create-zPCI-bus.patch * Backport 8: https://launchpadlibrarian.net/479700769/0008-s390-pci-adapt-events-for-zbus.patch * Backport 9: https://launchpadlibrarian.net/479700786/0009-s390-pci-Handling-multifunctions.patch * Backport 10: https://launchpadlibrarian.net/479700794/0010-s390-pci-Do-not-disable-PF-when-VFs-exist.patch * Backport 11: https://launchpadlibrarian.net/479700798/0011-s390-pci-Documentation-for-zPCI.patch * Backport 12: https://launchpadlibrarian.net/479700799/0012-s390-pci-removes-wrong-PCI-multifunction-assignment.patch [Test Case] * Prepare an IBM z13 or LinuxONE III (or newer) system with two or more RoCE Express PCI 2(.1) adapters. * Assign the adapters (and it's virtual functions) to an LPAR. * Verify whether the physical and virtual functions are grouped in arbitrary order or in consecutive order - physical first (for example with lspci -t ...) [Regression Potential] * The regression potential can be considered as moderate, since: * It is purely s390x specific code (arch/s390/* drivers/iommu/s390-iommu.c and drivers/pci/hotplug/s390_pci_hpc.c - and some doc adjustments, too). * It largely affects zPCI, the s390x specific PCI code layer. * PCI cards available for s390x are optional cards (RoCE and zEDC) and not very wide-spread. * The situation described above affects the RoCE adapters only (Mellanox based). * The patches are also upstream accepted and available via linux-next, but to apply them to focal kernel 5.4 the above backports are needed. * However, the code is modified by several patches (12), hence there is a chance to break zPCI with them. * For upfront testing a PPA got created with a focal (master-next) kernel that incl. all the above patches. Alexander Schmidt (1): s390/pci: Expose new port attribute for PCIe functions Niklas Schnelle (1): s390/pci: Improve handling of unset UID Pierre Morel (10): s390/pci: embedding hotplug_slot in zdev s390/pci: adaptation of iommu to multifunction s390/pci: define kernel parameters for PCI multifunction s390/pci: define RID and RID available s390/pci: create zPCI bus s390/pci: adapt events for zbus s390/pci: Handling multifunctions s390/pci: Do not disable PF when VFs exist s390/pci: Documentation for zPCI s390/pci: removes wrong PCI multifunction assignment .../admin-guide/kernel-parameters.txt | 2 + Documentation/s390/index.rst | 1 + Documentation/s390/pci.rst | 126 +++++++++ MAINTAINERS | 1 + arch/s390/include/asm/pci.h | 45 +++- arch/s390/include/asm/pci_clp.h | 12 +- arch/s390/pci/Makefile | 3 +- arch/s390/pci/pci.c | 198 ++++++-------- arch/s390/pci/pci_bus.c | 255 ++++++++++++++++++ arch/s390/pci/pci_bus.h | 31 +++ arch/s390/pci/pci_clp.c | 6 +- arch/s390/pci/pci_event.c | 39 ++- arch/s390/pci/pci_sysfs.c | 4 +- drivers/iommu/s390-iommu.c | 8 +- drivers/pci/hotplug/s390_pci_hpc.c | 105 +++----- 15 files changed, 618 insertions(+), 218 deletions(-) create mode 100644 Documentation/s390/pci.rst create mode 100644 arch/s390/pci/pci_bus.c create mode 100644 arch/s390/pci/pci_bus.h Acked-by: Stefan Bader Acked-by: Kleber Sacilotto de Souza