From patchwork Thu Aug 24 04:46:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kai-Heng Feng X-Patchwork-Id: 1825119 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=fqyLmg3d; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=2620:137:e000::1:20; helo=out1.vger.email; envelope-from=linux-pci-owner@vger.kernel.org; receiver=patchwork.ozlabs.org) Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by legolas.ozlabs.org (Postfix) with ESMTP id 4RWVvh6tXDz1yg8 for ; Thu, 24 Aug 2023 14:48:16 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239235AbjHXErm (ORCPT ); Thu, 24 Aug 2023 00:47:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56340 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239211AbjHXEri (ORCPT ); Thu, 24 Aug 2023 00:47:38 -0400 Received: from smtp-relay-canonical-0.canonical.com (smtp-relay-canonical-0.canonical.com [185.125.188.120]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7091E10F0; Wed, 23 Aug 2023 21:47:36 -0700 (PDT) Received: from localhost.localdomain (unknown [10.101.196.174]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-canonical-0.canonical.com (Postfix) with ESMTPSA id 68E883F326; Thu, 24 Aug 2023 04:47:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1692852454; bh=cwxju+w0lZghGRqmzAy7693vkTn397yDO/9OcFcmBOM=; h=From:To:Cc:Subject:Date:Message-Id:MIME-Version; b=fqyLmg3d/wvUKwdbxJ0fIfqne+U8bPsXbU4CmNNLYNj5m5av/aayyawJ4PhVCkyL6 hOwX7pyhi/sWK5MkPRAdazD3/v2EKdoJArfxjOqssXFpWHo+4UlBSXcAnO0fwAyDa9 KmKk6qaPW9hHXhKWGA2L0473ONEg+S2buzTUML5eve3q/lbyvQs8d4Xo/28cg3CYH6 W9hC9ksKEjyM5SwlWQ61uahLNHWY9vLEFvC3qErpnywlpDu8/0Zz+WYtsv2Po/hSGC 8r80BZn+hHxirjGVNc77VRUWEGQ/O43qksRXKGq+4eh26S2hX4CiMJE912Lh9uKqFn eLyC2WDCFAR2w== From: Kai-Heng Feng To: bhelgaas@google.com Cc: mika.westerberg@linux.intel.com, koba.ko@canonical.com, sathyanarayanan.kuppuswamy@linux.intel.com, Kai-Heng Feng , linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 1/3] PCI: Add helper to check if any of ancestor device support D3cold Date: Thu, 24 Aug 2023 12:46:43 +0800 Message-Id: <20230824044645.423378-1-kai.heng.feng@canonical.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org In addition to nearest upstream bridge, driver may want to know if the entire hierarchy can be powered off to perform different action. So walk higher up the hierarchy to find out if any device has valid _PR3. The user will be introduced in next patch. Signed-off-by: Kai-Heng Feng --- drivers/pci/pci.c | 16 ++++++++++++++++ include/linux/pci.h | 2 ++ 2 files changed, 18 insertions(+) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 60230da957e0..79961936ed1b 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -6500,6 +6500,22 @@ bool pci_pr3_present(struct pci_dev *pdev) acpi_has_method(adev->handle, "_PR3"); } EXPORT_SYMBOL_GPL(pci_pr3_present); + +bool pci_ancestor_pr3_present(struct pci_dev *pdev) +{ + struct pci_dev *parent = pdev; + + if (acpi_disabled) + return false; + + while ((parent = pci_upstream_bridge(parent))) { + if (pci_pr3_present(pdev)) + return true; + } + + return false; +} +EXPORT_SYMBOL_GPL(pci_ancestor_pr3_present); #endif /** diff --git a/include/linux/pci.h b/include/linux/pci.h index c69a2cc1f412..1f30c20753a3 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -2512,10 +2512,12 @@ struct irq_domain *pci_host_bridge_acpi_msi_domain(struct pci_bus *bus); void pci_msi_register_fwnode_provider(struct fwnode_handle *(*fn)(struct device *)); bool pci_pr3_present(struct pci_dev *pdev); +bool pci_ancestor_pr3_present(struct pci_dev *pdev); #else static inline struct irq_domain * pci_host_bridge_acpi_msi_domain(struct pci_bus *bus) { return NULL; } static inline bool pci_pr3_present(struct pci_dev *pdev) { return false; } +static inline bool pci_ancestor_pr3_present(struct pci_dev *pdev) { return false; } #endif #ifdef CONFIG_EEH From patchwork Thu Aug 24 04:46:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kai-Heng Feng X-Patchwork-Id: 1825120 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=TNfcEKTW; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=2620:137:e000::1:20; helo=out1.vger.email; envelope-from=linux-pci-owner@vger.kernel.org; receiver=patchwork.ozlabs.org) Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by legolas.ozlabs.org (Postfix) with ESMTP id 4RWVwH1xnVz1yg8 for ; Thu, 24 Aug 2023 14:48:47 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239293AbjHXEsQ (ORCPT ); Thu, 24 Aug 2023 00:48:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38138 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239214AbjHXErm (ORCPT ); Thu, 24 Aug 2023 00:47:42 -0400 Received: from smtp-relay-canonical-0.canonical.com (smtp-relay-canonical-0.canonical.com [185.125.188.120]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BCD7910E4; Wed, 23 Aug 2023 21:47:40 -0700 (PDT) Received: from localhost.localdomain (unknown [10.101.196.174]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-canonical-0.canonical.com (Postfix) with ESMTPSA id 7C121400F1; Thu, 24 Aug 2023 04:47:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1692852459; bh=lT+xOUvkqgknE0fDL1EWLyC+g3xCP3X2QeaziUTo0D0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=TNfcEKTW4L3BY06akVpl+Mj56NQYEVdA96PNmn8X2Bwm0aCHupWR+kxhmVqZ1bLNL 3389hkPLWmf3bqR+vFvDTfSmy9lnJHXmbpDqOFOxGYzE9UQSOAjqt34VQB2aWX5HvO 74b2J7i0K4GB+hYL8bnqtlgJJwrqZd67WP/kZeMqHaFJeyNDiv31ircWXa4SGj9/Bl M6S4YLXPBt4Ys+YpvD4a0qXqRE4A3XjpJvKSOwaRgj4lbTtTbJK9AGTzncVaPkemcq wFCNWaZnbm6xTX0oDg2dyRsecdmnJw8TIj50QrqtCMg06/IzwejuDZMKxpeAIiMb5a tCxYjdBFn8loA== From: Kai-Heng Feng To: bhelgaas@google.com Cc: mika.westerberg@linux.intel.com, koba.ko@canonical.com, sathyanarayanan.kuppuswamy@linux.intel.com, Kai-Heng Feng , Mahesh J Salgaonkar , "Oliver O'Halloran" , linuxppc-dev@lists.ozlabs.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 2/3] PCI/AER: Disable AER service on suspend Date: Thu, 24 Aug 2023 12:46:44 +0800 Message-Id: <20230824044645.423378-2-kai.heng.feng@canonical.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230824044645.423378-1-kai.heng.feng@canonical.com> References: <20230824044645.423378-1-kai.heng.feng@canonical.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org When the power rail gets cut off, the hardware can create some electric noise on the link that triggers AER. If IRQ is shared between AER with PME, such AER noise will cause a spurious wakeup on system suspend. When the power rail gets back, the firmware of the device resets itself and can create unexpected behavior like sending PTM messages. For this case, the driver will always be too late to toggle off features should be disabled. As Per PCIe Base Spec 5.0, section 5.2, titled "Link State Power Management", TLP and DLLP transmission are disabled for a Link in L2/L3 Ready (D3hot), L2 (D3cold with aux power) and L3 (D3cold) states. So if the power will be turned off during suspend process, disable AER service and re-enable it during the resume process. This should not affect the basic functionality. Link: https://bugzilla.kernel.org/show_bug.cgi?id=209149 Link: https://bugzilla.kernel.org/show_bug.cgi?id=216295 v7: - Wording - Disable AER completely (again) if power will be turned off v6: v5: - Wording. v4: v3: - No change. v2: - Only disable AER IRQ. - No more check on PME IRQ#. - Use helper. Signed-off-by: Kai-Heng Feng --- drivers/pci/pcie/aer.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/drivers/pci/pcie/aer.c b/drivers/pci/pcie/aer.c index f6c24ded134c..b5161bfedd8b 100644 --- a/drivers/pci/pcie/aer.c +++ b/drivers/pci/pcie/aer.c @@ -28,6 +28,7 @@ #include #include #include +#include #include #include @@ -1340,6 +1341,28 @@ static int aer_probe(struct pcie_device *dev) return 0; } +static int aer_suspend(struct pcie_device *dev) +{ + struct aer_rpc *rpc = get_service_data(dev); + struct pci_dev *pdev = rpc->rpd; + + if (pci_ancestor_pr3_present(pdev) || pm_suspend_via_firmware()) + aer_disable_rootport(rpc); + + return 0; +} + +static int aer_resume(struct pcie_device *dev) +{ + struct aer_rpc *rpc = get_service_data(dev); + struct pci_dev *pdev = rpc->rpd; + + if (pci_ancestor_pr3_present(pdev) || pm_resume_via_firmware()) + aer_enable_rootport(rpc); + + return 0; +} + /** * aer_root_reset - reset Root Port hierarchy, RCEC, or RCiEP * @dev: pointer to Root Port, RCEC, or RCiEP @@ -1411,6 +1434,8 @@ static struct pcie_port_service_driver aerdriver = { .service = PCIE_PORT_SERVICE_AER, .probe = aer_probe, + .suspend = aer_suspend, + .resume = aer_resume, .remove = aer_remove, }; From patchwork Thu Aug 24 04:46:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kai-Heng Feng X-Patchwork-Id: 1825121 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=acNwiT+K; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=2620:137:e000::1:20; helo=out1.vger.email; envelope-from=linux-pci-owner@vger.kernel.org; receiver=patchwork.ozlabs.org) Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by legolas.ozlabs.org (Postfix) with ESMTP id 4RWVwH42Whz1ygJ for ; Thu, 24 Aug 2023 14:48:47 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239150AbjHXEsQ (ORCPT ); Thu, 24 Aug 2023 00:48:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38140 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239178AbjHXErs (ORCPT ); Thu, 24 Aug 2023 00:47:48 -0400 Received: from smtp-relay-canonical-0.canonical.com (smtp-relay-canonical-0.canonical.com [185.125.188.120]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 552B810F0; Wed, 23 Aug 2023 21:47:45 -0700 (PDT) Received: from localhost.localdomain (unknown [10.101.196.174]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-canonical-0.canonical.com (Postfix) with ESMTPSA id 528C6401C0; Thu, 24 Aug 2023 04:47:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1692852464; bh=gzh8CFjOP3xGUTmBskUbhLru7uSiTvUwnZrR616fQVw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=acNwiT+K3CAJqMILNvkz//RH7LxUzOHRB996jP69N1l3uflJPrHjRjQOjO0XK3D6q eQKbcLJBxZsYJDlfN3NBUIlru/zxfRIvwMKzSVDRrnGYhaqJLgWIE2VomepbQcYW3O h00GK0E2KDuc8TVpJMqQlslavRm2djPPBmdQgpPdwL7rDLrmOoDlX7dWzbCqsYwm40 wBMWWYBlurTE7kLt+dZH1oZgO2eTgJH/vXXu2Afm6bzBDpjcW3snoWBWRKw+IJtEMG q/+or3+ZdhMi3L2WfyHs3qRmtPtL/Hx0P11qICmXSDHFXeHC+30GU44+j15jkSxI66 LX06IKU95LZYg== From: Kai-Heng Feng To: bhelgaas@google.com Cc: mika.westerberg@linux.intel.com, koba.ko@canonical.com, sathyanarayanan.kuppuswamy@linux.intel.com, Kai-Heng Feng , Mahesh J Salgaonkar , "Oliver O'Halloran" , linuxppc-dev@lists.ozlabs.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 3/3] PCI/DPC: Disable DPC service on suspend Date: Thu, 24 Aug 2023 12:46:45 +0800 Message-Id: <20230824044645.423378-3-kai.heng.feng@canonical.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230824044645.423378-1-kai.heng.feng@canonical.com> References: <20230824044645.423378-1-kai.heng.feng@canonical.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org When the power rail gets cut off, the hardware can create some electric noise on the link that triggers AER. If IRQ is shared between AER with PME, such AER noise will cause a spurious wakeup on system suspend. When the power rail gets back, the firmware of the device resets itself and can create unexpected behavior like sending PTM messages. For this case, the driver will always be too late to toggle off features should be disabled. As Per PCIe Base Spec 5.0, section 5.2, titled "Link State Power Management", TLP and DLLP transmission are disabled for a Link in L2/L3 Ready (D3hot), L2 (D3cold with aux power) and L3 (D3cold) states. So if the power will be turned off during suspend process, disable DPC service and re-enable it during the resume process. This should not affect the basic functionality. Since DPC depends on AER to function, also disable DPC. Link: https://bugzilla.kernel.org/show_bug.cgi?id=209149 Link: https://bugzilla.kernel.org/show_bug.cgi?id=216295 v7: - Wording. - Disable DPC completely (again) if power will be turned off v6: v5: - Wording. v4: v3: - No change. v2: - Only disable DPC IRQ. - No more check on PME IRQ#. Signed-off-by: Kai-Heng Feng --- drivers/pci/pcie/dpc.c | 56 ++++++++++++++++++++++++++++++++++-------- 1 file changed, 46 insertions(+), 10 deletions(-) diff --git a/drivers/pci/pcie/dpc.c b/drivers/pci/pcie/dpc.c index 3ceed8e3de41..73426addb2f1 100644 --- a/drivers/pci/pcie/dpc.c +++ b/drivers/pci/pcie/dpc.c @@ -13,6 +13,7 @@ #include #include #include +#include #include "portdrv.h" #include "../pci.h" @@ -347,13 +348,34 @@ void pci_dpc_init(struct pci_dev *pdev) } } +static void dpc_enable(struct pcie_device *dev) +{ + struct pci_dev *pdev = dev->port; + u16 ctl; + + pci_read_config_word(pdev, pdev->dpc_cap + PCI_EXP_DPC_CTL, &ctl); + + ctl = (ctl & 0xfff4) | PCI_EXP_DPC_CTL_EN_FATAL | PCI_EXP_DPC_CTL_INT_EN; + pci_write_config_word(pdev, pdev->dpc_cap + PCI_EXP_DPC_CTL, ctl); +} + +static void dpc_disable(struct pcie_device *dev) +{ + struct pci_dev *pdev = dev->port; + u16 ctl; + + pci_read_config_word(pdev, pdev->dpc_cap + PCI_EXP_DPC_CTL, &ctl); + ctl &= ~(PCI_EXP_DPC_CTL_EN_FATAL | PCI_EXP_DPC_CTL_INT_EN); + pci_write_config_word(pdev, pdev->dpc_cap + PCI_EXP_DPC_CTL, ctl); +} + #define FLAG(x, y) (((x) & (y)) ? '+' : '-') static int dpc_probe(struct pcie_device *dev) { struct pci_dev *pdev = dev->port; struct device *device = &dev->device; int status; - u16 ctl, cap; + u16 cap; if (!pcie_aer_is_native(pdev) && !pcie_ports_dpc_native) return -ENOTSUPP; @@ -368,10 +390,7 @@ static int dpc_probe(struct pcie_device *dev) } pci_read_config_word(pdev, pdev->dpc_cap + PCI_EXP_DPC_CAP, &cap); - pci_read_config_word(pdev, pdev->dpc_cap + PCI_EXP_DPC_CTL, &ctl); - - ctl = (ctl & 0xfff4) | PCI_EXP_DPC_CTL_EN_FATAL | PCI_EXP_DPC_CTL_INT_EN; - pci_write_config_word(pdev, pdev->dpc_cap + PCI_EXP_DPC_CTL, ctl); + dpc_enable(dev); pci_info(pdev, "enabled with IRQ %d\n", dev->irq); pci_info(pdev, "error containment capabilities: Int Msg #%d, RPExt%c PoisonedTLP%c SwTrigger%c RP PIO Log %d, DL_ActiveErr%c\n", @@ -384,14 +403,29 @@ static int dpc_probe(struct pcie_device *dev) return status; } -static void dpc_remove(struct pcie_device *dev) +static int dpc_suspend(struct pcie_device *dev) { struct pci_dev *pdev = dev->port; - u16 ctl; - pci_read_config_word(pdev, pdev->dpc_cap + PCI_EXP_DPC_CTL, &ctl); - ctl &= ~(PCI_EXP_DPC_CTL_EN_FATAL | PCI_EXP_DPC_CTL_INT_EN); - pci_write_config_word(pdev, pdev->dpc_cap + PCI_EXP_DPC_CTL, ctl); + if (pci_ancestor_pr3_present(pdev) || pm_suspend_via_firmware()) + dpc_disable(dev); + + return 0; +} + +static int dpc_resume(struct pcie_device *dev) +{ + struct pci_dev *pdev = dev->port; + + if (pci_ancestor_pr3_present(pdev) || pm_resume_via_firmware()) + dpc_enable(dev); + + return 0; +} + +static void dpc_remove(struct pcie_device *dev) +{ + dpc_disable(dev); } static struct pcie_port_service_driver dpcdriver = { @@ -399,6 +433,8 @@ static struct pcie_port_service_driver dpcdriver = { .port_type = PCIE_ANY_PORT, .service = PCIE_PORT_SERVICE_DPC, .probe = dpc_probe, + .suspend = dpc_suspend, + .resume = dpc_resume, .remove = dpc_remove, };