From patchwork Thu Apr 11 01:32:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: AceLan Kao X-Patchwork-Id: 1922321 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=185.125.189.65; helo=lists.ubuntu.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=patchwork.ozlabs.org) Received: from lists.ubuntu.com (lists.ubuntu.com [185.125.189.65]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VFMdy1hrCz1yYL for ; Thu, 11 Apr 2024 11:33:10 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=lists.ubuntu.com) by lists.ubuntu.com with esmtp (Exim 4.86_2) (envelope-from ) id 1rujJH-0004jP-OD; Thu, 11 Apr 2024 01:33:03 +0000 Received: from mail-oa1-f46.google.com ([209.85.160.46]) by lists.ubuntu.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1rujJE-0004gs-5o for kernel-team@lists.ubuntu.com; Thu, 11 Apr 2024 01:33:00 +0000 Received: by mail-oa1-f46.google.com with SMTP id 586e51a60fabf-22a96054726so5974514fac.0 for ; Wed, 10 Apr 2024 18:32:59 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712799178; x=1713403978; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=OwfGmXrH6ccAAc1kAQBfZE6HA87HngeFk4ltFLhFuE4=; b=MMDDSTbgZzTzCXCst7UsZKwe5btr29sj5+qNPpYnj+T03gh2WyPXBmFlj/N8vNhzLQ tvsjscpcXf7hLIl2fZncKLCzJGhu6XqiSeLM91DXQW4XCLgW4dOftDPg//VCXwS+8Lfg ynUOPXllKIn6yhW+MPzPP12G6kpDw2VEBHThyBHXxBtyCcyhmQdPWylIqoRJxBa8DjLl 8fIzCGHIUEzcJv7W9WURL/51TAGDx6kRL0WKUYf9VrG5SXSiDd3CEB12H7qegXcs1GZV cDkU66Tl+G/GsHm/RWc47rdmCuRcOgNgra+pf/rfGrb5YEC57PHuHyEvu5zwlAWIo1YT Jx9g== X-Gm-Message-State: AOJu0YyTT52fjv1mDzTRAuyuFVcd9ovqAPHQLbx0rpd3oyAT+eJPHr+x gaLWvWBVDB0sRDuaorKVujFRk+bswvqy+XQnMIleAG96Aqm4pP6La8ghpMSY X-Google-Smtp-Source: AGHT+IH6f2ZiPUkj58YzST49BCaPsRXXaUcIW0r/UBzjy9KqhT/yf7YRNLE7FMYWeE69VLxUDVjEmg== X-Received: by 2002:a05:6870:390c:b0:233:5a50:ba31 with SMTP id b12-20020a056870390c00b002335a50ba31mr768649oap.12.1712799178384; Wed, 10 Apr 2024 18:32:58 -0700 (PDT) Received: from localhost (211-75-139-218.hinet-ip.hinet.net. [211.75.139.218]) by smtp.gmail.com with ESMTPSA id n30-20020a056a000d5e00b006ecf217a5e1sm286965pfv.189.2024.04.10.18.32.57 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Apr 2024 18:32:58 -0700 (PDT) From: AceLan Kao To: kernel-team@lists.ubuntu.com Subject: [PATCH 1/7][Noble] PCI/ASPM: Move pci_configure_ltr() to aspm.c Date: Thu, 11 Apr 2024 09:32:47 +0800 Message-Id: <20240411013253.184137-2-acelan.kao@canonical.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240411013253.184137-1-acelan.kao@canonical.com> References: <20240411013253.184137-1-acelan.kao@canonical.com> MIME-Version: 1.0 Received-SPF: pass client-ip=209.85.160.46; envelope-from=acelan@gmail.com; helo=mail-oa1-f46.google.com X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: "David E. Box" BugLink: https://bugs.launchpad.net/bugs/2042500 The Latency Tolerance Reporting (LTR) mechanism supports the ASPM L1.2 state and is only configured when CONFIG_PCIEASPM is set. Move pci_configure_ltr() and pci_bridge_reconfigure_ltr() into aspm.c since they only build when CONFIG_PCIEASPM is set. No functional change intended. Suggested-by: Bjorn Helgaas Link: https://lore.kernel.org/r/20240128233212.1139663-2-david.e.box@linux.intel.com [bhelgaas: commit log, split build change from function moves] Link: https://lore.kernel.org/r/20240223205851.114931-2-helgaas@kernel.org Signed-off-by: David E. Box Signed-off-by: Bjorn Helgaas (cherry picked from commit fa84f4435a6202dd90248517f41e54bf3fb85bc5) Signed-off-by: Chia-Lin Kao (AceLan) --- drivers/pci/pci.c | 18 ---------- drivers/pci/pci.h | 5 ++- drivers/pci/pcie/aspm.c | 75 +++++++++++++++++++++++++++++++++++++++++ drivers/pci/probe.c | 61 --------------------------------- 4 files changed, 79 insertions(+), 80 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index c3585229c12a..898e892fa5d5 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -1652,24 +1652,6 @@ static int pci_save_pcie_state(struct pci_dev *dev) return 0; } -void pci_bridge_reconfigure_ltr(struct pci_dev *dev) -{ -#ifdef CONFIG_PCIEASPM - struct pci_dev *bridge; - u32 ctl; - - bridge = pci_upstream_bridge(dev); - if (bridge && bridge->ltr_path) { - pcie_capability_read_dword(bridge, PCI_EXP_DEVCTL2, &ctl); - if (!(ctl & PCI_EXP_DEVCTL2_LTR_EN)) { - pci_dbg(bridge, "re-enabling LTR\n"); - pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2, - PCI_EXP_DEVCTL2_LTR_EN); - } - } -#endif -} - static void pci_restore_pcie_state(struct pci_dev *dev) { int i = 0; diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index e9750b1b19ba..55b5eb6d04a1 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -97,7 +97,6 @@ void pci_msi_init(struct pci_dev *dev); void pci_msix_init(struct pci_dev *dev); bool pci_bridge_d3_possible(struct pci_dev *dev); void pci_bridge_d3_update(struct pci_dev *dev); -void pci_bridge_reconfigure_ltr(struct pci_dev *dev); int pci_bridge_wait_for_secondary_bus(struct pci_dev *dev, char *reset_type); static inline void pci_wakeup_event(struct pci_dev *dev) @@ -573,11 +572,15 @@ void pcie_aspm_init_link_state(struct pci_dev *pdev); void pcie_aspm_exit_link_state(struct pci_dev *pdev); void pcie_aspm_pm_state_change(struct pci_dev *pdev, bool locked); void pcie_aspm_powersave_config_link(struct pci_dev *pdev); +void pci_configure_ltr(struct pci_dev *pdev); +void pci_bridge_reconfigure_ltr(struct pci_dev *pdev); #else static inline void pcie_aspm_init_link_state(struct pci_dev *pdev) { } static inline void pcie_aspm_exit_link_state(struct pci_dev *pdev) { } static inline void pcie_aspm_pm_state_change(struct pci_dev *pdev, bool locked) { } static inline void pcie_aspm_powersave_config_link(struct pci_dev *pdev) { } +static inline void pci_configure_ltr(struct pci_dev *pdev) { } +static inline void pci_bridge_reconfigure_ltr(struct pci_dev *pdev) { } #endif #ifdef CONFIG_PCIE_ECRC diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index ed7b7bdd6b5b..b5d0a6d56983 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -938,6 +938,81 @@ void pcie_aspm_init_link_state(struct pci_dev *pdev) up_read(&pci_bus_sem); } +void pci_bridge_reconfigure_ltr(struct pci_dev *pdev) +{ + struct pci_dev *bridge; + u32 ctl; + + bridge = pci_upstream_bridge(pdev); + if (bridge && bridge->ltr_path) { + pcie_capability_read_dword(bridge, PCI_EXP_DEVCTL2, &ctl); + if (!(ctl & PCI_EXP_DEVCTL2_LTR_EN)) { + pci_dbg(bridge, "re-enabling LTR\n"); + pcie_capability_set_word(bridge, PCI_EXP_DEVCTL2, + PCI_EXP_DEVCTL2_LTR_EN); + } + } +} + +void pci_configure_ltr(struct pci_dev *pdev) +{ + struct pci_host_bridge *host = pci_find_host_bridge(pdev->bus); + struct pci_dev *bridge; + u32 cap, ctl; + + if (!pci_is_pcie(pdev)) + return; + + /* Read L1 PM substate capabilities */ + pdev->l1ss = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_L1SS); + + pcie_capability_read_dword(pdev, PCI_EXP_DEVCAP2, &cap); + if (!(cap & PCI_EXP_DEVCAP2_LTR)) + return; + + pcie_capability_read_dword(pdev, PCI_EXP_DEVCTL2, &ctl); + if (ctl & PCI_EXP_DEVCTL2_LTR_EN) { + if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT) { + pdev->ltr_path = 1; + return; + } + + bridge = pci_upstream_bridge(pdev); + if (bridge && bridge->ltr_path) + pdev->ltr_path = 1; + + return; + } + + if (!host->native_ltr) + return; + + /* + * Software must not enable LTR in an Endpoint unless the Root + * Complex and all intermediate Switches indicate support for LTR. + * PCIe r4.0, sec 6.18. + */ + if (pci_pcie_type(pdev) == PCI_EXP_TYPE_ROOT_PORT) { + pcie_capability_set_word(pdev, PCI_EXP_DEVCTL2, + PCI_EXP_DEVCTL2_LTR_EN); + pdev->ltr_path = 1; + return; + } + + /* + * If we're configuring a hot-added device, LTR was likely + * disabled in the upstream bridge, so re-enable it before enabling + * it in the new device. + */ + bridge = pci_upstream_bridge(pdev); + if (bridge && bridge->ltr_path) { + pci_bridge_reconfigure_ltr(pdev); + pcie_capability_set_word(pdev, PCI_EXP_DEVCTL2, + PCI_EXP_DEVCTL2_LTR_EN); + pdev->ltr_path = 1; + } +} + /* Recheck latencies and update aspm_capable for links under the root */ static void pcie_update_aspm_capable(struct pcie_link_state *root) { diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index b7335be56008..b809c0b0e0e5 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -2209,67 +2209,6 @@ static void pci_configure_relaxed_ordering(struct pci_dev *dev) } } -static void pci_configure_ltr(struct pci_dev *dev) -{ -#ifdef CONFIG_PCIEASPM - struct pci_host_bridge *host = pci_find_host_bridge(dev->bus); - struct pci_dev *bridge; - u32 cap, ctl; - - if (!pci_is_pcie(dev)) - return; - - /* Read L1 PM substate capabilities */ - dev->l1ss = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_L1SS); - - pcie_capability_read_dword(dev, PCI_EXP_DEVCAP2, &cap); - if (!(cap & PCI_EXP_DEVCAP2_LTR)) - return; - - pcie_capability_read_dword(dev, PCI_EXP_DEVCTL2, &ctl); - if (ctl & PCI_EXP_DEVCTL2_LTR_EN) { - if (pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT) { - dev->ltr_path = 1; - return; - } - - bridge = pci_upstream_bridge(dev); - if (bridge && bridge->ltr_path) - dev->ltr_path = 1; - - return; - } - - if (!host->native_ltr) - return; - - /* - * Software must not enable LTR in an Endpoint unless the Root - * Complex and all intermediate Switches indicate support for LTR. - * PCIe r4.0, sec 6.18. - */ - if (pci_pcie_type(dev) == PCI_EXP_TYPE_ROOT_PORT) { - pcie_capability_set_word(dev, PCI_EXP_DEVCTL2, - PCI_EXP_DEVCTL2_LTR_EN); - dev->ltr_path = 1; - return; - } - - /* - * If we're configuring a hot-added device, LTR was likely - * disabled in the upstream bridge, so re-enable it before enabling - * it in the new device. - */ - bridge = pci_upstream_bridge(dev); - if (bridge && bridge->ltr_path) { - pci_bridge_reconfigure_ltr(dev); - pcie_capability_set_word(dev, PCI_EXP_DEVCTL2, - PCI_EXP_DEVCTL2_LTR_EN); - dev->ltr_path = 1; - } -#endif -} - static void pci_configure_eetlp_prefix(struct pci_dev *dev) { #ifdef CONFIG_PCI_PASID From patchwork Thu Apr 11 01:32:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: AceLan Kao X-Patchwork-Id: 1922322 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=185.125.189.65; helo=lists.ubuntu.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=patchwork.ozlabs.org) Received: from lists.ubuntu.com (lists.ubuntu.com [185.125.189.65]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VFMdy25Svz1yYQ for ; Thu, 11 Apr 2024 11:33:10 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=lists.ubuntu.com) by lists.ubuntu.com with esmtp (Exim 4.86_2) (envelope-from ) id 1rujJI-0004jp-0e; Thu, 11 Apr 2024 01:33:04 +0000 Received: from mail-oa1-f46.google.com ([209.85.160.46]) by lists.ubuntu.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1rujJG-0004ht-3l for kernel-team@lists.ubuntu.com; Thu, 11 Apr 2024 01:33:02 +0000 Received: by mail-oa1-f46.google.com with SMTP id 586e51a60fabf-22efc6b8dc5so3126496fac.0 for ; Wed, 10 Apr 2024 18:33:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712799180; x=1713403980; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=+6X/77bQlIuKIqoqBh8UNVxuA7UAn62EVxr9gH94c4U=; b=FRsZ5Lm/jXfqyyAbEp5zlf5CU021+kynrBV/464MrykPaQWRCpsZ616yS1nrrYzTfN QuOIuDLQ8oYlsW1dUAshQ2VT+YCjQrZRNb9MLkuJ18kWu/FvRyrDhZPVdOQu5sQQNGfL EaVYGVZiuKcdu+XQ8ynjqIkyp2KY5kPLmYEPVR5TOlYeZl/ZxxOAiG8Wg/bf05Tayp2P vrebKpnrDzsEoXHZEM2NBY20J5XI4RH0zH7Kf2yjBHgcS13UshQjNWze2AYh8l1YMnI6 JWAb/ikbKl+8Gv6w8KJUgaLqe8pIHiRZ+pBmu0RJD6KsKkpc8GloLvmRjNyyaQ4Wvt9d /Cxw== X-Gm-Message-State: AOJu0YzZhhMqlQZrXOxuJpqo5TMCV7jn5P3+6moduwYiN+5c3NpmEmnY LdIYyWIGl17oRQ7h0O+7xlhA5q5lkca4Si6MAntb5JH4oO7VQg2+aE6oVBDV X-Google-Smtp-Source: AGHT+IFtG8iWs8IDz/SMVB6H67ZV+1N1IFmQaUVEf4TmAw4IrAi3Ocgh/h6cFicfTCR9sc79PSahhA== X-Received: by 2002:a05:6870:45a6:b0:22e:dfbc:4d90 with SMTP id y38-20020a05687045a600b0022edfbc4d90mr4480133oao.5.1712799180248; Wed, 10 Apr 2024 18:33:00 -0700 (PDT) Received: from localhost (211-75-139-218.hinet-ip.hinet.net. [211.75.139.218]) by smtp.gmail.com with ESMTPSA id a24-20020a056a0011d800b006eadf879a30sm295703pfu.179.2024.04.10.18.32.59 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Apr 2024 18:32:59 -0700 (PDT) From: AceLan Kao To: kernel-team@lists.ubuntu.com Subject: [PATCH 2/7][Noble] PCI/ASPM: Always build aspm.c Date: Thu, 11 Apr 2024 09:32:48 +0800 Message-Id: <20240411013253.184137-3-acelan.kao@canonical.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240411013253.184137-1-acelan.kao@canonical.com> References: <20240411013253.184137-1-acelan.kao@canonical.com> MIME-Version: 1.0 Received-SPF: pass client-ip=209.85.160.46; envelope-from=acelan@gmail.com; helo=mail-oa1-f46.google.com X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: "David E. Box" BugLink: https://bugs.launchpad.net/bugs/2042500 Some ASPM-related tasks, such as save and restore of LTR and L1SS capabilities, still need to be performed when CONFIG_PCIEASPM is not enabled. To prepare for these changes, wrap the current code in aspm.c with an #ifdef and always build the file. Link: https://lore.kernel.org/r/20240128233212.1139663-2-david.e.box@linux.intel.com [bhelgaas: split build change from function moves] Link: https://lore.kernel.org/r/20240223205851.114931-3-helgaas@kernel.org Signed-off-by: David E. Box Signed-off-by: Bjorn Helgaas (cherry picked from commit f3994bba8200b49e3eacbe5914cd13f228e0db37) Signed-off-by: Chia-Lin Kao (AceLan) --- drivers/pci/pcie/Makefile | 2 +- drivers/pci/pcie/aspm.c | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/drivers/pci/pcie/Makefile b/drivers/pci/pcie/Makefile index 8de4ed5f98f1..6461aa93fe76 100644 --- a/drivers/pci/pcie/Makefile +++ b/drivers/pci/pcie/Makefile @@ -6,7 +6,7 @@ pcieportdrv-y := portdrv.o rcec.o obj-$(CONFIG_PCIEPORTBUS) += pcieportdrv.o -obj-$(CONFIG_PCIEASPM) += aspm.o +obj-y += aspm.o obj-$(CONFIG_PCIEAER) += aer.o err.o obj-$(CONFIG_PCIEAER_INJECT) += aer_inject.o obj-$(CONFIG_PCIE_PME) += pme.o diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index b5d0a6d56983..91712c7572e0 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -24,6 +24,8 @@ #include "../pci.h" +#ifdef CONFIG_PCIEASPM + #ifdef MODULE_PARAM_PREFIX #undef MODULE_PARAM_PREFIX #endif @@ -1533,3 +1535,5 @@ bool pcie_aspm_support_enabled(void) { return aspm_support_enabled; } + +#endif /* CONFIG_PCIEASPM */ From patchwork Thu Apr 11 01:32:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: AceLan Kao X-Patchwork-Id: 1922323 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=185.125.189.65; helo=lists.ubuntu.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=patchwork.ozlabs.org) Received: from lists.ubuntu.com (lists.ubuntu.com [185.125.189.65]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VFMf35Q09z1yYL for ; Thu, 11 Apr 2024 11:33:15 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=lists.ubuntu.com) by lists.ubuntu.com with esmtp (Exim 4.86_2) (envelope-from ) id 1rujJN-0004uC-Lo; Thu, 11 Apr 2024 01:33:09 +0000 Received: from mail-pj1-f41.google.com ([209.85.216.41]) by lists.ubuntu.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1rujJJ-0004jy-6D for kernel-team@lists.ubuntu.com; Thu, 11 Apr 2024 01:33:05 +0000 Received: by mail-pj1-f41.google.com with SMTP id 98e67ed59e1d1-2a564ca6f67so2210165a91.2 for ; Wed, 10 Apr 2024 18:33:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712799183; x=1713403983; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=y5m93vdqSnd5TCIr7RnFjIK8TfW6j6XgcgGfpcnmywk=; b=kJOOZXhHTrjDL7aCdClyfM7hQBHHDqTPUp8Ty/l7dnZHWSWwQ87WlLDV9FMDpjWinu pi7l+IOdgUg3vKPbe0Ut07MOHGGwmuIQUh3DLZwW13I1cUfPmoedajycHHHoQnePMITE V9odCZ5CoarA0GInfJ54cmIvWhJe6ywQijJbFoiUZudfhiEkn6/oYcLULC9/6t+dwQxx 3Djc+r978lyYCFzouyP3Wqob5TvqjJzGcVhBVDPP6xkiVelOM1ZNoIc5OhfFphPBN2MN IsBnllz58jw1L3pwHIu5rLhCIxjg/ENOfgfqaX/N+rG+IN96kt/MZzY+OPNYSviNRtjx PHSQ== X-Gm-Message-State: AOJu0YzO9NmGaCD2O5KNLvZO7yIpTaEZbE4QZLL6JgO/6oDfM0HlK41d v302/kPKMhPcwzZJHPRqQqIF8TSj6TiEIqSkJEea11TSuPBXYsF4H7Yl++z6 X-Google-Smtp-Source: AGHT+IEggLbdUlbWTY67PPd9UTG0FOu1t+dzKs9aoONkyMhtqNWIu05iZtYiw+lrFXW6oi/FUtsyzA== X-Received: by 2002:a17:90a:460c:b0:2a2:bd4b:764f with SMTP id w12-20020a17090a460c00b002a2bd4b764fmr4129201pjg.3.1712799182775; Wed, 10 Apr 2024 18:33:02 -0700 (PDT) Received: from localhost (211-75-139-218.hinet-ip.hinet.net. [211.75.139.218]) by smtp.gmail.com with ESMTPSA id f22-20020a17090ace1600b002a25bf61931sm1980237pju.29.2024.04.10.18.33.02 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Apr 2024 18:33:02 -0700 (PDT) From: AceLan Kao To: kernel-team@lists.ubuntu.com Subject: [PATCH 3/7][Noble] PCI/ASPM: Move pci_save_ltr_state() to aspm.c Date: Thu, 11 Apr 2024 09:32:49 +0800 Message-Id: <20240411013253.184137-4-acelan.kao@canonical.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240411013253.184137-1-acelan.kao@canonical.com> References: <20240411013253.184137-1-acelan.kao@canonical.com> MIME-Version: 1.0 Received-SPF: pass client-ip=209.85.216.41; envelope-from=acelan@gmail.com; helo=mail-pj1-f41.google.com X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: "David E. Box" BugLink: https://bugs.launchpad.net/bugs/2042500 Even when CONFIG_PCIEASPM is not set, we save and restore the LTR Capability so that if ASPM L1.2 and LTR were configured by the platform, ASPM L1.2 will still work after suspend/resume, when that platform configuration may be lost. See dbbfadf23190 ("PCI/ASPM: Save LTR Capability for suspend/resume"). Since ASPM L1.2 depends on the LTR Capability, move the save/restore code to the part of aspm.c that is always compiled regardless of CONFIG_PCIEASPM. No functional change intended. Suggested-by: Bjorn Helgaas Link: https://lore.kernel.org/r/20240128233212.1139663-5-david.e.box@linux.intel.com [bhelgaas: commit log, reorder to make this a pure move] Link: https://lore.kernel.org/r/20240223205851.114931-4-helgaas@kernel.org Signed-off-by: David E. Box Signed-off-by: Bjorn Helgaas (cherry picked from commit 1e11b5494c3dbb1e5fce7e95021c1698799c7288) Signed-off-by: Chia-Lin Kao (AceLan) --- drivers/pci/pci.c | 40 ---------------------------------------- drivers/pci/pci.h | 5 +++++ drivers/pci/pcie/aspm.c | 40 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 45 insertions(+), 40 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 898e892fa5d5..b5ba026163e5 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -1715,46 +1715,6 @@ static void pci_restore_pcix_state(struct pci_dev *dev) pci_write_config_word(dev, pos + PCI_X_CMD, cap[i++]); } -static void pci_save_ltr_state(struct pci_dev *dev) -{ - int ltr; - struct pci_cap_saved_state *save_state; - u32 *cap; - - if (!pci_is_pcie(dev)) - return; - - ltr = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_LTR); - if (!ltr) - return; - - save_state = pci_find_saved_ext_cap(dev, PCI_EXT_CAP_ID_LTR); - if (!save_state) { - pci_err(dev, "no suspend buffer for LTR; ASPM issues possible after resume\n"); - return; - } - - /* Some broken devices only support dword access to LTR */ - cap = &save_state->cap.data[0]; - pci_read_config_dword(dev, ltr + PCI_LTR_MAX_SNOOP_LAT, cap); -} - -static void pci_restore_ltr_state(struct pci_dev *dev) -{ - struct pci_cap_saved_state *save_state; - int ltr; - u32 *cap; - - save_state = pci_find_saved_ext_cap(dev, PCI_EXT_CAP_ID_LTR); - ltr = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_LTR); - if (!save_state || !ltr) - return; - - /* Some broken devices only support dword access to LTR */ - cap = &save_state->cap.data[0]; - pci_write_config_dword(dev, ltr + PCI_LTR_MAX_SNOOP_LAT, *cap); -} - /** * pci_save_state - save the PCI configuration space of a device before * suspending diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 55b5eb6d04a1..c202d09b8ba0 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -567,6 +567,11 @@ pci_ers_result_t pcie_do_recovery(struct pci_dev *dev, bool pcie_wait_for_link(struct pci_dev *pdev, bool active); int pcie_retrain_link(struct pci_dev *pdev, bool use_lt); + +/* ASPM-related functionality we need even without CONFIG_PCIEASPM */ +void pci_save_ltr_state(struct pci_dev *dev); +void pci_restore_ltr_state(struct pci_dev *dev); + #ifdef CONFIG_PCIEASPM void pcie_aspm_init_link_state(struct pci_dev *pdev); void pcie_aspm_exit_link_state(struct pci_dev *pdev); diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index 91712c7572e0..8c557475297e 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -24,6 +24,46 @@ #include "../pci.h" +void pci_save_ltr_state(struct pci_dev *dev) +{ + int ltr; + struct pci_cap_saved_state *save_state; + u32 *cap; + + if (!pci_is_pcie(dev)) + return; + + ltr = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_LTR); + if (!ltr) + return; + + save_state = pci_find_saved_ext_cap(dev, PCI_EXT_CAP_ID_LTR); + if (!save_state) { + pci_err(dev, "no suspend buffer for LTR; ASPM issues possible after resume\n"); + return; + } + + /* Some broken devices only support dword access to LTR */ + cap = &save_state->cap.data[0]; + pci_read_config_dword(dev, ltr + PCI_LTR_MAX_SNOOP_LAT, cap); +} + +void pci_restore_ltr_state(struct pci_dev *dev) +{ + struct pci_cap_saved_state *save_state; + int ltr; + u32 *cap; + + save_state = pci_find_saved_ext_cap(dev, PCI_EXT_CAP_ID_LTR); + ltr = pci_find_ext_capability(dev, PCI_EXT_CAP_ID_LTR); + if (!save_state || !ltr) + return; + + /* Some broken devices only support dword access to LTR */ + cap = &save_state->cap.data[0]; + pci_write_config_dword(dev, ltr + PCI_LTR_MAX_SNOOP_LAT, *cap); +} + #ifdef CONFIG_PCIEASPM #ifdef MODULE_PARAM_PREFIX From patchwork Thu Apr 11 01:32:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: AceLan Kao X-Patchwork-Id: 1922324 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=185.125.189.65; helo=lists.ubuntu.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=patchwork.ozlabs.org) Received: from lists.ubuntu.com (lists.ubuntu.com [185.125.189.65]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VFMf51f8gz1yYL for ; Thu, 11 Apr 2024 11:33:17 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=lists.ubuntu.com) by lists.ubuntu.com with esmtp (Exim 4.86_2) (envelope-from ) id 1rujJP-0004xX-0y; Thu, 11 Apr 2024 01:33:11 +0000 Received: from mail-pf1-f182.google.com ([209.85.210.182]) by lists.ubuntu.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1rujJL-0004mt-39 for kernel-team@lists.ubuntu.com; Thu, 11 Apr 2024 01:33:07 +0000 Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-6ecee1f325bso6573944b3a.2 for ; Wed, 10 Apr 2024 18:33:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712799185; x=1713403985; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=aKc6Vn3e4mVs34tNgR4YeEGmahJkRJgfGqeRLcQ/Ln4=; b=fY9aCqgZQGqITT9DoXqR3Kpx65ly/65V255Bi0xngn3CeFq2W2s+lUJ8exo1dIe/NP wlmVcgAm67SXovDCo7aAeDC67MZ8Ji/Y9hQX1nvKuGbeMJHeH3qZAmqCiAG4CBndOeSC tcgqTSDOmxKxJAvspbO0Ang4j6hlnW2lf1CHhHnlsreQm2e5JaePTJrJN8nWpI3LnXHZ e1NcOikOjjKpKLuhOZHchLY0Nb9Mmeh+Rme7VZFV6LDDvqhXieS+7wmCAZ6PWO4lDd4S NLfLyCkis928BGt8cTEYa2coaTa0vRfiUb/Ip1JtfxsHoTgsmmiKoEbvxyBZFBI/t4hh Di/g== X-Gm-Message-State: AOJu0YxPEkr1GllaUc2A+jUjdUak9zerx7iEP05bRGOHHnhqRkF0E5r6 vVCIsSoN+ms7dapoaEX/Gg+JD1D+ls3nNTtegX6HDg0IfDs+rUIsrRWPy88v X-Google-Smtp-Source: AGHT+IGyfyYDAGswRGhjyLmgHZaoZaafDxLVYigY85ET5TDPK0KMwkIZ2u9fNTRW5zgWJo+R5Ek/tw== X-Received: by 2002:a05:6a00:2355:b0:6ea:f444:341b with SMTP id j21-20020a056a00235500b006eaf444341bmr4944019pfj.1.1712799184804; Wed, 10 Apr 2024 18:33:04 -0700 (PDT) Received: from localhost (211-75-139-218.hinet-ip.hinet.net. [211.75.139.218]) by smtp.gmail.com with ESMTPSA id gm8-20020a056a00640800b006e664031f10sm288515pfb.51.2024.04.10.18.33.04 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Apr 2024 18:33:04 -0700 (PDT) From: AceLan Kao To: kernel-team@lists.ubuntu.com Subject: [PATCH 4/7][Noble] PCI/ASPM: Save L1 PM Substates Capability for suspend/resume Date: Thu, 11 Apr 2024 09:32:50 +0800 Message-Id: <20240411013253.184137-5-acelan.kao@canonical.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240411013253.184137-1-acelan.kao@canonical.com> References: <20240411013253.184137-1-acelan.kao@canonical.com> MIME-Version: 1.0 Received-SPF: pass client-ip=209.85.210.182; envelope-from=acelan@gmail.com; helo=mail-pf1-f182.google.com X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: "David E. Box" BugLink: https://bugs.launchpad.net/bugs/2042500 4ff116d0d5fd ("PCI/ASPM: Save L1 PM Substates Capability for suspend/resume") restored the L1 PM Substates Capability after resume, which reduced power consumption by making the ASPM L1.x states work after resume. a7152be79b62 ("Revert "PCI/ASPM: Save L1 PM Substates Capability for suspend/resume"") reverted 4ff116d0d5fd because resume failed on some systems, so power consumption after resume increased again. a7152be79b62 mentioned that we restore L1 PM substate configuration even though ASPM L1 may already be enabled. This is due the fact that the pci_restore_aspm_l1ss_state() was called before pci_restore_pcie_state(). Save and restore the L1 PM Substates Capability, following PCIe r6.1, sec 5.5.4 more closely by: 1) Do not restore ASPM configuration in pci_restore_pcie_state() but do that after PCIe capability is restored in pci_restore_aspm_state() following PCIe r6.1, sec 5.5.4. 2) If BIOS reenables L1SS, particularly L1.2, we need to clear the enables in the right order, downstream before upstream. Defer restoring the L1SS config until we are at the downstream component. Then update the config for both ends of the link in the prescribed order. 3) Program ASPM L1 PM substate configuration before L1 enables. 4) Program ASPM L1 PM substate enables last, after rest of the fields in the capability are programmed. [bhelgaas: commit log, squash L1SS-related patches, do both LNKCTL restores in pci_restore_pcie_state()] Link: https://lore.kernel.org/r/20240128233212.1139663-3-david.e.box@linux.intel.com Link: https://lore.kernel.org/r/20240128233212.1139663-4-david.e.box@linux.intel.com Link: https://lore.kernel.org/r/20240223205851.114931-5-helgaas@kernel.org Closes: https://bugzilla.kernel.org/show_bug.cgi?id=217321 Link: https://bugzilla.kernel.org/show_bug.cgi?id=216782 Link: https://bugzilla.kernel.org/show_bug.cgi?id=216877 Co-developed-by: Mika Westerberg Co-developed-by: David E. Box Reported-by: Koba Ko Signed-off-by: Mika Westerberg Signed-off-by: David E. Box Signed-off-by: Bjorn Helgaas Tested-by: Tasev Nikola # Asus UX305FA Cc: Mark Enriquez Cc: Thomas Witt Cc: Werner Sembach Cc: Vidya Sagar (cherry picked from commit 17423360a27ae58c1850f588bdd8013bbfcd250b) Signed-off-by: Chia-Lin Kao (AceLan) --- drivers/pci/pci.c | 17 ++++++- drivers/pci/pci.h | 3 ++ drivers/pci/pcie/aspm.c | 102 ++++++++++++++++++++++++++++++++++++++-- drivers/pci/probe.c | 1 + include/linux/pci.h | 2 +- 5 files changed, 119 insertions(+), 6 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index b5ba026163e5..1d91498d8859 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -1649,6 +1649,8 @@ static int pci_save_pcie_state(struct pci_dev *dev) pcie_capability_read_word(dev, PCI_EXP_LNKCTL2, &cap[i++]); pcie_capability_read_word(dev, PCI_EXP_SLTCTL2, &cap[i++]); + pci_save_aspm_l1ss_state(dev); + return 0; } @@ -1656,7 +1658,7 @@ static void pci_restore_pcie_state(struct pci_dev *dev) { int i = 0; struct pci_cap_saved_state *save_state; - u16 *cap; + u16 *cap, lnkctl; save_state = pci_find_saved_cap(dev, PCI_CAP_ID_EXP); if (!save_state) @@ -1671,12 +1673,23 @@ static void pci_restore_pcie_state(struct pci_dev *dev) cap = (u16 *)&save_state->cap.data[0]; pcie_capability_write_word(dev, PCI_EXP_DEVCTL, cap[i++]); - pcie_capability_write_word(dev, PCI_EXP_LNKCTL, cap[i++]); + + /* Restore LNKCTL register with ASPM control field clear */ + lnkctl = cap[i++]; + pcie_capability_write_word(dev, PCI_EXP_LNKCTL, + lnkctl & ~PCI_EXP_LNKCTL_ASPMC); + pcie_capability_write_word(dev, PCI_EXP_SLTCTL, cap[i++]); pcie_capability_write_word(dev, PCI_EXP_RTCTL, cap[i++]); pcie_capability_write_word(dev, PCI_EXP_DEVCTL2, cap[i++]); pcie_capability_write_word(dev, PCI_EXP_LNKCTL2, cap[i++]); pcie_capability_write_word(dev, PCI_EXP_SLTCTL2, cap[i++]); + + pci_restore_aspm_l1ss_state(dev); + + /* Restore ASPM control after restoring L1SS state */ + pcie_capability_set_word(dev, PCI_EXP_LNKCTL, + lnkctl & PCI_EXP_LNKCTL_ASPMC); } static int pci_save_pcix_state(struct pci_dev *dev) diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index c202d09b8ba0..8ae24bb1156f 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -571,6 +571,9 @@ int pcie_retrain_link(struct pci_dev *pdev, bool use_lt); /* ASPM-related functionality we need even without CONFIG_PCIEASPM */ void pci_save_ltr_state(struct pci_dev *dev); void pci_restore_ltr_state(struct pci_dev *dev); +void pci_configure_aspm_l1ss(struct pci_dev *dev); +void pci_save_aspm_l1ss_state(struct pci_dev *dev); +void pci_restore_aspm_l1ss_state(struct pci_dev *dev); #ifdef CONFIG_PCIEASPM void pcie_aspm_init_link_state(struct pci_dev *pdev); diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index 8c557475297e..3770af63ed82 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -64,6 +64,105 @@ void pci_restore_ltr_state(struct pci_dev *dev) pci_write_config_dword(dev, ltr + PCI_LTR_MAX_SNOOP_LAT, *cap); } +void pci_configure_aspm_l1ss(struct pci_dev *pdev) +{ + int rc; + + pdev->l1ss = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_L1SS); + + rc = pci_add_ext_cap_save_buffer(pdev, PCI_EXT_CAP_ID_L1SS, + 2 * sizeof(u32)); + if (rc) + pci_err(pdev, "unable to allocate ASPM L1SS save buffer (%pe)\n", + ERR_PTR(rc)); +} + +void pci_save_aspm_l1ss_state(struct pci_dev *pdev) +{ + struct pci_cap_saved_state *save_state; + u16 l1ss = pdev->l1ss; + u32 *cap; + + /* + * Save L1 substate configuration. The ASPM L0s/L1 configuration + * in PCI_EXP_LNKCTL_ASPMC is saved by pci_save_pcie_state(). + */ + if (!l1ss) + return; + + save_state = pci_find_saved_ext_cap(pdev, PCI_EXT_CAP_ID_L1SS); + if (!save_state) + return; + + cap = &save_state->cap.data[0]; + pci_read_config_dword(pdev, l1ss + PCI_L1SS_CTL2, cap++); + pci_read_config_dword(pdev, l1ss + PCI_L1SS_CTL1, cap++); +} + +void pci_restore_aspm_l1ss_state(struct pci_dev *pdev) +{ + struct pci_cap_saved_state *pl_save_state, *cl_save_state; + struct pci_dev *parent = pdev->bus->self; + u32 *cap, pl_ctl1, pl_ctl2, pl_l1_2_enable; + u32 cl_ctl1, cl_ctl2, cl_l1_2_enable; + + /* + * In case BIOS enabled L1.2 when resuming, we need to disable it first + * on the downstream component before the upstream. So, don't attempt to + * restore either until we are at the downstream component. + */ + if (pcie_downstream_port(pdev) || !parent) + return; + + if (!pdev->l1ss || !parent->l1ss) + return; + + cl_save_state = pci_find_saved_ext_cap(pdev, PCI_EXT_CAP_ID_L1SS); + pl_save_state = pci_find_saved_ext_cap(parent, PCI_EXT_CAP_ID_L1SS); + if (!cl_save_state || !pl_save_state) + return; + + cap = &cl_save_state->cap.data[0]; + cl_ctl2 = *cap++; + cl_ctl1 = *cap; + cap = &pl_save_state->cap.data[0]; + pl_ctl2 = *cap++; + pl_ctl1 = *cap; + + /* + * Disable L1.2 on this downstream endpoint device first, followed + * by the upstream + */ + pci_clear_and_set_config_dword(pdev, pdev->l1ss + PCI_L1SS_CTL1, + PCI_L1SS_CTL1_L1_2_MASK, 0); + pci_clear_and_set_config_dword(parent, parent->l1ss + PCI_L1SS_CTL1, + PCI_L1SS_CTL1_L1_2_MASK, 0); + + /* + * In addition, Common_Mode_Restore_Time and LTR_L1.2_THRESHOLD + * in PCI_L1SS_CTL1 must be programmed *before* setting the L1.2 + * enable bits, even though they're all in PCI_L1SS_CTL1. + */ + pl_l1_2_enable = pl_ctl1 & PCI_L1SS_CTL1_L1_2_MASK; + pl_ctl1 &= ~PCI_L1SS_CTL1_L1_2_MASK; + cl_l1_2_enable = cl_ctl1 & PCI_L1SS_CTL1_L1_2_MASK; + cl_ctl1 &= ~PCI_L1SS_CTL1_L1_2_MASK; + + /* Write back without enables first (above we cleared them in ctl1) */ + pci_write_config_dword(parent, parent->l1ss + PCI_L1SS_CTL2, pl_ctl2); + pci_write_config_dword(pdev, pdev->l1ss + PCI_L1SS_CTL2, cl_ctl2); + pci_write_config_dword(parent, parent->l1ss + PCI_L1SS_CTL1, pl_ctl1); + pci_write_config_dword(pdev, pdev->l1ss + PCI_L1SS_CTL1, cl_ctl1); + + /* Then write back the enables */ + if (pl_l1_2_enable || cl_l1_2_enable) { + pci_write_config_dword(parent, parent->l1ss + PCI_L1SS_CTL1, + pl_ctl1 | pl_l1_2_enable); + pci_write_config_dword(pdev, pdev->l1ss + PCI_L1SS_CTL1, + cl_ctl1 | cl_l1_2_enable); + } +} + #ifdef CONFIG_PCIEASPM #ifdef MODULE_PARAM_PREFIX @@ -1005,9 +1104,6 @@ void pci_configure_ltr(struct pci_dev *pdev) if (!pci_is_pcie(pdev)) return; - /* Read L1 PM substate capabilities */ - pdev->l1ss = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_L1SS); - pcie_capability_read_dword(pdev, PCI_EXP_DEVCAP2, &cap); if (!(cap & PCI_EXP_DEVCAP2_LTR)) return; diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index b809c0b0e0e5..1434bf495db3 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -2259,6 +2259,7 @@ static void pci_configure_device(struct pci_dev *dev) pci_configure_extended_tags(dev, NULL); pci_configure_relaxed_ordering(dev); pci_configure_ltr(dev); + pci_configure_aspm_l1ss(dev); pci_configure_eetlp_prefix(dev); pci_configure_serr(dev); diff --git a/include/linux/pci.h b/include/linux/pci.h index 4e08961363e5..8ae950f910e6 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -390,9 +390,9 @@ struct pci_dev { unsigned int d3hot_delay; /* D3hot->D0 transition time in ms */ unsigned int d3cold_delay; /* D3cold->D0 transition time in ms */ + u16 l1ss; /* L1SS Capability pointer */ #ifdef CONFIG_PCIEASPM struct pcie_link_state *link_state; /* ASPM link state */ - u16 l1ss; /* L1SS Capability pointer */ unsigned int ltr_path:1; /* Latency Tolerance Reporting supported from root to here */ #endif From patchwork Thu Apr 11 01:32:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: AceLan Kao X-Patchwork-Id: 1922325 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=185.125.189.65; helo=lists.ubuntu.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=patchwork.ozlabs.org) Received: from lists.ubuntu.com (lists.ubuntu.com [185.125.189.65]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VFMf63wJVz1yYL for ; Thu, 11 Apr 2024 11:33:18 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=lists.ubuntu.com) by lists.ubuntu.com with esmtp (Exim 4.86_2) (envelope-from ) id 1rujJP-0004zb-Nx; Thu, 11 Apr 2024 01:33:11 +0000 Received: from mail-ot1-f44.google.com ([209.85.210.44]) by lists.ubuntu.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1rujJN-0004ql-EO for kernel-team@lists.ubuntu.com; Thu, 11 Apr 2024 01:33:09 +0000 Received: by mail-ot1-f44.google.com with SMTP id 46e09a7af769-6ea2ef1c6b6so942913a34.1 for ; Wed, 10 Apr 2024 18:33:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712799187; x=1713403987; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=7wE3702lhl92lmBQBEpEp5XFmL48cj9YetrFLu9UlA0=; b=o7rmHJNBU+HKoKAozKeN4pAkoqmvOfMIPyypbPdqpnf/TpMl/28sxL/GrpOUccQKTL GZVJWiIEWFeij39a++J6qrXz8mk6VsMH+DnGTG7drVDAe0/+pXHCXAkGD0/sY1U8CQbT f9SLFi4NkuSVWMB5aO+vaGJuTKh7ZlPxcILngpTULEK+capSpm0bF5u6WaH5nEKHZFdm p+WdeJVkuz3U1bWNu7pWtv4amk4a/0CZ3+lWMb7sGGuJKxI7blgAn36aVurjUw/YIf+T 93SskmoY8C3/SwKCYfOxRLHWAP/QamBcK3PYDxX+IOw/GRCQDQ8djWMQKmNS0TOsqs4N zvOw== X-Gm-Message-State: AOJu0Yww2mqtk/aaaIFFI0WYVDu5gQqN2F2UVN21J7WJjGg6h9KLg8l7 TVPJb78kwK5rmr7UZ2003kgogor2jlByOKz9/FmZPzSVQwpMSY0+6XOzI4eX X-Google-Smtp-Source: AGHT+IH8eKozHc1e15Pb/XraG5LlS1+Gyr+O2wOJ1AfczniJrpIjUyHZuXCNdoV+WvBD0nkOVzdYNg== X-Received: by 2002:a05:6870:796:b0:22e:9792:97ed with SMTP id en22-20020a056870079600b0022e979297edmr4955874oab.38.1712799187428; Wed, 10 Apr 2024 18:33:07 -0700 (PDT) Received: from localhost (211-75-139-218.hinet-ip.hinet.net. [211.75.139.218]) by smtp.gmail.com with ESMTPSA id gk11-20020a056a00848b00b006ed045af796sm285189pfb.88.2024.04.10.18.33.06 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Apr 2024 18:33:07 -0700 (PDT) From: AceLan Kao To: kernel-team@lists.ubuntu.com Subject: [PATCH 5/7][Noble] PCI/ASPM: Call pci_save_ltr_state() from pci_save_pcie_state() Date: Thu, 11 Apr 2024 09:32:51 +0800 Message-Id: <20240411013253.184137-6-acelan.kao@canonical.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240411013253.184137-1-acelan.kao@canonical.com> References: <20240411013253.184137-1-acelan.kao@canonical.com> MIME-Version: 1.0 Received-SPF: pass client-ip=209.85.210.44; envelope-from=acelan@gmail.com; helo=mail-ot1-f44.google.com X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: "David E. Box" BugLink: https://bugs.launchpad.net/bugs/2042500 ASPM state is saved and restored from pci_save/restore_pcie_state(). Since the LTR Capability is linked with ASPM, move the LTR save and restore calls there as well. No functional change intended. Suggested-by: Bjorn Helgaas Link: https://lore.kernel.org/r/20240128233212.1139663-6-david.e.box@linux.intel.com Link: https://lore.kernel.org/r/20240223205851.114931-6-helgaas@kernel.org Signed-off-by: David E. Box Signed-off-by: Bjorn Helgaas (cherry picked from commit c198fafa0125e97728d16411aa653602900ab0bc) Signed-off-by: Chia-Lin Kao (AceLan) --- drivers/pci/pci.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 1d91498d8859..2e930ef0210d 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -1650,6 +1650,7 @@ static int pci_save_pcie_state(struct pci_dev *dev) pcie_capability_read_word(dev, PCI_EXP_SLTCTL2, &cap[i++]); pci_save_aspm_l1ss_state(dev); + pci_save_ltr_state(dev); return 0; } @@ -1660,6 +1661,12 @@ static void pci_restore_pcie_state(struct pci_dev *dev) struct pci_cap_saved_state *save_state; u16 *cap, lnkctl; + /* + * Restore max latencies (in the LTR capability) before enabling + * LTR itself in PCI_EXP_DEVCTL2. + */ + pci_restore_ltr_state(dev); + save_state = pci_find_saved_cap(dev, PCI_CAP_ID_EXP); if (!save_state) return; @@ -1752,7 +1759,6 @@ int pci_save_state(struct pci_dev *dev) if (i != 0) return i; - pci_save_ltr_state(dev); pci_save_dpc_state(dev); pci_save_aer_state(dev); pci_save_ptm_state(dev); @@ -1853,12 +1859,6 @@ void pci_restore_state(struct pci_dev *dev) if (!dev->state_saved) return; - /* - * Restore max latencies (in the LTR capability) before enabling - * LTR itself (in the PCIe capability). - */ - pci_restore_ltr_state(dev); - pci_restore_pcie_state(dev); pci_restore_pasid_state(dev); pci_restore_pri_state(dev); From patchwork Thu Apr 11 01:32:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: AceLan Kao X-Patchwork-Id: 1922326 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=185.125.189.65; helo=lists.ubuntu.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=patchwork.ozlabs.org) Received: from lists.ubuntu.com (lists.ubuntu.com [185.125.189.65]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VFMf80Y93z1yYL for ; Thu, 11 Apr 2024 11:33:20 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=lists.ubuntu.com) by lists.ubuntu.com with esmtp (Exim 4.86_2) (envelope-from ) id 1rujJT-00057T-6X; Thu, 11 Apr 2024 01:33:15 +0000 Received: from mail-ot1-f44.google.com ([209.85.210.44]) by lists.ubuntu.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1rujJP-0004wB-4Z for kernel-team@lists.ubuntu.com; Thu, 11 Apr 2024 01:33:11 +0000 Received: by mail-ot1-f44.google.com with SMTP id 46e09a7af769-6ea1f98f3b9so1514282a34.1 for ; Wed, 10 Apr 2024 18:33:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712799189; x=1713403989; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=+zwuJldmVGOozaf3LNz//q5bGaawCtLaBhP+jMY9NlY=; b=QhsyWmfri8Ou3NoVkO1BgjUXKOgyi40a7PLMPNmue7MfnIJbq+P98e8moYsXItTu5X EMA+gvqTcbSIKgmeWXqbyjejhHmOvo+wggayOjkhoHekqWWIEY2yM//rv0mWa+Ieyr1o u8dbpCkKPKHUWTDCfZrtcm1nB9AwC9UOsM+Y00utjuuf//medhRf0ZTbZBnCn0QFZWKd RmdXki2y3o7m+kQPeDxu/dweBNNHrwTWFLY4Kj9QSn/U3kxSygqfuytSYWxI+YK5wlvV 25rqsT5gV3uvb0TFPSt3mNmIFrJvh+wydSOJ9tfsyskNTtJsp9/gKbaLQc8XnaIJFJbq 1NUw== X-Gm-Message-State: AOJu0YyZ75Ll4oBJLa8NF7PiJGc0InMsWwnfGsGG7t2UuIx2pGaRN3KR EZqG915PzTKA4SbIY74SCdivzDDHIBLVeJyI6K71akEssnVdFXNo0FOtBxYu X-Google-Smtp-Source: AGHT+IG7GK6wmM8Mv2g8i0zmLgYBCk/8ayzuwYVrvH/UkWopdYuXbFwC+XD+0dvQmqGkKZr0OInCfw== X-Received: by 2002:a05:6870:41cf:b0:22e:dde2:c05a with SMTP id z15-20020a05687041cf00b0022edde2c05amr4939658oac.3.1712799189180; Wed, 10 Apr 2024 18:33:09 -0700 (PDT) Received: from localhost (211-75-139-218.hinet-ip.hinet.net. [211.75.139.218]) by smtp.gmail.com with ESMTPSA id gu10-20020a056a004e4a00b006ea8cc9250bsm292135pfb.44.2024.04.10.18.33.08 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Apr 2024 18:33:08 -0700 (PDT) From: AceLan Kao To: kernel-team@lists.ubuntu.com Subject: [PATCH 6/7][Noble] PCI/ASPM: Disable L1 before configuring L1 Substates Date: Thu, 11 Apr 2024 09:32:52 +0800 Message-Id: <20240411013253.184137-7-acelan.kao@canonical.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240411013253.184137-1-acelan.kao@canonical.com> References: <20240411013253.184137-1-acelan.kao@canonical.com> MIME-Version: 1.0 Received-SPF: pass client-ip=209.85.210.44; envelope-from=acelan@gmail.com; helo=mail-ot1-f44.google.com X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: Bjorn Helgaas BugLink: https://bugs.launchpad.net/bugs/2042500 Per PCIe r6.1, sec 5.5.4, L1 must be disabled while setting ASPM L1 PM Substates enable bits. Previously this was enforced by clearing PCI_EXP_LNKCTL_ASPMC before calling pci_restore_aspm_l1ss_state(). Move the L1 (and L0s, although that doesn't seem required) disable into pci_restore_aspm_l1ss_state() itself so it's closer to the code that depends on it. Link: https://lore.kernel.org/r/20240223213733.GA115410@bhelgaas Signed-off-by: Bjorn Helgaas (cherry picked from commit 64dbb2d707444f691539fb12aacf81797786c10b) Signed-off-by: Chia-Lin Kao (AceLan) --- drivers/pci/pci.c | 16 +++------------- drivers/pci/pcie/aspm.c | 19 +++++++++++++++++++ 2 files changed, 22 insertions(+), 13 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index 2e930ef0210d..ba7bfc5f52db 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -1659,13 +1659,14 @@ static void pci_restore_pcie_state(struct pci_dev *dev) { int i = 0; struct pci_cap_saved_state *save_state; - u16 *cap, lnkctl; + u16 *cap; /* * Restore max latencies (in the LTR capability) before enabling * LTR itself in PCI_EXP_DEVCTL2. */ pci_restore_ltr_state(dev); + pci_restore_aspm_l1ss_state(dev); save_state = pci_find_saved_cap(dev, PCI_CAP_ID_EXP); if (!save_state) @@ -1680,23 +1681,12 @@ static void pci_restore_pcie_state(struct pci_dev *dev) cap = (u16 *)&save_state->cap.data[0]; pcie_capability_write_word(dev, PCI_EXP_DEVCTL, cap[i++]); - - /* Restore LNKCTL register with ASPM control field clear */ - lnkctl = cap[i++]; - pcie_capability_write_word(dev, PCI_EXP_LNKCTL, - lnkctl & ~PCI_EXP_LNKCTL_ASPMC); - + pcie_capability_write_word(dev, PCI_EXP_LNKCTL, cap[i++]); pcie_capability_write_word(dev, PCI_EXP_SLTCTL, cap[i++]); pcie_capability_write_word(dev, PCI_EXP_RTCTL, cap[i++]); pcie_capability_write_word(dev, PCI_EXP_DEVCTL2, cap[i++]); pcie_capability_write_word(dev, PCI_EXP_LNKCTL2, cap[i++]); pcie_capability_write_word(dev, PCI_EXP_SLTCTL2, cap[i++]); - - pci_restore_aspm_l1ss_state(dev); - - /* Restore ASPM control after restoring L1SS state */ - pcie_capability_set_word(dev, PCI_EXP_LNKCTL, - lnkctl & PCI_EXP_LNKCTL_ASPMC); } static int pci_save_pcix_state(struct pci_dev *dev) diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index 3770af63ed82..43447e356490 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -105,6 +105,7 @@ void pci_restore_aspm_l1ss_state(struct pci_dev *pdev) struct pci_dev *parent = pdev->bus->self; u32 *cap, pl_ctl1, pl_ctl2, pl_l1_2_enable; u32 cl_ctl1, cl_ctl2, cl_l1_2_enable; + u16 clnkctl, plnkctl; /* * In case BIOS enabled L1.2 when resuming, we need to disable it first @@ -129,6 +130,17 @@ void pci_restore_aspm_l1ss_state(struct pci_dev *pdev) pl_ctl2 = *cap++; pl_ctl1 = *cap; + /* Make sure L0s/L1 are disabled before updating L1SS config */ + pcie_capability_read_word(pdev, PCI_EXP_LNKCTL, &clnkctl); + pcie_capability_read_word(parent, PCI_EXP_LNKCTL, &plnkctl); + if (FIELD_GET(PCI_EXP_LNKCTL_ASPMC, clnkctl) || + FIELD_GET(PCI_EXP_LNKCTL_ASPMC, plnkctl)) { + pcie_capability_write_word(pdev, PCI_EXP_LNKCTL, + clnkctl & ~PCI_EXP_LNKCTL_ASPMC); + pcie_capability_write_word(parent, PCI_EXP_LNKCTL, + plnkctl & ~PCI_EXP_LNKCTL_ASPMC); + } + /* * Disable L1.2 on this downstream endpoint device first, followed * by the upstream @@ -161,6 +173,13 @@ void pci_restore_aspm_l1ss_state(struct pci_dev *pdev) pci_write_config_dword(pdev, pdev->l1ss + PCI_L1SS_CTL1, cl_ctl1 | cl_l1_2_enable); } + + /* Restore L0s/L1 if they were enabled */ + if (FIELD_GET(PCI_EXP_LNKCTL_ASPMC, clnkctl) || + FIELD_GET(PCI_EXP_LNKCTL_ASPMC, plnkctl)) { + pcie_capability_write_word(parent, PCI_EXP_LNKCTL, clnkctl); + pcie_capability_write_word(pdev, PCI_EXP_LNKCTL, plnkctl); + } } #ifdef CONFIG_PCIEASPM From patchwork Thu Apr 11 01:32:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: AceLan Kao X-Patchwork-Id: 1922327 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=185.125.189.65; helo=lists.ubuntu.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=patchwork.ozlabs.org) Received: from lists.ubuntu.com (lists.ubuntu.com [185.125.189.65]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VFMf83cV1z1yYQ for ; Thu, 11 Apr 2024 11:33:20 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=lists.ubuntu.com) by lists.ubuntu.com with esmtp (Exim 4.86_2) (envelope-from ) id 1rujJT-000583-JD; Thu, 11 Apr 2024 01:33:15 +0000 Received: from mail-pf1-f181.google.com ([209.85.210.181]) by lists.ubuntu.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1rujJR-00051V-En for kernel-team@lists.ubuntu.com; Thu, 11 Apr 2024 01:33:13 +0000 Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-6ecfeefe94cso293479b3a.0 for ; Wed, 10 Apr 2024 18:33:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712799191; x=1713403991; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:sender:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=Er17Bp4CwWasBj5F9Q6SoK+m2c9ldF0NcHl1oeELJ/E=; b=uB307O3GeTI+6/1jrmWLWDNdUBOQPD3Wmg1UqTq0l+GowUR1Uu/TvD+UZ26dmZsXrH 45QZHX2zoJEPWm2tLP4gUPsmAVRu2xYsuI8zOpD5vKAysbKOScDVW6e7+/XeleOng9x1 KzWsty70Ui8005lQgHnou561Qjan1ToAQdA/SV0M43HhGnje4O6iRP2+wJvb+wm10TLc SEZtCVDSSHE9NfCek22sGNWWnledxLrPB7XNEwGXb1vug4BP07tBry/c+kiWq0HKYPX0 1/6oK6igGDAWwmURl85id/ryPH78k3aQNuw5Mz0xIH0IYF5opeQdqYwCxVuTvy8SW87T Moiw== X-Gm-Message-State: AOJu0Yy8pIxuo8I/2lur1i/tix7RTr35hAQdeCeGawg1deUVO7cZqhOp Ae4xXufoNrABJcYJ46j0Cn5/4A6CypndIlxF6t9Cy7ZWTgYAp7Dvgx4jVtJe X-Google-Smtp-Source: AGHT+IHKVzlOfFdTixfkbnipb83ICER59Q5wSIgu5F5USnScW+Mps6OQYWpvU36c5iUi76dyRN+Tvw== X-Received: by 2002:a05:6a00:2da8:b0:6ed:de2c:77b9 with SMTP id fb40-20020a056a002da800b006edde2c77b9mr1803893pfb.14.1712799191209; Wed, 10 Apr 2024 18:33:11 -0700 (PDT) Received: from localhost (211-75-139-218.hinet-ip.hinet.net. [211.75.139.218]) by smtp.gmail.com with ESMTPSA id y21-20020aa78055000000b006ecfc3a5f2dsm291159pfm.46.2024.04.10.18.33.10 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 10 Apr 2024 18:33:10 -0700 (PDT) From: AceLan Kao To: kernel-team@lists.ubuntu.com Subject: [PATCH 7/7][Noble] PCI/ASPM: Update save_state when configuration changes Date: Thu, 11 Apr 2024 09:32:53 +0800 Message-Id: <20240411013253.184137-8-acelan.kao@canonical.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240411013253.184137-1-acelan.kao@canonical.com> References: <20240411013253.184137-1-acelan.kao@canonical.com> MIME-Version: 1.0 Received-SPF: pass client-ip=209.85.210.181; envelope-from=acelan@gmail.com; helo=mail-pf1-f181.google.com X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: Vidya Sagar BugLink: https://bugs.launchpad.net/bugs/2042500 Many PCIe device drivers save the configuration state of their device during probe and restore it when their .slot_reset() hook is called during PCIe error recovery. If the ASPM configuration is changed after the driver's probe is called and before an error event occurs, .slot_reset() restores the ASPM configuration to what it was at the time of probe, not to what it was just before the occurrence of the error event. This leads to a mismatch in ASPM configuration between the device and its upstream device. Update the saved configuration of the device when the ASPM configuration changes. Link: https://lore.kernel.org/r/20240222174436.3565146-1-vidyas@nvidia.com Signed-off-by: Vidya Sagar [bhelgaas: commit log, rebase to pci/aspm, rename to pci_update_aspm_saved_state() since it updates only LNKCTL, update only ASPMC and CLKREQ_EN in LNKCTL] Signed-off-by: Bjorn Helgaas Reviewed-by: Kuppuswamy Sathyanarayanan Reviewed-by: David E. Box (cherry picked from commit 6d4266675279b38c301243f3a4fac4a511b03246) Signed-off-by: Chia-Lin Kao (AceLan) --- drivers/pci/pcie/aspm.c | 34 +++++++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/drivers/pci/pcie/aspm.c b/drivers/pci/pcie/aspm.c index 43447e356490..42fe18e4f8a9 100644 --- a/drivers/pci/pcie/aspm.c +++ b/drivers/pci/pcie/aspm.c @@ -301,16 +301,42 @@ static int policy_to_clkpm_state(struct pcie_link_state *link) return 0; } +static void pci_update_aspm_saved_state(struct pci_dev *dev) +{ + struct pci_cap_saved_state *save_state; + u16 *cap, lnkctl, aspm_ctl; + + save_state = pci_find_saved_cap(dev, PCI_CAP_ID_EXP); + if (!save_state) + return; + + pcie_capability_read_word(dev, PCI_EXP_LNKCTL, &lnkctl); + + /* + * Update ASPM and CLKREQ bits of LNKCTL in save_state. We only + * write PCI_EXP_LNKCTL_CCC during enumeration, so it shouldn't + * change after being captured in save_state. + */ + aspm_ctl = lnkctl & (PCI_EXP_LNKCTL_ASPMC | PCI_EXP_LNKCTL_CLKREQ_EN); + lnkctl &= ~(PCI_EXP_LNKCTL_ASPMC | PCI_EXP_LNKCTL_CLKREQ_EN); + + /* Depends on pci_save_pcie_state(): cap[1] is LNKCTL */ + cap = (u16 *)&save_state->cap.data[0]; + cap[1] = lnkctl | aspm_ctl; +} + static void pcie_set_clkpm_nocheck(struct pcie_link_state *link, int enable) { struct pci_dev *child; struct pci_bus *linkbus = link->pdev->subordinate; u32 val = enable ? PCI_EXP_LNKCTL_CLKREQ_EN : 0; - list_for_each_entry(child, &linkbus->devices, bus_list) + list_for_each_entry(child, &linkbus->devices, bus_list) { pcie_capability_clear_and_set_word(child, PCI_EXP_LNKCTL, PCI_EXP_LNKCTL_CLKREQ_EN, val); + pci_update_aspm_saved_state(child); + } link->clkpm_enabled = !!enable; } @@ -929,6 +955,12 @@ static void pcie_config_aspm_link(struct pcie_link_state *link, u32 state) pcie_config_aspm_dev(parent, upstream); link->aspm_enabled = state; + + /* Update latest ASPM configuration in saved context */ + pci_save_aspm_l1ss_state(link->downstream); + pci_update_aspm_saved_state(link->downstream); + pci_save_aspm_l1ss_state(parent); + pci_update_aspm_saved_state(parent); } static void pcie_config_aspm_path(struct pcie_link_state *link)