| Message ID | 20240708153815.2757367-1-danielgeorgem@chromium.org |
|---|---|
| State | New |
| Headers | show |
| Series | PCI: r8169: add suspend/resume aspm quirk | expand |
[+cc r8169 folks] On Mon, Jul 08, 2024 at 03:38:15PM +0000, George-Daniel Matei wrote: > Added aspm suspend/resume hooks that run > before and after suspend and resume to change > the ASPM states of the PCI bus in order to allow > the system suspend while trying to prevent card hangs Why is this needed? Is there a r8169 defect we're working around? A BIOS defect? Is there a problem report you can reference here? s/Added/Add/ s/aspm/ASPM/ above s/PCI bus/device and parent/ Add period at end of sentence. Rewrap to fill 75 columns. > Signed-off-by: George-Daniel Matei <danielgeorgem@chromium.org> > --- > drivers/pci/quirks.c | 142 +++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 142 insertions(+) > > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c > index dc12d4a06e21..aa3dba2211d3 100644 > --- a/drivers/pci/quirks.c > +++ b/drivers/pci/quirks.c > @@ -6189,6 +6189,148 @@ DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56b0, aspm_l1_acceptable_latency > DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56b1, aspm_l1_acceptable_latency); > DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56c0, aspm_l1_acceptable_latency); > DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56c1, aspm_l1_acceptable_latency); > + > +static const struct dmi_system_id chromebox_match_table[] = { > + { > + .matches = { > + DMI_MATCH(DMI_PRODUCT_NAME, "Brask"), > + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > + } > + }, > + { > + .matches = { > + DMI_MATCH(DMI_PRODUCT_NAME, "Aurash"), > + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > + } > + }, > + { > + .matches = { > + DMI_MATCH(DMI_PRODUCT_NAME, "Bujia"), > + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > + } > + }, > + { > + .matches = { > + DMI_MATCH(DMI_PRODUCT_NAME, "Gaelin"), > + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > + } > + }, > + { > + .matches = { > + DMI_MATCH(DMI_PRODUCT_NAME, "Gladios"), > + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > + } > + }, > + { > + .matches = { > + DMI_MATCH(DMI_PRODUCT_NAME, "Hahn"), > + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > + } > + }, > + { > + .matches = { > + DMI_MATCH(DMI_PRODUCT_NAME, "Jeev"), > + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > + } > + }, > + { > + .matches = { > + DMI_MATCH(DMI_PRODUCT_NAME, "Kinox"), > + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > + } > + }, > + { > + .matches = { > + DMI_MATCH(DMI_PRODUCT_NAME, "Kuldax"), > + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > + } > + }, > + { > + .matches = { > + DMI_MATCH(DMI_PRODUCT_NAME, "Lisbon"), > + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > + } > + }, > + { > + .matches = { > + DMI_MATCH(DMI_PRODUCT_NAME, "Moli"), > + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > + } > + }, > + { } > +}; > + > +static void rtl8169_suspend_aspm_settings(struct pci_dev *dev) > +{ > + u16 val = 0; > + > + if (dmi_check_system(chromebox_match_table)) { > + //configure parent > + pcie_capability_clear_and_set_word(dev->bus->self, > + PCI_EXP_LNKCTL, > + PCI_EXP_LNKCTL_ASPMC, > + PCI_EXP_LNKCTL_ASPM_L1); > + > + pci_read_config_word(dev->bus->self, > + dev->bus->self->l1ss + PCI_L1SS_CTL1, > + &val); > + val = (val & ~PCI_L1SS_CTL1_L1SS_MASK) | > + PCI_L1SS_CTL1_PCIPM_L1_2 | PCI_L1SS_CTL1_PCIPM_L1_2 | > + PCI_L1SS_CTL1_ASPM_L1_1; > + pci_write_config_word(dev->bus->self, > + dev->bus->self->l1ss + PCI_L1SS_CTL1, > + val); > + > + //configure device > + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, > + PCI_EXP_LNKCTL_ASPMC, > + PCI_EXP_LNKCTL_ASPM_L1); > + > + pci_read_config_word(dev, dev->l1ss + PCI_L1SS_CTL1, &val); > + val = (val & ~PCI_L1SS_CTL1_L1SS_MASK) | > + PCI_L1SS_CTL1_PCIPM_L1_2 | PCI_L1SS_CTL1_PCIPM_L1_2 | > + PCI_L1SS_CTL1_ASPM_L1_1; > + pci_write_config_word(dev, dev->l1ss + PCI_L1SS_CTL1, val); > + } > +} > + > +DECLARE_PCI_FIXUP_SUSPEND(PCI_VENDOR_ID_REALTEK, 0x8168, > + rtl8169_suspend_aspm_settings); > + > +static void rtl8169_resume_aspm_settings(struct pci_dev *dev) > +{ > + u16 val = 0; > + > + if (dmi_check_system(chromebox_match_table)) { > + //configure device > + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, > + PCI_EXP_LNKCTL_ASPMC, 0); > + > + pci_read_config_word(dev->bus->self, > + dev->bus->self->l1ss + PCI_L1SS_CTL1, > + &val); > + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; > + pci_write_config_word(dev->bus->self, > + dev->bus->self->l1ss + PCI_L1SS_CTL1, > + val); > + > + //configure parent > + pcie_capability_clear_and_set_word(dev->bus->self, > + PCI_EXP_LNKCTL, > + PCI_EXP_LNKCTL_ASPMC, 0); > + > + pci_read_config_word(dev->bus->self, > + dev->bus->self->l1ss + PCI_L1SS_CTL1, > + &val); > + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; > + pci_write_config_word(dev->bus->self, > + dev->bus->self->l1ss + PCI_L1SS_CTL1, > + val); Updates the parent (dev->bus->self) twice; was the first one supposed to update the device (dev)? This doesn't restore the state as it existed before suspend. Does this rely on other parts of restore to do that? > + } > +} > + > +DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_REALTEK, 0x8168, > + rtl8169_resume_aspm_settings); > #endif > > #ifdef CONFIG_PCIE_DPC > -- > 2.45.2.803.g4e1b14247a-goog >
On 08.07.2024 19:23, Bjorn Helgaas wrote: > [+cc r8169 folks] > > On Mon, Jul 08, 2024 at 03:38:15PM +0000, George-Daniel Matei wrote: >> Added aspm suspend/resume hooks that run >> before and after suspend and resume to change >> the ASPM states of the PCI bus in order to allow >> the system suspend while trying to prevent card hangs > > Why is this needed? Is there a r8169 defect we're working around? > A BIOS defect? Is there a problem report you can reference here? > Basically the same question from my side. Apparently such a workaround isn't needed on any other system. And Realtek NICs can be found on more or less every consumer system. What's the root cause of the issue? A silicon bug on the host side? What is the RTL8168 chip version used on these systems? ASPM L1 is disabled per default in r8169. So why is the patch needed at all? > s/Added/Add/ > > s/aspm/ASPM/ above > > s/PCI bus/device and parent/ > > Add period at end of sentence. > > Rewrap to fill 75 columns. > >> Signed-off-by: George-Daniel Matei <danielgeorgem@chromium.org> >> --- >> drivers/pci/quirks.c | 142 +++++++++++++++++++++++++++++++++++++++++++ >> 1 file changed, 142 insertions(+) >> >> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c >> index dc12d4a06e21..aa3dba2211d3 100644 >> --- a/drivers/pci/quirks.c >> +++ b/drivers/pci/quirks.c >> @@ -6189,6 +6189,148 @@ DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56b0, aspm_l1_acceptable_latency >> DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56b1, aspm_l1_acceptable_latency); >> DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56c0, aspm_l1_acceptable_latency); >> DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56c1, aspm_l1_acceptable_latency); >> + >> +static const struct dmi_system_id chromebox_match_table[] = { >> + { >> + .matches = { >> + DMI_MATCH(DMI_PRODUCT_NAME, "Brask"), >> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >> + } >> + }, >> + { >> + .matches = { >> + DMI_MATCH(DMI_PRODUCT_NAME, "Aurash"), >> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >> + } >> + }, >> + { >> + .matches = { >> + DMI_MATCH(DMI_PRODUCT_NAME, "Bujia"), >> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >> + } >> + }, >> + { >> + .matches = { >> + DMI_MATCH(DMI_PRODUCT_NAME, "Gaelin"), >> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >> + } >> + }, >> + { >> + .matches = { >> + DMI_MATCH(DMI_PRODUCT_NAME, "Gladios"), >> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >> + } >> + }, >> + { >> + .matches = { >> + DMI_MATCH(DMI_PRODUCT_NAME, "Hahn"), >> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >> + } >> + }, >> + { >> + .matches = { >> + DMI_MATCH(DMI_PRODUCT_NAME, "Jeev"), >> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >> + } >> + }, >> + { >> + .matches = { >> + DMI_MATCH(DMI_PRODUCT_NAME, "Kinox"), >> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >> + } >> + }, >> + { >> + .matches = { >> + DMI_MATCH(DMI_PRODUCT_NAME, "Kuldax"), >> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >> + } >> + }, >> + { >> + .matches = { >> + DMI_MATCH(DMI_PRODUCT_NAME, "Lisbon"), >> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >> + } >> + }, >> + { >> + .matches = { >> + DMI_MATCH(DMI_PRODUCT_NAME, "Moli"), >> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >> + } >> + }, >> + { } >> +}; >> + >> +static void rtl8169_suspend_aspm_settings(struct pci_dev *dev) >> +{ >> + u16 val = 0; >> + >> + if (dmi_check_system(chromebox_match_table)) { >> + //configure parent >> + pcie_capability_clear_and_set_word(dev->bus->self, >> + PCI_EXP_LNKCTL, >> + PCI_EXP_LNKCTL_ASPMC, >> + PCI_EXP_LNKCTL_ASPM_L1); >> + >> + pci_read_config_word(dev->bus->self, >> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >> + &val); >> + val = (val & ~PCI_L1SS_CTL1_L1SS_MASK) | >> + PCI_L1SS_CTL1_PCIPM_L1_2 | PCI_L1SS_CTL1_PCIPM_L1_2 | >> + PCI_L1SS_CTL1_ASPM_L1_1; >> + pci_write_config_word(dev->bus->self, >> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >> + val); >> + >> + //configure device >> + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, >> + PCI_EXP_LNKCTL_ASPMC, >> + PCI_EXP_LNKCTL_ASPM_L1); >> + >> + pci_read_config_word(dev, dev->l1ss + PCI_L1SS_CTL1, &val); >> + val = (val & ~PCI_L1SS_CTL1_L1SS_MASK) | >> + PCI_L1SS_CTL1_PCIPM_L1_2 | PCI_L1SS_CTL1_PCIPM_L1_2 | >> + PCI_L1SS_CTL1_ASPM_L1_1; >> + pci_write_config_word(dev, dev->l1ss + PCI_L1SS_CTL1, val); >> + } >> +} >> + >> +DECLARE_PCI_FIXUP_SUSPEND(PCI_VENDOR_ID_REALTEK, 0x8168, >> + rtl8169_suspend_aspm_settings); >> + >> +static void rtl8169_resume_aspm_settings(struct pci_dev *dev) >> +{ >> + u16 val = 0; >> + >> + if (dmi_check_system(chromebox_match_table)) { >> + //configure device >> + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, >> + PCI_EXP_LNKCTL_ASPMC, 0); >> + >> + pci_read_config_word(dev->bus->self, >> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >> + &val); >> + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; >> + pci_write_config_word(dev->bus->self, >> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >> + val); >> + >> + //configure parent >> + pcie_capability_clear_and_set_word(dev->bus->self, >> + PCI_EXP_LNKCTL, >> + PCI_EXP_LNKCTL_ASPMC, 0); >> + >> + pci_read_config_word(dev->bus->self, >> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >> + &val); >> + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; >> + pci_write_config_word(dev->bus->self, >> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >> + val); > > Updates the parent (dev->bus->self) twice; was the first one supposed > to update the device (dev)? > > This doesn't restore the state as it existed before suspend. Does > this rely on other parts of restore to do that? > >> + } >> +} >> + >> +DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_REALTEK, 0x8168, >> + rtl8169_resume_aspm_settings); >> #endif >> >> #ifdef CONFIG_PCIE_DPC >> -- >> 2.45.2.803.g4e1b14247a-goog >>
On Mon, 8 Jul 2024, George-Daniel Matei wrote: > Added aspm suspend/resume hooks that run > before and after suspend and resume to change > the ASPM states of the PCI bus in order to allow > the system suspend while trying to prevent card hangs > > Signed-off-by: George-Daniel Matei <danielgeorgem@chromium.org> > --- > drivers/pci/quirks.c | 142 +++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 142 insertions(+) > > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c > index dc12d4a06e21..aa3dba2211d3 100644 > --- a/drivers/pci/quirks.c > +++ b/drivers/pci/quirks.c > @@ -6189,6 +6189,148 @@ DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56b0, aspm_l1_acceptable_latency > DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56b1, aspm_l1_acceptable_latency); > DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56c0, aspm_l1_acceptable_latency); > DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56c1, aspm_l1_acceptable_latency); > + > +static const struct dmi_system_id chromebox_match_table[] = { > + { > + .matches = { > + DMI_MATCH(DMI_PRODUCT_NAME, "Brask"), > + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > + } > + }, > + { > + .matches = { > + DMI_MATCH(DMI_PRODUCT_NAME, "Aurash"), > + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > + } > + }, > + { > + .matches = { > + DMI_MATCH(DMI_PRODUCT_NAME, "Bujia"), > + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > + } > + }, > + { > + .matches = { > + DMI_MATCH(DMI_PRODUCT_NAME, "Gaelin"), > + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > + } > + }, > + { > + .matches = { > + DMI_MATCH(DMI_PRODUCT_NAME, "Gladios"), > + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > + } > + }, > + { > + .matches = { > + DMI_MATCH(DMI_PRODUCT_NAME, "Hahn"), > + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > + } > + }, > + { > + .matches = { > + DMI_MATCH(DMI_PRODUCT_NAME, "Jeev"), > + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > + } > + }, > + { > + .matches = { > + DMI_MATCH(DMI_PRODUCT_NAME, "Kinox"), > + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > + } > + }, > + { > + .matches = { > + DMI_MATCH(DMI_PRODUCT_NAME, "Kuldax"), > + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > + } > + }, > + { > + .matches = { > + DMI_MATCH(DMI_PRODUCT_NAME, "Lisbon"), > + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > + } > + }, > + { > + .matches = { > + DMI_MATCH(DMI_PRODUCT_NAME, "Moli"), > + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > + } > + }, > + { } > +}; > + > +static void rtl8169_suspend_aspm_settings(struct pci_dev *dev) > +{ > + u16 val = 0; > + > + if (dmi_check_system(chromebox_match_table)) { > + //configure parent Missing space. > + pcie_capability_clear_and_set_word(dev->bus->self, > + PCI_EXP_LNKCTL, > + PCI_EXP_LNKCTL_ASPMC, > + PCI_EXP_LNKCTL_ASPM_L1); > + > + pci_read_config_word(dev->bus->self, > + dev->bus->self->l1ss + PCI_L1SS_CTL1, > + &val); > + val = (val & ~PCI_L1SS_CTL1_L1SS_MASK) | > + PCI_L1SS_CTL1_PCIPM_L1_2 | PCI_L1SS_CTL1_PCIPM_L1_2 | > + PCI_L1SS_CTL1_ASPM_L1_1; > + pci_write_config_word(dev->bus->self, > + dev->bus->self->l1ss + PCI_L1SS_CTL1, > + val); Touching ASPM state should be done through aspm driver, not by writing directly into LNKCTL and L1SS registers. > + //configure device Missing space. > + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, > + PCI_EXP_LNKCTL_ASPMC, > + PCI_EXP_LNKCTL_ASPM_L1); > + > + pci_read_config_word(dev, dev->l1ss + PCI_L1SS_CTL1, &val); > + val = (val & ~PCI_L1SS_CTL1_L1SS_MASK) | > + PCI_L1SS_CTL1_PCIPM_L1_2 | PCI_L1SS_CTL1_PCIPM_L1_2 | > + PCI_L1SS_CTL1_ASPM_L1_1; > + pci_write_config_word(dev, dev->l1ss + PCI_L1SS_CTL1, val); > + } > +} > + > +DECLARE_PCI_FIXUP_SUSPEND(PCI_VENDOR_ID_REALTEK, 0x8168, > + rtl8169_suspend_aspm_settings); > + > +static void rtl8169_resume_aspm_settings(struct pci_dev *dev) > +{ > + u16 val = 0; > + > + if (dmi_check_system(chromebox_match_table)) { > + //configure device Missing space > + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, > + PCI_EXP_LNKCTL_ASPMC, 0); pcie_capability_clear_word() > + > + pci_read_config_word(dev->bus->self, A copy-paste error for the device??? > + dev->bus->self->l1ss + PCI_L1SS_CTL1, > + &val); > + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; > + pci_write_config_word(dev->bus->self, > + dev->bus->self->l1ss + PCI_L1SS_CTL1, > + val); > + > + //configure parent Missing space > + pcie_capability_clear_and_set_word(dev->bus->self, > + PCI_EXP_LNKCTL, > + PCI_EXP_LNKCTL_ASPMC, 0); pcie_capability_clear_and_set_word() > + > + pci_read_config_word(dev->bus->self, > + dev->bus->self->l1ss + PCI_L1SS_CTL1, > + &val); > + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; > + pci_write_config_word(dev->bus->self, > + dev->bus->self->l1ss + PCI_L1SS_CTL1, > + val); Touching the same device twice here? > + } > +} > + > +DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_REALTEK, 0x8168, > + rtl8169_resume_aspm_settings); > #endif
Hi, >> Added aspm suspend/resume hooks that run >> before and after suspend and resume to change >> the ASPM states of the PCI bus in order to allow >> the system suspend while trying to prevent card hangs > > Why is this needed? Is there a r8169 defect we're working around? > A BIOS defect? Is there a problem report you can reference here? > We encountered this issue while upgrading from kernel v6.1 to v6.6. The system would not suspend with 6.6. We tracked down the problem to the NIC of the device, mainly that the following code was removed in 6.6: > else if (tp->mac_version >= RTL_GIGA_MAC_VER_46) > rc = pci_disable_link_state(pdev, PCIE_LINK_STATE_L1_2); For the listed devices, ASPM L1 is disabled entirely in 6.6. As for the reason, L1 was observed to cause some problems (https://bugzilla.kernel.org/show_bug.cgi?id=217814). We use a Raptor Lake soc and it won't change residency if the NIC doesn't have L1 enabled. I saw in 6.1 the following comment: > Chips from RTL8168h partially have issues with L1.2, but seem > to work fine with L1 and L1.1. I was thinking that disabling/enabling L1.1 on the fly before/after suspend could help mitigate the risk associated with L1/L1.1 . I know that ASPM settings are exposed in sysfs and that this could be done from outside the kernel, that was my first approach, but it was suggested to me that this kind of workaround would be better suited for quirks. I did around 1000 suspend/resume cycles of 16-30 seconds each (correcting the resume dev->bus->self being configured twice mistake) and did not notice any problems. What do you think, is this a good approach ... ? >> + //configure device >> + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, >> + PCI_EXP_LNKCTL_ASPMC, 0); >> + >> + pci_read_config_word(dev->bus->self, >> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >> + &val); >> + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; >> + pci_write_config_word(dev->bus->self, >> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >> + val); > Updates the parent (dev->bus->self) twice; was the first one supposed > to update the device (dev)? Yes, it was supposed to update the device (dev). It's my first time sending a patch and I messed something up while doing some style changes, I will correct it. I'm sorry for that. > This doesn't restore the state as it existed before suspend. Does > this rely on other parts of restore to do that? It operates on the assumption that after driver initialization PCI_EXP_LNKCTL_ASPMC is 0 and that there are no states enabled in CTL1. I did a lspci -vvv dump on the affected devices before and after the quirks ran and saw no difference. This could be improved. > What is the RTL8168 chip version used on these systems? It should be RTL8111H. > What's the root cause of the issue? > A silicon bug on the host side? I think it's the ASPM implementation of the soc. > ASPM L1 is disabled per default in r8169. So why is the patch needed > at all? Leaving it disabled all the time prevents the system from suspending. Thank you, George-Daniel Matei On Tue, Jul 9, 2024 at 12:15 AM Heiner Kallweit <hkallweit1@gmail.com> wrote: > > On 08.07.2024 19:23, Bjorn Helgaas wrote: > > [+cc r8169 folks] > > > > On Mon, Jul 08, 2024 at 03:38:15PM +0000, George-Daniel Matei wrote: > >> Added aspm suspend/resume hooks that run > >> before and after suspend and resume to change > >> the ASPM states of the PCI bus in order to allow > >> the system suspend while trying to prevent card hangs > > > > Why is this needed? Is there a r8169 defect we're working around? > > A BIOS defect? Is there a problem report you can reference here? > > > > Basically the same question from my side. Apparently such a workaround > isn't needed on any other system. And Realtek NICs can be found on more > or less every consumer system. What's the root cause of the issue? > A silicon bug on the host side? > > What is the RTL8168 chip version used on these systems? > > ASPM L1 is disabled per default in r8169. So why is the patch needed > at all? > > > s/Added/Add/ > > > > s/aspm/ASPM/ above > > > > s/PCI bus/device and parent/ > > > > Add period at end of sentence. > > > > Rewrap to fill 75 columns. > > > >> Signed-off-by: George-Daniel Matei <danielgeorgem@chromium.org> > >> --- > >> drivers/pci/quirks.c | 142 +++++++++++++++++++++++++++++++++++++++++++ > >> 1 file changed, 142 insertions(+) > >> > >> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c > >> index dc12d4a06e21..aa3dba2211d3 100644 > >> --- a/drivers/pci/quirks.c > >> +++ b/drivers/pci/quirks.c > >> @@ -6189,6 +6189,148 @@ DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56b0, aspm_l1_acceptable_latency > >> DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56b1, aspm_l1_acceptable_latency); > >> DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56c0, aspm_l1_acceptable_latency); > >> DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56c1, aspm_l1_acceptable_latency); > >> + > >> +static const struct dmi_system_id chromebox_match_table[] = { > >> + { > >> + .matches = { > >> + DMI_MATCH(DMI_PRODUCT_NAME, "Brask"), > >> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >> + } > >> + }, > >> + { > >> + .matches = { > >> + DMI_MATCH(DMI_PRODUCT_NAME, "Aurash"), > >> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >> + } > >> + }, > >> + { > >> + .matches = { > >> + DMI_MATCH(DMI_PRODUCT_NAME, "Bujia"), > >> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >> + } > >> + }, > >> + { > >> + .matches = { > >> + DMI_MATCH(DMI_PRODUCT_NAME, "Gaelin"), > >> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >> + } > >> + }, > >> + { > >> + .matches = { > >> + DMI_MATCH(DMI_PRODUCT_NAME, "Gladios"), > >> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >> + } > >> + }, > >> + { > >> + .matches = { > >> + DMI_MATCH(DMI_PRODUCT_NAME, "Hahn"), > >> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >> + } > >> + }, > >> + { > >> + .matches = { > >> + DMI_MATCH(DMI_PRODUCT_NAME, "Jeev"), > >> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >> + } > >> + }, > >> + { > >> + .matches = { > >> + DMI_MATCH(DMI_PRODUCT_NAME, "Kinox"), > >> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >> + } > >> + }, > >> + { > >> + .matches = { > >> + DMI_MATCH(DMI_PRODUCT_NAME, "Kuldax"), > >> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >> + } > >> + }, > >> + { > >> + .matches = { > >> + DMI_MATCH(DMI_PRODUCT_NAME, "Lisbon"), > >> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >> + } > >> + }, > >> + { > >> + .matches = { > >> + DMI_MATCH(DMI_PRODUCT_NAME, "Moli"), > >> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >> + } > >> + }, > >> + { } > >> +}; > >> + > >> +static void rtl8169_suspend_aspm_settings(struct pci_dev *dev) > >> +{ > >> + u16 val = 0; > >> + > >> + if (dmi_check_system(chromebox_match_table)) { > >> + //configure parent > >> + pcie_capability_clear_and_set_word(dev->bus->self, > >> + PCI_EXP_LNKCTL, > >> + PCI_EXP_LNKCTL_ASPMC, > >> + PCI_EXP_LNKCTL_ASPM_L1); > >> + > >> + pci_read_config_word(dev->bus->self, > >> + dev->bus->self->l1ss + PCI_L1SS_CTL1, > >> + &val); > >> + val = (val & ~PCI_L1SS_CTL1_L1SS_MASK) | > >> + PCI_L1SS_CTL1_PCIPM_L1_2 | PCI_L1SS_CTL1_PCIPM_L1_2 | > >> + PCI_L1SS_CTL1_ASPM_L1_1; > >> + pci_write_config_word(dev->bus->self, > >> + dev->bus->self->l1ss + PCI_L1SS_CTL1, > >> + val); > >> + > >> + //configure device > >> + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, > >> + PCI_EXP_LNKCTL_ASPMC, > >> + PCI_EXP_LNKCTL_ASPM_L1); > >> + > >> + pci_read_config_word(dev, dev->l1ss + PCI_L1SS_CTL1, &val); > >> + val = (val & ~PCI_L1SS_CTL1_L1SS_MASK) | > >> + PCI_L1SS_CTL1_PCIPM_L1_2 | PCI_L1SS_CTL1_PCIPM_L1_2 | > >> + PCI_L1SS_CTL1_ASPM_L1_1; > >> + pci_write_config_word(dev, dev->l1ss + PCI_L1SS_CTL1, val); > >> + } > >> +} > >> + > >> +DECLARE_PCI_FIXUP_SUSPEND(PCI_VENDOR_ID_REALTEK, 0x8168, > >> + rtl8169_suspend_aspm_settings); > >> + > >> +static void rtl8169_resume_aspm_settings(struct pci_dev *dev) > >> +{ > >> + u16 val = 0; > >> + > >> + if (dmi_check_system(chromebox_match_table)) { > >> + //configure device > >> + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, > >> + PCI_EXP_LNKCTL_ASPMC, 0); > >> + > >> + pci_read_config_word(dev->bus->self, > >> + dev->bus->self->l1ss + PCI_L1SS_CTL1, > >> + &val); > >> + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; > >> + pci_write_config_word(dev->bus->self, > >> + dev->bus->self->l1ss + PCI_L1SS_CTL1, > >> + val); > >> + > >> + //configure parent > >> + pcie_capability_clear_and_set_word(dev->bus->self, > >> + PCI_EXP_LNKCTL, > >> + PCI_EXP_LNKCTL_ASPMC, 0); > >> + > >> + pci_read_config_word(dev->bus->self, > >> + dev->bus->self->l1ss + PCI_L1SS_CTL1, > >> + &val); > >> + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; > >> + pci_write_config_word(dev->bus->self, > >> + dev->bus->self->l1ss + PCI_L1SS_CTL1, > >> + val); > > > > Updates the parent (dev->bus->self) twice; was the first one supposed > > to update the device (dev)? > > > > This doesn't restore the state as it existed before suspend. Does > > this rely on other parts of restore to do that? > > > >> + } > >> +} > >> + > >> +DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_REALTEK, 0x8168, > >> + rtl8169_resume_aspm_settings); > >> #endif > >> > >> #ifdef CONFIG_PCIE_DPC > >> -- > >> 2.45.2.803.g4e1b14247a-goog > >> >
On 10.07.2024 17:09, George-Daniel Matei wrote: > Hi, > >>> Added aspm suspend/resume hooks that run >>> before and after suspend and resume to change >>> the ASPM states of the PCI bus in order to allow >>> the system suspend while trying to prevent card hangs >> >> Why is this needed? Is there a r8169 defect we're working around? >> A BIOS defect? Is there a problem report you can reference here? >> > > We encountered this issue while upgrading from kernel v6.1 to v6.6. > The system would not suspend with 6.6. We tracked down the problem to > the NIC of the device, mainly that the following code was removed in > 6.6: >> else if (tp->mac_version >= RTL_GIGA_MAC_VER_46) >> rc = pci_disable_link_state(pdev, PCIE_LINK_STATE_L1_2); > For the listed devices, ASPM L1 is disabled entirely in 6.6. As for > the reason, L1 was observed to cause some problems > (https://bugzilla.kernel.org/show_bug.cgi?id=217814). We use a Raptor > Lake soc and it won't change residency if the NIC doesn't have L1 > enabled. I saw in 6.1 the following comment: With residency you refer to the package power saving state? >> Chips from RTL8168h partially have issues with L1.2, but seem >> to work fine with L1 and L1.1. > I was thinking that disabling/enabling L1.1 on the fly before/after > suspend could help mitigate the risk associated with L1/L1.1 . I know > that ASPM settings are exposed in sysfs and that this could be done > from outside the kernel, that was my first approach, but it was > suggested to me that this kind of workaround would be better suited > for quirks. I did around 1000 suspend/resume cycles of 16-30 seconds > each (correcting the resume dev->bus->self being configured twice > mistake) and did not notice any problems. What do you think, is this a > good approach ... ? > If the root cause really should be in the SoC's ASPM implementation, then: - Other systems with the same SoC may suffer from the same problem, but are not covered by the quirk. - The issue may occur also with other devices than a RTL8168 NIC. How about e.g. RTL8125? Or completely different PCI devices? What I understand so far from your description: W/o ASPM L1 the SoC doesn't change "residency". See comment above, please elaborate on this. And w/ ASPM L1 the NIC hangs on suspend? What's the dmesg entries related to this hang? Tx timeout? Or card not accessible at all? My perspective so far: It's a relatively complex quirk that covers only a part of the potentially affected systems, and the issue isn't well understood. And most likely there are lots of systems out there with a Raptor Lake CPU and a RTL8168 on board. Therefore it's surprising that there hasn't been a similar report before. >>> + //configure device >>> + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, >>> + PCI_EXP_LNKCTL_ASPMC, 0); >>> + >>> + pci_read_config_word(dev->bus->self, >>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>> + &val); >>> + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; >>> + pci_write_config_word(dev->bus->self, >>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>> + val); >> Updates the parent (dev->bus->self) twice; was the first one supposed >> to update the device (dev)? > Yes, it was supposed to update the device (dev). It's my first time > sending a patch and I messed something up while doing some style > changes, I will correct it. I'm sorry for that. > >> This doesn't restore the state as it existed before suspend. Does >> this rely on other parts of restore to do that? > It operates on the assumption that after driver initialization > PCI_EXP_LNKCTL_ASPMC is 0 and that there are no states enabled in > CTL1. I did a lspci -vvv dump on the affected devices before and after > the quirks ran and saw no difference. This could be improved. > >> What is the RTL8168 chip version used on these systems? > It should be RTL8111H. > >> What's the root cause of the issue? >> A silicon bug on the host side? > I think it's the ASPM implementation of the soc. > >> ASPM L1 is disabled per default in r8169. So why is the patch needed >> at all? > Leaving it disabled all the time prevents the system from suspending. > This is not clear to me. You refer to STR? Why should a system not suspend just because one PCI device doesn't have ASPM L1 enabled? > Thank you, > George-Daniel Matei > > > > > > On Tue, Jul 9, 2024 at 12:15 AM Heiner Kallweit <hkallweit1@gmail.com> wrote: >> >> On 08.07.2024 19:23, Bjorn Helgaas wrote: >>> [+cc r8169 folks] >>> >>> On Mon, Jul 08, 2024 at 03:38:15PM +0000, George-Daniel Matei wrote: >>>> Added aspm suspend/resume hooks that run >>>> before and after suspend and resume to change >>>> the ASPM states of the PCI bus in order to allow >>>> the system suspend while trying to prevent card hangs >>> >>> Why is this needed? Is there a r8169 defect we're working around? >>> A BIOS defect? Is there a problem report you can reference here? >>> >> >> Basically the same question from my side. Apparently such a workaround >> isn't needed on any other system. And Realtek NICs can be found on more >> or less every consumer system. What's the root cause of the issue? >> A silicon bug on the host side? >> >> What is the RTL8168 chip version used on these systems? >> >> ASPM L1 is disabled per default in r8169. So why is the patch needed >> at all? >> >>> s/Added/Add/ >>> >>> s/aspm/ASPM/ above >>> >>> s/PCI bus/device and parent/ >>> >>> Add period at end of sentence. >>> >>> Rewrap to fill 75 columns. >>> >>>> Signed-off-by: George-Daniel Matei <danielgeorgem@chromium.org> >>>> --- >>>> drivers/pci/quirks.c | 142 +++++++++++++++++++++++++++++++++++++++++++ >>>> 1 file changed, 142 insertions(+) >>>> >>>> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c >>>> index dc12d4a06e21..aa3dba2211d3 100644 >>>> --- a/drivers/pci/quirks.c >>>> +++ b/drivers/pci/quirks.c >>>> @@ -6189,6 +6189,148 @@ DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56b0, aspm_l1_acceptable_latency >>>> DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56b1, aspm_l1_acceptable_latency); >>>> DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56c0, aspm_l1_acceptable_latency); >>>> DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56c1, aspm_l1_acceptable_latency); >>>> + >>>> +static const struct dmi_system_id chromebox_match_table[] = { >>>> + { >>>> + .matches = { >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Brask"), >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>> + } >>>> + }, >>>> + { >>>> + .matches = { >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Aurash"), >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>> + } >>>> + }, >>>> + { >>>> + .matches = { >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Bujia"), >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>> + } >>>> + }, >>>> + { >>>> + .matches = { >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Gaelin"), >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>> + } >>>> + }, >>>> + { >>>> + .matches = { >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Gladios"), >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>> + } >>>> + }, >>>> + { >>>> + .matches = { >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Hahn"), >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>> + } >>>> + }, >>>> + { >>>> + .matches = { >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Jeev"), >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>> + } >>>> + }, >>>> + { >>>> + .matches = { >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Kinox"), >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>> + } >>>> + }, >>>> + { >>>> + .matches = { >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Kuldax"), >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>> + } >>>> + }, >>>> + { >>>> + .matches = { >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Lisbon"), >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>> + } >>>> + }, >>>> + { >>>> + .matches = { >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Moli"), >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>> + } >>>> + }, >>>> + { } >>>> +}; >>>> + >>>> +static void rtl8169_suspend_aspm_settings(struct pci_dev *dev) >>>> +{ >>>> + u16 val = 0; >>>> + >>>> + if (dmi_check_system(chromebox_match_table)) { >>>> + //configure parent >>>> + pcie_capability_clear_and_set_word(dev->bus->self, >>>> + PCI_EXP_LNKCTL, >>>> + PCI_EXP_LNKCTL_ASPMC, >>>> + PCI_EXP_LNKCTL_ASPM_L1); >>>> + >>>> + pci_read_config_word(dev->bus->self, >>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>>> + &val); >>>> + val = (val & ~PCI_L1SS_CTL1_L1SS_MASK) | >>>> + PCI_L1SS_CTL1_PCIPM_L1_2 | PCI_L1SS_CTL1_PCIPM_L1_2 | >>>> + PCI_L1SS_CTL1_ASPM_L1_1; >>>> + pci_write_config_word(dev->bus->self, >>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>>> + val); >>>> + >>>> + //configure device >>>> + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, >>>> + PCI_EXP_LNKCTL_ASPMC, >>>> + PCI_EXP_LNKCTL_ASPM_L1); >>>> + >>>> + pci_read_config_word(dev, dev->l1ss + PCI_L1SS_CTL1, &val); >>>> + val = (val & ~PCI_L1SS_CTL1_L1SS_MASK) | >>>> + PCI_L1SS_CTL1_PCIPM_L1_2 | PCI_L1SS_CTL1_PCIPM_L1_2 | >>>> + PCI_L1SS_CTL1_ASPM_L1_1; >>>> + pci_write_config_word(dev, dev->l1ss + PCI_L1SS_CTL1, val); >>>> + } >>>> +} >>>> + >>>> +DECLARE_PCI_FIXUP_SUSPEND(PCI_VENDOR_ID_REALTEK, 0x8168, >>>> + rtl8169_suspend_aspm_settings); >>>> + >>>> +static void rtl8169_resume_aspm_settings(struct pci_dev *dev) >>>> +{ >>>> + u16 val = 0; >>>> + >>>> + if (dmi_check_system(chromebox_match_table)) { >>>> + //configure device >>>> + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, >>>> + PCI_EXP_LNKCTL_ASPMC, 0); >>>> + >>>> + pci_read_config_word(dev->bus->self, >>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>>> + &val); >>>> + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; >>>> + pci_write_config_word(dev->bus->self, >>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>>> + val); >>>> + >>>> + //configure parent >>>> + pcie_capability_clear_and_set_word(dev->bus->self, >>>> + PCI_EXP_LNKCTL, >>>> + PCI_EXP_LNKCTL_ASPMC, 0); >>>> + >>>> + pci_read_config_word(dev->bus->self, >>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>>> + &val); >>>> + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; >>>> + pci_write_config_word(dev->bus->self, >>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>>> + val); >>> >>> Updates the parent (dev->bus->self) twice; was the first one supposed >>> to update the device (dev)? >>> >>> This doesn't restore the state as it existed before suspend. Does >>> this rely on other parts of restore to do that? >>> >>>> + } >>>> +} >>>> + >>>> +DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_REALTEK, 0x8168, >>>> + rtl8169_resume_aspm_settings); >>>> #endif >>>> >>>> #ifdef CONFIG_PCIE_DPC >>>> -- >>>> 2.45.2.803.g4e1b14247a-goog >>>> >>
On Wed, Jul 10, 2024 at 05:09:08PM +0200, George-Daniel Matei wrote: > >> Added aspm suspend/resume hooks that run > >> before and after suspend and resume to change > >> the ASPM states of the PCI bus in order to allow > >> the system suspend while trying to prevent card hangs > > > > Why is this needed? Is there a r8169 defect we're working around? > > A BIOS defect? Is there a problem report you can reference here? > > We encountered this issue while upgrading from kernel v6.1 to v6.6. > The system would not suspend with 6.6. We tracked down the problem to > the NIC of the device, mainly that the following code was removed in > 6.6: > > > else if (tp->mac_version >= RTL_GIGA_MAC_VER_46) > > rc = pci_disable_link_state(pdev, PCIE_LINK_STATE_L1_2); > > For the listed devices, ASPM L1 is disabled entirely in 6.6. As for > the reason, L1 was observed to cause some problems > (https://bugzilla.kernel.org/show_bug.cgi?id=217814). We use a Raptor > Lake soc and it won't change residency if the NIC doesn't have L1 > enabled. I saw in 6.1 the following comment: Can you verify that the problem still exists in a current kernel, e.g., v6.9? If this is a regression that's still present in v6.9, we need to identify the commit that broke it. Maybe it's 90ca51e8c654 ("r8169: fix ASPM-related issues on a number of systems with NIC version from RTL8168h")? > > Chips from RTL8168h partially have issues with L1.2, but seem > > to work fine with L1 and L1.1. > > I was thinking that disabling/enabling L1.1 on the fly before/after > suspend could help mitigate the risk associated with L1/L1.1 . I know > that ASPM settings are exposed in sysfs and that this could be done > from outside the kernel, that was my first approach, but it was > suggested to me that this kind of workaround would be better suited > for quirks. I did around 1000 suspend/resume cycles of 16-30 seconds > each (correcting the resume dev->bus->self being configured twice > mistake) and did not notice any problems. What do you think, is this a > good approach ... ? Whatever the problem is, it definitely should be fixed in the kernel, and Ilpo is right that it *should* be done in the PCI core ASPM support (aspm.c) or at least with interfaces it supplies. Generally speaking, drivers should not need to touch ASPM at all except to work around hardware defects in their device, but r8169 has a long history of weird ASPM stuff. I dunno if that stuff is related to hardware defects in the r8169 devices or if it is workarounds for past or current defects in aspm.c. > > This doesn't restore the state as it existed before suspend. Does > > this rely on other parts of restore to do that? > > It operates on the assumption that after driver initialization > PCI_EXP_LNKCTL_ASPMC is 0 and that there are no states enabled in > CTL1. I did a lspci -vvv dump on the affected devices before and after > the quirks ran and saw no difference. This could be improved. Yep, we can't assume any of that because the PCI core owns ASPM config, not the driver itself. > > What's the root cause of the issue? > > A silicon bug on the host side? > > I think it's the ASPM implementation of the soc. As Heiner pointed out, if it's a SoC defect, it would potentially affect all devices and a workaround would have to cover them all. Side note: oops, quoting error below, see note about top-posting here: https://people.kernel.org/tglx/notes-about-netiquette > On Tue, Jul 9, 2024 at 12:15 AM Heiner Kallweit <hkallweit1@gmail.com> wrote: > > > > On 08.07.2024 19:23, Bjorn Helgaas wrote: > > > [+cc r8169 folks] > > > > > > On Mon, Jul 08, 2024 at 03:38:15PM +0000, George-Daniel Matei wrote: > > >> Added aspm suspend/resume hooks that run > > >> before and after suspend and resume to change > > >> the ASPM states of the PCI bus in order to allow > > >> the system suspend while trying to prevent card hangs > > > > > > Why is this needed? Is there a r8169 defect we're working around? > > > A BIOS defect? Is there a problem report you can reference here? > ...
On 10.07.2024 17:09, George-Daniel Matei wrote: > Hi, > >>> Added aspm suspend/resume hooks that run >>> before and after suspend and resume to change >>> the ASPM states of the PCI bus in order to allow >>> the system suspend while trying to prevent card hangs >> >> Why is this needed? Is there a r8169 defect we're working around? >> A BIOS defect? Is there a problem report you can reference here? >> > > We encountered this issue while upgrading from kernel v6.1 to v6.6. > The system would not suspend with 6.6. We tracked down the problem to > the NIC of the device, mainly that the following code was removed in > 6.6: >> else if (tp->mac_version >= RTL_GIGA_MAC_VER_46) >> rc = pci_disable_link_state(pdev, PCIE_LINK_STATE_L1_2);1 With this (older) 6.1 version everything is ok? Would mean that L1.1 is active and the system suspends (STR?) properly also with L1.1 being active. Under 6.6 per default L1 (incl. sub-states) is disabled. Then you manually enable L1 (incl. L1.1, but not L1.2?) via sysfs, and now the system hangs on suspend? Is this what you're saying? Would be strange because in both cases L1.1 is active when suspending. > For the listed devices, ASPM L1 is disabled entirely in 6.6. As for > the reason, L1 was observed to cause some problems > (https://bugzilla.kernel.org/show_bug.cgi?id=217814). We use a Raptor > Lake soc and it won't change residency if the NIC doesn't have L1 > enabled. I saw in 6.1 the following comment: >> Chips from RTL8168h partially have issues with L1.2, but seem >> to work fine with L1 and L1.1. > I was thinking that disabling/enabling L1.1 on the fly before/after > suspend could help mitigate the risk associated with L1/L1.1 . I know > that ASPM settings are exposed in sysfs and that this could be done > from outside the kernel, that was my first approach, but it was > suggested to me that this kind of workaround would be better suited > for quirks. I did around 1000 suspend/resume cycles of 16-30 seconds > each (correcting the resume dev->bus->self being configured twice > mistake) and did not notice any problems. What do you think, is this a > good approach ... ? > >>> + //configure device >>> + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, >>> + PCI_EXP_LNKCTL_ASPMC, 0); >>> + >>> + pci_read_config_word(dev->bus->self, >>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>> + &val); >>> + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; >>> + pci_write_config_word(dev->bus->self, >>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>> + val); >> Updates the parent (dev->bus->self) twice; was the first one supposed >> to update the device (dev)? > Yes, it was supposed to update the device (dev). It's my first time > sending a patch and I messed something up while doing some style > changes, I will correct it. I'm sorry for that. > >> This doesn't restore the state as it existed before suspend. Does >> this rely on other parts of restore to do that? > It operates on the assumption that after driver initialization > PCI_EXP_LNKCTL_ASPMC is 0 and that there are no states enabled in > CTL1. I did a lspci -vvv dump on the affected devices before and after > the quirks ran and saw no difference. This could be improved. > >> What is the RTL8168 chip version used on these systems? > It should be RTL8111H. > >> What's the root cause of the issue? >> A silicon bug on the host side? > I think it's the ASPM implementation of the soc. > >> ASPM L1 is disabled per default in r8169. So why is the patch needed >> at all? > Leaving it disabled all the time prevents the system from suspending. > > Thank you, > George-Daniel Matei > > > > > > On Tue, Jul 9, 2024 at 12:15 AM Heiner Kallweit <hkallweit1@gmail.com> wrote: >> >> On 08.07.2024 19:23, Bjorn Helgaas wrote: >>> [+cc r8169 folks] >>> >>> On Mon, Jul 08, 2024 at 03:38:15PM +0000, George-Daniel Matei wrote: >>>> Added aspm suspend/resume hooks that run >>>> before and after suspend and resume to change >>>> the ASPM states of the PCI bus in order to allow >>>> the system suspend while trying to prevent card hangs >>> >>> Why is this needed? Is there a r8169 defect we're working around? >>> A BIOS defect? Is there a problem report you can reference here? >>> >> >> Basically the same question from my side. Apparently such a workaround >> isn't needed on any other system. And Realtek NICs can be found on more >> or less every consumer system. What's the root cause of the issue? >> A silicon bug on the host side? >> >> What is the RTL8168 chip version used on these systems? >> >> ASPM L1 is disabled per default in r8169. So why is the patch needed >> at all? >> >>> s/Added/Add/ >>> >>> s/aspm/ASPM/ above >>> >>> s/PCI bus/device and parent/ >>> >>> Add period at end of sentence. >>> >>> Rewrap to fill 75 columns. >>> >>>> Signed-off-by: George-Daniel Matei <danielgeorgem@chromium.org> >>>> --- >>>> drivers/pci/quirks.c | 142 +++++++++++++++++++++++++++++++++++++++++++ >>>> 1 file changed, 142 insertions(+) >>>> >>>> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c >>>> index dc12d4a06e21..aa3dba2211d3 100644 >>>> --- a/drivers/pci/quirks.c >>>> +++ b/drivers/pci/quirks.c >>>> @@ -6189,6 +6189,148 @@ DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56b0, aspm_l1_acceptable_latency >>>> DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56b1, aspm_l1_acceptable_latency); >>>> DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56c0, aspm_l1_acceptable_latency); >>>> DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56c1, aspm_l1_acceptable_latency); >>>> + >>>> +static const struct dmi_system_id chromebox_match_table[] = { >>>> + { >>>> + .matches = { >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Brask"), >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>> + } >>>> + }, >>>> + { >>>> + .matches = { >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Aurash"), >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>> + } >>>> + }, >>>> + { >>>> + .matches = { >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Bujia"), >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>> + } >>>> + }, >>>> + { >>>> + .matches = { >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Gaelin"), >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>> + } >>>> + }, >>>> + { >>>> + .matches = { >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Gladios"), >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>> + } >>>> + }, >>>> + { >>>> + .matches = { >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Hahn"), >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>> + } >>>> + }, >>>> + { >>>> + .matches = { >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Jeev"), >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>> + } >>>> + }, >>>> + { >>>> + .matches = { >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Kinox"), >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>> + } >>>> + }, >>>> + { >>>> + .matches = { >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Kuldax"), >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>> + } >>>> + }, >>>> + { >>>> + .matches = { >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Lisbon"), >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>> + } >>>> + }, >>>> + { >>>> + .matches = { >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Moli"), >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>> + } >>>> + }, >>>> + { } >>>> +}; >>>> + >>>> +static void rtl8169_suspend_aspm_settings(struct pci_dev *dev) >>>> +{ >>>> + u16 val = 0; >>>> + >>>> + if (dmi_check_system(chromebox_match_table)) { >>>> + //configure parent >>>> + pcie_capability_clear_and_set_word(dev->bus->self, >>>> + PCI_EXP_LNKCTL, >>>> + PCI_EXP_LNKCTL_ASPMC, >>>> + PCI_EXP_LNKCTL_ASPM_L1); >>>> + >>>> + pci_read_config_word(dev->bus->self, >>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>>> + &val); >>>> + val = (val & ~PCI_L1SS_CTL1_L1SS_MASK) | >>>> + PCI_L1SS_CTL1_PCIPM_L1_2 | PCI_L1SS_CTL1_PCIPM_L1_2 | >>>> + PCI_L1SS_CTL1_ASPM_L1_1; >>>> + pci_write_config_word(dev->bus->self, >>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>>> + val); >>>> + >>>> + //configure device >>>> + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, >>>> + PCI_EXP_LNKCTL_ASPMC, >>>> + PCI_EXP_LNKCTL_ASPM_L1); >>>> + >>>> + pci_read_config_word(dev, dev->l1ss + PCI_L1SS_CTL1, &val); >>>> + val = (val & ~PCI_L1SS_CTL1_L1SS_MASK) | >>>> + PCI_L1SS_CTL1_PCIPM_L1_2 | PCI_L1SS_CTL1_PCIPM_L1_2 | >>>> + PCI_L1SS_CTL1_ASPM_L1_1; >>>> + pci_write_config_word(dev, dev->l1ss + PCI_L1SS_CTL1, val); >>>> + } >>>> +} >>>> + >>>> +DECLARE_PCI_FIXUP_SUSPEND(PCI_VENDOR_ID_REALTEK, 0x8168, >>>> + rtl8169_suspend_aspm_settings); >>>> + >>>> +static void rtl8169_resume_aspm_settings(struct pci_dev *dev) >>>> +{ >>>> + u16 val = 0; >>>> + >>>> + if (dmi_check_system(chromebox_match_table)) { >>>> + //configure device >>>> + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, >>>> + PCI_EXP_LNKCTL_ASPMC, 0); >>>> + >>>> + pci_read_config_word(dev->bus->self, >>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>>> + &val); >>>> + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; >>>> + pci_write_config_word(dev->bus->self, >>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>>> + val); >>>> + >>>> + //configure parent >>>> + pcie_capability_clear_and_set_word(dev->bus->self, >>>> + PCI_EXP_LNKCTL, >>>> + PCI_EXP_LNKCTL_ASPMC, 0); >>>> + >>>> + pci_read_config_word(dev->bus->self, >>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>>> + &val); >>>> + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; >>>> + pci_write_config_word(dev->bus->self, >>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>>> + val); >>> >>> Updates the parent (dev->bus->self) twice; was the first one supposed >>> to update the device (dev)? >>> >>> This doesn't restore the state as it existed before suspend. Does >>> this rely on other parts of restore to do that? >>> >>>> + } >>>> +} >>>> + >>>> +DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_REALTEK, 0x8168, >>>> + rtl8169_resume_aspm_settings); >>>> #endif >>>> >>>> #ifdef CONFIG_PCIE_DPC >>>> -- >>>> 2.45.2.803.g4e1b14247a-goog >>>> >>
On Thu, Jul 11, 2024 at 7:45 AM Heiner Kallweit <hkallweit1@gmail.com> wrote: > > On 10.07.2024 17:09, George-Daniel Matei wrote: > > Hi, > > > >>> Added aspm suspend/resume hooks that run > >>> before and after suspend and resume to change > >>> the ASPM states of the PCI bus in order to allow > >>> the system suspend while trying to prevent card hangs > >> > >> Why is this needed? Is there a r8169 defect we're working around? > >> A BIOS defect? Is there a problem report you can reference here? > >> > > > > We encountered this issue while upgrading from kernel v6.1 to v6.6. > > The system would not suspend with 6.6. We tracked down the problem to > > the NIC of the device, mainly that the following code was removed in > > 6.6: > >> else if (tp->mac_version >= RTL_GIGA_MAC_VER_46) > >> rc = pci_disable_link_state(pdev, PCIE_LINK_STATE_L1_2);1 > > With this (older) 6.1 version everything is ok? > Would mean that L1.1 is active and the system suspends (STR?) properly > also with L1.1 being active. > Yes, with 6.1 everything was ok. L1 was active and just the L1.1 substate was enabled, L1.2 was disabled. > Under 6.6 per default L1 (incl. sub-states) is disabled. > Then you manually enable L1 (incl. L1.1, but not L1.2?) via sysfs, > and now the system hangs on suspend? > Yes, in 6.6 L1 (+substates) is disabled. Like Bjorn mentioned, I think that is because of 90ca51e8c654 ("r8169: fix ASPM-related issues on a number of systems with NIC version from RTL8168h". With L1 disabled the system would not suspend so I enabled back L1 along with just L1.1 substate through sysfs, just to test, and saw that the system could suspend again. L1 is disabled by default for a reason, that's because it could cause tx timeouts. So to try to work around the possible timeouts I thought of changing the ASPM states before suspending and then restoring on resume. > Is this what you're saying? Would be strange because in both cases > L1.1 is active when suspending. > > > > For the listed devices, ASPM L1 is disabled entirely in 6.6. As for > > the reason, L1 was observed to cause some problems > > (https://bugzilla.kernel.org/show_bug.cgi?id=217814). We use a Raptor > > Lake soc and it won't change residency if the NIC doesn't have L1 > > enabled. I saw in 6.1 the following comment: > >> Chips from RTL8168h partially have issues with L1.2, but seem > >> to work fine with L1 and L1.1. > > I was thinking that disabling/enabling L1.1 on the fly before/after > > suspend could help mitigate the risk associated with L1/L1.1 . I know > > that ASPM settings are exposed in sysfs and that this could be done > > from outside the kernel, that was my first approach, but it was > > suggested to me that this kind of workaround would be better suited > > for quirks. I did around 1000 suspend/resume cycles of 16-30 seconds > > each (correcting the resume dev->bus->self being configured twice > > mistake) and did not notice any problems. What do you think, is this a > > good approach ... ? > > > >>> + //configure device > >>> + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, > >>> + PCI_EXP_LNKCTL_ASPMC, 0); > >>> + > >>> + pci_read_config_word(dev->bus->self, > >>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, > >>> + &val); > >>> + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; > >>> + pci_write_config_word(dev->bus->self, > >>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, > >>> + val); > >> Updates the parent (dev->bus->self) twice; was the first one supposed > >> to update the device (dev)? > > Yes, it was supposed to update the device (dev). It's my first time > > sending a patch and I messed something up while doing some style > > changes, I will correct it. I'm sorry for that. > > > >> This doesn't restore the state as it existed before suspend. Does > >> this rely on other parts of restore to do that? > > It operates on the assumption that after driver initialization > > PCI_EXP_LNKCTL_ASPMC is 0 and that there are no states enabled in > > CTL1. I did a lspci -vvv dump on the affected devices before and after > > the quirks ran and saw no difference. This could be improved. > > > >> What is the RTL8168 chip version used on these systems? > > It should be RTL8111H. > > > >> What's the root cause of the issue? > >> A silicon bug on the host side? > > I think it's the ASPM implementation of the soc. > > > >> ASPM L1 is disabled per default in r8169. So why is the patch needed > >> at all? > > Leaving it disabled all the time prevents the system from suspending. > > > > Thank you, > > George-Daniel Matei > > > > > > > > > > > > On Tue, Jul 9, 2024 at 12:15 AM Heiner Kallweit <hkallweit1@gmail.com> wrote: > >> > >> On 08.07.2024 19:23, Bjorn Helgaas wrote: > >>> [+cc r8169 folks] > >>> > >>> On Mon, Jul 08, 2024 at 03:38:15PM +0000, George-Daniel Matei wrote: > >>>> Added aspm suspend/resume hooks that run > >>>> before and after suspend and resume to change > >>>> the ASPM states of the PCI bus in order to allow > >>>> the system suspend while trying to prevent card hangs > >>> > >>> Why is this needed? Is there a r8169 defect we're working around? > >>> A BIOS defect? Is there a problem report you can reference here? > >>> > >> > >> Basically the same question from my side. Apparently such a workaround > >> isn't needed on any other system. And Realtek NICs can be found on more > >> or less every consumer system. What's the root cause of the issue? > >> A silicon bug on the host side? > >> > >> What is the RTL8168 chip version used on these systems? > >> > >> ASPM L1 is disabled per default in r8169. So why is the patch needed > >> at all? > >> > >>> s/Added/Add/ > >>> > >>> s/aspm/ASPM/ above > >>> > >>> s/PCI bus/device and parent/ > >>> > >>> Add period at end of sentence. > >>> > >>> Rewrap to fill 75 columns. > >>> > >>>> Signed-off-by: George-Daniel Matei <danielgeorgem@chromium.org> > >>>> --- > >>>> drivers/pci/quirks.c | 142 +++++++++++++++++++++++++++++++++++++++++++ > >>>> 1 file changed, 142 insertions(+) > >>>> > >>>> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c > >>>> index dc12d4a06e21..aa3dba2211d3 100644 > >>>> --- a/drivers/pci/quirks.c > >>>> +++ b/drivers/pci/quirks.c > >>>> @@ -6189,6 +6189,148 @@ DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56b0, aspm_l1_acceptable_latency > >>>> DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56b1, aspm_l1_acceptable_latency); > >>>> DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56c0, aspm_l1_acceptable_latency); > >>>> DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56c1, aspm_l1_acceptable_latency); > >>>> + > >>>> +static const struct dmi_system_id chromebox_match_table[] = { > >>>> + { > >>>> + .matches = { > >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Brask"), > >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >>>> + } > >>>> + }, > >>>> + { > >>>> + .matches = { > >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Aurash"), > >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >>>> + } > >>>> + }, > >>>> + { > >>>> + .matches = { > >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Bujia"), > >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >>>> + } > >>>> + }, > >>>> + { > >>>> + .matches = { > >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Gaelin"), > >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >>>> + } > >>>> + }, > >>>> + { > >>>> + .matches = { > >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Gladios"), > >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >>>> + } > >>>> + }, > >>>> + { > >>>> + .matches = { > >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Hahn"), > >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >>>> + } > >>>> + }, > >>>> + { > >>>> + .matches = { > >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Jeev"), > >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >>>> + } > >>>> + }, > >>>> + { > >>>> + .matches = { > >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Kinox"), > >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >>>> + } > >>>> + }, > >>>> + { > >>>> + .matches = { > >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Kuldax"), > >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >>>> + } > >>>> + }, > >>>> + { > >>>> + .matches = { > >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Lisbon"), > >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >>>> + } > >>>> + }, > >>>> + { > >>>> + .matches = { > >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Moli"), > >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >>>> + } > >>>> + }, > >>>> + { } > >>>> +}; > >>>> + > >>>> +static void rtl8169_suspend_aspm_settings(struct pci_dev *dev) > >>>> +{ > >>>> + u16 val = 0; > >>>> + > >>>> + if (dmi_check_system(chromebox_match_table)) { > >>>> + //configure parent > >>>> + pcie_capability_clear_and_set_word(dev->bus->self, > >>>> + PCI_EXP_LNKCTL, > >>>> + PCI_EXP_LNKCTL_ASPMC, > >>>> + PCI_EXP_LNKCTL_ASPM_L1); > >>>> + > >>>> + pci_read_config_word(dev->bus->self, > >>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, > >>>> + &val); > >>>> + val = (val & ~PCI_L1SS_CTL1_L1SS_MASK) | > >>>> + PCI_L1SS_CTL1_PCIPM_L1_2 | PCI_L1SS_CTL1_PCIPM_L1_2 | > >>>> + PCI_L1SS_CTL1_ASPM_L1_1; > >>>> + pci_write_config_word(dev->bus->self, > >>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, > >>>> + val); > >>>> + > >>>> + //configure device > >>>> + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, > >>>> + PCI_EXP_LNKCTL_ASPMC, > >>>> + PCI_EXP_LNKCTL_ASPM_L1); > >>>> + > >>>> + pci_read_config_word(dev, dev->l1ss + PCI_L1SS_CTL1, &val); > >>>> + val = (val & ~PCI_L1SS_CTL1_L1SS_MASK) | > >>>> + PCI_L1SS_CTL1_PCIPM_L1_2 | PCI_L1SS_CTL1_PCIPM_L1_2 | > >>>> + PCI_L1SS_CTL1_ASPM_L1_1; > >>>> + pci_write_config_word(dev, dev->l1ss + PCI_L1SS_CTL1, val); > >>>> + } > >>>> +} > >>>> + > >>>> +DECLARE_PCI_FIXUP_SUSPEND(PCI_VENDOR_ID_REALTEK, 0x8168, > >>>> + rtl8169_suspend_aspm_settings); > >>>> + > >>>> +static void rtl8169_resume_aspm_settings(struct pci_dev *dev) > >>>> +{ > >>>> + u16 val = 0; > >>>> + > >>>> + if (dmi_check_system(chromebox_match_table)) { > >>>> + //configure device > >>>> + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, > >>>> + PCI_EXP_LNKCTL_ASPMC, 0); > >>>> + > >>>> + pci_read_config_word(dev->bus->self, > >>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, > >>>> + &val); > >>>> + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; > >>>> + pci_write_config_word(dev->bus->self, > >>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, > >>>> + val); > >>>> + > >>>> + //configure parent > >>>> + pcie_capability_clear_and_set_word(dev->bus->self, > >>>> + PCI_EXP_LNKCTL, > >>>> + PCI_EXP_LNKCTL_ASPMC, 0); > >>>> + > >>>> + pci_read_config_word(dev->bus->self, > >>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, > >>>> + &val); > >>>> + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; > >>>> + pci_write_config_word(dev->bus->self, > >>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, > >>>> + val); > >>> > >>> Updates the parent (dev->bus->self) twice; was the first one supposed > >>> to update the device (dev)? > >>> > >>> This doesn't restore the state as it existed before suspend. Does > >>> this rely on other parts of restore to do that? > >>> > >>>> + } > >>>> +} > >>>> + > >>>> +DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_REALTEK, 0x8168, > >>>> + rtl8169_resume_aspm_settings); > >>>> #endif > >>>> > >>>> #ifdef CONFIG_PCIE_DPC > >>>> -- > >>>> 2.45.2.803.g4e1b14247a-goog > >>>> > >> >
On 16.07.2024 14:13, George-Daniel Matei wrote: > On Thu, Jul 11, 2024 at 7:45 AM Heiner Kallweit <hkallweit1@gmail.com> wrote: >> >> On 10.07.2024 17:09, George-Daniel Matei wrote: >>> Hi, >>> >>>>> Added aspm suspend/resume hooks that run >>>>> before and after suspend and resume to change >>>>> the ASPM states of the PCI bus in order to allow >>>>> the system suspend while trying to prevent card hangs >>>> >>>> Why is this needed? Is there a r8169 defect we're working around? >>>> A BIOS defect? Is there a problem report you can reference here? >>>> >>> >>> We encountered this issue while upgrading from kernel v6.1 to v6.6. >>> The system would not suspend with 6.6. We tracked down the problem to >>> the NIC of the device, mainly that the following code was removed in >>> 6.6: >>>> else if (tp->mac_version >= RTL_GIGA_MAC_VER_46) >>>> rc = pci_disable_link_state(pdev, PCIE_LINK_STATE_L1_2);1 >> >> With this (older) 6.1 version everything is ok? >> Would mean that L1.1 is active and the system suspends (STR?) properly >> also with L1.1 being active. >> > Yes, with 6.1 everything was ok. L1 was active and just the L1.1 substate > was enabled, L1.2 was disabled. > >> Under 6.6 per default L1 (incl. sub-states) is disabled. >> Then you manually enable L1 (incl. L1.1, but not L1.2?) via sysfs, >> and now the system hangs on suspend? >> > Yes, in 6.6 L1 (+substates) is disabled. Like Bjorn mentioned, I > think that is because of 90ca51e8c654 ("r8169: > fix ASPM-related issues on a number of systems with NIC version from > RTL8168h". With L1 disabled the system would not suspend so I enabled > back L1 along with just L1.1 substate through sysfs, just to test, and > saw that the system could It still sounds very weird that a system does not suspend to ram just because ASPM L1 is disabled for a single device. What if a PCI device is used which doesn't support ASPM? Which subsystem fails to suspend? Can you provide a log showing the suspend error? > suspend again. L1 is disabled by default for a reason, that's because > it could cause tx timeouts. So to try to work around the possible timeouts > I thought of changing the ASPM states before suspending and then > restoring on resume. > >> Is this what you're saying? Would be strange because in both cases >> L1.1 is active when suspending. >> >> >>> For the listed devices, ASPM L1 is disabled entirely in 6.6. As for >>> the reason, L1 was observed to cause some problems >>> (https://bugzilla.kernel.org/show_bug.cgi?id=217814). We use a Raptor >>> Lake soc and it won't change residency if the NIC doesn't have L1 >>> enabled. I saw in 6.1 the following comment: >>>> Chips from RTL8168h partially have issues with L1.2, but seem >>>> to work fine with L1 and L1.1. >>> I was thinking that disabling/enabling L1.1 on the fly before/after >>> suspend could help mitigate the risk associated with L1/L1.1 . I know >>> that ASPM settings are exposed in sysfs and that this could be done >>> from outside the kernel, that was my first approach, but it was >>> suggested to me that this kind of workaround would be better suited >>> for quirks. I did around 1000 suspend/resume cycles of 16-30 seconds >>> each (correcting the resume dev->bus->self being configured twice >>> mistake) and did not notice any problems. What do you think, is this a >>> good approach ... ? >>> >>>>> + //configure device >>>>> + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, >>>>> + PCI_EXP_LNKCTL_ASPMC, 0); >>>>> + >>>>> + pci_read_config_word(dev->bus->self, >>>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>>>> + &val); >>>>> + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; >>>>> + pci_write_config_word(dev->bus->self, >>>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>>>> + val); >>>> Updates the parent (dev->bus->self) twice; was the first one supposed >>>> to update the device (dev)? >>> Yes, it was supposed to update the device (dev). It's my first time >>> sending a patch and I messed something up while doing some style >>> changes, I will correct it. I'm sorry for that. >>> >>>> This doesn't restore the state as it existed before suspend. Does >>>> this rely on other parts of restore to do that? >>> It operates on the assumption that after driver initialization >>> PCI_EXP_LNKCTL_ASPMC is 0 and that there are no states enabled in >>> CTL1. I did a lspci -vvv dump on the affected devices before and after >>> the quirks ran and saw no difference. This could be improved. >>> >>>> What is the RTL8168 chip version used on these systems? >>> It should be RTL8111H. >>> >>>> What's the root cause of the issue? >>>> A silicon bug on the host side? >>> I think it's the ASPM implementation of the soc. >>> >>>> ASPM L1 is disabled per default in r8169. So why is the patch needed >>>> at all? >>> Leaving it disabled all the time prevents the system from suspending. >>> >>> Thank you, >>> George-Daniel Matei >>> >>> >>> >>> >>> >>> On Tue, Jul 9, 2024 at 12:15 AM Heiner Kallweit <hkallweit1@gmail.com> wrote: >>>> >>>> On 08.07.2024 19:23, Bjorn Helgaas wrote: >>>>> [+cc r8169 folks] >>>>> >>>>> On Mon, Jul 08, 2024 at 03:38:15PM +0000, George-Daniel Matei wrote: >>>>>> Added aspm suspend/resume hooks that run >>>>>> before and after suspend and resume to change >>>>>> the ASPM states of the PCI bus in order to allow >>>>>> the system suspend while trying to prevent card hangs >>>>> >>>>> Why is this needed? Is there a r8169 defect we're working around? >>>>> A BIOS defect? Is there a problem report you can reference here? >>>>> >>>> >>>> Basically the same question from my side. Apparently such a workaround >>>> isn't needed on any other system. And Realtek NICs can be found on more >>>> or less every consumer system. What's the root cause of the issue? >>>> A silicon bug on the host side? >>>> >>>> What is the RTL8168 chip version used on these systems? >>>> >>>> ASPM L1 is disabled per default in r8169. So why is the patch needed >>>> at all? >>>> >>>>> s/Added/Add/ >>>>> >>>>> s/aspm/ASPM/ above >>>>> >>>>> s/PCI bus/device and parent/ >>>>> >>>>> Add period at end of sentence. >>>>> >>>>> Rewrap to fill 75 columns. >>>>> >>>>>> Signed-off-by: George-Daniel Matei <danielgeorgem@chromium.org> >>>>>> --- >>>>>> drivers/pci/quirks.c | 142 +++++++++++++++++++++++++++++++++++++++++++ >>>>>> 1 file changed, 142 insertions(+) >>>>>> >>>>>> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c >>>>>> index dc12d4a06e21..aa3dba2211d3 100644 >>>>>> --- a/drivers/pci/quirks.c >>>>>> +++ b/drivers/pci/quirks.c >>>>>> @@ -6189,6 +6189,148 @@ DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56b0, aspm_l1_acceptable_latency >>>>>> DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56b1, aspm_l1_acceptable_latency); >>>>>> DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56c0, aspm_l1_acceptable_latency); >>>>>> DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56c1, aspm_l1_acceptable_latency); >>>>>> + >>>>>> +static const struct dmi_system_id chromebox_match_table[] = { >>>>>> + { >>>>>> + .matches = { >>>>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Brask"), >>>>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>>>> + } >>>>>> + }, >>>>>> + { >>>>>> + .matches = { >>>>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Aurash"), >>>>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>>>> + } >>>>>> + }, >>>>>> + { >>>>>> + .matches = { >>>>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Bujia"), >>>>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>>>> + } >>>>>> + }, >>>>>> + { >>>>>> + .matches = { >>>>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Gaelin"), >>>>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>>>> + } >>>>>> + }, >>>>>> + { >>>>>> + .matches = { >>>>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Gladios"), >>>>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>>>> + } >>>>>> + }, >>>>>> + { >>>>>> + .matches = { >>>>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Hahn"), >>>>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>>>> + } >>>>>> + }, >>>>>> + { >>>>>> + .matches = { >>>>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Jeev"), >>>>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>>>> + } >>>>>> + }, >>>>>> + { >>>>>> + .matches = { >>>>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Kinox"), >>>>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>>>> + } >>>>>> + }, >>>>>> + { >>>>>> + .matches = { >>>>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Kuldax"), >>>>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>>>> + } >>>>>> + }, >>>>>> + { >>>>>> + .matches = { >>>>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Lisbon"), >>>>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>>>> + } >>>>>> + }, >>>>>> + { >>>>>> + .matches = { >>>>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Moli"), >>>>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>>>> + } >>>>>> + }, >>>>>> + { } >>>>>> +}; >>>>>> + >>>>>> +static void rtl8169_suspend_aspm_settings(struct pci_dev *dev) >>>>>> +{ >>>>>> + u16 val = 0; >>>>>> + >>>>>> + if (dmi_check_system(chromebox_match_table)) { >>>>>> + //configure parent >>>>>> + pcie_capability_clear_and_set_word(dev->bus->self, >>>>>> + PCI_EXP_LNKCTL, >>>>>> + PCI_EXP_LNKCTL_ASPMC, >>>>>> + PCI_EXP_LNKCTL_ASPM_L1); >>>>>> + >>>>>> + pci_read_config_word(dev->bus->self, >>>>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>>>>> + &val); >>>>>> + val = (val & ~PCI_L1SS_CTL1_L1SS_MASK) | >>>>>> + PCI_L1SS_CTL1_PCIPM_L1_2 | PCI_L1SS_CTL1_PCIPM_L1_2 | >>>>>> + PCI_L1SS_CTL1_ASPM_L1_1; >>>>>> + pci_write_config_word(dev->bus->self, >>>>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>>>>> + val); >>>>>> + >>>>>> + //configure device >>>>>> + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, >>>>>> + PCI_EXP_LNKCTL_ASPMC, >>>>>> + PCI_EXP_LNKCTL_ASPM_L1); >>>>>> + >>>>>> + pci_read_config_word(dev, dev->l1ss + PCI_L1SS_CTL1, &val); >>>>>> + val = (val & ~PCI_L1SS_CTL1_L1SS_MASK) | >>>>>> + PCI_L1SS_CTL1_PCIPM_L1_2 | PCI_L1SS_CTL1_PCIPM_L1_2 | >>>>>> + PCI_L1SS_CTL1_ASPM_L1_1; >>>>>> + pci_write_config_word(dev, dev->l1ss + PCI_L1SS_CTL1, val); >>>>>> + } >>>>>> +} >>>>>> + >>>>>> +DECLARE_PCI_FIXUP_SUSPEND(PCI_VENDOR_ID_REALTEK, 0x8168, >>>>>> + rtl8169_suspend_aspm_settings); >>>>>> + >>>>>> +static void rtl8169_resume_aspm_settings(struct pci_dev *dev) >>>>>> +{ >>>>>> + u16 val = 0; >>>>>> + >>>>>> + if (dmi_check_system(chromebox_match_table)) { >>>>>> + //configure device >>>>>> + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, >>>>>> + PCI_EXP_LNKCTL_ASPMC, 0); >>>>>> + >>>>>> + pci_read_config_word(dev->bus->self, >>>>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>>>>> + &val); >>>>>> + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; >>>>>> + pci_write_config_word(dev->bus->self, >>>>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>>>>> + val); >>>>>> + >>>>>> + //configure parent >>>>>> + pcie_capability_clear_and_set_word(dev->bus->self, >>>>>> + PCI_EXP_LNKCTL, >>>>>> + PCI_EXP_LNKCTL_ASPMC, 0); >>>>>> + >>>>>> + pci_read_config_word(dev->bus->self, >>>>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>>>>> + &val); >>>>>> + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; >>>>>> + pci_write_config_word(dev->bus->self, >>>>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>>>>> + val); >>>>> >>>>> Updates the parent (dev->bus->self) twice; was the first one supposed >>>>> to update the device (dev)? >>>>> >>>>> This doesn't restore the state as it existed before suspend. Does >>>>> this rely on other parts of restore to do that? >>>>> >>>>>> + } >>>>>> +} >>>>>> + >>>>>> +DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_REALTEK, 0x8168, >>>>>> + rtl8169_resume_aspm_settings); >>>>>> #endif >>>>>> >>>>>> #ifdef CONFIG_PCIE_DPC >>>>>> -- >>>>>> 2.45.2.803.g4e1b14247a-goog >>>>>> >>>> >>
On Wed, Jul 10, 2024 at 9:59 PM Heiner Kallweit <hkallweit1@gmail.com> wrote: > > On 10.07.2024 17:09, George-Daniel Matei wrote: > > Hi, > > > >>> Added aspm suspend/resume hooks that run > >>> before and after suspend and resume to change > >>> the ASPM states of the PCI bus in order to allow > >>> the system suspend while trying to prevent card hangs > >> > >> Why is this needed? Is there a r8169 defect we're working around? > >> A BIOS defect? Is there a problem report you can reference here? > >> > > > > We encountered this issue while upgrading from kernel v6.1 to v6.6. > > The system would not suspend with 6.6. We tracked down the problem to > > the NIC of the device, mainly that the following code was removed in > > 6.6: > >> else if (tp->mac_version >= RTL_GIGA_MAC_VER_46) > >> rc = pci_disable_link_state(pdev, PCIE_LINK_STATE_L1_2); > > For the listed devices, ASPM L1 is disabled entirely in 6.6. As for > > the reason, L1 was observed to cause some problems > > (https://bugzilla.kernel.org/show_bug.cgi?id=217814). We use a Raptor > > Lake soc and it won't change residency if the NIC doesn't have L1 > > enabled. I saw in 6.1 the following comment: > > With residency you refer to the package power saving state? > Yes, by residency I'm referring to the package power saving state. > >> Chips from RTL8168h partially have issues with L1.2, but seem > >> to work fine with L1 and L1.1. > > I was thinking that disabling/enabling L1.1 on the fly before/after > > suspend could help mitigate the risk associated with L1/L1.1 . I know > > that ASPM settings are exposed in sysfs and that this could be done > > from outside the kernel, that was my first approach, but it was > > suggested to me that this kind of workaround would be better suited > > for quirks. I did around 1000 suspend/resume cycles of 16-30 seconds > > each (correcting the resume dev->bus->self being configured twice > > mistake) and did not notice any problems. What do you think, is this a > > good approach ... ? > > > If the root cause really should be in the SoC's ASPM implementation, then: > - Other systems with the same SoC may suffer from the same problem, > but are not covered by the quirk. > - The issue may occur also with other devices than a RTL8168 NIC. > How about e.g. RTL8125? Or completely different PCI devices? > > What I understand so far from your description: > > W/o ASPM L1 the SoC doesn't change "residency". See comment above, > please elaborate on this. > And w/ ASPM L1 the NIC hangs on suspend? > What's the dmesg entries related to this hang? Tx timeout? > Or card not accessible at all? > W/o ASPM L1 the SoC doesn't change power states, yes. With ASPM L1 the SoC changes power states. But for the L1 substates, L1.1 could cause tx timeouts as per 90ca51e8c654 ("r8169: fix ASPM-related issues on a number of systems with NIC version from RTL8168h". So L1 (and L1.1) couldn't be simply turned back on without the possibility of running into these tx timeouts. I never observed them while I experimented and tested with L1 and L1.1. > My perspective so far: > It's a relatively complex quirk that covers only a part of the potentially > affected systems, and the issue isn't well understood. > > And most likely there are lots of systems out there with a Raptor Lake CPU > and a RTL8168 on board. Therefore it's surprising that there hasn't been > a similar report before. > > > >>> + //configure device > >>> + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, > >>> + PCI_EXP_LNKCTL_ASPMC, 0); > >>> + > >>> + pci_read_config_word(dev->bus->self, > >>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, > >>> + &val); > >>> + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; > >>> + pci_write_config_word(dev->bus->self, > >>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, > >>> + val); > >> Updates the parent (dev->bus->self) twice; was the first one supposed > >> to update the device (dev)? > > Yes, it was supposed to update the device (dev). It's my first time > > sending a patch and I messed something up while doing some style > > changes, I will correct it. I'm sorry for that. > > > >> This doesn't restore the state as it existed before suspend. Does > >> this rely on other parts of restore to do that? > > It operates on the assumption that after driver initialization > > PCI_EXP_LNKCTL_ASPMC is 0 and that there are no states enabled in > > CTL1. I did a lspci -vvv dump on the affected devices before and after > > the quirks ran and saw no difference. This could be improved. > > > >> What is the RTL8168 chip version used on these systems? > > It should be RTL8111H. > > > >> What's the root cause of the issue? > >> A silicon bug on the host side? > > I think it's the ASPM implementation of the soc. > > > >> ASPM L1 is disabled per default in r8169. So why is the patch needed > >> at all? > > Leaving it disabled all the time prevents the system from suspending. > > > This is not clear to me. You refer to STR? > Why should a system not suspend just because one PCI device doesn't > have ASPM L1 enabled? I think some of my previous comments have been misleading. The actual problem is that the SoC doesn't change power states. intel_pmc_core INT33A1:00: CPU did not enter SLP_S0!!! In the case of the system I observed the problem, there is an EC, separate from the SoC, which will trigger the system to resume if following a suspend there is no power state transition detected after a period of time. That's why I was mentioning "can't suspend" but actually is a "can't achieve s0ix" problem. I searched for s0ix problems, I found: https://bugzilla.kernel.org/show_bug.cgi?id=207893 and it looks similar. What do you think ? Somehow not having L1 enabled makes it that the PCI controller have a state that doesn't meet the criteria to transition to s0ix. > > > Thank you, > > George-Daniel Matei > > > > > > > > > > > > On Tue, Jul 9, 2024 at 12:15 AM Heiner Kallweit <hkallweit1@gmail.com> wrote: > >> > >> On 08.07.2024 19:23, Bjorn Helgaas wrote: > >>> [+cc r8169 folks] > >>> > >>> On Mon, Jul 08, 2024 at 03:38:15PM +0000, George-Daniel Matei wrote: > >>>> Added aspm suspend/resume hooks that run > >>>> before and after suspend and resume to change > >>>> the ASPM states of the PCI bus in order to allow > >>>> the system suspend while trying to prevent card hangs > >>> > >>> Why is this needed? Is there a r8169 defect we're working around? > >>> A BIOS defect? Is there a problem report you can reference here? > >>> > >> > >> Basically the same question from my side. Apparently such a workaround > >> isn't needed on any other system. And Realtek NICs can be found on more > >> or less every consumer system. What's the root cause of the issue? > >> A silicon bug on the host side? > >> > >> What is the RTL8168 chip version used on these systems? > >> > >> ASPM L1 is disabled per default in r8169. So why is the patch needed > >> at all? > >> > >>> s/Added/Add/ > >>> > >>> s/aspm/ASPM/ above > >>> > >>> s/PCI bus/device and parent/ > >>> > >>> Add period at end of sentence. > >>> > >>> Rewrap to fill 75 columns. > >>> > >>>> Signed-off-by: George-Daniel Matei <danielgeorgem@chromium.org> > >>>> --- > >>>> drivers/pci/quirks.c | 142 +++++++++++++++++++++++++++++++++++++++++++ > >>>> 1 file changed, 142 insertions(+) > >>>> > >>>> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c > >>>> index dc12d4a06e21..aa3dba2211d3 100644 > >>>> --- a/drivers/pci/quirks.c > >>>> +++ b/drivers/pci/quirks.c > >>>> @@ -6189,6 +6189,148 @@ DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56b0, aspm_l1_acceptable_latency > >>>> DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56b1, aspm_l1_acceptable_latency); > >>>> DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56c0, aspm_l1_acceptable_latency); > >>>> DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56c1, aspm_l1_acceptable_latency); > >>>> + > >>>> +static const struct dmi_system_id chromebox_match_table[] = { > >>>> + { > >>>> + .matches = { > >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Brask"), > >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >>>> + } > >>>> + }, > >>>> + { > >>>> + .matches = { > >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Aurash"), > >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >>>> + } > >>>> + }, > >>>> + { > >>>> + .matches = { > >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Bujia"), > >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >>>> + } > >>>> + }, > >>>> + { > >>>> + .matches = { > >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Gaelin"), > >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >>>> + } > >>>> + }, > >>>> + { > >>>> + .matches = { > >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Gladios"), > >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >>>> + } > >>>> + }, > >>>> + { > >>>> + .matches = { > >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Hahn"), > >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >>>> + } > >>>> + }, > >>>> + { > >>>> + .matches = { > >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Jeev"), > >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >>>> + } > >>>> + }, > >>>> + { > >>>> + .matches = { > >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Kinox"), > >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >>>> + } > >>>> + }, > >>>> + { > >>>> + .matches = { > >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Kuldax"), > >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >>>> + } > >>>> + }, > >>>> + { > >>>> + .matches = { > >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Lisbon"), > >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >>>> + } > >>>> + }, > >>>> + { > >>>> + .matches = { > >>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Moli"), > >>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), > >>>> + } > >>>> + }, > >>>> + { } > >>>> +}; > >>>> + > >>>> +static void rtl8169_suspend_aspm_settings(struct pci_dev *dev) > >>>> +{ > >>>> + u16 val = 0; > >>>> + > >>>> + if (dmi_check_system(chromebox_match_table)) { > >>>> + //configure parent > >>>> + pcie_capability_clear_and_set_word(dev->bus->self, > >>>> + PCI_EXP_LNKCTL, > >>>> + PCI_EXP_LNKCTL_ASPMC, > >>>> + PCI_EXP_LNKCTL_ASPM_L1); > >>>> + > >>>> + pci_read_config_word(dev->bus->self, > >>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, > >>>> + &val); > >>>> + val = (val & ~PCI_L1SS_CTL1_L1SS_MASK) | > >>>> + PCI_L1SS_CTL1_PCIPM_L1_2 | PCI_L1SS_CTL1_PCIPM_L1_2 | > >>>> + PCI_L1SS_CTL1_ASPM_L1_1; > >>>> + pci_write_config_word(dev->bus->self, > >>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, > >>>> + val); > >>>> + > >>>> + //configure device > >>>> + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, > >>>> + PCI_EXP_LNKCTL_ASPMC, > >>>> + PCI_EXP_LNKCTL_ASPM_L1); > >>>> + > >>>> + pci_read_config_word(dev, dev->l1ss + PCI_L1SS_CTL1, &val); > >>>> + val = (val & ~PCI_L1SS_CTL1_L1SS_MASK) | > >>>> + PCI_L1SS_CTL1_PCIPM_L1_2 | PCI_L1SS_CTL1_PCIPM_L1_2 | > >>>> + PCI_L1SS_CTL1_ASPM_L1_1; > >>>> + pci_write_config_word(dev, dev->l1ss + PCI_L1SS_CTL1, val); > >>>> + } > >>>> +} > >>>> + > >>>> +DECLARE_PCI_FIXUP_SUSPEND(PCI_VENDOR_ID_REALTEK, 0x8168, > >>>> + rtl8169_suspend_aspm_settings); > >>>> + > >>>> +static void rtl8169_resume_aspm_settings(struct pci_dev *dev) > >>>> +{ > >>>> + u16 val = 0; > >>>> + > >>>> + if (dmi_check_system(chromebox_match_table)) { > >>>> + //configure device > >>>> + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, > >>>> + PCI_EXP_LNKCTL_ASPMC, 0); > >>>> + > >>>> + pci_read_config_word(dev->bus->self, > >>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, > >>>> + &val); > >>>> + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; > >>>> + pci_write_config_word(dev->bus->self, > >>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, > >>>> + val); > >>>> + > >>>> + //configure parent > >>>> + pcie_capability_clear_and_set_word(dev->bus->self, > >>>> + PCI_EXP_LNKCTL, > >>>> + PCI_EXP_LNKCTL_ASPMC, 0); > >>>> + > >>>> + pci_read_config_word(dev->bus->self, > >>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, > >>>> + &val); > >>>> + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; > >>>> + pci_write_config_word(dev->bus->self, > >>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, > >>>> + val); > >>> > >>> Updates the parent (dev->bus->self) twice; was the first one supposed > >>> to update the device (dev)? > >>> > >>> This doesn't restore the state as it existed before suspend. Does > >>> this rely on other parts of restore to do that? > >>> > >>>> + } > >>>> +} > >>>> + > >>>> +DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_REALTEK, 0x8168, > >>>> + rtl8169_resume_aspm_settings); > >>>> #endif > >>>> > >>>> #ifdef CONFIG_PCIE_DPC > >>>> -- > >>>> 2.45.2.803.g4e1b14247a-goog > >>>> > >> >
On 17.07.2024 19:12, George-Daniel Matei wrote: > On Wed, Jul 10, 2024 at 9:59 PM Heiner Kallweit <hkallweit1@gmail.com> wrote: >> >> On 10.07.2024 17:09, George-Daniel Matei wrote: >>> Hi, >>> >>>>> Added aspm suspend/resume hooks that run >>>>> before and after suspend and resume to change >>>>> the ASPM states of the PCI bus in order to allow >>>>> the system suspend while trying to prevent card hangs >>>> >>>> Why is this needed? Is there a r8169 defect we're working around? >>>> A BIOS defect? Is there a problem report you can reference here? >>>> >>> >>> We encountered this issue while upgrading from kernel v6.1 to v6.6. >>> The system would not suspend with 6.6. We tracked down the problem to >>> the NIC of the device, mainly that the following code was removed in >>> 6.6: >>>> else if (tp->mac_version >= RTL_GIGA_MAC_VER_46) >>>> rc = pci_disable_link_state(pdev, PCIE_LINK_STATE_L1_2); >>> For the listed devices, ASPM L1 is disabled entirely in 6.6. As for >>> the reason, L1 was observed to cause some problems >>> (https://bugzilla.kernel.org/show_bug.cgi?id=217814). We use a Raptor >>> Lake soc and it won't change residency if the NIC doesn't have L1 >>> enabled. I saw in 6.1 the following comment: >> >> With residency you refer to the package power saving state? >> > Yes, by residency I'm referring to the package power saving state. > >>>> Chips from RTL8168h partially have issues with L1.2, but seem >>>> to work fine with L1 and L1.1. >>> I was thinking that disabling/enabling L1.1 on the fly before/after >>> suspend could help mitigate the risk associated with L1/L1.1 . I know >>> that ASPM settings are exposed in sysfs and that this could be done >>> from outside the kernel, that was my first approach, but it was >>> suggested to me that this kind of workaround would be better suited >>> for quirks. I did around 1000 suspend/resume cycles of 16-30 seconds >>> each (correcting the resume dev->bus->self being configured twice >>> mistake) and did not notice any problems. What do you think, is this a >>> good approach ... ? >>> >> If the root cause really should be in the SoC's ASPM implementation, then: >> - Other systems with the same SoC may suffer from the same problem, >> but are not covered by the quirk. >> - The issue may occur also with other devices than a RTL8168 NIC. >> How about e.g. RTL8125? Or completely different PCI devices? >> >> What I understand so far from your description: >> >> W/o ASPM L1 the SoC doesn't change "residency". See comment above, >> please elaborate on this. >> And w/ ASPM L1 the NIC hangs on suspend? >> What's the dmesg entries related to this hang? Tx timeout? >> Or card not accessible at all? >> > W/o ASPM L1 the SoC doesn't change power states, yes. > With ASPM L1 the SoC changes power states. But for the L1 > substates, L1.1 could cause tx timeouts as per 90ca51e8c654 ("r8169: > fix ASPM-related issues on a number of systems with NIC version from > RTL8168h". So L1 (and L1.1) couldn't be simply turned back on without the > possibility of running into these tx timeouts. I never observed them while > I experimented and tested with L1 and L1.1. > >> My perspective so far: >> It's a relatively complex quirk that covers only a part of the potentially >> affected systems, and the issue isn't well understood. >> >> And most likely there are lots of systems out there with a Raptor Lake CPU >> and a RTL8168 on board. Therefore it's surprising that there hasn't been >> a similar report before. >> >> >>>>> + //configure device >>>>> + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, >>>>> + PCI_EXP_LNKCTL_ASPMC, 0); >>>>> + >>>>> + pci_read_config_word(dev->bus->self, >>>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>>>> + &val); >>>>> + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; >>>>> + pci_write_config_word(dev->bus->self, >>>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>>>> + val); >>>> Updates the parent (dev->bus->self) twice; was the first one supposed >>>> to update the device (dev)? >>> Yes, it was supposed to update the device (dev). It's my first time >>> sending a patch and I messed something up while doing some style >>> changes, I will correct it. I'm sorry for that. >>> >>>> This doesn't restore the state as it existed before suspend. Does >>>> this rely on other parts of restore to do that? >>> It operates on the assumption that after driver initialization >>> PCI_EXP_LNKCTL_ASPMC is 0 and that there are no states enabled in >>> CTL1. I did a lspci -vvv dump on the affected devices before and after >>> the quirks ran and saw no difference. This could be improved. >>> >>>> What is the RTL8168 chip version used on these systems? >>> It should be RTL8111H. >>> >>>> What's the root cause of the issue? >>>> A silicon bug on the host side? >>> I think it's the ASPM implementation of the soc. >>> >>>> ASPM L1 is disabled per default in r8169. So why is the patch needed >>>> at all? >>> Leaving it disabled all the time prevents the system from suspending. >>> >> This is not clear to me. You refer to STR? >> Why should a system not suspend just because one PCI device doesn't >> have ASPM L1 enabled? > I think some of my previous comments have been misleading. > The actual problem is that the SoC doesn't change power states. > This makes sense, thanks for the clarification. > intel_pmc_core INT33A1:00: CPU did not enter SLP_S0!!! > > In the case of the system I observed the problem, there is an > EC, separate from the SoC, which will trigger the system to resume > if following a suspend there is no power state transition > detected after a period of time. That's why I was mentioning > "can't suspend" but actually is a "can't achieve s0ix" problem. > > I searched for s0ix problems, I found: > https://bugzilla.kernel.org/show_bug.cgi?id=207893 > and it looks similar. What do you think ? > > Somehow not having L1 enabled makes it that the PCI controller > have a state that doesn't meet the criteria to transition to s0ix. > It's a known issue that disabling certain ASPM states can result in the the system not reaching deeper package power saving states. It has been reported several times, especially for notebooks and mini-pc servers, where users try to keep energy consumption at a minimum. It's a tradeoff which ASPM states to enable, and one of the reasons why ASPM states can be controlled from user space. The RTL8168 NIC family has a long history of ASPM issues. Partially it also seems to depend on the host system chipset. Root cause isn't known, Realtek doesn't provide datasheets or errata information. > > > >> >>> Thank you, >>> George-Daniel Matei >>> >>> >>> >>> >>> >>> On Tue, Jul 9, 2024 at 12:15 AM Heiner Kallweit <hkallweit1@gmail.com> wrote: >>>> >>>> On 08.07.2024 19:23, Bjorn Helgaas wrote: >>>>> [+cc r8169 folks] >>>>> >>>>> On Mon, Jul 08, 2024 at 03:38:15PM +0000, George-Daniel Matei wrote: >>>>>> Added aspm suspend/resume hooks that run >>>>>> before and after suspend and resume to change >>>>>> the ASPM states of the PCI bus in order to allow >>>>>> the system suspend while trying to prevent card hangs >>>>> >>>>> Why is this needed? Is there a r8169 defect we're working around? >>>>> A BIOS defect? Is there a problem report you can reference here? >>>>> >>>> >>>> Basically the same question from my side. Apparently such a workaround >>>> isn't needed on any other system. And Realtek NICs can be found on more >>>> or less every consumer system. What's the root cause of the issue? >>>> A silicon bug on the host side? >>>> >>>> What is the RTL8168 chip version used on these systems? >>>> >>>> ASPM L1 is disabled per default in r8169. So why is the patch needed >>>> at all? >>>> >>>>> s/Added/Add/ >>>>> >>>>> s/aspm/ASPM/ above >>>>> >>>>> s/PCI bus/device and parent/ >>>>> >>>>> Add period at end of sentence. >>>>> >>>>> Rewrap to fill 75 columns. >>>>> >>>>>> Signed-off-by: George-Daniel Matei <danielgeorgem@chromium.org> >>>>>> --- >>>>>> drivers/pci/quirks.c | 142 +++++++++++++++++++++++++++++++++++++++++++ >>>>>> 1 file changed, 142 insertions(+) >>>>>> >>>>>> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c >>>>>> index dc12d4a06e21..aa3dba2211d3 100644 >>>>>> --- a/drivers/pci/quirks.c >>>>>> +++ b/drivers/pci/quirks.c >>>>>> @@ -6189,6 +6189,148 @@ DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56b0, aspm_l1_acceptable_latency >>>>>> DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56b1, aspm_l1_acceptable_latency); >>>>>> DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56c0, aspm_l1_acceptable_latency); >>>>>> DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56c1, aspm_l1_acceptable_latency); >>>>>> + >>>>>> +static const struct dmi_system_id chromebox_match_table[] = { >>>>>> + { >>>>>> + .matches = { >>>>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Brask"), >>>>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>>>> + } >>>>>> + }, >>>>>> + { >>>>>> + .matches = { >>>>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Aurash"), >>>>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>>>> + } >>>>>> + }, >>>>>> + { >>>>>> + .matches = { >>>>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Bujia"), >>>>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>>>> + } >>>>>> + }, >>>>>> + { >>>>>> + .matches = { >>>>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Gaelin"), >>>>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>>>> + } >>>>>> + }, >>>>>> + { >>>>>> + .matches = { >>>>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Gladios"), >>>>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>>>> + } >>>>>> + }, >>>>>> + { >>>>>> + .matches = { >>>>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Hahn"), >>>>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>>>> + } >>>>>> + }, >>>>>> + { >>>>>> + .matches = { >>>>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Jeev"), >>>>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>>>> + } >>>>>> + }, >>>>>> + { >>>>>> + .matches = { >>>>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Kinox"), >>>>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>>>> + } >>>>>> + }, >>>>>> + { >>>>>> + .matches = { >>>>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Kuldax"), >>>>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>>>> + } >>>>>> + }, >>>>>> + { >>>>>> + .matches = { >>>>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Lisbon"), >>>>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>>>> + } >>>>>> + }, >>>>>> + { >>>>>> + .matches = { >>>>>> + DMI_MATCH(DMI_PRODUCT_NAME, "Moli"), >>>>>> + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), >>>>>> + } >>>>>> + }, >>>>>> + { } >>>>>> +}; >>>>>> + >>>>>> +static void rtl8169_suspend_aspm_settings(struct pci_dev *dev) >>>>>> +{ >>>>>> + u16 val = 0; >>>>>> + >>>>>> + if (dmi_check_system(chromebox_match_table)) { >>>>>> + //configure parent >>>>>> + pcie_capability_clear_and_set_word(dev->bus->self, >>>>>> + PCI_EXP_LNKCTL, >>>>>> + PCI_EXP_LNKCTL_ASPMC, >>>>>> + PCI_EXP_LNKCTL_ASPM_L1); >>>>>> + >>>>>> + pci_read_config_word(dev->bus->self, >>>>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>>>>> + &val); >>>>>> + val = (val & ~PCI_L1SS_CTL1_L1SS_MASK) | >>>>>> + PCI_L1SS_CTL1_PCIPM_L1_2 | PCI_L1SS_CTL1_PCIPM_L1_2 | >>>>>> + PCI_L1SS_CTL1_ASPM_L1_1; >>>>>> + pci_write_config_word(dev->bus->self, >>>>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>>>>> + val); >>>>>> + >>>>>> + //configure device >>>>>> + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, >>>>>> + PCI_EXP_LNKCTL_ASPMC, >>>>>> + PCI_EXP_LNKCTL_ASPM_L1); >>>>>> + >>>>>> + pci_read_config_word(dev, dev->l1ss + PCI_L1SS_CTL1, &val); >>>>>> + val = (val & ~PCI_L1SS_CTL1_L1SS_MASK) | >>>>>> + PCI_L1SS_CTL1_PCIPM_L1_2 | PCI_L1SS_CTL1_PCIPM_L1_2 | >>>>>> + PCI_L1SS_CTL1_ASPM_L1_1; >>>>>> + pci_write_config_word(dev, dev->l1ss + PCI_L1SS_CTL1, val); >>>>>> + } >>>>>> +} >>>>>> + >>>>>> +DECLARE_PCI_FIXUP_SUSPEND(PCI_VENDOR_ID_REALTEK, 0x8168, >>>>>> + rtl8169_suspend_aspm_settings); >>>>>> + >>>>>> +static void rtl8169_resume_aspm_settings(struct pci_dev *dev) >>>>>> +{ >>>>>> + u16 val = 0; >>>>>> + >>>>>> + if (dmi_check_system(chromebox_match_table)) { >>>>>> + //configure device >>>>>> + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, >>>>>> + PCI_EXP_LNKCTL_ASPMC, 0); >>>>>> + >>>>>> + pci_read_config_word(dev->bus->self, >>>>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>>>>> + &val); >>>>>> + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; >>>>>> + pci_write_config_word(dev->bus->self, >>>>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>>>>> + val); >>>>>> + >>>>>> + //configure parent >>>>>> + pcie_capability_clear_and_set_word(dev->bus->self, >>>>>> + PCI_EXP_LNKCTL, >>>>>> + PCI_EXP_LNKCTL_ASPMC, 0); >>>>>> + >>>>>> + pci_read_config_word(dev->bus->self, >>>>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>>>>> + &val); >>>>>> + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; >>>>>> + pci_write_config_word(dev->bus->self, >>>>>> + dev->bus->self->l1ss + PCI_L1SS_CTL1, >>>>>> + val); >>>>> >>>>> Updates the parent (dev->bus->self) twice; was the first one supposed >>>>> to update the device (dev)? >>>>> >>>>> This doesn't restore the state as it existed before suspend. Does >>>>> this rely on other parts of restore to do that? >>>>> >>>>>> + } >>>>>> +} >>>>>> + >>>>>> +DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_REALTEK, 0x8168, >>>>>> + rtl8169_resume_aspm_settings); >>>>>> #endif >>>>>> >>>>>> #ifdef CONFIG_PCIE_DPC >>>>>> -- >>>>>> 2.45.2.803.g4e1b14247a-goog >>>>>> >>>> >>
On Wed, Jul 10, 2024 at 11:38 PM Bjorn Helgaas <helgaas@kernel.org> wrote: > > On Wed, Jul 10, 2024 at 05:09:08PM +0200, George-Daniel Matei wrote: > > >> Added aspm suspend/resume hooks that run > > >> before and after suspend and resume to change > > >> the ASPM states of the PCI bus in order to allow > > >> the system suspend while trying to prevent card hangs > > > > > > Why is this needed? Is there a r8169 defect we're working around? > > > A BIOS defect? Is there a problem report you can reference here? > > > > We encountered this issue while upgrading from kernel v6.1 to v6.6. > > The system would not suspend with 6.6. We tracked down the problem to > > the NIC of the device, mainly that the following code was removed in > > 6.6: > > > > > else if (tp->mac_version >= RTL_GIGA_MAC_VER_46) > > > rc = pci_disable_link_state(pdev, PCIE_LINK_STATE_L1_2); > > > > For the listed devices, ASPM L1 is disabled entirely in 6.6. As for > > the reason, L1 was observed to cause some problems > > (https://bugzilla.kernel.org/show_bug.cgi?id=217814). We use a Raptor > > Lake soc and it won't change residency if the NIC doesn't have L1 > > enabled. I saw in 6.1 the following comment: > > Can you verify that the problem still exists in a current kernel, > e.g., v6.9? > I tested it with v6.9, still the same problem. > If this is a regression that's still present in v6.9, we need to > identify the commit that broke it. Maybe it's 90ca51e8c654 ("r8169: > fix ASPM-related issues on a number of systems with NIC version from > RTL8168h")? > I also tried v6.9 with 90ca51e8c654 reverted and it works ok. > > > Chips from RTL8168h partially have issues with L1.2, but seem > > > to work fine with L1 and L1.1. > > > > I was thinking that disabling/enabling L1.1 on the fly before/after > > suspend could help mitigate the risk associated with L1/L1.1 . I know > > that ASPM settings are exposed in sysfs and that this could be done > > from outside the kernel, that was my first approach, but it was > > suggested to me that this kind of workaround would be better suited > > for quirks. I did around 1000 suspend/resume cycles of 16-30 seconds > > each (correcting the resume dev->bus->self being configured twice > > mistake) and did not notice any problems. What do you think, is this a > > good approach ... ? > > Whatever the problem is, it definitely should be fixed in the kernel, > and Ilpo is right that it *should* be done in the PCI core ASPM > support (aspm.c) or at least with interfaces it supplies. > The problem is actually the system not being able to reach depper power saving states without certain ASPM states enabled. It was mentioned in the other thread replies that this kind of problem has been reported several times in the past. > Generally speaking, drivers should not need to touch ASPM at all > except to work around hardware defects in their device, but r8169 has > a long history of weird ASPM stuff. I dunno if that stuff is related > to hardware defects in the r8169 devices or if it is workarounds for > past or current defects in aspm.c. > What would be a good approach to move forward with this issue to get a fix approved? Make a general version of this toggle workaround in the aspm core that would be controllable & configurable for each pci device individually? Keep the quirks and fix the aforementioned comments? > > > This doesn't restore the state as it existed before suspend. Does > > > this rely on other parts of restore to do that? > > > > It operates on the assumption that after driver initialization > > PCI_EXP_LNKCTL_ASPMC is 0 and that there are no states enabled in > > CTL1. I did a lspci -vvv dump on the affected devices before and after > > the quirks ran and saw no difference. This could be improved. > > Yep, we can't assume any of that because the PCI core owns ASPM > config, not the driver itself. > > > > What's the root cause of the issue? > > > A silicon bug on the host side? > > > > I think it's the ASPM implementation of the soc. > > As Heiner pointed out, if it's a SoC defect, it would potentially > affect all devices and a workaround would have to cover them all. > > Side note: oops, quoting error below, see note about top-posting here: > https://people.kernel.org/tglx/notes-about-netiquette > > > On Tue, Jul 9, 2024 at 12:15 AM Heiner Kallweit <hkallweit1@gmail.com> wrote: > > > > > > On 08.07.2024 19:23, Bjorn Helgaas wrote: > > > > [+cc r8169 folks] > > > > > > > > On Mon, Jul 08, 2024 at 03:38:15PM +0000, George-Daniel Matei wrote: > > > >> Added aspm suspend/resume hooks that run > > > >> before and after suspend and resume to change > > > >> the ASPM states of the PCI bus in order to allow > > > >> the system suspend while trying to prevent card hangs > > > > > > > > Why is this needed? Is there a r8169 defect we're working around? > > > > A BIOS defect? Is there a problem report you can reference here? > > ...
[+cc Rafael in case you have suspend debug help] On Tue, Jul 16, 2024 at 09:25:40PM +0200, Heiner Kallweit wrote: > On 16.07.2024 14:13, George-Daniel Matei wrote: > > On Thu, Jul 11, 2024 at 7:45 AM Heiner Kallweit <hkallweit1@gmail.com> wrote: > >> On 10.07.2024 17:09, George-Daniel Matei wrote: > >>>>> Added aspm suspend/resume hooks that run > >>>>> before and after suspend and resume to change > >>>>> the ASPM states of the PCI bus in order to allow > >>>>> the system suspend while trying to prevent card hangs > >>>> > >>>> Why is this needed? Is there a r8169 defect we're working around? > >>>> A BIOS defect? Is there a problem report you can reference here? > >>> > >>> We encountered this issue while upgrading from kernel v6.1 to v6.6. > >>> The system would not suspend with 6.6. We tracked down the problem to > >>> the NIC of the device, mainly that the following code was removed in > >>> 6.6: > >>>> else if (tp->mac_version >= RTL_GIGA_MAC_VER_46) > >>>> rc = pci_disable_link_state(pdev, PCIE_LINK_STATE_L1_2);1 > >> > >> With this (older) 6.1 version everything is ok? > >> Would mean that L1.1 is active and the system suspends (STR?) properly > >> also with L1.1 being active. > >> > > Yes, with 6.1 everything was ok. L1 was active and just the L1.1 substate > > was enabled, L1.2 was disabled. > > > >> Under 6.6 per default L1 (incl. sub-states) is disabled. > >> Then you manually enable L1 (incl. L1.1, but not L1.2?) via sysfs, > >> and now the system hangs on suspend? > >> > > Yes, in 6.6 L1 (+substates) is disabled. Like Bjorn mentioned, I > > think that is because of 90ca51e8c654 ("r8169: > > fix ASPM-related issues on a number of systems with NIC version from > > RTL8168h". With L1 disabled the system would not suspend so I enabled > > back L1 along with just L1.1 substate through sysfs, just to test, and > > saw that the system could > > It still sounds very weird that a system does not suspend to ram > just because ASPM L1 is disabled for a single device. > What if a PCI device is used which doesn't support ASPM? > > Which subsystem fails to suspend? Can you provide a log showing > the suspend error? Can we push on this a little bit? The fact that suspend fails is super interesting to me. I'd like to know exactly how this fails and whether it's in the kernel or in firmware. If we're violating some assumption firmware is making, maybe there would be a more generic fix. How exactly do you suspend? Is there any debugging output you can collect while doing that? Maybe [1] has hints. I see a bunch of trace_suspend_resume() calls, and I think they're connected to [2], which looks like it might generate console/dmesg output, but I don't know how to enable that. [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/power/basic-pm-debugging.rst?id=v6.10 [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/trace/events/power.h?id=v6.10#n247
On 25.07.2024 14:56, George-Daniel Matei wrote: > On Wed, Jul 10, 2024 at 11:38 PM Bjorn Helgaas <helgaas@kernel.org> wrote: >> >> On Wed, Jul 10, 2024 at 05:09:08PM +0200, George-Daniel Matei wrote: >>>>> Added aspm suspend/resume hooks that run >>>>> before and after suspend and resume to change >>>>> the ASPM states of the PCI bus in order to allow >>>>> the system suspend while trying to prevent card hangs >>>> >>>> Why is this needed? Is there a r8169 defect we're working around? >>>> A BIOS defect? Is there a problem report you can reference here? >>> >>> We encountered this issue while upgrading from kernel v6.1 to v6.6. >>> The system would not suspend with 6.6. We tracked down the problem to >>> the NIC of the device, mainly that the following code was removed in >>> 6.6: >>> >>>> else if (tp->mac_version >= RTL_GIGA_MAC_VER_46) >>>> rc = pci_disable_link_state(pdev, PCIE_LINK_STATE_L1_2); >>> >>> For the listed devices, ASPM L1 is disabled entirely in 6.6. As for >>> the reason, L1 was observed to cause some problems >>> (https://bugzilla.kernel.org/show_bug.cgi?id=217814). We use a Raptor >>> Lake soc and it won't change residency if the NIC doesn't have L1 >>> enabled. I saw in 6.1 the following comment: >> >> Can you verify that the problem still exists in a current kernel, >> e.g., v6.9? >> > I tested it with v6.9, still the same problem. > >> If this is a regression that's still present in v6.9, we need to >> identify the commit that broke it. Maybe it's 90ca51e8c654 ("r8169: >> fix ASPM-related issues on a number of systems with NIC version from >> RTL8168h")? >> > I also tried v6.9 with 90ca51e8c654 reverted and it works ok. > >>>> Chips from RTL8168h partially have issues with L1.2, but seem >>>> to work fine with L1 and L1.1. >>> >>> I was thinking that disabling/enabling L1.1 on the fly before/after >>> suspend could help mitigate the risk associated with L1/L1.1 . I know >>> that ASPM settings are exposed in sysfs and that this could be done >>> from outside the kernel, that was my first approach, but it was >>> suggested to me that this kind of workaround would be better suited >>> for quirks. I did around 1000 suspend/resume cycles of 16-30 seconds >>> each (correcting the resume dev->bus->self being configured twice >>> mistake) and did not notice any problems. What do you think, is this a >>> good approach ... ? >> >> Whatever the problem is, it definitely should be fixed in the kernel, >> and Ilpo is right that it *should* be done in the PCI core ASPM >> support (aspm.c) or at least with interfaces it supplies. >> > The problem is actually the system not being able to reach > depper power saving states without certain ASPM states enabled. > It was mentioned in the other thread replies that this kind of problem > has been reported several times in the past. > This is a known side effect of disabling ASPM L1 per default. There isn't really something broken, the system just consumes some more power. If this is an issue for you and ASPM L1 causes no issues on your system, you can use sysfs to enable ASPM L1. The general issue is that there are at least hundreds of combinations of RTL8168 NICs, host chipsets, and BIOS versions. Several of these combinations cause serious issues if ASPM L1 is enabled. It's just unknown which ones, and why. Therefore ASPM L1 is disabled per default. >> Generally speaking, drivers should not need to touch ASPM at all >> except to work around hardware defects in their device, but r8169 has >> a long history of weird ASPM stuff. I dunno if that stuff is related >> to hardware defects in the r8169 devices or if it is workarounds for >> past or current defects in aspm.c. >> > What would be a good approach to move forward with this issue to > get a fix approved? > Propose a patch which fixes your power consumption issue and is guaranteed not to have negative impact on any other user. > Make a general version of this toggle workaround in the aspm core > that would be controllable & configurable for each pci device individually? > Keep the quirks and fix the aforementioned comments? > >>>> This doesn't restore the state as it existed before suspend. Does >>>> this rely on other parts of restore to do that? >>> >>> It operates on the assumption that after driver initialization >>> PCI_EXP_LNKCTL_ASPMC is 0 and that there are no states enabled in >>> CTL1. I did a lspci -vvv dump on the affected devices before and after >>> the quirks ran and saw no difference. This could be improved. >> >> Yep, we can't assume any of that because the PCI core owns ASPM >> config, not the driver itself. >> >>>> What's the root cause of the issue? >>>> A silicon bug on the host side? >>> >>> I think it's the ASPM implementation of the soc. >> >> As Heiner pointed out, if it's a SoC defect, it would potentially >> affect all devices and a workaround would have to cover them all. >> >> Side note: oops, quoting error below, see note about top-posting here: >> https://people.kernel.org/tglx/notes-about-netiquette >> >>> On Tue, Jul 9, 2024 at 12:15 AM Heiner Kallweit <hkallweit1@gmail.com> wrote: >>>> >>>> On 08.07.2024 19:23, Bjorn Helgaas wrote: >>>>> [+cc r8169 folks] >>>>> >>>>> On Mon, Jul 08, 2024 at 03:38:15PM +0000, George-Daniel Matei wrote: >>>>>> Added aspm suspend/resume hooks that run >>>>>> before and after suspend and resume to change >>>>>> the ASPM states of the PCI bus in order to allow >>>>>> the system suspend while trying to prevent card hangs >>>>> >>>>> Why is this needed? Is there a r8169 defect we're working around? >>>>> A BIOS defect? Is there a problem report you can reference here? >>> ...
diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index dc12d4a06e21..aa3dba2211d3 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -6189,6 +6189,148 @@ DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56b0, aspm_l1_acceptable_latency DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56b1, aspm_l1_acceptable_latency); DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56c0, aspm_l1_acceptable_latency); DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x56c1, aspm_l1_acceptable_latency); + +static const struct dmi_system_id chromebox_match_table[] = { + { + .matches = { + DMI_MATCH(DMI_PRODUCT_NAME, "Brask"), + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), + } + }, + { + .matches = { + DMI_MATCH(DMI_PRODUCT_NAME, "Aurash"), + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), + } + }, + { + .matches = { + DMI_MATCH(DMI_PRODUCT_NAME, "Bujia"), + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), + } + }, + { + .matches = { + DMI_MATCH(DMI_PRODUCT_NAME, "Gaelin"), + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), + } + }, + { + .matches = { + DMI_MATCH(DMI_PRODUCT_NAME, "Gladios"), + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), + } + }, + { + .matches = { + DMI_MATCH(DMI_PRODUCT_NAME, "Hahn"), + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), + } + }, + { + .matches = { + DMI_MATCH(DMI_PRODUCT_NAME, "Jeev"), + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), + } + }, + { + .matches = { + DMI_MATCH(DMI_PRODUCT_NAME, "Kinox"), + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), + } + }, + { + .matches = { + DMI_MATCH(DMI_PRODUCT_NAME, "Kuldax"), + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), + } + }, + { + .matches = { + DMI_MATCH(DMI_PRODUCT_NAME, "Lisbon"), + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), + } + }, + { + .matches = { + DMI_MATCH(DMI_PRODUCT_NAME, "Moli"), + DMI_MATCH(DMI_BIOS_VENDOR, "coreboot"), + } + }, + { } +}; + +static void rtl8169_suspend_aspm_settings(struct pci_dev *dev) +{ + u16 val = 0; + + if (dmi_check_system(chromebox_match_table)) { + //configure parent + pcie_capability_clear_and_set_word(dev->bus->self, + PCI_EXP_LNKCTL, + PCI_EXP_LNKCTL_ASPMC, + PCI_EXP_LNKCTL_ASPM_L1); + + pci_read_config_word(dev->bus->self, + dev->bus->self->l1ss + PCI_L1SS_CTL1, + &val); + val = (val & ~PCI_L1SS_CTL1_L1SS_MASK) | + PCI_L1SS_CTL1_PCIPM_L1_2 | PCI_L1SS_CTL1_PCIPM_L1_2 | + PCI_L1SS_CTL1_ASPM_L1_1; + pci_write_config_word(dev->bus->self, + dev->bus->self->l1ss + PCI_L1SS_CTL1, + val); + + //configure device + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, + PCI_EXP_LNKCTL_ASPMC, + PCI_EXP_LNKCTL_ASPM_L1); + + pci_read_config_word(dev, dev->l1ss + PCI_L1SS_CTL1, &val); + val = (val & ~PCI_L1SS_CTL1_L1SS_MASK) | + PCI_L1SS_CTL1_PCIPM_L1_2 | PCI_L1SS_CTL1_PCIPM_L1_2 | + PCI_L1SS_CTL1_ASPM_L1_1; + pci_write_config_word(dev, dev->l1ss + PCI_L1SS_CTL1, val); + } +} + +DECLARE_PCI_FIXUP_SUSPEND(PCI_VENDOR_ID_REALTEK, 0x8168, + rtl8169_suspend_aspm_settings); + +static void rtl8169_resume_aspm_settings(struct pci_dev *dev) +{ + u16 val = 0; + + if (dmi_check_system(chromebox_match_table)) { + //configure device + pcie_capability_clear_and_set_word(dev, PCI_EXP_LNKCTL, + PCI_EXP_LNKCTL_ASPMC, 0); + + pci_read_config_word(dev->bus->self, + dev->bus->self->l1ss + PCI_L1SS_CTL1, + &val); + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; + pci_write_config_word(dev->bus->self, + dev->bus->self->l1ss + PCI_L1SS_CTL1, + val); + + //configure parent + pcie_capability_clear_and_set_word(dev->bus->self, + PCI_EXP_LNKCTL, + PCI_EXP_LNKCTL_ASPMC, 0); + + pci_read_config_word(dev->bus->self, + dev->bus->self->l1ss + PCI_L1SS_CTL1, + &val); + val = val & ~PCI_L1SS_CTL1_L1SS_MASK; + pci_write_config_word(dev->bus->self, + dev->bus->self->l1ss + PCI_L1SS_CTL1, + val); + } +} + +DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_REALTEK, 0x8168, + rtl8169_resume_aspm_settings); #endif #ifdef CONFIG_PCIE_DPC
Added aspm suspend/resume hooks that run before and after suspend and resume to change the ASPM states of the PCI bus in order to allow the system suspend while trying to prevent card hangs Signed-off-by: George-Daniel Matei <danielgeorgem@chromium.org> --- drivers/pci/quirks.c | 142 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 142 insertions(+)