| Message ID | 20211126083119.16570-5-kishon@ti.com |
|---|---|
| State | New |
| Headers | show |
| Series | PCI: Keystone: Misc fixes for TI's AM65x PCIe | expand |
On Fri, Nov 26, 2021 at 02:01:18PM +0530, Kishon Vijay Abraham I wrote: > AM654 RootComplex has a hard coded 64 bit BAR of size 1MB and also has > both MSI and MSI-X capability in it's config space. If PCIEPORTBUS is > enabled, it tries to configure MSI-X and msix_mask_all() adds about 10 > Second boot up delay when it tries to write to undefined location. > > Add quirk to mark AM654 RC BAR flag as IORESOURCE_UNSET so that > msix_map_region() returns NULL for Root Complex and avoid un-desirable > writes to MSI-X table. I don't think this is the right fix (it is not even a fix, just a plaster to workaround an issue). What do you mean by "writing to an undefined location" ? What does "a hard coded BAR" mean ? What happens if we _rightly_ write into it (ie to size it) ? Lorenzo > Signed-off-by: Kishon Vijay Abraham I <kishon@ti.com> > --- > drivers/pci/controller/dwc/pci-keystone.c | 8 +++++++- > 1 file changed, 7 insertions(+), 1 deletion(-) > > diff --git a/drivers/pci/controller/dwc/pci-keystone.c b/drivers/pci/controller/dwc/pci-keystone.c > index 52d20fe17ee9..73e6626a0d8f 100644 > --- a/drivers/pci/controller/dwc/pci-keystone.c > +++ b/drivers/pci/controller/dwc/pci-keystone.c > @@ -557,8 +557,14 @@ static void ks_pcie_quirk(struct pci_dev *dev) > { 0, }, > }; > > - if (pci_is_root_bus(bus)) > + if (pci_is_root_bus(bus)) { > bridge = dev; > + if (pci_match_id(am6_pci_devids, bridge)) { > + struct resource *r = &dev->resource[0]; > + > + r->flags |= IORESOURCE_UNSET; > + } > + } > > /* look for the host bridge */ > while (!pci_is_root_bus(bus)) { > -- > 2.17.1 >
On Fri, Nov 26, 2021 at 2:31 AM Kishon Vijay Abraham I <kishon@ti.com> wrote: > > AM654 RootComplex has a hard coded 64 bit BAR of size 1MB and also has > both MSI and MSI-X capability in it's config space. If PCIEPORTBUS is > enabled, it tries to configure MSI-X and msix_mask_all() adds about 10 > Second boot up delay when it tries to write to undefined location. > > Add quirk to mark AM654 RC BAR flag as IORESOURCE_UNSET so that > msix_map_region() returns NULL for Root Complex and avoid un-desirable > writes to MSI-X table. > > Signed-off-by: Kishon Vijay Abraham I <kishon@ti.com> > --- > drivers/pci/controller/dwc/pci-keystone.c | 8 +++++++- > 1 file changed, 7 insertions(+), 1 deletion(-) > > diff --git a/drivers/pci/controller/dwc/pci-keystone.c b/drivers/pci/controller/dwc/pci-keystone.c > index 52d20fe17ee9..73e6626a0d8f 100644 > --- a/drivers/pci/controller/dwc/pci-keystone.c > +++ b/drivers/pci/controller/dwc/pci-keystone.c > @@ -557,8 +557,14 @@ static void ks_pcie_quirk(struct pci_dev *dev) > { 0, }, > }; > > - if (pci_is_root_bus(bus)) > + if (pci_is_root_bus(bus)) { The existing quirk has to be called for every device. But this quirk applies to just the RC, so can't you add another quirk and use the existing quirk infrastructure matching mechanism? > bridge = dev; > + if (pci_match_id(am6_pci_devids, bridge)) { > + struct resource *r = &dev->resource[0]; > + > + r->flags |= IORESOURCE_UNSET; > + } > + } > > /* look for the host bridge */ > while (!pci_is_root_bus(bus)) { > -- > 2.17.1 >
Hi Lorenzo, On 04/01/22 9:27 pm, Lorenzo Pieralisi wrote: > On Fri, Nov 26, 2021 at 02:01:18PM +0530, Kishon Vijay Abraham I wrote: >> AM654 RootComplex has a hard coded 64 bit BAR of size 1MB and also has >> both MSI and MSI-X capability in it's config space. If PCIEPORTBUS is >> enabled, it tries to configure MSI-X and msix_mask_all() adds about 10 >> Second boot up delay when it tries to write to undefined location. >> >> Add quirk to mark AM654 RC BAR flag as IORESOURCE_UNSET so that >> msix_map_region() returns NULL for Root Complex and avoid un-desirable >> writes to MSI-X table. > > I don't think this is the right fix (it is not even a fix, just a > plaster to workaround an issue). > > What do you mean by "writing to an undefined location" ? > > What does "a hard coded BAR" mean ? > > What happens if we _rightly_ write into it (ie to size it) ? There are two parts w.r.t setting the BAR; one is during the configuration and the other is during the enumeration. i) During the configuration, the size of the BAR is configured and the inbound ATU is configured to map the BAR to a physical memory. ii) During the enumeration, the size of the BAR is obtained and an address is allocated and programmed in the BAR. In the case of RC, for (i) above, the BAR size is configured as '0' https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/dwc/pcie-designware-host.c#n556 and the inbound ATU is not programmed at all. However, in the case of AM654, the HW configures BAR0 for a fixed size of 1MB (irrespective of what SW programmed in [i]). While this was done more for a endpoint usecase, since the same IP is configured for both RC mode and EP mode, the fixed BAR size is seen with RC mode as well. AM654 also has MSI-X capability for RC mode (the IP should have been ideally configured to have MSI-X capability for EP mode). This results in PCIEPORTBUS doing some undesired access in msix_mask_all(). Here I configure IORESOURCE_UNSET so that memory is not allocated for RC BAR. Thank You, Kishon > > Lorenzo > >> Signed-off-by: Kishon Vijay Abraham I <kishon@ti.com> >> --- >> drivers/pci/controller/dwc/pci-keystone.c | 8 +++++++- >> 1 file changed, 7 insertions(+), 1 deletion(-) >> >> diff --git a/drivers/pci/controller/dwc/pci-keystone.c b/drivers/pci/controller/dwc/pci-keystone.c >> index 52d20fe17ee9..73e6626a0d8f 100644 >> --- a/drivers/pci/controller/dwc/pci-keystone.c >> +++ b/drivers/pci/controller/dwc/pci-keystone.c >> @@ -557,8 +557,14 @@ static void ks_pcie_quirk(struct pci_dev *dev) >> { 0, }, >> }; >> >> - if (pci_is_root_bus(bus)) >> + if (pci_is_root_bus(bus)) { >> bridge = dev; >> + if (pci_match_id(am6_pci_devids, bridge)) { >> + struct resource *r = &dev->resource[0]; >> + >> + r->flags |= IORESOURCE_UNSET; >> + } >> + } >> >> /* look for the host bridge */ >> while (!pci_is_root_bus(bus)) { >> -- >> 2.17.1 >>
Hi Lorenzo, On 11/01/22 11:53 am, Kishon Vijay Abraham I wrote: > Hi Lorenzo, > > On 04/01/22 9:27 pm, Lorenzo Pieralisi wrote: >> On Fri, Nov 26, 2021 at 02:01:18PM +0530, Kishon Vijay Abraham I wrote: >>> AM654 RootComplex has a hard coded 64 bit BAR of size 1MB and also has >>> both MSI and MSI-X capability in it's config space. If PCIEPORTBUS is >>> enabled, it tries to configure MSI-X and msix_mask_all() adds about 10 >>> Second boot up delay when it tries to write to undefined location. >>> >>> Add quirk to mark AM654 RC BAR flag as IORESOURCE_UNSET so that >>> msix_map_region() returns NULL for Root Complex and avoid un-desirable >>> writes to MSI-X table. >> >> I don't think this is the right fix (it is not even a fix, just a >> plaster to workaround an issue). >> >> What do you mean by "writing to an undefined location" ? >> >> What does "a hard coded BAR" mean ? >> >> What happens if we _rightly_ write into it (ie to size it) ? > > There are two parts w.r.t setting the BAR; one is during the configuration and > the other is during the enumeration. > i) During the configuration, the size of the BAR is configured and the inbound > ATU is configured to map the BAR to a physical memory. > ii) During the enumeration, the size of the BAR is obtained and an address is > allocated and programmed in the BAR. > > In the case of RC, for (i) above, the BAR size is configured as '0' > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/dwc/pcie-designware-host.c#n556 > and the inbound ATU is not programmed at all. > > However, in the case of AM654, the HW configures BAR0 for a fixed size of 1MB > (irrespective of what SW programmed in [i]). While this was done more for a > endpoint usecase, since the same IP is configured for both RC mode and EP mode, > the fixed BAR size is seen with RC mode as well. AM654 also has MSI-X capability > for RC mode (the IP should have been ideally configured to have MSI-X capability > for EP mode). This results in PCIEPORTBUS doing some undesired access in > msix_mask_all(). > > Here I configure IORESOURCE_UNSET so that memory is not allocated for RC BAR. Do you need further clarifications on this? Thank You, Kishon > > Thank You, > Kishon > > >> >> Lorenzo >> >>> Signed-off-by: Kishon Vijay Abraham I <kishon@ti.com> >>> --- >>> drivers/pci/controller/dwc/pci-keystone.c | 8 +++++++- >>> 1 file changed, 7 insertions(+), 1 deletion(-) >>> >>> diff --git a/drivers/pci/controller/dwc/pci-keystone.c b/drivers/pci/controller/dwc/pci-keystone.c >>> index 52d20fe17ee9..73e6626a0d8f 100644 >>> --- a/drivers/pci/controller/dwc/pci-keystone.c >>> +++ b/drivers/pci/controller/dwc/pci-keystone.c >>> @@ -557,8 +557,14 @@ static void ks_pcie_quirk(struct pci_dev *dev) >>> { 0, }, >>> }; >>> >>> - if (pci_is_root_bus(bus)) >>> + if (pci_is_root_bus(bus)) { >>> bridge = dev; >>> + if (pci_match_id(am6_pci_devids, bridge)) { >>> + struct resource *r = &dev->resource[0]; >>> + >>> + r->flags |= IORESOURCE_UNSET; >>> + } >>> + } >>> >>> /* look for the host bridge */ >>> while (!pci_is_root_bus(bus)) { >>> -- >>> 2.17.1 >>>
On Fri, Feb 04, 2022 at 08:38:46PM +0530, Kishon Vijay Abraham I wrote: > Hi Lorenzo, > > On 11/01/22 11:53 am, Kishon Vijay Abraham I wrote: > > Hi Lorenzo, > > > > On 04/01/22 9:27 pm, Lorenzo Pieralisi wrote: > >> On Fri, Nov 26, 2021 at 02:01:18PM +0530, Kishon Vijay Abraham I wrote: > >>> AM654 RootComplex has a hard coded 64 bit BAR of size 1MB and also has > >>> both MSI and MSI-X capability in it's config space. If PCIEPORTBUS is > >>> enabled, it tries to configure MSI-X and msix_mask_all() adds about 10 > >>> Second boot up delay when it tries to write to undefined location. > >>> > >>> Add quirk to mark AM654 RC BAR flag as IORESOURCE_UNSET so that > >>> msix_map_region() returns NULL for Root Complex and avoid un-desirable > >>> writes to MSI-X table. > >> > >> I don't think this is the right fix (it is not even a fix, just a > >> plaster to workaround an issue). > >> > >> What do you mean by "writing to an undefined location" ? > >> > >> What does "a hard coded BAR" mean ? > >> > >> What happens if we _rightly_ write into it (ie to size it) ? > > > > There are two parts w.r.t setting the BAR; one is during the configuration and > > the other is during the enumeration. > > i) During the configuration, the size of the BAR is configured and the inbound > > ATU is configured to map the BAR to a physical memory. > > ii) During the enumeration, the size of the BAR is obtained and an address is > > allocated and programmed in the BAR. > > > > In the case of RC, for (i) above, the BAR size is configured as '0' > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/dwc/pcie-designware-host.c#n556 > > and the inbound ATU is not programmed at all. > > > > However, in the case of AM654, the HW configures BAR0 for a fixed size of 1MB > > (irrespective of what SW programmed in [i]). While this was done more for a > > endpoint usecase, since the same IP is configured for both RC mode and EP mode, > > the fixed BAR size is seen with RC mode as well. AM654 also has MSI-X capability > > for RC mode (the IP should have been ideally configured to have MSI-X capability > > for EP mode). This results in PCIEPORTBUS doing some undesired access in > > msix_mask_all(). > > > > Here I configure IORESOURCE_UNSET so that memory is not allocated for RC BAR. > > Do you need further clarifications on this? There are two things here: 1) As Rob mentioned, you can write it as a quirk applying only to the bridge _only_ 2) What you want is that the BAR should not be visible to the OS since it is not an actual resource. What I am questioning is whether your way of doing that complies with how this is done in the kernel for other devices/bridges. I need Bjorn's input on this since he knows better (especially wrt IORESOURCE_UNSET usage). I don't want to add any other IORESOURCE_UNSET usage that deviates from what's expected from it Lorenzo > > > >> > >> Lorenzo > >> > >>> Signed-off-by: Kishon Vijay Abraham I <kishon@ti.com> > >>> --- > >>> drivers/pci/controller/dwc/pci-keystone.c | 8 +++++++- > >>> 1 file changed, 7 insertions(+), 1 deletion(-) > >>> > >>> diff --git a/drivers/pci/controller/dwc/pci-keystone.c b/drivers/pci/controller/dwc/pci-keystone.c > >>> index 52d20fe17ee9..73e6626a0d8f 100644 > >>> --- a/drivers/pci/controller/dwc/pci-keystone.c > >>> +++ b/drivers/pci/controller/dwc/pci-keystone.c > >>> @@ -557,8 +557,14 @@ static void ks_pcie_quirk(struct pci_dev *dev) > >>> { 0, }, > >>> }; > >>> > >>> - if (pci_is_root_bus(bus)) > >>> + if (pci_is_root_bus(bus)) { > >>> bridge = dev; > >>> + if (pci_match_id(am6_pci_devids, bridge)) { > >>> + struct resource *r = &dev->resource[0]; > >>> + > >>> + r->flags |= IORESOURCE_UNSET; > >>> + } > >>> + } > >>> > >>> /* look for the host bridge */ > >>> while (!pci_is_root_bus(bus)) { > >>> -- > >>> 2.17.1 > >>>
On Tue, Feb 08, 2022 at 11:53:44AM +0000, Lorenzo Pieralisi wrote: > On Fri, Feb 04, 2022 at 08:38:46PM +0530, Kishon Vijay Abraham I wrote: > > On 11/01/22 11:53 am, Kishon Vijay Abraham I wrote: > > > On 04/01/22 9:27 pm, Lorenzo Pieralisi wrote: > > >> On Fri, Nov 26, 2021 at 02:01:18PM +0530, Kishon Vijay Abraham I wrote: > > >>> AM654 RootComplex has a hard coded 64 bit BAR of size 1MB and > > >>> also has both MSI and MSI-X capability in it's config space. > > >>> If PCIEPORTBUS is enabled, it tries to configure MSI-X and > > >>> msix_mask_all() adds about 10 Second boot up delay when it > > >>> tries to write to undefined location. s/AM654 RootComplex/The AM654 Root Complex/ But Root Complexes are not normally materialized as PCI devices with their own bus/device/function address, config space, BARs, etc. Sounds like this might really be a Root *Port*, not a Root Complex? s/it's config/its config/ s/10 Second/10 second/ > > >>> Add quirk to mark AM654 RC BAR flag as IORESOURCE_UNSET so > > >>> that msix_map_region() returns NULL for Root Complex and avoid > > >>> un-desirable writes to MSI-X table. > > >> > > >> I don't think this is the right fix (it is not even a fix, just > > >> a plaster to workaround an issue). > > >> > > >> What do you mean by "writing to an undefined location" ? > > >> > > >> What does "a hard coded BAR" mean ? > > >> > > >> What happens if we _rightly_ write into it (ie to size it) ? > > > > > > There are two parts w.r.t setting the BAR; one is during the > > > configuration and the other is during the enumeration. > > > i) During the configuration, the size of the BAR is configured > > > and the inbound ATU is configured to map the BAR to a > > > physical memory. > > > ii) During the enumeration, the size of the BAR is obtained and > > > an address is allocated and programmed in the BAR. > > > > > > In the case of RC, for (i) above, the BAR size is configured as '0' > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/controller/dwc/pcie-designware-host.c#n556 > > > and the inbound ATU is not programmed at all. > > > > > > However, in the case of AM654, the HW configures BAR0 for a > > > fixed size of 1MB (irrespective of what SW programmed in [i]). > > > While this was done more for a endpoint usecase, since the same > > > IP is configured for both RC mode and EP mode, the fixed BAR > > > size is seen with RC mode as well. This doesn't seem to quite answer Lorenzo's question. On AM654, does __pci_read_base() discover the 1MB size correctly for both RC and EP mode? What value should BAR0 contain in RC mode? Does the device respond at that address? Do we still need to ensure that 1MB address space is not assigned to any other device? > > > AM654 also has MSI-X capability for RC mode (the IP should have > > > been ideally configured to have MSI-X capability for EP mode). > > > This results in PCIEPORTBUS doing some undesired access in > > > msix_mask_all(). We have several quirks that set dev->no_msi. Maybe that would be a better way to prevent use of MSI-X (it would also prevent use of MSI; not sure whether that needs to be avoided as well)? > > > Here I configure IORESOURCE_UNSET so that memory is not > > > allocated for RC BAR. I guess this implies that this device (RP? I don't think RCs have BARs) never responds to PCI address space described by BAR0? > > Do you need further clarifications on this? > > There are two things here: > > 1) As Rob mentioned, you can write it as a quirk applying only to > the bridge _only_ > 2) What you want is that the BAR should not be visible to the OS > since it is not an actual resource. What I am questioning is > whether your way of doing that complies with how this is done in > the kernel for other devices/bridges. I need Bjorn's input on > this since he knows better (especially wrt IORESOURCE_UNSET > usage). I don't want to add any other IORESOURCE_UNSET usage that > deviates from what's expected from it If BAR0 should not exist at all as far as the OS is concerned, I think you should just set r->flags = 0. But that assumes the device never uses the value in the BAR, and it never responds using whatever address is there, even when PCI_COMMAND_MEMORY is set. > > >>> Signed-off-by: Kishon Vijay Abraham I <kishon@ti.com> > > >>> --- > > >>> drivers/pci/controller/dwc/pci-keystone.c | 8 +++++++- > > >>> 1 file changed, 7 insertions(+), 1 deletion(-) > > >>> > > >>> diff --git a/drivers/pci/controller/dwc/pci-keystone.c b/drivers/pci/controller/dwc/pci-keystone.c > > >>> index 52d20fe17ee9..73e6626a0d8f 100644 > > >>> --- a/drivers/pci/controller/dwc/pci-keystone.c > > >>> +++ b/drivers/pci/controller/dwc/pci-keystone.c > > >>> @@ -557,8 +557,14 @@ static void ks_pcie_quirk(struct pci_dev *dev) > > >>> { 0, }, > > >>> }; > > >>> > > >>> - if (pci_is_root_bus(bus)) > > >>> + if (pci_is_root_bus(bus)) { > > >>> bridge = dev; > > >>> + if (pci_match_id(am6_pci_devids, bridge)) { > > >>> + struct resource *r = &dev->resource[0]; > > >>> + > > >>> + r->flags |= IORESOURCE_UNSET; > > >>> + } > > >>> + } > > >>> > > >>> /* look for the host bridge */ > > >>> while (!pci_is_root_bus(bus)) { > > >>> -- > > >>> 2.17.1 > > >>>
diff --git a/drivers/pci/controller/dwc/pci-keystone.c b/drivers/pci/controller/dwc/pci-keystone.c index 52d20fe17ee9..73e6626a0d8f 100644 --- a/drivers/pci/controller/dwc/pci-keystone.c +++ b/drivers/pci/controller/dwc/pci-keystone.c @@ -557,8 +557,14 @@ static void ks_pcie_quirk(struct pci_dev *dev) { 0, }, }; - if (pci_is_root_bus(bus)) + if (pci_is_root_bus(bus)) { bridge = dev; + if (pci_match_id(am6_pci_devids, bridge)) { + struct resource *r = &dev->resource[0]; + + r->flags |= IORESOURCE_UNSET; + } + } /* look for the host bridge */ while (!pci_is_root_bus(bus)) {
AM654 RootComplex has a hard coded 64 bit BAR of size 1MB and also has both MSI and MSI-X capability in it's config space. If PCIEPORTBUS is enabled, it tries to configure MSI-X and msix_mask_all() adds about 10 Second boot up delay when it tries to write to undefined location. Add quirk to mark AM654 RC BAR flag as IORESOURCE_UNSET so that msix_map_region() returns NULL for Root Complex and avoid un-desirable writes to MSI-X table. Signed-off-by: Kishon Vijay Abraham I <kishon@ti.com> --- drivers/pci/controller/dwc/pci-keystone.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)