| Message ID | 20191209160638.141431-1-andre.przywara@arm.com |
|---|---|
| State | New |
| Headers | show |
| Series | pcie: Add quirk for the Arm Neoverse N1SDP platform | expand |
On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote: > From: Deepak Pandey <Deepak.Pandey@arm.com> > > The Arm N1SDP SoC suffers from some PCIe integration issues, most > prominently config space accesses to not existing BDFs being answered > with a bus abort, resulting in an SError. "Do as I say, not as I do"? > To mitigate this, the firmware scans the bus before boot (catching the > SErrors) and creates a table with valid BDFs, which acts as a filter for > Linux' config space accesses. > > Add code consulting the table as an ACPI PCIe quirk, also register the > corresponding device tree based description of the host controller. > Also fix the other two minor issues on the way, namely not being fully > ECAM compliant and config space accesses being restricted to 32-bit > accesses only. > > This allows the Arm Neoverse N1SDP board to boot Linux without crashing > and to access *any* devices (there are no platform devices except UART). > > Signed-off-by: Deepak Pandey <Deepak.Pandey@arm.com> > [Sudipto: extend to cover the CCIX root port as well] > Signed-off-by: Sudipto Paul <sudipto.paul@arm.com> > [Andre: fix coding style issues, rewrite some parts, add DT support] > Signed-off-by: Andre Przywara <andre.przywara@arm.com> > --- > arch/arm64/configs/defconfig | 1 + > drivers/acpi/pci_mcfg.c | 7 + > drivers/pci/controller/Kconfig | 11 ++ > drivers/pci/controller/Makefile | 1 + > drivers/pci/controller/pcie-n1sdp.c | 196 ++++++++++++++++++++++++++++ > include/linux/pci-ecam.h | 2 + > 6 files changed, 218 insertions(+) > create mode 100644 drivers/pci/controller/pcie-n1sdp.c Where can I buy one of these? They're "unreleased" according to: https://community.arm.com/developer/tools-software/oss-platforms/w/docs/440/neoverse-n1-sdp and I don't think we should wreck upstream because of a platform that doesn't exist. Will
On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote: > From: Deepak Pandey <Deepak.Pandey@arm.com> > > The Arm N1SDP SoC suffers from some PCIe integration issues, most > prominently config space accesses to not existing BDFs being answered > with a bus abort, resulting in an SError. Can we tease this apart a little more? Linux doesn't program all the bits that control error signaling, so even on hardware that works perfectly, much of this behavior is determined by what firmware did. I wonder if Linux could be more careful about this. "Bus abort" is not a term used in PCIe. IIUC, a config read to a device that doesn't exist should terminate with an Unsupported Request completion, e.g., see the implementation note in PCIe r5.0 sec 2.3.1. The UR should be an uncorrectable non-fatal error (Table 6-5), and Figures 6-2 and 6-3 show how it should be handled and when it should be signaled as a system error. In case you don't have a copy of the spec, I extracted those two figures and put them at [1]. Can you collect "lspci -vvxxx" output to see if we can correlate it with those figures and the behavior you see? [1] https://drive.google.com/file/d/1ihhdQvr0a7ZEJG-3gPddw1Tq7cTFAsah/view?usp=sharing > To mitigate this, the firmware scans the bus before boot (catching the > SErrors) and creates a table with valid BDFs, which acts as a filter for > Linux' config space accesses. > > Add code consulting the table as an ACPI PCIe quirk, also register the > corresponding device tree based description of the host controller. > Also fix the other two minor issues on the way, namely not being fully > ECAM compliant and config space accesses being restricted to 32-bit > accesses only. As I'm sure you've noticed, controllers that support only 32-bit config writes are not spec compliant and devices may not work correctly. The comment in pci_generic_config_write32() explains why. You may not trip over this problem frequently, but I wouldn't call it a "minor" issue because when you *do* trip over it, you have no indication that a register was corrupted. Even ECAM compliance is not really minor -- if this controller were fully compliant with the spec, you would need ZERO Linux changes to support it. Every quirk like this means additional maintenance burden, and it's not just a one-time thing. It means old kernels that *should* "just work" on your system will not work unless somebody backports the quirk. > This allows the Arm Neoverse N1SDP board to boot Linux without crashing > and to access *any* devices (there are no platform devices except UART).
On Tue, 10 Dec 2019 08:41:15 -0600 Bjorn Helgaas <helgaas@kernel.org> wrote: Hi Bjorn, > On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote: > > From: Deepak Pandey <Deepak.Pandey@arm.com> > > > > The Arm N1SDP SoC suffers from some PCIe integration issues, most > > prominently config space accesses to not existing BDFs being answered > > with a bus abort, resulting in an SError. > > Can we tease this apart a little more? Linux doesn't program all the > bits that control error signaling, so even on hardware that works > perfectly, much of this behavior is determined by what firmware did. > I wonder if Linux could be more careful about this. > > "Bus abort" is not a term used in PCIe. Yes, sorry, that was my sloppy term, also aiming more at the CPU side of the bus, between the cores and the RC. > IIUC, a config read to a > device that doesn't exist should terminate with an Unsupported Request > completion, e.g., see the implementation note in PCIe r5.0 sec 2.3.1. Yes, that's what Lorenzo mentioned as well. > The UR should be an uncorrectable non-fatal error (Table 6-5), and > Figures 6-2 and 6-3 show how it should be handled and when it should > be signaled as a system error. In case you don't have a copy of the > spec, I extracted those two figures and put them at [1]. Thanks for that. So in the last few months we tossed several ideas around how to work-around this without kernel intervention, all of them turned out to be not working. There are indeed registers in the RC that influence error reporting to the CPU side, but even if we could suppress (or catch) the SError, we can't recover and fixup the read transaction to the CPU. Even Lorenzo gave up on this ;-) As far as I understood this, there are gates missing which are supposed to translate this specific UR into a valid "all-1s" response. > Can you collect "lspci -vvxxx" output to see if we can correlate it > with those figures and the behavior you see? Attached. > > [1] https://drive.google.com/file/d/1ihhdQvr0a7ZEJG-3gPddw1Tq7cTFAsah/view?usp=sharing > > > To mitigate this, the firmware scans the bus before boot (catching the > > SErrors) and creates a table with valid BDFs, which acts as a filter for > > Linux' config space accesses. > > > > Add code consulting the table as an ACPI PCIe quirk, also register the > > corresponding device tree based description of the host controller. > > Also fix the other two minor issues on the way, namely not being fully > > ECAM compliant and config space accesses being restricted to 32-bit > > accesses only. > > As I'm sure you've noticed, controllers that support only 32-bit > config writes are not spec compliant and devices may not work > correctly. The comment in pci_generic_config_write32() explains why. Yes, I understand. Actually it seems to work fine in my experiments without that part of the patch, but the hardware guys insisted that it's needed. I will double check. > You may not trip over this problem frequently, but I wouldn't call it > a "minor" issue because when you *do* trip over it, you have no > indication that a register was corrupted. > > Even ECAM compliance is not really minor -- if this controller were > fully compliant with the spec, you would need ZERO Linux changes to > support it. Every quirk like this means additional maintenance > burden, and it's not just a one-time thing. It means old kernels that > *should* "just work" on your system will not work unless somebody > backports the quirk. I am well aware of that, and we had quite some discussions internally, with quite some opposition. The point is that this board has virtually everything behind PCI (which is good!), so not having working PCI renders this virtually useless. And installing Linux brings back warm and fuzzy memories of 1990's boot floppies - just without the actual disks ;-) So anything that improves this situation in any way is helpful. On the technical side this is a quirk, in the ACPI quirk framework(!), so it shouldn't affect anyone without the magic ACPI ID strings. People could even compile it out if needed. Plus the actual code is contained in a single, new file. So, yes, I see that it's not pretty and we should not have broken hardware in the first place, but this is probably as good as this will get. Cheers, Andre > > This allows the Arm Neoverse N1SDP board to boot Linux without crashing > > and to access *any* devices (there are no platform devices except UART). 0000:00:00.0 PCI bridge: ARM Device 0100 (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 30 Bus: primary=00, secondary=01, subordinate=19, sec-latency=0 I/O behind bridge: 00001000-00004fff Memory behind bridge: 71200000-71bfffff Prefetchable memory behind bridge: 0000000900000000-00000009006fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR+ NoISA+ VGA+ MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [80] Power Management version 3 Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [90] MSI: Enable+ Count=1/1 Maskable+ 64bit+ Address: 00000000fffff040 Data: 0000 Masking: 00000001 Pending: 00000000 Capabilities: [b0] MSI-X: Enable- Count=1 Masked- Vector table: BAR=0 offset=00000000 PBA: BAR=0 offset=00000008 Capabilities: [c0] Express (v2) Root Port (Slot-), MSI 00 DevCap: MaxPayload 1024 bytes, PhantFunc 0 ExtTag+ RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <256ns, L1 <8us ClockPM- Surprise- LLActRep- BwNot+ ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt+ ABWMgmt- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR+, OBFF Via message ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [150 v1] Device Serial Number 00-00-00-00-00-00-00-00 Capabilities: [274 v1] Transaction Processing Hints Interrupt vector mode supported Device specific mode supported Steering table in TPH capability structure Capabilities: [300 v1] #19 Capabilities: [900 v1] L1 PM Substates L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+ PortCommonModeRestoreTime=255us PortTPowerOnTime=26us L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- T_CommonMode=0us LTR1.2_Threshold=0ns L1SubCtl2: T_PwrOn=10us Kernel driver in use: pcieport 00: b5 13 00 01 07 04 10 00 00 00 04 06 00 00 01 00 10: 00 00 00 00 00 00 00 00 00 01 19 00 10 40 00 00 20: 20 71 b0 71 01 00 61 00 09 00 00 00 09 00 00 00 30: 00 00 00 00 80 00 00 00 00 00 00 00 ff 01 1e 00 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 80: 01 90 03 5a 08 00 00 00 00 00 00 00 00 00 00 00 90: 05 b0 81 01 40 f0 ff ff 00 00 00 00 00 00 00 00 a0: 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 b0: 11 c0 00 00 00 00 00 00 08 00 00 00 00 00 00 00 c0: 10 00 42 00 23 80 00 00 10 29 00 00 03 ad 61 00 d0: 00 00 02 41 00 00 00 00 00 00 40 00 00 00 00 00 e0: 00 00 00 00 32 18 75 00 00 00 00 00 0e 00 00 00 f0: 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0000:01:00.0 PCI bridge: PLX Technology, Inc. Device 8749 (rev ca) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 24 Region 0: Memory at 71b00000 (32-bit, non-prefetchable) [size=256K] Bus: primary=01, secondary=02, subordinate=08, sec-latency=0 I/O behind bridge: 00001000-00004fff Memory behind bridge: 71200000-71afffff Prefetchable memory behind bridge: 0000000900000000-00000009006fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [68] Express (v2) Upstream Port, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0 ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 0.000W DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L1, Exit Latency L0s <2us, L1 <4us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8749 Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [138 v1] Power Budgeting <?> Capabilities: [10c v1] #19 Capabilities: [148 v1] Virtual Channel Caps: LPEVC=1 RefClk=100ns PATEntryBits=8 Arb: Fixed+ WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=03 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=WRR64 TC/VC=ff Status: NegoPending- InProgress- Port Arbitration Table <?> VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00 Status: NegoPending+ InProgress- Capabilities: [e00 v1] #12 Capabilities: [b00 v1] Latency Tolerance Reporting Max snoop latency: 0ns Max no snoop latency: 0ns Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport 00: b5 10 49 87 07 00 10 00 ca 00 04 06 00 00 81 00 10: 00 00 b0 71 00 00 00 00 01 02 08 00 11 41 00 00 20: 20 71 a0 71 01 00 61 00 09 00 00 00 09 00 00 00 30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 02 00 40: 01 48 03 c8 08 00 00 00 05 68 86 01 00 00 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60: 00 00 00 00 00 00 00 00 10 a4 52 00 01 80 00 00 70: 10 08 09 00 02 59 41 00 00 00 02 01 00 00 00 00 80: 00 00 00 00 00 00 00 00 00 00 00 00 40 08 04 00 90: 00 00 00 00 06 01 00 00 02 00 00 00 00 00 00 00 a0: 00 00 00 00 0d 00 00 00 b5 10 49 87 00 00 00 00 b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0000:01:00.1 System peripheral: PLX Technology, Inc. Device 87d0 (rev ca) Subsystem: PLX Technology, Inc. Device 87d0 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin B routed to IRQ 255 Region 0: Memory at 71b40000 (32-bit, non-prefetchable) [disabled] [size=8K] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [68] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L1, Exit Latency L0s <2us, L1 <4us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via message DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [1f0 v1] Vendor Specific Information: ID=0010 Rev=0 Len=0c4 <?> Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> 00: b5 10 d0 87 00 00 10 00 ca 00 80 08 00 00 80 00 10: 00 00 b4 71 00 00 00 00 00 00 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 b5 10 d0 87 30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 02 00 00 40: 01 48 03 00 08 00 00 00 05 68 86 01 00 00 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60: 00 00 00 00 00 00 00 00 10 00 02 00 e4 8f 00 10 70: 10 29 00 00 02 59 41 00 00 00 02 01 00 00 00 00 80: 00 00 00 00 00 00 00 00 00 00 00 00 1f 08 04 00 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0000:01:00.2 System peripheral: PLX Technology, Inc. Device 87d0 (rev ca) Subsystem: PLX Technology, Inc. Device 87d0 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin B routed to IRQ 255 Region 0: Memory at 71b42000 (32-bit, non-prefetchable) [disabled] [size=8K] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [68] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L1, Exit Latency L0s <2us, L1 <4us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via message DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [1f0 v1] Vendor Specific Information: ID=0010 Rev=0 Len=0c4 <?> Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> 00: b5 10 d0 87 00 00 10 00 ca 00 80 08 00 00 80 00 10: 00 20 b4 71 00 00 00 00 00 00 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 b5 10 d0 87 30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 02 00 00 40: 01 48 03 00 08 00 00 00 05 68 86 01 00 00 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60: 00 00 00 00 00 00 00 00 10 00 02 00 e4 8f 00 10 70: 10 29 00 00 02 59 41 00 00 00 02 01 00 00 00 00 80: 00 00 00 00 00 00 00 00 00 00 00 00 1f 08 04 00 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0000:01:00.3 System peripheral: PLX Technology, Inc. Device 87d0 (rev ca) Subsystem: PLX Technology, Inc. Device 87d0 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin B routed to IRQ 255 Region 0: Memory at 71b44000 (32-bit, non-prefetchable) [disabled] [size=8K] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [68] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L1, Exit Latency L0s <2us, L1 <4us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via message DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [1f0 v1] Vendor Specific Information: ID=0010 Rev=0 Len=0c4 <?> Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> 00: b5 10 d0 87 00 00 10 00 ca 00 80 08 00 00 80 00 10: 00 40 b4 71 00 00 00 00 00 00 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 b5 10 d0 87 30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 02 00 00 40: 01 48 03 00 08 00 00 00 05 68 86 01 00 00 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60: 00 00 00 00 00 00 00 00 10 00 02 00 e4 8f 00 10 70: 10 29 00 00 02 59 41 00 00 00 02 01 00 00 00 00 80: 00 00 00 00 00 00 00 00 00 00 00 00 1f 08 04 00 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0000:01:00.4 System peripheral: PLX Technology, Inc. Device 87d0 (rev ca) Subsystem: PLX Technology, Inc. Device 87d0 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin B routed to IRQ 255 Region 0: Memory at 71b46000 (32-bit, non-prefetchable) [disabled] [size=8K] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [68] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L1, Exit Latency L0s <2us, L1 <4us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via message DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [1f0 v1] Vendor Specific Information: ID=0010 Rev=0 Len=0c4 <?> Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> 00: b5 10 d0 87 00 00 10 00 ca 00 80 08 00 00 80 00 10: 00 60 b4 71 00 00 00 00 00 00 00 00 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 b5 10 d0 87 30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 02 00 00 40: 01 48 03 00 08 00 00 00 05 68 86 01 00 00 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60: 00 00 00 00 00 00 00 00 10 00 02 00 e4 8f 00 10 70: 10 29 00 00 02 59 41 00 00 00 02 01 00 00 00 00 80: 00 00 00 00 00 00 00 00 00 00 00 00 1f 08 04 00 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0000:02:08.0 PCI bridge: PLX Technology, Inc. Device 8749 (rev ca) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 31 Bus: primary=02, secondary=03, subordinate=03, sec-latency=0 I/O behind bridge: 00001000-00001fff Memory behind bridge: 71200000-713fffff Prefetchable memory behind bridge: 0000000900000000-00000009001fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+ Address: 00000000fffff040 Data: 0000 Masking: 000000ff Pending: 00000000 Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #8, Speed 5GT/s, Width x16, ASPM L1, Exit Latency L0s <4us, L1 <4us ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+ LnkCtl: ASPM Disabled; Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- SltCap: AttnBtn+ PwrCtrl+ MRL+ AttnInd+ PwrInd+ HotPlug+ Surprise- Slot #8, PowerLimit 25.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Off, PwrInd Off, Power+ Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL+ CmdCplt- PresDet- Interlock- Changed: MRL- PresDet- LinkState- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8749 Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [138 v1] Power Budgeting <?> Capabilities: [10c v1] #19 Capabilities: [148 v1] Virtual Channel Caps: LPEVC=1 RefClk=100ns PATEntryBits=8 Arb: Fixed+ WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=03 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=WRR64 TC/VC=ff Status: NegoPending+ InProgress- Port Arbitration Table <?> VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00 Status: NegoPending+ InProgress- Capabilities: [e00 v1] #12 Capabilities: [f24 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport 00: b5 10 49 87 07 04 10 00 ca 00 04 06 00 00 01 00 10: 00 00 00 00 00 00 00 00 02 03 03 00 11 11 00 00 20: 20 71 30 71 01 00 11 00 09 00 00 00 09 00 00 00 30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 02 00 40: 01 48 03 c8 08 00 00 00 05 68 87 01 40 f0 ff ff 50: 00 00 00 00 00 00 00 00 ff 00 00 00 00 00 00 00 60: 00 00 00 00 00 00 00 00 10 a4 62 01 01 80 00 00 70: 10 08 09 00 02 69 79 08 00 00 01 00 df 0c 40 00 80: c0 07 20 00 00 00 00 00 00 00 00 00 60 08 04 00 90: 00 00 00 00 06 01 00 00 02 00 01 00 00 00 00 00 a0: 00 00 00 00 0d 00 00 00 b5 10 49 87 00 00 00 00 b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0000:02:10.0 PCI bridge: PLX Technology, Inc. Device 8749 (rev ca) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 32 Bus: primary=02, secondary=04, subordinate=04, sec-latency=0 Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+ Address: 00000000fffff040 Data: 0000 Masking: 000000ff Pending: 00000000 Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #16, Speed 5GT/s, Width x8, ASPM L1, Exit Latency L0s <4us, L1 <4us ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+ LnkCtl: ASPM Disabled; Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #16, PowerLimit 25.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock- Changed: MRL- PresDet- LinkState- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8749 Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [138 v1] Power Budgeting <?> Capabilities: [10c v1] #19 Capabilities: [148 v1] Virtual Channel Caps: LPEVC=1 RefClk=100ns PATEntryBits=8 Arb: Fixed+ WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=03 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=WRR64 TC/VC=ff Status: NegoPending+ InProgress- Port Arbitration Table <?> VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00 Status: NegoPending+ InProgress- Capabilities: [e00 v1] #12 Capabilities: [f24 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport 00: b5 10 49 87 07 04 10 00 ca 00 04 06 00 00 01 00 10: 00 00 00 00 00 00 00 00 02 04 04 00 f1 01 00 00 20: f0 ff 00 00 f1 ff 01 00 00 00 00 00 00 00 00 00 30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 02 00 40: 01 48 03 c8 08 00 00 00 05 68 87 01 40 f0 ff ff 50: 00 00 00 00 00 00 00 00 ff 00 00 00 00 00 00 00 60: 00 00 00 00 00 00 00 00 10 a4 62 01 01 80 00 00 70: 10 08 09 00 82 68 79 10 00 00 01 00 80 0c 80 00 80: 00 00 00 00 00 00 00 00 00 00 00 00 60 08 04 00 90: 00 00 00 00 06 01 00 00 02 00 01 00 00 00 00 00 a0: 00 00 00 00 0d 00 00 00 b5 10 49 87 00 00 00 00 b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0000:02:11.0 PCI bridge: PLX Technology, Inc. Device 8749 (rev ca) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 34 Bus: primary=02, secondary=05, subordinate=05, sec-latency=0 I/O behind bridge: 00002000-00002fff Memory behind bridge: 71400000-715fffff Prefetchable memory behind bridge: 0000000900200000-00000009003fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+ Address: 00000000fffff040 Data: 0000 Masking: 000000ff Pending: 00000000 Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #17, Speed 5GT/s, Width x2, ASPM L1, Exit Latency L0s <4us, L1 <4us ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+ LnkCtl: ASPM Disabled; Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- SltCap: AttnBtn+ PwrCtrl+ MRL+ AttnInd+ PwrInd+ HotPlug+ Surprise- Slot #17, PowerLimit 25.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Off, PwrInd Off, Power+ Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL+ CmdCplt- PresDet- Interlock- Changed: MRL- PresDet- LinkState- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8749 Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [138 v1] Power Budgeting <?> Capabilities: [10c v1] #19 Capabilities: [148 v1] Virtual Channel Caps: LPEVC=1 RefClk=100ns PATEntryBits=8 Arb: Fixed+ WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=03 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=WRR64 TC/VC=ff Status: NegoPending+ InProgress- Port Arbitration Table <?> VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00 Status: NegoPending+ InProgress- Capabilities: [e00 v1] #12 Capabilities: [f24 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport 00: b5 10 49 87 07 04 10 00 ca 00 04 06 00 00 01 00 10: 00 00 00 00 00 00 00 00 02 05 05 00 21 21 00 00 20: 40 71 50 71 21 00 31 00 09 00 00 00 09 00 00 00 30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 02 00 40: 01 48 03 c8 08 00 00 00 05 68 87 01 40 f0 ff ff 50: 00 00 00 00 00 00 00 00 ff 00 00 00 00 00 00 00 60: 00 00 00 00 00 00 00 00 10 a4 62 01 01 80 00 00 70: 10 08 09 00 22 68 79 11 00 00 01 00 df 0c 88 00 80: c0 07 20 00 00 00 00 00 00 00 00 00 60 08 04 00 90: 00 00 00 00 06 01 00 00 02 00 01 00 00 00 00 00 a0: 00 00 00 00 0d 00 00 00 b5 10 49 87 00 00 00 00 b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0000:02:12.0 PCI bridge: PLX Technology, Inc. Device 8749 (rev ca) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 35 Bus: primary=02, secondary=06, subordinate=06, sec-latency=0 I/O behind bridge: 00003000-00003fff Memory behind bridge: 71600000-716fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+ Address: 00000000fffff040 Data: 0000 Masking: 000000ff Pending: 00000000 Capabilities: [68] Express (v2) Downstream Port (Slot-), MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #18, Speed 5GT/s, Width x2, ASPM L1, Exit Latency L0s <2us, L1 <4us ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+ LnkCtl: ASPM Disabled; Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk- DLActive+ BWMgmt+ ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8749 Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [138 v1] Power Budgeting <?> Capabilities: [10c v1] #19 Capabilities: [148 v1] Virtual Channel Caps: LPEVC=1 RefClk=100ns PATEntryBits=1 Arb: Fixed+ WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00 Status: NegoPending+ InProgress- Capabilities: [e00 v1] #12 Capabilities: [f24 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport 00: b5 10 49 87 07 04 10 00 ca 00 04 06 00 00 01 00 10: 00 00 00 00 00 00 00 00 02 06 06 00 31 31 00 00 20: 60 71 60 71 f1 ff 01 00 00 00 00 00 00 00 00 00 30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 02 00 40: 01 48 03 c8 08 00 00 00 05 68 87 01 40 f0 ff ff 50: 00 00 00 00 00 00 00 00 ff 00 00 00 00 00 00 00 60: 00 00 00 00 00 00 00 00 10 a4 62 00 01 80 00 00 70: 10 08 09 00 22 58 79 12 00 00 12 60 80 0c 00 00 80: 00 00 40 00 00 00 00 00 00 00 00 00 60 08 04 00 90: 00 00 00 00 06 01 00 00 02 00 00 00 00 00 00 00 a0: 00 00 00 00 0d 00 00 00 b5 10 49 87 00 00 00 00 b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0000:02:13.0 PCI bridge: PLX Technology, Inc. Device 8749 (rev ca) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 36 Bus: primary=02, secondary=07, subordinate=07, sec-latency=0 I/O behind bridge: 00004000-00004fff Memory behind bridge: 71700000-717fffff Prefetchable memory behind bridge: 0000000900400000-00000009004fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+ Address: 00000000fffff040 Data: 0000 Masking: 000000ff Pending: 00000000 Capabilities: [68] Express (v2) Downstream Port (Slot-), MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #19, Speed 5GT/s, Width x2, ASPM L1, Exit Latency L0s <4us, L1 <4us ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+ LnkCtl: ASPM Disabled; Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive+ BWMgmt+ ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8749 Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [138 v1] Power Budgeting <?> Capabilities: [10c v1] #19 Capabilities: [148 v1] Virtual Channel Caps: LPEVC=1 RefClk=100ns PATEntryBits=1 Arb: Fixed+ WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00 Status: NegoPending+ InProgress- Capabilities: [e00 v1] #12 Capabilities: [f24 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport 00: b5 10 49 87 07 04 10 00 ca 00 04 06 00 00 01 00 10: 00 00 00 00 00 00 00 00 02 07 07 00 41 41 00 00 20: 70 71 70 71 41 00 41 00 09 00 00 00 09 00 00 00 30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 02 00 40: 01 48 03 c8 08 00 00 00 05 68 87 01 40 f0 ff ff 50: 00 00 00 00 00 00 00 00 ff 00 00 00 00 00 00 00 60: 00 00 00 00 00 00 00 00 10 a4 62 00 01 80 00 00 70: 10 08 09 00 22 68 79 13 00 00 11 60 80 0c 00 00 80: 00 00 40 00 00 00 00 00 00 00 00 00 60 08 04 00 90: 00 00 00 00 06 01 00 00 02 00 01 00 00 00 00 00 a0: 00 00 00 00 0d 00 00 00 b5 10 49 87 00 00 00 00 b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0000:02:14.0 PCI bridge: PLX Technology, Inc. Device 8749 (rev ca) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 37 Bus: primary=02, secondary=08, subordinate=08, sec-latency=0 Memory behind bridge: 71800000-718fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+ Address: 00000000fffff040 Data: 0000 Masking: 000000ff Pending: 00000000 Capabilities: [68] Express (v2) Downstream Port (Slot-), MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #20, Speed 5GT/s, Width x2, ASPM L1, Exit Latency L0s <2us, L1 <4us ClockPM- Surprise+ LLActRep+ BwNot+ ASPMOptComp+ LnkCtl: ASPM Disabled; Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk- DLActive+ BWMgmt+ ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8749 Capabilities: [100 v1] Device Serial Number ca-87-00-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [138 v1] Power Budgeting <?> Capabilities: [10c v1] #19 Capabilities: [148 v1] Virtual Channel Caps: LPEVC=1 RefClk=100ns PATEntryBits=1 Arb: Fixed+ WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00 Status: NegoPending+ InProgress- Capabilities: [e00 v1] #12 Capabilities: [f24 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+ ACSCtl: SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans- Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport 00: b5 10 49 87 07 04 10 00 ca 00 04 06 00 00 01 00 10: 00 00 00 00 00 00 00 00 02 08 08 00 f1 01 00 00 20: 80 71 80 71 f1 ff 01 00 00 00 00 00 00 00 00 00 30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 02 00 40: 01 48 03 c8 08 00 00 00 05 68 87 01 40 f0 ff ff 50: 00 00 00 00 00 00 00 00 ff 00 00 00 00 00 00 00 60: 00 00 00 00 00 00 00 00 10 a4 62 00 01 80 00 00 70: 10 08 09 00 22 58 79 14 00 00 12 60 80 0c 00 00 80: 00 00 40 00 00 00 00 00 00 00 00 00 60 08 04 00 90: 00 00 00 00 06 01 00 00 02 00 00 00 00 00 00 00 a0: 00 00 00 00 0d 00 00 00 b5 10 49 87 00 00 00 00 b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0000:06:00.0 SATA controller: Marvell Technology Group Ltd. Device 9170 (rev 12) (prog-if 01 [AHCI 1.0]) Subsystem: Marvell Technology Group Ltd. Device 9170 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 21 Region 0: I/O ports at 3010 [size=8] Region 1: I/O ports at 3020 [size=4] Region 2: I/O ports at 3018 [size=8] Region 3: I/O ports at 3024 [size=4] Region 4: I/O ports at 3000 [size=16] Region 5: Memory at 71610000 (32-bit, non-prefetchable) [size=512] Expansion ROM at 71600000 [disabled] [size=64K] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit- Address: fffff040 Data: 0000 Capabilities: [70] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Kernel driver in use: ahci 00: 4b 1b 70 91 07 04 10 00 12 01 06 01 00 00 00 00 10: 11 30 00 00 21 30 00 00 19 30 00 00 25 30 00 00 20: 01 30 00 00 00 00 61 71 00 00 00 00 4b 1b 70 91 30: 00 00 ff ff 40 00 00 00 00 00 00 00 ff 01 00 00 40: 01 50 03 40 00 00 00 00 00 00 00 00 00 00 00 00 50: 05 70 01 00 40 f0 ff ff 00 00 00 00 00 00 00 00 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 70: 10 00 12 00 02 87 64 00 10 20 09 00 12 3c 03 00 80: 00 00 12 10 00 00 00 00 00 00 00 00 00 00 00 00 90: 00 00 00 00 10 00 00 00 00 00 00 00 00 00 00 00 a0: 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0000:07:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c) Subsystem: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 22 Region 0: I/O ports at 4000 [size=256] Region 2: Memory at 71700000 (64-bit, non-prefetchable) [size=4K] Region 4: Memory at 900400000 (64-bit, prefetchable) [size=16K] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [70] Express (v2) Endpoint, MSI 01 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 25.000W DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via message/WAKE# DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [b0] MSI-X: Enable+ Count=4 Masked- Vector table: BAR=4 offset=00000000 PBA: BAR=4 offset=00000800 Capabilities: [d0] Vital Product Data Not readable Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [140 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- Capabilities: [160 v1] Device Serial Number 01-00-00-00-68-4c-e0-00 Capabilities: [170 v1] Latency Tolerance Reporting Max snoop latency: 0ns Max no snoop latency: 0ns Kernel driver in use: r8169 00: ec 10 68 81 07 04 10 00 0c 00 00 02 10 00 00 00 10: 01 40 00 00 00 00 00 00 04 00 70 71 00 00 00 00 20: 0c 00 40 00 09 00 00 00 00 00 00 00 ec 10 23 01 30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 00 00 40: 01 50 c3 ff 08 00 00 00 00 00 00 00 00 00 00 00 50: 05 70 80 00 00 00 00 00 00 00 00 00 00 00 00 00 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 70: 10 b0 02 02 c0 8c 64 00 10 20 19 00 11 7c 47 00 80: 00 00 11 10 00 00 00 00 00 00 00 00 00 00 00 00 90: 00 00 00 00 1f 08 0c 00 00 00 00 00 02 00 00 00 a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 b0: 11 d0 03 80 04 00 00 00 04 08 00 00 00 00 00 00 c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 d0: 03 00 00 80 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0000:08:00.0 USB controller: Texas Instruments TUSB73x0 SuperSpeed USB 3.0 xHCI Host Controller (rev 02) (prog-if 30 [XHCI]) Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 24 Region 0: Memory at 71800000 (64-bit, non-prefetchable) [size=64K] Region 2: Memory at 71810000 (64-bit, non-prefetchable) [size=8K] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable- Count=1/8 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [70] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 1024 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 25.000W DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <64us ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [c0] MSI-X: Enable+ Count=8 Masked- Vector table: BAR=2 offset=00000000 PBA: BAR=2 offset=00001000 Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [150 v1] Device Serial Number 08-00-28-00-00-20-00-00 Kernel driver in use: xhci_hcd 00: 4c 10 41 82 06 04 10 00 02 30 03 0c 10 00 00 00 10: 04 00 80 71 00 00 00 00 04 00 81 71 00 00 00 00 20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 30: 00 00 00 00 40 00 00 00 00 00 00 00 ff 01 00 00 40: 01 48 03 7e 08 00 00 00 05 70 86 00 00 00 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60: 30 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 70: 10 c0 02 00 c3 8f 64 00 10 20 09 00 12 5c 07 00 80: 00 00 12 10 00 00 00 00 00 00 00 00 00 00 00 00 90: 00 00 00 00 10 00 00 00 00 00 00 00 00 00 00 00 a0: 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 b0: 00 00 00 08 00 00 0f 00 00 00 00 00 00 00 00 00 c0: 11 00 07 80 02 00 00 00 02 10 00 00 00 00 00 00 d0: 00 00 00 00 ab 0d 00 00 1b 00 00 00 3f 00 00 00 e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0001:00:00.0 Host bridge: ARM Device 0100 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Interrupt: pin A routed to IRQ 255 Region 0: Memory at 2900000000 (64-bit, non-prefetchable) [size=4M] Bus: primary=00, secondary=01, subordinate=11, sec-latency=0 I/O behind bridge: 00000000-00000fff Prefetchable memory behind bridge: 00000000-000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR+ NoISA+ VGA+ MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [80] Power Management version 3 Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA PME(D0+,D1+,D2-,D3hot+,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [90] MSI: Enable- Count=1/1 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [b0] MSI-X: Enable- Count=1 Masked- Vector table: BAR=0 offset=00000000 PBA: BAR=0 offset=00000008 Capabilities: [c0] Express (v2) Root Port (Slot-), MSI 00 DevCap: MaxPayload 1024 bytes, PhantFunc 0 ExtTag+ RBE+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <256ns, L1 <8us ClockPM- Surprise- LLActRep- BwNot+ ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible- RootCap: CRSVisible- RootSta: PME ReqID 0000, PMEStatus- PMEPending- DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR+, OBFF Via message ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 16GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [150 v1] Device Serial Number 00-00-00-00-00-00-00-00 Capabilities: [274 v1] Transaction Processing Hints Interrupt vector mode supported Device specific mode supported Steering table in TPH capability structure Capabilities: [300 v1] #19 Capabilities: [4c0 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00 Status: NegoPending- InProgress- Capabilities: [5c0 v1] Address Translation Service (ATS) ATSCap: Invalidate Queue Depth: 01 ATSCtl: Enable-, Smallest Translation Unit: 00 Capabilities: [640 v1] Page Request Interface (PRI) PRICtl: Enable- Reset- PRISta: RF- UPRGI- Stopped+ Page Request Capacity: 00000001, Page Request Allocation: 00000000 Capabilities: [900 v1] L1 PM Substates L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+ PortCommonModeRestoreTime=255us PortTPowerOnTime=26us L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- T_CommonMode=0us LTR1.2_Threshold=0ns L1SubCtl2: T_PwrOn=10us Capabilities: [910 v1] #25 Capabilities: [920 v1] #27 Capabilities: [9c0 v1] #26 00: b5 13 00 01 07 00 10 00 00 00 00 06 00 00 01 00 10: 04 00 00 00 29 00 00 00 00 01 11 00 00 00 00 00 20: f0 ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 30: 00 00 00 00 80 00 00 00 00 00 00 00 ff 01 1e 00 40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 80: 01 90 03 5a 08 00 00 00 00 00 00 00 00 00 00 00 90: 05 b0 80 01 00 00 00 00 00 00 00 00 00 00 00 00 a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 b0: 11 c0 00 00 00 00 00 00 08 00 00 00 00 00 00 00 c0: 10 00 42 00 23 80 00 00 10 29 00 00 04 ad 61 00 d0: 00 00 01 01 00 00 00 00 00 00 40 00 00 00 00 00 e0: 00 00 00 00 32 18 75 00 00 00 00 00 1e 00 80 01 f0: 04 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
On Wed, Dec 11, 2019 at 11:00:49AM +0000, Andre Przywara wrote: > On Tue, 10 Dec 2019 08:41:15 -0600 > Bjorn Helgaas <helgaas@kernel.org> wrote: > > On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote: > > > From: Deepak Pandey <Deepak.Pandey@arm.com> > > > > > > The Arm N1SDP SoC suffers from some PCIe integration issues, most > > > prominently config space accesses to not existing BDFs being answered > > > with a bus abort, resulting in an SError. > > > > Can we tease this apart a little more? Linux doesn't program all the > > bits that control error signaling, so even on hardware that works > > perfectly, much of this behavior is determined by what firmware did. > > I wonder if Linux could be more careful about this. > > > > "Bus abort" is not a term used in PCIe. > > Yes, sorry, that was my sloppy term, also aiming more at the CPU > side of the bus, between the cores and the RC. > > > IIUC, a config read to a > > device that doesn't exist should terminate with an Unsupported Request > > completion, e.g., see the implementation note in PCIe r5.0 sec 2.3.1. > > Yes, that's what Lorenzo mentioned as well. > > > The UR should be an uncorrectable non-fatal error (Table 6-5), and > > Figures 6-2 and 6-3 show how it should be handled and when it should > > be signaled as a system error. In case you don't have a copy of the > > spec, I extracted those two figures and put them at [1]. > > Thanks for that. > So in the last few months we tossed several ideas around how to > work-around this without kernel intervention, all of them turned out > to be not working. There are indeed registers in the RC that > influence error reporting to the CPU side, but even if we could > suppress (or catch) the SError, we can't recover and fixup the read > transaction to the CPU. Even Lorenzo gave up on this ;-) As far as I > understood this, there are gates missing which are supposed to > translate this specific UR into a valid "all-1s" response. But the commit log says firmware scanned the bus (catching the SErrors). Shouldn't Linux be able to catch them the same way? The "all-1s" response directly from hardware is typical of most platforms, but I don't think it's strictly required by the PCIe spec and I don't think it's absolutely essential even to Linux. If you can catch the SErrors, isn't there a way for software to fabricate that all-1s data and continue after the read? > > Even ECAM compliance is not really minor -- if this controller were > > fully compliant with the spec, you would need ZERO Linux changes to > > support it. Every quirk like this means additional maintenance > > burden, and it's not just a one-time thing. It means old kernels that > > *should* "just work" on your system will not work unless somebody > > backports the quirk. > > I am well aware of that, and we had quite some discussions > internally, with quite some opposition. ... The main point is that *future* silicon should be designed to avoid this issue. I hope at least that part was not controversial. If we want to take advantage of the generic PCI code supplied by Linux, we have to expect that the hardware will play by the rules of PCI. Bjorn
On 11/12/2019 20:17, Bjorn Helgaas wrote: Hi Bjorn, > On Wed, Dec 11, 2019 at 11:00:49AM +0000, Andre Przywara wrote: >> On Tue, 10 Dec 2019 08:41:15 -0600 >> Bjorn Helgaas <helgaas@kernel.org> wrote: >>> On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote: >>>> From: Deepak Pandey <Deepak.Pandey@arm.com> >>>> >>>> The Arm N1SDP SoC suffers from some PCIe integration issues, most >>>> prominently config space accesses to not existing BDFs being answered >>>> with a bus abort, resulting in an SError. >>> >>> Can we tease this apart a little more? Linux doesn't program all the >>> bits that control error signaling, so even on hardware that works >>> perfectly, much of this behavior is determined by what firmware did. >>> I wonder if Linux could be more careful about this. >>> >>> "Bus abort" is not a term used in PCIe. >> >> Yes, sorry, that was my sloppy term, also aiming more at the CPU >> side of the bus, between the cores and the RC. >> >>> IIUC, a config read to a >>> device that doesn't exist should terminate with an Unsupported Request >>> completion, e.g., see the implementation note in PCIe r5.0 sec 2.3.1. >> >> Yes, that's what Lorenzo mentioned as well. >> >>> The UR should be an uncorrectable non-fatal error (Table 6-5), and >>> Figures 6-2 and 6-3 show how it should be handled and when it should >>> be signaled as a system error. In case you don't have a copy of the >>> spec, I extracted those two figures and put them at [1]. >> >> Thanks for that. >> So in the last few months we tossed several ideas around how to >> work-around this without kernel intervention, all of them turned out >> to be not working. There are indeed registers in the RC that >> influence error reporting to the CPU side, but even if we could >> suppress (or catch) the SError, we can't recover and fixup the read >> transaction to the CPU. Even Lorenzo gave up on this ;-) As far as I >> understood this, there are gates missing which are supposed to >> translate this specific UR into a valid "all-1s" response. > > But the commit log says firmware scanned the bus (catching the > SErrors). Shouldn't Linux be able to catch them the same way? Not really. The scanning is done by the SCP management processor, which is a Cortex-M class core on the same bus. So it's a simple, single core running very early after power-on, when the actual AP cores are still off. The SError handler is set to just increase a value, then to return. This value is then checked before and after a config space access for a given BDF: https://git.linaro.org/landing-teams/working/arm/n1sdp-pcie-quirk.git/tree/scp On the AP cores that run Linux later on this is quite different: The SError is asynchronous, imprecise (inexact) and has no syndrome information. That means we can't attribute this anymore to the faulting instruction, we don't even know if it happened due to this config space access. The CPU might have executed later instructions already, so the state is broken at this point. SError basically means: the system is screwed up. Because this is quite common for SErrors, we don't even allow to register SError handlers in arm64 Linux. So even if we could somehow handle this is in Linux, it would be a much greater and intrusive hack, so I'd rather stick with this version. > The "all-1s" response directly from hardware is typical of most > platforms, but I don't think it's strictly required by the PCIe spec > and I don't think it's absolutely essential even to Linux. If you can > catch the SErrors, isn't there a way for software to fabricate that > all-1s data and continue after the read? That was an idea we had as well, but due to the points mentioned above this is not possible. >>> Even ECAM compliance is not really minor -- if this controller were >>> fully compliant with the spec, you would need ZERO Linux changes to >>> support it. Every quirk like this means additional maintenance >>> burden, and it's not just a one-time thing. It means old kernels that >>> *should* "just work" on your system will not work unless somebody >>> backports the quirk. >> >> I am well aware of that, and we had quite some discussions >> internally, with quite some opposition. ... > > The main point is that *future* silicon should be designed to avoid > this issue. I hope at least that part was not controversial. Yes, the design goal was to be completely standards (SBSA, ACPI, ECAM) compliant, it was just the usual "things happen" that wasn't spotted in time. > If we want to take advantage of the generic PCI code supplied by > Linux, we have to expect that the hardware will play by the rules of > PCI. You don't need to convince me ;-), but I think the lesson has been learned. Cheers, Andre.
On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote: > From: Deepak Pandey <Deepak.Pandey@arm.com> > > The Arm N1SDP SoC suffers from some PCIe integration issues, most > prominently config space accesses to not existing BDFs being answered > with a bus abort, resulting in an SError. It wouldn't be a surprise if the host controller handled UR completions differently depending on if they were in response to a Type 0 configuration request or a Type 1 configuration request. (I think I've seen this before). Have you verified that you still get a bus abort when you attempt to perform a config read of a non-existent device downstream of the PCIe switch? (and thus as a response to a Type 1 request). I ask because if this is the case, and knowing that the PCIe switch is fixed, then it would be possible to simplify this quirk (by just making assumptions of the presence of devices in bus 0 rather than all the busses). > To mitigate this, the firmware scans the bus before boot (catching the > SErrors) and creates a table with valid BDFs, which acts as a filter for > Linux' config space accesses. > > Add code consulting the table as an ACPI PCIe quirk, also register the > corresponding device tree based description of the host controller. > Also fix the other two minor issues on the way, namely not being fully > ECAM compliant and config space accesses being restricted to 32-bit > accesses only. > > This allows the Arm Neoverse N1SDP board to boot Linux without crashing > and to access *any* devices (there are no platform devices except UART). This implies that this quirk has no side-effects and everything will work as expected - but this is only true for the simple case. For example hot plug won't work, SR-IOV, and others won't work. Also what happens for devices that return CRS? - does this also result in an abort? Does that mean that the firmware will consider these devices as not present instead of not ready yet? If this is an issue, then FLR of devices will also create issues (resulting in SErrors for users). I think it would be helpful to update this commit message to indicate that this makes it work better, but it may be broken in certain ways. > > Signed-off-by: Deepak Pandey <Deepak.Pandey@arm.com> > [Sudipto: extend to cover the CCIX root port as well] > Signed-off-by: Sudipto Paul <sudipto.paul@arm.com> > [Andre: fix coding style issues, rewrite some parts, add DT support] > Signed-off-by: Andre Przywara <andre.przywara@arm.com> > --- > arch/arm64/configs/defconfig | 1 + > drivers/acpi/pci_mcfg.c | 7 + > drivers/pci/controller/Kconfig | 11 ++ > drivers/pci/controller/Makefile | 1 + > drivers/pci/controller/pcie-n1sdp.c | 196 ++++++++++++++++++++++++++++ > include/linux/pci-ecam.h | 2 + > 6 files changed, 218 insertions(+) > create mode 100644 drivers/pci/controller/pcie-n1sdp.c > > diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig > index 6a83ba2aea3e..58124ef5070b 100644 > --- a/arch/arm64/configs/defconfig > +++ b/arch/arm64/configs/defconfig > @@ -177,6 +177,7 @@ CONFIG_NET_9P=y > CONFIG_NET_9P_VIRTIO=y > CONFIG_PCI=y > CONFIG_PCIEPORTBUS=y > +CONFIG_PCI_QUIRKS=y > CONFIG_PCI_IOV=y > CONFIG_HOTPLUG_PCI=y > CONFIG_HOTPLUG_PCI_ACPI=y > diff --git a/drivers/acpi/pci_mcfg.c b/drivers/acpi/pci_mcfg.c > index 6b347d9920cc..7a2b41b9ab57 100644 > --- a/drivers/acpi/pci_mcfg.c > +++ b/drivers/acpi/pci_mcfg.c > @@ -142,6 +142,13 @@ static struct mcfg_fixup mcfg_quirks[] = { > XGENE_V2_ECAM_MCFG(4, 0), > XGENE_V2_ECAM_MCFG(4, 1), > XGENE_V2_ECAM_MCFG(4, 2), > + > +#define N1SDP_ECAM_MCFG(rev, seg, ops) \ > + {"ARMLTD", "ARMN1SDP", rev, seg, MCFG_BUS_ANY, ops } > + > + /* N1SDP SoC with v1 PCIe controller */ > + N1SDP_ECAM_MCFG(0x20181101, 0, &pci_n1sdp_pcie_ecam_ops), > + N1SDP_ECAM_MCFG(0x20181101, 1, &pci_n1sdp_ccix_ecam_ops), > }; > > static char mcfg_oem_id[ACPI_OEM_ID_SIZE]; > diff --git a/drivers/pci/controller/Kconfig b/drivers/pci/controller/Kconfig > index c77069c8ee5d..45700d32f02e 100644 > --- a/drivers/pci/controller/Kconfig > +++ b/drivers/pci/controller/Kconfig > @@ -37,6 +37,17 @@ config PCI_FTPCI100 > depends on OF > default ARCH_GEMINI > > +config PCIE_HOST_N1SDP_ECAM > + bool "ARM N1SDP PCIe Controller" > + depends on ARM64 > + depends on OF || (ACPI && PCI_QUIRKS) > + select PCI_HOST_COMMON > + default y if ARCH_VEXPRESS > + help > + Say Y here if you want PCIe support for the Arm N1SDP platform. > + The controller is ECAM compliant, but needs a quirk to workaround > + an integration issue. Again - please indicate the scope of support provided. > + > config PCI_TEGRA > bool "NVIDIA Tegra PCIe controller" > depends on ARCH_TEGRA || COMPILE_TEST > diff --git a/drivers/pci/controller/Makefile b/drivers/pci/controller/Makefile > index 3d4f597f15ce..5f47fefbd67d 100644 > --- a/drivers/pci/controller/Makefile > +++ b/drivers/pci/controller/Makefile > @@ -28,6 +28,7 @@ obj-$(CONFIG_PCIE_MEDIATEK) += pcie-mediatek.o > obj-$(CONFIG_PCIE_MOBIVEIL) += pcie-mobiveil.o > obj-$(CONFIG_PCIE_TANGO_SMP8759) += pcie-tango.o > obj-$(CONFIG_VMD) += vmd.o > +obj-$(CONFIG_PCIE_HOST_N1SDP_ECAM) += pcie-n1sdp.o > # pcie-hisi.o quirks are needed even without CONFIG_PCIE_DW > obj-y += dwc/ > > diff --git a/drivers/pci/controller/pcie-n1sdp.c b/drivers/pci/controller/pcie-n1sdp.c > new file mode 100644 > index 000000000000..620ab221466c > --- /dev/null > +++ b/drivers/pci/controller/pcie-n1sdp.c > @@ -0,0 +1,196 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/* > + * Copyright (C) 2018/2019 ARM Ltd. > + * > + * This quirk is to mask the following issues: > + * - PCIE SLVERR: config space accesses to invalid PCIe BDFs cause a bus > + * error (signalled as an asynchronous SError) > + * - MCFG BDF mapping: the root complex is mapped separately from the device > + * config space > + * - Non 32-bit accesses to config space are not supported. > + * > + * At boot time the SCP board firmware creates a discovery table with > + * the root complex' base address and the valid BDF values, discovered while > + * scanning the config space and catching the SErrors. > + * Linux responds only to the EPs listed in this table, returning NULL NIT: it will respond to the switch devices which aren't EPs. > + * for the rest. > + */ > + > +#include <linux/kernel.h> > +#include <linux/init.h> > +#include <linux/ioport.h> > +#include <linux/sizes.h> > +#include <linux/of_pci.h> > +#include <linux/of.h> > +#include <linux/pci-ecam.h> > +#include <linux/platform_device.h> > +#include <linux/module.h> > + > +/* Platform specific values as hardcoded in the firmware. */ > +#define AP_NS_SHARED_MEM_BASE 0x06000000 > +#define MAX_SEGMENTS 2 /* Two PCIe root complexes. */ > +#define BDF_TABLE_SIZE SZ_16K > + > +/* > + * Shared memory layout as written by the SCP upon boot time: > + * ---- > + * Discover data header --> RC base address > + * \-> BDF Count > + * Discover data --> BDF 0...n > + * ---- > + */ > +struct pcie_discovery_data { > + u32 rc_base_addr; > + u32 nr_bdfs; > + u32 valid_bdfs[0]; > +} *pcie_discovery_data[MAX_SEGMENTS]; > + > +void __iomem *rc_remapped_addr[MAX_SEGMENTS]; > + > +/* > + * map_bus() is called before we do a config space access for a certain > + * device. We use this to check whether this device is valid, avoiding > + * config space accesses which would result in an SError otherwise. > + */ > +static void __iomem *pci_n1sdp_map_bus(struct pci_bus *bus, unsigned int devfn, > + int where) > +{ > + struct pci_config_window *cfg = bus->sysdata; > + unsigned int devfn_shift = cfg->ops->bus_shift - 8; > + unsigned int busn = bus->number; > + unsigned int segment = bus->domain_nr; > + unsigned int bdf_addr; > + unsigned int table_count, i; > + > + if (segment >= MAX_SEGMENTS || > + busn < cfg->busr.start || busn > cfg->busr.end) > + return NULL; > + > + /* The PCIe root complex has a separate config space mapping. */ > + if (busn == 0 && devfn == 0) > + return rc_remapped_addr[segment] + where; > + > + busn -= cfg->busr.start; > + bdf_addr = (busn << cfg->ops->bus_shift) + (devfn << devfn_shift); > + table_count = pcie_discovery_data[segment]->nr_bdfs; > + for (i = 0; i < table_count; i++) { > + if (bdf_addr == pcie_discovery_data[segment]->valid_bdfs[i]) > + return pci_ecam_map_bus(bus, devfn, where); > + } > + > + return NULL; > +} > + > +static int pci_n1sdp_init(struct pci_config_window *cfg, unsigned int segment) > +{ > + phys_addr_t table_base; > + struct device *dev = cfg->parent; > + struct pcie_discovery_data *shared_data; > + size_t bdfs_size; > + > + if (segment >= MAX_SEGMENTS) > + return -ENODEV; > + > + table_base = AP_NS_SHARED_MEM_BASE + segment * BDF_TABLE_SIZE; How can you be sure that this table is populated and isn't junk? I.e. using an older SCP version? > + > + if (!request_mem_region(table_base, BDF_TABLE_SIZE, > + "PCIe valid BDFs")) { > + dev_err(dev, "PCIe BDF shared region request failed\n"); > + return -ENOMEM; > + } > + > + shared_data = devm_ioremap(dev, > + table_base, BDF_TABLE_SIZE); > + if (!shared_data) > + return -ENOMEM; > + > + /* Copy the valid BDFs structure to allocated normal memory. */ > + bdfs_size = sizeof(struct pcie_discovery_data) + > + sizeof(u32) * shared_data->nr_bdfs; > + pcie_discovery_data[segment] = devm_kmalloc(dev, bdfs_size, GFP_KERNEL); > + if (!pcie_discovery_data[segment]) > + return -ENOMEM; > + > + memcpy_fromio(pcie_discovery_data[segment], shared_data, bdfs_size); > + > + rc_remapped_addr[segment] = devm_ioremap_nocache(dev, > + shared_data->rc_base_addr, > + PCI_CFG_SPACE_EXP_SIZE); > + if (!rc_remapped_addr[segment]) { > + dev_err(dev, "Cannot remap root port base\n"); > + return -ENOMEM; > + } > + > + devm_iounmap(dev, shared_data); > + > + return 0; > +} > + > +static int pci_n1sdp_pcie_init(struct pci_config_window *cfg) > +{ > + return pci_n1sdp_init(cfg, 0); > +} > + > +static int pci_n1sdp_ccix_init(struct pci_config_window *cfg) > +{ > + return pci_n1sdp_init(cfg, 1); > +} > + > +struct pci_ecam_ops pci_n1sdp_pcie_ecam_ops = { > + .bus_shift = 20, > + .init = pci_n1sdp_pcie_init, > + .pci_ops = { > + .map_bus = pci_n1sdp_map_bus, > + .read = pci_generic_config_read32, > + .write = pci_generic_config_write32, > + } > +}; > + > +struct pci_ecam_ops pci_n1sdp_ccix_ecam_ops = { > + .bus_shift = 20, > + .init = pci_n1sdp_ccix_init, > + .pci_ops = { > + .map_bus = pci_n1sdp_map_bus, > + .read = pci_generic_config_read32, > + .write = pci_generic_config_write32, > + } > +}; > + > +static const struct of_device_id n1sdp_pcie_of_match[] = { > + { .compatible = "arm,n1sdp-pcie" }, > + { }, > +}; > +MODULE_DEVICE_TABLE(of, n1sdp_pcie_of_match); > + > +static int n1sdp_pcie_probe(struct platform_device *pdev) > +{ > + const struct device_node *of_node = pdev->dev.of_node; > + u32 segment; > + > + if (of_property_read_u32(of_node, "linux,pci-domain", &segment)) { > + dev_err(&pdev->dev, "N1SDP PCI controllers require linux,pci-domain property\n"); > + return -EINVAL; > + } Can you use of_get_pci_domain_nr here? Thanks, Andrew Murray > + > + switch (segment) { > + case 0: > + return pci_host_common_probe(pdev, &pci_n1sdp_pcie_ecam_ops); > + case 1: > + return pci_host_common_probe(pdev, &pci_n1sdp_ccix_ecam_ops); > + } > + > + dev_err(&pdev->dev, "Invalid segment number, must be smaller than %d\n", > + MAX_SEGMENTS); > + > + return -EINVAL; > +} > + > +static struct platform_driver n1sdp_pcie_driver = { > + .driver = { > + .name = KBUILD_MODNAME, > + .of_match_table = n1sdp_pcie_of_match, > + .suppress_bind_attrs = true, > + }, > + .probe = n1sdp_pcie_probe, > +}; > +builtin_platform_driver(n1sdp_pcie_driver); > diff --git a/include/linux/pci-ecam.h b/include/linux/pci-ecam.h > index a73164c85e78..03cdea69f4e8 100644 > --- a/include/linux/pci-ecam.h > +++ b/include/linux/pci-ecam.h > @@ -57,6 +57,8 @@ extern struct pci_ecam_ops pci_thunder_ecam_ops; /* Cavium ThunderX 1.x */ > extern struct pci_ecam_ops xgene_v1_pcie_ecam_ops; /* APM X-Gene PCIe v1 */ > extern struct pci_ecam_ops xgene_v2_pcie_ecam_ops; /* APM X-Gene PCIe v2.x */ > extern struct pci_ecam_ops al_pcie_ops; /* Amazon Annapurna Labs PCIe */ > +extern struct pci_ecam_ops pci_n1sdp_pcie_ecam_ops; /* Arm N1SDP PCIe */ > +extern struct pci_ecam_ops pci_n1sdp_ccix_ecam_ops; /* Arm N1SDP PCIe */ > #endif > > #ifdef CONFIG_PCI_HOST_COMMON > -- > 2.17.1 >
On 12/12/2019 11:05 am, Andre Przywara wrote: > On 11/12/2019 20:17, Bjorn Helgaas wrote: > > Hi Bjorn, > >> On Wed, Dec 11, 2019 at 11:00:49AM +0000, Andre Przywara wrote: >>> On Tue, 10 Dec 2019 08:41:15 -0600 >>> Bjorn Helgaas <helgaas@kernel.org> wrote: >>>> On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote: >>>>> From: Deepak Pandey <Deepak.Pandey@arm.com> >>>>> >>>>> The Arm N1SDP SoC suffers from some PCIe integration issues, most >>>>> prominently config space accesses to not existing BDFs being answered >>>>> with a bus abort, resulting in an SError. >>>> >>>> Can we tease this apart a little more? Linux doesn't program all the >>>> bits that control error signaling, so even on hardware that works >>>> perfectly, much of this behavior is determined by what firmware did. >>>> I wonder if Linux could be more careful about this. >>>> >>>> "Bus abort" is not a term used in PCIe. >>> >>> Yes, sorry, that was my sloppy term, also aiming more at the CPU >>> side of the bus, between the cores and the RC. >>> >>>> IIUC, a config read to a >>>> device that doesn't exist should terminate with an Unsupported Request >>>> completion, e.g., see the implementation note in PCIe r5.0 sec 2.3.1. >>> >>> Yes, that's what Lorenzo mentioned as well. >>> >>>> The UR should be an uncorrectable non-fatal error (Table 6-5), and >>>> Figures 6-2 and 6-3 show how it should be handled and when it should >>>> be signaled as a system error. In case you don't have a copy of the >>>> spec, I extracted those two figures and put them at [1]. >>> >>> Thanks for that. >>> So in the last few months we tossed several ideas around how to >>> work-around this without kernel intervention, all of them turned out >>> to be not working. There are indeed registers in the RC that >>> influence error reporting to the CPU side, but even if we could >>> suppress (or catch) the SError, we can't recover and fixup the read >>> transaction to the CPU. Even Lorenzo gave up on this ;-) As far as I >>> understood this, there are gates missing which are supposed to >>> translate this specific UR into a valid "all-1s" response. >> >> But the commit log says firmware scanned the bus (catching the >> SErrors). Shouldn't Linux be able to catch them the same way? > > Not really. The scanning is done by the SCP management processor, which is a Cortex-M class core on the same bus. So it's a simple, single core running very early after power-on, when the actual AP cores are still off. The SError handler is set to just increase a value, then to return. This value is then checked before and after a config space access for a given > BDF: https://git.linaro.org/landing-teams/working/arm/n1sdp-pcie-quirk.git/tree/scp > > On the AP cores that run Linux later on this is quite different: The SError is asynchronous, imprecise (inexact) and has no syndrome information. That means we can't attribute this anymore to the faulting instruction, we don't even know if it happened due to this config space access. The CPU might have executed later instructions already, so the state is broken at this point. SError basically means: the system is screwed up. > Because this is quite common for SErrors, we don't even allow to register SError handlers in arm64 Linux. Furthermore, on the main application processor, SError might be delivered to EL3 firmware well beyond the reach of Linux, so we can make zero assumptions about how it's handled and whether we'll ever see it, or survive the result (EL3 is at liberty to say "oh, something went wrong, I'll reset the system immediately"). Robin. > So even if we could somehow handle this is in Linux, it would be a much greater and intrusive hack, so I'd rather stick with this version. > >> The "all-1s" response directly from hardware is typical of most >> platforms, but I don't think it's strictly required by the PCIe spec >> and I don't think it's absolutely essential even to Linux. If you can >> catch the SErrors, isn't there a way for software to fabricate that >> all-1s data and continue after the read? > > That was an idea we had as well, but due to the points mentioned above this is not possible. > >>>> Even ECAM compliance is not really minor -- if this controller were >>>> fully compliant with the spec, you would need ZERO Linux changes to >>>> support it. Every quirk like this means additional maintenance >>>> burden, and it's not just a one-time thing. It means old kernels that >>>> *should* "just work" on your system will not work unless somebody >>>> backports the quirk. >>> >>> I am well aware of that, and we had quite some discussions >>> internally, with quite some opposition. ... >> >> The main point is that *future* silicon should be designed to avoid >> this issue. I hope at least that part was not controversial. > > Yes, the design goal was to be completely standards (SBSA, ACPI, ECAM) compliant, it was just the usual "things happen" that wasn't spotted in time. > >> If we want to take advantage of the generic PCI code supplied by >> Linux, we have to expect that the hardware will play by the rules of >> PCI. > > You don't need to convince me ;-), but I think the lesson has been learned. > > Cheers, > Andre. > > _______________________________________________ > linux-arm-kernel mailing list > linux-arm-kernel@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel >
On Tue, Dec 10, 2019 at 08:41:15AM -0600, Bjorn Helgaas wrote: > On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote: > > From: Deepak Pandey <Deepak.Pandey@arm.com> > > > > The Arm N1SDP SoC suffers from some PCIe integration issues, most > > prominently config space accesses to not existing BDFs being answered > > with a bus abort, resulting in an SError. > > Can we tease this apart a little more? Linux doesn't program all the > bits that control error signaling, so even on hardware that works > perfectly, much of this behavior is determined by what firmware did. > I wonder if Linux could be more careful about this. > > "Bus abort" is not a term used in PCIe. IIUC, a config read to a > device that doesn't exist should terminate with an Unsupported Request > completion, e.g., see the implementation note in PCIe r5.0 sec 2.3.1. > > The UR should be an uncorrectable non-fatal error (Table 6-5), and > Figures 6-2 and 6-3 show how it should be handled and when it should > be signaled as a system error. In case you don't have a copy of the > spec, I extracted those two figures and put them at [1]. > > Can you collect "lspci -vvxxx" output to see if we can correlate it > with those figures and the behavior you see? > > [1] https://drive.google.com/file/d/1ihhdQvr0a7ZEJG-3gPddw1Tq7cTFAsah/view?usp=sharing > > > To mitigate this, the firmware scans the bus before boot (catching the > > SErrors) and creates a table with valid BDFs, which acts as a filter for > > Linux' config space accesses. > > > > Add code consulting the table as an ACPI PCIe quirk, also register the > > corresponding device tree based description of the host controller. > > Also fix the other two minor issues on the way, namely not being fully > > ECAM compliant and config space accesses being restricted to 32-bit > > accesses only. > > As I'm sure you've noticed, controllers that support only 32-bit > config writes are not spec compliant and devices may not work > correctly. The comment in pci_generic_config_write32() explains why. > > You may not trip over this problem frequently, but I wouldn't call it > a "minor" issue because when you *do* trip over it, you have no > indication that a register was corrupted. > > Even ECAM compliance is not really minor -- if this controller were > fully compliant with the spec, you would need ZERO Linux changes to > support it. Every quirk like this means additional maintenance > burden, and it's not just a one-time thing. It means old kernels that > *should* "just work" on your system will not work unless somebody > backports the quirk. With regards to URs resulting in unwanted aborts or similar - this seems to be a very common theme amongst ARM PCI controller drivers. For example both ARM32 imx6 and ARM32 keystone have fault handlers to handle an abort and fabricate a 0xffffffff read value. The ARM32 rcar driver, whilst it doesn't appear to produce an abort, does read the PCI_STATUS register after making a config read to determine if any aborts have happened - in which case it reports PCIBIOS_DEVICE_NOT_FOUND. And as recently reported [1], the rockchip driver also appears to produce aborts. I suspect that this ARM64 controller driver won't be the last either. Thus any solution here may form the basis of copy-cat solutions for subsequent controllers. From my understanding of the issues, the ARM64 serrors are imprecise and as a result there isn't a sensible way of using them to determine that a read is a UR. So where there are no other solutions to suppress the generation of an abort by the controller, the only solutions that seem to exist are 1) pre-scan the devices in firmware and only talk to those devices in Linux - a safe option but limiting - perhaps with side effects for CRS and 2) the approach rcar takes in using the PCI_STATUS register - though you'd end up having to mask the serror (PSTATE.A) for a limited period of time - a risky option (you'll miss real serrors) - but with no side effects. (I don't know if option 2 is feasible in this case by the way). [1] https://lore.kernel.org/linux-pci/2a381384-9d47-a7e2-679c-780950cd862d@rock-chips.com/2-0001-WFT-PCI-rockchip-play-game-with-unsupported-request-.patch Thanks, Andrew Murray > > > This allows the Arm Neoverse N1SDP board to boot Linux without crashing > > and to access *any* devices (there are no platform devices except UART).
On Thu, 12 Dec 2019 12:37:48 +0000 Andrew Murray <andrew.murray@arm.com> wrote: Hi Andrew, > On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote: > > From: Deepak Pandey <Deepak.Pandey@arm.com> > > > > The Arm N1SDP SoC suffers from some PCIe integration issues, most > > prominently config space accesses to not existing BDFs being answered > > with a bus abort, resulting in an SError. > > It wouldn't be a surprise if the host controller handled UR completions > differently depending on if they were in response to a Type 0 configuration > request or a Type 1 configuration request. (I think I've seen this before). Yeah, that rings a bell here as well, I remember something with the RK3399 PCIe RC. > Have you verified that you still get a bus abort when you attempt to > perform a config read of a non-existent device downstream of the PCIe switch? > (and thus as a response to a Type 1 request). I think I have checked this (please confirm if my experiment is valid): I get SErrors for both probing non-existent devices on bus 0 and other busses. > I ask because if this is the case, and knowing that the PCIe switch is > fixed, then it would be possible to simplify this quirk (by just making > assumptions of the presence of devices in bus 0 rather than all the busses). Yeah, thanks for trying, but no luck here ;-) > > To mitigate this, the firmware scans the bus before boot (catching the > > SErrors) and creates a table with valid BDFs, which acts as a filter for > > Linux' config space accesses. > > > > Add code consulting the table as an ACPI PCIe quirk, also register the > > corresponding device tree based description of the host controller. > > Also fix the other two minor issues on the way, namely not being fully > > ECAM compliant and config space accesses being restricted to 32-bit > > accesses only. > > > > This allows the Arm Neoverse N1SDP board to boot Linux without crashing > > and to access *any* devices (there are no platform devices except UART). > > This implies that this quirk has no side-effects and everything will work > as expected - but this is only true for the simple case. For example hot > plug won't work, SR-IOV, and others won't work. Yeah, good point, I should have mentioned this. This is really a best effort hack^Wworkaround to make the system work as best as possible. Yes, SR-IOV won't work. We are about to evaluate our options here, but for now it's just not supported on this system. I don't think hot plug is of a particular concern here, since the hardware doesn't support physical hot plug. I am not sure if a Thunderbolt add-in card would trigger this problem, but in the worst case it just wouldn't work. > Also what happens for devices that return CRS? - does this also result in an > abort? I haven't tried, and it looks like it's hard to trigger? But chances are indeed that it could generate SErrors. > Does that mean that the firmware will consider these devices as not > present instead of not ready yet? If this is an issue, then FLR of devices > will also create issues (resulting in SErrors for users). > > I think it would be helpful to update this commit message to indicate that > this makes it work better, but it may be broken in certain ways. Good point. This is really a pragmatic patch to make the system usable at all. > > Signed-off-by: Deepak Pandey <Deepak.Pandey@arm.com> > > [Sudipto: extend to cover the CCIX root port as well] > > Signed-off-by: Sudipto Paul <sudipto.paul@arm.com> > > [Andre: fix coding style issues, rewrite some parts, add DT support] > > Signed-off-by: Andre Przywara <andre.przywara@arm.com> > > --- > > arch/arm64/configs/defconfig | 1 + > > drivers/acpi/pci_mcfg.c | 7 + > > drivers/pci/controller/Kconfig | 11 ++ > > drivers/pci/controller/Makefile | 1 + > > drivers/pci/controller/pcie-n1sdp.c | 196 ++++++++++++++++++++++++++++ > > include/linux/pci-ecam.h | 2 + > > 6 files changed, 218 insertions(+) > > create mode 100644 drivers/pci/controller/pcie-n1sdp.c > > > > diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig > > index 6a83ba2aea3e..58124ef5070b 100644 > > --- a/arch/arm64/configs/defconfig > > +++ b/arch/arm64/configs/defconfig > > @@ -177,6 +177,7 @@ CONFIG_NET_9P=y > > CONFIG_NET_9P_VIRTIO=y > > CONFIG_PCI=y > > CONFIG_PCIEPORTBUS=y > > +CONFIG_PCI_QUIRKS=y > > CONFIG_PCI_IOV=y > > CONFIG_HOTPLUG_PCI=y > > CONFIG_HOTPLUG_PCI_ACPI=y > > diff --git a/drivers/acpi/pci_mcfg.c b/drivers/acpi/pci_mcfg.c > > index 6b347d9920cc..7a2b41b9ab57 100644 > > --- a/drivers/acpi/pci_mcfg.c > > +++ b/drivers/acpi/pci_mcfg.c > > @@ -142,6 +142,13 @@ static struct mcfg_fixup mcfg_quirks[] = { > > XGENE_V2_ECAM_MCFG(4, 0), > > XGENE_V2_ECAM_MCFG(4, 1), > > XGENE_V2_ECAM_MCFG(4, 2), > > + > > +#define N1SDP_ECAM_MCFG(rev, seg, ops) \ > > + {"ARMLTD", "ARMN1SDP", rev, seg, MCFG_BUS_ANY, ops } > > + > > + /* N1SDP SoC with v1 PCIe controller */ > > + N1SDP_ECAM_MCFG(0x20181101, 0, &pci_n1sdp_pcie_ecam_ops), > > + N1SDP_ECAM_MCFG(0x20181101, 1, &pci_n1sdp_ccix_ecam_ops), > > }; > > > > static char mcfg_oem_id[ACPI_OEM_ID_SIZE]; > > diff --git a/drivers/pci/controller/Kconfig b/drivers/pci/controller/Kconfig > > index c77069c8ee5d..45700d32f02e 100644 > > --- a/drivers/pci/controller/Kconfig > > +++ b/drivers/pci/controller/Kconfig > > @@ -37,6 +37,17 @@ config PCI_FTPCI100 > > depends on OF > > default ARCH_GEMINI > > > > +config PCIE_HOST_N1SDP_ECAM > > + bool "ARM N1SDP PCIe Controller" > > + depends on ARM64 > > + depends on OF || (ACPI && PCI_QUIRKS) > > + select PCI_HOST_COMMON > > + default y if ARCH_VEXPRESS > > + help > > + Say Y here if you want PCIe support for the Arm N1SDP platform. > > + The controller is ECAM compliant, but needs a quirk to workaround > > + an integration issue. > > Again - please indicate the scope of support provided. > > > + > > config PCI_TEGRA > > bool "NVIDIA Tegra PCIe controller" > > depends on ARCH_TEGRA || COMPILE_TEST > > diff --git a/drivers/pci/controller/Makefile b/drivers/pci/controller/Makefile > > index 3d4f597f15ce..5f47fefbd67d 100644 > > --- a/drivers/pci/controller/Makefile > > +++ b/drivers/pci/controller/Makefile > > @@ -28,6 +28,7 @@ obj-$(CONFIG_PCIE_MEDIATEK) += pcie-mediatek.o > > obj-$(CONFIG_PCIE_MOBIVEIL) += pcie-mobiveil.o > > obj-$(CONFIG_PCIE_TANGO_SMP8759) += pcie-tango.o > > obj-$(CONFIG_VMD) += vmd.o > > +obj-$(CONFIG_PCIE_HOST_N1SDP_ECAM) += pcie-n1sdp.o > > # pcie-hisi.o quirks are needed even without CONFIG_PCIE_DW > > obj-y += dwc/ > > > > diff --git a/drivers/pci/controller/pcie-n1sdp.c b/drivers/pci/controller/pcie-n1sdp.c > > new file mode 100644 > > index 000000000000..620ab221466c > > --- /dev/null > > +++ b/drivers/pci/controller/pcie-n1sdp.c > > @@ -0,0 +1,196 @@ > > +// SPDX-License-Identifier: GPL-2.0 > > +/* > > + * Copyright (C) 2018/2019 ARM Ltd. > > + * > > + * This quirk is to mask the following issues: > > + * - PCIE SLVERR: config space accesses to invalid PCIe BDFs cause a bus > > + * error (signalled as an asynchronous SError) > > + * - MCFG BDF mapping: the root complex is mapped separately from the device > > + * config space > > + * - Non 32-bit accesses to config space are not supported. > > + * > > + * At boot time the SCP board firmware creates a discovery table with > > + * the root complex' base address and the valid BDF values, discovered while > > + * scanning the config space and catching the SErrors. > > + * Linux responds only to the EPs listed in this table, returning NULL > > NIT: it will respond to the switch devices which aren't EPs. > > > > + * for the rest. > > + */ > > + > > +#include <linux/kernel.h> > > +#include <linux/init.h> > > +#include <linux/ioport.h> > > +#include <linux/sizes.h> > > +#include <linux/of_pci.h> > > +#include <linux/of.h> > > +#include <linux/pci-ecam.h> > > +#include <linux/platform_device.h> > > +#include <linux/module.h> > > + > > +/* Platform specific values as hardcoded in the firmware. */ > > +#define AP_NS_SHARED_MEM_BASE 0x06000000 > > +#define MAX_SEGMENTS 2 /* Two PCIe root complexes. */ > > +#define BDF_TABLE_SIZE SZ_16K > > + > > +/* > > + * Shared memory layout as written by the SCP upon boot time: > > + * ---- > > + * Discover data header --> RC base address > > + * \-> BDF Count > > + * Discover data --> BDF 0...n > > + * ---- > > + */ > > +struct pcie_discovery_data { > > + u32 rc_base_addr; > > + u32 nr_bdfs; > > + u32 valid_bdfs[0]; > > +} *pcie_discovery_data[MAX_SEGMENTS]; > > + > > +void __iomem *rc_remapped_addr[MAX_SEGMENTS]; > > + > > +/* > > + * map_bus() is called before we do a config space access for a certain > > + * device. We use this to check whether this device is valid, avoiding > > + * config space accesses which would result in an SError otherwise. > > + */ > > +static void __iomem *pci_n1sdp_map_bus(struct pci_bus *bus, unsigned int devfn, > > + int where) > > +{ > > + struct pci_config_window *cfg = bus->sysdata; > > + unsigned int devfn_shift = cfg->ops->bus_shift - 8; > > + unsigned int busn = bus->number; > > + unsigned int segment = bus->domain_nr; > > + unsigned int bdf_addr; > > + unsigned int table_count, i; > > + > > + if (segment >= MAX_SEGMENTS || > > + busn < cfg->busr.start || busn > cfg->busr.end) > > + return NULL; > > + > > + /* The PCIe root complex has a separate config space mapping. */ > > + if (busn == 0 && devfn == 0) > > + return rc_remapped_addr[segment] + where; > > + > > + busn -= cfg->busr.start; > > + bdf_addr = (busn << cfg->ops->bus_shift) + (devfn << devfn_shift); > > + table_count = pcie_discovery_data[segment]->nr_bdfs; > > + for (i = 0; i < table_count; i++) { > > + if (bdf_addr == pcie_discovery_data[segment]->valid_bdfs[i]) > > + return pci_ecam_map_bus(bus, devfn, where); > > + } > > + > > + return NULL; > > +} > > + > > +static int pci_n1sdp_init(struct pci_config_window *cfg, unsigned int segment) > > +{ > > + phys_addr_t table_base; > > + struct device *dev = cfg->parent; > > + struct pcie_discovery_data *shared_data; > > + size_t bdfs_size; > > + > > + if (segment >= MAX_SEGMENTS) > > + return -ENODEV; > > + > > + table_base = AP_NS_SHARED_MEM_BASE + segment * BDF_TABLE_SIZE; > > How can you be sure that this table is populated and isn't junk? I.e. using an older > SCP version? This is just (and always was) part of the firmware. If not, it won't work. Technically we tie this to the DT compatible or the ACPI signature. > > + > > + if (!request_mem_region(table_base, BDF_TABLE_SIZE, > > + "PCIe valid BDFs")) { > > + dev_err(dev, "PCIe BDF shared region request failed\n"); > > + return -ENOMEM; > > + } > > + > > + shared_data = devm_ioremap(dev, > > + table_base, BDF_TABLE_SIZE); > > + if (!shared_data) > > + return -ENOMEM; > > + > > + /* Copy the valid BDFs structure to allocated normal memory. */ > > + bdfs_size = sizeof(struct pcie_discovery_data) + > > + sizeof(u32) * shared_data->nr_bdfs; > > + pcie_discovery_data[segment] = devm_kmalloc(dev, bdfs_size, GFP_KERNEL); > > + if (!pcie_discovery_data[segment]) > > + return -ENOMEM; > > + > > + memcpy_fromio(pcie_discovery_data[segment], shared_data, bdfs_size); > > + > > + rc_remapped_addr[segment] = devm_ioremap_nocache(dev, > > + shared_data->rc_base_addr, > > + PCI_CFG_SPACE_EXP_SIZE); > > + if (!rc_remapped_addr[segment]) { > > + dev_err(dev, "Cannot remap root port base\n"); > > + return -ENOMEM; > > + } > > + > > + devm_iounmap(dev, shared_data); > > + > > + return 0; > > +} > > + > > +static int pci_n1sdp_pcie_init(struct pci_config_window *cfg) > > +{ > > + return pci_n1sdp_init(cfg, 0); > > +} > > + > > +static int pci_n1sdp_ccix_init(struct pci_config_window *cfg) > > +{ > > + return pci_n1sdp_init(cfg, 1); > > +} > > + > > +struct pci_ecam_ops pci_n1sdp_pcie_ecam_ops = { > > + .bus_shift = 20, > > + .init = pci_n1sdp_pcie_init, > > + .pci_ops = { > > + .map_bus = pci_n1sdp_map_bus, > > + .read = pci_generic_config_read32, > > + .write = pci_generic_config_write32, > > + } > > +}; > > + > > +struct pci_ecam_ops pci_n1sdp_ccix_ecam_ops = { > > + .bus_shift = 20, > > + .init = pci_n1sdp_ccix_init, > > + .pci_ops = { > > + .map_bus = pci_n1sdp_map_bus, > > + .read = pci_generic_config_read32, > > + .write = pci_generic_config_write32, > > + } > > +}; > > + > > +static const struct of_device_id n1sdp_pcie_of_match[] = { > > + { .compatible = "arm,n1sdp-pcie" }, > > + { }, > > +}; > > +MODULE_DEVICE_TABLE(of, n1sdp_pcie_of_match); > > + > > +static int n1sdp_pcie_probe(struct platform_device *pdev) > > +{ > > + const struct device_node *of_node = pdev->dev.of_node; > > + u32 segment; > > + > > + if (of_property_read_u32(of_node, "linux,pci-domain", &segment)) { > > + dev_err(&pdev->dev, "N1SDP PCI controllers require linux,pci-domain property\n"); > > + return -EINVAL; > > + } > > Can you use of_get_pci_domain_nr here? Ah, indeed. There seems to be an of_... function for everything ;-) Cheers, Andre. > > + > > + switch (segment) { > > + case 0: > > + return pci_host_common_probe(pdev, &pci_n1sdp_pcie_ecam_ops); > > + case 1: > > + return pci_host_common_probe(pdev, &pci_n1sdp_ccix_ecam_ops); > > + } > > + > > + dev_err(&pdev->dev, "Invalid segment number, must be smaller than %d\n", > > + MAX_SEGMENTS); > > + > > + return -EINVAL; > > +} > > + > > +static struct platform_driver n1sdp_pcie_driver = { > > + .driver = { > > + .name = KBUILD_MODNAME, > > + .of_match_table = n1sdp_pcie_of_match, > > + .suppress_bind_attrs = true, > > + }, > > + .probe = n1sdp_pcie_probe, > > +}; > > +builtin_platform_driver(n1sdp_pcie_driver); > > diff --git a/include/linux/pci-ecam.h b/include/linux/pci-ecam.h > > index a73164c85e78..03cdea69f4e8 100644 > > --- a/include/linux/pci-ecam.h > > +++ b/include/linux/pci-ecam.h > > @@ -57,6 +57,8 @@ extern struct pci_ecam_ops pci_thunder_ecam_ops; /* Cavium ThunderX 1.x */ > > extern struct pci_ecam_ops xgene_v1_pcie_ecam_ops; /* APM X-Gene PCIe v1 */ > > extern struct pci_ecam_ops xgene_v2_pcie_ecam_ops; /* APM X-Gene PCIe v2.x */ > > extern struct pci_ecam_ops al_pcie_ops; /* Amazon Annapurna Labs PCIe */ > > +extern struct pci_ecam_ops pci_n1sdp_pcie_ecam_ops; /* Arm N1SDP PCIe */ > > +extern struct pci_ecam_ops pci_n1sdp_ccix_ecam_ops; /* Arm N1SDP PCIe */ > > #endif > > > > #ifdef CONFIG_PCI_HOST_COMMON > > -- > > 2.17.1 > >
On Thu, 12 Dec 2019 21:07:24 +0000 Andrew Murray <andrew.murray@arm.com> wrote: Hi, > On Tue, Dec 10, 2019 at 08:41:15AM -0600, Bjorn Helgaas wrote: > > On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote: [ ... ] > > Even ECAM compliance is not really minor -- if this controller were > > fully compliant with the spec, you would need ZERO Linux changes to > > support it. Every quirk like this means additional maintenance > > burden, and it's not just a one-time thing. It means old kernels that > > *should* "just work" on your system will not work unless somebody > > backports the quirk. > > With regards to URs resulting in unwanted aborts or similar - this seems > to be a very common theme amongst ARM PCI controller drivers. For example > both ARM32 imx6 and ARM32 keystone have fault handlers to handle an abort > and fabricate a 0xffffffff read value. > > The ARM32 rcar driver, whilst it doesn't appear to produce an abort, does > read the PCI_STATUS register after making a config read to determine if > any aborts have happened - in which case it reports > PCIBIOS_DEVICE_NOT_FOUND. > > And as recently reported [1], the rockchip driver also appears to produce > aborts. > > I suspect that this ARM64 controller driver won't be the last either. Thus > any solution here may form the basis of copy-cat solutions for subsequent > controllers. Well, I think Bjorn is aware of them, but was actually hoping that those broken controllers would go away at some point ;-) And just to make this clear: I would categorise this issue as an integration bug, which just can't be fixed in hardware or firmware easily. It was never meant to be this way. So I am not sure we should promote generic solutions here. > From my understanding of the issues, the ARM64 serrors are imprecise and > as a result there isn't a sensible way of using them to determine that a > read is a UR. So where there are no other solutions to suppress the > generation of an abort by the controller, the only solutions that seem to > exist are 1) pre-scan the devices in firmware and only talk to those devices > in Linux - a safe option but limiting - perhaps with side effects for CRS > and 2) the approach rcar takes in using the PCI_STATUS register - though > you'd end up having to mask the serror (PSTATE.A) for a limited period of > time - a risky option (you'll miss real serrors) - but with no side effects. > > (I don't know if option 2 is feasible in this case by the way). Interesting, we might evaluate this, but mostly out of curiosity or for debugging. I don't think it's really a better option. If there is a safe way of making this work in the majority of cases, that should be the way to go. Setting PSTATE.A sounds quite wacky to me. Thanks, Andre. > [1] https://lore.kernel.org/linux-pci/2a381384-9d47-a7e2-679c-780950cd862d@rock-chips.com/2-0001-WFT-PCI-rockchip-play-game-with-unsupported-request-.patch > > Thanks, > > Andrew Murray > > > > > > This allows the Arm Neoverse N1SDP board to boot Linux without crashing > > > and to access *any* devices (there are no platform devices except UART).
On Thu, Dec 12, 2019 at 11:05:31AM +0000, Andre Przywara wrote: > On 11/12/2019 20:17, Bjorn Helgaas wrote: > > On Wed, Dec 11, 2019 at 11:00:49AM +0000, Andre Przywara wrote: > >> On Tue, 10 Dec 2019 08:41:15 -0600 > >> Bjorn Helgaas <helgaas@kernel.org> wrote: > >>> On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote: > >>>> From: Deepak Pandey <Deepak.Pandey@arm.com> > >>>> > >>>> The Arm N1SDP SoC suffers from some PCIe integration issues, most > >>>> prominently config space accesses to not existing BDFs being answered > >>>> with a bus abort, resulting in an SError. > >>> > >>> Can we tease this apart a little more? Linux doesn't program all the > >>> bits that control error signaling, so even on hardware that works > >>> perfectly, much of this behavior is determined by what firmware did. > >>> I wonder if Linux could be more careful about this. > >>> > >>> "Bus abort" is not a term used in PCIe. > >> > >> Yes, sorry, that was my sloppy term, also aiming more at the CPU > >> side of the bus, between the cores and the RC. > >> > >>> IIUC, a config read to a > >>> device that doesn't exist should terminate with an Unsupported Request > >>> completion, e.g., see the implementation note in PCIe r5.0 sec 2.3.1. > >> > >> Yes, that's what Lorenzo mentioned as well. > >> > >>> The UR should be an uncorrectable non-fatal error (Table 6-5), and > >>> Figures 6-2 and 6-3 show how it should be handled and when it should > >>> be signaled as a system error. In case you don't have a copy of the > >>> spec, I extracted those two figures and put them at [1]. > >> > >> Thanks for that. > >> So in the last few months we tossed several ideas around how to > >> work-around this without kernel intervention, all of them turned out > >> to be not working. There are indeed registers in the RC that > >> influence error reporting to the CPU side, but even if we could > >> suppress (or catch) the SError, we can't recover and fixup the read > >> transaction to the CPU. Even Lorenzo gave up on this ;-) As far as I > >> understood this, there are gates missing which are supposed to > >> translate this specific UR into a valid "all-1s" response. > > > > But the commit log says firmware scanned the bus (catching the > > SErrors). Shouldn't Linux be able to catch them the same way? > > Not really. The scanning is done by the SCP management processor, > which is a Cortex-M class core on the same bus. So it's a simple, > single core running very early after power-on, when the actual AP > cores are still off. The SError handler is set to just increase a > value, then to return. This value is then checked before and after a > config space access for a given BDF: > https://git.linaro.org/landing-teams/working/arm/n1sdp-pcie-quirk.git/tree/scp > > On the AP cores that run Linux later on this is quite different: The > SError is asynchronous, imprecise (inexact) and has no syndrome > information. That means we can't attribute this anymore to the > faulting instruction, we don't even know if it happened due to this > config space access. The CPU might have executed later instructions > already, so the state is broken at this point. SError basically > means: the system is screwed up. Because this is quite common for > SErrors, we don't even allow to register SError handlers in arm64 > Linux. > > So even if we could somehow handle this is in Linux, it would be a > much greater and intrusive hack, so I'd rather stick with this > version. The problem is that from a PCIe point of view, UR is something we should be able to tolerate. It happens during enumeration and also during hotplug. It definitely does not mean "the system is screwed up and must be rebooted." To go back to Figure 6-3, I'm getting the impression that the "System Error" shown at the top is *not* the "SError" you're referring to. If they were the same, the Root Control enable bits should gate it, but according to your lspci, those enable bits are cleared, yet you still take SErrors. SError is asynchronous and imprecise. Is there no way to do the config access in a way that makes it precise, by adding some kind of sync? There's no reason we can't single-thread config accesses and maybe even MMIO/IO port accesses as well if necessary. Bjorn
On 12/9/19 11:26 AM, Will Deacon wrote: > On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote: >> From: Deepak Pandey <Deepak.Pandey@arm.com> >> >> The Arm N1SDP SoC suffers from some PCIe integration issues, most >> prominently config space accesses to not existing BDFs being answered >> with a bus abort, resulting in an SError. > > "Do as I say, not as I do"? In my former role I asked nicely that these patches not be posted upstream, but I see that they ended up being posted anyway. Hacking up upstream Linux to cover for the fact that a (reference) platform is non-standard is not only not good form but it actively harms the community. You'll have people consume this platform and not realize that it's broken, IP won't get fixed, and generally it'll be a mess. Yes, it's unfortunate, but so was taping out that platform without working PCI. We all know what should have happened, and what the right move ahead is. Jon.
On Tue, 17 Dec 2019 21:21:17 -0500 Jon Masters <jcm@jonmasters.org> wrote: Hi Jon, > On 12/9/19 11:26 AM, Will Deacon wrote: > > On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote: > >> From: Deepak Pandey <Deepak.Pandey@arm.com> > >> > >> The Arm N1SDP SoC suffers from some PCIe integration issues, most > >> prominently config space accesses to not existing BDFs being answered > >> with a bus abort, resulting in an SError. > > > > "Do as I say, not as I do"? > > In my former role I asked nicely that these patches not be posted > upstream, but I see that they ended up being posted anyway. Hacking up > upstream Linux to cover for the fact that a (reference) platform is > non-standard is not only not good form but it actively harms the community. Please keep in mind that this platform was designed to be standards compliant, it is just due to an integration problem that this is not the case with this silicon. So we end up with the usual hardware errata, which the kernel can fix up. I agree it's not nice, and I also want it fixed in hardware, but I guess that's the usual software guy's pipe dream. > You'll have people consume this platform and not realize that it's > broken, IP won't get fixed, and generally it'll be a mess. I don't see how yet another ACPI quirk in the ACPI quirk framework(!) will make a mess. Actually the rest of the system is standards compliant (it even uses ACPI from the very beginning ;-), so it's just this problem that prevents us from using the system in the proper, standards compliant way. Effectively we are back to the embedded times with compiling your own kernel and somehow getting a root filesystem on the hard drive. If there would be mainline kernel support, all of this would go away and would could use standard distro installers (given they backport the patch). > Yes, it's > unfortunate, but so was taping out that platform without working PCI. We > all know what should have happened, and what the right move ahead is. That may come as a surprise to some, but Arm Ltd. is actually not really in the business of *producing silicon*, so a respin of the chip was and is not an option. I too wish it would be different, but that's how it is. Cheers, Andre.
On 2019-12-18 10:22, Andre Przywara wrote: > On Tue, 17 Dec 2019 21:21:17 -0500 > Jon Masters <jcm@jonmasters.org> wrote: > > Hi Jon, > >> On 12/9/19 11:26 AM, Will Deacon wrote: >> > On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote: >> >> From: Deepak Pandey <Deepak.Pandey@arm.com> >> >> >> >> The Arm N1SDP SoC suffers from some PCIe integration issues, most >> >> prominently config space accesses to not existing BDFs being >> answered >> >> with a bus abort, resulting in an SError. >> > >> > "Do as I say, not as I do"? >> >> In my former role I asked nicely that these patches not be posted >> upstream, but I see that they ended up being posted anyway. Hacking >> up >> upstream Linux to cover for the fact that a (reference) platform is >> non-standard is not only not good form but it actively harms the >> community. > > Please keep in mind that this platform was designed to be standards > compliant, it is just due to an integration problem that this is not > the case with this silicon. So we end up with the usual hardware > errata, which the kernel can fix up. I agree it's not nice, and I > also > want it fixed in hardware, but I guess that's the usual software > guy's > pipe dream. > >> You'll have people consume this platform and not realize that it's >> broken, IP won't get fixed, and generally it'll be a mess. > > I don't see how yet another ACPI quirk in the ACPI quirk framework(!) > will make a mess. > Actually the rest of the system is standards compliant (it even uses > ACPI from the very beginning ;-), so it's just this problem that > prevents us from using the system in the proper, standards compliant > way. Effectively we are back to the embedded times with compiling > your > own kernel and somehow getting a root filesystem on the hard drive. > If there would be mainline kernel support, all of this would go away > and would could use standard distro installers (given they backport > the patch). > >> Yes, it's >> unfortunate, but so was taping out that platform without working >> PCI. We >> all know what should have happened, and what the right move ahead >> is. > > That may come as a surprise to some, but Arm Ltd. is actually not > really in the business of *producing silicon*, so a respin of the > chip > was and is not an option. I too wish it would be different, but > that's > how it is. In all honesty, I really wonder why we are even having this discussion: - The HW is unavailable to the mere mortal. And even most superheroes cannot get their hands on it - Even with this terrible son of a hack, essential PCIe features cannot work (and yes, I do consider SR-IOV as an essential feature) - If we take this hack and somehow get this thing to run with mainline, we will never be able to say "no" to this kind of unfinished, *prototype* implementations ever again To sum it up: a hack that doesn't really work for a prototype that you can't really buy? Why should we even care? M.
On Fri, 13 Dec 2019 15:07:07 -0600 Bjorn Helgaas <helgaas@kernel.org> wrote: Hi Bjorn, > On Thu, Dec 12, 2019 at 11:05:31AM +0000, Andre Przywara wrote: > > On 11/12/2019 20:17, Bjorn Helgaas wrote: > > > On Wed, Dec 11, 2019 at 11:00:49AM +0000, Andre Przywara wrote: > > >> On Tue, 10 Dec 2019 08:41:15 -0600 > > >> Bjorn Helgaas <helgaas@kernel.org> wrote: > > >>> On Mon, Dec 09, 2019 at 04:06:38PM +0000, Andre Przywara wrote: > > >>>> From: Deepak Pandey <Deepak.Pandey@arm.com> > > >>>> > > >>>> The Arm N1SDP SoC suffers from some PCIe integration issues, most > > >>>> prominently config space accesses to not existing BDFs being answered > > >>>> with a bus abort, resulting in an SError. > > >>> > > >>> Can we tease this apart a little more? Linux doesn't program all the > > >>> bits that control error signaling, so even on hardware that works > > >>> perfectly, much of this behavior is determined by what firmware did. > > >>> I wonder if Linux could be more careful about this. > > >>> > > >>> "Bus abort" is not a term used in PCIe. > > >> > > >> Yes, sorry, that was my sloppy term, also aiming more at the CPU > > >> side of the bus, between the cores and the RC. > > >> > > >>> IIUC, a config read to a > > >>> device that doesn't exist should terminate with an Unsupported Request > > >>> completion, e.g., see the implementation note in PCIe r5.0 sec 2.3.1. > > >> > > >> Yes, that's what Lorenzo mentioned as well. > > >> > > >>> The UR should be an uncorrectable non-fatal error (Table 6-5), and > > >>> Figures 6-2 and 6-3 show how it should be handled and when it should > > >>> be signaled as a system error. In case you don't have a copy of the > > >>> spec, I extracted those two figures and put them at [1]. > > >> > > >> Thanks for that. > > >> So in the last few months we tossed several ideas around how to > > >> work-around this without kernel intervention, all of them turned out > > >> to be not working. There are indeed registers in the RC that > > >> influence error reporting to the CPU side, but even if we could > > >> suppress (or catch) the SError, we can't recover and fixup the read > > >> transaction to the CPU. Even Lorenzo gave up on this ;-) As far as I > > >> understood this, there are gates missing which are supposed to > > >> translate this specific UR into a valid "all-1s" response. > > > > > > But the commit log says firmware scanned the bus (catching the > > > SErrors). Shouldn't Linux be able to catch them the same way? > > > > Not really. The scanning is done by the SCP management processor, > > which is a Cortex-M class core on the same bus. So it's a simple, > > single core running very early after power-on, when the actual AP > > cores are still off. The SError handler is set to just increase a > > value, then to return. This value is then checked before and after a > > config space access for a given BDF: > > https://git.linaro.org/landing-teams/working/arm/n1sdp-pcie-quirk.git/tree/scp > > > > On the AP cores that run Linux later on this is quite different: The > > SError is asynchronous, imprecise (inexact) and has no syndrome > > information. That means we can't attribute this anymore to the > > faulting instruction, we don't even know if it happened due to this > > config space access. The CPU might have executed later instructions > > already, so the state is broken at this point. SError basically > > means: the system is screwed up. Because this is quite common for > > SErrors, we don't even allow to register SError handlers in arm64 > > Linux. > > > > So even if we could somehow handle this is in Linux, it would be a > > much greater and intrusive hack, so I'd rather stick with this > > version. > > The problem is that from a PCIe point of view, UR is something we > should be able to tolerate. It happens during enumeration and also > during hotplug. It definitely does not mean "the system is screwed up > and must be rebooted." I agree, I am also wondering why an (ARM) SError was considered an appropriate answer in the first place. Maybe it's due to some confusion between the ARM architecture term "SError" and the PCI #SERR signal? > To go back to Figure 6-3, I'm getting the impression that the "System > Error" shown at the top is *not* the "SError" you're referring to. If > they were the same, the Root Control enable bits should gate it, but > according to your lspci, those enable bits are cleared, yet you still > take SErrors. Yes, "System Error" in PCI lingo is definitely something different and unrelated to the ARM architecture SError. I also checked the (legacy) PCI command register, the PCIe baseline error handling and AER, all of them seem to not propagate the SERR signal. So I reckon it's something which does this on the root complex integration level, outside of generic PCIe. So far I also couldn't find a masking bit in the RC data sheet as well. > SError is asynchronous and imprecise. Is there no way to do the > config access in a way that makes it precise, by adding some kind of > sync? There's no reason we can't single-thread config accesses and > maybe even MMIO/IO port accesses as well if necessary. There is, on this core, but it becomes even more dodgy: While you can *block* delivery of an SError (by setting PSTATE.A), you couldn't clear it so far. ARMv8.2 introduced the "esb" instruction, which allows us to actually consume the SError, so clearing the A bit afterwards would continue operation as normal. This particular core supports v8.2, but the Rockchip SoC does not. So it would not help in the general case, also has that bitter taste of possibly masking an unrelated SError. So I managed to hack some proof of concept up where I can indeed fence in the SError around the config space access, within the kernel, so I can *almost* certainly say whether this readl/readw triggered this. But that would require aarch64 specific code (even instructions that require an ARMv8.2 compatible core) in the PCI driver, and so is probably more ugly than the existing solution. Plus we would need to iron out the remaining uncertainty, not sure that's actually possible. Cheers, Andre.
diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig index 6a83ba2aea3e..58124ef5070b 100644 --- a/arch/arm64/configs/defconfig +++ b/arch/arm64/configs/defconfig @@ -177,6 +177,7 @@ CONFIG_NET_9P=y CONFIG_NET_9P_VIRTIO=y CONFIG_PCI=y CONFIG_PCIEPORTBUS=y +CONFIG_PCI_QUIRKS=y CONFIG_PCI_IOV=y CONFIG_HOTPLUG_PCI=y CONFIG_HOTPLUG_PCI_ACPI=y diff --git a/drivers/acpi/pci_mcfg.c b/drivers/acpi/pci_mcfg.c index 6b347d9920cc..7a2b41b9ab57 100644 --- a/drivers/acpi/pci_mcfg.c +++ b/drivers/acpi/pci_mcfg.c @@ -142,6 +142,13 @@ static struct mcfg_fixup mcfg_quirks[] = { XGENE_V2_ECAM_MCFG(4, 0), XGENE_V2_ECAM_MCFG(4, 1), XGENE_V2_ECAM_MCFG(4, 2), + +#define N1SDP_ECAM_MCFG(rev, seg, ops) \ + {"ARMLTD", "ARMN1SDP", rev, seg, MCFG_BUS_ANY, ops } + + /* N1SDP SoC with v1 PCIe controller */ + N1SDP_ECAM_MCFG(0x20181101, 0, &pci_n1sdp_pcie_ecam_ops), + N1SDP_ECAM_MCFG(0x20181101, 1, &pci_n1sdp_ccix_ecam_ops), }; static char mcfg_oem_id[ACPI_OEM_ID_SIZE]; diff --git a/drivers/pci/controller/Kconfig b/drivers/pci/controller/Kconfig index c77069c8ee5d..45700d32f02e 100644 --- a/drivers/pci/controller/Kconfig +++ b/drivers/pci/controller/Kconfig @@ -37,6 +37,17 @@ config PCI_FTPCI100 depends on OF default ARCH_GEMINI +config PCIE_HOST_N1SDP_ECAM + bool "ARM N1SDP PCIe Controller" + depends on ARM64 + depends on OF || (ACPI && PCI_QUIRKS) + select PCI_HOST_COMMON + default y if ARCH_VEXPRESS + help + Say Y here if you want PCIe support for the Arm N1SDP platform. + The controller is ECAM compliant, but needs a quirk to workaround + an integration issue. + config PCI_TEGRA bool "NVIDIA Tegra PCIe controller" depends on ARCH_TEGRA || COMPILE_TEST diff --git a/drivers/pci/controller/Makefile b/drivers/pci/controller/Makefile index 3d4f597f15ce..5f47fefbd67d 100644 --- a/drivers/pci/controller/Makefile +++ b/drivers/pci/controller/Makefile @@ -28,6 +28,7 @@ obj-$(CONFIG_PCIE_MEDIATEK) += pcie-mediatek.o obj-$(CONFIG_PCIE_MOBIVEIL) += pcie-mobiveil.o obj-$(CONFIG_PCIE_TANGO_SMP8759) += pcie-tango.o obj-$(CONFIG_VMD) += vmd.o +obj-$(CONFIG_PCIE_HOST_N1SDP_ECAM) += pcie-n1sdp.o # pcie-hisi.o quirks are needed even without CONFIG_PCIE_DW obj-y += dwc/ diff --git a/drivers/pci/controller/pcie-n1sdp.c b/drivers/pci/controller/pcie-n1sdp.c new file mode 100644 index 000000000000..620ab221466c --- /dev/null +++ b/drivers/pci/controller/pcie-n1sdp.c @@ -0,0 +1,196 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (C) 2018/2019 ARM Ltd. + * + * This quirk is to mask the following issues: + * - PCIE SLVERR: config space accesses to invalid PCIe BDFs cause a bus + * error (signalled as an asynchronous SError) + * - MCFG BDF mapping: the root complex is mapped separately from the device + * config space + * - Non 32-bit accesses to config space are not supported. + * + * At boot time the SCP board firmware creates a discovery table with + * the root complex' base address and the valid BDF values, discovered while + * scanning the config space and catching the SErrors. + * Linux responds only to the EPs listed in this table, returning NULL + * for the rest. + */ + +#include <linux/kernel.h> +#include <linux/init.h> +#include <linux/ioport.h> +#include <linux/sizes.h> +#include <linux/of_pci.h> +#include <linux/of.h> +#include <linux/pci-ecam.h> +#include <linux/platform_device.h> +#include <linux/module.h> + +/* Platform specific values as hardcoded in the firmware. */ +#define AP_NS_SHARED_MEM_BASE 0x06000000 +#define MAX_SEGMENTS 2 /* Two PCIe root complexes. */ +#define BDF_TABLE_SIZE SZ_16K + +/* + * Shared memory layout as written by the SCP upon boot time: + * ---- + * Discover data header --> RC base address + * \-> BDF Count + * Discover data --> BDF 0...n + * ---- + */ +struct pcie_discovery_data { + u32 rc_base_addr; + u32 nr_bdfs; + u32 valid_bdfs[0]; +} *pcie_discovery_data[MAX_SEGMENTS]; + +void __iomem *rc_remapped_addr[MAX_SEGMENTS]; + +/* + * map_bus() is called before we do a config space access for a certain + * device. We use this to check whether this device is valid, avoiding + * config space accesses which would result in an SError otherwise. + */ +static void __iomem *pci_n1sdp_map_bus(struct pci_bus *bus, unsigned int devfn, + int where) +{ + struct pci_config_window *cfg = bus->sysdata; + unsigned int devfn_shift = cfg->ops->bus_shift - 8; + unsigned int busn = bus->number; + unsigned int segment = bus->domain_nr; + unsigned int bdf_addr; + unsigned int table_count, i; + + if (segment >= MAX_SEGMENTS || + busn < cfg->busr.start || busn > cfg->busr.end) + return NULL; + + /* The PCIe root complex has a separate config space mapping. */ + if (busn == 0 && devfn == 0) + return rc_remapped_addr[segment] + where; + + busn -= cfg->busr.start; + bdf_addr = (busn << cfg->ops->bus_shift) + (devfn << devfn_shift); + table_count = pcie_discovery_data[segment]->nr_bdfs; + for (i = 0; i < table_count; i++) { + if (bdf_addr == pcie_discovery_data[segment]->valid_bdfs[i]) + return pci_ecam_map_bus(bus, devfn, where); + } + + return NULL; +} + +static int pci_n1sdp_init(struct pci_config_window *cfg, unsigned int segment) +{ + phys_addr_t table_base; + struct device *dev = cfg->parent; + struct pcie_discovery_data *shared_data; + size_t bdfs_size; + + if (segment >= MAX_SEGMENTS) + return -ENODEV; + + table_base = AP_NS_SHARED_MEM_BASE + segment * BDF_TABLE_SIZE; + + if (!request_mem_region(table_base, BDF_TABLE_SIZE, + "PCIe valid BDFs")) { + dev_err(dev, "PCIe BDF shared region request failed\n"); + return -ENOMEM; + } + + shared_data = devm_ioremap(dev, + table_base, BDF_TABLE_SIZE); + if (!shared_data) + return -ENOMEM; + + /* Copy the valid BDFs structure to allocated normal memory. */ + bdfs_size = sizeof(struct pcie_discovery_data) + + sizeof(u32) * shared_data->nr_bdfs; + pcie_discovery_data[segment] = devm_kmalloc(dev, bdfs_size, GFP_KERNEL); + if (!pcie_discovery_data[segment]) + return -ENOMEM; + + memcpy_fromio(pcie_discovery_data[segment], shared_data, bdfs_size); + + rc_remapped_addr[segment] = devm_ioremap_nocache(dev, + shared_data->rc_base_addr, + PCI_CFG_SPACE_EXP_SIZE); + if (!rc_remapped_addr[segment]) { + dev_err(dev, "Cannot remap root port base\n"); + return -ENOMEM; + } + + devm_iounmap(dev, shared_data); + + return 0; +} + +static int pci_n1sdp_pcie_init(struct pci_config_window *cfg) +{ + return pci_n1sdp_init(cfg, 0); +} + +static int pci_n1sdp_ccix_init(struct pci_config_window *cfg) +{ + return pci_n1sdp_init(cfg, 1); +} + +struct pci_ecam_ops pci_n1sdp_pcie_ecam_ops = { + .bus_shift = 20, + .init = pci_n1sdp_pcie_init, + .pci_ops = { + .map_bus = pci_n1sdp_map_bus, + .read = pci_generic_config_read32, + .write = pci_generic_config_write32, + } +}; + +struct pci_ecam_ops pci_n1sdp_ccix_ecam_ops = { + .bus_shift = 20, + .init = pci_n1sdp_ccix_init, + .pci_ops = { + .map_bus = pci_n1sdp_map_bus, + .read = pci_generic_config_read32, + .write = pci_generic_config_write32, + } +}; + +static const struct of_device_id n1sdp_pcie_of_match[] = { + { .compatible = "arm,n1sdp-pcie" }, + { }, +}; +MODULE_DEVICE_TABLE(of, n1sdp_pcie_of_match); + +static int n1sdp_pcie_probe(struct platform_device *pdev) +{ + const struct device_node *of_node = pdev->dev.of_node; + u32 segment; + + if (of_property_read_u32(of_node, "linux,pci-domain", &segment)) { + dev_err(&pdev->dev, "N1SDP PCI controllers require linux,pci-domain property\n"); + return -EINVAL; + } + + switch (segment) { + case 0: + return pci_host_common_probe(pdev, &pci_n1sdp_pcie_ecam_ops); + case 1: + return pci_host_common_probe(pdev, &pci_n1sdp_ccix_ecam_ops); + } + + dev_err(&pdev->dev, "Invalid segment number, must be smaller than %d\n", + MAX_SEGMENTS); + + return -EINVAL; +} + +static struct platform_driver n1sdp_pcie_driver = { + .driver = { + .name = KBUILD_MODNAME, + .of_match_table = n1sdp_pcie_of_match, + .suppress_bind_attrs = true, + }, + .probe = n1sdp_pcie_probe, +}; +builtin_platform_driver(n1sdp_pcie_driver); diff --git a/include/linux/pci-ecam.h b/include/linux/pci-ecam.h index a73164c85e78..03cdea69f4e8 100644 --- a/include/linux/pci-ecam.h +++ b/include/linux/pci-ecam.h @@ -57,6 +57,8 @@ extern struct pci_ecam_ops pci_thunder_ecam_ops; /* Cavium ThunderX 1.x */ extern struct pci_ecam_ops xgene_v1_pcie_ecam_ops; /* APM X-Gene PCIe v1 */ extern struct pci_ecam_ops xgene_v2_pcie_ecam_ops; /* APM X-Gene PCIe v2.x */ extern struct pci_ecam_ops al_pcie_ops; /* Amazon Annapurna Labs PCIe */ +extern struct pci_ecam_ops pci_n1sdp_pcie_ecam_ops; /* Arm N1SDP PCIe */ +extern struct pci_ecam_ops pci_n1sdp_ccix_ecam_ops; /* Arm N1SDP PCIe */ #endif #ifdef CONFIG_PCI_HOST_COMMON