Message ID | 20240416205810.28754-1-gbatra@linux.ibm.com (mailing list archive) |
---|---|
State | Changes Requested |
Headers | show |
Series | powerpc/pseries/iommu: LPAR panics when rebooted with a frozen PE | expand |
Context | Check | Description |
---|---|---|
snowpatch_ozlabs/github-powerpc_ppctests | success | Successfully ran 8 jobs. |
snowpatch_ozlabs/github-powerpc_selftests | success | Successfully ran 8 jobs. |
snowpatch_ozlabs/github-powerpc_sparse | success | Successfully ran 4 jobs. |
snowpatch_ozlabs/github-powerpc_clang | success | Successfully ran 6 jobs. |
snowpatch_ozlabs/github-powerpc_kernel_qemu | success | Successfully ran 23 jobs. |
Gaurav Batra <gbatra@linux.ibm.com> writes: > At the time of LPAR reboot, partition firmware provides Open Firmware > property ibm,dma-window for the PE. This property is provided on the PCI > bus the PE is attached to. AFAICS you're actually describing a bug that happens during boot *up*? Describing it as "reboot" makes me think you're talking about the shutdown path. I think that will confuse people, me at least :) cheers > There are execptions where the partition firmware might not provide this > property for the PE at the time of LPAR reboot. One of the scenario is > where the firmware has frozen the PE due to some error conditions. This > PE is frozen for 24 hours or unless the whole system is reinitialized. > > Within this time frame, if the LPAR is rebooted, the frozen PE will be > presented to the LPAR but ibm,dma-window property could be missing. > > Today, under these circumstances, the LPAR oopses with NULL pointer > dereference, when configuring the PCI bus the PE is attached to. > > BUG: Kernel NULL pointer dereference on read at 0x000000c8 > Faulting instruction address: 0xc0000000001024c0 > Oops: Kernel access of bad area, sig: 7 [#1] > LE PAGE_SIZE=64K MMU=Radix SMP NR_CPUS=2048 NUMA pSeries > Modules linked in: > Supported: Yes > CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.4.0-150600.9-default #1 > Hardware name: IBM,9043-MRX POWER10 (raw) 0x800200 0xf000006 of:IBM,FW1060.00 (NM1060_023) hv:phyp pSeries > NIP: c0000000001024c0 LR: c0000000001024b0 CTR: c000000000102450 > REGS: c0000000037db5c0 TRAP: 0300 Not tainted (6.4.0-150600.9-default) > MSR: 8000000002009033 <SF,VEC,EE,ME,IR,DR,RI,LE> CR: 28000822 XER: 00000000 > CFAR: c00000000010254c DAR: 00000000000000c8 DSISR: 00080000 IRQMASK: 0 > ... > NIP [c0000000001024c0] pci_dma_bus_setup_pSeriesLP+0x70/0x2a0 > LR [c0000000001024b0] pci_dma_bus_setup_pSeriesLP+0x60/0x2a0 > Call Trace: > pci_dma_bus_setup_pSeriesLP+0x60/0x2a0 (unreliable) > pcibios_setup_bus_self+0x1c0/0x370 > __of_scan_bus+0x2f8/0x330 > pcibios_scan_phb+0x280/0x3d0 > pcibios_init+0x88/0x12c > do_one_initcall+0x60/0x320 > kernel_init_freeable+0x344/0x3e4 > kernel_init+0x34/0x1d0 > ret_from_kernel_user_thread+0x14/0x1c > > Fixes: b1fc44eaa9ba ("pseries/iommu/ddw: Fix kdump to work in absence of ibm,dma-window") > Signed-off-by: Gaurav Batra <gbatra@linux.ibm.com> > --- > arch/powerpc/platforms/pseries/iommu.c | 8 ++++++++ > 1 file changed, 8 insertions(+) > > diff --git a/arch/powerpc/platforms/pseries/iommu.c b/arch/powerpc/platforms/pseries/iommu.c > index e8c4129697b1..e808d5b1fa49 100644 > --- a/arch/powerpc/platforms/pseries/iommu.c > +++ b/arch/powerpc/platforms/pseries/iommu.c > @@ -786,8 +786,16 @@ static void pci_dma_bus_setup_pSeriesLP(struct pci_bus *bus) > * parent bus. During reboot, there will be ibm,dma-window property to > * define DMA window. For kdump, there will at least be default window or DDW > * or both. > + * There is an exception to the above. In case the PE goes into frozen > + * state, firmware may not provide ibm,dma-window property at the time > + * of LPAR reboot. > */ > > + if (!pdn) { > + pr_debug(" no ibm,dma-window property !\n"); > + return; > + } > + > ppci = PCI_DN(pdn); > > pr_debug(" parent is %pOF, iommu_table: 0x%p\n", > > base-commit: 2c71fdf02a95b3dd425b42f28fd47fb2b1d22702 > -- > 2.39.3 (Apple Git-146)
Hello, On Fri, Apr 19, 2024 at 04:12:46PM +1000, Michael Ellerman wrote: > Gaurav Batra <gbatra@linux.ibm.com> writes: > > At the time of LPAR reboot, partition firmware provides Open Firmware > > property ibm,dma-window for the PE. This property is provided on the PCI > > bus the PE is attached to. > > AFAICS you're actually describing a bug that happens during boot *up*? > > Describing it as "reboot" makes me think you're talking about the > shutdown path. I think that will confuse people, me at least :) there is probably an assumption that it must have been running previously for the errors to happen in the first place but given the error state persists for a day it may be a very long 'reboot'. Thanks Michal > > cheers > > > There are execptions where the partition firmware might not provide this > > property for the PE at the time of LPAR reboot. One of the scenario is > > where the firmware has frozen the PE due to some error conditions. This > > PE is frozen for 24 hours or unless the whole system is reinitialized. > > > > Within this time frame, if the LPAR is rebooted, the frozen PE will be > > presented to the LPAR but ibm,dma-window property could be missing. > > > > Today, under these circumstances, the LPAR oopses with NULL pointer > > dereference, when configuring the PCI bus the PE is attached to. > > > > BUG: Kernel NULL pointer dereference on read at 0x000000c8 > > Faulting instruction address: 0xc0000000001024c0 > > Oops: Kernel access of bad area, sig: 7 [#1] > > LE PAGE_SIZE=64K MMU=Radix SMP NR_CPUS=2048 NUMA pSeries > > Modules linked in: > > Supported: Yes > > CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.4.0-150600.9-default #1 > > Hardware name: IBM,9043-MRX POWER10 (raw) 0x800200 0xf000006 of:IBM,FW1060.00 (NM1060_023) hv:phyp pSeries > > NIP: c0000000001024c0 LR: c0000000001024b0 CTR: c000000000102450 > > REGS: c0000000037db5c0 TRAP: 0300 Not tainted (6.4.0-150600.9-default) > > MSR: 8000000002009033 <SF,VEC,EE,ME,IR,DR,RI,LE> CR: 28000822 XER: 00000000 > > CFAR: c00000000010254c DAR: 00000000000000c8 DSISR: 00080000 IRQMASK: 0 > > ... > > NIP [c0000000001024c0] pci_dma_bus_setup_pSeriesLP+0x70/0x2a0 > > LR [c0000000001024b0] pci_dma_bus_setup_pSeriesLP+0x60/0x2a0 > > Call Trace: > > pci_dma_bus_setup_pSeriesLP+0x60/0x2a0 (unreliable) > > pcibios_setup_bus_self+0x1c0/0x370 > > __of_scan_bus+0x2f8/0x330 > > pcibios_scan_phb+0x280/0x3d0 > > pcibios_init+0x88/0x12c > > do_one_initcall+0x60/0x320 > > kernel_init_freeable+0x344/0x3e4 > > kernel_init+0x34/0x1d0 > > ret_from_kernel_user_thread+0x14/0x1c > > > > Fixes: b1fc44eaa9ba ("pseries/iommu/ddw: Fix kdump to work in absence of ibm,dma-window") > > Signed-off-by: Gaurav Batra <gbatra@linux.ibm.com> > > --- > > arch/powerpc/platforms/pseries/iommu.c | 8 ++++++++ > > 1 file changed, 8 insertions(+) > > > > diff --git a/arch/powerpc/platforms/pseries/iommu.c b/arch/powerpc/platforms/pseries/iommu.c > > index e8c4129697b1..e808d5b1fa49 100644 > > --- a/arch/powerpc/platforms/pseries/iommu.c > > +++ b/arch/powerpc/platforms/pseries/iommu.c > > @@ -786,8 +786,16 @@ static void pci_dma_bus_setup_pSeriesLP(struct pci_bus *bus) > > * parent bus. During reboot, there will be ibm,dma-window property to > > * define DMA window. For kdump, there will at least be default window or DDW > > * or both. > > + * There is an exception to the above. In case the PE goes into frozen > > + * state, firmware may not provide ibm,dma-window property at the time > > + * of LPAR reboot. > > */ > > > > + if (!pdn) { > > + pr_debug(" no ibm,dma-window property !\n"); > > + return; > > + } > > + > > ppci = PCI_DN(pdn); > > > > pr_debug(" parent is %pOF, iommu_table: 0x%p\n", > > > > base-commit: 2c71fdf02a95b3dd425b42f28fd47fb2b1d22702 > > -- > > 2.39.3 (Apple Git-146)
You are right. I think, the "reboot" should be replaced with just "boot up". If there are no other comments, or code changes, I can re-word the commit message and submit for review. Thanks, Gaurav On 4/19/24 6:11 AM, Michal Suchánek wrote: > Hello, > > On Fri, Apr 19, 2024 at 04:12:46PM +1000, Michael Ellerman wrote: >> Gaurav Batra <gbatra@linux.ibm.com> writes: >>> At the time of LPAR reboot, partition firmware provides Open Firmware >>> property ibm,dma-window for the PE. This property is provided on the PCI >>> bus the PE is attached to. >> AFAICS you're actually describing a bug that happens during boot *up*? >> >> Describing it as "reboot" makes me think you're talking about the >> shutdown path. I think that will confuse people, me at least :) > there is probably an assumption that it must have been running > previously for the errors to happen in the first place but given the > error state persists for a day it may be a very long 'reboot'. > > Thanks > > Michal >> cheers >> >>> There are execptions where the partition firmware might not provide this >>> property for the PE at the time of LPAR reboot. One of the scenario is >>> where the firmware has frozen the PE due to some error conditions. This >>> PE is frozen for 24 hours or unless the whole system is reinitialized. >>> >>> Within this time frame, if the LPAR is rebooted, the frozen PE will be >>> presented to the LPAR but ibm,dma-window property could be missing. >>> >>> Today, under these circumstances, the LPAR oopses with NULL pointer >>> dereference, when configuring the PCI bus the PE is attached to. >>> >>> BUG: Kernel NULL pointer dereference on read at 0x000000c8 >>> Faulting instruction address: 0xc0000000001024c0 >>> Oops: Kernel access of bad area, sig: 7 [#1] >>> LE PAGE_SIZE=64K MMU=Radix SMP NR_CPUS=2048 NUMA pSeries >>> Modules linked in: >>> Supported: Yes >>> CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.4.0-150600.9-default #1 >>> Hardware name: IBM,9043-MRX POWER10 (raw) 0x800200 0xf000006 of:IBM,FW1060.00 (NM1060_023) hv:phyp pSeries >>> NIP: c0000000001024c0 LR: c0000000001024b0 CTR: c000000000102450 >>> REGS: c0000000037db5c0 TRAP: 0300 Not tainted (6.4.0-150600.9-default) >>> MSR: 8000000002009033 <SF,VEC,EE,ME,IR,DR,RI,LE> CR: 28000822 XER: 00000000 >>> CFAR: c00000000010254c DAR: 00000000000000c8 DSISR: 00080000 IRQMASK: 0 >>> ... >>> NIP [c0000000001024c0] pci_dma_bus_setup_pSeriesLP+0x70/0x2a0 >>> LR [c0000000001024b0] pci_dma_bus_setup_pSeriesLP+0x60/0x2a0 >>> Call Trace: >>> pci_dma_bus_setup_pSeriesLP+0x60/0x2a0 (unreliable) >>> pcibios_setup_bus_self+0x1c0/0x370 >>> __of_scan_bus+0x2f8/0x330 >>> pcibios_scan_phb+0x280/0x3d0 >>> pcibios_init+0x88/0x12c >>> do_one_initcall+0x60/0x320 >>> kernel_init_freeable+0x344/0x3e4 >>> kernel_init+0x34/0x1d0 >>> ret_from_kernel_user_thread+0x14/0x1c >>> >>> Fixes: b1fc44eaa9ba ("pseries/iommu/ddw: Fix kdump to work in absence of ibm,dma-window") >>> Signed-off-by: Gaurav Batra <gbatra@linux.ibm.com> >>> --- >>> arch/powerpc/platforms/pseries/iommu.c | 8 ++++++++ >>> 1 file changed, 8 insertions(+) >>> >>> diff --git a/arch/powerpc/platforms/pseries/iommu.c b/arch/powerpc/platforms/pseries/iommu.c >>> index e8c4129697b1..e808d5b1fa49 100644 >>> --- a/arch/powerpc/platforms/pseries/iommu.c >>> +++ b/arch/powerpc/platforms/pseries/iommu.c >>> @@ -786,8 +786,16 @@ static void pci_dma_bus_setup_pSeriesLP(struct pci_bus *bus) >>> * parent bus. During reboot, there will be ibm,dma-window property to >>> * define DMA window. For kdump, there will at least be default window or DDW >>> * or both. >>> + * There is an exception to the above. In case the PE goes into frozen >>> + * state, firmware may not provide ibm,dma-window property at the time >>> + * of LPAR reboot. >>> */ >>> >>> + if (!pdn) { >>> + pr_debug(" no ibm,dma-window property !\n"); >>> + return; >>> + } >>> + >>> ppci = PCI_DN(pdn); >>> >>> pr_debug(" parent is %pOF, iommu_table: 0x%p\n", >>> >>> base-commit: 2c71fdf02a95b3dd425b42f28fd47fb2b1d22702 >>> -- >>> 2.39.3 (Apple Git-146)
Michal Suchánek <msuchanek@suse.de> writes: > Hello, > > On Fri, Apr 19, 2024 at 04:12:46PM +1000, Michael Ellerman wrote: >> Gaurav Batra <gbatra@linux.ibm.com> writes: >> > At the time of LPAR reboot, partition firmware provides Open Firmware >> > property ibm,dma-window for the PE. This property is provided on the PCI >> > bus the PE is attached to. >> >> AFAICS you're actually describing a bug that happens during boot *up*? >> >> Describing it as "reboot" makes me think you're talking about the >> shutdown path. I think that will confuse people, me at least :) > > there is probably an assumption that it must have been running > previously for the errors to happen in the first place but given the > error state persists for a day it may be a very long 'reboot'. Yeah. Which is good detail, but the actual change is to the boot up path so I think it's better described that way. cheers
Gaurav Batra <gbatra@linux.ibm.com> writes: > You are right. I think, the "reboot" should be replaced with just "boot > up". If there are no other comments, or code changes, I can re-word the > commit message and submit for review. Yeah thanks. The change looks fine, just the change log needs a tweak. It's fine to mention that the bug happens when a system has been running, a device has been frozen, then the LPAR is rebooted, and *then* we hit the bug at boot up. cheers
diff --git a/arch/powerpc/platforms/pseries/iommu.c b/arch/powerpc/platforms/pseries/iommu.c index e8c4129697b1..e808d5b1fa49 100644 --- a/arch/powerpc/platforms/pseries/iommu.c +++ b/arch/powerpc/platforms/pseries/iommu.c @@ -786,8 +786,16 @@ static void pci_dma_bus_setup_pSeriesLP(struct pci_bus *bus) * parent bus. During reboot, there will be ibm,dma-window property to * define DMA window. For kdump, there will at least be default window or DDW * or both. + * There is an exception to the above. In case the PE goes into frozen + * state, firmware may not provide ibm,dma-window property at the time + * of LPAR reboot. */ + if (!pdn) { + pr_debug(" no ibm,dma-window property !\n"); + return; + } + ppci = PCI_DN(pdn); pr_debug(" parent is %pOF, iommu_table: 0x%p\n",
At the time of LPAR reboot, partition firmware provides Open Firmware property ibm,dma-window for the PE. This property is provided on the PCI bus the PE is attached to. There are execptions where the partition firmware might not provide this property for the PE at the time of LPAR reboot. One of the scenario is where the firmware has frozen the PE due to some error conditions. This PE is frozen for 24 hours or unless the whole system is reinitialized. Within this time frame, if the LPAR is rebooted, the frozen PE will be presented to the LPAR but ibm,dma-window property could be missing. Today, under these circumstances, the LPAR oopses with NULL pointer dereference, when configuring the PCI bus the PE is attached to. BUG: Kernel NULL pointer dereference on read at 0x000000c8 Faulting instruction address: 0xc0000000001024c0 Oops: Kernel access of bad area, sig: 7 [#1] LE PAGE_SIZE=64K MMU=Radix SMP NR_CPUS=2048 NUMA pSeries Modules linked in: Supported: Yes CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.4.0-150600.9-default #1 Hardware name: IBM,9043-MRX POWER10 (raw) 0x800200 0xf000006 of:IBM,FW1060.00 (NM1060_023) hv:phyp pSeries NIP: c0000000001024c0 LR: c0000000001024b0 CTR: c000000000102450 REGS: c0000000037db5c0 TRAP: 0300 Not tainted (6.4.0-150600.9-default) MSR: 8000000002009033 <SF,VEC,EE,ME,IR,DR,RI,LE> CR: 28000822 XER: 00000000 CFAR: c00000000010254c DAR: 00000000000000c8 DSISR: 00080000 IRQMASK: 0 ... NIP [c0000000001024c0] pci_dma_bus_setup_pSeriesLP+0x70/0x2a0 LR [c0000000001024b0] pci_dma_bus_setup_pSeriesLP+0x60/0x2a0 Call Trace: pci_dma_bus_setup_pSeriesLP+0x60/0x2a0 (unreliable) pcibios_setup_bus_self+0x1c0/0x370 __of_scan_bus+0x2f8/0x330 pcibios_scan_phb+0x280/0x3d0 pcibios_init+0x88/0x12c do_one_initcall+0x60/0x320 kernel_init_freeable+0x344/0x3e4 kernel_init+0x34/0x1d0 ret_from_kernel_user_thread+0x14/0x1c Fixes: b1fc44eaa9ba ("pseries/iommu/ddw: Fix kdump to work in absence of ibm,dma-window") Signed-off-by: Gaurav Batra <gbatra@linux.ibm.com> --- arch/powerpc/platforms/pseries/iommu.c | 8 ++++++++ 1 file changed, 8 insertions(+) base-commit: 2c71fdf02a95b3dd425b42f28fd47fb2b1d22702