| Message ID | 20240923212218.116979-1-mkhalfella@purestorage.com |
|---|---|
| State | Superseded |
| Headers | show |
| Series | igb: Do not bring the device up after non-fatal error | expand |
On 2024-09-23 16:11:14 -0700, Jacob Keller wrote: > > > On 9/23/2024 2:22 PM, Mohamed Khalfella wrote: > > Commit 004d25060c78 ("igb: Fix igb_down hung on surprise removal") > > changed igb_io_error_detected() to ignore non-fatal pcie errors in order > > to avoid hung task that can happen when igb_down() is called multiple > > times. This caused an issue when processing transient non-fatal errors. > > igb_io_resume(), which is called after igb_io_error_detected(), assumes > > that device is brought down by igb_io_error_detected() if the interface > > is up. This resulted in panic with stacktrace below. > > > > [ T3256] igb 0000:09:00.0 haeth0: igb: haeth0 NIC Link is Down > > [ T292] pcieport 0000:00:1c.5: AER: Uncorrected (Non-Fatal) error received: 0000:09:00.0 > > [ T292] igb 0000:09:00.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID) > > [ T292] igb 0000:09:00.0: device [8086:1537] error status/mask=00004000/00000000 > > [ T292] igb 0000:09:00.0: [14] CmpltTO [ 200.105524,009][ T292] igb 0000:09:00.0: AER: TLP Header: 00000000 00000000 00000000 00000000 > > [ T292] pcieport 0000:00:1c.5: AER: broadcast error_detected message > > [ T292] igb 0000:09:00.0: Non-correctable non-fatal error reported. > > [ T292] pcieport 0000:00:1c.5: AER: broadcast mmio_enabled message > > [ T292] pcieport 0000:00:1c.5: AER: broadcast resume message > > [ T292] ------------[ cut here ]------------ > > [ T292] kernel BUG at net/core/dev.c:6539! > > [ T292] invalid opcode: 0000 [#1] PREEMPT SMP > > [ T292] RIP: 0010:napi_enable+0x37/0x40 > > [ T292] Call Trace: > > [ T292] <TASK> > > [ T292] ? die+0x33/0x90 > > [ T292] ? do_trap+0xdc/0x110 > > [ T292] ? napi_enable+0x37/0x40 > > [ T292] ? do_error_trap+0x70/0xb0 > > [ T292] ? napi_enable+0x37/0x40 > > [ T292] ? napi_enable+0x37/0x40 > > [ T292] ? exc_invalid_op+0x4e/0x70 > > [ T292] ? napi_enable+0x37/0x40 > > [ T292] ? asm_exc_invalid_op+0x16/0x20 > > [ T292] ? napi_enable+0x37/0x40 > > [ T292] igb_up+0x41/0x150 > > [ T292] igb_io_resume+0x25/0x70 > > [ T292] report_resume+0x54/0x70 > > [ T292] ? report_frozen_detected+0x20/0x20 > > [ T292] pci_walk_bus+0x6c/0x90 > > [ T292] ? aer_print_port_info+0xa0/0xa0 > > [ T292] pcie_do_recovery+0x22f/0x380 > > [ T292] aer_process_err_devices+0x110/0x160 > > [ T292] aer_isr+0x1c1/0x1e0 > > [ T292] ? disable_irq_nosync+0x10/0x10 > > [ T292] irq_thread_fn+0x1a/0x60 > > [ T292] irq_thread+0xe3/0x1a0 > > [ T292] ? irq_set_affinity_notifier+0x120/0x120 > > [ T292] ? irq_affinity_notify+0x100/0x100 > > [ T292] kthread+0xe2/0x110 > > [ T292] ? kthread_complete_and_exit+0x20/0x20 > > [ T292] ret_from_fork+0x2d/0x50 > > [ T292] ? kthread_complete_and_exit+0x20/0x20 > > [ T292] ret_from_fork_asm+0x11/0x20 > > [ T292] </TASK> > > > > To fix this issue igb_io_resume() checks if the interface is running and > > the device is not down this means igb_io_error_detected() did not bring > > the device down and there is no need to bring it up. > > > > Signed-off-by: Mohamed Khalfella <mkhalfella@purestorage.com> > > Reviewed-by: Yuanyuan Zhong<yzhong@purestorage.com> > > Fixes: 004d25060c78 ("igb: Fix igb_down hung on surprise removal") > > --- > > drivers/net/ethernet/intel/igb/igb_main.c | 4 ++++ > > 1 file changed, 4 insertions(+) > > > > diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c > > index 1ef4cb871452..8c6bc3db9a3d 100644 > > --- a/drivers/net/ethernet/intel/igb/igb_main.c > > +++ b/drivers/net/ethernet/intel/igb/igb_main.c > > @@ -9651,6 +9651,10 @@ static void igb_io_resume(struct pci_dev *pdev) > > struct igb_adapter *adapter = netdev_priv(netdev); > > > > if (netif_running(netdev)) { > > + if (!test_bit(__IGB_DOWN, &adapter->state)) { > > + dev_info(&pdev->dev, "Resuming from non-fatal error, do nothing.\n"); > > + return; > > I'm not sure this needs to be a dev_info. I am okay with changing it to dev_warn() to match 004d25060c78 ("igb: Fix igb_down hung on surprise removal"). Is that okay? > > Reviewed-by: Jacob Keller <jacob.e.keller@intel.com>
On 2024-09-23 17:10:36 -0700, Jacob Keller wrote: > > > On 9/23/2024 4:11 PM, Jacob Keller wrote: > > > > > > On 9/23/2024 2:22 PM, Mohamed Khalfella wrote: > >> Commit 004d25060c78 ("igb: Fix igb_down hung on surprise removal") > >> changed igb_io_error_detected() to ignore non-fatal pcie errors in order > >> to avoid hung task that can happen when igb_down() is called multiple > >> times. This caused an issue when processing transient non-fatal errors. > >> igb_io_resume(), which is called after igb_io_error_detected(), assumes > >> that device is brought down by igb_io_error_detected() if the interface > >> is up. This resulted in panic with stacktrace below. > >> > >> [ T3256] igb 0000:09:00.0 haeth0: igb: haeth0 NIC Link is Down > >> [ T292] pcieport 0000:00:1c.5: AER: Uncorrected (Non-Fatal) error received: 0000:09:00.0 > >> [ T292] igb 0000:09:00.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID) > >> [ T292] igb 0000:09:00.0: device [8086:1537] error status/mask=00004000/00000000 > >> [ T292] igb 0000:09:00.0: [14] CmpltTO [ 200.105524,009][ T292] igb 0000:09:00.0: AER: TLP Header: 00000000 00000000 00000000 00000000 > >> [ T292] pcieport 0000:00:1c.5: AER: broadcast error_detected message > >> [ T292] igb 0000:09:00.0: Non-correctable non-fatal error reported. > >> [ T292] pcieport 0000:00:1c.5: AER: broadcast mmio_enabled message > >> [ T292] pcieport 0000:00:1c.5: AER: broadcast resume message > >> [ T292] ------------[ cut here ]------------ > >> [ T292] kernel BUG at net/core/dev.c:6539! > >> [ T292] invalid opcode: 0000 [#1] PREEMPT SMP > >> [ T292] RIP: 0010:napi_enable+0x37/0x40 > >> [ T292] Call Trace: > >> [ T292] <TASK> > >> [ T292] ? die+0x33/0x90 > >> [ T292] ? do_trap+0xdc/0x110 > >> [ T292] ? napi_enable+0x37/0x40 > >> [ T292] ? do_error_trap+0x70/0xb0 > >> [ T292] ? napi_enable+0x37/0x40 > >> [ T292] ? napi_enable+0x37/0x40 > >> [ T292] ? exc_invalid_op+0x4e/0x70 > >> [ T292] ? napi_enable+0x37/0x40 > >> [ T292] ? asm_exc_invalid_op+0x16/0x20 > >> [ T292] ? napi_enable+0x37/0x40 > >> [ T292] igb_up+0x41/0x150 > >> [ T292] igb_io_resume+0x25/0x70 > >> [ T292] report_resume+0x54/0x70 > >> [ T292] ? report_frozen_detected+0x20/0x20 > >> [ T292] pci_walk_bus+0x6c/0x90 > >> [ T292] ? aer_print_port_info+0xa0/0xa0 > >> [ T292] pcie_do_recovery+0x22f/0x380 > >> [ T292] aer_process_err_devices+0x110/0x160 > >> [ T292] aer_isr+0x1c1/0x1e0 > >> [ T292] ? disable_irq_nosync+0x10/0x10 > >> [ T292] irq_thread_fn+0x1a/0x60 > >> [ T292] irq_thread+0xe3/0x1a0 > >> [ T292] ? irq_set_affinity_notifier+0x120/0x120 > >> [ T292] ? irq_affinity_notify+0x100/0x100 > >> [ T292] kthread+0xe2/0x110 > >> [ T292] ? kthread_complete_and_exit+0x20/0x20 > >> [ T292] ret_from_fork+0x2d/0x50 > >> [ T292] ? kthread_complete_and_exit+0x20/0x20 > >> [ T292] ret_from_fork_asm+0x11/0x20 > >> [ T292] </TASK> > >> > >> To fix this issue igb_io_resume() checks if the interface is running and > >> the device is not down this means igb_io_error_detected() did not bring > >> the device down and there is no need to bring it up. > >> > >> Signed-off-by: Mohamed Khalfella <mkhalfella@purestorage.com> > >> Reviewed-by: Yuanyuan Zhong<yzhong@purestorage.com> > >> Fixes: 004d25060c78 ("igb: Fix igb_down hung on surprise removal") > >> --- > >> drivers/net/ethernet/intel/igb/igb_main.c | 4 ++++ > >> 1 file changed, 4 insertions(+) > >> > >> diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c > >> index 1ef4cb871452..8c6bc3db9a3d 100644 > >> --- a/drivers/net/ethernet/intel/igb/igb_main.c > >> +++ b/drivers/net/ethernet/intel/igb/igb_main.c > >> @@ -9651,6 +9651,10 @@ static void igb_io_resume(struct pci_dev *pdev) > >> struct igb_adapter *adapter = netdev_priv(netdev); > >> > >> if (netif_running(netdev)) { > >> + if (!test_bit(__IGB_DOWN, &adapter->state)) { > >> + dev_info(&pdev->dev, "Resuming from non-fatal error, do nothing.\n"); > >> + return; > > > > I'm not sure this needs to be a dev_info. > > > > I was thinking dev_dbg, because I don't really see why its relevant to > inform the user we did nothing. Seems like its log spam to me. Good point. I changed it to dev_dbg() in v2. v2: https://lore.kernel.org/all/20240924210604.123175-1-mkhalfella@purestorage.com/ > > > Reviewed-by: Jacob Keller <jacob.e.keller@intel.com> > > >
diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c index 1ef4cb871452..8c6bc3db9a3d 100644 --- a/drivers/net/ethernet/intel/igb/igb_main.c +++ b/drivers/net/ethernet/intel/igb/igb_main.c @@ -9651,6 +9651,10 @@ static void igb_io_resume(struct pci_dev *pdev) struct igb_adapter *adapter = netdev_priv(netdev); if (netif_running(netdev)) { + if (!test_bit(__IGB_DOWN, &adapter->state)) { + dev_info(&pdev->dev, "Resuming from non-fatal error, do nothing.\n"); + return; + } if (igb_up(adapter)) { dev_err(&pdev->dev, "igb_up failed after reset\n"); return;
Commit 004d25060c78 ("igb: Fix igb_down hung on surprise removal") changed igb_io_error_detected() to ignore non-fatal pcie errors in order to avoid hung task that can happen when igb_down() is called multiple times. This caused an issue when processing transient non-fatal errors. igb_io_resume(), which is called after igb_io_error_detected(), assumes that device is brought down by igb_io_error_detected() if the interface is up. This resulted in panic with stacktrace below. [ T3256] igb 0000:09:00.0 haeth0: igb: haeth0 NIC Link is Down [ T292] pcieport 0000:00:1c.5: AER: Uncorrected (Non-Fatal) error received: 0000:09:00.0 [ T292] igb 0000:09:00.0: PCIe Bus Error: severity=Uncorrected (Non-Fatal), type=Transaction Layer, (Requester ID) [ T292] igb 0000:09:00.0: device [8086:1537] error status/mask=00004000/00000000 [ T292] igb 0000:09:00.0: [14] CmpltTO [ 200.105524,009][ T292] igb 0000:09:00.0: AER: TLP Header: 00000000 00000000 00000000 00000000 [ T292] pcieport 0000:00:1c.5: AER: broadcast error_detected message [ T292] igb 0000:09:00.0: Non-correctable non-fatal error reported. [ T292] pcieport 0000:00:1c.5: AER: broadcast mmio_enabled message [ T292] pcieport 0000:00:1c.5: AER: broadcast resume message [ T292] ------------[ cut here ]------------ [ T292] kernel BUG at net/core/dev.c:6539! [ T292] invalid opcode: 0000 [#1] PREEMPT SMP [ T292] RIP: 0010:napi_enable+0x37/0x40 [ T292] Call Trace: [ T292] <TASK> [ T292] ? die+0x33/0x90 [ T292] ? do_trap+0xdc/0x110 [ T292] ? napi_enable+0x37/0x40 [ T292] ? do_error_trap+0x70/0xb0 [ T292] ? napi_enable+0x37/0x40 [ T292] ? napi_enable+0x37/0x40 [ T292] ? exc_invalid_op+0x4e/0x70 [ T292] ? napi_enable+0x37/0x40 [ T292] ? asm_exc_invalid_op+0x16/0x20 [ T292] ? napi_enable+0x37/0x40 [ T292] igb_up+0x41/0x150 [ T292] igb_io_resume+0x25/0x70 [ T292] report_resume+0x54/0x70 [ T292] ? report_frozen_detected+0x20/0x20 [ T292] pci_walk_bus+0x6c/0x90 [ T292] ? aer_print_port_info+0xa0/0xa0 [ T292] pcie_do_recovery+0x22f/0x380 [ T292] aer_process_err_devices+0x110/0x160 [ T292] aer_isr+0x1c1/0x1e0 [ T292] ? disable_irq_nosync+0x10/0x10 [ T292] irq_thread_fn+0x1a/0x60 [ T292] irq_thread+0xe3/0x1a0 [ T292] ? irq_set_affinity_notifier+0x120/0x120 [ T292] ? irq_affinity_notify+0x100/0x100 [ T292] kthread+0xe2/0x110 [ T292] ? kthread_complete_and_exit+0x20/0x20 [ T292] ret_from_fork+0x2d/0x50 [ T292] ? kthread_complete_and_exit+0x20/0x20 [ T292] ret_from_fork_asm+0x11/0x20 [ T292] </TASK> To fix this issue igb_io_resume() checks if the interface is running and the device is not down this means igb_io_error_detected() did not bring the device down and there is no need to bring it up. Signed-off-by: Mohamed Khalfella <mkhalfella@purestorage.com> Reviewed-by: Yuanyuan Zhong<yzhong@purestorage.com> Fixes: 004d25060c78 ("igb: Fix igb_down hung on surprise removal") --- drivers/net/ethernet/intel/igb/igb_main.c | 4 ++++ 1 file changed, 4 insertions(+)