| Message ID | 20240821160640.115552-1-dawid.osuchowski@linux.intel.com |
|---|---|
| State | Accepted |
| Delegated to: | Anthony Nguyen |
| Headers | show |
| Series | [iwl-net,v5] ice: Add netif_device_attach/detach into PF reset flow | expand |
> ----Original Message----- > From: Intel-wired-lan <intel-wired-lan-bounces@osuosl.org> On Behalf Of Dawid Osuchowski > Sent: Wednesday, August 21, 2024 9:37 PM > To: intel-wired-lan@lists.osuosl.org > Cc: Fijalkowski, Maciej <maciej.fijalkowski@intel.com>; Zaremba, Larysa <larysa.zaremba@intel.com>; netdev@vger.kernel.org; kalesh-anakkur.purayil@broadcom.com; Bagnucki, Igor <igor.bagnucki@intel.com>; Jakub Kicinski <kuba@kernel.org>; Dawid Osuchowski <dawid.osuchowski@linux.intel.com> > Subject: [Intel-wired-lan] [PATCH iwl-net v5] ice: Add netif_device_attach/detach into PF reset flow > > Ethtool callbacks can be executed while reset is in progress and try to access deleted resources, e.g. getting coalesce settings can result in a NULL pointer dereference seen below. > > Reproduction steps: > Once the driver is fully initialized, trigger reset: > # echo 1 > /sys/class/net/<interface>/device/reset > when reset is in progress try to get coalesce settings using ethtool: > # ethtool -c <interface> > > BUG: kernel NULL pointer dereference, address: 0000000000000020 PGD 0 P4D 0 > Oops: Oops: 0000 [#1] PREEMPT SMP PTI > CPU: 11 PID: 19713 Comm: ethtool Tainted: G S 6.10.0-rc7+ #7 > RIP: 0010:ice_get_q_coalesce+0x2e/0xa0 [ice] > RSP: 0018:ffffbab1e9bcf6a8 EFLAGS: 00010206 > RAX: 000000000000000c RBX: ffff94512305b028 RCX: 0000000000000000 > RDX: 0000000000000000 RSI: ffff9451c3f2e588 RDI: ffff9451c3f2e588 > RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 > R10: ffff9451c3f2e580 R11: 000000000000001f R12: ffff945121fa9000 > R13: ffffbab1e9bcf760 R14: 0000000000000013 R15: ffffffff9e65dd40 > FS: 00007faee5fbe740(0000) GS:ffff94546fd80000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: 0000000000000020 CR3: 0000000106c2e005 CR4: 00000000001706f0 Call Trace: > <TASK> > ice_get_coalesce+0x17/0x30 [ice] > coalesce_prepare_data+0x61/0x80 > ethnl_default_doit+0xde/0x340 > genl_family_rcv_msg_doit+0xf2/0x150 > genl_rcv_msg+0x1b3/0x2c0 > netlink_rcv_skb+0x5b/0x110 > genl_rcv+0x28/0x40 > netlink_unicast+0x19c/0x290 > netlink_sendmsg+0x222/0x490 > __sys_sendto+0x1df/0x1f0 > __x64_sys_sendto+0x24/0x30 > do_syscall_64+0x82/0x160 > entry_SYSCALL_64_after_hwframe+0x76/0x7e > RIP: 0033:0x7faee60d8e27 > > Calling netif_device_detach() before reset makes the net core not call the driver when ethtool command is issued, the attempt to execute an ethtool command during reset will result in the following message: > > netlink error: No such device > > instead of NULL pointer dereference. Once reset is done and > ice_rebuild() is executing, the netif_device_attach() is called to allow for ethtool operations to occur again in a safe manner. > > Fixes: fcea6f3da546 ("ice: Add stats and ethtool support") > Suggested-by: Jakub Kicinski <kuba@kernel.org> > Reviewed-by: Igor Bagnucki <igor.bagnucki@intel.com> > Signed-off-by: Dawid Osuchowski <dawid.osuchowski@linux.intel.com> > --- > Changes since v1: > * Changed Fixes tag to point to another commit > * Minified the stacktrace > > Changes since v2: > * Moved netif_device_attach() directly into ice_rebuild() and perform it > only on main vsi > > Changes since v3: > * Style changes requested by Przemek Kitszel > > Changes since v4: > * Applied reverse xmas tree rule to declaration of ice_vsi *vsi variable > > Suggestion from Kuba: https://lore.kernel.org/netdev/20240610194756.5be5be90@kernel.org/ > Previous attempt (dropped because it introduced regression with link up): https://lore.kernel.org/netdev/20240722122839.51342-1-dawid.osuchowski@linux.intel.com/ > --- > drivers/net/ethernet/intel/ice/ice_main.c | 7 +++++++ > 1 file changed, 7 insertions(+) > Tested-by: Pucha Himasekhar Reddy <himasekharx.reddy.pucha@intel.com> (A Contingent worker at Intel)
On Wed, Aug 21, 2024 at 6:07 PM Dawid Osuchowski <dawid.osuchowski@linux.intel.com> wrote: > > Ethtool callbacks can be executed while reset is in progress and try to > access deleted resources, e.g. getting coalesce settings can result in a > NULL pointer dereference seen below. > > Reproduction steps: > Once the driver is fully initialized, trigger reset: > # echo 1 > /sys/class/net/<interface>/device/reset > when reset is in progress try to get coalesce settings using ethtool: > # ethtool -c <interface> > > BUG: kernel NULL pointer dereference, address: 0000000000000020 > PGD 0 P4D 0 > Oops: Oops: 0000 [#1] PREEMPT SMP PTI > CPU: 11 PID: 19713 Comm: ethtool Tainted: G S 6.10.0-rc7+ #7 > RIP: 0010:ice_get_q_coalesce+0x2e/0xa0 [ice] > RSP: 0018:ffffbab1e9bcf6a8 EFLAGS: 00010206 > RAX: 000000000000000c RBX: ffff94512305b028 RCX: 0000000000000000 > RDX: 0000000000000000 RSI: ffff9451c3f2e588 RDI: ffff9451c3f2e588 > RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 > R10: ffff9451c3f2e580 R11: 000000000000001f R12: ffff945121fa9000 > R13: ffffbab1e9bcf760 R14: 0000000000000013 R15: ffffffff9e65dd40 > FS: 00007faee5fbe740(0000) GS:ffff94546fd80000(0000) knlGS:0000000000000000 > CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > CR2: 0000000000000020 CR3: 0000000106c2e005 CR4: 00000000001706f0 > Call Trace: > <TASK> > ice_get_coalesce+0x17/0x30 [ice] > coalesce_prepare_data+0x61/0x80 > ethnl_default_doit+0xde/0x340 > genl_family_rcv_msg_doit+0xf2/0x150 > genl_rcv_msg+0x1b3/0x2c0 > netlink_rcv_skb+0x5b/0x110 > genl_rcv+0x28/0x40 > netlink_unicast+0x19c/0x290 > netlink_sendmsg+0x222/0x490 > __sys_sendto+0x1df/0x1f0 > __x64_sys_sendto+0x24/0x30 > do_syscall_64+0x82/0x160 > entry_SYSCALL_64_after_hwframe+0x76/0x7e > RIP: 0033:0x7faee60d8e27 > > Calling netif_device_detach() before reset makes the net core not call > the driver when ethtool command is issued, the attempt to execute an > ethtool command during reset will result in the following message: > > netlink error: No such device > > instead of NULL pointer dereference. Once reset is done and > ice_rebuild() is executing, the netif_device_attach() is called to allow > for ethtool operations to occur again in a safe manner. > > Fixes: fcea6f3da546 ("ice: Add stats and ethtool support") > Suggested-by: Jakub Kicinski <kuba@kernel.org> > Reviewed-by: Igor Bagnucki <igor.bagnucki@intel.com> > Signed-off-by: Dawid Osuchowski <dawid.osuchowski@linux.intel.com> > --- > Changes since v1: > * Changed Fixes tag to point to another commit > * Minified the stacktrace > > Changes since v2: > * Moved netif_device_attach() directly into ice_rebuild() and perform it > only on main vsi > > Changes since v3: > * Style changes requested by Przemek Kitszel > > Changes since v4: > * Applied reverse xmas tree rule to declaration of ice_vsi *vsi variable > > Suggestion from Kuba: https://lore.kernel.org/netdev/20240610194756.5be5be90@kernel.org/ > Previous attempt (dropped because it introduced regression with link up): https://lore.kernel.org/netdev/20240722122839.51342-1-dawid.osuchowski@linux.intel.com/ This v5 passes the tests that the previous attempt referenced above failed. The patch looks sane. Reviewed-by: Michal Schmidt <mschmidt@redhat.com>
diff --git a/drivers/net/ethernet/intel/ice/ice_main.c b/drivers/net/ethernet/intel/ice/ice_main.c index eaa73cc200f4..71bd7bbfb447 100644 --- a/drivers/net/ethernet/intel/ice/ice_main.c +++ b/drivers/net/ethernet/intel/ice/ice_main.c @@ -608,6 +608,9 @@ ice_prepare_for_reset(struct ice_pf *pf, enum ice_reset_req reset_type) memset(&vsi->mqprio_qopt, 0, sizeof(vsi->mqprio_qopt)); } } + + if (vsi->netdev) + netif_device_detach(vsi->netdev); skip: /* clear SW filtering DB */ @@ -7589,6 +7592,7 @@ static void ice_update_pf_netdev_link(struct ice_pf *pf) */ static void ice_rebuild(struct ice_pf *pf, enum ice_reset_req reset_type) { + struct ice_vsi *vsi = ice_get_main_vsi(pf); struct device *dev = ice_pf_to_dev(pf); struct ice_hw *hw = &pf->hw; bool dvm; @@ -7731,6 +7735,9 @@ static void ice_rebuild(struct ice_pf *pf, enum ice_reset_req reset_type) ice_rebuild_arfs(pf); } + if (vsi && vsi->netdev) + netif_device_attach(vsi->netdev); + ice_update_pf_netdev_link(pf); /* tell the firmware we are up */
Ethtool callbacks can be executed while reset is in progress and try to access deleted resources, e.g. getting coalesce settings can result in a NULL pointer dereference seen below. Reproduction steps: Once the driver is fully initialized, trigger reset: # echo 1 > /sys/class/net/<interface>/device/reset when reset is in progress try to get coalesce settings using ethtool: # ethtool -c <interface> BUG: kernel NULL pointer dereference, address: 0000000000000020 PGD 0 P4D 0 Oops: Oops: 0000 [#1] PREEMPT SMP PTI CPU: 11 PID: 19713 Comm: ethtool Tainted: G S 6.10.0-rc7+ #7 RIP: 0010:ice_get_q_coalesce+0x2e/0xa0 [ice] RSP: 0018:ffffbab1e9bcf6a8 EFLAGS: 00010206 RAX: 000000000000000c RBX: ffff94512305b028 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff9451c3f2e588 RDI: ffff9451c3f2e588 RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 R10: ffff9451c3f2e580 R11: 000000000000001f R12: ffff945121fa9000 R13: ffffbab1e9bcf760 R14: 0000000000000013 R15: ffffffff9e65dd40 FS: 00007faee5fbe740(0000) GS:ffff94546fd80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000020 CR3: 0000000106c2e005 CR4: 00000000001706f0 Call Trace: <TASK> ice_get_coalesce+0x17/0x30 [ice] coalesce_prepare_data+0x61/0x80 ethnl_default_doit+0xde/0x340 genl_family_rcv_msg_doit+0xf2/0x150 genl_rcv_msg+0x1b3/0x2c0 netlink_rcv_skb+0x5b/0x110 genl_rcv+0x28/0x40 netlink_unicast+0x19c/0x290 netlink_sendmsg+0x222/0x490 __sys_sendto+0x1df/0x1f0 __x64_sys_sendto+0x24/0x30 do_syscall_64+0x82/0x160 entry_SYSCALL_64_after_hwframe+0x76/0x7e RIP: 0033:0x7faee60d8e27 Calling netif_device_detach() before reset makes the net core not call the driver when ethtool command is issued, the attempt to execute an ethtool command during reset will result in the following message: netlink error: No such device instead of NULL pointer dereference. Once reset is done and ice_rebuild() is executing, the netif_device_attach() is called to allow for ethtool operations to occur again in a safe manner. Fixes: fcea6f3da546 ("ice: Add stats and ethtool support") Suggested-by: Jakub Kicinski <kuba@kernel.org> Reviewed-by: Igor Bagnucki <igor.bagnucki@intel.com> Signed-off-by: Dawid Osuchowski <dawid.osuchowski@linux.intel.com> --- Changes since v1: * Changed Fixes tag to point to another commit * Minified the stacktrace Changes since v2: * Moved netif_device_attach() directly into ice_rebuild() and perform it only on main vsi Changes since v3: * Style changes requested by Przemek Kitszel Changes since v4: * Applied reverse xmas tree rule to declaration of ice_vsi *vsi variable Suggestion from Kuba: https://lore.kernel.org/netdev/20240610194756.5be5be90@kernel.org/ Previous attempt (dropped because it introduced regression with link up): https://lore.kernel.org/netdev/20240722122839.51342-1-dawid.osuchowski@linux.intel.com/ --- drivers/net/ethernet/intel/ice/ice_main.c | 7 +++++++ 1 file changed, 7 insertions(+)