| Message ID | 20201001195247.66636-2-saeed@kernel.org |
|---|---|
| State | Changes Requested |
| Delegated to: | David Miller |
| Headers | show |
| Series | [net,V2,01/15] net/mlx5: Don't allow health work when device is uninitialized | expand |
On Thu, 1 Oct 2020 12:52:33 -0700 saeed@kernel.org wrote: > From: Shay Drory <shayd@mellanox.com> > > On error flow due to failure on driver load, driver can be > un-initializing while a health work is running in the background, > health work shouldn't be allowed at this point, as it needs resources to > be initialized and there is no point to recover on driver load failures. > > Therefore, introducing a new state bit to indicated if device is > initialized, for health work to check before trying to recover the driver. Can't you cancel this work? Or make sure it's not scheduled? IMHO those "INITILIZED" bits are an anti-pattern. > Fixes: b6e0b6bebe07 ("net/mlx5: Fix fatal error handling during device load") > Signed-off-by: Shay Drory <shayd@mellanox.com> > Signed-off-by: Saeed Mahameed <saeedm@mellanox.com> > Signed-off-by: Saeed Mahameed <saeedm@nvidia.com> You signed off twice :) We should teach verify_signoff to catch that..
On Thu, 2020-10-01 at 16:15 -0700, Jakub Kicinski wrote: > On Thu, 1 Oct 2020 12:52:33 -0700 saeed@kernel.org wrote: > > From: Shay Drory <shayd@mellanox.com> > > > > On error flow due to failure on driver load, driver can be > > un-initializing while a health work is running in the background, > > health work shouldn't be allowed at this point, as it needs > > resources to > > be initialized and there is no point to recover on driver load > > failures. > > > > Therefore, introducing a new state bit to indicated if device is > > initialized, for health work to check before trying to recover the > > driver. > > Can't you cancel this work? Or make sure it's not scheduled? > IMHO those "INITILIZED" bits are an anti-pattern. > Shay didn't want to make this patch complicated for net, since this health work should start as early as possible and should be kept running after driver is initialized, even if the driver instance reloads after .. the main issue of the design is that we initialize + allocate the driver structures once on the first boot, after that all reloads will reuse the same structure, so there is some asymmetry that we need to deal with, but nothing is impossible, the solution will be more complicated but won't be too big to make it to net (i hope), I will drop this patch for now. > > Fixes: b6e0b6bebe07 ("net/mlx5: Fix fatal error handling during > > device load") > > Signed-off-by: Shay Drory <shayd@mellanox.com> > > Signed-off-by: Saeed Mahameed <saeedm@mellanox.com> > > Signed-off-by: Saeed Mahameed <saeedm@nvidia.com> > > You signed off twice :) > Will fix this, old mellanox email :/ > We should teach verify_signoff to catch that.. it is not exactly twice, different emails..
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/health.c b/drivers/net/ethernet/mellanox/mlx5/core/health.c index b31f769d2df9..bbaffe7af619 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/health.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/health.c @@ -190,6 +190,11 @@ static bool reset_fw_if_needed(struct mlx5_core_dev *dev) return true; } +static bool mlx5_is_device_initialized(struct mlx5_core_dev *dev) +{ + return test_bit(MLX5_INTERFACE_STATE_INITIALIZED, &dev->intf_state); +} + void mlx5_enter_error_state(struct mlx5_core_dev *dev, bool force) { bool err_detected = false; @@ -201,6 +206,9 @@ void mlx5_enter_error_state(struct mlx5_core_dev *dev, bool force) err_detected = true; } mutex_lock(&dev->intf_state_mutex); + if (!mlx5_is_device_initialized(dev)) + goto unlock; + if (!err_detected && dev->state == MLX5_DEVICE_STATE_INTERNAL_ERROR) goto unlock;/* a previous error is still being handled */ if (dev->state == MLX5_DEVICE_STATE_UNINITIALIZED) { @@ -609,6 +617,9 @@ static void mlx5_fw_fatal_reporter_err_work(struct work_struct *work) dev = container_of(priv, struct mlx5_core_dev, priv); mlx5_enter_error_state(dev, false); + if (!mlx5_is_device_initialized(dev)) + return; + if (IS_ERR_OR_NULL(health->fw_fatal_reporter)) { if (mlx5_health_try_recover(dev)) mlx5_core_err(dev, "health recovery failed\n"); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c index ce43e3feccd9..8ebb8d27d3ac 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c @@ -878,6 +878,7 @@ static int mlx5_init_once(struct mlx5_core_dev *dev) dev->tracer = mlx5_fw_tracer_create(dev); dev->hv_vhca = mlx5_hv_vhca_create(dev); dev->rsc_dump = mlx5_rsc_dump_create(dev); + set_bit(MLX5_INTERFACE_STATE_INITIALIZED, &dev->intf_state); return 0; @@ -906,6 +907,7 @@ static int mlx5_init_once(struct mlx5_core_dev *dev) static void mlx5_cleanup_once(struct mlx5_core_dev *dev) { + clear_bit(MLX5_INTERFACE_STATE_INITIALIZED, &dev->intf_state); mlx5_rsc_dump_destroy(dev); mlx5_hv_vhca_destroy(dev->hv_vhca); mlx5_fw_tracer_destroy(dev->tracer); diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h index c145de0473bc..223aaaaaf8a6 100644 --- a/include/linux/mlx5/driver.h +++ b/include/linux/mlx5/driver.h @@ -594,6 +594,7 @@ enum mlx5_device_state { enum mlx5_interface_state { MLX5_INTERFACE_STATE_UP = BIT(0), + MLX5_INTERFACE_STATE_INITIALIZED = BIT(1), }; enum mlx5_pci_status {