From patchwork Thu Jan 17 15:05:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eran Ben Elisha X-Patchwork-Id: 1026701 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43gS7X5DnXz9sD9 for ; Fri, 18 Jan 2019 02:05:44 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728034AbfAQPFm (ORCPT ); Thu, 17 Jan 2019 10:05:42 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:55231 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728160AbfAQPFj (ORCPT ); Thu, 17 Jan 2019 10:05:39 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from eranbe@mellanox.com) with ESMTPS (AES256-SHA encrypted); 17 Jan 2019 17:05:33 +0200 Received: from dev-l-vrt-198.mtl.labs.mlnx (dev-l-vrt-198.mtl.labs.mlnx [10.134.198.1]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x0HF5WoZ029214; Thu, 17 Jan 2019 17:05:33 +0200 From: Eran Ben Elisha To: netdev@vger.kernel.org, Jiri Pirko , "David S. Miller" , Ariel Almog , Aya Levin , Eran Ben Elisha , Moshe Shemesh Subject: [PATCH net-next 26/27] net/mlx5: Report devlink health on FW fatal issues Date: Thu, 17 Jan 2019 17:05:20 +0200 Message-Id: <1547737521-29888-27-git-send-email-eranbe@mellanox.com> X-Mailer: git-send-email 1.8.4.3 In-Reply-To: <1547737521-29888-1-git-send-email-eranbe@mellanox.com> References: <1547737521-29888-1-git-send-email-eranbe@mellanox.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Moshe Shemesh Report devlink health on FW fatal issues via FW_fatal_reporter. The driver recover flow for FW fatal error is now being handled by the devlink health. Having the recovery controlled by devlink health, the user has the ability to cancel the auto-recovery for debug session and run it manually. Signed-off-by: Moshe Shemesh Signed-off-by: Eran Ben Elisha Reviewed-by: Saeed Mahameed --- .../net/ethernet/mellanox/mlx5/core/devlink.c | 19 +++++++++++++++ .../net/ethernet/mellanox/mlx5/core/devlink.h | 1 + .../net/ethernet/mellanox/mlx5/core/health.c | 23 +------------------ 3 files changed, 21 insertions(+), 22 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c index 406856002dd1..e03160626171 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c @@ -235,6 +235,25 @@ static const struct devlink_health_reporter_ops mlx5_fw_reporter_ops = { .diagnose = mlx5_fw_reporter_diagnose, }; +void mlx5_fw_fatal_reporter_work(struct work_struct *work) +{ + struct mlx5_fw_reporter_ctx fw_reporter_ctx; + struct mlx5_core_health *health; + struct delayed_work *dwork; + struct mlx5_core_dev *dev; + struct mlx5_priv *priv; + + dwork = container_of(work, struct delayed_work, work); + health = container_of(dwork, struct mlx5_core_health, recover_work); + priv = container_of(health, struct mlx5_priv, health); + dev = container_of(priv, struct mlx5_core_dev, priv); + + fw_reporter_ctx.err_synd = health->synd; + fw_reporter_ctx.miss_counter = health->miss_counter; + devlink_health_report(dev->fw_fatal_reporter, "FW recovery", + &fw_reporter_ctx); +} + static int mlx5_fw_fatal_reporter_recover(struct devlink_health_reporter *reporter, void *priv_ctx) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.h b/drivers/net/ethernet/mellanox/mlx5/core/devlink.h index 9b544f677aa7..9c5eba632293 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.h @@ -17,5 +17,6 @@ void mlx5_devlink_unregister(struct devlink *devlink); int mlx5_fw_reporters_create(struct mlx5_core_dev *dev); void mlx5_fw_reporters_destroy(struct mlx5_core_dev *dev); void mlx5_fw_reporter_err_work(struct work_struct *work); +void mlx5_fw_fatal_reporter_work(struct work_struct *work); #endif /* __MLX5_DEVLINK_H__ */ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/health.c b/drivers/net/ethernet/mellanox/mlx5/core/health.c index d3ea624bd03c..189000545162 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/health.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/health.c @@ -303,27 +303,6 @@ static void mlx5_handle_bad_state(struct mlx5_core_dev *dev) mlx5_disable_device(dev); } -static void health_recover(struct work_struct *work) -{ - struct mlx5_core_health *health; - struct delayed_work *dwork; - struct mlx5_core_dev *dev; - struct mlx5_priv *priv; - - dwork = container_of(work, struct delayed_work, work); - health = container_of(dwork, struct mlx5_core_health, recover_work); - priv = container_of(health, struct mlx5_priv, health); - dev = container_of(priv, struct mlx5_core_dev, priv); - - if (mlx5_sensor_pci_not_working(dev)) { - dev_err(&dev->pdev->dev, "health recovery flow aborted, PCI reads still not working\n"); - return; - } - - dev_err(&dev->pdev->dev, "starting health recovery flow\n"); - mlx5_recover_device(dev); -} - /* How much time to wait until health resetting the driver (in msecs) */ #define MLX5_RECOVERY_DELAY_MSECS 60000 #define MLX5_RECOVERY_NO_DELAY 0 @@ -586,7 +565,7 @@ int mlx5_health_init(struct mlx5_core_dev *dev) spin_lock_init(&health->wq_lock); INIT_WORK(&health->work, health_care); INIT_WORK(&health->report_work, mlx5_fw_reporter_err_work); - INIT_DELAYED_WORK(&health->recover_work, health_recover); + INIT_DELAYED_WORK(&health->recover_work, mlx5_fw_fatal_reporter_work); health->crdump = NULL; return 0;