From patchwork Thu Jan 10 10:28:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saeed Mahameed X-Patchwork-Id: 1022840 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43b2Ky2YjLz9sMQ for ; Thu, 10 Jan 2019 21:29:26 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728134AbfAJK3Z (ORCPT ); Thu, 10 Jan 2019 05:29:25 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:40916 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726255AbfAJK3Y (ORCPT ); Thu, 10 Jan 2019 05:29:24 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from saeedm@mellanox.com) with ESMTPS (AES256-SHA encrypted); 10 Jan 2019 12:29:21 +0200 Received: from sx1.mtl.com ([172.16.5.7]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x0AATDGF031830; Thu, 10 Jan 2019 12:29:18 +0200 From: Saeed Mahameed To: "David S. Miller" Cc: netdev@vger.kernel.org, Alex Vesker , Feras Daoud , Saeed Mahameed Subject: [net-next 1/9] net/mlx5: Move all devlink related functions calls to devlink.c Date: Thu, 10 Jan 2019 12:28:57 +0200 Message-Id: <20190110102906.3751-2-saeedm@mellanox.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190110102906.3751-1-saeedm@mellanox.com> References: <20190110102906.3751-1-saeedm@mellanox.com> MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Alex Vesker Centralize all devlink related callbacks in one file. In the downstream patch, some more functionality will be added, this patch is preparing the driver infrastructure for it. Currently, move devlink un/register functions calls into this file. Signed-off-by: Alex Vesker Reviewed-by: Feras Daoud Signed-off-by: Saeed Mahameed --- drivers/net/ethernet/mellanox/mlx5/core/devlink.c | 14 ++++++++++++++ drivers/net/ethernet/mellanox/mlx5/core/devlink.h | 12 ++++++++++++ drivers/net/ethernet/mellanox/mlx5/core/main.c | 5 +++-- 3 files changed, 29 insertions(+), 2 deletions(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/devlink.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/devlink.h diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c new file mode 100644 index 000000000000..ddfaf06c910c --- /dev/null +++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c @@ -0,0 +1,14 @@ +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB +/* Copyright (c) 2018 Mellanox Technologies */ + +#include + +int mlx5_devlink_register(struct devlink *devlink, struct device *dev) +{ + return devlink_register(devlink, dev); +} + +void mlx5_devlink_unregister(struct devlink *devlink) +{ + devlink_unregister(devlink); +} diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.h b/drivers/net/ethernet/mellanox/mlx5/core/devlink.h new file mode 100644 index 000000000000..6174113e3fe4 --- /dev/null +++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.h @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* Copyright (c) 2018 Mellanox Technologies */ + +#ifndef __MLX5_DEVLINK_H__ +#define __MLX5_DEVLINK_H__ + +#include + +int mlx5_devlink_register(struct devlink *devlink, struct device *dev); +void mlx5_devlink_unregister(struct devlink *devlink); + +#endif /* __MLX5_DEVLINK_H__ */ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c index 77896c11f6f3..bd50d4adc6bf 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c @@ -56,6 +56,7 @@ #include "fs_core.h" #include "lib/mpfs.h" #include "eswitch.h" +#include "devlink.h" #include "lib/mlx5.h" #include "fpga/core.h" #include "fpga/ipsec.h" @@ -1204,7 +1205,7 @@ static int init_one(struct pci_dev *pdev, request_module_nowait(MLX5_IB_MOD); - err = devlink_register(devlink, &pdev->dev); + err = mlx5_devlink_register(devlink, &pdev->dev); if (err) goto clean_load; @@ -1231,7 +1232,7 @@ static void remove_one(struct pci_dev *pdev) struct devlink *devlink = priv_to_devlink(dev); struct mlx5_priv *priv = &dev->priv; - devlink_unregister(devlink); + mlx5_devlink_unregister(devlink); mlx5_unregister_device(dev); if (mlx5_unload_one(dev, priv, true)) { From patchwork Thu Jan 10 10:28:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saeed Mahameed X-Patchwork-Id: 1022841 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43b2L40DBnz9sMQ for ; Thu, 10 Jan 2019 21:29:31 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728148AbfAJK3a (ORCPT ); Thu, 10 Jan 2019 05:29:30 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:40942 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726255AbfAJK3a (ORCPT ); Thu, 10 Jan 2019 05:29:30 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from saeedm@mellanox.com) with ESMTPS (AES256-SHA encrypted); 10 Jan 2019 12:29:26 +0200 Received: from sx1.mtl.com ([172.16.5.7]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x0AATDGG031830; Thu, 10 Jan 2019 12:29:22 +0200 From: Saeed Mahameed To: "David S. Miller" Cc: netdev@vger.kernel.org, Alex Vesker , Feras Daoud , Saeed Mahameed Subject: [net-next 2/9] net/mlx5: Add Vendor Specific Capability access gateway Date: Thu, 10 Jan 2019 12:28:58 +0200 Message-Id: <20190110102906.3751-3-saeedm@mellanox.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190110102906.3751-1-saeedm@mellanox.com> References: <20190110102906.3751-1-saeedm@mellanox.com> MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Alex Vesker The Vendor Specific Capability (VSC) is used to activate a gateway interfacing with the device. The gateway is used to read or write device configurations, which are organized in different domains (spaces). A configuration access may result in multiple actions, reads, writes. Example usages are accessing the Crspace domain to read the crspace or locking a device semaphore using the Semaphore domain. The configuration access use pci_cfg_access to prevent parallel access to the VSC space by the driver and userspace calls. Signed-off-by: Alex Vesker Signed-off-by: Feras Daoud Signed-off-by: Saeed Mahameed --- .../net/ethernet/mellanox/mlx5/core/Makefile | 2 +- .../ethernet/mellanox/mlx5/core/lib/pci_vsc.c | 282 ++++++++++++++++++ .../ethernet/mellanox/mlx5/core/lib/pci_vsc.h | 25 ++ .../net/ethernet/mellanox/mlx5/core/main.c | 3 + include/linux/mlx5/driver.h | 1 + 5 files changed, 312 insertions(+), 1 deletion(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/lib/pci_vsc.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/lib/pci_vsc.h diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Makefile b/drivers/net/ethernet/mellanox/mlx5/core/Makefile index 9de9abacf7f6..3ed5d4b6b4c0 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/Makefile +++ b/drivers/net/ethernet/mellanox/mlx5/core/Makefile @@ -15,7 +15,7 @@ mlx5_core-y := main.o cmd.o debugfs.o fw.o eq.o uar.o pagealloc.o \ health.o mcg.o cq.o alloc.o qp.o port.o mr.o pd.o \ mad.o transobj.o vport.o sriov.o fs_cmd.o fs_core.o \ fs_counters.o rl.o lag.o dev.o events.o wq.o lib/gid.o \ - lib/devcom.o diag/fs_tracepoint.o diag/fw_tracer.o + lib/devcom.o lib/pci_vsc.o diag/fs_tracepoint.o diag/fw_tracer.o # # Netdev basic diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/pci_vsc.c b/drivers/net/ethernet/mellanox/mlx5/core/lib/pci_vsc.c new file mode 100644 index 000000000000..b26e14c4456b --- /dev/null +++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/pci_vsc.c @@ -0,0 +1,282 @@ +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB +/* Copyright (c) 2018 Mellanox Technologies */ + +#include +#include "mlx5_core.h" +#include "pci_vsc.h" + +#define MLX5_EXTRACT_C(source, offset, size) \ + ((((unsigned)(source)) >> (offset)) & MLX5_ONES32(size)) +#define MLX5_EXTRACT(src, start, len) \ + (((len) == 32) ? (src) : MLX5_EXTRACT_C(src, start, len)) +#define MLX5_ONES32(size) \ + ((size) ? (0xffffffff >> (32 - (size))) : 0) +#define MLX5_MASK32(offset, size) \ + (MLX5_ONES32(size) << (offset)) +#define MLX5_MERGE_C(rsrc1, rsrc2, start, len) \ + ((((rsrc2) << (start)) & (MLX5_MASK32((start), (len)))) | \ + ((rsrc1) & (~MLX5_MASK32((start), (len))))) +#define MLX5_MERGE(rsrc1, rsrc2, start, len) \ + (((len) == 32) ? (rsrc2) : MLX5_MERGE_C(rsrc1, rsrc2, start, len)) +#define vsc_read(dev, offset, val) \ + pci_read_config_dword((dev)->pdev, (dev)->vsc_addr + (offset), (val)) +#define vsc_write(dev, offset, val) \ + pci_write_config_dword((dev)->pdev, (dev)->vsc_addr + (offset), (val)) +#define VSC_MAX_RETRIES 2048 + +enum mlx5_vsc_state { + MLX5_VSC_UNLOCK, + MLX5_VSC_LOCK, +}; + +enum { + VSC_CTRL_OFFSET = 0x4, + VSC_COUNTER_OFFSET = 0x8, + VSC_SEMAPHORE_OFFSET = 0xc, + VSC_ADDR_OFFSET = 0x10, + VSC_DATA_OFFSET = 0x14, + + VSC_FLAG_BIT_OFFS = 31, + VSC_FLAG_BIT_LEN = 1, + + VSC_SYND_BIT_OFFS = 30, + VSC_SYND_BIT_LEN = 1, + + VSC_ADDR_BIT_OFFS = 0, + VSC_ADDR_BIT_LEN = 30, + + VSC_SPACE_BIT_OFFS = 0, + VSC_SPACE_BIT_LEN = 16, + + VSC_SIZE_VLD_BIT_OFFS = 28, + VSC_SIZE_VLD_BIT_LEN = 1, + + VSC_STATUS_BIT_OFFS = 29, + VSC_STATUS_BIT_LEN = 3, +}; + +void mlx5_vsc_init(struct mlx5_core_dev *dev) +{ + dev->vsc_addr = pci_find_capability(dev->pdev, + PCI_CAP_ID_VNDR); + if (!dev->vsc_addr) + mlx5_core_warn(dev, "Failed to get valid vendor specific ID\n"); +} + +int mlx5_vsc_gw_lock(struct mlx5_core_dev *dev) +{ + u32 counter = 0; + int retries = 0; + u32 lock_val; + int ret; + + pci_cfg_access_lock(dev->pdev); + do { + if (retries > VSC_MAX_RETRIES) { + ret = -EBUSY; + goto pci_unlock; + } + + /* Check if semaphore is already locked */ + ret = vsc_read(dev, VSC_SEMAPHORE_OFFSET, &lock_val); + if (ret) + goto pci_unlock; + + if (lock_val) { + retries++; + usleep_range(1000, 2000); + continue; + } + + /* + * Read and write counter value, if written value is + * the same, semaphore was acquired successfully. + */ + ret = vsc_read(dev, VSC_COUNTER_OFFSET, &counter); + if (ret) + goto pci_unlock; + + ret = vsc_write(dev, VSC_SEMAPHORE_OFFSET, counter); + if (ret) + goto pci_unlock; + + ret = vsc_read(dev, VSC_SEMAPHORE_OFFSET, &lock_val); + if (ret) + goto pci_unlock; + + retries++; + } while (counter != lock_val); + + return 0; + +pci_unlock: + pci_cfg_access_unlock(dev->pdev); + return ret; +} + +int mlx5_vsc_gw_unlock(struct mlx5_core_dev *dev) +{ + int ret; + + ret = vsc_write(dev, VSC_SEMAPHORE_OFFSET, MLX5_VSC_UNLOCK); + pci_cfg_access_unlock(dev->pdev); + return ret; +} + +int mlx5_vsc_gw_set_space(struct mlx5_core_dev *dev, u16 space, + u32 *ret_space_size) +{ + int ret; + u32 val = 0; + + if (!mlx5_vsc_accessible(dev)) + return -EINVAL; + + if (ret_space_size) + *ret_space_size = 0; + + /* Get a unique val */ + ret = vsc_read(dev, VSC_CTRL_OFFSET, &val); + if (ret) + goto out; + + /* Try to modify the lock */ + val = MLX5_MERGE(val, space, VSC_SPACE_BIT_OFFS, VSC_SPACE_BIT_LEN); + ret = vsc_write(dev, VSC_CTRL_OFFSET, val); + if (ret) + goto out; + + /* Verify lock was modified */ + ret = vsc_read(dev, VSC_CTRL_OFFSET, &val); + if (ret) + goto out; + + if (MLX5_EXTRACT(val, VSC_STATUS_BIT_OFFS, VSC_STATUS_BIT_LEN) == 0) + return -EINVAL; + + /* Get space max address if indicated by size valid bit */ + if (ret_space_size && + MLX5_EXTRACT(val, VSC_SIZE_VLD_BIT_OFFS, VSC_SIZE_VLD_BIT_LEN)) { + ret = vsc_read(dev, VSC_ADDR_OFFSET, &val); + if (ret) { + mlx5_core_warn(dev, "Failed to get max space size\n"); + goto out; + } + *ret_space_size = MLX5_EXTRACT(val, VSC_ADDR_BIT_OFFS, + VSC_ADDR_BIT_LEN); + } + return 0; + +out: + return ret; +} + +static int mlx5_vsc_wait_on_flag(struct mlx5_core_dev *dev, u8 expected_val) +{ + int retries = 0; + u32 flag; + int ret; + + do { + if (retries > VSC_MAX_RETRIES) + return -EBUSY; + + ret = vsc_read(dev, VSC_ADDR_OFFSET, &flag); + flag = MLX5_EXTRACT(flag, VSC_FLAG_BIT_OFFS, VSC_FLAG_BIT_LEN); + retries++; + + if ((retries & 0xf) == 0) + usleep_range(1000, 2000); + + } while (flag != expected_val); + + return 0; +} + +static int mlx5_vsc_gw_write(struct mlx5_core_dev *dev, unsigned int address, + u32 data) +{ + int ret; + + if (MLX5_EXTRACT(address, VSC_SYND_BIT_OFFS, + VSC_FLAG_BIT_LEN + VSC_SYND_BIT_LEN)) + return -EINVAL; + + /* Set flag to 0x1 */ + address = MLX5_MERGE(address, 1, VSC_FLAG_BIT_OFFS, 1); + ret = vsc_write(dev, VSC_DATA_OFFSET, data); + if (ret) + goto out; + + ret = vsc_write(dev, VSC_ADDR_OFFSET, address); + if (ret) + goto out; + + /* Wait for the flag to be cleared */ + ret = mlx5_vsc_wait_on_flag(dev, 0); + +out: + return ret; +} + +static int mlx5_vsc_gw_read(struct mlx5_core_dev *dev, unsigned int address, + u32 *data) +{ + int ret; + + if (MLX5_EXTRACT(address, VSC_SYND_BIT_OFFS, + VSC_FLAG_BIT_LEN + VSC_SYND_BIT_LEN)) + return -EINVAL; + + ret = vsc_write(dev, VSC_ADDR_OFFSET, address); + if (ret) + goto out; + + ret = mlx5_vsc_wait_on_flag(dev, 1); + if (ret) + goto out; + + ret = vsc_read(dev, VSC_DATA_OFFSET, data); +out: + return ret; +} + +static int mlx5_vsc_gw_read_fast(struct mlx5_core_dev *dev, + unsigned int read_addr, + unsigned int *next_read_addr, + u32 *data) +{ + int ret; + + ret = mlx5_vsc_gw_read(dev, read_addr, data); + if (ret) + goto out; + + ret = vsc_read(dev, VSC_ADDR_OFFSET, next_read_addr); + if (ret) + goto out; + + *next_read_addr = MLX5_EXTRACT(*next_read_addr, VSC_ADDR_BIT_OFFS, + VSC_ADDR_BIT_LEN); + + if (*next_read_addr <= read_addr) + ret = EINVAL; +out: + return ret; +} + +int mlx5_vsc_gw_read_block_fast(struct mlx5_core_dev *dev, u32 *data, + int length) +{ + unsigned int next_read_addr = 0; + unsigned int read_addr = 0; + + while (read_addr < length) { + if (mlx5_vsc_gw_read_fast(dev, read_addr, &next_read_addr, + &data[(read_addr >> 2)])) + return read_addr; + + read_addr = next_read_addr; + } + return length; +} diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/pci_vsc.h b/drivers/net/ethernet/mellanox/mlx5/core/lib/pci_vsc.h new file mode 100644 index 000000000000..c30ff69ffa3e --- /dev/null +++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/pci_vsc.h @@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* Copyright (c) 2018 Mellanox Technologies */ + +#ifndef __MLX5_PCI_VSC_H__ +#define __MLX5_PCI_VSC_H__ + +enum { + MLX5_VSC_SPACE_SCAN_CRSPACE = 0x7, +}; + +void mlx5_vsc_init(struct mlx5_core_dev *dev); +void mlx5_vsc_cleanup(struct mlx5_core_dev *dev); +int mlx5_vsc_gw_lock(struct mlx5_core_dev *dev); +int mlx5_vsc_gw_unlock(struct mlx5_core_dev *dev); +int mlx5_vsc_gw_set_space(struct mlx5_core_dev *dev, u16 space, + u32 *ret_space_size); +int mlx5_vsc_gw_read_block_fast(struct mlx5_core_dev *dev, u32 *data, + int length); + +static inline bool mlx5_vsc_accessible(struct mlx5_core_dev *dev) +{ + return !!dev->vsc_addr; +} + +#endif /* __MLX5_PCI_VSC_H__ */ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c index bd50d4adc6bf..e49afa9571f7 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c @@ -66,6 +66,7 @@ #include "lib/vxlan.h" #include "lib/devcom.h" #include "diag/fw_tracer.h" +#include "lib/pci_vsc.h" MODULE_AUTHOR("Eli Cohen "); MODULE_DESCRIPTION("Mellanox 5th generation network adapters (ConnectX series) core driver"); @@ -1205,6 +1206,8 @@ static int init_one(struct pci_dev *pdev, request_module_nowait(MLX5_IB_MOD); + mlx5_vsc_init(dev); + err = mlx5_devlink_register(devlink, &pdev->dev); if (err) goto clean_load; diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h index 4d16ba04790e..1ef75c3eddc9 100644 --- a/include/linux/mlx5/driver.h +++ b/include/linux/mlx5/driver.h @@ -677,6 +677,7 @@ struct mlx5_core_dev { struct mlx5_ib_clock_info *clock_info; struct page *clock_info_page; struct mlx5_fw_tracer *tracer; + u32 vsc_addr; }; struct mlx5_db { From patchwork Thu Jan 10 10:28:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saeed Mahameed X-Patchwork-Id: 1022842 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43b2L85B8Cz9sMp for ; Thu, 10 Jan 2019 21:29:36 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728161AbfAJK3f (ORCPT ); Thu, 10 Jan 2019 05:29:35 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:40952 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726255AbfAJK3f (ORCPT ); Thu, 10 Jan 2019 05:29:35 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from saeedm@mellanox.com) with ESMTPS (AES256-SHA encrypted); 10 Jan 2019 12:29:30 +0200 Received: from sx1.mtl.com ([172.16.5.7]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x0AATDGH031830; Thu, 10 Jan 2019 12:29:27 +0200 From: Saeed Mahameed To: "David S. Miller" Cc: netdev@vger.kernel.org, Alex Vesker , Feras Daoud , Saeed Mahameed Subject: [net-next 3/9] net/mlx5: Add Crdump FW snapshot support Date: Thu, 10 Jan 2019 12:28:59 +0200 Message-Id: <20190110102906.3751-4-saeedm@mellanox.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190110102906.3751-1-saeedm@mellanox.com> References: <20190110102906.3751-1-saeedm@mellanox.com> MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Alex Vesker Crdump allows the driver to create a snapshot of the FW PCI crspace. This is useful in case of catastrophic issues which require FW reset. The snapshot can be used for later debug. The snapshot is exposed using devlink, cr-space address regions are registered on init and snapshots are attached once a new snapshot is collected by the driver. Signed-off-by: Alex Vesker Reviewed-by: Feras Daoud Signed-off-by: Saeed Mahameed --- .../net/ethernet/mellanox/mlx5/core/Makefile | 3 +- .../ethernet/mellanox/mlx5/core/diag/crdump.c | 173 ++++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/health.c | 1 + .../ethernet/mellanox/mlx5/core/lib/mlx5.h | 2 + .../net/ethernet/mellanox/mlx5/core/main.c | 5 + include/linux/mlx5/driver.h | 4 + 6 files changed, 187 insertions(+), 1 deletion(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/diag/crdump.c diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Makefile b/drivers/net/ethernet/mellanox/mlx5/core/Makefile index 3ed5d4b6b4c0..f5415e229614 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/Makefile +++ b/drivers/net/ethernet/mellanox/mlx5/core/Makefile @@ -15,7 +15,8 @@ mlx5_core-y := main.o cmd.o debugfs.o fw.o eq.o uar.o pagealloc.o \ health.o mcg.o cq.o alloc.o qp.o port.o mr.o pd.o \ mad.o transobj.o vport.o sriov.o fs_cmd.o fs_core.o \ fs_counters.o rl.o lag.o dev.o events.o wq.o lib/gid.o \ - lib/devcom.o lib/pci_vsc.o diag/fs_tracepoint.o diag/fw_tracer.o + lib/devcom.o lib/pci_vsc.o diag/fs_tracepoint.o diag/fw_tracer.o \ + diag/crdump.o # # Netdev basic diff --git a/drivers/net/ethernet/mellanox/mlx5/core/diag/crdump.c b/drivers/net/ethernet/mellanox/mlx5/core/diag/crdump.c new file mode 100644 index 000000000000..78c92d235641 --- /dev/null +++ b/drivers/net/ethernet/mellanox/mlx5/core/diag/crdump.c @@ -0,0 +1,173 @@ +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB +/* Copyright (c) 2018 Mellanox Technologies */ + +#include +#include +#include +#include "mlx5_core.h" +#include "lib/pci_vsc.h" + +#define BAD_ACCESS 0xBADACCE5 +#define MLX5_PROTECTED_CR_SCAN_CRSPACE 0x7 +#define MAX_NUM_OF_DUMPS_TO_STORE (8) + +static const char *region_cr_space_str = "cr-space"; + +struct mlx5_fw_crdump { + u32 size; + struct devlink_region *region_crspace; +}; + +bool mlx5_crdump_enbaled(struct mlx5_core_dev *dev) +{ + struct mlx5_priv *priv = &dev->priv; + + return (!!priv->health.crdump); +} + +static int mlx5_crdump_fill(struct mlx5_core_dev *dev) +{ + struct devlink *devlink = priv_to_devlink(dev); + struct mlx5_priv *priv = &dev->priv; + struct mlx5_fw_crdump *crdump = priv->health.crdump; + int i, ret = 0; + u32 *cr_data; + u32 id; + + cr_data = kvmalloc(crdump->size, GFP_KERNEL); + if (!cr_data) + return -ENOMEM; + + for (i = 0; i < (crdump->size / 4); i++) + cr_data[i] = BAD_ACCESS; + + ret = mlx5_vsc_gw_read_block_fast(dev, cr_data, crdump->size); + if (ret <= 0) + goto free_data; + + if (crdump->size != ret) { + mlx5_core_warn(dev, "failed to read full dump, read %d out of %u\n", + ret, crdump->size); + ret = -EINVAL; + goto free_data; + } + + /* Get the available snapshot ID for the dumps */ + id = devlink_region_shapshot_id_get(devlink); + ret = devlink_region_snapshot_create(crdump->region_crspace, + crdump->size, (u8 *)cr_data, + id, &kvfree); + if (ret) { + mlx5_core_warn(dev, "crdump: devlink create %s snapshot id %d err %d\n", + region_cr_space_str, id, ret); + goto free_data; + } else { + mlx5_core_info(dev, "crdump: added snapshot %d to devlink region %s\n", + id, region_cr_space_str); + } + return 0; + +free_data: + kvfree(cr_data); + return ret; +} + +int mlx5_crdump_collect(struct mlx5_core_dev *dev) +{ + int ret = 0; + + if (!mlx5_crdump_enbaled(dev)) + return -ENODEV; + + ret = mlx5_vsc_gw_lock(dev); + if (ret) { + mlx5_core_warn(dev, "crdump: failed to lock vsc gw err %d\n", + ret); + return ret; + } + + ret = mlx5_vsc_gw_set_space(dev, MLX5_VSC_SPACE_SCAN_CRSPACE, NULL); + if (ret) + goto unlock; + + ret = mlx5_crdump_fill(dev); + +unlock: + mlx5_vsc_gw_unlock(dev); + return ret; +} + +int mlx5_crdump_init(struct mlx5_core_dev *dev) +{ + struct devlink *devlink = priv_to_devlink(dev); + struct mlx5_priv *priv = &dev->priv; + struct mlx5_fw_crdump *crdump; + u32 space_size; + int ret; + + if (!mlx5_core_is_pf(dev) || !mlx5_vsc_accessible(dev) || + mlx5_crdump_enbaled(dev)) + return 0; + + ret = mlx5_vsc_gw_lock(dev); + if (ret) + return ret; + + /* Check if space is supported and get space size */ + ret = mlx5_vsc_gw_set_space(dev, MLX5_VSC_SPACE_SCAN_CRSPACE, + &space_size); + if (ret) { + /* Unlock and mask error since space is not supported */ + mlx5_vsc_gw_unlock(dev); + return 0; + } + + if (!space_size) { + mlx5_core_warn(dev, "Invalid Crspace size, zero\n"); + mlx5_vsc_gw_unlock(dev); + return -EINVAL; + } + + ret = mlx5_vsc_gw_unlock(dev); + if (ret) + return ret; + + crdump = kzalloc(sizeof(*crdump), GFP_KERNEL); + if (!crdump) + return -ENOMEM; + + /* Create cr-space region */ + crdump->size = space_size; + crdump->region_crspace = + devlink_region_create(devlink, + region_cr_space_str, + MAX_NUM_OF_DUMPS_TO_STORE, + space_size); + if (IS_ERR(crdump->region_crspace)) { + mlx5_core_warn(dev, + "crdump: create devlink region %s err %ld\n", + region_cr_space_str, + PTR_ERR(crdump->region_crspace)); + ret = PTR_ERR(crdump->region_crspace); + goto free_crdump; + } + priv->health.crdump = crdump; + return 0; + +free_crdump: + kfree(crdump); + return ret; +} + +void mlx5_crdump_cleanup(struct mlx5_core_dev *dev) +{ + struct mlx5_priv *priv = &dev->priv; + struct mlx5_fw_crdump *crdump = priv->health.crdump; + + if (!crdump) + return; + + devlink_region_destroy(crdump->region_crspace); + kfree(crdump); + priv->health.crdump = NULL; +} diff --git a/drivers/net/ethernet/mellanox/mlx5/core/health.c b/drivers/net/ethernet/mellanox/mlx5/core/health.c index 196c07383082..883c1e8ffdc2 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/health.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/health.c @@ -378,6 +378,7 @@ int mlx5_health_init(struct mlx5_core_dev *dev) spin_lock_init(&health->wq_lock); INIT_WORK(&health->work, health_care); INIT_DELAYED_WORK(&health->recover_work, health_recover); + health->crdump = NULL; return 0; } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/mlx5.h b/drivers/net/ethernet/mellanox/mlx5/core/lib/mlx5.h index 397a2847867a..29f4ed7e5034 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/lib/mlx5.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/mlx5.h @@ -41,6 +41,8 @@ int mlx5_core_reserve_gids(struct mlx5_core_dev *dev, unsigned int count); void mlx5_core_unreserve_gids(struct mlx5_core_dev *dev, unsigned int count); int mlx5_core_reserved_gid_alloc(struct mlx5_core_dev *dev, int *gid_index); void mlx5_core_reserved_gid_free(struct mlx5_core_dev *dev, int gid_index); +int mlx5_crdump_init(struct mlx5_core_dev *dev); +void mlx5_crdump_cleanup(struct mlx5_core_dev *dev); /* TODO move to lib/events.h */ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c index e49afa9571f7..42a51902c26a 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c @@ -1212,6 +1212,10 @@ static int init_one(struct pci_dev *pdev, if (err) goto clean_load; + err = mlx5_crdump_init(dev); + if (err) + dev_err(&pdev->dev, "mlx5_crdump_init failed with error code %d\n", err); + pci_save_state(pdev); return 0; @@ -1235,6 +1239,7 @@ static void remove_one(struct pci_dev *pdev) struct devlink *devlink = priv_to_devlink(dev); struct mlx5_priv *priv = &dev->priv; + mlx5_crdump_cleanup(dev); mlx5_devlink_unregister(devlink); mlx5_unregister_device(dev); diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h index 1ef75c3eddc9..2f41603dc728 100644 --- a/include/linux/mlx5/driver.h +++ b/include/linux/mlx5/driver.h @@ -53,6 +53,7 @@ #include #include #include +#include enum { MLX5_BOARD_ID_LEN = 64, @@ -425,6 +426,8 @@ struct mlx5_sq_bfreg { unsigned int offset; }; +struct mlx5_fw_crdump; + struct mlx5_core_health { struct health_buffer __iomem *health; __be32 __iomem *health_counter; @@ -438,6 +441,7 @@ struct mlx5_core_health { unsigned long flags; struct work_struct work; struct delayed_work recover_work; + struct mlx5_fw_crdump *crdump; }; struct mlx5_qp_table { From patchwork Thu Jan 10 10:29:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saeed Mahameed X-Patchwork-Id: 1022843 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43b2LG0c5vz9sN4 for ; Thu, 10 Jan 2019 21:29:42 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728170AbfAJK3l (ORCPT ); Thu, 10 Jan 2019 05:29:41 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:40968 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726255AbfAJK3k (ORCPT ); Thu, 10 Jan 2019 05:29:40 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from saeedm@mellanox.com) with ESMTPS (AES256-SHA encrypted); 10 Jan 2019 12:29:34 +0200 Received: from sx1.mtl.com ([172.16.5.7]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x0AATDGI031830; Thu, 10 Jan 2019 12:29:31 +0200 From: Saeed Mahameed To: "David S. Miller" Cc: netdev@vger.kernel.org, Alex Vesker , Feras Daoud , Saeed Mahameed Subject: [net-next 4/9] net/mlx5: Use devlink region_snapshot parameter Date: Thu, 10 Jan 2019 12:29:00 +0200 Message-Id: <20190110102906.3751-5-saeedm@mellanox.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190110102906.3751-1-saeedm@mellanox.com> References: <20190110102906.3751-1-saeedm@mellanox.com> MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Alex Vesker This parameter enables capturing region snapshot of the crspace during critical errors. The default value of this parameter is disabled, it can be enabled using devlink param commands. It is possible to configure during runtime and also driver init. Signed-off-by: Alex Vesker Reviewed-by: Feras Daoud Reviewed-by: Alex Vesker Signed-off-by: Saeed Mahameed --- .../net/ethernet/mellanox/mlx5/core/Makefile | 2 +- .../net/ethernet/mellanox/mlx5/core/devlink.c | 60 ++++++++++++++++++- .../ethernet/mellanox/mlx5/core/diag/crdump.c | 22 +++++++ .../ethernet/mellanox/mlx5/core/lib/mlx5.h | 2 + 4 files changed, 84 insertions(+), 2 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Makefile b/drivers/net/ethernet/mellanox/mlx5/core/Makefile index f5415e229614..767c411a040c 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/Makefile +++ b/drivers/net/ethernet/mellanox/mlx5/core/Makefile @@ -14,7 +14,7 @@ obj-$(CONFIG_MLX5_CORE) += mlx5_core.o mlx5_core-y := main.o cmd.o debugfs.o fw.o eq.o uar.o pagealloc.o \ health.o mcg.o cq.o alloc.o qp.o port.o mr.o pd.o \ mad.o transobj.o vport.o sriov.o fs_cmd.o fs_core.o \ - fs_counters.o rl.o lag.o dev.o events.o wq.o lib/gid.o \ + fs_counters.o rl.o lag.o dev.o events.o wq.o devlink.o lib/gid.o \ lib/devcom.o lib/pci_vsc.o diag/fs_tracepoint.o diag/fw_tracer.o \ diag/crdump.o diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c index ddfaf06c910c..88190acae873 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c @@ -2,13 +2,71 @@ /* Copyright (c) 2018 Mellanox Technologies */ #include +#include +#include "lib/mlx5.h" + +static int mlx5_devlink_get_crdump_snapshot(struct devlink *devlink, u32 id, + struct devlink_param_gset_ctx *ctx) +{ + struct mlx5_core_dev *dev = devlink_priv(devlink); + + ctx->val.vbool = mlx5_crdump_is_snapshot_enabled(dev); + return 0; +} + +static int mlx5_devlink_set_crdump_snapshot(struct devlink *devlink, u32 id, + struct devlink_param_gset_ctx *ctx) +{ + struct mlx5_core_dev *dev = devlink_priv(devlink); + + return mlx5_crdump_set_snapshot_enabled(dev, ctx->val.vbool); +} + +static const struct devlink_param mlx5_devlink_params[] = { + DEVLINK_PARAM_GENERIC(REGION_SNAPSHOT, + BIT(DEVLINK_PARAM_CMODE_RUNTIME) | + BIT(DEVLINK_PARAM_CMODE_DRIVERINIT), + mlx5_devlink_get_crdump_snapshot, + mlx5_devlink_set_crdump_snapshot, NULL), +}; int mlx5_devlink_register(struct devlink *devlink, struct device *dev) { - return devlink_register(devlink, dev); + union devlink_param_value init_val; + int err; + + err = devlink_register(devlink, dev); + if (err) { + dev_warn(dev, + "devlink register failed (err = %d)", err); + return err; + } + + err = devlink_params_register(devlink, mlx5_devlink_params, + ARRAY_SIZE(mlx5_devlink_params)); + if (err) { + dev_err(dev, "devlink_params_register failed, err = %d\n", err); + goto unregister; + } + + init_val.vbool = false; + err = devlink_param_driverinit_value_set(devlink, + DEVLINK_PARAM_GENERIC_ID_REGION_SNAPSHOT, + init_val); + if (err) + dev_warn(dev, + "devlink param init failed (err = %d)", err); + + return 0; + +unregister: + devlink_unregister(devlink); + return err; } void mlx5_devlink_unregister(struct devlink *devlink) { + devlink_params_unregister(devlink, mlx5_devlink_params, + ARRAY_SIZE(mlx5_devlink_params)); devlink_unregister(devlink); } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/diag/crdump.c b/drivers/net/ethernet/mellanox/mlx5/core/diag/crdump.c index 78c92d235641..9a87db775c36 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/diag/crdump.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/diag/crdump.c @@ -15,6 +15,7 @@ static const char *region_cr_space_str = "cr-space"; struct mlx5_fw_crdump { u32 size; + bool snapshot_enable; struct devlink_region *region_crspace; }; @@ -97,6 +98,27 @@ int mlx5_crdump_collect(struct mlx5_core_dev *dev) return ret; } +bool mlx5_crdump_is_snapshot_enabled(struct mlx5_core_dev *dev) +{ + struct mlx5_priv *priv = &dev->priv; + + if (mlx5_crdump_enbaled(dev)) + return priv->health.crdump->snapshot_enable; + + return false; +} + +int mlx5_crdump_set_snapshot_enabled(struct mlx5_core_dev *dev, bool value) +{ + struct mlx5_priv *priv = &dev->priv; + + if (!mlx5_crdump_enbaled(dev)) + return -ENODEV; + + priv->health.crdump->snapshot_enable = value; + return 0; +} + int mlx5_crdump_init(struct mlx5_core_dev *dev) { struct devlink *devlink = priv_to_devlink(dev); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/mlx5.h b/drivers/net/ethernet/mellanox/mlx5/core/lib/mlx5.h index 29f4ed7e5034..98ef3341bdf1 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/lib/mlx5.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/mlx5.h @@ -43,6 +43,8 @@ int mlx5_core_reserved_gid_alloc(struct mlx5_core_dev *dev, int *gid_index); void mlx5_core_reserved_gid_free(struct mlx5_core_dev *dev, int gid_index); int mlx5_crdump_init(struct mlx5_core_dev *dev); void mlx5_crdump_cleanup(struct mlx5_core_dev *dev); +bool mlx5_crdump_is_snapshot_enabled(struct mlx5_core_dev *dev); +int mlx5_crdump_set_snapshot_enabled(struct mlx5_core_dev *dev, bool value); /* TODO move to lib/events.h */ From patchwork Thu Jan 10 10:29:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saeed Mahameed X-Patchwork-Id: 1022844 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43b2LL2S1Rz9sMQ for ; Thu, 10 Jan 2019 21:29:46 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728175AbfAJK3p (ORCPT ); Thu, 10 Jan 2019 05:29:45 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:40986 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726255AbfAJK3p (ORCPT ); Thu, 10 Jan 2019 05:29:45 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from saeedm@mellanox.com) with ESMTPS (AES256-SHA encrypted); 10 Jan 2019 12:29:38 +0200 Received: from sx1.mtl.com ([172.16.5.7]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x0AATDGJ031830; Thu, 10 Jan 2019 12:29:35 +0200 From: Saeed Mahameed To: "David S. Miller" Cc: netdev@vger.kernel.org, Feras Daoud , Saeed Mahameed Subject: [net-next 5/9] Documentation: mlx5: Update kernel documentation Date: Thu, 10 Jan 2019 12:29:01 +0200 Message-Id: <20190110102906.3751-6-saeedm@mellanox.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190110102906.3751-1-saeedm@mellanox.com> References: <20190110102906.3751-1-saeedm@mellanox.com> MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Feras Daoud Initial kernel documentation for mlx5 core driver. This change includes also documentation for the new devlink region param : DEVLINK_PARAM_GENERIC_ID_REGION_SNAPSHOT. Signed-off-by: Feras Daoud Signed-off-by: Saeed Mahameed --- Documentation/networking/mlx5.rst | 39 +++++++++++++++++++++++++++++++ MAINTAINERS | 1 + 2 files changed, 40 insertions(+) create mode 100644 Documentation/networking/mlx5.rst diff --git a/Documentation/networking/mlx5.rst b/Documentation/networking/mlx5.rst new file mode 100644 index 000000000000..4c44996d2132 --- /dev/null +++ b/Documentation/networking/mlx5.rst @@ -0,0 +1,39 @@ +.. SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB + +Linux* Base Driver for Mellanox Core VPI Network Connection +=========================================================== + +Copyright (c) 2018, Mellanox Technologies inc. + +Contents +======== + +- Command Line Parameters + +Command Line Parameters +======================= + +Devlink tool +------------ +The driver utilizes the devlink device configuration tool for setting driver +configuration, as well as displaying device attributes. + +:Devlink supported parameters: + +- DEVLINK_PARAM_GENERIC_ID_REGION_SNAPSHOT: + + - This parameter enables capturing region snapshot of the crspace during critical errors. + - The default value of this parameter is disabled. + + - Example: + devlink region show + List available address regions and snapshot. + + devlink region del pci/0000:00:05.0/cr-space snapshot 1 + Delete snapshot id 1 from cr-space address region from device pci/0000:00:05.0. + + devlink region dump pci/0000:00:05.0/cr-space snapshot 1 + Dump the snapshot taken from cr-space address region with ID 1 + + devlink region read pci/0000:00:05.0/cr-space snapshot 1 address 0x10 legth 16 + Read from address 0x10, 16 Bytes of snapshot ID 1 taken from cr-space address region diff --git a/MAINTAINERS b/MAINTAINERS index 7a9804a891fd..78b2cc7b26ab 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -9676,6 +9676,7 @@ Q: http://patchwork.ozlabs.org/project/netdev/list/ S: Supported F: drivers/net/ethernet/mellanox/mlx5/core/ F: include/linux/mlx5/ +F: Documentation/networking/mlx5.rst MELLANOX MLX5 IB driver M: Leon Romanovsky From patchwork Thu Jan 10 10:29:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saeed Mahameed X-Patchwork-Id: 1022845 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43b2LR56wtz9sMQ for ; Thu, 10 Jan 2019 21:29:51 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728193AbfAJK3u (ORCPT ); Thu, 10 Jan 2019 05:29:50 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:41005 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726255AbfAJK3u (ORCPT ); Thu, 10 Jan 2019 05:29:50 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from saeedm@mellanox.com) with ESMTPS (AES256-SHA encrypted); 10 Jan 2019 12:29:43 +0200 Received: from sx1.mtl.com ([172.16.5.7]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x0AATDGK031830; Thu, 10 Jan 2019 12:29:39 +0200 From: Saeed Mahameed To: "David S. Miller" Cc: netdev@vger.kernel.org, Feras Daoud , Alex Vesker , Daniel Jurgens , Saeed Mahameed Subject: [net-next 6/9] net/mlx5: Handle SW reset of FW in error flow Date: Thu, 10 Jan 2019 12:29:02 +0200 Message-Id: <20190110102906.3751-7-saeedm@mellanox.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190110102906.3751-1-saeedm@mellanox.com> References: <20190110102906.3751-1-saeedm@mellanox.com> MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Feras Daoud New mlx5 adapters allow the driver to reset the FW in the event of an error, this action called "SW Reset". When an SW reset is issued on any PF all PFs enter reset state which is a recoverable condition. The existing recovery flow was designed to allow the recovery of a VF after a PF driver reload. This patch adds the sw reset to the NIC states as a preparation for sw reset handling. When a software reset is issued the following occurs: 1. The NIC interface mode is set to 7 while the reset is in progress. 2. Once the reset completes the NIC interface mode is set to 1. Signed-off-by: Alex Vesker Signed-off-by: Daniel Jurgens Reviewed-by: Feras Daoud Reviewed-by: Alex Vesker Signed-off-by: Saeed Mahameed --- .../ethernet/mellanox/mlx5/core/en_selftest.c | 2 +- .../net/ethernet/mellanox/mlx5/core/health.c | 57 +++++++++++++------ .../ethernet/mellanox/mlx5/core/mlx5_core.h | 2 +- include/linux/mlx5/driver.h | 2 +- 4 files changed, 44 insertions(+), 19 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_selftest.c b/drivers/net/ethernet/mellanox/mlx5/core/en_selftest.c index 4382ef85488c..840ec945ccba 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_selftest.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_selftest.c @@ -64,7 +64,7 @@ static int mlx5e_test_health_info(struct mlx5e_priv *priv) { struct mlx5_core_health *health = &priv->mdev->priv.health; - return health->sick ? 1 : 0; + return health->fatal_error ? 1 : 0; } static int mlx5e_test_link_state(struct mlx5e_priv *priv) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/health.c b/drivers/net/ethernet/mellanox/mlx5/core/health.c index 883c1e8ffdc2..afa3fe6eef8f 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/health.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/health.c @@ -65,9 +65,16 @@ enum { MLX5_DROP_NEW_RECOVERY_WORK, }; +enum { + MLX5_SENSOR_NO_ERR = 0, + MLX5_SENSOR_PCI_COMM_ERR = 1, + MLX5_SENSOR_NIC_DISABLED = 2, + MLX5_SENSOR_NIC_SW_RESET = 3, +}; + u8 mlx5_get_nic_state(struct mlx5_core_dev *dev) { - return (ioread32be(&dev->iseg->cmdq_addr_l_sz) >> 8) & 3; + return (ioread32be(&dev->iseg->cmdq_addr_l_sz) >> 8) & 7; } void mlx5_set_nic_state(struct mlx5_core_dev *dev, u8 state) @@ -80,18 +87,25 @@ void mlx5_set_nic_state(struct mlx5_core_dev *dev, u8 state) &dev->iseg->cmdq_addr_l_sz); } -static int in_fatal(struct mlx5_core_dev *dev) +static bool sensor_pci_not_working(struct mlx5_core_dev *dev) { struct mlx5_core_health *health = &dev->priv.health; struct health_buffer __iomem *h = health->health; - if (mlx5_get_nic_state(dev) == MLX5_NIC_IFC_DISABLED) - return 1; + /* Offline PCI reads return 0xffffffff */ + return (ioread32be(&h->fw_ver) == 0xffffffff); +} - if (ioread32be(&h->fw_ver) == 0xffffffff) - return 1; +static u32 check_fatal_sensors(struct mlx5_core_dev *dev) +{ + if (sensor_pci_not_working(dev)) + return MLX5_SENSOR_PCI_COMM_ERR; + if (mlx5_get_nic_state(dev) == MLX5_NIC_IFC_DISABLED) + return MLX5_SENSOR_NIC_DISABLED; + if (mlx5_get_nic_state(dev) == MLX5_NIC_IFC_SW_RESET) + return MLX5_SENSOR_NIC_SW_RESET; - return 0; + return MLX5_SENSOR_NO_ERR; } void mlx5_enter_error_state(struct mlx5_core_dev *dev, bool force) @@ -101,7 +115,8 @@ void mlx5_enter_error_state(struct mlx5_core_dev *dev, bool force) goto unlock; mlx5_core_err(dev, "start\n"); - if (pci_channel_offline(dev->pdev) || in_fatal(dev) || force) { + if (pci_channel_offline(dev->pdev) || + dev->priv.health.fatal_error != MLX5_SENSOR_NO_ERR || force) { dev->state = MLX5_DEVICE_STATE_INTERNAL_ERROR; mlx5_cmd_trigger_completions(dev); } @@ -143,16 +158,14 @@ static void health_recover(struct work_struct *work) struct delayed_work *dwork; struct mlx5_core_dev *dev; struct mlx5_priv *priv; - u8 nic_state; dwork = container_of(work, struct delayed_work, work); health = container_of(dwork, struct mlx5_core_health, recover_work); priv = container_of(health, struct mlx5_priv, health); dev = container_of(priv, struct mlx5_core_dev, priv); - nic_state = mlx5_get_nic_state(dev); - if (nic_state == MLX5_NIC_IFC_INVALID) { - dev_err(&dev->pdev->dev, "health recovery flow aborted since the nic state is invalid\n"); + if (sensor_pci_not_working(dev)) { + dev_err(&dev->pdev->dev, "health recovery flow aborted, PCI reads still not working\n"); return; } @@ -162,10 +175,17 @@ static void health_recover(struct work_struct *work) /* How much time to wait until health resetting the driver (in msecs) */ #define MLX5_RECOVERY_DELAY_MSECS 60000 +#define MLX5_RECOVERY_NO_DELAY 0 +static unsigned long get_recovery_delay(struct mlx5_core_dev *dev) +{ + return dev->priv.health.fatal_error == MLX5_SENSOR_PCI_COMM_ERR ? + MLX5_RECOVERY_DELAY_MSECS : MLX5_RECOVERY_NO_DELAY; +} + static void health_care(struct work_struct *work) { - unsigned long recover_delay = msecs_to_jiffies(MLX5_RECOVERY_DELAY_MSECS); struct mlx5_core_health *health; + unsigned long recover_delay; struct mlx5_core_dev *dev; struct mlx5_priv *priv; unsigned long flags; @@ -175,6 +195,7 @@ static void health_care(struct work_struct *work) dev = container_of(priv, struct mlx5_core_dev, priv); mlx5_core_warn(dev, "handling bad device here\n"); mlx5_handle_bad_state(dev); + recover_delay = msecs_to_jiffies(get_recovery_delay(dev)); spin_lock_irqsave(&health->wq_lock, flags); if (!test_bit(MLX5_DROP_NEW_RECOVERY_WORK, &health->flags)) @@ -271,6 +292,7 @@ static void poll_health(struct timer_list *t) { struct mlx5_core_dev *dev = from_timer(dev, t, priv.health.timer); struct mlx5_core_health *health = &dev->priv.health; + u32 fatal_error; u32 count; if (dev->state == MLX5_DEVICE_STATE_INTERNAL_ERROR) @@ -288,8 +310,11 @@ static void poll_health(struct timer_list *t) print_health_info(dev); } - if (in_fatal(dev) && !health->sick) { - health->sick = true; + fatal_error = check_fatal_sensors(dev); + + if (fatal_error && !health->fatal_error) { + mlx5_core_err(dev, "Fatal error %u detected\n", fatal_error); + dev->priv.health.fatal_error = fatal_error; print_health_info(dev); mlx5_trigger_health_work(dev); } @@ -303,7 +328,7 @@ void mlx5_start_health_poll(struct mlx5_core_dev *dev) struct mlx5_core_health *health = &dev->priv.health; timer_setup(&health->timer, poll_health, 0); - health->sick = 0; + health->fatal_error = MLX5_SENSOR_NO_ERR; clear_bit(MLX5_DROP_NEW_HEALTH_WORK, &health->flags); clear_bit(MLX5_DROP_NEW_RECOVERY_WORK, &health->flags); health->health = &dev->iseg->health; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h index c68dcea5985b..29a2a4546fb6 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h @@ -194,7 +194,7 @@ enum { MLX5_NIC_IFC_FULL = 0, MLX5_NIC_IFC_DISABLED = 1, MLX5_NIC_IFC_NO_DRAM_NIC = 2, - MLX5_NIC_IFC_INVALID = 3 + MLX5_NIC_IFC_SW_RESET = 7 }; u8 mlx5_get_nic_state(struct mlx5_core_dev *dev); diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h index 2f41603dc728..71cb52bae1ae 100644 --- a/include/linux/mlx5/driver.h +++ b/include/linux/mlx5/driver.h @@ -434,7 +434,7 @@ struct mlx5_core_health { struct timer_list timer; u32 prev; int miss_counter; - bool sick; + u32 fatal_error; /* wq spinlock to synchronize draining */ spinlock_t wq_lock; struct workqueue_struct *wq; From patchwork Thu Jan 10 10:29:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saeed Mahameed X-Patchwork-Id: 1022846 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43b2LT1CWyz9sMQ for ; Thu, 10 Jan 2019 21:29:53 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728184AbfAJK3u (ORCPT ); Thu, 10 Jan 2019 05:29:50 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:41006 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728176AbfAJK3u (ORCPT ); Thu, 10 Jan 2019 05:29:50 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from saeedm@mellanox.com) with ESMTPS (AES256-SHA encrypted); 10 Jan 2019 12:29:47 +0200 Received: from sx1.mtl.com ([172.16.5.7]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x0AATDGL031830; Thu, 10 Jan 2019 12:29:44 +0200 From: Saeed Mahameed To: "David S. Miller" Cc: netdev@vger.kernel.org, Feras Daoud , Saeed Mahameed Subject: [net-next 7/9] net/mlx5: Control CR-space access by different PFs Date: Thu, 10 Jan 2019 12:29:03 +0200 Message-Id: <20190110102906.3751-8-saeedm@mellanox.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190110102906.3751-1-saeedm@mellanox.com> References: <20190110102906.3751-1-saeedm@mellanox.com> MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Feras Daoud Since the FW can be shared between different PFs/VFs it is common that more than one health poll will detected a failure, this can lead to multiple resets which are unneeded. The solution is to use a FW locking mechanism using semaphore space to provide a way to allow only one device to collect the cr-dump and to issue a sw-reset. Signed-off-by: Feras Daoud Signed-off-by: Saeed Mahameed --- .../ethernet/mellanox/mlx5/core/lib/pci_vsc.c | 40 ++++++++++++++++--- .../ethernet/mellanox/mlx5/core/lib/pci_vsc.h | 8 ++++ .../ethernet/mellanox/mlx5/core/mlx5_core.h | 4 ++ 3 files changed, 47 insertions(+), 5 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/pci_vsc.c b/drivers/net/ethernet/mellanox/mlx5/core/lib/pci_vsc.c index b26e14c4456b..2bdbe89a77f0 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/lib/pci_vsc.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/pci_vsc.c @@ -24,11 +24,6 @@ pci_write_config_dword((dev)->pdev, (dev)->vsc_addr + (offset), (val)) #define VSC_MAX_RETRIES 2048 -enum mlx5_vsc_state { - MLX5_VSC_UNLOCK, - MLX5_VSC_LOCK, -}; - enum { VSC_CTRL_OFFSET = 0x4, VSC_COUNTER_OFFSET = 0x8, @@ -280,3 +275,38 @@ int mlx5_vsc_gw_read_block_fast(struct mlx5_core_dev *dev, u32 *data, } return length; } + +int mlx5_vsc_sem_set_space(struct mlx5_core_dev *dev, u16 space, + enum mlx5_vsc_state state) +{ + u32 data, id = 0; + int ret; + + ret = mlx5_vsc_gw_set_space(dev, MLX5_SEMAPHORE_SPACE_DOMAIN, NULL); + if (ret) { + mlx5_core_warn(dev, "Failed to set gw space %d\n", ret); + return ret; + } + + if (state == MLX5_VSC_LOCK) { + /* Get a unique ID based on the counter */ + ret = vsc_read(dev, VSC_COUNTER_OFFSET, &id); + if (ret) + return ret; + } + + /* Try to modify lock */ + ret = mlx5_vsc_gw_write(dev, space, id); + if (ret) + return ret; + + /* Verify lock was modified */ + ret = mlx5_vsc_gw_read(dev, space, &data); + if (ret) + return -EINVAL; + + if (data != id) + return -EBUSY; + + return 0; +} diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/pci_vsc.h b/drivers/net/ethernet/mellanox/mlx5/core/lib/pci_vsc.h index c30ff69ffa3e..1f96db9b982b 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/lib/pci_vsc.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/pci_vsc.h @@ -4,6 +4,11 @@ #ifndef __MLX5_PCI_VSC_H__ #define __MLX5_PCI_VSC_H__ +enum mlx5_vsc_state { + MLX5_VSC_UNLOCK, + MLX5_VSC_LOCK, +}; + enum { MLX5_VSC_SPACE_SCAN_CRSPACE = 0x7, }; @@ -22,4 +27,7 @@ static inline bool mlx5_vsc_accessible(struct mlx5_core_dev *dev) return !!dev->vsc_addr; } +int mlx5_vsc_sem_set_space(struct mlx5_core_dev *dev, u16 space, + enum mlx5_vsc_state state); + #endif /* __MLX5_PCI_VSC_H__ */ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h index 29a2a4546fb6..e5350c885ab7 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h @@ -97,6 +97,10 @@ enum { MLX5_DRIVER_SYND = 0xbadd00de, }; +enum mlx5_semaphore_space_address { + MLX5_SEMAPHORE_SPACE_DOMAIN = 0xA, +}; + int mlx5_query_hca_caps(struct mlx5_core_dev *dev); int mlx5_query_board_id(struct mlx5_core_dev *dev); int mlx5_cmd_init_hca(struct mlx5_core_dev *dev, uint32_t *sw_owner_id); From patchwork Thu Jan 10 10:29:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saeed Mahameed X-Patchwork-Id: 1022848 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43b2Lg37G5z9sMQ for ; Thu, 10 Jan 2019 21:30:03 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728223AbfAJKaC (ORCPT ); Thu, 10 Jan 2019 05:30:02 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:41034 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1728194AbfAJKaA (ORCPT ); Thu, 10 Jan 2019 05:30:00 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from saeedm@mellanox.com) with ESMTPS (AES256-SHA encrypted); 10 Jan 2019 12:29:55 +0200 Received: from sx1.mtl.com ([172.16.5.7]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x0AATDGM031830; Thu, 10 Jan 2019 12:29:48 +0200 From: Saeed Mahameed To: "David S. Miller" Cc: netdev@vger.kernel.org, Feras Daoud , Alex Vesker , Daniel Jurgens , Saeed Mahameed Subject: [net-next 8/9] net/mlx5: Issue SW reset on FW assert Date: Thu, 10 Jan 2019 12:29:04 +0200 Message-Id: <20190110102906.3751-9-saeedm@mellanox.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190110102906.3751-1-saeedm@mellanox.com> References: <20190110102906.3751-1-saeedm@mellanox.com> MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Feras Daoud If a FW assert is considered fatal, indicated by a new bit in the health buffer, reset the FW. After the reset go through the normal recovery flow. Only one PF needs to issue the reset, so an attempt is made to prevent the 2nd function from also issuing the reset. It's not an error if that happens, it just slows recovery. Signed-off-by: Alex Vesker Signed-off-by: Daniel Jurgens Reviewed-by: Feras Daoud Signed-off-by: Saeed Mahameed --- .../net/ethernet/mellanox/mlx5/core/health.c | 174 +++++++++++++++++- .../ethernet/mellanox/mlx5/core/lib/mlx5.h | 1 + .../ethernet/mellanox/mlx5/core/mlx5_core.h | 1 + include/linux/mlx5/device.h | 10 +- include/linux/mlx5/driver.h | 1 + 5 files changed, 177 insertions(+), 10 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/health.c b/drivers/net/ethernet/mellanox/mlx5/core/health.c index afa3fe6eef8f..74de30246eee 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/health.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/health.c @@ -40,6 +40,7 @@ #include "mlx5_core.h" #include "lib/eq.h" #include "lib/mlx5.h" +#include "lib/pci_vsc.h" enum { MLX5_HEALTH_POLL_INTERVAL = 2 * HZ, @@ -68,8 +69,10 @@ enum { enum { MLX5_SENSOR_NO_ERR = 0, MLX5_SENSOR_PCI_COMM_ERR = 1, - MLX5_SENSOR_NIC_DISABLED = 2, - MLX5_SENSOR_NIC_SW_RESET = 3, + MLX5_SENSOR_PCI_ERR = 2, + MLX5_SENSOR_NIC_DISABLED = 3, + MLX5_SENSOR_NIC_SW_RESET = 4, + MLX5_SENSOR_FW_SYND_RFR = 5, }; u8 mlx5_get_nic_state(struct mlx5_core_dev *dev) @@ -96,34 +99,169 @@ static bool sensor_pci_not_working(struct mlx5_core_dev *dev) return (ioread32be(&h->fw_ver) == 0xffffffff); } +static bool sensor_fw_synd_rfr(struct mlx5_core_dev *dev) +{ + struct mlx5_core_health *health = &dev->priv.health; + struct health_buffer __iomem *h = health->health; + u32 rfr = ioread32be(&h->rfr) >> MLX5_RFR_OFFSET; + u8 synd = ioread8(&h->synd); + + if (rfr && synd) + mlx5_core_dbg(dev, "FW requests reset, synd: %d\n", synd); + return rfr && synd; +} + static u32 check_fatal_sensors(struct mlx5_core_dev *dev) { if (sensor_pci_not_working(dev)) return MLX5_SENSOR_PCI_COMM_ERR; + if (pci_channel_offline(dev->pdev)) + return MLX5_SENSOR_PCI_ERR; if (mlx5_get_nic_state(dev) == MLX5_NIC_IFC_DISABLED) return MLX5_SENSOR_NIC_DISABLED; if (mlx5_get_nic_state(dev) == MLX5_NIC_IFC_SW_RESET) return MLX5_SENSOR_NIC_SW_RESET; + if (sensor_fw_synd_rfr(dev)) + return MLX5_SENSOR_FW_SYND_RFR; return MLX5_SENSOR_NO_ERR; } +static int lock_sem_sw_reset(struct mlx5_core_dev *dev, bool lock) +{ + int ret; + + if (!mlx5_core_is_pf(dev)) + return -EBUSY; + + /* Try to lock GW access, this stage doesn't return + * EBUSY because locked GW does not mean that other PF + * already started the reset. + */ + ret = mlx5_vsc_gw_lock(dev); + if (ret == -EBUSY) + return -EINVAL; + if (ret) + return ret; + + /* At this stage, if the return status == EBUSY, then we know + * for sure that another PF started the reset, so don't allow + * another reset. + */ + ret = mlx5_vsc_sem_set_space(dev, MLX5_SEMAPHORE_SW_RESET, lock); + if (ret) + mlx5_core_warn(dev, "Failed to lock SW reset semaphore\n"); + + /* Unlock GW access */ + mlx5_vsc_gw_unlock(dev); + + return ret; +} + +static bool reset_fw_if_needed(struct mlx5_core_dev *dev) +{ + bool supported = (ioread32be(&dev->iseg->initializing) >> + MLX5_FW_RESET_SUPPORTED_OFFSET) & 1; + u32 fatal_error; + + if (!supported) + return false; + + /* The reset only needs to be issued by one PF. The health buffer is + * shared between all functions, and will be cleared during a reset. + * Check again to avoid a redundant 2nd reset. If the fatal erros was + * PCI related a reset won't help. + */ + fatal_error = check_fatal_sensors(dev); + if (fatal_error == MLX5_SENSOR_PCI_COMM_ERR || + fatal_error == MLX5_SENSOR_NIC_DISABLED || + fatal_error == MLX5_SENSOR_NIC_SW_RESET) { + mlx5_core_warn(dev, "Not issuing FW reset. Either it's already done or won't help."); + return false; + } + + mlx5_core_warn(dev, "Issuing FW Reset\n"); + /* Write the NIC interface field to initiate the reset, the command + * interface address also resides here, don't overwrite it. + */ + mlx5_set_nic_state(dev, MLX5_NIC_IFC_SW_RESET); + + return true; +} + +#define MLX5_CRDUMP_WAIT_MS 60000 +#define MLX5_FW_RESET_WAIT_MS 1000 void mlx5_enter_error_state(struct mlx5_core_dev *dev, bool force) { + unsigned long end, delay_ms = MLX5_FW_RESET_WAIT_MS; + u32 fatal_error, err; + int lock = -EBUSY; + mutex_lock(&dev->intf_state_mutex); if (dev->state == MLX5_DEVICE_STATE_INTERNAL_ERROR) goto unlock; + if (dev->state == MLX5_DEVICE_STATE_UNINITIALIZED) { + dev->state = MLX5_DEVICE_STATE_INTERNAL_ERROR; + goto unlock; + } + + if (force) + mlx5_core_dbg(dev, "start\n"); + else + mlx5_core_err(dev, "start\n"); - mlx5_core_err(dev, "start\n"); - if (pci_channel_offline(dev->pdev) || - dev->priv.health.fatal_error != MLX5_SENSOR_NO_ERR || force) { + fatal_error = check_fatal_sensors(dev); + + if (fatal_error || force) { dev->state = MLX5_DEVICE_STATE_INTERNAL_ERROR; mlx5_cmd_trigger_completions(dev); } mlx5_notifier_call_chain(dev->priv.events, MLX5_DEV_EVENT_SYS_ERROR, (void *)1); - mlx5_core_err(dev, "end\n"); + if (force) + goto err_state_done; + + if (fatal_error == MLX5_SENSOR_FW_SYND_RFR) { + /* Get cr-dump and reset FW semaphore */ + lock = lock_sem_sw_reset(dev, true); + + /* Execute cr-dump and SW reset */ + if (lock == -EBUSY) { + delay_ms = MLX5_CRDUMP_WAIT_MS; + goto recover_from_sw_reset; + } + err = mlx5_crdump_collect(dev); + if (err) + mlx5_core_err(dev, "Failed to collect crdump area err %d\n", + err); + reset_fw_if_needed(dev); + } + +recover_from_sw_reset: + /* Recover from SW reset */ + end = jiffies + msecs_to_jiffies(delay_ms); + do { + if (mlx5_get_nic_state(dev) == MLX5_NIC_IFC_DISABLED) + break; + + cond_resched(); + } while (!time_after(jiffies, end)); + + if (mlx5_get_nic_state(dev) != MLX5_NIC_IFC_DISABLED) { + dev_err(&dev->pdev->dev, "NIC IFC still %d after %lums.\n", + mlx5_get_nic_state(dev), delay_ms); + } + + /* Release FW semaphore if you are the lock owner */ + if (!lock) + lock_sem_sw_reset(dev, false); + +err_state_done: + if (force) + mlx5_core_dbg(dev, "end\n"); + else + mlx5_core_err(dev, "end\n"); unlock: mutex_unlock(&dev->intf_state_mutex); } @@ -144,6 +282,20 @@ static void mlx5_handle_bad_state(struct mlx5_core_dev *dev) case MLX5_NIC_IFC_NO_DRAM_NIC: mlx5_core_warn(dev, "Expected to see disabled NIC but it is no dram nic\n"); break; + + case MLX5_NIC_IFC_SW_RESET: + /* The IFC mode field is 3 bits, so it will read 0x7 in 2 cases: + * 1. PCI has been disabled (ie. PCI-AER, PF driver unloaded + * and this is a VF), this is not recoverable by SW reset. + * Logging of this is handled elsewhere. + * 2. FW reset has been issued by another function, driver can + * be reloaded to recover after the mode switches to + * MLX5_NIC_IFC_DISABLED. + */ + if (dev->priv.health.fatal_error != MLX5_SENSOR_PCI_COMM_ERR) + mlx5_core_warn(dev, "NIC SW reset in progress\n"); + break; + default: mlx5_core_warn(dev, "Expected to see disabled NIC but it is has invalid value %d\n", nic_interface); @@ -178,7 +330,8 @@ static void health_recover(struct work_struct *work) #define MLX5_RECOVERY_NO_DELAY 0 static unsigned long get_recovery_delay(struct mlx5_core_dev *dev) { - return dev->priv.health.fatal_error == MLX5_SENSOR_PCI_COMM_ERR ? + return dev->priv.health.fatal_error == MLX5_SENSOR_PCI_ERR || + MLX5_SENSOR_PCI_COMM_ERR ? MLX5_RECOVERY_DELAY_MSECS : MLX5_RECOVERY_NO_DELAY; } @@ -198,11 +351,14 @@ static void health_care(struct work_struct *work) recover_delay = msecs_to_jiffies(get_recovery_delay(dev)); spin_lock_irqsave(&health->wq_lock, flags); - if (!test_bit(MLX5_DROP_NEW_RECOVERY_WORK, &health->flags)) + if (!test_bit(MLX5_DROP_NEW_RECOVERY_WORK, &health->flags)) { + mlx5_core_warn(dev, "Scheduling recovery work with %lums delay\n", + recover_delay); schedule_delayed_work(&health->recover_work, recover_delay); - else + } else { dev_err(&dev->pdev->dev, "new health works are not permitted at this stage\n"); + } spin_unlock_irqrestore(&health->wq_lock, flags); } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lib/mlx5.h b/drivers/net/ethernet/mellanox/mlx5/core/lib/mlx5.h index 98ef3341bdf1..a5d1c1be9c00 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/lib/mlx5.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/lib/mlx5.h @@ -45,6 +45,7 @@ int mlx5_crdump_init(struct mlx5_core_dev *dev); void mlx5_crdump_cleanup(struct mlx5_core_dev *dev); bool mlx5_crdump_is_snapshot_enabled(struct mlx5_core_dev *dev); int mlx5_crdump_set_snapshot_enabled(struct mlx5_core_dev *dev, bool value); +int mlx5_crdump_collect(struct mlx5_core_dev *dev); /* TODO move to lib/events.h */ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h index e5350c885ab7..bf118e85aefe 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h @@ -99,6 +99,7 @@ enum { enum mlx5_semaphore_space_address { MLX5_SEMAPHORE_SPACE_DOMAIN = 0xA, + MLX5_SEMAPHORE_SW_RESET = 0x20, }; int mlx5_query_hca_caps(struct mlx5_core_dev *dev); diff --git a/include/linux/mlx5/device.h b/include/linux/mlx5/device.h index 8c4a820bd4c1..cac801609d76 100644 --- a/include/linux/mlx5/device.h +++ b/include/linux/mlx5/device.h @@ -507,6 +507,10 @@ struct mlx5_cmd_layout { u8 status_own; }; +enum mlx5_fatal_assert_bit_offsets { + MLX5_RFR_OFFSET = 31, +}; + struct health_buffer { __be32 assert_var[5]; __be32 rsvd0[3]; @@ -515,12 +519,16 @@ struct health_buffer { __be32 rsvd1[2]; __be32 fw_ver; __be32 hw_id; - __be32 rsvd2; + __be32 rfr; u8 irisc_index; u8 synd; __be16 ext_synd; }; +enum mlx5_initializing_bit_offsets { + MLX5_FW_RESET_SUPPORTED_OFFSET = 30, +}; + enum mlx5_cmd_addr_l_sz_offset { MLX5_NIC_IFC_OFFSET = 8, }; diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h index 71cb52bae1ae..2ea6732c1d4d 100644 --- a/include/linux/mlx5/driver.h +++ b/include/linux/mlx5/driver.h @@ -575,6 +575,7 @@ struct mlx5_priv { }; enum mlx5_device_state { + MLX5_DEVICE_STATE_UNINITIALIZED, MLX5_DEVICE_STATE_UP, MLX5_DEVICE_STATE_INTERNAL_ERROR, }; From patchwork Thu Jan 10 10:29:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Saeed Mahameed X-Patchwork-Id: 1022847 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43b2Ld3mdxz9sMQ for ; Thu, 10 Jan 2019 21:30:01 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728208AbfAJKaA (ORCPT ); Thu, 10 Jan 2019 05:30:00 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:41035 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726255AbfAJKaA (ORCPT ); Thu, 10 Jan 2019 05:30:00 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from saeedm@mellanox.com) with ESMTPS (AES256-SHA encrypted); 10 Jan 2019 12:29:56 +0200 Received: from sx1.mtl.com ([172.16.5.7]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id x0AATDGN031830; Thu, 10 Jan 2019 12:29:53 +0200 From: Saeed Mahameed To: "David S. Miller" Cc: netdev@vger.kernel.org, Feras Daoud , Saeed Mahameed Subject: [net-next 9/9] net/mlx5: Protect against infinite recovery requests Date: Thu, 10 Jan 2019 12:29:05 +0200 Message-Id: <20190110102906.3751-10-saeedm@mellanox.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190110102906.3751-1-saeedm@mellanox.com> References: <20190110102906.3751-1-saeedm@mellanox.com> MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Feras Daoud A buggy HW may cause infinite recovery requests loop that may terminate the driver. The following change protects against that by adding a timestamp variable that will remember the last recover request timestamp, and allow recovery only if the period between two sequential requests is bigger than 20 min. Signed-off-by: Feras Daoud Signed-off-by: Saeed Mahameed --- .../net/ethernet/mellanox/mlx5/core/health.c | 24 +++++++++++++++++++ include/linux/mlx5/driver.h | 1 + 2 files changed, 25 insertions(+) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/health.c b/drivers/net/ethernet/mellanox/mlx5/core/health.c index 74de30246eee..b43070e4f519 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/health.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/health.c @@ -191,9 +191,26 @@ static bool reset_fw_if_needed(struct mlx5_core_dev *dev) #define MLX5_CRDUMP_WAIT_MS 60000 #define MLX5_FW_RESET_WAIT_MS 1000 +#define MLX5_RECOVERY_TIMEOUT_MS 1200000 + +static bool mlx5_health_allow_recover(struct mlx5_core_health *health) +{ + bool ret = false; + + ret = health->last_recover_tstamp ? + time_after(jiffies, health->last_recover_tstamp + + msecs_to_jiffies(MLX5_RECOVERY_TIMEOUT_MS)) : + true; + + health->last_recover_tstamp = jiffies; + + return ret; +} + void mlx5_enter_error_state(struct mlx5_core_dev *dev, bool force) { unsigned long end, delay_ms = MLX5_FW_RESET_WAIT_MS; + struct mlx5_core_health *health = &dev->priv.health; u32 fatal_error, err; int lock = -EBUSY; @@ -212,6 +229,12 @@ void mlx5_enter_error_state(struct mlx5_core_dev *dev, bool force) fatal_error = check_fatal_sensors(dev); + if (fatal_error == MLX5_SENSOR_FW_SYND_RFR && + !mlx5_health_allow_recover(health)) { + mlx5_core_warn_once(dev, "Device recovery ignored\n"); + goto err_state_done; + } + if (fatal_error || force) { dev->state = MLX5_DEVICE_STATE_INTERNAL_ERROR; mlx5_cmd_trigger_completions(dev); @@ -560,6 +583,7 @@ int mlx5_health_init(struct mlx5_core_dev *dev) INIT_WORK(&health->work, health_care); INIT_DELAYED_WORK(&health->recover_work, health_recover); health->crdump = NULL; + health->last_recover_tstamp = 0; return 0; } diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h index 2ea6732c1d4d..06ab2647f790 100644 --- a/include/linux/mlx5/driver.h +++ b/include/linux/mlx5/driver.h @@ -442,6 +442,7 @@ struct mlx5_core_health { struct work_struct work; struct delayed_work recover_work; struct mlx5_fw_crdump *crdump; + unsigned long last_recover_tstamp; }; struct mlx5_qp_table {