From patchwork Mon Dec 31 14:31:55 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eran Ben Elisha X-Patchwork-Id: 1019578 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43T0CS6gzyz9s55 for ; Tue, 1 Jan 2019 01:32:52 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727351AbeLaOcw (ORCPT ); Mon, 31 Dec 2018 09:32:52 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:52200 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726061AbeLaOc0 (ORCPT ); Mon, 31 Dec 2018 09:32:26 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from eranbe@mellanox.com) with ESMTPS (AES256-SHA encrypted); 31 Dec 2018 16:32:21 +0200 Received: from dev-l-vrt-198.mtl.labs.mlnx (dev-l-vrt-198.mtl.labs.mlnx [10.134.198.1]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id wBVEWLYn017529; Mon, 31 Dec 2018 16:32:21 +0200 From: Eran Ben Elisha To: netdev@vger.kernel.org, "David S. Miller" , Jiri Pirko Cc: Moshe Shemesh , Aya Levin , Eran Ben Elisha , Tal Alon , Ariel Almog Subject: [PATCH RFC net-next 01/19] devlink: Add health buffer support Date: Mon, 31 Dec 2018 16:31:55 +0200 Message-Id: <1546266733-9512-2-git-send-email-eranbe@mellanox.com> X-Mailer: git-send-email 1.8.4.3 In-Reply-To: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> References: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Devlink health buffer is a mechanism to pass descriptors between drivers and devlink. The API allows the driver to add objects, object pair, value array (nested attributes), value and name. Driver can use this API to fill the buffers in a format which can be translated by the devlink to the netlink message. In order to fulfill it, an internal buffer descriptor is defined. This will hold the data and metadata per each attribute and by used to pass actual commands to the netlink. This mechanism will be later used in devlink health for objdump and diagnose data store by the drivers. Signed-off-by: Eran Ben Elisha --- include/net/devlink.h | 76 ++++++ include/uapi/linux/devlink.h | 8 + net/core/devlink.c | 501 +++++++++++++++++++++++++++++++++++ 3 files changed, 585 insertions(+) diff --git a/include/net/devlink.h b/include/net/devlink.h index 67f4293bc970..77c77319290a 100644 --- a/include/net/devlink.h +++ b/include/net/devlink.h @@ -423,6 +423,8 @@ struct devlink_region; typedef void devlink_snapshot_data_dest_t(const void *data); +struct devlink_health_buffer; + struct devlink_ops { int (*reload)(struct devlink *devlink, struct netlink_ext_ack *extack); int (*port_type_set)(struct devlink_port *devlink_port, @@ -584,6 +586,22 @@ int devlink_region_snapshot_create(struct devlink_region *region, u64 data_len, u8 *data, u32 snapshot_id, devlink_snapshot_data_dest_t *data_destructor); +int devlink_health_buffer_nest_start(struct devlink_health_buffer *buffer, + int attrtype); +void devlink_health_buffer_nest_end(struct devlink_health_buffer *buffer); +void devlink_health_buffer_nest_cancel(struct devlink_health_buffer *buffer); +int devlink_health_buffer_put_object_name(struct devlink_health_buffer *buffer, + char *name); +int devlink_health_buffer_put_value_u8(struct devlink_health_buffer *buffer, + u8 value); +int devlink_health_buffer_put_value_u32(struct devlink_health_buffer *buffer, + u32 value); +int devlink_health_buffer_put_value_u64(struct devlink_health_buffer *buffer, + u64 value); +int devlink_health_buffer_put_value_string(struct devlink_health_buffer *buffer, + char *name); +int devlink_health_buffer_put_value_data(struct devlink_health_buffer *buffer, + void *data, int len); #else static inline struct devlink *devlink_alloc(const struct devlink_ops *ops, @@ -844,6 +862,64 @@ devlink_region_snapshot_create(struct devlink_region *region, u64 data_len, return 0; } +static inline int +devlink_health_buffer_nest_start(struct devlink_health_buffer *buffer, + int attrtype) +{ + return 0; +} + +static inline void +devlink_health_buffer_nest_end(struct devlink_health_buffer *buffer) +{ +} + +static inline void +devlink_health_buffer_nest_cancel(struct devlink_health_buffer *buffer) +{ +} + +static inline int +devlink_health_buffer_put_object_name(struct devlink_health_buffer *buffer, + char *name) +{ + return 0; +} + +static inline int +devlink_health_buffer_put_value_u8(struct devlink_health_buffer *buffer, + u8 value) +{ + return 0; +} + +static inline int +devlink_health_buffer_put_value_u32(struct devlink_health_buffer *buffer, + u32 value) +{ + return 0; +} + +static inline int +devlink_health_buffer_put_value_u64(struct devlink_health_buffer *buffer, + u64 value) +{ + return 0; +} + +static inline int +devlink_health_buffer_put_value_string(struct devlink_health_buffer *buffer, + char *name) +{ + return 0; +} + +static inline int +devlink_health_buffer_put_value_data(struct devlink_health_buffer *buffer, + void *data, int len) +{ + return 0; +} #endif #endif /* _NET_DEVLINK_H_ */ diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h index 6e52d3660654..cff0e0cb5ac2 100644 --- a/include/uapi/linux/devlink.h +++ b/include/uapi/linux/devlink.h @@ -285,6 +285,14 @@ enum devlink_attr { DEVLINK_ATTR_REGION_CHUNK_ADDR, /* u64 */ DEVLINK_ATTR_REGION_CHUNK_LEN, /* u64 */ + DEVLINK_ATTR_HEALTH_BUFFER_OBJECT, /* nested */ + DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_PAIR, /* nested */ + DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_NAME, /* string */ + DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE, /* nested */ + DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE_ARRAY, /* nested */ + DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE_TYPE, /* u8 */ + DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE_DATA, /* dynamic */ + /* add new attributes above here, update the policy in devlink.c */ __DEVLINK_ATTR_MAX, diff --git a/net/core/devlink.c b/net/core/devlink.c index abb0da9d7b4b..36018af086a0 100644 --- a/net/core/devlink.c +++ b/net/core/devlink.c @@ -3597,6 +3597,507 @@ static int devlink_nl_cmd_region_read_dumpit(struct sk_buff *skb, return 0; } +#define DEVLINK_HEALTH_BUFFER_SIZE (4096 - GENL_HDRLEN) +#define DEVLINK_HEALTH_BUFFER_DATA_SIZE (DEVLINK_HEALTH_BUFFER_SIZE / 2) +#define DEVLINK_HEALTH_SIZE_TO_BUFFERS(size) DIV_ROUND_UP(size, DEVLINK_HEALTH_BUFFER_DATA_SIZE) +#define DEVLINK_HEALTH_BUFFER_MAX_CHUNK 1024 + +struct devlink_health_buffer { + void *data; + u64 offset; + u64 bytes_left; + u64 bytes_left_metadata; + u64 max_nested_depth; + u64 curr_nest; +}; + +struct devlink_health_buffer_desc { + int attrtype; + u16 len; + u8 nla_type; + u8 nest_end; + int value[0]; +}; + +static void +devlink_health_buffers_reset(struct devlink_health_buffer **buffers_list, + u64 num_of_buffers) +{ + u64 i; + + for (i = 0; i < num_of_buffers; i++) { + memset(buffers_list[i]->data, 0, DEVLINK_HEALTH_BUFFER_SIZE); + buffers_list[i]->offset = 0; + buffers_list[i]->bytes_left = DEVLINK_HEALTH_BUFFER_DATA_SIZE; + buffers_list[i]->bytes_left_metadata = + DEVLINK_HEALTH_BUFFER_DATA_SIZE; + buffers_list[i]->max_nested_depth = 0; + buffers_list[i]->curr_nest = 0; + } +} + +static void +devlink_health_buffers_destroy(struct devlink_health_buffer **buffers_list, + u64 size); + +static struct devlink_health_buffer ** +devlink_health_buffers_create(u64 size) +{ + struct devlink_health_buffer **buffers_list; + u64 num_of_buffers = DEVLINK_HEALTH_SIZE_TO_BUFFERS(size); + u64 i; + + buffers_list = kcalloc(num_of_buffers, + sizeof(struct devlink_health_buffer *), + GFP_KERNEL); + if (!buffers_list) + return NULL; + + for (i = 0; i < num_of_buffers; i++) { + struct devlink_health_buffer *buffer; + void *data; + + buffer = kzalloc(sizeof(*buffer), GFP_KERNEL); + data = kzalloc(DEVLINK_HEALTH_BUFFER_SIZE, GFP_KERNEL); + if (!buffer || !data) { + kfree(buffer); + kfree(data); + goto buffers_cleanup; + } + buffers_list[i] = buffer; + buffer->data = data; + } + devlink_health_buffers_reset(buffers_list, num_of_buffers); + + return buffers_list; + +buffers_cleanup: + devlink_health_buffers_destroy(buffers_list, --i); + kfree(buffers_list); + return NULL; +} + +static void +devlink_health_buffers_destroy(struct devlink_health_buffer **buffers_list, + u64 num_of_buffers) +{ + u64 i; + + for (i = 0; i < num_of_buffers; i++) { + kfree(buffers_list[i]->data); + kfree(buffers_list[i]); + } +} + +void +devlink_health_buffer_offset_inc(struct devlink_health_buffer *buffer, + int len) +{ + buffer->offset += len; +} + +/* In order to store a nest, need two descriptors, for start and end */ +#define DEVLINK_HEALTH_BUFFER_NEST_SIZE (sizeof(struct devlink_health_buffer_desc) * 2) + +int devlink_health_buffer_verify_len(struct devlink_health_buffer *buffer, + int len, int metadata_len) +{ + if (len > DEVLINK_HEALTH_BUFFER_DATA_SIZE) + return -EINVAL; + + if (buffer->bytes_left < len || + buffer->bytes_left_metadata < metadata_len) + return -ENOMEM; + + return 0; +} + +static struct devlink_health_buffer_desc * +devlink_health_buffer_get_desc_from_offset(struct devlink_health_buffer *buffer) +{ + return buffer->data + buffer->offset; +} + +int +devlink_health_buffer_nest_start(struct devlink_health_buffer *buffer, + int attrtype) +{ + struct devlink_health_buffer_desc *desc; + int err; + + err = devlink_health_buffer_verify_len(buffer, 0, + DEVLINK_HEALTH_BUFFER_NEST_SIZE); + if (err) + return err; + + if (attrtype != DEVLINK_ATTR_HEALTH_BUFFER_OBJECT && + attrtype != DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_PAIR && + attrtype != DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE && + attrtype != DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE_ARRAY) + return -EINVAL; + + desc = devlink_health_buffer_get_desc_from_offset(buffer); + + desc->attrtype = attrtype; + buffer->bytes_left_metadata -= DEVLINK_HEALTH_BUFFER_NEST_SIZE; + devlink_health_buffer_offset_inc(buffer, sizeof(*desc)); + + buffer->curr_nest++; + buffer->max_nested_depth = max(buffer->max_nested_depth, + buffer->curr_nest); + + return 0; +} +EXPORT_SYMBOL_GPL(devlink_health_buffer_nest_start); + +enum devlink_health_buffer_nest_end_cancel { + DEVLINK_HEALTH_BUFFER_NEST_END = 1, + DEVLINK_HEALTH_BUFFER_NEST_CANCEL, +}; + +static void +devlink_health_buffer_nest_end_cancel(struct devlink_health_buffer *buffer, + enum devlink_health_buffer_nest_end_cancel nest) +{ + struct devlink_health_buffer_desc *desc; + + WARN_ON(!buffer->curr_nest); + buffer->curr_nest--; + + desc = devlink_health_buffer_get_desc_from_offset(buffer); + desc->nest_end = nest; + devlink_health_buffer_offset_inc(buffer, sizeof(*desc)); +} + +void devlink_health_buffer_nest_end(struct devlink_health_buffer *buffer) +{ + devlink_health_buffer_nest_end_cancel(buffer, + DEVLINK_HEALTH_BUFFER_NEST_END); +} +EXPORT_SYMBOL_GPL(devlink_health_buffer_nest_end); + +void devlink_health_buffer_nest_cancel(struct devlink_health_buffer *buffer) +{ + devlink_health_buffer_nest_end_cancel(buffer, + DEVLINK_HEALTH_BUFFER_NEST_CANCEL); +} +EXPORT_SYMBOL_GPL(devlink_health_buffer_nest_cancel); + +int +devlink_health_buffer_put_object_name(struct devlink_health_buffer *buffer, + char *name) +{ + struct devlink_health_buffer_desc *desc; + int err; + + err = devlink_health_buffer_verify_len(buffer, strlen(name) + 1, + sizeof(*desc)); + if (err) + return err; + + desc = devlink_health_buffer_get_desc_from_offset(buffer); + desc->attrtype = DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_NAME; + desc->nla_type = NLA_NUL_STRING; + desc->len = strlen(name) + 1; + memcpy(&desc->value, name, desc->len); + devlink_health_buffer_offset_inc(buffer, sizeof(*desc) + desc->len); + + buffer->bytes_left_metadata -= sizeof(*desc); + buffer->bytes_left -= (strlen(name) + 1); + + return 0; +} +EXPORT_SYMBOL_GPL(devlink_health_buffer_put_object_name); + +static int +devlink_health_buffer_put_value(struct devlink_health_buffer *buffer, + u8 nla_type, void *value, int len) +{ + struct devlink_health_buffer_desc *desc; + int err; + + err = devlink_health_buffer_verify_len(buffer, len, sizeof(*desc)); + if (err) + return err; + + desc = devlink_health_buffer_get_desc_from_offset(buffer); + desc->attrtype = DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE_DATA; + desc->nla_type = nla_type; + desc->len = len; + memcpy(&desc->value, value, len); + devlink_health_buffer_offset_inc(buffer, sizeof(*desc) + desc->len); + + buffer->bytes_left_metadata -= sizeof(*desc); + buffer->bytes_left -= len; + + return 0; +} + +int +devlink_health_buffer_put_value_u8(struct devlink_health_buffer *buffer, + u8 value) +{ + int err; + + err = devlink_health_buffer_put_value(buffer, NLA_U8, &value, + sizeof(value)); + if (err) + return err; + + return 0; +} +EXPORT_SYMBOL_GPL(devlink_health_buffer_put_value_u8); + +int +devlink_health_buffer_put_value_u32(struct devlink_health_buffer *buffer, + u32 value) +{ + int err; + + err = devlink_health_buffer_put_value(buffer, NLA_U32, &value, + sizeof(value)); + if (err) + return err; + + return 0; +} +EXPORT_SYMBOL_GPL(devlink_health_buffer_put_value_u32); + +int +devlink_health_buffer_put_value_u64(struct devlink_health_buffer *buffer, + u64 value) +{ + int err; + + err = devlink_health_buffer_put_value(buffer, NLA_U64, &value, + sizeof(value)); + if (err) + return err; + + return 0; +} +EXPORT_SYMBOL_GPL(devlink_health_buffer_put_value_u64); + +int +devlink_health_buffer_put_value_string(struct devlink_health_buffer *buffer, + char *name) +{ + int err; + + if (strlen(name) + 1 > DEVLINK_HEALTH_BUFFER_MAX_CHUNK) + return -EINVAL; + + err = devlink_health_buffer_put_value(buffer, NLA_NUL_STRING, name, + strlen(name) + 1); + if (err) + return err; + + return 0; +} +EXPORT_SYMBOL_GPL(devlink_health_buffer_put_value_string); + +int +devlink_health_buffer_put_value_data(struct devlink_health_buffer *buffer, + void *data, int len) +{ + int err; + + if (len > DEVLINK_HEALTH_BUFFER_MAX_CHUNK) + return -EINVAL; + + err = devlink_health_buffer_put_value(buffer, NLA_BINARY, data, len); + if (err) + return err; + + return 0; +} +EXPORT_SYMBOL_GPL(devlink_health_buffer_put_value_data); + +static int +devlink_health_buffer_fill_data(struct sk_buff *skb, + struct devlink_health_buffer_desc *desc) +{ + int err = -EINVAL; + + switch (desc->nla_type) { + case NLA_U8: + err = nla_put_u8(skb, DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE_DATA, + *(u8 *)desc->value); + break; + case NLA_U32: + err = nla_put_u32(skb, DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE_DATA, + *(u32 *)desc->value); + break; + case NLA_U64: + err = nla_put_u64_64bit(skb, + DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE_DATA, + *(u64 *)desc->value, DEVLINK_ATTR_PAD); + break; + case NLA_NUL_STRING: + err = nla_put_string(skb, + DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE_DATA, + (char *)&desc->value); + break; + case NLA_BINARY: + err = nla_put(skb, DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE_DATA, + desc->len, (void *)&desc->value); + break; + } + + return err; +} + +static int +devlink_health_buffer_fill_type(struct sk_buff *skb, + struct devlink_health_buffer_desc *desc) +{ + int err = -EINVAL; + + switch (desc->nla_type) { + case NLA_U8: + err = nla_put_u8(skb, DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE_TYPE, + NLA_U8); + break; + case NLA_U32: + err = nla_put_u8(skb, DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE_TYPE, + NLA_U32); + break; + case NLA_U64: + err = nla_put_u8(skb, DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE_TYPE, + NLA_U64); + break; + case NLA_NUL_STRING: + err = nla_put_u8(skb, DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE_TYPE, + NLA_NUL_STRING); + break; + case NLA_BINARY: + err = nla_put_u8(skb, DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE_TYPE, + NLA_BINARY); + break; + } + + return err; +} + +static inline struct devlink_health_buffer_desc * +devlink_health_buffer_get_next_desc(struct devlink_health_buffer_desc *desc) +{ + return (void *)&desc->value + desc->len; +} + +static int +devlink_health_buffer_prepare_skb(struct sk_buff *skb, + struct devlink_health_buffer *buffer) +{ + struct devlink_health_buffer_desc *last_desc, *desc; + struct nlattr **buffer_nlattr; + int err; + int i = 0; + + buffer_nlattr = kcalloc(buffer->max_nested_depth, + sizeof(*buffer_nlattr), GFP_KERNEL); + if (!buffer_nlattr) + return -EINVAL; + + last_desc = devlink_health_buffer_get_desc_from_offset(buffer); + desc = buffer->data; + while (desc != last_desc) { + switch (desc->attrtype) { + case DEVLINK_ATTR_HEALTH_BUFFER_OBJECT: + case DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_PAIR: + case DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE: + case DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE_ARRAY: + buffer_nlattr[i] = nla_nest_start(skb, desc->attrtype); + if (!buffer_nlattr[i]) + goto nla_put_failure; + i++; + break; + case DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE_DATA: + err = devlink_health_buffer_fill_data(skb, desc); + if (err) + goto nla_put_failure; + err = devlink_health_buffer_fill_type(skb, desc); + if (err) + goto nla_put_failure; + break; + case DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_NAME: + err = nla_put_string(skb, desc->attrtype, + (char *)&desc->value); + if (err) + goto nla_put_failure; + break; + default: + WARN_ON(!desc->nest_end); + WARN_ON(i <= 0); + if (desc->nest_end == DEVLINK_HEALTH_BUFFER_NEST_END) + nla_nest_end(skb, buffer_nlattr[--i]); + else + nla_nest_cancel(skb, buffer_nlattr[--i]); + break; + } + desc = devlink_health_buffer_get_next_desc(desc); + }; + + return 0; + +nla_put_failure: + kfree(buffer_nlattr); + return err; +} + +static int +devlink_health_buffer_snd(struct genl_info *info, + enum devlink_command cmd, int flags, + struct devlink_health_buffer **buffers_array, + u64 num_of_buffers) +{ + struct sk_buff *skb; + struct nlmsghdr *nlh; + void *hdr; + int err; + u64 i; + + for (i = 0; i < num_of_buffers; i++) { + /* Skip buffer if driver did not fill it up with any data */ + if (!buffers_array[i]->offset) + continue; + + skb = genlmsg_new(GENLMSG_DEFAULT_SIZE, GFP_KERNEL); + if (!skb) + return -ENOMEM; + + hdr = genlmsg_put(skb, info->snd_portid, info->snd_seq, + &devlink_nl_family, NLM_F_MULTI, cmd); + if (!hdr) + goto nla_put_failure; + + err = devlink_health_buffer_prepare_skb(skb, buffers_array[i]); + if (err) + goto nla_put_failure; + + genlmsg_end(skb, hdr); + err = genlmsg_reply(skb, info); + if (err) + return err; + } + + skb = genlmsg_new(GENLMSG_DEFAULT_SIZE, GFP_KERNEL); + if (!skb) + return -ENOMEM; + nlh = nlmsg_put(skb, info->snd_portid, info->snd_seq, + NLMSG_DONE, 0, flags | NLM_F_MULTI); + err = genlmsg_reply(skb, info); + if (err) + return err; + + return 0; + +nla_put_failure: + err = -EIO; + nlmsg_free(skb); + return err; +} + static const struct nla_policy devlink_nl_policy[DEVLINK_ATTR_MAX + 1] = { [DEVLINK_ATTR_BUS_NAME] = { .type = NLA_NUL_STRING }, [DEVLINK_ATTR_DEV_NAME] = { .type = NLA_NUL_STRING }, From patchwork Mon Dec 31 14:31:56 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eran Ben Elisha X-Patchwork-Id: 1019573 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43T0CP37H4z9sBQ for ; Tue, 1 Jan 2019 01:32:49 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727318AbeLaOcf (ORCPT ); Mon, 31 Dec 2018 09:32:35 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:52199 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726355AbeLaOc1 (ORCPT ); Mon, 31 Dec 2018 09:32:27 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from eranbe@mellanox.com) with ESMTPS (AES256-SHA encrypted); 31 Dec 2018 16:32:21 +0200 Received: from dev-l-vrt-198.mtl.labs.mlnx (dev-l-vrt-198.mtl.labs.mlnx [10.134.198.1]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id wBVEWLYo017529; Mon, 31 Dec 2018 16:32:21 +0200 From: Eran Ben Elisha To: netdev@vger.kernel.org, "David S. Miller" , Jiri Pirko Cc: Moshe Shemesh , Aya Levin , Eran Ben Elisha , Tal Alon , Ariel Almog Subject: [PATCH RFC net-next 02/19] devlink: Add health reporter create/destroy functionality Date: Mon, 31 Dec 2018 16:31:56 +0200 Message-Id: <1546266733-9512-3-git-send-email-eranbe@mellanox.com> X-Mailer: git-send-email 1.8.4.3 In-Reply-To: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> References: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Devlink health reporter is an instance for reporting, diagnosing and recovering from run time errors discovered by the reporters. Define it's data structure and supported operations. In addition, expose devlink API to create and destroy a reporter. Each devlink instance will hold it's own reporters list. As part of the allocation, driver shall provide a set of callbacks which will be used the devlink in order to handle health reports and user commands related to this reporter. In addition, driver is entitled to provide some priv pointer, which can be fetched from the reporter by devlink_health_reporter_priv function. For each reporter, devlink will hold a metadata of statistics, buffers and status. Signed-off-by: Eran Ben Elisha --- include/net/devlink.h | 58 ++++++++++++++++++++++ net/core/devlink.c | 112 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 170 insertions(+) diff --git a/include/net/devlink.h b/include/net/devlink.h index 77c77319290a..6884a2571348 100644 --- a/include/net/devlink.h +++ b/include/net/devlink.h @@ -30,6 +30,7 @@ struct devlink { struct list_head param_list; struct list_head region_list; u32 snapshot_id; + struct list_head reporter_list; struct devlink_dpipe_headers *dpipe_headers; const struct devlink_ops *ops; struct device *dev; @@ -424,6 +425,34 @@ struct devlink_region; typedef void devlink_snapshot_data_dest_t(const void *data); struct devlink_health_buffer; +struct devlink_health_reporter; + +/** + * struct devlink_health_reporter_ops - Reporter operations + * @name: reporter name + * objdump_size: objdump buffer size allocated by the devlink + * diagnose_size: diagnose buffer size allocated by the devlink + * recover: callback to recover from reported error + * if priv_ctx is NULL, run a full recover + * objdump: callback to dump an object + * if priv_ctx is NULL, run a full objdump + * diagnose: callback to diagnose the current status + */ + +struct devlink_health_reporter_ops { + char *name; + unsigned int objdump_size; + unsigned int diagnose_size; + int (*recover)(struct devlink_health_reporter *reporter, + void *priv_ctx); + int (*objdump)(struct devlink_health_reporter *reporter, + struct devlink_health_buffer **buffers_array, + unsigned int buffer_size, unsigned int num_buffers, + void *priv_ctx); + int (*diagnose)(struct devlink_health_reporter *reporter, + struct devlink_health_buffer **buffers_array, + unsigned int buffer_size, unsigned int num_buffers); +}; struct devlink_ops { int (*reload)(struct devlink *devlink, struct netlink_ext_ack *extack); @@ -602,6 +631,16 @@ int devlink_health_buffer_put_value_string(struct devlink_health_buffer *buffer, char *name); int devlink_health_buffer_put_value_data(struct devlink_health_buffer *buffer, void *data, int len); +struct devlink_health_reporter * +devlink_health_reporter_create(struct devlink *devlink, + const struct devlink_health_reporter_ops *ops, + u64 graceful_period, bool auto_recover, + void *priv); +void +devlink_health_reporter_destroy(struct devlink_health_reporter *reporter); + +void * +devlink_health_reporter_priv(struct devlink_health_reporter *reporter); #else static inline struct devlink *devlink_alloc(const struct devlink_ops *ops, @@ -919,6 +958,25 @@ devlink_health_buffer_put_value_data(struct devlink_health_buffer *buffer, void *data, int len) { return 0; + +static inline struct devlink_health_reporter * +devlink_health_reporter_create(struct devlink *devlink, + const struct devlink_health_reporter_ops *ops, + u64 graceful_period, bool auto_recover, + void *priv) +{ + return NULL; +} + +static inline void +devlink_health_reporter_destroy(struct devlink_health_reporter *reporter) +{ +} + +static inline void * +devlink_health_reporter_priv(struct devlink_health_reporter *reporter) +{ + return NULL; } #endif diff --git a/net/core/devlink.c b/net/core/devlink.c index 36018af086a0..a69d4679211f 100644 --- a/net/core/devlink.c +++ b/net/core/devlink.c @@ -4098,6 +4098,117 @@ devlink_health_buffer_snd(struct genl_info *info, return err; } +struct devlink_health_reporter { + struct list_head list; + struct devlink_health_buffer **objdump_buffers_array; + struct mutex objdump_lock; /* lock parallel read/write from objdump buffers */ + struct devlink_health_buffer **diagnose_buffers_array; + struct mutex diagnose_lock; /* lock parallel read/write from diagnose buffers */ + void *priv; + const struct devlink_health_reporter_ops *ops; + struct devlink *devlink; + u64 graceful_period; + bool auto_recover; + u8 health_state; +}; + +void * +devlink_health_reporter_priv(struct devlink_health_reporter *reporter) +{ + return reporter->priv; +} +EXPORT_SYMBOL_GPL(devlink_health_reporter_priv); + +static struct devlink_health_reporter * +devlink_health_reporter_find_by_name(struct devlink *devlink, + const char *reporter_name) +{ + struct devlink_health_reporter *reporter; + + list_for_each_entry(reporter, &devlink->reporter_list, list) + if (!strcmp(reporter->ops->name, reporter_name)) + return reporter; + return NULL; +} + +/** + * devlink_health_reporter_create - create devlink health reporter + * + * @devlink: devlink + * @ops: ops + * @graceful_period: to avoid recovery loops, in msecs + * @auto_recover: auto recover when error occurs + * @priv: priv + */ +struct devlink_health_reporter * +devlink_health_reporter_create(struct devlink *devlink, + const struct devlink_health_reporter_ops *ops, + u64 graceful_period, bool auto_recover, + void *priv) +{ + struct devlink_health_reporter *reporter; + + if (devlink_health_reporter_find_by_name(devlink, ops->name)) + return ERR_PTR(-EEXIST); + + if (WARN_ON(ops->objdump && !ops->objdump_size) || + WARN_ON(ops->diagnose && !ops->diagnose_size) || + WARN_ON(auto_recover && !ops->recover) || + WARN_ON(graceful_period && !ops->recover)) + return ERR_PTR(-EINVAL); + + reporter = kzalloc(sizeof(*reporter), GFP_KERNEL); + if (!reporter) + return ERR_PTR(-ENOMEM); + + if (ops->objdump) { + reporter->objdump_buffers_array = + devlink_health_buffers_create(ops->objdump_size); + if (!reporter->objdump_buffers_array) + return ERR_PTR(-ENOMEM); + } + + if (ops->diagnose) { + reporter->diagnose_buffers_array = + devlink_health_buffers_create(ops->diagnose_size); + if (!reporter->diagnose_buffers_array) { + devlink_health_buffers_destroy(reporter->diagnose_buffers_array, + DEVLINK_HEALTH_SIZE_TO_BUFFERS(ops->objdump_size)); + return ERR_PTR(-ENOMEM); + } + } + + list_add_tail(&reporter->list, &devlink->reporter_list); + mutex_init(&reporter->objdump_lock); + mutex_init(&reporter->diagnose_lock); + + reporter->priv = priv; + reporter->ops = ops; + reporter->devlink = devlink; + reporter->graceful_period = graceful_period; + reporter->auto_recover = auto_recover; + + return reporter; +} +EXPORT_SYMBOL_GPL(devlink_health_reporter_create); + +/** + * devlink_health_reporter_destroy - destroy devlink health reporter + * + * @reporter: devlink health reporter to destroy + */ +void +devlink_health_reporter_destroy(struct devlink_health_reporter *reporter) +{ + list_del(&reporter->list); + devlink_health_buffers_destroy(reporter->diagnose_buffers_array, + DEVLINK_HEALTH_SIZE_TO_BUFFERS(reporter->ops->objdump_size)); + devlink_health_buffers_destroy(reporter->diagnose_buffers_array, + DEVLINK_HEALTH_SIZE_TO_BUFFERS(reporter->ops->diagnose_size)); + kfree(reporter); +} +EXPORT_SYMBOL_GPL(devlink_health_reporter_destroy); + static const struct nla_policy devlink_nl_policy[DEVLINK_ATTR_MAX + 1] = { [DEVLINK_ATTR_BUS_NAME] = { .type = NLA_NUL_STRING }, [DEVLINK_ATTR_DEV_NAME] = { .type = NLA_NUL_STRING }, @@ -4383,6 +4494,7 @@ struct devlink *devlink_alloc(const struct devlink_ops *ops, size_t priv_size) INIT_LIST_HEAD(&devlink->resource_list); INIT_LIST_HEAD(&devlink->param_list); INIT_LIST_HEAD(&devlink->region_list); + INIT_LIST_HEAD(&devlink->reporter_list); mutex_init(&devlink->lock); return devlink; } From patchwork Mon Dec 31 14:31:57 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eran Ben Elisha X-Patchwork-Id: 1019572 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43T0CN5ckKz9s55 for ; Tue, 1 Jan 2019 01:32:48 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727313AbeLaOcd (ORCPT ); Mon, 31 Dec 2018 09:32:33 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:52201 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726151AbeLaOc1 (ORCPT ); Mon, 31 Dec 2018 09:32:27 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from eranbe@mellanox.com) with ESMTPS (AES256-SHA encrypted); 31 Dec 2018 16:32:21 +0200 Received: from dev-l-vrt-198.mtl.labs.mlnx (dev-l-vrt-198.mtl.labs.mlnx [10.134.198.1]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id wBVEWLYp017529; Mon, 31 Dec 2018 16:32:21 +0200 From: Eran Ben Elisha To: netdev@vger.kernel.org, "David S. Miller" , Jiri Pirko Cc: Moshe Shemesh , Aya Levin , Eran Ben Elisha , Tal Alon , Ariel Almog Subject: [PATCH RFC net-next 03/19] devlink: Add health report functionality Date: Mon, 31 Dec 2018 16:31:57 +0200 Message-Id: <1546266733-9512-4-git-send-email-eranbe@mellanox.com> X-Mailer: git-send-email 1.8.4.3 In-Reply-To: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> References: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Upon error discover, every driver can report it to the devlink health mechanism via devlink_health_report function, using the appropriate reporter registered to it. Driver can pass error specific context which will be delivered to it as part of the objdump / recovery callbacks. Once an error is reported, devlink health will do the following actions: * A log is being send to the kernel trace events buffer * Health status and statistics are being updated for the reporter instance * Object dump is being taken and stored at the reporter instance (as long as there is no other Objdump which is already stored) * Auto recovery attempt is being done. depends on: - Auto Recovery configuration - Grace period vs. time since last recover Signed-off-by: Eran Ben Elisha --- include/net/devlink.h | 8 +++ include/trace/events/devlink.h | 62 +++++++++++++++++++++++ net/core/devlink.c | 93 ++++++++++++++++++++++++++++++++++ 3 files changed, 163 insertions(+) diff --git a/include/net/devlink.h b/include/net/devlink.h index 6884a2571348..40f988f4c940 100644 --- a/include/net/devlink.h +++ b/include/net/devlink.h @@ -641,6 +641,8 @@ devlink_health_reporter_destroy(struct devlink_health_reporter *reporter); void * devlink_health_reporter_priv(struct devlink_health_reporter *reporter); +int devlink_health_report(struct devlink_health_reporter *reporter, + const char *msg, void *priv_ctx); #else static inline struct devlink *devlink_alloc(const struct devlink_ops *ops, @@ -977,6 +979,12 @@ static inline void * devlink_health_reporter_priv(struct devlink_health_reporter *reporter) { return NULL; + +static inline int +devlink_health_report(struct devlink_health_reporter *reporter, + const char *msg, void *priv_ctx) +{ + return 0; } #endif diff --git a/include/trace/events/devlink.h b/include/trace/events/devlink.h index 44acfbca1266..7e39d2fc7c75 100644 --- a/include/trace/events/devlink.h +++ b/include/trace/events/devlink.h @@ -46,6 +46,65 @@ TRACE_EVENT(devlink_hwmsg, (int) __entry->len, __get_dynamic_array(buf), __entry->len) ); +TRACE_EVENT(devlink_health_report, + TP_PROTO(const struct devlink *devlink, const char *reporter_name, + const char *msg), + + TP_ARGS(devlink, reporter_name, msg), + + TP_STRUCT__entry( + __string(bus_name, devlink->dev->bus->name) + __string(dev_name, dev_name(devlink->dev)) + __string(driver_name, devlink->dev->driver->name) + __string(reporter_name, msg) + __string(msg, msg) + ), + + TP_fast_assign( + __assign_str(bus_name, devlink->dev->bus->name); + __assign_str(dev_name, dev_name(devlink->dev)); + __assign_str(driver_name, devlink->dev->driver->name); + __assign_str(reporter_name, reporter_name); + __assign_str(msg, msg); + ), + + TP_printk("bus_name=%s dev_name=%s driver_name=%s reporter_name=%s: %s", + __get_str(bus_name), __get_str(dev_name), + __get_str(driver_name), __get_str(reporter_name), + __get_str(msg)) +); + +TRACE_EVENT(devlink_health_recover_aborted, + TP_PROTO(const struct devlink *devlink, const char *reporter_name, + bool health_state, u64 time_since_last_recover), + + TP_ARGS(devlink, reporter_name, health_state, time_since_last_recover), + + TP_STRUCT__entry( + __string(bus_name, devlink->dev->bus->name) + __string(dev_name, dev_name(devlink->dev)) + __string(driver_name, devlink->dev->driver->name) + __string(reporter_name, reporter_name) + __field(bool, health_state) + __field(u64, time_since_last_recover) + ), + + TP_fast_assign( + __assign_str(bus_name, devlink->dev->bus->name); + __assign_str(dev_name, dev_name(devlink->dev)); + __assign_str(driver_name, devlink->dev->driver->name); + __assign_str(reporter_name, reporter_name); + __entry->health_state = health_state; + __entry->time_since_last_recover = time_since_last_recover; + ), + + TP_printk("bus_name=%s dev_name=%s driver_name=%s reporter_name=%s: health_state=%d time_since_last_recover = %llu recover aborted", + __get_str(bus_name), __get_str(dev_name), + __get_str(driver_name), __get_str(reporter_name), + __entry->health_state, + __entry->time_since_last_recover) +); + #endif /* _TRACE_DEVLINK_H */ /* This part must be outside protection */ @@ -64,6 +123,9 @@ static inline void trace_devlink_hwmsg(const struct devlink *devlink, { } +static inline void trace_devlink_health(const char *msg) +{ +} #endif /* _TRACE_DEVLINK_H */ #endif diff --git a/net/core/devlink.c b/net/core/devlink.c index a69d4679211f..5150d127da5f 100644 --- a/net/core/devlink.c +++ b/net/core/devlink.c @@ -4110,6 +4110,16 @@ struct devlink_health_reporter { u64 graceful_period; bool auto_recover; u8 health_state; + u8 objdump_avail; + u64 objdump_ts; + u64 error_count; + u64 recovery_count; + u64 last_recovery_ts; +}; + +enum devlink_health_reporter_state { + DEVLINK_HEALTH_REPORTER_STATE_HEALTHY, + DEVLINK_HEALTH_REPORTER_STATE_ERROR, }; void * @@ -4209,6 +4219,89 @@ devlink_health_reporter_destroy(struct devlink_health_reporter *reporter) } EXPORT_SYMBOL_GPL(devlink_health_reporter_destroy); +static int +devlink_health_reporter_recover(struct devlink_health_reporter *reporter, + void *priv_ctx) +{ + int err; + + if (!reporter->ops->recover) + return -EOPNOTSUPP; + + err = reporter->ops->recover(reporter, priv_ctx); + if (err) + return err; + + reporter->recovery_count++; + reporter->health_state = DEVLINK_HEALTH_REPORTER_STATE_HEALTHY; + reporter->last_recovery_ts = jiffies; + + return 0; +} + +static int devlink_health_do_objdump(struct devlink_health_reporter *reporter, + void *priv_ctx) +{ + int err; + + if (!reporter->ops->objdump) + return 0; + + if (reporter->objdump_avail) + return 0; + + devlink_health_buffers_reset(reporter->objdump_buffers_array, + DEVLINK_HEALTH_SIZE_TO_BUFFERS(reporter->ops->objdump_size)); + err = reporter->ops->objdump(reporter, reporter->objdump_buffers_array, + DEVLINK_HEALTH_BUFFER_SIZE, + DEVLINK_HEALTH_SIZE_TO_BUFFERS(reporter->ops->objdump_size), + priv_ctx); + if (!err) { + reporter->objdump_avail = true; + reporter->objdump_ts = jiffies; + } + + return err; +} + +int devlink_health_report(struct devlink_health_reporter *reporter, + const char *msg, void *priv_ctx) +{ + struct devlink *devlink = reporter->devlink; + int err = 0; + + /* write a log message of the current error */ + WARN_ON(!msg); + trace_devlink_health_report(devlink, reporter->ops->name, msg); + reporter->error_count++; + + /* abort if the previous error wasn't recovered */ + if (reporter->auto_recover && + (reporter->health_state != DEVLINK_HEALTH_REPORTER_STATE_HEALTHY || + jiffies - reporter->last_recovery_ts < + msecs_to_jiffies(reporter->graceful_period))) { + trace_devlink_health_recover_aborted(devlink, + reporter->ops->name, + reporter->health_state, + jiffies - + reporter->last_recovery_ts); + return -ECANCELED; + } + + reporter->health_state = DEVLINK_HEALTH_REPORTER_STATE_ERROR; + + mutex_lock(&reporter->objdump_lock); + /* store current objdump of current error, for later analysis */ + devlink_health_do_objdump(reporter, priv_ctx); + mutex_unlock(&reporter->objdump_lock); + + if (reporter->auto_recover) + err = devlink_health_reporter_recover(reporter, priv_ctx); + + return err; +} +EXPORT_SYMBOL_GPL(devlink_health_report); + static const struct nla_policy devlink_nl_policy[DEVLINK_ATTR_MAX + 1] = { [DEVLINK_ATTR_BUS_NAME] = { .type = NLA_NUL_STRING }, [DEVLINK_ATTR_DEV_NAME] = { .type = NLA_NUL_STRING }, From patchwork Mon Dec 31 14:31:58 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eran Ben Elisha X-Patchwork-Id: 1019571 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43T0CN0yvRz9sBQ for ; Tue, 1 Jan 2019 01:32:48 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727299AbeLaOc3 (ORCPT ); Mon, 31 Dec 2018 09:32:29 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:52203 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1725899AbeLaOc2 (ORCPT ); Mon, 31 Dec 2018 09:32:28 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from eranbe@mellanox.com) with ESMTPS (AES256-SHA encrypted); 31 Dec 2018 16:32:21 +0200 Received: from dev-l-vrt-198.mtl.labs.mlnx (dev-l-vrt-198.mtl.labs.mlnx [10.134.198.1]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id wBVEWLYq017529; Mon, 31 Dec 2018 16:32:21 +0200 From: Eran Ben Elisha To: netdev@vger.kernel.org, "David S. Miller" , Jiri Pirko Cc: Moshe Shemesh , Aya Levin , Eran Ben Elisha , Tal Alon , Ariel Almog Subject: [PATCH RFC net-next 04/19] devlink: Add health get command Date: Mon, 31 Dec 2018 16:31:58 +0200 Message-Id: <1546266733-9512-5-git-send-email-eranbe@mellanox.com> X-Mailer: git-send-email 1.8.4.3 In-Reply-To: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> References: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Add devlink health get command to provide reporter/s data for user space. Add the ability to get data per reporter or dump data from all available reporters. Signed-off-by: Eran Ben Elisha --- include/uapi/linux/devlink.h | 12 +++ net/core/devlink.c | 151 +++++++++++++++++++++++++++++++++++ 2 files changed, 163 insertions(+) diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h index cff0e0cb5ac2..5d35fabf25bf 100644 --- a/include/uapi/linux/devlink.h +++ b/include/uapi/linux/devlink.h @@ -89,6 +89,8 @@ enum devlink_command { DEVLINK_CMD_REGION_DEL, DEVLINK_CMD_REGION_READ, + DEVLINK_CMD_HEALTH_REPORTER_GET, + /* add new commands above here */ __DEVLINK_CMD_MAX, DEVLINK_CMD_MAX = __DEVLINK_CMD_MAX - 1 @@ -293,6 +295,16 @@ enum devlink_attr { DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE_TYPE, /* u8 */ DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE_DATA, /* dynamic */ + DEVLINK_ATTR_HEALTH_REPORTER, /* nested */ + DEVLINK_ATTR_HEALTH_REPORTER_NAME, /* string */ + DEVLINK_ATTR_HEALTH_REPORTER_STATE, /* u8 */ + DEVLINK_ATTR_HEALTH_REPORTER_ERR, /* u64 */ + DEVLINK_ATTR_HEALTH_REPORTER_RECOVER, /* u64 */ + DEVLINK_ATTR_HEALTH_REPORTER_OBJDUMP_AVAIL, /* u8 */ + DEVLINK_ATTR_HEALTH_REPORTER_OBJDUMP_TS, /* u64 */ + DEVLINK_ATTR_HEALTH_REPORTER_GRACEFUL_PERIOD, /* u64 */ + DEVLINK_ATTR_HEALTH_REPORTER_AUTO_RECOVER, /* u8 */ + /* add new attributes above here, update the policy in devlink.c */ __DEVLINK_ATTR_MAX, diff --git a/net/core/devlink.c b/net/core/devlink.c index 5150d127da5f..4e6624f889aa 100644 --- a/net/core/devlink.c +++ b/net/core/devlink.c @@ -4302,6 +4302,148 @@ int devlink_health_report(struct devlink_health_reporter *reporter, } EXPORT_SYMBOL_GPL(devlink_health_report); +static struct devlink_health_reporter * +devlink_health_reporter_get_from_info(struct devlink *devlink, + struct genl_info *info) +{ + char *reporter_name; + + if (!info->attrs[DEVLINK_ATTR_HEALTH_REPORTER_NAME]) + return NULL; + + reporter_name = + nla_data(info->attrs[DEVLINK_ATTR_HEALTH_REPORTER_NAME]); + return devlink_health_reporter_find_by_name(devlink, reporter_name); +} + +static int +devlink_nl_health_reporter_fill(struct sk_buff *msg, + struct devlink *devlink, + struct devlink_health_reporter *reporter, + enum devlink_command cmd, u32 portid, + u32 seq, int flags) +{ + struct nlattr *reporter_attr; + void *hdr; + + hdr = genlmsg_put(msg, portid, seq, &devlink_nl_family, flags, cmd); + if (!hdr) + return -EMSGSIZE; + + if (devlink_nl_put_handle(msg, devlink)) + goto genlmsg_cancel; + + reporter_attr = nla_nest_start(msg, DEVLINK_ATTR_HEALTH_REPORTER); + if (!reporter_attr) + goto genlmsg_cancel; + if (nla_put_string(msg, DEVLINK_ATTR_HEALTH_REPORTER_NAME, + reporter->ops->name)) + goto reporter_nest_cancel; + if (nla_put_u8(msg, DEVLINK_ATTR_HEALTH_REPORTER_STATE, + reporter->health_state)) + goto reporter_nest_cancel; + if (nla_put_u64_64bit(msg, DEVLINK_ATTR_HEALTH_REPORTER_ERR, + reporter->error_count, DEVLINK_ATTR_PAD)) + goto reporter_nest_cancel; + if (nla_put_u64_64bit(msg, DEVLINK_ATTR_HEALTH_REPORTER_RECOVER, + reporter->recovery_count, DEVLINK_ATTR_PAD)) + goto reporter_nest_cancel; + if (nla_put_u64_64bit(msg, DEVLINK_ATTR_HEALTH_REPORTER_GRACEFUL_PERIOD, + reporter->graceful_period, + DEVLINK_ATTR_PAD)) + goto reporter_nest_cancel; + if (nla_put_u8(msg, DEVLINK_ATTR_HEALTH_REPORTER_AUTO_RECOVER, + reporter->auto_recover)) + goto reporter_nest_cancel; + if (nla_put_u8(msg, DEVLINK_ATTR_HEALTH_REPORTER_OBJDUMP_AVAIL, + reporter->objdump_avail)) + goto reporter_nest_cancel; + if (reporter->objdump_avail && + nla_put_u64_64bit(msg, DEVLINK_ATTR_HEALTH_REPORTER_OBJDUMP_TS, + reporter->objdump_ts, DEVLINK_ATTR_PAD)) + goto reporter_nest_cancel; + + nla_nest_end(msg, reporter_attr); + genlmsg_end(msg, hdr); + return 0; + +reporter_nest_cancel: + nla_nest_end(msg, reporter_attr); +genlmsg_cancel: + genlmsg_cancel(msg, hdr); + return -EMSGSIZE; +} + +static int devlink_nl_cmd_health_reporter_get_doit(struct sk_buff *skb, + struct genl_info *info) +{ + struct devlink *devlink = info->user_ptr[0]; + struct devlink_health_reporter *reporter; + struct sk_buff *msg; + int err; + + reporter = devlink_health_reporter_get_from_info(devlink, info); + if (!reporter) + return -EINVAL; + + msg = nlmsg_new(NLMSG_DEFAULT_SIZE, GFP_KERNEL); + if (!msg) + return -ENOMEM; + + err = devlink_nl_health_reporter_fill(msg, devlink, reporter, + DEVLINK_CMD_HEALTH_REPORTER_GET, + info->snd_portid, info->snd_seq, + 0); + if (err) { + nlmsg_free(msg); + return err; + } + + return genlmsg_reply(msg, info); +} + +static int +devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg, + struct netlink_callback *cb) +{ + struct devlink_health_reporter *reporter; + struct devlink *devlink; + int start = cb->args[0]; + int idx = 0; + int err; + + mutex_lock(&devlink_mutex); + list_for_each_entry(devlink, &devlink_list, list) { + if (!net_eq(devlink_net(devlink), sock_net(msg->sk))) + continue; + mutex_lock(&devlink->lock); + list_for_each_entry(reporter, &devlink->reporter_list, + list) { + if (idx < start) { + idx++; + continue; + } + err = devlink_nl_health_reporter_fill(msg, devlink, + reporter, + DEVLINK_CMD_HEALTH_REPORTER_GET, + NETLINK_CB(cb->skb).portid, + cb->nlh->nlmsg_seq, + NLM_F_MULTI); + if (err) { + mutex_unlock(&devlink->lock); + goto out; + } + idx++; + } + mutex_unlock(&devlink->lock); + } +out: + mutex_unlock(&devlink_mutex); + + cb->args[0] = idx; + return msg->len; +} + static const struct nla_policy devlink_nl_policy[DEVLINK_ATTR_MAX + 1] = { [DEVLINK_ATTR_BUS_NAME] = { .type = NLA_NUL_STRING }, [DEVLINK_ATTR_DEV_NAME] = { .type = NLA_NUL_STRING }, @@ -4327,6 +4469,7 @@ static const struct nla_policy devlink_nl_policy[DEVLINK_ATTR_MAX + 1] = { [DEVLINK_ATTR_PARAM_VALUE_CMODE] = { .type = NLA_U8 }, [DEVLINK_ATTR_REGION_NAME] = { .type = NLA_NUL_STRING }, [DEVLINK_ATTR_REGION_SNAPSHOT_ID] = { .type = NLA_U32 }, + [DEVLINK_ATTR_HEALTH_REPORTER_NAME] = { .type = NLA_NUL_STRING }, }; static const struct genl_ops devlink_nl_ops[] = { @@ -4547,6 +4690,14 @@ static const struct genl_ops devlink_nl_ops[] = { .flags = GENL_ADMIN_PERM, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, }, + { + .cmd = DEVLINK_CMD_HEALTH_REPORTER_GET, + .doit = devlink_nl_cmd_health_reporter_get_doit, + .dumpit = devlink_nl_cmd_health_reporter_get_dumpit, + .policy = devlink_nl_policy, + .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, + /* can be retrieved by unprivileged users */ + }, }; static struct genl_family devlink_nl_family __ro_after_init = { From patchwork Mon Dec 31 14:31:59 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eran Ben Elisha X-Patchwork-Id: 1019567 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43T0CK3GH3z9sBQ for ; Tue, 1 Jan 2019 01:32:45 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727083AbeLaOc0 (ORCPT ); Mon, 31 Dec 2018 09:32:26 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:52204 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726625AbeLaOcZ (ORCPT ); Mon, 31 Dec 2018 09:32:25 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from eranbe@mellanox.com) with ESMTPS (AES256-SHA encrypted); 31 Dec 2018 16:32:21 +0200 Received: from dev-l-vrt-198.mtl.labs.mlnx (dev-l-vrt-198.mtl.labs.mlnx [10.134.198.1]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id wBVEWLYr017529; Mon, 31 Dec 2018 16:32:21 +0200 From: Eran Ben Elisha To: netdev@vger.kernel.org, "David S. Miller" , Jiri Pirko Cc: Moshe Shemesh , Aya Levin , Eran Ben Elisha , Tal Alon , Ariel Almog Subject: [PATCH RFC net-next 05/19] devlink: Add health set command Date: Mon, 31 Dec 2018 16:31:59 +0200 Message-Id: <1546266733-9512-6-git-send-email-eranbe@mellanox.com> X-Mailer: git-send-email 1.8.4.3 In-Reply-To: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> References: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Add devlink health set command, in order to set configuration parameters for a specific reporter. Supported parameters are: - graceful_period: Time interval between auto recoveries (in msec) - auto_recover: Determines if the devlink shall execute recover upon receiving error for the reporter Signed-off-by: Eran Ben Elisha --- include/uapi/linux/devlink.h | 1 + net/core/devlink.c | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h index 5d35fabf25bf..7b6253c3fc89 100644 --- a/include/uapi/linux/devlink.h +++ b/include/uapi/linux/devlink.h @@ -90,6 +90,7 @@ enum devlink_command { DEVLINK_CMD_REGION_READ, DEVLINK_CMD_HEALTH_REPORTER_GET, + DEVLINK_CMD_HEALTH_REPORTER_SET, /* add new commands above here */ __DEVLINK_CMD_MAX, diff --git a/net/core/devlink.c b/net/core/devlink.c index 4e6624f889aa..c0959242d891 100644 --- a/net/core/devlink.c +++ b/net/core/devlink.c @@ -4444,6 +4444,33 @@ devlink_nl_cmd_health_reporter_get_dumpit(struct sk_buff *msg, return msg->len; } +static int +devlink_nl_cmd_health_reporter_set_doit(struct sk_buff *skb, + struct genl_info *info) +{ + struct devlink *devlink = info->user_ptr[0]; + struct devlink_health_reporter *reporter; + + reporter = devlink_health_reporter_get_from_info(devlink, info); + if (!reporter) + return -EINVAL; + + if (!reporter->ops->recover && + (info->attrs[DEVLINK_ATTR_HEALTH_REPORTER_GRACEFUL_PERIOD] || + info->attrs[DEVLINK_ATTR_HEALTH_REPORTER_AUTO_RECOVER])) + return -EINVAL; + + if (info->attrs[DEVLINK_ATTR_HEALTH_REPORTER_GRACEFUL_PERIOD]) + reporter->graceful_period = + nla_get_u64(info->attrs[DEVLINK_ATTR_HEALTH_REPORTER_GRACEFUL_PERIOD]); + + if (info->attrs[DEVLINK_ATTR_HEALTH_REPORTER_AUTO_RECOVER]) + reporter->auto_recover = + nla_get_u8(info->attrs[DEVLINK_ATTR_HEALTH_REPORTER_AUTO_RECOVER]); + + return 0; +} + static const struct nla_policy devlink_nl_policy[DEVLINK_ATTR_MAX + 1] = { [DEVLINK_ATTR_BUS_NAME] = { .type = NLA_NUL_STRING }, [DEVLINK_ATTR_DEV_NAME] = { .type = NLA_NUL_STRING }, @@ -4470,6 +4497,8 @@ static const struct nla_policy devlink_nl_policy[DEVLINK_ATTR_MAX + 1] = { [DEVLINK_ATTR_REGION_NAME] = { .type = NLA_NUL_STRING }, [DEVLINK_ATTR_REGION_SNAPSHOT_ID] = { .type = NLA_U32 }, [DEVLINK_ATTR_HEALTH_REPORTER_NAME] = { .type = NLA_NUL_STRING }, + [DEVLINK_ATTR_HEALTH_REPORTER_GRACEFUL_PERIOD] = { .type = NLA_U64 }, + [DEVLINK_ATTR_HEALTH_REPORTER_AUTO_RECOVER] = { .type = NLA_U8 }, }; static const struct genl_ops devlink_nl_ops[] = { @@ -4698,6 +4727,13 @@ static const struct genl_ops devlink_nl_ops[] = { .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, /* can be retrieved by unprivileged users */ }, + { + .cmd = DEVLINK_CMD_HEALTH_REPORTER_SET, + .doit = devlink_nl_cmd_health_reporter_set_doit, + .policy = devlink_nl_policy, + .flags = GENL_ADMIN_PERM, + .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, + }, }; static struct genl_family devlink_nl_family __ro_after_init = { From patchwork Mon Dec 31 14:32:00 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eran Ben Elisha X-Patchwork-Id: 1019580 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43T0CV36Sdz9s55 for ; Tue, 1 Jan 2019 01:32:54 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726999AbeLaOc0 (ORCPT ); Mon, 31 Dec 2018 09:32:26 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:52212 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726456AbeLaOcZ (ORCPT ); Mon, 31 Dec 2018 09:32:25 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from eranbe@mellanox.com) with ESMTPS (AES256-SHA encrypted); 31 Dec 2018 16:32:21 +0200 Received: from dev-l-vrt-198.mtl.labs.mlnx (dev-l-vrt-198.mtl.labs.mlnx [10.134.198.1]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id wBVEWLYs017529; Mon, 31 Dec 2018 16:32:21 +0200 From: Eran Ben Elisha To: netdev@vger.kernel.org, "David S. Miller" , Jiri Pirko Cc: Moshe Shemesh , Aya Levin , Eran Ben Elisha , Tal Alon , Ariel Almog Subject: [PATCH RFC net-next 06/19] devlink: Add health recover command Date: Mon, 31 Dec 2018 16:32:00 +0200 Message-Id: <1546266733-9512-7-git-send-email-eranbe@mellanox.com> X-Mailer: git-send-email 1.8.4.3 In-Reply-To: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> References: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Add devlink health recover command to the uapi, in order to allow the user to execute a recover operation over a specific reporter. Signed-off-by: Eran Ben Elisha --- include/uapi/linux/devlink.h | 1 + net/core/devlink.c | 20 ++++++++++++++++++++ 2 files changed, 21 insertions(+) diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h index 7b6253c3fc89..243819560715 100644 --- a/include/uapi/linux/devlink.h +++ b/include/uapi/linux/devlink.h @@ -91,6 +91,7 @@ enum devlink_command { DEVLINK_CMD_HEALTH_REPORTER_GET, DEVLINK_CMD_HEALTH_REPORTER_SET, + DEVLINK_CMD_HEALTH_REPORTER_RECOVER, /* add new commands above here */ __DEVLINK_CMD_MAX, diff --git a/net/core/devlink.c b/net/core/devlink.c index c0959242d891..31fff86390e3 100644 --- a/net/core/devlink.c +++ b/net/core/devlink.c @@ -4471,6 +4471,19 @@ devlink_nl_cmd_health_reporter_set_doit(struct sk_buff *skb, return 0; } +static int devlink_nl_cmd_health_reporter_recover_doit(struct sk_buff *skb, + struct genl_info *info) +{ + struct devlink *devlink = info->user_ptr[0]; + struct devlink_health_reporter *reporter; + + reporter = devlink_health_reporter_get_from_info(devlink, info); + if (!reporter) + return -EINVAL; + + return devlink_health_reporter_recover(reporter, NULL); +} + static const struct nla_policy devlink_nl_policy[DEVLINK_ATTR_MAX + 1] = { [DEVLINK_ATTR_BUS_NAME] = { .type = NLA_NUL_STRING }, [DEVLINK_ATTR_DEV_NAME] = { .type = NLA_NUL_STRING }, @@ -4734,6 +4747,13 @@ static const struct genl_ops devlink_nl_ops[] = { .flags = GENL_ADMIN_PERM, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, }, + { + .cmd = DEVLINK_CMD_HEALTH_REPORTER_RECOVER, + .doit = devlink_nl_cmd_health_reporter_recover_doit, + .policy = devlink_nl_policy, + .flags = GENL_ADMIN_PERM, + .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, + }, }; static struct genl_family devlink_nl_family __ro_after_init = { From patchwork Mon Dec 31 14:32:01 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eran Ben Elisha X-Patchwork-Id: 1019581 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43T0CW6M9jz9s55 for ; Tue, 1 Jan 2019 01:32:55 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726873AbeLaOc0 (ORCPT ); Mon, 31 Dec 2018 09:32:26 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:52220 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726549AbeLaOcZ (ORCPT ); Mon, 31 Dec 2018 09:32:25 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from eranbe@mellanox.com) with ESMTPS (AES256-SHA encrypted); 31 Dec 2018 16:32:21 +0200 Received: from dev-l-vrt-198.mtl.labs.mlnx (dev-l-vrt-198.mtl.labs.mlnx [10.134.198.1]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id wBVEWLYt017529; Mon, 31 Dec 2018 16:32:21 +0200 From: Eran Ben Elisha To: netdev@vger.kernel.org, "David S. Miller" , Jiri Pirko Cc: Moshe Shemesh , Aya Levin , Eran Ben Elisha , Tal Alon , Ariel Almog Subject: [PATCH RFC net-next 07/19] devlink: Add health diagnose command Date: Mon, 31 Dec 2018 16:32:01 +0200 Message-Id: <1546266733-9512-8-git-send-email-eranbe@mellanox.com> X-Mailer: git-send-email 1.8.4.3 In-Reply-To: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> References: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Add devlink health diagnose command, in order to run a diagnose operation over a specific reporter. It is expected from driver's callback for diagnose command to fill it via the buffer descriptors API. Devlink will parse it and convert isto netlink nla API in order to pass it to the user. Signed-off-by: Eran Ben Elisha --- include/uapi/linux/devlink.h | 1 + net/core/devlink.c | 51 ++++++++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+) diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h index 243819560715..73faabe37488 100644 --- a/include/uapi/linux/devlink.h +++ b/include/uapi/linux/devlink.h @@ -92,6 +92,7 @@ enum devlink_command { DEVLINK_CMD_HEALTH_REPORTER_GET, DEVLINK_CMD_HEALTH_REPORTER_SET, DEVLINK_CMD_HEALTH_REPORTER_RECOVER, + DEVLINK_CMD_HEALTH_REPORTER_DIAGNOSE, /* add new commands above here */ __DEVLINK_CMD_MAX, diff --git a/net/core/devlink.c b/net/core/devlink.c index 31fff86390e3..343ebbd38ba8 100644 --- a/net/core/devlink.c +++ b/net/core/devlink.c @@ -4484,6 +4484,50 @@ static int devlink_nl_cmd_health_reporter_recover_doit(struct sk_buff *skb, return devlink_health_reporter_recover(reporter, NULL); } +static int devlink_nl_cmd_health_reporter_diagnose_doit(struct sk_buff *skb, + struct genl_info *info) +{ + struct devlink *devlink = info->user_ptr[0]; + struct devlink_health_reporter *reporter; + u64 num_of_buffers; + int err; + + reporter = devlink_health_reporter_get_from_info(devlink, info); + if (!reporter) + return -EINVAL; + + if (!reporter->ops->diagnose) + return -EOPNOTSUPP; + + num_of_buffers = + DEVLINK_HEALTH_SIZE_TO_BUFFERS(reporter->ops->diagnose_size); + + mutex_lock(&reporter->diagnose_lock); + devlink_health_buffers_reset(reporter->diagnose_buffers_array, + num_of_buffers); + + err = reporter->ops->diagnose(reporter, + reporter->diagnose_buffers_array, + DEVLINK_HEALTH_BUFFER_SIZE, + num_of_buffers); + if (err) + goto out; + + err = devlink_health_buffer_snd(info, + DEVLINK_CMD_HEALTH_REPORTER_DIAGNOSE, + 0, reporter->diagnose_buffers_array, + num_of_buffers); + if (err) + goto out; + + mutex_unlock(&reporter->diagnose_lock); + return 0; + +out: + mutex_unlock(&reporter->diagnose_lock); + return err; +} + static const struct nla_policy devlink_nl_policy[DEVLINK_ATTR_MAX + 1] = { [DEVLINK_ATTR_BUS_NAME] = { .type = NLA_NUL_STRING }, [DEVLINK_ATTR_DEV_NAME] = { .type = NLA_NUL_STRING }, @@ -4754,6 +4798,13 @@ static const struct genl_ops devlink_nl_ops[] = { .flags = GENL_ADMIN_PERM, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, }, + { + .cmd = DEVLINK_CMD_HEALTH_REPORTER_DIAGNOSE, + .doit = devlink_nl_cmd_health_reporter_diagnose_doit, + .policy = devlink_nl_policy, + .flags = GENL_ADMIN_PERM, + .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, + }, }; static struct genl_family devlink_nl_family __ro_after_init = { From patchwork Mon Dec 31 14:32:02 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eran Ben Elisha X-Patchwork-Id: 1019584 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43T0Cf03HZz9s55 for ; Tue, 1 Jan 2019 01:33:02 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727369AbeLaOdB (ORCPT ); Mon, 31 Dec 2018 09:33:01 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:52229 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726220AbeLaOcZ (ORCPT ); Mon, 31 Dec 2018 09:32:25 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from eranbe@mellanox.com) with ESMTPS (AES256-SHA encrypted); 31 Dec 2018 16:32:21 +0200 Received: from dev-l-vrt-198.mtl.labs.mlnx (dev-l-vrt-198.mtl.labs.mlnx [10.134.198.1]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id wBVEWLYu017529; Mon, 31 Dec 2018 16:32:21 +0200 From: Eran Ben Elisha To: netdev@vger.kernel.org, "David S. Miller" , Jiri Pirko Cc: Moshe Shemesh , Aya Levin , Eran Ben Elisha , Tal Alon , Ariel Almog Subject: [PATCH RFC net-next 08/19] devlink: Add health objdump {get, clear} commands Date: Mon, 31 Dec 2018 16:32:02 +0200 Message-Id: <1546266733-9512-9-git-send-email-eranbe@mellanox.com> X-Mailer: git-send-email 1.8.4.3 In-Reply-To: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> References: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Add devlink health objdump commands, in order to run an objdump operation over a specific reporter. The supported operations are objdump_get in order to get last saved objdump (if not exist, dump now) and objdump_clear to clear last saved objdump. It is expected from driver's callback for diagnose command to fill it via the buffer descriptors API. Devlink will parse it and convert isto netlink nla API in order to pass it to the user. Signed-off-by: Eran Ben Elisha --- include/uapi/linux/devlink.h | 2 + net/core/devlink.c | 73 ++++++++++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+) diff --git a/include/uapi/linux/devlink.h b/include/uapi/linux/devlink.h index 73faabe37488..0fecb7618d18 100644 --- a/include/uapi/linux/devlink.h +++ b/include/uapi/linux/devlink.h @@ -93,6 +93,8 @@ enum devlink_command { DEVLINK_CMD_HEALTH_REPORTER_SET, DEVLINK_CMD_HEALTH_REPORTER_RECOVER, DEVLINK_CMD_HEALTH_REPORTER_DIAGNOSE, + DEVLINK_CMD_HEALTH_REPORTER_OBJDUMP_GET, + DEVLINK_CMD_HEALTH_REPORTER_OBJDUMP_CLEAR, /* add new commands above here */ __DEVLINK_CMD_MAX, diff --git a/net/core/devlink.c b/net/core/devlink.c index 343ebbd38ba8..d45a284aa608 100644 --- a/net/core/devlink.c +++ b/net/core/devlink.c @@ -4528,6 +4528,65 @@ static int devlink_nl_cmd_health_reporter_diagnose_doit(struct sk_buff *skb, return err; } +static void +devlink_health_objdump_clear(struct devlink_health_reporter *reporter) +{ + reporter->objdump_avail = false; + reporter->objdump_ts = 0; + devlink_health_buffers_reset(reporter->objdump_buffers_array, + DEVLINK_HEALTH_SIZE_TO_BUFFERS(reporter->ops->objdump_size)); +} + +static int devlink_nl_cmd_health_reporter_objdump_get_doit(struct sk_buff *skb, + struct genl_info *info) +{ + struct devlink *devlink = info->user_ptr[0]; + struct devlink_health_reporter *reporter; + u64 num_of_buffers; + int err; + + reporter = devlink_health_reporter_get_from_info(devlink, info); + if (!reporter) + return -EINVAL; + + if (!reporter->ops->objdump) + return -EOPNOTSUPP; + + num_of_buffers = + DEVLINK_HEALTH_SIZE_TO_BUFFERS(reporter->ops->objdump_size); + + mutex_lock(&reporter->objdump_lock); + err = devlink_health_do_objdump(reporter, NULL); + if (err) + goto out; + + err = devlink_health_buffer_snd(info, + DEVLINK_CMD_HEALTH_REPORTER_OBJDUMP_GET, + 0, reporter->objdump_buffers_array, + num_of_buffers); + +out: + mutex_unlock(&reporter->objdump_lock); + return err; +} + +static int +devlink_nl_cmd_health_reporter_objdump_clear_doit(struct sk_buff *skb, + struct genl_info *info) +{ + struct devlink *devlink = info->user_ptr[0]; + struct devlink_health_reporter *reporter; + + reporter = devlink_health_reporter_get_from_info(devlink, info); + if (!reporter) + return -EINVAL; + + mutex_lock(&reporter->objdump_lock); + devlink_health_objdump_clear(reporter); + mutex_unlock(&reporter->objdump_lock); + return 0; +} + static const struct nla_policy devlink_nl_policy[DEVLINK_ATTR_MAX + 1] = { [DEVLINK_ATTR_BUS_NAME] = { .type = NLA_NUL_STRING }, [DEVLINK_ATTR_DEV_NAME] = { .type = NLA_NUL_STRING }, @@ -4805,6 +4864,20 @@ static const struct genl_ops devlink_nl_ops[] = { .flags = GENL_ADMIN_PERM, .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, }, + { + .cmd = DEVLINK_CMD_HEALTH_REPORTER_OBJDUMP_GET, + .doit = devlink_nl_cmd_health_reporter_objdump_get_doit, + .policy = devlink_nl_policy, + .flags = GENL_ADMIN_PERM, + .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, + }, + { + .cmd = DEVLINK_CMD_HEALTH_REPORTER_OBJDUMP_CLEAR, + .doit = devlink_nl_cmd_health_reporter_objdump_clear_doit, + .policy = devlink_nl_policy, + .flags = GENL_ADMIN_PERM, + .internal_flags = DEVLINK_NL_FLAG_NEED_DEVLINK, + }, }; static struct genl_family devlink_nl_family __ro_after_init = { From patchwork Mon Dec 31 14:32:03 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eran Ben Elisha X-Patchwork-Id: 1019576 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43T0CR2rdrz9s55 for ; Tue, 1 Jan 2019 01:32:51 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727334AbeLaOcm (ORCPT ); Mon, 31 Dec 2018 09:32:42 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:52244 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1725996AbeLaOc0 (ORCPT ); Mon, 31 Dec 2018 09:32:26 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from eranbe@mellanox.com) with ESMTPS (AES256-SHA encrypted); 31 Dec 2018 16:32:21 +0200 Received: from dev-l-vrt-198.mtl.labs.mlnx (dev-l-vrt-198.mtl.labs.mlnx [10.134.198.1]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id wBVEWLYv017529; Mon, 31 Dec 2018 16:32:21 +0200 From: Eran Ben Elisha To: netdev@vger.kernel.org, "David S. Miller" , Jiri Pirko Cc: Moshe Shemesh , Aya Levin , Eran Ben Elisha , Tal Alon , Ariel Almog Subject: [PATCH RFC net-next 09/19] net/mlx5e: Add TX reporter support Date: Mon, 31 Dec 2018 16:32:03 +0200 Message-Id: <1546266733-9512-10-git-send-email-eranbe@mellanox.com> X-Mailer: git-send-email 1.8.4.3 In-Reply-To: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> References: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Add mlx5e tx reporter to devlink health reporters. This reporter will be responsible for diagnosing, reporting and recovering of TX errors. This patch declares the TX reporter operations and allocate it using the devlink health API. Currently, this reporter supports reporting and recovering from send error CQE only. In addition, it adds diagnose information for the open SQs. For a local SQ recover (due to driver error report), in case of SQ recover failure, the recover operation will be considered as a failure. For a full TX recover, an attempt to close and open the channels will be done. If this one passed successfully, it will be considered as a successful recover. The SQ recover from error CQE flow is not a new feature in the driver, this patch re-organize the functions and adapt them for the devlink health API. For this purpose, move code from en_main.c to a new file named reporter_tx.c. Signed-off-by: Eran Ben Elisha --- .../net/ethernet/mellanox/mlx5/core/Makefile | 2 +- drivers/net/ethernet/mellanox/mlx5/core/en.h | 18 +- .../ethernet/mellanox/mlx5/core/en/reporter.h | 14 + .../mellanox/mlx5/core/en/reporter_tx.c | 321 ++++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/en_main.c | 135 +------- .../net/ethernet/mellanox/mlx5/core/en_tx.c | 2 +- 6 files changed, 367 insertions(+), 125 deletions(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/reporter.h create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/en/reporter_tx.c diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Makefile b/drivers/net/ethernet/mellanox/mlx5/core/Makefile index 9de9abacf7f6..6bb2a860b15b 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/Makefile +++ b/drivers/net/ethernet/mellanox/mlx5/core/Makefile @@ -22,7 +22,7 @@ mlx5_core-y := main.o cmd.o debugfs.o fw.o eq.o uar.o pagealloc.o \ # mlx5_core-$(CONFIG_MLX5_CORE_EN) += en_main.o en_common.o en_fs.o en_ethtool.o \ en_tx.o en_rx.o en_dim.o en_txrx.o en/xdp.o en_stats.o \ - en_selftest.o en/port.o en/monitor_stats.o + en_selftest.o en/port.o en/monitor_stats.o en/reporter_tx.o # # Netdev extra diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h index 8fa8fdd30b85..27e276c9bf84 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h @@ -388,10 +388,7 @@ struct mlx5e_txqsq { struct mlx5e_channel *channel; int txq_ix; u32 rate_limit; - struct mlx5e_txqsq_recover { - struct work_struct recover_work; - u64 last_recover; - } recover; + struct work_struct recover_work; } ____cacheline_aligned_in_smp; struct mlx5e_dma_info { @@ -682,6 +679,13 @@ struct mlx5e_rss_params { u8 hfunc; }; +struct mlx5e_modify_sq_param { + int curr_state; + int next_state; + int rl_update; + int rl_index; +}; + struct mlx5e_priv { /* priv data path fields - start */ struct mlx5e_txqsq *txq2sq[MLX5E_MAX_NUM_CHANNELS * MLX5E_MAX_NUM_TC]; @@ -737,6 +741,7 @@ struct mlx5e_priv { #ifdef CONFIG_MLX5_EN_TLS struct mlx5e_tls *tls; #endif + struct devlink_health_reporter *tx_reporter; }; struct mlx5e_profile { @@ -866,6 +871,11 @@ void mlx5e_set_rq_type(struct mlx5_core_dev *mdev, struct mlx5e_params *params); void mlx5e_init_rq_type_params(struct mlx5_core_dev *mdev, struct mlx5e_params *params); +int mlx5e_modify_sq(struct mlx5_core_dev *mdev, u32 sqn, + struct mlx5e_modify_sq_param *p); +void mlx5e_activate_txqsq(struct mlx5e_txqsq *sq); +void mlx5e_tx_disable_queue(struct netdev_queue *txq); + static inline bool mlx5e_tunnel_inner_ft_supported(struct mlx5_core_dev *mdev) { return (MLX5_CAP_ETH(mdev, tunnel_stateless_gre) && diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/reporter.h b/drivers/net/ethernet/mellanox/mlx5/core/en/reporter.h new file mode 100644 index 000000000000..74083e120ab3 --- /dev/null +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/reporter.h @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* Copyright (c) 2018 Mellanox Technologies. */ + +#ifndef __MLX5E_EN_REPORTER_H +#define __MLX5E_EN_REPORTER_H + +#include +#include "en.h" + +int mlx5e_tx_reporter_create(struct mlx5e_priv *priv); +void mlx5e_tx_reporter_destroy(struct mlx5e_priv *priv); +void mlx5e_tx_reporter_err_cqe(struct mlx5e_txqsq *sq); + +#endif diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_tx.c new file mode 100644 index 000000000000..b88539528f12 --- /dev/null +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_tx.c @@ -0,0 +1,321 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* Copyright (c) 2018 Mellanox Technologies. */ + +#include +#include "reporter.h" +#include "lib/eq.h" + +#define MLX5E_TX_REPORTER_PER_SQ_MAX_LEN 256 + +struct mlx5e_tx_err_ctx { + int (*recover)(struct mlx5e_txqsq *sq); + struct mlx5e_txqsq *sq; +}; + +static int mlx5e_wait_for_sq_flush(struct mlx5e_txqsq *sq) +{ + unsigned long exp_time = jiffies + msecs_to_jiffies(2000); + + while (time_before(jiffies, exp_time)) { + if (sq->cc == sq->pc) + return 0; + + msleep(20); + } + + netdev_err(sq->channel->netdev, + "Wait for SQ 0x%x flush timeout (sq cc = 0x%x, sq pc = 0x%x)\n", + sq->sqn, sq->cc, sq->pc); + + return -ETIMEDOUT; +} + +static void mlx5e_reset_txqsq_cc_pc(struct mlx5e_txqsq *sq) +{ + WARN_ONCE(sq->cc != sq->pc, + "SQ 0x%x: cc (0x%x) != pc (0x%x)\n", + sq->sqn, sq->cc, sq->pc); + sq->cc = 0; + sq->dma_fifo_cc = 0; + sq->pc = 0; +} + +static int mlx5e_sq_to_ready(struct mlx5e_txqsq *sq, int curr_state) +{ + struct mlx5_core_dev *mdev = sq->channel->mdev; + struct net_device *dev = sq->channel->netdev; + struct mlx5e_modify_sq_param msp = {0}; + int err; + + msp.curr_state = curr_state; + msp.next_state = MLX5_SQC_STATE_RST; + + err = mlx5e_modify_sq(mdev, sq->sqn, &msp); + if (err) { + netdev_err(dev, "Failed to move sq 0x%x to reset\n", sq->sqn); + return err; + } + + memset(&msp, 0, sizeof(msp)); + msp.curr_state = MLX5_SQC_STATE_RST; + msp.next_state = MLX5_SQC_STATE_RDY; + + err = mlx5e_modify_sq(mdev, sq->sqn, &msp); + if (err) { + netdev_err(dev, "Failed to move sq 0x%x to ready\n", sq->sqn); + return err; + } + + return 0; +} + +static int mlx5e_tx_reporter_err_cqe_recover(struct mlx5e_txqsq *sq) +{ + struct mlx5_core_dev *mdev = sq->channel->mdev; + struct net_device *dev = sq->channel->netdev; + u8 state; + int err; + + if (!test_bit(MLX5E_SQ_STATE_RECOVERING, &sq->state)) + return 0; + + err = mlx5_core_query_sq_state(mdev, sq->sqn, &state); + if (err) { + netdev_err(dev, "Failed to query SQ 0x%x state. err = %d\n", + sq->sqn, err); + return err; + } + + if (state != MLX5_RQC_STATE_ERR) { + netdev_err(dev, "SQ 0x%x not in ERROR state\n", sq->sqn); + return -EINVAL; + } + + mlx5e_tx_disable_queue(sq->txq); + + err = mlx5e_wait_for_sq_flush(sq); + if (err) + return err; + + /* At this point, no new packets will arrive from the stack as TXQ is + * marked with QUEUE_STATE_DRV_XOFF. In addition, NAPI cleared all + * pending WQEs. SQ can safely reset the SQ. + */ + + err = mlx5e_sq_to_ready(sq, state); + if (err) + return err; + + mlx5e_reset_txqsq_cc_pc(sq); + sq->stats->recover++; + mlx5e_activate_txqsq(sq); + + return 0; +} + +void mlx5e_tx_reporter_err_cqe(struct mlx5e_txqsq *sq) +{ + char err_str[MLX5E_TX_REPORTER_PER_SQ_MAX_LEN]; + struct mlx5e_tx_err_ctx err_ctx = {0}; + + err_ctx.sq = sq; + err_ctx.recover = mlx5e_tx_reporter_err_cqe_recover; + sprintf(err_str, "ERR CQE on SQ: 0x%x", sq->sqn); + + devlink_health_report(sq->channel->priv->tx_reporter, err_str, + &err_ctx); +} + +/* state lock cannot be grabbed within this function. + * It can cause a dead lock or a read-after-free. + */ +int mlx5e_tx_reporter_recover_from_ctx(struct mlx5e_tx_err_ctx *err_ctx) +{ + return err_ctx->recover(err_ctx->sq); +} + +static int mlx5e_tx_reporter_recover_all(struct mlx5e_priv *priv) +{ + int err; + + mutex_lock(&priv->state_lock); + mlx5e_close_locked(priv->netdev); + err = mlx5e_open_locked(priv->netdev); + mutex_unlock(&priv->state_lock); + + return err; +} + +static int mlx5e_tx_reporter_recover(struct devlink_health_reporter *reporter, + void *context) +{ + struct mlx5e_priv *priv = devlink_health_reporter_priv(reporter); + struct mlx5e_tx_err_ctx *err_ctx = context; + + return err_ctx ? mlx5e_tx_reporter_recover_from_ctx(err_ctx) : + mlx5e_tx_reporter_recover_all(priv); +} + +static int +mlx5e_tx_reporter_build_diagnose_output(struct devlink_health_buffer *buffer, + u32 sqn, u8 state, u8 stopped) +{ + int err, i; + int nest = 0; + char name[20]; + + err = devlink_health_buffer_nest_start(buffer, + DEVLINK_ATTR_HEALTH_BUFFER_OBJECT); + if (err) + goto buffer_error; + nest++; + + err = devlink_health_buffer_nest_start(buffer, + DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_PAIR); + if (err) + goto buffer_error; + nest++; + + sprintf(name, "SQ 0x%x", sqn); + err = devlink_health_buffer_put_object_name(buffer, name); + if (err) + goto buffer_error; + + err = devlink_health_buffer_nest_start(buffer, + DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE); + if (err) + goto buffer_error; + nest++; + + err = devlink_health_buffer_nest_start(buffer, + DEVLINK_ATTR_HEALTH_BUFFER_OBJECT); + if (err) + goto buffer_error; + nest++; + + err = devlink_health_buffer_nest_start(buffer, + DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_PAIR); + if (err) + goto buffer_error; + nest++; + + err = devlink_health_buffer_put_object_name(buffer, "HW state"); + if (err) + goto buffer_error; + + err = devlink_health_buffer_nest_start(buffer, + DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE); + if (err) + goto buffer_error; + nest++; + + err = devlink_health_buffer_put_value_u8(buffer, state); + if (err) + goto buffer_error; + + devlink_health_buffer_nest_end(buffer); /* DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE */ + nest--; + + devlink_health_buffer_nest_end(buffer); /* DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_PAIR */ + nest--; + + err = devlink_health_buffer_nest_start(buffer, + DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_PAIR); + if (err) + goto buffer_error; + nest++; + + err = devlink_health_buffer_put_object_name(buffer, "stopped"); + if (err) + goto buffer_error; + + err = devlink_health_buffer_nest_start(buffer, + DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE); + if (err) + goto buffer_error; + nest++; + + err = devlink_health_buffer_put_value_u8(buffer, stopped); + if (err) + goto buffer_error; + + for (i = 0; i < nest; i++) + devlink_health_buffer_nest_end(buffer); + + return 0; + +buffer_error: + for (i = 0; i < nest; i++) + devlink_health_buffer_nest_cancel(buffer); + return err; +} + +static int mlx5e_tx_reporter_diagnose(struct devlink_health_reporter *reporter, + struct devlink_health_buffer **buffers_array, + unsigned int buffer_size, + unsigned int num_buffers) +{ + struct mlx5e_priv *priv = devlink_health_reporter_priv(reporter); + unsigned int buff = 0; + int i = 0, err = 0; + + if (buffer_size < MLX5E_TX_REPORTER_PER_SQ_MAX_LEN) + return -ENOMEM; + + mutex_lock(&priv->state_lock); + + if (!test_bit(MLX5E_STATE_OPENED, &priv->state)) { + mutex_unlock(&priv->state_lock); + return 0; + } + + while (i < priv->channels.num * priv->channels.params.num_tc) { + struct mlx5e_txqsq *sq = priv->txq2sq[i]; + u8 state; + + err = mlx5_core_query_sq_state(priv->mdev, sq->sqn, &state); + if (err) + break; + + err = mlx5e_tx_reporter_build_diagnose_output(buffers_array[buff], + sq->sqn, state, + netif_xmit_stopped(sq->txq)); + if (err) { + if (++buff == num_buffers) + break; + } else { + i++; + } + } + + mutex_unlock(&priv->state_lock); + return err; +} + +static const struct devlink_health_reporter_ops mlx5_tx_reporter_ops = { + .name = "TX", + .recover = mlx5e_tx_reporter_recover, + .diagnose_size = MLX5E_MAX_NUM_CHANNELS * MLX5E_MAX_NUM_TC * + MLX5E_TX_REPORTER_PER_SQ_MAX_LEN, + .diagnose = mlx5e_tx_reporter_diagnose, + .objdump_size = 0, + .objdump = NULL, +}; + +#define MLX5_REPORTER_TX_GRACEFUL_PERIOD 500 +int mlx5e_tx_reporter_create(struct mlx5e_priv *priv) +{ + struct mlx5_core_dev *mdev = priv->mdev; + struct devlink *devlink = priv_to_devlink(mdev); + + priv->tx_reporter = + devlink_health_reporter_create(devlink, &mlx5_tx_reporter_ops, + MLX5_REPORTER_TX_GRACEFUL_PERIOD, + true, priv); + return PTR_ERR_OR_ZERO(priv->tx_reporter); +} + +void mlx5e_tx_reporter_destroy(struct mlx5e_priv *priv) +{ + devlink_health_reporter_destroy(priv->tx_reporter); +} diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c index 8cfd2ec7c0a2..4c4507384fed 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c @@ -51,6 +51,7 @@ #include "en/xdp.h" #include "lib/eq.h" #include "en/monitor_stats.h" +#include "en/reporter.h" struct mlx5e_rq_param { u32 rqc[MLX5_ST_SZ_DW(rqc)]; @@ -1160,7 +1161,7 @@ static int mlx5e_alloc_txqsq_db(struct mlx5e_txqsq *sq, int numa) return 0; } -static void mlx5e_sq_recover(struct work_struct *work); +static void mlx5e_tx_err_cqe_work(struct work_struct *recover_work); static int mlx5e_alloc_txqsq(struct mlx5e_channel *c, int txq_ix, struct mlx5e_params *params, @@ -1182,7 +1183,7 @@ static int mlx5e_alloc_txqsq(struct mlx5e_channel *c, sq->uar_map = mdev->mlx5e_res.bfreg.map; sq->min_inline_mode = params->tx_min_inline_mode; sq->stats = &c->priv->channel_stats[c->ix].sq[tc]; - INIT_WORK(&sq->recover.recover_work, mlx5e_sq_recover); + INIT_WORK(&sq->recover_work, mlx5e_tx_err_cqe_work); if (MLX5_IPSEC_DEV(c->priv->mdev)) set_bit(MLX5E_SQ_STATE_IPSEC, &sq->state); if (mlx5_accel_is_tls_device(c->priv->mdev)) @@ -1270,15 +1271,8 @@ static int mlx5e_create_sq(struct mlx5_core_dev *mdev, return err; } -struct mlx5e_modify_sq_param { - int curr_state; - int next_state; - bool rl_update; - int rl_index; -}; - -static int mlx5e_modify_sq(struct mlx5_core_dev *mdev, u32 sqn, - struct mlx5e_modify_sq_param *p) +int mlx5e_modify_sq(struct mlx5_core_dev *mdev, u32 sqn, + struct mlx5e_modify_sq_param *p) { void *in; void *sqc; @@ -1376,17 +1370,7 @@ static int mlx5e_open_txqsq(struct mlx5e_channel *c, return err; } -static void mlx5e_reset_txqsq_cc_pc(struct mlx5e_txqsq *sq) -{ - WARN_ONCE(sq->cc != sq->pc, - "SQ 0x%x: cc (0x%x) != pc (0x%x)\n", - sq->sqn, sq->cc, sq->pc); - sq->cc = 0; - sq->dma_fifo_cc = 0; - sq->pc = 0; -} - -static void mlx5e_activate_txqsq(struct mlx5e_txqsq *sq) +void mlx5e_activate_txqsq(struct mlx5e_txqsq *sq) { sq->txq = netdev_get_tx_queue(sq->channel->netdev, sq->txq_ix); clear_bit(MLX5E_SQ_STATE_RECOVERING, &sq->state); @@ -1395,7 +1379,7 @@ static void mlx5e_activate_txqsq(struct mlx5e_txqsq *sq) netif_tx_start_queue(sq->txq); } -static inline void netif_tx_disable_queue(struct netdev_queue *txq) +void mlx5e_tx_disable_queue(struct netdev_queue *txq) { __netif_tx_lock_bh(txq); netif_tx_stop_queue(txq); @@ -1411,7 +1395,7 @@ static void mlx5e_deactivate_txqsq(struct mlx5e_txqsq *sq) /* prevent netif_tx_wake_queue */ napi_synchronize(&c->napi); - netif_tx_disable_queue(sq->txq); + mlx5e_tx_disable_queue(sq->txq); /* last doorbell out, godspeed .. */ if (mlx5e_wqc_has_room_for(wq, sq->cc, sq->pc, 1)) { @@ -1431,6 +1415,7 @@ static void mlx5e_close_txqsq(struct mlx5e_txqsq *sq) struct mlx5_rate_limit rl = {0}; cancel_work_sync(&sq->dim.work); + cancel_work_sync(&sq->recover_work); mlx5e_destroy_sq(mdev, sq->sqn); if (sq->rate_limit) { rl.rate = sq->rate_limit; @@ -1440,105 +1425,15 @@ static void mlx5e_close_txqsq(struct mlx5e_txqsq *sq) mlx5e_free_txqsq(sq); } -static int mlx5e_wait_for_sq_flush(struct mlx5e_txqsq *sq) +static void mlx5e_tx_err_cqe_work(struct work_struct *recover_work) { - unsigned long exp_time = jiffies + msecs_to_jiffies(2000); - - while (time_before(jiffies, exp_time)) { - if (sq->cc == sq->pc) - return 0; - - msleep(20); - } - - netdev_err(sq->channel->netdev, - "Wait for SQ 0x%x flush timeout (sq cc = 0x%x, sq pc = 0x%x)\n", - sq->sqn, sq->cc, sq->pc); - - return -ETIMEDOUT; -} - -static int mlx5e_sq_to_ready(struct mlx5e_txqsq *sq, int curr_state) -{ - struct mlx5_core_dev *mdev = sq->channel->mdev; - struct net_device *dev = sq->channel->netdev; - struct mlx5e_modify_sq_param msp = {0}; - int err; - - msp.curr_state = curr_state; - msp.next_state = MLX5_SQC_STATE_RST; - - err = mlx5e_modify_sq(mdev, sq->sqn, &msp); - if (err) { - netdev_err(dev, "Failed to move sq 0x%x to reset\n", sq->sqn); - return err; - } - - memset(&msp, 0, sizeof(msp)); - msp.curr_state = MLX5_SQC_STATE_RST; - msp.next_state = MLX5_SQC_STATE_RDY; + struct mlx5e_txqsq *sq = container_of(recover_work, struct mlx5e_txqsq, + recover_work); - err = mlx5e_modify_sq(mdev, sq->sqn, &msp); - if (err) { - netdev_err(dev, "Failed to move sq 0x%x to ready\n", sq->sqn); - return err; - } - - return 0; -} - -static void mlx5e_sq_recover(struct work_struct *work) -{ - struct mlx5e_txqsq_recover *recover = - container_of(work, struct mlx5e_txqsq_recover, - recover_work); - struct mlx5e_txqsq *sq = container_of(recover, struct mlx5e_txqsq, - recover); - struct mlx5_core_dev *mdev = sq->channel->mdev; - struct net_device *dev = sq->channel->netdev; - u8 state; - int err; - - err = mlx5_core_query_sq_state(mdev, sq->sqn, &state); - if (err) { - netdev_err(dev, "Failed to query SQ 0x%x state. err = %d\n", - sq->sqn, err); - return; - } - - if (state != MLX5_RQC_STATE_ERR) { - netdev_err(dev, "SQ 0x%x not in ERROR state\n", sq->sqn); - return; - } - - netif_tx_disable_queue(sq->txq); - - if (mlx5e_wait_for_sq_flush(sq)) - return; - - /* If the interval between two consecutive recovers per SQ is too - * short, don't recover to avoid infinite loop of ERR_CQE -> recover. - * If we reached this state, there is probably a bug that needs to be - * fixed. let's keep the queue close and let tx timeout cleanup. - */ - if (jiffies_to_msecs(jiffies - recover->last_recover) < - MLX5E_SQ_RECOVER_MIN_INTERVAL) { - netdev_err(dev, "Recover SQ 0x%x canceled, too many error CQEs\n", - sq->sqn); - return; - } - - /* At this point, no new packets will arrive from the stack as TXQ is - * marked with QUEUE_STATE_DRV_XOFF. In addition, NAPI cleared all - * pending WQEs. SQ can safely reset the SQ. - */ - if (mlx5e_sq_to_ready(sq, state)) + if (!sq->channel->priv->tx_reporter) return; - mlx5e_reset_txqsq_cc_pc(sq); - sq->stats->recover++; - recover->last_recover = jiffies; - mlx5e_activate_txqsq(sq); + mlx5e_tx_reporter_err_cqe(sq); } static int mlx5e_open_icosq(struct mlx5e_channel *c, @@ -3207,6 +3102,7 @@ static void mlx5e_cleanup_nic_tx(struct mlx5e_priv *priv) { int tc; + mlx5e_tx_reporter_destroy(priv); for (tc = 0; tc < priv->profile->max_tc; tc++) mlx5e_destroy_tis(priv->mdev, priv->tisn[tc]); } @@ -4908,6 +4804,7 @@ static int mlx5e_init_nic_tx(struct mlx5e_priv *priv) #ifdef CONFIG_MLX5_CORE_EN_DCB mlx5e_dcbnl_initialize(priv); #endif + mlx5e_tx_reporter_create(priv); return 0; } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c index 598ad7e4d5c9..a8e052a5ce36 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tx.c @@ -514,7 +514,7 @@ bool mlx5e_poll_tx_cq(struct mlx5e_cq *cq, int napi_budget) mlx5e_dump_error_cqe(sq, (struct mlx5_err_cqe *)cqe); queue_work(cq->channel->priv->wq, - &sq->recover.recover_work); + &sq->recover_work); } stats->cqe_err++; } From patchwork Mon Dec 31 14:32:04 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eran Ben Elisha X-Patchwork-Id: 1019582 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43T0CX6v6Rz9sBQ for ; Tue, 1 Jan 2019 01:32:56 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727358AbeLaOcz (ORCPT ); Mon, 31 Dec 2018 09:32:55 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:52233 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726383AbeLaOc0 (ORCPT ); Mon, 31 Dec 2018 09:32:26 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from eranbe@mellanox.com) with ESMTPS (AES256-SHA encrypted); 31 Dec 2018 16:32:21 +0200 Received: from dev-l-vrt-198.mtl.labs.mlnx (dev-l-vrt-198.mtl.labs.mlnx [10.134.198.1]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id wBVEWLYw017529; Mon, 31 Dec 2018 16:32:21 +0200 From: Eran Ben Elisha To: netdev@vger.kernel.org, "David S. Miller" , Jiri Pirko Cc: Moshe Shemesh , Aya Levin , Eran Ben Elisha , Tal Alon , Ariel Almog Subject: [PATCH RFC net-next 10/19] net/mlx5e: Add TX timeout support for mlx5e TX reporter Date: Mon, 31 Dec 2018 16:32:04 +0200 Message-Id: <1546266733-9512-11-git-send-email-eranbe@mellanox.com> X-Mailer: git-send-email 1.8.4.3 In-Reply-To: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> References: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org With this patch, ndo_tx_timeout callback will be redirected to the TX reporter in order to detect a TX timeout error and report it to the devlink health. (The watchdog detects TX timeouts, but the driver verify the issue still exists before launching any recover method). In addition, recover from TX timeout in case of lost interrupt was added to the TX reporter recover method. The TX timeout recover from lost interrupt is not a new feature in the driver, this patch re-organize the functionality and move it to the TX reporter recovery flow. TX timeout example: (with auto_recover set to false, if set to true, the manual recover and diagnose sections are irrelevant) $cat /sys/kernel/debug/tracing/trace ... devlink_health_report: bus_name=pci dev_name=0000:00:09.0 driver_name=mlx5_core reporter_name=TX: TX timeout on queue: 0, SQ: 0xd8a, CQ: 0x406, SQ Cons: 0x2 SQ Prod: 0x2, usecs since last trans: 13972000 $devlink health diagnose pci/0000:00:09 reporter TX SQ 0xd8a: HW state: 1, stopped: 1 SQ 0xe44: HW state: 1, stopped: 0 SQ 0xeb4: HW state: 1, stopped: 0 SQ 0xf1f: HW state: 1, stopped: 0 SQ 0xf80: HW state: 1, stopped: 0 SQ 0xfe5: HW state: 1, stopped: 0 $devlink health recover pci/0000:00:09 reporter TX $devlink health show pci/0000:00:09.0: name TX state healthy #err 1 #recover 1 last_objdump_ts N/A objdump_available false attributes: grace_period 500 auto_recover false Signed-off-by: Eran Ben Elisha --- .../ethernet/mellanox/mlx5/core/en/reporter.h | 1 + .../mellanox/mlx5/core/en/reporter_tx.c | 35 +++++++++++++ .../net/ethernet/mellanox/mlx5/core/en_main.c | 51 +++---------------- 3 files changed, 43 insertions(+), 44 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/reporter.h b/drivers/net/ethernet/mellanox/mlx5/core/en/reporter.h index 74083e120ab3..2335c5b48820 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/reporter.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/reporter.h @@ -10,5 +10,6 @@ int mlx5e_tx_reporter_create(struct mlx5e_priv *priv); void mlx5e_tx_reporter_destroy(struct mlx5e_priv *priv); void mlx5e_tx_reporter_err_cqe(struct mlx5e_txqsq *sq); +void mlx5e_tx_reporter_timeout(struct mlx5e_txqsq *sq); #endif diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_tx.c b/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_tx.c index b88539528f12..5cbb0a611c5b 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_tx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/reporter_tx.c @@ -126,6 +126,41 @@ void mlx5e_tx_reporter_err_cqe(struct mlx5e_txqsq *sq) &err_ctx); } +static int mlx5e_tx_reporter_timeout_recover(struct mlx5e_txqsq *sq) +{ + struct mlx5_eq_comp *eq = sq->cq.mcq.eq; + u32 eqe_count; + + netdev_err(sq->channel->netdev, "EQ 0x%x: Cons = 0x%x, irqn = 0x%x\n", + eq->core.eqn, eq->core.cons_index, eq->core.irqn); + + eqe_count = mlx5_eq_poll_irq_disabled(eq); + if (!eqe_count) { + clear_bit(MLX5E_SQ_STATE_ENABLED, &sq->state); + return 1; + } + + netdev_err(sq->channel->netdev, "Recover %d eqes on EQ 0x%x\n", + eqe_count, eq->core.eqn); + sq->channel->stats->eq_rearm++; + return 0; +} + +void mlx5e_tx_reporter_timeout(struct mlx5e_txqsq *sq) +{ + struct mlx5e_tx_err_ctx err_ctx; + char err_str[MLX5E_TX_REPORTER_PER_SQ_MAX_LEN]; + + err_ctx.sq = sq; + err_ctx.recover = mlx5e_tx_reporter_timeout_recover; + sprintf(err_str, + "TX timeout on queue: %d, SQ: 0x%x, CQ: 0x%x, SQ Cons: 0x%x SQ Prod: 0x%x, usecs since last trans: %u\n", + sq->channel->ix, sq->sqn, sq->cq.mcq.cqn, sq->cc, sq->pc, + jiffies_to_usecs(jiffies - sq->txq->trans_start)); + devlink_health_report(sq->channel->priv->tx_reporter, err_str, + &err_ctx); +} + /* state lock cannot be grabbed within this function. * It can cause a dead lock or a read-after-free. */ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c index 4c4507384fed..dee0c8f3d4e9 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c @@ -4074,31 +4074,14 @@ netdev_features_t mlx5e_features_check(struct sk_buff *skb, return features; } -static bool mlx5e_tx_timeout_eq_recover(struct net_device *dev, - struct mlx5e_txqsq *sq) -{ - struct mlx5_eq_comp *eq = sq->cq.mcq.eq; - u32 eqe_count; - - netdev_err(dev, "EQ 0x%x: Cons = 0x%x, irqn = 0x%x\n", - eq->core.eqn, eq->core.cons_index, eq->core.irqn); - - eqe_count = mlx5_eq_poll_irq_disabled(eq); - if (!eqe_count) - return false; - - netdev_err(dev, "Recover %d eqes on EQ 0x%x\n", eqe_count, eq->core.eqn); - sq->channel->stats->eq_rearm++; - return true; -} - static void mlx5e_tx_timeout_work(struct work_struct *work) { struct mlx5e_priv *priv = container_of(work, struct mlx5e_priv, tx_timeout_work); - struct net_device *dev = priv->netdev; - bool reopen_channels = false; - int i, err; + int i; + + if (!priv->tx_reporter) + return; rtnl_lock(); mutex_lock(&priv->state_lock); @@ -4107,36 +4090,16 @@ static void mlx5e_tx_timeout_work(struct work_struct *work) goto unlock; for (i = 0; i < priv->channels.num * priv->channels.params.num_tc; i++) { - struct netdev_queue *dev_queue = netdev_get_tx_queue(dev, i); + struct netdev_queue *dev_queue = + netdev_get_tx_queue(priv->netdev, i); struct mlx5e_txqsq *sq = priv->txq2sq[i]; if (!netif_xmit_stopped(dev_queue)) continue; - netdev_err(dev, - "TX timeout on queue: %d, SQ: 0x%x, CQ: 0x%x, SQ Cons: 0x%x SQ Prod: 0x%x, usecs since last trans: %u\n", - i, sq->sqn, sq->cq.mcq.cqn, sq->cc, sq->pc, - jiffies_to_usecs(jiffies - dev_queue->trans_start)); - - /* If we recover a lost interrupt, most likely TX timeout will - * be resolved, skip reopening channels - */ - if (!mlx5e_tx_timeout_eq_recover(dev, sq)) { - clear_bit(MLX5E_SQ_STATE_ENABLED, &sq->state); - reopen_channels = true; - } + mlx5e_tx_reporter_timeout(sq); } - if (!reopen_channels) - goto unlock; - - mlx5e_close_locked(dev); - err = mlx5e_open_locked(dev); - if (err) - netdev_err(priv->netdev, - "mlx5e_open_locked failed recovering from a tx_timeout, err(%d).\n", - err); - unlock: mutex_unlock(&priv->state_lock); rtnl_unlock(); From patchwork Mon Dec 31 14:32:05 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eran Ben Elisha X-Patchwork-Id: 1019566 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43T0CJ4kx7z9s55 for ; Tue, 1 Jan 2019 01:32:44 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727119AbeLaOc0 (ORCPT ); Mon, 31 Dec 2018 09:32:26 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:52259 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726283AbeLaOcZ (ORCPT ); Mon, 31 Dec 2018 09:32:25 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from eranbe@mellanox.com) with ESMTPS (AES256-SHA encrypted); 31 Dec 2018 16:32:21 +0200 Received: from dev-l-vrt-198.mtl.labs.mlnx (dev-l-vrt-198.mtl.labs.mlnx [10.134.198.1]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id wBVEWLYx017529; Mon, 31 Dec 2018 16:32:21 +0200 From: Eran Ben Elisha To: netdev@vger.kernel.org, "David S. Miller" , Jiri Pirko Cc: Moshe Shemesh , Aya Levin , Eran Ben Elisha , Tal Alon , Ariel Almog Subject: [PATCH RFC net-next 11/19] net/mlx5: Move all devlink related functions calls to devlink.c Date: Mon, 31 Dec 2018 16:32:05 +0200 Message-Id: <1546266733-9512-12-git-send-email-eranbe@mellanox.com> X-Mailer: git-send-email 1.8.4.3 In-Reply-To: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> References: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Centralize all devlink related callbacks in one file. In the downstream patch, some more functionality will be added, this patch is preparing the driver infrastructure for it. Currently, move devlink un/register functions calls into this file. Signed-off-by: Eran Ben Elisha Reviewed-by: Moshe Shemesh --- drivers/net/ethernet/mellanox/mlx5/core/Makefile | 2 +- drivers/net/ethernet/mellanox/mlx5/core/devlink.c | 14 ++++++++++++++ drivers/net/ethernet/mellanox/mlx5/core/devlink.h | 12 ++++++++++++ drivers/net/ethernet/mellanox/mlx5/core/main.c | 5 +++-- 4 files changed, 30 insertions(+), 3 deletions(-) create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/devlink.c create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/devlink.h diff --git a/drivers/net/ethernet/mellanox/mlx5/core/Makefile b/drivers/net/ethernet/mellanox/mlx5/core/Makefile index 6bb2a860b15b..2049f332c7e3 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/Makefile +++ b/drivers/net/ethernet/mellanox/mlx5/core/Makefile @@ -15,7 +15,7 @@ mlx5_core-y := main.o cmd.o debugfs.o fw.o eq.o uar.o pagealloc.o \ health.o mcg.o cq.o alloc.o qp.o port.o mr.o pd.o \ mad.o transobj.o vport.o sriov.o fs_cmd.o fs_core.o \ fs_counters.o rl.o lag.o dev.o events.o wq.o lib/gid.o \ - lib/devcom.o diag/fs_tracepoint.o diag/fw_tracer.o + lib/devcom.o diag/fs_tracepoint.o diag/fw_tracer.o devlink.o # # Netdev basic diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c new file mode 100644 index 000000000000..2daf686bcc98 --- /dev/null +++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c @@ -0,0 +1,14 @@ +// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB +/* Copyright (c) 2018, Mellanox Technologies inc. All rights reserved. */ + +#include + +int mlx5_devlink_register(struct devlink *devlink, struct device *dev) +{ + return devlink_register(devlink, dev); +} + +void mlx5_devlink_unregister(struct devlink *devlink) +{ + devlink_unregister(devlink); +} diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.h b/drivers/net/ethernet/mellanox/mlx5/core/devlink.h new file mode 100644 index 000000000000..455bfa4e89c0 --- /dev/null +++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.h @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ +/* Copyright (c) 2018, Mellanox Technologies inc. All rights reserved. */ + +#ifndef __MLX5_DEVLINK_H__ +#define __MLX5_DEVLINK_H__ + +#include + +int mlx5_devlink_register(struct devlink *devlink, struct device *dev); +void mlx5_devlink_unregister(struct devlink *devlink); + +#endif /* __MLX5_DEVLINK_H__ */ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c index 77896c11f6f3..bd50d4adc6bf 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c @@ -56,6 +56,7 @@ #include "fs_core.h" #include "lib/mpfs.h" #include "eswitch.h" +#include "devlink.h" #include "lib/mlx5.h" #include "fpga/core.h" #include "fpga/ipsec.h" @@ -1204,7 +1205,7 @@ static int init_one(struct pci_dev *pdev, request_module_nowait(MLX5_IB_MOD); - err = devlink_register(devlink, &pdev->dev); + err = mlx5_devlink_register(devlink, &pdev->dev); if (err) goto clean_load; @@ -1231,7 +1232,7 @@ static void remove_one(struct pci_dev *pdev) struct devlink *devlink = priv_to_devlink(dev); struct mlx5_priv *priv = &dev->priv; - devlink_unregister(devlink); + mlx5_devlink_unregister(devlink); mlx5_unregister_device(dev); if (mlx5_unload_one(dev, priv, true)) { From patchwork Mon Dec 31 14:32:06 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eran Ben Elisha X-Patchwork-Id: 1019569 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43T0CL5kQWz9sBQ for ; Tue, 1 Jan 2019 01:32:46 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727269AbeLaOc1 (ORCPT ); Mon, 31 Dec 2018 09:32:27 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:52245 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726127AbeLaOc1 (ORCPT ); Mon, 31 Dec 2018 09:32:27 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from eranbe@mellanox.com) with ESMTPS (AES256-SHA encrypted); 31 Dec 2018 16:32:21 +0200 Received: from dev-l-vrt-198.mtl.labs.mlnx (dev-l-vrt-198.mtl.labs.mlnx [10.134.198.1]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id wBVEWLZ0017529; Mon, 31 Dec 2018 16:32:21 +0200 From: Eran Ben Elisha To: netdev@vger.kernel.org, "David S. Miller" , Jiri Pirko Cc: Moshe Shemesh , Aya Levin , Eran Ben Elisha , Tal Alon , Ariel Almog Subject: [PATCH RFC net-next 12/19] net/mlx5: Refactor print health info Date: Mon, 31 Dec 2018 16:32:06 +0200 Message-Id: <1546266733-9512-13-git-send-email-eranbe@mellanox.com> X-Mailer: git-send-email 1.8.4.3 In-Reply-To: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> References: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Moshe Shemesh Refactor print health info code, split to two functions: 1. mlx5_get_health_info() - writes the health info into a buffer. 2. mlx5_print_health_info() - prints the health info to kernel log. This refactoring is done to enable using the health info data by devlink health reporter diagnose() in the downstream patch. Signed-off-by: Moshe Shemesh --- .../net/ethernet/mellanox/mlx5/core/health.c | 44 +++++++++++++------ .../ethernet/mellanox/mlx5/core/mlx5_core.h | 7 +++ include/linux/mlx5/device.h | 1 + include/linux/mlx5/mlx5_ifc.h | 4 +- 4 files changed, 42 insertions(+), 14 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/health.c b/drivers/net/ethernet/mellanox/mlx5/core/health.c index 196c07383082..2fa78edde1fe 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/health.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/health.c @@ -215,7 +215,8 @@ static const char *hsynd_str(u8 synd) } } -static void print_health_info(struct mlx5_core_dev *dev) +void mlx5_get_health_info(struct mlx5_core_dev *dev, u8 *synd, + char (*lines_buf)[HEALTH_INFO_MAX_LINE]) { struct mlx5_core_health *health = &dev->priv.health; struct health_buffer __iomem *h = health->health; @@ -223,23 +224,40 @@ static void print_health_info(struct mlx5_core_dev *dev) u32 fw; int i; + *synd = ioread8(&h->synd); /* If the syndrome is 0, the device is OK and no need to print buffer */ - if (!ioread8(&h->synd)) + if (!synd) return; for (i = 0; i < ARRAY_SIZE(h->assert_var); i++) - dev_err(&dev->pdev->dev, "assert_var[%d] 0x%08x\n", i, ioread32be(h->assert_var + i)); + sprintf(lines_buf[i], "assert_var[%d] 0x%08x\n", i, ioread32be(h->assert_var + i)); - dev_err(&dev->pdev->dev, "assert_exit_ptr 0x%08x\n", ioread32be(&h->assert_exit_ptr)); - dev_err(&dev->pdev->dev, "assert_callra 0x%08x\n", ioread32be(&h->assert_callra)); + sprintf(lines_buf[i++], "assert_exit_ptr 0x%08x\n", ioread32be(&h->assert_exit_ptr)); + sprintf(lines_buf[i++], "assert_callra 0x%08x\n", ioread32be(&h->assert_callra)); sprintf(fw_str, "%d.%d.%d", fw_rev_maj(dev), fw_rev_min(dev), fw_rev_sub(dev)); - dev_err(&dev->pdev->dev, "fw_ver %s\n", fw_str); - dev_err(&dev->pdev->dev, "hw_id 0x%08x\n", ioread32be(&h->hw_id)); - dev_err(&dev->pdev->dev, "irisc_index %d\n", ioread8(&h->irisc_index)); - dev_err(&dev->pdev->dev, "synd 0x%x: %s\n", ioread8(&h->synd), hsynd_str(ioread8(&h->synd))); - dev_err(&dev->pdev->dev, "ext_synd 0x%04x\n", ioread16be(&h->ext_synd)); + sprintf(lines_buf[i++], "fw_ver %s\n", fw_str); + sprintf(lines_buf[i++], "hw_id 0x%08x\n", ioread32be(&h->hw_id)); + sprintf(lines_buf[i++], "irisc_index %d\n", ioread8(&h->irisc_index)); + sprintf(lines_buf[i++], "synd 0x%x: %s\n", ioread8(&h->synd), hsynd_str(ioread8(&h->synd))); + sprintf(lines_buf[i++], "ext_synd 0x%04x\n", ioread16be(&h->ext_synd)); fw = ioread32be(&h->fw_ver); - dev_err(&dev->pdev->dev, "raw fw_ver 0x%08x\n", fw); + sprintf(lines_buf[i], "raw fw_ver 0x%08x\n", fw); +} + +static void mlx5_print_health_info(struct mlx5_core_dev *dev) +{ + char lines_buf[HEALTH_INFO_LINES][HEALTH_INFO_MAX_LINE] = {}; + u8 synd; + int i; + + mlx5_get_health_info(dev, &synd, lines_buf); + + if (!synd) + return; + + for (i = 0; i < HEALTH_INFO_LINES; i++) + dev_err(&dev->pdev->dev, lines_buf[i]); + } static unsigned long get_next_poll_jiffies(void) @@ -285,12 +303,12 @@ static void poll_health(struct timer_list *t) health->prev = count; if (health->miss_counter == MAX_MISSES) { dev_err(&dev->pdev->dev, "device's health compromised - reached miss count\n"); - print_health_info(dev); + mlx5_print_health_info(dev); } if (in_fatal(dev) && !health->sick) { health->sick = true; - print_health_info(dev); + mlx5_print_health_info(dev); mlx5_trigger_health_work(dev); } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h index c68dcea5985b..ea53195c89a6 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h @@ -199,4 +199,11 @@ enum { u8 mlx5_get_nic_state(struct mlx5_core_dev *dev); void mlx5_set_nic_state(struct mlx5_core_dev *dev, u8 state); + +#define HEALTH_INFO_MAX_LINE 80 +#define HEALTH_INFO_LINES (MLX5_FLD_SZ_DW(health_buffer, assert_var) + 8) +#define HEALTH_INFO_MAX_BUFF (HEALTH_INFO_MAX_LINE * HEALTH_INFO_LINES) +void mlx5_get_health_info(struct mlx5_core_dev *dev, u8 *synd, + char (*lines_buf)[HEALTH_INFO_MAX_LINE]); + #endif /* __MLX5_CORE_H__ */ diff --git a/include/linux/mlx5/device.h b/include/linux/mlx5/device.h index 8c4a820bd4c1..bb68ab6dffe0 100644 --- a/include/linux/mlx5/device.h +++ b/include/linux/mlx5/device.h @@ -61,6 +61,7 @@ #define __mlx5_st_sz_bits(typ) sizeof(struct mlx5_ifc_##typ##_bits) #define MLX5_FLD_SZ_BYTES(typ, fld) (__mlx5_bit_sz(typ, fld) / 8) +#define MLX5_FLD_SZ_DW(typ, fld) (__mlx5_bit_sz(typ, fld) / 32) #define MLX5_ST_SZ_BYTES(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 8) #define MLX5_ST_SZ_DW(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 32) #define MLX5_ST_SZ_QW(typ) (sizeof(struct mlx5_ifc_##typ##_bits) / 64) diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h index 821b751485fb..505a5f6cecdf 100644 --- a/include/linux/mlx5/mlx5_ifc.h +++ b/include/linux/mlx5/mlx5_ifc.h @@ -3360,7 +3360,9 @@ union mlx5_ifc_event_auto_bits { }; struct mlx5_ifc_health_buffer_bits { - u8 reserved_at_0[0x100]; + u8 assert_var[0xa0]; + + u8 reserved_at_a0[0x60]; u8 assert_existptr[0x20]; From patchwork Mon Dec 31 14:32:07 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eran Ben Elisha X-Patchwork-Id: 1019568 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43T0CL17j4z9s55 for ; Tue, 1 Jan 2019 01:32:46 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727199AbeLaOc1 (ORCPT ); Mon, 31 Dec 2018 09:32:27 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:52260 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726332AbeLaOcZ (ORCPT ); Mon, 31 Dec 2018 09:32:25 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from eranbe@mellanox.com) with ESMTPS (AES256-SHA encrypted); 31 Dec 2018 16:32:21 +0200 Received: from dev-l-vrt-198.mtl.labs.mlnx (dev-l-vrt-198.mtl.labs.mlnx [10.134.198.1]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id wBVEWLZ1017529; Mon, 31 Dec 2018 16:32:21 +0200 From: Eran Ben Elisha To: netdev@vger.kernel.org, "David S. Miller" , Jiri Pirko Cc: Moshe Shemesh , Aya Levin , Eran Ben Elisha , Tal Alon , Ariel Almog Subject: [PATCH RFC net-next 13/19] net/mlx5: Create FW devlink_health_reporter Date: Mon, 31 Dec 2018 16:32:07 +0200 Message-Id: <1546266733-9512-14-git-send-email-eranbe@mellanox.com> X-Mailer: git-send-email 1.8.4.3 In-Reply-To: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> References: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Moshe Shemesh Create mlx5_devlink_health_reporter for FW reporter. The FW reporter implements devlink_health_reporter diagnose callback. Signed-off-by: Moshe Shemesh --- .../net/ethernet/mellanox/mlx5/core/devlink.c | 115 ++++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/devlink.h | 8 ++ .../net/ethernet/mellanox/mlx5/core/main.c | 5 + include/linux/mlx5/driver.h | 1 + 4 files changed, 129 insertions(+) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c index 2daf686bcc98..4ec5d092a332 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c @@ -2,6 +2,121 @@ /* Copyright (c) 2018, Mellanox Technologies inc. All rights reserved. */ #include +#include "mlx5_core.h" + +static int +mlx5_devlink_health_buffer_fill_syndrom(struct devlink_health_buffer *dh_buffer, + u8 syndrom) +{ + int err; + + err = devlink_health_buffer_nest_start(dh_buffer, + DEVLINK_ATTR_HEALTH_BUFFER_OBJECT); + if (err) + return err; + err = devlink_health_buffer_nest_start(dh_buffer, + DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_PAIR); + if (err) + return err; + err = devlink_health_buffer_put_object_name(dh_buffer, "Syndrom"); + if (err) + return err; + err = devlink_health_buffer_nest_start(dh_buffer, + DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE); + if (err) + return err; + err = devlink_health_buffer_put_value_u8(dh_buffer, syndrom); + if (err) + return err; + devlink_health_buffer_nest_end(dh_buffer); + devlink_health_buffer_nest_end(dh_buffer); + devlink_health_buffer_nest_end(dh_buffer); + + return 0; +} + +static int +mlx5_fw_reporter_diagnose(struct devlink_health_reporter *reporter, + struct devlink_health_buffer **buffers_array, + unsigned int buff_size, unsigned int num_buffers) +{ + struct mlx5_core_dev *dev = devlink_health_reporter_priv(reporter); + char lines_buf[HEALTH_INFO_LINES][HEALTH_INFO_MAX_LINE] = {}; + struct devlink_health_buffer *buffer; + u8 synd; + int err; + int i; + + if (!buffers_array || buff_size < HEALTH_INFO_MAX_BUFF || + num_buffers < 1 || !buffers_array[0]) + return -EINVAL; + + buffer = buffers_array[0]; + mlx5_get_health_info(dev, &synd, lines_buf); + mlx5_devlink_health_buffer_fill_syndrom(buffer, synd); + + if (!synd) + return 0; + + err = devlink_health_buffer_nest_start(buffer, + DEVLINK_ATTR_HEALTH_BUFFER_OBJECT); + if (err) + return err; + err = devlink_health_buffer_nest_start(buffer, + DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_PAIR); + if (err) + return err; + err = devlink_health_buffer_put_object_name(buffer, "diagnose data"); + if (err) + return err; + err = devlink_health_buffer_nest_start(buffer, + DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE); + if (err) + return err; + err = devlink_health_buffer_nest_start(buffer, + DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE_ARRAY); + if (err) + return err; + + for (i = 0; i < HEALTH_INFO_LINES; i++) { + err = devlink_health_buffer_nest_start(buffer, DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE); + if (err) + return err; + err = devlink_health_buffer_put_value_string(buffer, lines_buf[i]); + if (err) + return err; + devlink_health_buffer_nest_end(buffer); + } + devlink_health_buffer_nest_end(buffer); /* DEVLINK_HEALTH_BUFFER_OBJECT_VALUE_ARRAY */ + devlink_health_buffer_nest_end(buffer); /* DEVLINK_HEALTH_BUFFER_OBJECT_VALUE */ + devlink_health_buffer_nest_end(buffer); /* DEVLINK_HEALTH_BUFFER_OBJECT_PAIR */ + devlink_health_buffer_nest_end(buffer); /* DEVLINK_HEALTH_BUFFER_OBJECT */ + + return 0; +} + +static const struct devlink_health_reporter_ops mlx5_fw_reporter_ops = { + .name = "FW", + .diagnose_size = HEALTH_INFO_MAX_BUFF, + .diagnose = mlx5_fw_reporter_diagnose, +}; + +int mlx5_fw_reporter_create(struct mlx5_core_dev *dev) +{ + struct devlink *devlink = priv_to_devlink(dev); + + dev->fw_reporter = devlink_health_reporter_create(devlink, &mlx5_fw_reporter_ops, + 0, false, dev); + return PTR_ERR_OR_ZERO(dev->fw_reporter); +} + +void mlx5_fw_reporter_destroy(struct mlx5_core_dev *dev) +{ + if (!dev->fw_reporter) + return; + + devlink_health_reporter_destroy(dev->fw_reporter); +} int mlx5_devlink_register(struct devlink *devlink, struct device *dev) { diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.h b/drivers/net/ethernet/mellanox/mlx5/core/devlink.h index 455bfa4e89c0..34f6bfed1cfb 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.h @@ -5,8 +5,16 @@ #define __MLX5_DEVLINK_H__ #include +#include "mlx5_core.h" + +struct mlx5_fw_reporter_ctx { + u8 err_synd; + int miss_counter; +}; int mlx5_devlink_register(struct devlink *devlink, struct device *dev); void mlx5_devlink_unregister(struct devlink *devlink); +int mlx5_fw_reporter_create(struct mlx5_core_dev *dev); +void mlx5_fw_reporter_destroy(struct mlx5_core_dev *dev); #endif /* __MLX5_DEVLINK_H__ */ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c index bd50d4adc6bf..ca5f4c661f6d 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c @@ -981,6 +981,10 @@ static int mlx5_load_one(struct mlx5_core_dev *dev, struct mlx5_priv *priv, goto err_fw_tracer; } + err = mlx5_fw_reporter_create(dev); + if (err) + dev_warn(&pdev->dev, "Failed to create FW reporter\n"); + err = mlx5_fpga_device_start(dev); if (err) { dev_err(&pdev->dev, "fpga device start failed %d\n", err); @@ -1113,6 +1117,7 @@ static int mlx5_unload_one(struct mlx5_core_dev *dev, struct mlx5_priv *priv, mlx5_accel_ipsec_cleanup(dev); mlx5_accel_tls_cleanup(dev); mlx5_fpga_device_stop(dev); + mlx5_fw_reporter_destroy(dev); mlx5_fw_tracer_cleanup(dev->tracer); mlx5_eq_table_destroy(dev); mlx5_pagealloc_stop(dev); diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h index 4d16ba04790e..98c17e74fd4d 100644 --- a/include/linux/mlx5/driver.h +++ b/include/linux/mlx5/driver.h @@ -677,6 +677,7 @@ struct mlx5_core_dev { struct mlx5_ib_clock_info *clock_info; struct page *clock_info_page; struct mlx5_fw_tracer *tracer; + struct devlink_health_reporter *fw_reporter; }; struct mlx5_db { From patchwork Mon Dec 31 14:32:08 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eran Ben Elisha X-Patchwork-Id: 1019577 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43T0CS1kJjz9sBQ for ; Tue, 1 Jan 2019 01:32:52 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727030AbeLaOc0 (ORCPT ); Mon, 31 Dec 2018 09:32:26 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:52273 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726603AbeLaOcZ (ORCPT ); Mon, 31 Dec 2018 09:32:25 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from eranbe@mellanox.com) with ESMTPS (AES256-SHA encrypted); 31 Dec 2018 16:32:21 +0200 Received: from dev-l-vrt-198.mtl.labs.mlnx (dev-l-vrt-198.mtl.labs.mlnx [10.134.198.1]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id wBVEWLZ2017529; Mon, 31 Dec 2018 16:32:21 +0200 From: Eran Ben Elisha To: netdev@vger.kernel.org, "David S. Miller" , Jiri Pirko Cc: Moshe Shemesh , Aya Levin , Eran Ben Elisha , Tal Alon , Ariel Almog Subject: [PATCH RFC net-next 14/19] net/mlx5: Add core dump register access functions Date: Mon, 31 Dec 2018 16:32:08 +0200 Message-Id: <1546266733-9512-15-git-send-email-eranbe@mellanox.com> X-Mailer: git-send-email 1.8.4.3 In-Reply-To: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> References: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Moshe Shemesh Add access functions to core dump register to enable trigger FW core dump. Signed-off-by: Moshe Shemesh --- .../mellanox/mlx5/core/diag/fw_tracer.c | 31 +++++++++++++++++++ include/linux/mlx5/driver.h | 1 + include/linux/mlx5/mlx5_ifc.h | 17 +++++++++- 3 files changed, 48 insertions(+), 1 deletion(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c b/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c index 6999f4486e9e..d0f8449019af 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c @@ -786,6 +786,37 @@ static void mlx5_fw_tracer_ownership_change(struct work_struct *work) mlx5_fw_tracer_start(tracer); } +static int mlx5_fw_tracer_set_core_dump_reg(struct mlx5_core_dev *dev, u32 *in, int size_in) +{ + u32 out[MLX5_ST_SZ_DW(core_dump_reg)] = {0}; + + if (!MLX5_CAP_DEBUG(dev, core_dump_general) && + !MLX5_CAP_DEBUG(dev, core_dump_qp)) + return -EOPNOTSUPP; + + return mlx5_core_access_reg(dev, in, size_in, out, sizeof(out), + MLX5_REG_CORE_DUMP, 0, 1); +} + +int mlx5_fw_tracer_trigger_core_dump_general(struct mlx5_core_dev *dev) +{ + struct mlx5_fw_tracer *tracer = dev->tracer; + u32 in[MLX5_ST_SZ_DW(core_dump_reg)] = {0}; + int err; + + if (!MLX5_CAP_DEBUG(dev, core_dump_general) || !tracer) + return -EOPNOTSUPP; + + MLX5_SET(core_dump_reg, in, core_dump_type, 0x0); + + err = mlx5_fw_tracer_set_core_dump_reg(dev, in, sizeof(in)); + if (err) + return err; + queue_work(tracer->work_queue, &tracer->handle_traces_work); + flush_workqueue(tracer->work_queue); + return 0; +} + /* Create software resources (Buffers, etc ..) */ struct mlx5_fw_tracer *mlx5_fw_tracer_create(struct mlx5_core_dev *dev) { diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h index 98c17e74fd4d..bedc9bc08963 100644 --- a/include/linux/mlx5/driver.h +++ b/include/linux/mlx5/driver.h @@ -108,6 +108,7 @@ enum { MLX5_REG_FPGA_CAP = 0x4022, MLX5_REG_FPGA_CTRL = 0x4023, MLX5_REG_FPGA_ACCESS_REG = 0x4024, + MLX5_REG_CORE_DUMP = 0x402e, MLX5_REG_PCAP = 0x5001, MLX5_REG_PMTU = 0x5003, MLX5_REG_PTYS = 0x5004, diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h index 505a5f6cecdf..d1f7d4b2e3eb 100644 --- a/include/linux/mlx5/mlx5_ifc.h +++ b/include/linux/mlx5/mlx5_ifc.h @@ -671,7 +671,9 @@ struct mlx5_ifc_qos_cap_bits { }; struct mlx5_ifc_debug_cap_bits { - u8 reserved_at_0[0x20]; + u8 core_dump_general[0x1]; + u8 core_dump_qp[0x1]; + u8 reserved_at_2[0x1e]; u8 reserved_at_20[0x2]; u8 stall_detect[0x1]; @@ -2453,6 +2455,7 @@ union mlx5_ifc_hca_cap_union_bits { struct mlx5_ifc_e_switch_cap_bits e_switch_cap; struct mlx5_ifc_vector_calc_cap_bits vector_calc_cap; struct mlx5_ifc_qos_cap_bits qos_cap; + struct mlx5_ifc_debug_cap_bits debug_cap; struct mlx5_ifc_fpga_cap_bits fpga_cap; u8 reserved_at_0[0x8000]; }; @@ -8410,6 +8413,18 @@ struct mlx5_ifc_qcam_reg_bits { u8 reserved_at_1c0[0x80]; }; +struct mlx5_ifc_core_dump_reg_bits { + u8 reserved_at_0[0x18]; + u8 core_dump_type[0x8]; + + u8 reserved_at_20[0x30]; + u8 vhca_id[0x10]; + + u8 reserved_at_60[0x8]; + u8 qpn[0x18]; + u8 reserved_at_80[0x180]; +}; + struct mlx5_ifc_pcap_reg_bits { u8 reserved_at_0[0x8]; u8 local_port[0x8]; From patchwork Mon Dec 31 14:32:09 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eran Ben Elisha X-Patchwork-Id: 1019574 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43T0CQ0l8Cz9s55 for ; Tue, 1 Jan 2019 01:32:50 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727330AbeLaOcl (ORCPT ); Mon, 31 Dec 2018 09:32:41 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:52274 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726585AbeLaOc0 (ORCPT ); Mon, 31 Dec 2018 09:32:26 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from eranbe@mellanox.com) with ESMTPS (AES256-SHA encrypted); 31 Dec 2018 16:32:21 +0200 Received: from dev-l-vrt-198.mtl.labs.mlnx (dev-l-vrt-198.mtl.labs.mlnx [10.134.198.1]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id wBVEWLZ3017529; Mon, 31 Dec 2018 16:32:21 +0200 From: Eran Ben Elisha To: netdev@vger.kernel.org, "David S. Miller" , Jiri Pirko Cc: Moshe Shemesh , Aya Levin , Eran Ben Elisha , Tal Alon , Ariel Almog Subject: [PATCH RFC net-next 15/19] net/mlx5: Add support for FW reporter objdump Date: Mon, 31 Dec 2018 16:32:09 +0200 Message-Id: <1546266733-9512-16-git-send-email-eranbe@mellanox.com> X-Mailer: git-send-email 1.8.4.3 In-Reply-To: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> References: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Moshe Shemesh Add support of objdump callback for mlx5 FW reporter. Once we trigger FW dump, the FW will write the core dump to its raw data buffer. The tracer translates the raw data to traces and save it to a buffer. Once dump is done, the saved traces data is filled as objects into the objdump buffer. Signed-off-by: Moshe Shemesh --- .../net/ethernet/mellanox/mlx5/core/devlink.c | 112 ++++++++++++++++++ .../mellanox/mlx5/core/diag/fw_tracer.c | 44 +++++++ .../mellanox/mlx5/core/diag/fw_tracer.h | 13 ++ 3 files changed, 169 insertions(+) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c index 4ec5d092a332..07bc473a8ebb 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c @@ -3,6 +3,7 @@ #include #include "mlx5_core.h" +#include "diag/fw_tracer.h" static int mlx5_devlink_health_buffer_fill_syndrom(struct devlink_health_buffer *dh_buffer, @@ -35,6 +36,115 @@ mlx5_devlink_health_buffer_fill_syndrom(struct devlink_health_buffer *dh_buffer, return 0; } +int mlx5_devlink_health_buffer_fill_trace(struct devlink_health_buffer *dh_buffer, + char *trace) +{ + int nest = 0; + int err = 0; + int i; + + err = devlink_health_buffer_nest_start(dh_buffer, + DEVLINK_ATTR_HEALTH_BUFFER_OBJECT); + if (err) + goto nest_cancel; + nest++; + + err = devlink_health_buffer_nest_start(dh_buffer, + DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_PAIR); + if (err) + goto nest_cancel; + nest++; + + err = devlink_health_buffer_put_object_name(dh_buffer, "trace"); + if (err) + goto nest_cancel; + + err = devlink_health_buffer_nest_start(dh_buffer, + DEVLINK_ATTR_HEALTH_BUFFER_OBJECT_VALUE); + if (err) + goto nest_cancel; + nest++; + + err = devlink_health_buffer_put_value_string(dh_buffer, trace); + if (err) + goto nest_cancel; + + for (i = 0; i < nest; i++) + devlink_health_buffer_nest_end(dh_buffer); + return 0; +nest_cancel: + for (i = 0; i < nest; i++) + devlink_health_buffer_nest_cancel(dh_buffer); + + return err; +} + +int mlx5_fw_tracer_get_saved_traces_objects(struct mlx5_fw_tracer *tracer, + struct devlink_health_buffer **buffers_array, + unsigned int num_buffers) +{ + u32 saved_traces_index = tracer->sbuff.saved_traces_index; + char *saved_traces = tracer->sbuff.traces_buff; + u32 index, start_index, end_index; + u32 dh_buffer_index = 0; + int err = 0; + + if (!saved_traces[0]) + return -ENOMSG; + + if (saved_traces[saved_traces_index * TRACE_STR_LINE]) + start_index = saved_traces_index; + else + start_index = 0; + end_index = (saved_traces_index - 1) & (SAVED_TRACES_NUM - 1); + + index = start_index; + while (index <= end_index) { + err = mlx5_devlink_health_buffer_fill_trace(buffers_array[dh_buffer_index], + saved_traces + index * TRACE_STR_LINE); + if (err) { + dh_buffer_index++; + if (dh_buffer_index == num_buffers) + break; + } else { + index++; + } + } + + return err; +} + +static int +mlx5_fw_reporter_objdump(struct devlink_health_reporter *reporter, + struct devlink_health_buffer **buffers_array, + unsigned int buff_size, unsigned int num_buffers, + void *priv_ctx) +{ + struct mlx5_core_dev *dev = devlink_health_reporter_priv(reporter); + struct devlink_health_buffer *buffer; + int err; + + if (!buffers_array || buff_size < TRACE_STR_LINE || num_buffers < 1) + return -EINVAL; + + err = mlx5_fw_tracer_trigger_core_dump_general(dev); + if (err) + return err; + + buffer = buffers_array[0]; + if (priv_ctx) { + struct mlx5_fw_reporter_ctx *fw_reporter_ctx = priv_ctx; + + err = mlx5_devlink_health_buffer_fill_syndrom(buffer, + fw_reporter_ctx->err_synd); + if (err) + return err; + } + + return mlx5_fw_tracer_get_saved_traces_objects(dev->tracer, buffers_array, + num_buffers); +} + static int mlx5_fw_reporter_diagnose(struct devlink_health_reporter *reporter, struct devlink_health_buffer **buffers_array, @@ -97,7 +207,9 @@ mlx5_fw_reporter_diagnose(struct devlink_health_reporter *reporter, static const struct devlink_health_reporter_ops mlx5_fw_reporter_ops = { .name = "FW", + .objdump_size = SAVED_TRACES_BUFFER_SIZE_BYTE, .diagnose_size = HEALTH_INFO_MAX_BUFF, + .objdump = mlx5_fw_reporter_objdump, .diagnose = mlx5_fw_reporter_diagnose, }; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c b/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c index d0f8449019af..b704df545d01 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.c @@ -243,6 +243,24 @@ static int mlx5_fw_tracer_allocate_strings_db(struct mlx5_fw_tracer *tracer) return -ENOMEM; } +static int mlx5_fw_tracer_allocate_saved_traces_buff(struct mlx5_fw_tracer *tracer) +{ + int traces_buff_size = SAVED_TRACES_BUFFER_SIZE_BYTE; + + tracer->sbuff.traces_buff = kzalloc(traces_buff_size, GFP_KERNEL); + if (!tracer->sbuff.traces_buff) + return -ENOMEM; + tracer->sbuff.saved_traces_index = 0; + + return 0; +} + +static void mlx5_fw_tracer_free_saved_traces_buff(struct mlx5_fw_tracer *tracer) +{ + kfree(tracer->sbuff.traces_buff); + tracer->sbuff.traces_buff = NULL; +} + static void mlx5_tracer_read_strings_db(struct work_struct *work) { struct mlx5_fw_tracer *tracer = container_of(work, struct mlx5_fw_tracer, @@ -522,6 +540,20 @@ static void mlx5_fw_tracer_clean_ready_list(struct mlx5_fw_tracer *tracer) list_del(&str_frmt->list); } +static void mlx5_fw_tracer_save_trace(struct mlx5_fw_tracer *tracer, u64 timestamp, + bool lost, u8 event_id, char *msg) +{ + char *saved_traces = tracer->sbuff.traces_buff; + u32 offset; + + offset = tracer->sbuff.saved_traces_index * TRACE_STR_LINE; + snprintf(saved_traces + offset, TRACE_STR_LINE, "%s [0x%llx] %d [0x%x] %s", + dev_name(&tracer->dev->pdev->dev), timestamp, lost, event_id, msg); + + tracer->sbuff.saved_traces_index = + (tracer->sbuff.saved_traces_index + 1) & (SAVED_TRACES_NUM - 1); +} + static void mlx5_tracer_print_trace(struct tracer_string_format *str_frmt, struct mlx5_core_dev *dev, u64 trace_timestamp) @@ -540,6 +572,9 @@ static void mlx5_tracer_print_trace(struct tracer_string_format *str_frmt, trace_mlx5_fw(dev->tracer, trace_timestamp, str_frmt->lost, str_frmt->event_id, tmp); + mlx5_fw_tracer_save_trace(dev->tracer, trace_timestamp, + str_frmt->lost, str_frmt->event_id, tmp); + /* remove it from hash */ mlx5_tracer_clean_message(str_frmt); } @@ -864,10 +899,18 @@ struct mlx5_fw_tracer *mlx5_fw_tracer_create(struct mlx5_core_dev *dev) goto free_log_buf; } + err = mlx5_fw_tracer_allocate_saved_traces_buff(tracer); + if (err) { + mlx5_core_warn(dev, "FWTracer: Create saved traces buffer failed %d\n", err); + goto free_strings_db; + } + mlx5_core_dbg(dev, "FWTracer: Tracer created\n"); return tracer; +free_strings_db: + mlx5_fw_tracer_free_strings_db(tracer); free_log_buf: mlx5_fw_tracer_destroy_log_buf(tracer); destroy_workqueue: @@ -948,6 +991,7 @@ void mlx5_fw_tracer_destroy(struct mlx5_fw_tracer *tracer) cancel_work_sync(&tracer->read_fw_strings_work); mlx5_fw_tracer_clean_ready_list(tracer); mlx5_fw_tracer_clean_print_hash(tracer); + mlx5_fw_tracer_free_saved_traces_buff(tracer); mlx5_fw_tracer_free_strings_db(tracer); mlx5_fw_tracer_destroy_log_buf(tracer); flush_workqueue(tracer->work_queue); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.h b/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.h index a8b8747f2b61..ad817932cc8e 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/diag/fw_tracer.h @@ -46,6 +46,10 @@ #define TRACER_BLOCK_SIZE_BYTE 256 #define TRACES_PER_BLOCK 32 +#define TRACE_STR_LINE 256 +#define SAVED_TRACES_NUM 1024 +#define SAVED_TRACES_BUFFER_SIZE_BYTE (SAVED_TRACES_NUM * TRACE_STR_LINE) + #define TRACER_MAX_PARAMS 7 #define MESSAGE_HASH_BITS 6 #define MESSAGE_HASH_SIZE BIT(MESSAGE_HASH_BITS) @@ -83,6 +87,12 @@ struct mlx5_fw_tracer { u32 consumer_index; } buff; + /* Saved Tarces Buffer */ + struct { + void *traces_buff; + u32 saved_traces_index; + } sbuff; + u64 last_timestamp; struct work_struct handle_traces_work; struct hlist_head hash[MESSAGE_HASH_SIZE]; @@ -171,5 +181,8 @@ struct mlx5_fw_tracer *mlx5_fw_tracer_create(struct mlx5_core_dev *dev); int mlx5_fw_tracer_init(struct mlx5_fw_tracer *tracer); void mlx5_fw_tracer_cleanup(struct mlx5_fw_tracer *tracer); void mlx5_fw_tracer_destroy(struct mlx5_fw_tracer *tracer); +int mlx5_fw_tracer_trigger_core_dump_general(struct mlx5_core_dev *dev); +int mlx5_fw_tracer_get_saved_traces(struct mlx5_fw_tracer *tracer, + char *buff, unsigned int buff_size); #endif From patchwork Mon Dec 31 14:32:10 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eran Ben Elisha X-Patchwork-Id: 1019583 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43T0Cb4QN7z9s55 for ; Tue, 1 Jan 2019 01:32:59 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726839AbeLaOc0 (ORCPT ); Mon, 31 Dec 2018 09:32:26 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:52288 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726722AbeLaOcZ (ORCPT ); Mon, 31 Dec 2018 09:32:25 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from eranbe@mellanox.com) with ESMTPS (AES256-SHA encrypted); 31 Dec 2018 16:32:22 +0200 Received: from dev-l-vrt-198.mtl.labs.mlnx (dev-l-vrt-198.mtl.labs.mlnx [10.134.198.1]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id wBVEWLZ4017529; Mon, 31 Dec 2018 16:32:21 +0200 From: Eran Ben Elisha To: netdev@vger.kernel.org, "David S. Miller" , Jiri Pirko Cc: Moshe Shemesh , Aya Levin , Eran Ben Elisha , Tal Alon , Ariel Almog Subject: [PATCH RFC net-next 16/19] net/mlx5: Report devlink health on FW issues Date: Mon, 31 Dec 2018 16:32:10 +0200 Message-Id: <1546266733-9512-17-git-send-email-eranbe@mellanox.com> X-Mailer: git-send-email 1.8.4.3 In-Reply-To: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> References: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Moshe Shemesh Use devlink_health_report() to report any symptom of FW issue as FW counter miss or new health syndrom. Signed-off-by: Moshe Shemesh --- .../net/ethernet/mellanox/mlx5/core/devlink.c | 21 +++++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/devlink.h | 1 + .../net/ethernet/mellanox/mlx5/core/health.c | 10 +++++++++ include/linux/mlx5/driver.h | 2 ++ 4 files changed, 34 insertions(+) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c index 07bc473a8ebb..04c904214e4c 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c @@ -205,6 +205,27 @@ mlx5_fw_reporter_diagnose(struct devlink_health_reporter *reporter, return 0; } +void mlx5_fw_reporter_err_work(struct work_struct *work) +{ + struct mlx5_fw_reporter_ctx fw_reporter_ctx; + struct mlx5_core_health *health; + struct mlx5_core_dev *dev; + struct mlx5_priv *priv; + + health = container_of(work, struct mlx5_core_health, report_work); + priv = container_of(health, struct mlx5_priv, health); + dev = container_of(priv, struct mlx5_core_dev, priv); + + fw_reporter_ctx.err_synd = health->synd; + fw_reporter_ctx.miss_counter = health->miss_counter; + if (fw_reporter_ctx.err_synd) + devlink_health_report(dev->fw_reporter, "FW syndrom reported", + &fw_reporter_ctx); + else if (fw_reporter_ctx.miss_counter) + devlink_health_report(dev->fw_reporter, "FW miss counter reported", + &fw_reporter_ctx); +} + static const struct devlink_health_reporter_ops mlx5_fw_reporter_ops = { .name = "FW", .objdump_size = SAVED_TRACES_BUFFER_SIZE_BYTE, diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.h b/drivers/net/ethernet/mellanox/mlx5/core/devlink.h index 34f6bfed1cfb..082a648a3af3 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.h @@ -16,5 +16,6 @@ int mlx5_devlink_register(struct devlink *devlink, struct device *dev); void mlx5_devlink_unregister(struct devlink *devlink); int mlx5_fw_reporter_create(struct mlx5_core_dev *dev); void mlx5_fw_reporter_destroy(struct mlx5_core_dev *dev); +void mlx5_fw_reporter_err_work(struct work_struct *work); #endif /* __MLX5_DEVLINK_H__ */ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/health.c b/drivers/net/ethernet/mellanox/mlx5/core/health.c index 2fa78edde1fe..4d0ad792b226 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/health.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/health.c @@ -38,6 +38,7 @@ #include #include #include "mlx5_core.h" +#include "devlink.h" #include "lib/eq.h" #include "lib/mlx5.h" @@ -289,7 +290,9 @@ static void poll_health(struct timer_list *t) { struct mlx5_core_dev *dev = from_timer(dev, t, priv.health.timer); struct mlx5_core_health *health = &dev->priv.health; + struct health_buffer __iomem *h = health->health; u32 count; + u8 synd; if (dev->state == MLX5_DEVICE_STATE_INTERNAL_ERROR) goto out; @@ -304,8 +307,13 @@ static void poll_health(struct timer_list *t) if (health->miss_counter == MAX_MISSES) { dev_err(&dev->pdev->dev, "device's health compromised - reached miss count\n"); mlx5_print_health_info(dev); + queue_work(health->wq, &health->report_work); } + synd = ioread8(&h->synd); + if (synd && synd != health->synd) + queue_work(health->wq, &health->report_work); + if (in_fatal(dev) && !health->sick) { health->sick = true; mlx5_print_health_info(dev); @@ -356,6 +364,7 @@ void mlx5_drain_health_wq(struct mlx5_core_dev *dev) set_bit(MLX5_DROP_NEW_RECOVERY_WORK, &health->flags); spin_unlock_irqrestore(&health->wq_lock, flags); cancel_delayed_work_sync(&health->recover_work); + cancel_work_sync(&health->report_work); cancel_work_sync(&health->work); } @@ -395,6 +404,7 @@ int mlx5_health_init(struct mlx5_core_dev *dev) return -ENOMEM; spin_lock_init(&health->wq_lock); INIT_WORK(&health->work, health_care); + INIT_WORK(&health->report_work, mlx5_fw_reporter_err_work); INIT_DELAYED_WORK(&health->recover_work, health_recover); return 0; diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h index bedc9bc08963..b2dc32b553b4 100644 --- a/include/linux/mlx5/driver.h +++ b/include/linux/mlx5/driver.h @@ -432,12 +432,14 @@ struct mlx5_core_health { struct timer_list timer; u32 prev; int miss_counter; + u8 synd; bool sick; /* wq spinlock to synchronize draining */ spinlock_t wq_lock; struct workqueue_struct *wq; unsigned long flags; struct work_struct work; + struct work_struct report_work; struct delayed_work recover_work; }; From patchwork Mon Dec 31 14:32:11 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eran Ben Elisha X-Patchwork-Id: 1019570 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43T0CM3JTJz9s55 for ; Tue, 1 Jan 2019 01:32:47 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727281AbeLaOc2 (ORCPT ); Mon, 31 Dec 2018 09:32:28 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:52287 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726686AbeLaOc1 (ORCPT ); Mon, 31 Dec 2018 09:32:27 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from eranbe@mellanox.com) with ESMTPS (AES256-SHA encrypted); 31 Dec 2018 16:32:22 +0200 Received: from dev-l-vrt-198.mtl.labs.mlnx (dev-l-vrt-198.mtl.labs.mlnx [10.134.198.1]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id wBVEWLZ5017529; Mon, 31 Dec 2018 16:32:21 +0200 From: Eran Ben Elisha To: netdev@vger.kernel.org, "David S. Miller" , Jiri Pirko Cc: Moshe Shemesh , Aya Levin , Eran Ben Elisha , Tal Alon , Ariel Almog Subject: [PATCH RFC net-next 17/19] net/mlx5: Add FW fatal devlink_health_reporter Date: Mon, 31 Dec 2018 16:32:11 +0200 Message-Id: <1546266733-9512-18-git-send-email-eranbe@mellanox.com> X-Mailer: git-send-email 1.8.4.3 In-Reply-To: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> References: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Moshe Shemesh Create mlx5_devlink_health_reporter for FW fatal reporter. The FW fatal reporter is added in addition to the fw reporter and implements only the recover callback. The point of having two reporters for FW issues, is that we don't want to run FW recover on any issue, but only fatal ones. Signed-off-by: Moshe Shemesh --- .../net/ethernet/mellanox/mlx5/core/devlink.c | 51 +++++++++++++++---- .../net/ethernet/mellanox/mlx5/core/devlink.h | 4 +- .../net/ethernet/mellanox/mlx5/core/main.c | 6 +-- include/linux/mlx5/driver.h | 1 + 4 files changed, 48 insertions(+), 14 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c index 04c904214e4c..2e6c74ed1f04 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c @@ -234,21 +234,54 @@ static const struct devlink_health_reporter_ops mlx5_fw_reporter_ops = { .diagnose = mlx5_fw_reporter_diagnose, }; -int mlx5_fw_reporter_create(struct mlx5_core_dev *dev) +static int +mlx5_fw_fatal_reporter_recover(struct devlink_health_reporter *reporter, + void *priv_ctx) { - struct devlink *devlink = priv_to_devlink(dev); + struct mlx5_core_dev *dev = devlink_health_reporter_priv(reporter); + u8 nic_state; - dev->fw_reporter = devlink_health_reporter_create(devlink, &mlx5_fw_reporter_ops, - 0, false, dev); - return PTR_ERR_OR_ZERO(dev->fw_reporter); + nic_state = mlx5_get_nic_state(dev); + if (nic_state == MLX5_NIC_IFC_INVALID) { + dev_err(&dev->pdev->dev, "health recovery flow aborted since the nic state is invalid\n"); + return -ECANCELED; + } + dev_err(&dev->pdev->dev, "starting health recovery flow\n"); + + mlx5_recover_device(dev); + + return 0; } -void mlx5_fw_reporter_destroy(struct mlx5_core_dev *dev) +static const struct devlink_health_reporter_ops mlx5_fw_fatal_reporter_ops = { + .name = "FW_fatal", + .recover = mlx5_fw_fatal_reporter_recover, +}; + +#define MLX5_REPORTER_FW_GRACEFUL_PERIOD 120000 +int mlx5_fw_reporters_create(struct mlx5_core_dev *dev) { - if (!dev->fw_reporter) - return; + struct devlink *devlink = priv_to_devlink(dev); - devlink_health_reporter_destroy(dev->fw_reporter); + dev->fw_reporter = + devlink_health_reporter_create(devlink, &mlx5_fw_reporter_ops, + 0, false, dev); + if (IS_ERR(dev->fw_reporter)) + return PTR_ERR(dev->fw_reporter); + + dev->fw_fatal_reporter = + devlink_health_reporter_create(devlink, &mlx5_fw_fatal_reporter_ops, + MLX5_REPORTER_FW_GRACEFUL_PERIOD, + true, dev); + return PTR_ERR_OR_ZERO(dev->fw_fatal_reporter); +} + +void mlx5_fw_reporters_destroy(struct mlx5_core_dev *dev) +{ + if (dev->fw_reporter) + devlink_health_reporter_destroy(dev->fw_reporter); + if (dev->fw_fatal_reporter) + devlink_health_reporter_destroy(dev->fw_fatal_reporter); } int mlx5_devlink_register(struct devlink *devlink, struct device *dev) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.h b/drivers/net/ethernet/mellanox/mlx5/core/devlink.h index 082a648a3af3..9b544f677aa7 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.h @@ -14,8 +14,8 @@ struct mlx5_fw_reporter_ctx { int mlx5_devlink_register(struct devlink *devlink, struct device *dev); void mlx5_devlink_unregister(struct devlink *devlink); -int mlx5_fw_reporter_create(struct mlx5_core_dev *dev); -void mlx5_fw_reporter_destroy(struct mlx5_core_dev *dev); +int mlx5_fw_reporters_create(struct mlx5_core_dev *dev); +void mlx5_fw_reporters_destroy(struct mlx5_core_dev *dev); void mlx5_fw_reporter_err_work(struct work_struct *work); #endif /* __MLX5_DEVLINK_H__ */ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c index ca5f4c661f6d..8f12c761a485 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c @@ -981,9 +981,9 @@ static int mlx5_load_one(struct mlx5_core_dev *dev, struct mlx5_priv *priv, goto err_fw_tracer; } - err = mlx5_fw_reporter_create(dev); + err = mlx5_fw_reporters_create(dev); if (err) - dev_warn(&pdev->dev, "Failed to create FW reporter\n"); + dev_warn(&pdev->dev, "Failed to create FW reporters\n"); err = mlx5_fpga_device_start(dev); if (err) { @@ -1117,7 +1117,7 @@ static int mlx5_unload_one(struct mlx5_core_dev *dev, struct mlx5_priv *priv, mlx5_accel_ipsec_cleanup(dev); mlx5_accel_tls_cleanup(dev); mlx5_fpga_device_stop(dev); - mlx5_fw_reporter_destroy(dev); + mlx5_fw_reporters_destroy(dev); mlx5_fw_tracer_cleanup(dev->tracer); mlx5_eq_table_destroy(dev); mlx5_pagealloc_stop(dev); diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h index b2dc32b553b4..8ab66bb40a17 100644 --- a/include/linux/mlx5/driver.h +++ b/include/linux/mlx5/driver.h @@ -681,6 +681,7 @@ struct mlx5_core_dev { struct page *clock_info_page; struct mlx5_fw_tracer *tracer; struct devlink_health_reporter *fw_reporter; + struct devlink_health_reporter *fw_fatal_reporter; }; struct mlx5_db { From patchwork Mon Dec 31 14:32:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eran Ben Elisha X-Patchwork-Id: 1019575 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43T0CQ5Kczz9sBQ for ; Tue, 1 Jan 2019 01:32:50 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727342AbeLaOcn (ORCPT ); Mon, 31 Dec 2018 09:32:43 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:52301 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726740AbeLaOc0 (ORCPT ); Mon, 31 Dec 2018 09:32:26 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from eranbe@mellanox.com) with ESMTPS (AES256-SHA encrypted); 31 Dec 2018 16:32:22 +0200 Received: from dev-l-vrt-198.mtl.labs.mlnx (dev-l-vrt-198.mtl.labs.mlnx [10.134.198.1]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id wBVEWLZ6017529; Mon, 31 Dec 2018 16:32:22 +0200 From: Eran Ben Elisha To: netdev@vger.kernel.org, "David S. Miller" , Jiri Pirko Cc: Moshe Shemesh , Aya Levin , Eran Ben Elisha , Tal Alon , Ariel Almog Subject: [PATCH RFC net-next 18/19] net/mlx5: Report devlink health on FW fatal issues Date: Mon, 31 Dec 2018 16:32:12 +0200 Message-Id: <1546266733-9512-19-git-send-email-eranbe@mellanox.com> X-Mailer: git-send-email 1.8.4.3 In-Reply-To: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> References: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Moshe Shemesh Report devlink health on FW fatal issues via FW_fatal_reporter. The driver recover flow for FW fatal error is now being handled by the devlink health. Having the recovery controlled by devlink health, the user has the ability to cancel the auto-recovery for debug session and run it manually. Signed-off-by: Moshe Shemesh --- .../net/ethernet/mellanox/mlx5/core/devlink.c | 15 +++++++++++ .../net/ethernet/mellanox/mlx5/core/devlink.h | 1 + .../net/ethernet/mellanox/mlx5/core/health.c | 25 +------------------ 3 files changed, 17 insertions(+), 24 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c index 2e6c74ed1f04..4e41b8844d61 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.c @@ -234,6 +234,21 @@ static const struct devlink_health_reporter_ops mlx5_fw_reporter_ops = { .diagnose = mlx5_fw_reporter_diagnose, }; +void mlx5_fw_fatal_reporter_work(struct work_struct *work) +{ + struct mlx5_core_health *health; + struct delayed_work *dwork; + struct mlx5_core_dev *dev; + struct mlx5_priv *priv; + + dwork = container_of(work, struct delayed_work, work); + health = container_of(dwork, struct mlx5_core_health, recover_work); + priv = container_of(health, struct mlx5_priv, health); + dev = container_of(priv, struct mlx5_core_dev, priv); + + devlink_health_report(dev->fw_fatal_reporter, "FW recovery", NULL); +} + static int mlx5_fw_fatal_reporter_recover(struct devlink_health_reporter *reporter, void *priv_ctx) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/devlink.h b/drivers/net/ethernet/mellanox/mlx5/core/devlink.h index 9b544f677aa7..9c5eba632293 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/devlink.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/devlink.h @@ -17,5 +17,6 @@ void mlx5_devlink_unregister(struct devlink *devlink); int mlx5_fw_reporters_create(struct mlx5_core_dev *dev); void mlx5_fw_reporters_destroy(struct mlx5_core_dev *dev); void mlx5_fw_reporter_err_work(struct work_struct *work); +void mlx5_fw_fatal_reporter_work(struct work_struct *work); #endif /* __MLX5_DEVLINK_H__ */ diff --git a/drivers/net/ethernet/mellanox/mlx5/core/health.c b/drivers/net/ethernet/mellanox/mlx5/core/health.c index 4d0ad792b226..f3d030de5695 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/health.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/health.c @@ -138,29 +138,6 @@ static void mlx5_handle_bad_state(struct mlx5_core_dev *dev) mlx5_disable_device(dev); } -static void health_recover(struct work_struct *work) -{ - struct mlx5_core_health *health; - struct delayed_work *dwork; - struct mlx5_core_dev *dev; - struct mlx5_priv *priv; - u8 nic_state; - - dwork = container_of(work, struct delayed_work, work); - health = container_of(dwork, struct mlx5_core_health, recover_work); - priv = container_of(health, struct mlx5_priv, health); - dev = container_of(priv, struct mlx5_core_dev, priv); - - nic_state = mlx5_get_nic_state(dev); - if (nic_state == MLX5_NIC_IFC_INVALID) { - dev_err(&dev->pdev->dev, "health recovery flow aborted since the nic state is invalid\n"); - return; - } - - dev_err(&dev->pdev->dev, "starting health recovery flow\n"); - mlx5_recover_device(dev); -} - /* How much time to wait until health resetting the driver (in msecs) */ #define MLX5_RECOVERY_DELAY_MSECS 60000 static void health_care(struct work_struct *work) @@ -405,7 +382,7 @@ int mlx5_health_init(struct mlx5_core_dev *dev) spin_lock_init(&health->wq_lock); INIT_WORK(&health->work, health_care); INIT_WORK(&health->report_work, mlx5_fw_reporter_err_work); - INIT_DELAYED_WORK(&health->recover_work, health_recover); + INIT_DELAYED_WORK(&health->recover_work, mlx5_fw_fatal_reporter_work); return 0; } From patchwork Mon Dec 31 14:32:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eran Ben Elisha X-Patchwork-Id: 1019585 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=mellanox.com Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43T0Cg3Pp7z9s55 for ; Tue, 1 Jan 2019 01:33:03 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727365AbeLaOdA (ORCPT ); Mon, 31 Dec 2018 09:33:00 -0500 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:52302 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726771AbeLaOcZ (ORCPT ); Mon, 31 Dec 2018 09:32:25 -0500 Received: from Internal Mail-Server by MTLPINE1 (envelope-from eranbe@mellanox.com) with ESMTPS (AES256-SHA encrypted); 31 Dec 2018 16:32:22 +0200 Received: from dev-l-vrt-198.mtl.labs.mlnx (dev-l-vrt-198.mtl.labs.mlnx [10.134.198.1]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id wBVEWLZ7017529; Mon, 31 Dec 2018 16:32:22 +0200 From: Eran Ben Elisha To: netdev@vger.kernel.org, "David S. Miller" , Jiri Pirko Cc: Moshe Shemesh , Aya Levin , Eran Ben Elisha , Tal Alon , Ariel Almog Subject: [PATCH RFC net-next 19/19] devlink: Add Documentation/networking/devlink-health.txt Date: Mon, 31 Dec 2018 16:32:13 +0200 Message-Id: <1546266733-9512-20-git-send-email-eranbe@mellanox.com> X-Mailer: git-send-email 1.8.4.3 In-Reply-To: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> References: <1546266733-9512-1-git-send-email-eranbe@mellanox.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Aya Levin This patch adds a new file to add information about devlink health mechanism. Signed-off-by: Aya Levin --- Documentation/networking/devlink-health.txt | 86 +++++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 Documentation/networking/devlink-health.txt diff --git a/Documentation/networking/devlink-health.txt b/Documentation/networking/devlink-health.txt new file mode 100644 index 000000000000..ea8a9cc773a2 --- /dev/null +++ b/Documentation/networking/devlink-health.txt @@ -0,0 +1,86 @@ +The health mechanism is targeted for Real Time Alerting, in order to know when +something bad had happened to a PCI device +- Provide alert debug information +- Self healing +- If problem needs vendor support, provide a way to gather all needed debugging + information. + +The main idea is to unify and centralize driver health reports in the +generic devlink instance and allow the user to set different +attributes of the health reporting and recovery procedures. + +The devlink health reporter: +Device driver creates a "health reporter" per each error/health type. +Error/Health type can be a known/generic (eg pci error, fw error, rx/tx error) +or unknown (driver specific). +For each registered health reporter a driver can issue error/health reports +asynchronously. All health reports handling is done by devlink. +Device driver can provide specific callbacks for each "health reporter", e.g. + - Recovery procedures + - Diagnostics and object dump procedures + - OOB initial parameters +Different parts of the driver can register different types of health reporters +with different handlers. + +Once an error is reported, devlink health will do the following actions: + * A log is being send to the kernel trace events buffer + * Health status and statistics are being updated for the reporter instance + * Object dump is being taken and saved at the reporter instance (as long as + there is no other Objdump which is already stored) + * Auto recovery attempt is being done. Depends on: + - Auto-recovery configuration + - Grace period vs. time passed since last recover + +The user interface: +User can access/change each reporter's parameters and driver specific callbacks +via devlink, e.g per error type (per health reporter) + - Configure reporter's generic parameters (like: disable/enable auto recovery) + - Invoke recovery procedure + - Run diagnostics + - Object dump + +The devlink health interface (via netlink): +DEVLINK_CMD_HEALTH_REPORTER_GET + Retrieves status and configuration info per DEV and reporter. +DEVLINK_CMD_HEALTH_REPORTER_SET + Allows reporter-related configuration setting. +DEVLINK_CMD_HEALTH_REPORTER_RECOVER + Triggers a reporter's recovery procedure. +DEVLINK_CMD_HEALTH_REPORTER_DIAGNOSE + Retrieves diagnostics data from a reporter on a device. +DEVLINK_CMD_HEALTH_REPORTER_OBJDUMP_GET + Retrieves the last stored objdump. Devlink health + saves a single objdump. If an objdump is not already stored by the devlink + for this reporter, devlink generates a new objdump. + Objdump output is defined by the reporter. +DEVLINK_CMD_HEALTH_REPORTER_OBJDUMP_CLEAR + Clears the last saved objdump file for the specified reporter. + + + netlink + +--------------------------+ + | | + | + | + | | | + +--------------------------+ + |request for ops + |(diagnose, + mlx5_core devlink |recover, + |dump) ++--------+ +--------------------------+ +| | | reporter| | +| | | +---------v----------+ | +| | ops execution | | | | +| <----------------------------------+ | | +| | | | | | +| | | + ^------------------+ | +| | | | request for ops | +| | | | (recover, dump) | +| | | | | +| | | +-+------------------+ | +| | health report | | health handler | | +| +-------------------------------> | | +| | | +--------------------+ | +| | health reporter create | | +| +----------------------------> | ++--------+ +--------------------------+