From patchwork Thu Jul 7 15:38:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Finn, Emma" X-Patchwork-Id: 1653672 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=enXMkgCR; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lf0vY2HtKz9s0w for ; Fri, 8 Jul 2022 01:39:21 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 47A4941BA8; Thu, 7 Jul 2022 15:39:18 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 47A4941BA8 Authentication-Results: smtp4.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=enXMkgCR X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id on2FE_vycVi0; Thu, 7 Jul 2022 15:39:15 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id A4F7641B8B; Thu, 7 Jul 2022 15:39:13 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org A4F7641B8B Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 6C2B1C0033; Thu, 7 Jul 2022 15:39:13 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id B99F7C002D for ; Thu, 7 Jul 2022 15:39:12 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 7C9BC4100C for ; Thu, 7 Jul 2022 15:39:12 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 7C9BC4100C Authentication-Results: smtp2.osuosl.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=enXMkgCR X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4m6oj38fC8DL for ; Thu, 7 Jul 2022 15:39:11 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 259AE4051C Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by smtp2.osuosl.org (Postfix) with ESMTPS id 259AE4051C for ; Thu, 7 Jul 2022 15:39:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657208351; x=1688744351; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=F5pgPDzSaySE62wE0GCL3inZCgasS2yjU9md/QSuq9M=; b=enXMkgCRW4tnK/0ImQoIN9QLf4SD4ymUJF/DEXDMrX4VS898VuhnkuTm kBcEofdDvItiMeYcDfUFU/r3Z7fyzpNeD7ucG9NK6gGmtXUG3exenRYdm mwjMjv+5bU8/wCvHtaYjb2vIL+ynKBsnqT6MdNJ3fW/T0qHqJnhxTZC1r X54YMtbxYc5F25CgASyRu2fO0qooRzD8G/H8f2k1cIBa0smdoA+IzEaxi pdKdcIzA9wHuwVJNT2y8d2aKXofFkbZ2HvwH2Ze67ZjhMtpLJPssgdkE3 Y5Ntk8CLeJo1n+/U397RZ//sV2qrqnvJWHhgQsWOeQmp6d80MFxcl8kQW g==; X-IronPort-AV: E=McAfee;i="6400,9594,10400"; a="267083948" X-IronPort-AV: E=Sophos;i="5.92,253,1650956400"; d="scan'208";a="267083948" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jul 2022 08:39:10 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,253,1650956400"; d="scan'208";a="920633431" Received: from silpixa00401384.ir.intel.com ([10.243.22.75]) by fmsmga005.fm.intel.com with ESMTP; 07 Jul 2022 08:39:08 -0700 From: Emma Finn To: dev@openvswitch.org, echaudro@redhat.com, harry.van.haaren@intel.com, kumar.amber@intel.com Date: Thu, 7 Jul 2022 15:38:51 +0000 Message-Id: <20220707153900.3147694-2-emma.finn@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220707153900.3147694-1-emma.finn@intel.com> References: <20220614115743.1143341-2-emma.finn@intel.com> <20220707153900.3147694-1-emma.finn@intel.com> MIME-Version: 1.0 Cc: i.maximets@ovn.org Subject: [ovs-dev] [v8 01/10] odp-execute: Add function pointers to odp-execute for different action implementations. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This commit introduces the initial infrastructure required to allow different implementations for OvS actions. The patch introduces action function pointers which allows user to switch between different action implementations available. This will allow for more performance and flexibility so the user can choose the action implementation to best suite their use case. Signed-off-by: Emma Finn Acked-by: Harry van Haaren --- lib/automake.mk | 2 + lib/dpif-netdev.c | 4 ++ lib/odp-execute-private.c | 92 +++++++++++++++++++++++++++++++++++++++ lib/odp-execute-private.h | 76 ++++++++++++++++++++++++++++++++ lib/odp-execute.c | 51 +++++++++++++++++++++- lib/odp-execute.h | 7 +++ 6 files changed, 231 insertions(+), 1 deletion(-) create mode 100644 lib/odp-execute-private.c create mode 100644 lib/odp-execute-private.h diff --git a/lib/automake.mk b/lib/automake.mk index 1d00cfa20..23ba4fab0 100644 --- a/lib/automake.mk +++ b/lib/automake.mk @@ -216,6 +216,8 @@ lib_libopenvswitch_la_SOURCES = \ lib/object-collection.h \ lib/odp-execute.c \ lib/odp-execute.h \ + lib/odp-execute-private.c \ + lib/odp-execute-private.h \ lib/odp-util.c \ lib/odp-util.h \ lib/ofp-actions.c \ diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index d09138f2c..58be9d0f0 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -1678,6 +1678,10 @@ create_dpif_netdev(struct dp_netdev *dp) dpif->dp = dp; dpif->last_port_seq = seq_read(dp->port_seq); + /* Called once at initialization time. This handles setting up the state + * of the actions functions at init time. */ + odp_execute_init(); + return &dpif->dpif; } diff --git a/lib/odp-execute-private.c b/lib/odp-execute-private.c new file mode 100644 index 000000000..b10880ed9 --- /dev/null +++ b/lib/odp-execute-private.c @@ -0,0 +1,92 @@ +/* + * Copyright (c) 2022 Intel. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include +#include +#include +#include + +#include "dpdk.h" +#include "dp-packet.h" +#include "odp-execute-private.h" +#include "odp-netlink.h" +#include "odp-util.h" +#include "openvswitch/vlog.h" + +VLOG_DEFINE_THIS_MODULE(odp_execute_impl); +static int active_action_impl_index; + +static struct odp_execute_action_impl action_impls[] = { + [ACTION_IMPL_SCALAR] = { + .available = false, + .name = "scalar", + .init_func = NULL, + }, +}; + +static void +action_impl_copy_funcs(struct odp_execute_action_impl *src, + const struct odp_execute_action_impl *dest) +{ + for (int i = 0; i < __OVS_ACTION_ATTR_MAX; i++) { + atomic_store_relaxed(&src->funcs[i], dest->funcs[i]); + } +} + +struct odp_execute_action_impl * +odp_execute_action_set(const char *name) +{ + for (int i = 0; i < ACTION_IMPL_MAX; i++) { + /* String compare, and set ptrs atomically. */ + if (!strcmp(action_impls[i].name, name)) { + active_action_impl_index = i; + + VLOG_INFO("Action implementation set to %s", name); + return &action_impls[i]; + } + } + return NULL; +} + +void +odp_execute_action_init(void) +{ + /* Each impl's function array is initialized to reflect the scalar + * implementation. This simplifies adding optimized implementations, + * as the autovalidator can always compare all actions. + * + * Below will check if impl is available and copies the scalar functions + * to all other implementations. + */ + for (int i = 0; i < ACTION_IMPL_MAX; i++) { + bool avail = true; + + if (i != ACTION_IMPL_SCALAR) { + action_impl_copy_funcs(&action_impls[i], + &action_impls[ACTION_IMPL_SCALAR]); + } + + if (action_impls[i].init_func) { + /* Return zero is success, non-zero means error. */ + avail = (action_impls[i].init_func(&action_impls[i]) == 0); + } + + action_impls[i].available = avail; + + VLOG_INFO("Action implementation %s (available: %s)", + action_impls[i].name, avail ? "Yes" : "No"); + } +} diff --git a/lib/odp-execute-private.h b/lib/odp-execute-private.h new file mode 100644 index 000000000..24126cdca --- /dev/null +++ b/lib/odp-execute-private.h @@ -0,0 +1,76 @@ +/* + * Copyright (c) 2022 Intel. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef ODP_EXTRACT_PRIVATE +#define ODP_EXTRACT_PRIVATE 1 + +#include "dp-packet.h" +#include "odp-execute.h" +#include "odp-netlink.h" +#include "ovs-atomic.h" + +/* Forward declaration for typedef. */ +struct odp_execute_action_impl; + +/* Typedef for an initialization function that can initialize each + * implementation, checking requirements such as CPU ISA. + */ +typedef int (*odp_execute_action_init_func) + (struct odp_execute_action_impl *self); + +/* Structure represents an implementation of the odp actions. */ +struct odp_execute_action_impl { + /* When set, the CPU ISA required for this implementation is available + * and the implementation can be used. + */ + bool available; + + /* Name of the implementation. */ + const char *name; + + /* Function is used to detect if this CPU has the ISA required + * to run the optimized action implementation and if available, initializes + * the implementation for use. + */ + odp_execute_action_init_func init_func; + + /* An array of callback functions, one for each action. */ + ATOMIC(odp_execute_action_cb) funcs[__OVS_ACTION_ATTR_MAX]; +}; + +/* Order of Actions implementations. */ +enum odp_execute_action_impl_idx { + ACTION_IMPL_SCALAR, + /* See ACTION_IMPL_BEGIN below, for "first to-be-validated" impl. + * Do not change the autovalidator position in this list without updating + * the define below. + */ + + ACTION_IMPL_MAX, +}; + +/* Index to start verifying implementations from. */ +BUILD_ASSERT_DECL(ACTION_IMPL_SCALAR == 0); + +/* Odp execute init handles setting up the state of the actions functions at + * initialization time. It cannot return errors, as it must always succeed in + * initializing the scalar/generic codepath. + */ +void odp_execute_action_init(void); + +struct odp_execute_action_impl * odp_execute_action_set(const char *name); + +#endif /* ODP_EXTRACT_PRIVATE */ diff --git a/lib/odp-execute.c b/lib/odp-execute.c index 7da56793d..f6f5bea48 100644 --- a/lib/odp-execute.c +++ b/lib/odp-execute.c @@ -17,6 +17,7 @@ #include #include "odp-execute.h" +#include "odp-execute-private.h" #include #include #include @@ -833,6 +834,38 @@ requires_datapath_assistance(const struct nlattr *a) return false; } +/* The active function pointers on the datapath. ISA optimized implementations + * are enabled by plugging them into this static arary, which is consulted when + * applying actions on the datapath. + */ +static struct odp_execute_action_impl *actions_active_impl; + +static int +odp_actions_impl_set(const char *name) +{ + struct odp_execute_action_impl *active; + active = odp_execute_action_set(name); + if (!active) { + VLOG_ERR("Failed setting action implementation to %s", name); + return 1; + } + + actions_active_impl = active; + return 0; + +} + +void +odp_execute_init(void) +{ + static struct ovsthread_once once = OVSTHREAD_ONCE_INITIALIZER; + if (ovsthread_once_start(&once)) { + odp_execute_action_init(); + odp_actions_impl_set("scalar"); + ovsthread_once_done(&once); + } +} + /* Executes all of the 'actions_len' bytes of datapath actions in 'actions' on * the packets in 'batch'. If 'steal' is true, possibly modifies and * definitely free the packets in 'batch', otherwise leaves 'batch' unchanged. @@ -857,6 +890,7 @@ odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, NL_ATTR_FOR_EACH_UNSAFE (a, left, actions, actions_len) { int type = nl_attr_type(a); + enum ovs_action_attr attr_type = (enum ovs_action_attr) type; bool last_action = (left <= NLA_ALIGN(a->nla_len)); if (requires_datapath_assistance(a)) { @@ -879,8 +913,20 @@ odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, continue; } - switch ((enum ovs_action_attr) type) { + /* If type is set in the active actions implementation, call the + * function-pointer and continue to the next action. + */ + if (actions_active_impl->funcs[attr_type] && + attr_type <= OVS_ACTION_ATTR_MAX) { + actions_active_impl->funcs[attr_type](batch, a); + continue; + } + /* If the action was not handled by the active function pointers above, + * process them by switching on the type below. + */ + + switch (attr_type) { case OVS_ACTION_ATTR_HASH: { const struct ovs_action_hash *hash_act = nl_attr_get(a); @@ -1094,6 +1140,9 @@ odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, case __OVS_ACTION_ATTR_MAX: OVS_NOT_REACHED(); } + + /* Do not add any generic processing here, as it won't be executed when + * an ISA-specific action implementation exists. */ } dp_packet_delete_batch(batch, steal); diff --git a/lib/odp-execute.h b/lib/odp-execute.h index a3578a575..0921ee924 100644 --- a/lib/odp-execute.h +++ b/lib/odp-execute.h @@ -28,6 +28,13 @@ struct dp_packet; struct pkt_metadata; struct dp_packet_batch; + +/* Called once at initialization time. */ +void odp_execute_init(void); + +typedef void (*odp_execute_action_cb)(struct dp_packet_batch *batch, + const struct nlattr *action); + typedef void (*odp_execute_cb)(void *dp, struct dp_packet_batch *batch, const struct nlattr *action, bool should_steal); From patchwork Thu Jul 7 15:38:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Finn, Emma" X-Patchwork-Id: 1653675 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=k3cdSB6N; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lf0vh1HmSz9s2R for ; Fri, 8 Jul 2022 01:39:28 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id D70B5612A1; Thu, 7 Jul 2022 15:39:22 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org D70B5612A1 Authentication-Results: smtp3.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=k3cdSB6N X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cUyo9eMbOXw6; Thu, 7 Jul 2022 15:39:20 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id 37DF260BAF; Thu, 7 Jul 2022 15:39:18 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 37DF260BAF Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 388AAC0086; Thu, 7 Jul 2022 15:39:17 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 120D1C0033 for ; Thu, 7 Jul 2022 15:39:15 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id D04054100C for ; Thu, 7 Jul 2022 15:39:13 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org D04054100C Authentication-Results: smtp2.osuosl.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=k3cdSB6N X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Tm9Q__G_sNpG for ; Thu, 7 Jul 2022 15:39:13 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 0C9364051C Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by smtp2.osuosl.org (Postfix) with ESMTPS id 0C9364051C for ; Thu, 7 Jul 2022 15:39:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657208353; x=1688744353; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=C7WJcnewQ91qNcsbJOfnTn7FN11I7Qhq5yJlePPrbSM=; b=k3cdSB6NRT3lMjq7YsLnbTCUMBFLFmjpi4iAht56g1tagjSVmBw72xy0 QXwvX8JXBQ/bjHQyFrZAa2sUzCt4wt47c9V6u3kJ6QaPJ+vVZGZGnrfsg /Z43mUie/Sv6M1+tTEfxUxvpqAfILS8N86KYPltBThSBnZJ8ix8F1+Yhq rcLaNNwGqRMa41taFTj+2k12eeUhAPa8IGDiMSBjQecO32zPkYv8+Gc6K 9f5IpgyAwJ3ptP53hnavBbARRvTPQs5AxJHX61MDZR4wWo4fYbIR2CIH2 E2MVi/ObygDIObgqiHSN1LodciT+Koj0k19FNUisYX5BwQn1L1GilmS8k g==; X-IronPort-AV: E=McAfee;i="6400,9594,10400"; a="267083953" X-IronPort-AV: E=Sophos;i="5.92,253,1650956400"; d="scan'208";a="267083953" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jul 2022 08:39:12 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,253,1650956400"; d="scan'208";a="920633437" Received: from silpixa00401384.ir.intel.com ([10.243.22.75]) by fmsmga005.fm.intel.com with ESMTP; 07 Jul 2022 08:39:10 -0700 From: Emma Finn To: dev@openvswitch.org, echaudro@redhat.com, harry.van.haaren@intel.com, kumar.amber@intel.com Date: Thu, 7 Jul 2022 15:38:52 +0000 Message-Id: <20220707153900.3147694-3-emma.finn@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220707153900.3147694-1-emma.finn@intel.com> References: <20220614115743.1143341-2-emma.finn@intel.com> <20220707153900.3147694-1-emma.finn@intel.com> MIME-Version: 1.0 Cc: i.maximets@ovn.org Subject: [ovs-dev] [v8 02/10] odp-execute: Add function pointer for pop_vlan action. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This commit removes the pop_vlan action from the large switch and creates a separate function for batched processing. A function pointer is also added to call the new batched function for the pop_vlan action. Signed-off-by: Emma Finn Acked-by: Harry van Haaren --- lib/odp-execute-private.c | 16 +++++++++++++++- lib/odp-execute-private.h | 5 +++++ lib/odp-execute.c | 32 ++++++++++++++++++++++++++------ 3 files changed, 46 insertions(+), 7 deletions(-) diff --git a/lib/odp-execute-private.c b/lib/odp-execute-private.c index b10880ed9..017257a2d 100644 --- a/lib/odp-execute-private.c +++ b/lib/odp-execute-private.c @@ -33,7 +33,7 @@ static struct odp_execute_action_impl action_impls[] = { [ACTION_IMPL_SCALAR] = { .available = false, .name = "scalar", - .init_func = NULL, + .init_func = odp_action_scalar_init, }, }; @@ -88,5 +88,19 @@ odp_execute_action_init(void) VLOG_INFO("Action implementation %s (available: %s)", action_impls[i].name, avail ? "Yes" : "No"); + + /* The following is a run-time check to make sure a scalar + * implementation exists for the given ISA implementation. This is to + * make sure the autovalidator works as expected. */ + if (avail && i != ACTION_IMPL_SCALAR) { + for (int j = 0; j < __OVS_ACTION_ATTR_MAX; j++) { + /* No ovs_assert(), as it can be compiled out. */ + if (action_impls[ACTION_IMPL_SCALAR].funcs[j] == NULL + && action_impls[i].funcs[j] != NULL) { + ovs_assert_failure(OVS_SOURCE_LOCATOR, __func__, + "Missing scalar action function!"); + } + } + } } } diff --git a/lib/odp-execute-private.h b/lib/odp-execute-private.h index 24126cdca..ae06fbc09 100644 --- a/lib/odp-execute-private.h +++ b/lib/odp-execute-private.h @@ -71,6 +71,11 @@ BUILD_ASSERT_DECL(ACTION_IMPL_SCALAR == 0); */ void odp_execute_action_init(void); +/* Init functions for the action implementations. Initializes the function + * pointers for optimized action types. + */ +int odp_action_scalar_init(struct odp_execute_action_impl *self); + struct odp_execute_action_impl * odp_execute_action_set(const char *name); #endif /* ODP_EXTRACT_PRIVATE */ diff --git a/lib/odp-execute.c b/lib/odp-execute.c index f6f5bea48..e6839df6d 100644 --- a/lib/odp-execute.c +++ b/lib/odp-execute.c @@ -834,6 +834,30 @@ requires_datapath_assistance(const struct nlattr *a) return false; } +static void +action_pop_vlan(struct dp_packet_batch *batch, + const struct nlattr *a OVS_UNUSED) +{ + struct dp_packet *packet; + + DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { + eth_pop_vlan(packet); + } +} + +/* Implementation of the scalar actions impl init function. Build up the + * array of func ptrs here. + */ +int +odp_action_scalar_init(struct odp_execute_action_impl *self) +{ + /* Set function pointers for actions that can be applied directly, these + * are identified by OVS_ACTION_ATTR_*. */ + self->funcs[OVS_ACTION_ATTR_POP_VLAN] = action_pop_vlan; + + return 0; +} + /* The active function pointers on the datapath. ISA optimized implementations * are enabled by plugging them into this static arary, which is consulted when * applying actions on the datapath. @@ -982,12 +1006,6 @@ odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, break; } - case OVS_ACTION_ATTR_POP_VLAN: - DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { - eth_pop_vlan(packet); - } - break; - case OVS_ACTION_ATTR_PUSH_MPLS: { const struct ovs_action_push_mpls *mpls = nl_attr_get(a); @@ -1138,6 +1156,8 @@ odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, case OVS_ACTION_ATTR_CT: case OVS_ACTION_ATTR_UNSPEC: case __OVS_ACTION_ATTR_MAX: + /* The following actions are handled by the scalar implementation. */ + case OVS_ACTION_ATTR_POP_VLAN: OVS_NOT_REACHED(); } From patchwork Thu Jul 7 15:38:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Finn, Emma" X-Patchwork-Id: 1653674 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=JXK4dkK/; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lf0vg5KXtz9s0w for ; Fri, 8 Jul 2022 01:39:27 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id A726441BE7; Thu, 7 Jul 2022 15:39:23 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org A726441BE7 Authentication-Results: smtp4.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=JXK4dkK/ X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OUNq50aALoJe; Thu, 7 Jul 2022 15:39:22 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTPS id 00AAC41B98; Thu, 7 Jul 2022 15:39:19 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 00AAC41B98 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 0C07BC0070; Thu, 7 Jul 2022 15:39:18 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 360AFC0033 for ; Thu, 7 Jul 2022 15:39:16 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 0DCBC41026 for ; Thu, 7 Jul 2022 15:39:16 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 0DCBC41026 Authentication-Results: smtp2.osuosl.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=JXK4dkK/ X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XQcQHzyv448n for ; Thu, 7 Jul 2022 15:39:15 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 1E89E4100C Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by smtp2.osuosl.org (Postfix) with ESMTPS id 1E89E4100C for ; Thu, 7 Jul 2022 15:39:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657208355; x=1688744355; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=8MpdotD0uWWloJnebz98L4y9e3TjuX+Xa6MRWynRNDQ=; b=JXK4dkK/yBw+8Ddox3AuGKegw5vp4oe4VAj7lwN3MqSAXXKTySIjchJi RaEfe2c0ZMMcKoVGqQka/Jh/9igwdE2pv6oCvTMwE/x3eiUjN+Ww93RcG +GOGpejrP3RhhClSVbSDnOZgZkTQ+01DbUlDihkdKzoDhTr6CQGH+phzd c3e6ufzihPjE6GZuJutdKEHiD754rrvEa6BYT121a81bkyRp9Y/UCvPtq FiglkQ3PB5ZS51ffQyZ2FZW6D0KOyB3E+ydpPpJh7wzyldQ1jgLpuLYgU +/xfR/aeQyuwRz3pS9WkwmZxOAMnpJJbEO9tD7rl5z/1O3VgxN3Dzt8Aq g==; X-IronPort-AV: E=McAfee;i="6400,9594,10400"; a="267083960" X-IronPort-AV: E=Sophos;i="5.92,253,1650956400"; d="scan'208";a="267083960" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jul 2022 08:39:14 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,253,1650956400"; d="scan'208";a="920633445" Received: from silpixa00401384.ir.intel.com ([10.243.22.75]) by fmsmga005.fm.intel.com with ESMTP; 07 Jul 2022 08:39:12 -0700 From: Emma Finn To: dev@openvswitch.org, echaudro@redhat.com, harry.van.haaren@intel.com, kumar.amber@intel.com Date: Thu, 7 Jul 2022 15:38:53 +0000 Message-Id: <20220707153900.3147694-4-emma.finn@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220707153900.3147694-1-emma.finn@intel.com> References: <20220614115743.1143341-2-emma.finn@intel.com> <20220707153900.3147694-1-emma.finn@intel.com> MIME-Version: 1.0 Cc: i.maximets@ovn.org Subject: [ovs-dev] [v8 03/10] odp-execute: Add auto validation function for actions. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This commit introduced the auto-validation function which allows users to compare the batch of packets obtained from different action implementations against the linear action implementation. The autovalidator function can be triggered at runtime using the following command: $ ovs-appctl odp-execute/action-impl-set autovalidator Signed-off-by: Emma Finn Acked-by: Harry van Haaren --- NEWS | 2 + lib/dp-packet.c | 24 ++++++++++ lib/dp-packet.h | 4 ++ lib/odp-execute-private.c | 99 +++++++++++++++++++++++++++++++++++++++ lib/odp-execute-private.h | 6 +++ 5 files changed, 135 insertions(+) diff --git a/NEWS b/NEWS index 994fdf6a9..1d0364fff 100644 --- a/NEWS +++ b/NEWS @@ -43,6 +43,8 @@ Post-v2.17.0 * 'dpif-netdev/subtable-lookup-prio-get' appctl command renamed to 'dpif-netdev/subtable-lookup-info-get' to better reflect its purpose. The old variant is kept for backward compatibility. + * Add actions auto-validator function to compare different actions + implementations against default implementation. v2.17.0 - 17 Feb 2022 diff --git a/lib/dp-packet.c b/lib/dp-packet.c index 35c72542a..4538d2a61 100644 --- a/lib/dp-packet.c +++ b/lib/dp-packet.c @@ -506,3 +506,27 @@ dp_packet_resize_l2(struct dp_packet *b, int increment) dp_packet_adjust_layer_offset(&b->l2_5_ofs, increment); return dp_packet_data(b); } + +bool +dp_packet_compare_offsets(struct dp_packet *b1, struct dp_packet *b2, + struct ds *err_str) +{ + if ((b1->l2_pad_size != b2->l2_pad_size) || + (b1->l2_5_ofs != b2->l2_5_ofs) || + (b1->l3_ofs != b2->l3_ofs) || + (b1->l4_ofs != b2->l4_ofs)) { + if (err_str) { + ds_put_format(err_str, "Packet offset comparison failed\n"); + ds_put_format(err_str, "Buffer 1 offsets: l2_pad_size %u," + " l2_5_ofs : %u l3_ofs %u, l4_ofs %u\n", + b1->l2_pad_size, b1->l2_5_ofs, + b1->l3_ofs, b1->l4_ofs); + ds_put_format(err_str, "Buffer 2 offsets: l2_pad_size %u," + " l2_5_ofs : %u l3_ofs %u, l4_ofs %u\n", + b2->l2_pad_size, b2->l2_5_ofs, + b2->l3_ofs, b2->l4_ofs); + } + return false; + } + return true; +} diff --git a/lib/dp-packet.h b/lib/dp-packet.h index eea5a9215..55eeaab2c 100644 --- a/lib/dp-packet.h +++ b/lib/dp-packet.h @@ -236,6 +236,10 @@ void *dp_packet_steal_data(struct dp_packet *); static inline bool dp_packet_equal(const struct dp_packet *, const struct dp_packet *); +bool dp_packet_compare_offsets(struct dp_packet *good, + struct dp_packet *test, + struct ds *err_str); + /* Frees memory that 'b' points to, as well as 'b' itself. */ static inline void diff --git a/lib/odp-execute-private.c b/lib/odp-execute-private.c index 017257a2d..01722dc9b 100644 --- a/lib/odp-execute-private.c +++ b/lib/odp-execute-private.c @@ -30,6 +30,12 @@ VLOG_DEFINE_THIS_MODULE(odp_execute_impl); static int active_action_impl_index; static struct odp_execute_action_impl action_impls[] = { + [ACTION_IMPL_AUTOVALIDATOR] = { + .available = false, + .name = "autovalidator", + .init_func = action_autoval_init, + }, + [ACTION_IMPL_SCALAR] = { .available = false, .name = "scalar", @@ -104,3 +110,96 @@ odp_execute_action_init(void) } } } + +/* Init sequence required to be scalar first to pick up the default scalar +* implementations, allowing over-riding of the optimized functions later. +*/ +BUILD_ASSERT_DECL(ACTION_IMPL_SCALAR == 0); +BUILD_ASSERT_DECL(ACTION_IMPL_AUTOVALIDATOR == 1); + +/* Loop over packets, and validate each one for the given action. */ +static void +action_autoval_generic(struct dp_packet_batch *batch, const struct nlattr *a) +{ + struct odp_execute_action_impl *scalar = &action_impls[ACTION_IMPL_SCALAR]; + enum ovs_action_attr attr_type = nl_attr_type(a); + struct dp_packet_batch original_batch; + bool failed = false; + + dp_packet_batch_clone(&original_batch, batch); + + scalar->funcs[attr_type](batch, a); + + for (int impl = ACTION_IMPL_BEGIN; impl < ACTION_IMPL_MAX; impl++) { + /* Clone original batch and execute implementation under test. */ + struct dp_packet_batch test_batch; + + dp_packet_batch_clone(&test_batch, &original_batch); + action_impls[impl].funcs[attr_type](&test_batch, a); + + /* Loop over implementations, checking each one. */ + for (int pidx = 0; pidx < original_batch.count; pidx++) { + struct dp_packet *good_pkt = batch->packets[pidx]; + struct dp_packet *test_pkt = test_batch.packets[pidx]; + + struct ds log_msg = DS_EMPTY_INITIALIZER; + + /* Compare packet length and payload contents. */ + bool eq = dp_packet_equal(good_pkt, test_pkt); + + if (!eq) { + ds_put_format(&log_msg, "Packet: %d\nAction : ", pidx); + format_odp_actions(&log_msg, a, a->nla_len, NULL); + ds_put_format(&log_msg, "\nGood hex:\n"); + ds_put_hex_dump(&log_msg, dp_packet_data(good_pkt), + dp_packet_size(good_pkt), 0, false); + ds_put_format(&log_msg, "Test hex:\n"); + ds_put_hex_dump(&log_msg, dp_packet_data(test_pkt), + dp_packet_size(test_pkt), 0, false); + + failed = true; + } + + /* Compare offsets and RSS */ + if (!dp_packet_compare_offsets(good_pkt, test_pkt, &log_msg)) { + failed = true; + } + + if (dp_packet_rss_valid(good_pkt)) { + uint32_t good_hash = dp_packet_get_rss_hash(good_pkt); + uint32_t test_hash = dp_packet_get_rss_hash(test_pkt); + + if (good_hash != test_hash) { + ds_put_format(&log_msg, + "Autovalidation rss hash failed\n"); + ds_put_format(&log_msg, "Good RSS hash : %u\n", good_hash); + ds_put_format(&log_msg, "Test RSS hash : %u\n", test_hash); + + failed = true; + } + } + + if (failed) { + VLOG_ERR("Autovalidation of %s failed. Details:\n%s", + action_impls[impl].name, ds_cstr(&log_msg)); + ds_destroy(&log_msg); + failed = false; + } + } + dp_packet_delete_batch(&test_batch, true); + } + dp_packet_delete_batch(&original_batch, true); +} + +int +action_autoval_init(struct odp_execute_action_impl *self) +{ + /* Set function pointers for actions that can be applied directly, these + * are identified by OVS_ACTION_ATTR_*. */ + for (int i = 0; i < __OVS_ACTION_ATTR_MAX; i++) { + if (action_impls[ACTION_IMPL_SCALAR].funcs[i]) { + self->funcs[i] = action_autoval_generic; + } + } + return 0; +} diff --git a/lib/odp-execute-private.h b/lib/odp-execute-private.h index ae06fbc09..074a8d67e 100644 --- a/lib/odp-execute-private.h +++ b/lib/odp-execute-private.h @@ -54,6 +54,7 @@ struct odp_execute_action_impl { /* Order of Actions implementations. */ enum odp_execute_action_impl_idx { ACTION_IMPL_SCALAR, + ACTION_IMPL_AUTOVALIDATOR, /* See ACTION_IMPL_BEGIN below, for "first to-be-validated" impl. * Do not change the autovalidator position in this list without updating * the define below. @@ -64,6 +65,9 @@ enum odp_execute_action_impl_idx { /* Index to start verifying implementations from. */ BUILD_ASSERT_DECL(ACTION_IMPL_SCALAR == 0); +BUILD_ASSERT_DECL(ACTION_IMPL_AUTOVALIDATOR == 1); + +#define ACTION_IMPL_BEGIN (ACTION_IMPL_AUTOVALIDATOR + 1) /* Odp execute init handles setting up the state of the actions functions at * initialization time. It cannot return errors, as it must always succeed in @@ -78,4 +82,6 @@ int odp_action_scalar_init(struct odp_execute_action_impl *self); struct odp_execute_action_impl * odp_execute_action_set(const char *name); +int action_autoval_init(struct odp_execute_action_impl *self); + #endif /* ODP_EXTRACT_PRIVATE */ From patchwork Thu Jul 7 15:38:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Finn, Emma" X-Patchwork-Id: 1653677 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=fjEG+MUS; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lf0vl68kqz9s2R for ; Fri, 8 Jul 2022 01:39:31 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 77A95410C7; Thu, 7 Jul 2022 15:39:29 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 77A95410C7 Authentication-Results: smtp2.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=fjEG+MUS X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YOWh14iqOIHC; Thu, 7 Jul 2022 15:39:27 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id 6730D41083; Thu, 7 Jul 2022 15:39:25 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 6730D41083 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 03CC6C0070; Thu, 7 Jul 2022 15:39:25 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 409C9C002D for ; Thu, 7 Jul 2022 15:39:22 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 321C44105F for ; Thu, 7 Jul 2022 15:39:21 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 321C44105F X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id K0z4hdk5DPk8 for ; Thu, 7 Jul 2022 15:39:19 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org AB23D41052 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by smtp2.osuosl.org (Postfix) with ESMTPS id AB23D41052 for ; Thu, 7 Jul 2022 15:39:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657208358; x=1688744358; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=TrUKKUYbJke+Hs8YZPsrq+0HlF19Z6R+ztXQdaPgRcQ=; b=fjEG+MUS/iXGMqDaRhf4iiSpjBUthdyJASXzzgwbmEeVtc2dsVU2V0hv g7jnO1EOoHrTwNLH8UFPLKpCFTUGMOAa932S/L04MBlTd4hUuHkOS7tuF UapQRPycdtr399tAQj7pRSTtLmGoLeNgAuBRw+gmaD+UcrPFH1VMOyFeB apAaDVKsHCo6slujVB6KOkTl/vip560Pr/rHer8bzI3DgXSLxuQo+MWUy rZQYvFgw+oB2HkZJLsub8FUkoM22UuaGHhzDH8L4/gfsmsk4PkGMaCfEn 1cHrkN67lTEFKvwP9BDFpcBR4P56W9wXv4l/dNSJCKxtAALgj/jcghy2j g==; X-IronPort-AV: E=McAfee;i="6400,9594,10400"; a="267083970" X-IronPort-AV: E=Sophos;i="5.92,253,1650956400"; d="scan'208";a="267083970" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jul 2022 08:39:18 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,253,1650956400"; d="scan'208";a="920633455" Received: from silpixa00401384.ir.intel.com ([10.243.22.75]) by fmsmga005.fm.intel.com with ESMTP; 07 Jul 2022 08:39:14 -0700 From: Emma Finn To: dev@openvswitch.org, echaudro@redhat.com, harry.van.haaren@intel.com, kumar.amber@intel.com Date: Thu, 7 Jul 2022 15:38:54 +0000 Message-Id: <20220707153900.3147694-5-emma.finn@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220707153900.3147694-1-emma.finn@intel.com> References: <20220614115743.1143341-2-emma.finn@intel.com> <20220707153900.3147694-1-emma.finn@intel.com> MIME-Version: 1.0 Cc: i.maximets@ovn.org Subject: [ovs-dev] [v8 04/10] odp-execute: Add command to switch action implementation. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This commit adds a new command to allow the user to switch the active action implementation at runtime. Usage: $ ovs-appctl odp-execute/action-impl-set scalar This commit also adds a new command to retrieve the list of available action implementations. This can be used by to check what implementations of actions are available and what implementation is active during runtime. Usage: $ ovs-appctl odp-execute/action-impl-show Added separate test-case for ovs-actions show/set commands: PMD - ovs-actions configuration Signed-off-by: Emma Finn Signed-off-by: Kumar Amber Signed-off-by: Sunil Pai G Co-authored-by: Kumar Amber Co-authored-by: Sunil Pai G Acked-by: Harry van Haaren --- NEWS | 2 ++ lib/automake.mk | 1 + lib/dpif-netdev.c | 1 + lib/odp-execute-private.c | 12 ++++++++++ lib/odp-execute-private.h | 2 ++ lib/odp-execute-unixctl.man | 10 +++++++++ lib/odp-execute.c | 44 +++++++++++++++++++++++++++++++++++++ tests/pmd.at | 39 ++++++++++++++++++++++++++++++++ vswitchd/ovs-vswitchd.8.in | 1 + 9 files changed, 112 insertions(+) create mode 100644 lib/odp-execute-unixctl.man diff --git a/NEWS b/NEWS index 1d0364fff..a279cbc29 100644 --- a/NEWS +++ b/NEWS @@ -45,6 +45,8 @@ Post-v2.17.0 The old variant is kept for backward compatibility. * Add actions auto-validator function to compare different actions implementations against default implementation. + * Add command line option to switch between different actions + implementations available at run time. v2.17.0 - 17 Feb 2022 diff --git a/lib/automake.mk b/lib/automake.mk index 23ba4fab0..5c3b05f6b 100644 --- a/lib/automake.mk +++ b/lib/automake.mk @@ -584,6 +584,7 @@ MAN_FRAGMENTS += \ lib/netdev-dpdk-unixctl.man \ lib/dpif-netdev-unixctl.man \ lib/dpif-netlink-unixctl.man \ + lib/odp-execute-unixctl.man \ lib/ofp-version.man \ lib/ovs.tmac \ lib/ovs-replay.man \ diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 58be9d0f0..b618ecba0 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -63,6 +63,7 @@ #include "netdev-vport.h" #include "netlink.h" #include "odp-execute.h" +#include "odp-execute-private.h" #include "odp-util.h" #include "openvswitch/dynamic-string.h" #include "openvswitch/list.h" diff --git a/lib/odp-execute-private.c b/lib/odp-execute-private.c index 01722dc9b..442837fa5 100644 --- a/lib/odp-execute-private.c +++ b/lib/odp-execute-private.c @@ -67,6 +67,18 @@ odp_execute_action_set(const char *name) return NULL; } +void +odp_execute_action_get_info(struct ds *string) +{ + ds_put_cstr(string, "Available Actions implementations:\n"); + for (int i = 0; i < ACTION_IMPL_MAX; i++) { + ds_put_format(string, " %s (available: %s, active: %s)\n", + action_impls[i].name, + action_impls[i].available ? "Yes" : "No", + i == active_action_impl_index ? "Yes" : "No"); + } +} + void odp_execute_action_init(void) { diff --git a/lib/odp-execute-private.h b/lib/odp-execute-private.h index 074a8d67e..d6eebbf37 100644 --- a/lib/odp-execute-private.h +++ b/lib/odp-execute-private.h @@ -84,4 +84,6 @@ struct odp_execute_action_impl * odp_execute_action_set(const char *name); int action_autoval_init(struct odp_execute_action_impl *self); +void odp_execute_action_get_info(struct ds *name); + #endif /* ODP_EXTRACT_PRIVATE */ diff --git a/lib/odp-execute-unixctl.man b/lib/odp-execute-unixctl.man new file mode 100644 index 000000000..82d51e1d3 --- /dev/null +++ b/lib/odp-execute-unixctl.man @@ -0,0 +1,10 @@ +.SS "ODP-EXECUTE COMMANDS" +These commands manage the "odp-execute" component. + +.IP "\fBodp-execute/action-impl-show\fR +Lists the actions implementations that are available and highlights the +currently enabled one. +. +.IP "\fBodp-execute/action-impl-set\fR \fIaction_impl\fR" +Sets the action implementation to any available implementation. By default +"scalar" is used. diff --git a/lib/odp-execute.c b/lib/odp-execute.c index e6839df6d..6f8beae7d 100644 --- a/lib/odp-execute.c +++ b/lib/odp-execute.c @@ -39,6 +39,7 @@ #include "csum.h" #include "conntrack.h" #include "openvswitch/vlog.h" +#include "unixctl.h" VLOG_DEFINE_THIS_MODULE(odp_execute); COVERAGE_DEFINE(datapath_drop_sample_error); @@ -879,6 +880,48 @@ odp_actions_impl_set(const char *name) } +static void +action_impl_set(struct unixctl_conn *conn, int argc OVS_UNUSED, + const char *argv[] OVS_UNUSED, void *aux OVS_UNUSED) +{ + struct ds reply = DS_EMPTY_INITIALIZER; + + int err = odp_actions_impl_set(argv[1]); + if (err) { + ds_put_format(&reply, + "Error: unknown action implementation, %s, specified!\n", + argv[1]); + unixctl_command_reply_error(conn, ds_cstr(&reply)); + } else { + ds_put_format(&reply, "Action implementation set to %s.\n", argv[1]); + unixctl_command_reply(conn, ds_cstr(&reply)); + } + + ds_destroy(&reply); +} + +static void +action_impl_show(struct unixctl_conn *conn, int argc OVS_UNUSED, + const char *argv[] OVS_UNUSED, void *aux OVS_UNUSED) +{ + struct ds reply = DS_EMPTY_INITIALIZER; + + odp_execute_action_get_info(&reply); + unixctl_command_reply(conn, ds_cstr(&reply)); + ds_destroy(&reply); +} + +static void +odp_execute_unixctl_init(void) +{ + unixctl_command_register("odp-execute/action-impl-set", "name", + 1, 1, action_impl_set, + NULL); + unixctl_command_register("odp-execute/action-impl-show", "", + 0, 0, action_impl_show, + NULL); +} + void odp_execute_init(void) { @@ -886,6 +929,7 @@ odp_execute_init(void) if (ovsthread_once_start(&once)) { odp_execute_action_init(); odp_actions_impl_set("scalar"); + odp_execute_unixctl_init(); ovsthread_once_done(&once); } } diff --git a/tests/pmd.at b/tests/pmd.at index 4342c50e0..4ca926581 100644 --- a/tests/pmd.at +++ b/tests/pmd.at @@ -1192,3 +1192,42 @@ ovs-appctl: ovs-vswitchd: server returned an error OVS_VSWITCHD_STOP AT_CLEANUP + +AT_SETUP([PMD - ovs-actions configuration]) +OVS_VSWITCHD_START([], [], [], [--dummy-numa 0,0]) +AT_CHECK([ovs-vsctl add-port br0 p1 -- set Interface p1 type=dummy-pmd]) + +AT_CHECK([ovs-vsctl show], [], [stdout]) + +dnl Set the scalar first, so we always have the scalar impl as Active. +AT_CHECK([ovs-appctl odp-execute/action-impl-set scalar], [0], [dnl +Action implementation set to scalar. +]) +AT_CHECK([ovs-appctl odp-execute/action-impl-show | grep "scalar"], [], [dnl + scalar (available: Yes, active: Yes) +]) + +AT_CHECK([ovs-appctl odp-execute/action-impl-show | grep "autovalidator"], [], [dnl + autovalidator (available: Yes, active: No) +]) + +dnl Set the autovalidator impl to active. +AT_CHECK([ovs-appctl odp-execute/action-impl-set autovalidator], [0], [dnl +Action implementation set to autovalidator. +]) + +AT_CHECK([ovs-appctl odp-execute/action-impl-show | grep "scalar"], [], [dnl + scalar (available: Yes, active: No) +]) + +AT_CHECK([ovs-appctl odp-execute/action-impl-show | grep "autovalidator"], [], [dnl + autovalidator (available: Yes, active: Yes) +]) + +AT_CHECK([ovs-appctl odp-execute/action-impl-set invalid_implementation], [2], [], [dnl +Error: unknown action implementation, invalid_implementation, specified! +ovs-appctl: ovs-vswitchd: server returned an error +]) + +OVS_VSWITCHD_STOP(["/Failed setting action implementation to invalid_implementation/d"]) +AT_CLEANUP diff --git a/vswitchd/ovs-vswitchd.8.in b/vswitchd/ovs-vswitchd.8.in index 1a32402be..9569265fc 100644 --- a/vswitchd/ovs-vswitchd.8.in +++ b/vswitchd/ovs-vswitchd.8.in @@ -282,6 +282,7 @@ type). .so lib/dpif-netdev-unixctl.man .so lib/dpif-netlink-unixctl.man .so lib/netdev-dpdk-unixctl.man +.so lib/odp-execute-unixctl.man .so ofproto/ofproto-dpif-unixctl.man .so ofproto/ofproto-unixctl.man .so lib/vlog-unixctl.man From patchwork Thu Jul 7 15:38:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Finn, Emma" X-Patchwork-Id: 1653676 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=CT42LgAC; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lf0vl1mNBz9s0w for ; Fri, 8 Jul 2022 01:39:31 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 28011612AD; Thu, 7 Jul 2022 15:39:26 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 28011612AD Authentication-Results: smtp3.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=CT42LgAC X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NN9mMP9c5iuH; Thu, 7 Jul 2022 15:39:23 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id E80DF6129C; Thu, 7 Jul 2022 15:39:21 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org E80DF6129C Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id B37E7C0033; Thu, 7 Jul 2022 15:39:21 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1DC62C0033 for ; Thu, 7 Jul 2022 15:39:21 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id D23484105F for ; Thu, 7 Jul 2022 15:39:20 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org D23484105F Authentication-Results: smtp2.osuosl.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=CT42LgAC X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id v5sjm2p9SkLv for ; Thu, 7 Jul 2022 15:39:19 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 6C85B4102A Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by smtp2.osuosl.org (Postfix) with ESMTPS id 6C85B4102A for ; Thu, 7 Jul 2022 15:39:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657208359; x=1688744359; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=fKmhw+MaS3IbRkua0YV4RIWcrSeOLN2G1CRuH/k5Sqs=; b=CT42LgACgcmZdLNoEwdXuGh4QH8cerHBjQqU3UNySlntyRqOgmWPIcWE NeY/BDexIC5fUzMbZ4PrKD9H7RzE5r1odXgGmmjxFjB9bKalnwuXO0scR o9lTj3Xa164Ytjrvm739udIWMeBJFqJA3AvKCSazjZHZvPrSu96tbubIX 0d1Pe+kWNBLYVwF11hlAZ49XYZe/TNle/ZiN4ji+GR6ps7CthbiMZu2bh xUysA0GBkUgUH4K/OGD2dCjIJasrx6OuqyJvatZCthHot+ur/LrgKjQEK Tmt/ksroLbYq4rR3ODrUsa04eJoTc8ptkNdLpN4D8Xqdj8jA2FM+4Dh1G Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10400"; a="267083974" X-IronPort-AV: E=Sophos;i="5.92,253,1650956400"; d="scan'208";a="267083974" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jul 2022 08:39:19 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,253,1650956400"; d="scan'208";a="920633461" Received: from silpixa00401384.ir.intel.com ([10.243.22.75]) by fmsmga005.fm.intel.com with ESMTP; 07 Jul 2022 08:39:17 -0700 From: Emma Finn To: dev@openvswitch.org, echaudro@redhat.com, harry.van.haaren@intel.com, kumar.amber@intel.com Date: Thu, 7 Jul 2022 15:38:55 +0000 Message-Id: <20220707153900.3147694-6-emma.finn@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220707153900.3147694-1-emma.finn@intel.com> References: <20220614115743.1143341-2-emma.finn@intel.com> <20220707153900.3147694-1-emma.finn@intel.com> MIME-Version: 1.0 Cc: i.maximets@ovn.org Subject: [ovs-dev] [v8 05/10] dpif-netdev: Add configure option to enable actions autovalidator at build time. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" From: Kumar Amber This commit adds a new command to allow the user to enable the actions autovalidator by default at build time thus allowing for running unit test by default. $ ./configure --enable-actions-default-autovalidator Signed-off-by: Kumar Amber Acked-by: Harry van Haaren --- NEWS | 2 ++ acinclude.m4 | 21 +++++++++++++++++++++ configure.ac | 1 + lib/odp-execute.c | 4 ++++ 4 files changed, 28 insertions(+) diff --git a/NEWS b/NEWS index a279cbc29..607514874 100644 --- a/NEWS +++ b/NEWS @@ -47,6 +47,8 @@ Post-v2.17.0 implementations against default implementation. * Add command line option to switch between different actions implementations available at run time. + * Add build time configure command to enable auto-validator as default + actions implementation at build time. v2.17.0 - 17 Feb 2022 diff --git a/acinclude.m4 b/acinclude.m4 index b518aa624..e63494e1d 100644 --- a/acinclude.m4 +++ b/acinclude.m4 @@ -14,6 +14,27 @@ # See the License for the specific language governing permissions and # limitations under the License. +dnl Set OVS Actions Autovalidator as the default action implementation +at compile time? +dnl This enables automatically running all unit tests with all actions +dnl implementations. +AC_DEFUN([OVS_CHECK_ACTIONS_AUTOVALIDATOR], [ + AC_ARG_ENABLE([actions-default-autovalidator], + [AC_HELP_STRING([--enable-actions-default-autovalidator], + [Enable actions autovalidator as default + ovs actions implementation.])], + [autovalidator=yes],[autovalidator=no]) + AC_MSG_CHECKING([whether actions Autovalidator is default implementation]) + if test "$autovalidator" != yes; then + AC_MSG_RESULT([no]) + else + AC_DEFINE([ACTIONS_AUTOVALIDATOR_DEFAULT], [1], + [Autovalidator for actions is a default implementation.]) + AC_MSG_RESULT([yes]) + fi +]) + + dnl Set OVS MFEX Autovalidator as default miniflow extract at compile time? dnl This enables automatically running all unit tests with all MFEX dnl implementations. diff --git a/configure.ac b/configure.ac index 59ea0a281..ab8e1bd12 100644 --- a/configure.ac +++ b/configure.ac @@ -184,6 +184,7 @@ OVS_CONDITIONAL_CC_OPTION([-Wno-unused-parameter], [HAVE_WNO_UNUSED_PARAMETER]) OVS_ENABLE_WERROR OVS_ENABLE_SPARSE OVS_CTAGS_IDENTIFIERS +OVS_CHECK_ACTIONS_AUTOVALIDATOR OVS_CHECK_DPCLS_AUTOVALIDATOR OVS_CHECK_DPIF_AVX512_DEFAULT OVS_CHECK_MFEX_AUTOVALIDATOR diff --git a/lib/odp-execute.c b/lib/odp-execute.c index 6f8beae7d..49d92f012 100644 --- a/lib/odp-execute.c +++ b/lib/odp-execute.c @@ -928,7 +928,11 @@ odp_execute_init(void) static struct ovsthread_once once = OVSTHREAD_ONCE_INITIALIZER; if (ovsthread_once_start(&once)) { odp_execute_action_init(); +#ifdef ACTIONS_AUTOVALIDATOR_DEFAULT + odp_actions_impl_set("autovalidator"); +#else odp_actions_impl_set("scalar"); +#endif odp_execute_unixctl_init(); ovsthread_once_done(&once); } From patchwork Thu Jul 7 15:38:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Finn, Emma" X-Patchwork-Id: 1653678 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=G5OU2VC6; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lf0vr34yCz9s0w for ; Fri, 8 Jul 2022 01:39:36 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 430A3840FB; Thu, 7 Jul 2022 15:39:34 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 430A3840FB Authentication-Results: smtp1.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=G5OU2VC6 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mCVa0O_k7QJT; Thu, 7 Jul 2022 15:39:32 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTPS id 3775E8410A; Thu, 7 Jul 2022 15:39:31 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 3775E8410A Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 04491C0033; Thu, 7 Jul 2022 15:39:31 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 0882DC0033 for ; Thu, 7 Jul 2022 15:39:30 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id C72BF41040 for ; Thu, 7 Jul 2022 15:39:24 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org C72BF41040 Authentication-Results: smtp2.osuosl.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=G5OU2VC6 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id VCOXbsRKMjYF for ; Thu, 7 Jul 2022 15:39:21 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 8ED8D4107F Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by smtp2.osuosl.org (Postfix) with ESMTPS id 8ED8D4107F for ; Thu, 7 Jul 2022 15:39:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657208361; x=1688744361; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=I0SYhU5sROu6sXC3krdvU7J9fFbR0ffaMwBVfpWGOY4=; b=G5OU2VC6lCHeKXvUqzzeapANndx+kmvCqQO0LrXAubKL2SEc9oAc3E5Y BaZVhQuw82d0Bu8jXVBhxQuZZV7Y3WrQPQ+lG/hiklqiGt5OC30lwMnWQ aDk+WIS0FENdHFeU0Phgm2Rw9S8/MnxBeN4oNvHS6o1D88NJEDhNZ8rya Kf5aHD3aS8PVJFddlCyM3qyMNvHCTqaxwlq7qS9pmwm1f3SXEe0JYtHX4 9NCu6n3cPCO8eN4A7tpVXslkmp26BAAW+cCxZmbMKQGOhglgWbSchQGgF SzRrX+Ap62eD/Bzfnka5N8YjdsoFzNjkVsfgUYog9JdD9ZrgzFF79HZEI w==; X-IronPort-AV: E=McAfee;i="6400,9594,10400"; a="267083978" X-IronPort-AV: E=Sophos;i="5.92,253,1650956400"; d="scan'208";a="267083978" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jul 2022 08:39:21 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,253,1650956400"; d="scan'208";a="920633469" Received: from silpixa00401384.ir.intel.com ([10.243.22.75]) by fmsmga005.fm.intel.com with ESMTP; 07 Jul 2022 08:39:19 -0700 From: Emma Finn To: dev@openvswitch.org, echaudro@redhat.com, harry.van.haaren@intel.com, kumar.amber@intel.com Date: Thu, 7 Jul 2022 15:38:56 +0000 Message-Id: <20220707153900.3147694-7-emma.finn@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220707153900.3147694-1-emma.finn@intel.com> References: <20220614115743.1143341-2-emma.finn@intel.com> <20220707153900.3147694-1-emma.finn@intel.com> MIME-Version: 1.0 Cc: i.maximets@ovn.org Subject: [ovs-dev] [v8 06/10] odp-execute: Add ISA implementation of actions. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This commit adds the AVX512 implementation of the action functionality. Usage: $ ovs-appctl odp-execute/action-impl-set avx512 Signed-off-by: Emma Finn Acked-by: Harry van Haaren --- Documentation/ref/ovs-actions.7.rst | 26 +++++++++++++++++ Documentation/topics/testing.rst | 24 ++++++++++------ NEWS | 1 + lib/automake.mk | 6 +++- lib/cpu.c | 1 + lib/cpu.h | 1 + lib/odp-execute-avx512.c | 32 +++++++++++++++++++++ lib/odp-execute-private.c | 43 +++++++++++++++++++++++++++++ lib/odp-execute-private.h | 8 ++++++ 9 files changed, 133 insertions(+), 9 deletions(-) create mode 100644 lib/odp-execute-avx512.c diff --git a/Documentation/ref/ovs-actions.7.rst b/Documentation/ref/ovs-actions.7.rst index b59b7634f..2410acc4a 100644 --- a/Documentation/ref/ovs-actions.7.rst +++ b/Documentation/ref/ovs-actions.7.rst @@ -125,6 +125,32 @@ the one added to the set later replaces the earlier action: An action set may only contain the actions listed above. +Actions Implementations (Experimental) +-------------------------------------- + +Actions are used in OpenFlow flows to describe what to do when the flow +matches a packet. Just like with the datapath interface, SIMD instructions +with the userspace datapath can be applied to the action implementation to +improve performance. + +OVS provides multiple implementations of the actions. +Available implementations can be listed with the following command:: + + $ ovs-appctl odp-execute/action-impl-show + Available Actions implementations: + scalar (available: Yes, active: Yes) + autovalidator (available: Yes, active: No) + avx512 (available: Yes, active: No) + +By default, ``scalar`` is used. Implementations can be selected by +name:: + + $ ovs-appctl odp-execute/action-impl-set avx512 + Action implementation set to avx512. + + $ ovs-appctl odp-execute/action-impl-set scalar + Action implementation set to scalar. + Error Handling -------------- diff --git a/Documentation/topics/testing.rst b/Documentation/topics/testing.rst index c15d5b38f..a6c747b18 100644 --- a/Documentation/topics/testing.rst +++ b/Documentation/topics/testing.rst @@ -361,12 +361,12 @@ testsuite. Userspace datapath: Testing and Validation of CPU-specific Optimizations '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' -As multiple versions of the datapath classifier and packet parsing functions -can co-exist, each with different CPU ISA optimizations, it is important to -validate that they all give the exact same results. To easily test all the -implementations, an ``autovalidator`` implementation of them exists. This -implementation runs all other available implementations, and verifies that the -results are identical. +As multiple versions of the datapath classifier, packet parsing functions and +actions can co-exist, each with different CPU ISA optimizations, it is +important to validate that they all give the exact same results. To easily +test all the implementations, an ``autovalidator`` implementation of them +exists. This implementation runs all other available implementations, and +verifies that the results are identical. Running the OVS unit tests with the autovalidator enabled ensures all implementations provide the same results. Note that the performance of the @@ -382,18 +382,26 @@ To set the autovalidator for the packet parser, use this command:: $ ovs-appctl dpif-netdev/miniflow-parser-set autovalidator +To set the autovalidator for actions, use this command:: + + $ ovs-appctl odp-execute/action-impl-set autovalidator + To run the OVS unit test suite with the autovalidator as the default implementation, it is required to recompile OVS. During the recompilation, the default priority of the `autovalidator` implementation is set to the -maximum priority, ensuring every test will be run with every implementation:: +maximum priority, ensuring every test will be run with every implementation. +Priority is only related to mfex autovalidator and not the actions +autovalidator.:: - $ ./configure --enable-autovalidator --enable-mfex-default-autovalidator + $ ./configure --enable-autovalidator --enable-mfex-default-autovalidator \ + --enable-actions-default-autovalidator The following line should be seen in the configuration log when the above options are used:: checking whether DPCLS Autovalidator is default implementation... yes checking whether MFEX Autovalidator is default implementation... yes + checking whether actions Autovalidator is default implementation... yes Compile OVS in debug mode to have `ovs_assert` statements error out if there is a mis-match in the datapath classifier lookup or packet parser diff --git a/NEWS b/NEWS index 607514874..751951ac9 100644 --- a/NEWS +++ b/NEWS @@ -49,6 +49,7 @@ Post-v2.17.0 implementations available at run time. * Add build time configure command to enable auto-validator as default actions implementation at build time. + * Add AVX512 implementation of actions. v2.17.0 - 17 Feb 2022 diff --git a/lib/automake.mk b/lib/automake.mk index 5c3b05f6b..e6335ccac 100644 --- a/lib/automake.mk +++ b/lib/automake.mk @@ -31,6 +31,9 @@ lib_LTLIBRARIES += lib/libopenvswitchavx512.la lib_libopenvswitch_la_LIBADD += lib/libopenvswitchavx512.la lib_libopenvswitchavx512_la_CFLAGS = \ -mavx512f \ + -mavx512bw \ + -mavx512vl \ + -mavx512dq \ -mbmi \ -mbmi2 \ -fPIC \ @@ -44,7 +47,8 @@ lib_libopenvswitchavx512_la_CFLAGS += \ -mavx512vl lib_libopenvswitchavx512_la_SOURCES += \ lib/dpif-netdev-extract-avx512.c \ - lib/dpif-netdev-lookup-avx512-gather.c + lib/dpif-netdev-lookup-avx512-gather.c \ + lib/odp-execute-avx512.c endif # HAVE_AVX512VL endif # HAVE_AVX512BW lib_libopenvswitchavx512_la_LDFLAGS = \ diff --git a/lib/cpu.c b/lib/cpu.c index 2df003c51..0292f715e 100644 --- a/lib/cpu.c +++ b/lib/cpu.c @@ -53,6 +53,7 @@ X86_ISA(X86_EXT_FEATURES_LEAF, EBX, 16, OVS_CPU_ISA_X86_AVX512F) X86_ISA(X86_EXT_FEATURES_LEAF, EBX, 30, OVS_CPU_ISA_X86_AVX512BW) X86_ISA(X86_EXT_FEATURES_LEAF, ECX, 1, OVS_CPU_ISA_X86_AVX512VBMI) X86_ISA(X86_EXT_FEATURES_LEAF, ECX, 14, OVS_CPU_ISA_X86_VPOPCNTDQ) +X86_ISA(X86_EXT_FEATURES_LEAF, EBX, 31, OVS_CPU_ISA_X86_AVX512VL) #endif bool diff --git a/lib/cpu.h b/lib/cpu.h index 92897bb71..3215229bc 100644 --- a/lib/cpu.h +++ b/lib/cpu.h @@ -25,6 +25,7 @@ enum ovs_cpu_isa { OVS_CPU_ISA_X86_AVX512F, OVS_CPU_ISA_X86_AVX512BW, OVS_CPU_ISA_X86_AVX512VBMI, + OVS_CPU_ISA_X86_AVX512VL, OVS_CPU_ISA_X86_VPOPCNTDQ, OVS_CPU_ISA_X86_LAST = OVS_CPU_ISA_X86_VPOPCNTDQ, }; diff --git a/lib/odp-execute-avx512.c b/lib/odp-execute-avx512.c new file mode 100644 index 000000000..33c9078cf --- /dev/null +++ b/lib/odp-execute-avx512.c @@ -0,0 +1,32 @@ +/* + * Copyright (c) 2022 Intel. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include +#include + +#include "dp-packet.h" +#include "immintrin.h" +#include "odp-execute-private.h" +#include "odp-netlink.h" +#include "openvswitch/vlog.h" + +int +action_avx512_init(struct odp_execute_action_impl *self OVS_UNUSED) +{ + /* Set function pointers for actions that can be applied directly, these + * are identified by OVS_ACTION_ATTR_*. */ + return 0; +} diff --git a/lib/odp-execute-private.c b/lib/odp-execute-private.c index 442837fa5..d99a94a93 100644 --- a/lib/odp-execute-private.c +++ b/lib/odp-execute-private.c @@ -19,6 +19,7 @@ #include #include +#include "cpu.h" #include "dpdk.h" #include "dp-packet.h" #include "odp-execute-private.h" @@ -29,6 +30,40 @@ VLOG_DEFINE_THIS_MODULE(odp_execute_impl); static int active_action_impl_index; +#ifdef ACTION_IMPL_AVX512_CHECK +/* Probe functions to check ISA requirements. */ +static bool +action_avx512_isa_probe(void) +{ + static enum ovs_cpu_isa isa_required[] = { + OVS_CPU_ISA_X86_AVX512F, + OVS_CPU_ISA_X86_AVX512BW, + OVS_CPU_ISA_X86_BMI2, + OVS_CPU_ISA_X86_AVX512VL, + }; + + for (int i = 0; i < ARRAY_SIZE(isa_required); i++) { + if (!cpu_has_isa(isa_required[i])) { + return false; + } + } + + return true; +} + +static int +action_avx512_probe(struct odp_execute_action_impl *self) +{ + if (!action_avx512_isa_probe()) { + return -ENOTSUP; + } else { + action_avx512_init(self); + } + + return 0; +} +#endif + static struct odp_execute_action_impl action_impls[] = { [ACTION_IMPL_AUTOVALIDATOR] = { .available = false, @@ -41,6 +76,14 @@ static struct odp_execute_action_impl action_impls[] = { .name = "scalar", .init_func = odp_action_scalar_init, }, + +#ifdef ACTION_IMPL_AVX512_CHECK + [ACTION_IMPL_AVX512] = { + .available = false, + .name = "avx512", + .init_func = action_avx512_probe, + }, +#endif }; static void diff --git a/lib/odp-execute-private.h b/lib/odp-execute-private.h index d6eebbf37..3ece71e7b 100644 --- a/lib/odp-execute-private.h +++ b/lib/odp-execute-private.h @@ -22,6 +22,9 @@ #include "odp-netlink.h" #include "ovs-atomic.h" +#define ACTION_IMPL_AVX512_CHECK (__x86_64__ && HAVE_AVX512F \ + && HAVE_LD_AVX512_GOOD && __SSE4_2__) + /* Forward declaration for typedef. */ struct odp_execute_action_impl; @@ -59,6 +62,9 @@ enum odp_execute_action_impl_idx { * Do not change the autovalidator position in this list without updating * the define below. */ +#ifdef ACTION_IMPL_AVX512_CHECK + ACTION_IMPL_AVX512, +#endif ACTION_IMPL_MAX, }; @@ -84,6 +90,8 @@ struct odp_execute_action_impl * odp_execute_action_set(const char *name); int action_autoval_init(struct odp_execute_action_impl *self); +int action_avx512_init(struct odp_execute_action_impl *self OVS_UNUSED); + void odp_execute_action_get_info(struct ds *name); #endif /* ODP_EXTRACT_PRIVATE */ From patchwork Thu Jul 7 15:38:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Finn, Emma" X-Patchwork-Id: 1653679 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=AlnwjrLv; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lf0w017jBz9s0w for ; Fri, 8 Jul 2022 01:39:44 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 32D9384111; Thu, 7 Jul 2022 15:39:42 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 32D9384111 Authentication-Results: smtp1.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=AlnwjrLv X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id O02J8eSZTq_P; Thu, 7 Jul 2022 15:39:40 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTPS id 41B1E8412D; Thu, 7 Jul 2022 15:39:38 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 41B1E8412D Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 16127C007D; Thu, 7 Jul 2022 15:39:38 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 482FCC002D for ; Thu, 7 Jul 2022 15:39:37 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 18B244108D for ; Thu, 7 Jul 2022 15:39:26 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 18B244108D Authentication-Results: smtp2.osuosl.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=AlnwjrLv X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id uuCF26Hibvr2 for ; Thu, 7 Jul 2022 15:39:24 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 695C341051 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by smtp2.osuosl.org (Postfix) with ESMTPS id 695C341051 for ; Thu, 7 Jul 2022 15:39:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657208363; x=1688744363; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=I28Xqu0fLqB1JnecWG6QKXkevDC0LUbvhL8letcENLE=; b=AlnwjrLvYncWsd3pthLIIiGMZMUTmAB4EcMJvAUqKks1cllZEsRN1TsJ 7xdpkfofpirEaN4lfO/A+QrTilkeea7v9fafOfn+qSWAPjdYUSm3erMWt HsK1ZmzNkkFSWAhr4UeFPtRHQ1eSZc+kTX54ULLACHftIaJAgjJhODAmM HU1sD41/5Spv9SZsBrCv+ja9z6Yn79/1deCK50KDstLn05U3jyYyJR9ma wcyv02TAGpw9Ny8KFUcNEelNzmcNQIueug8gAelFb5035efu9wJufo71U 1cOpK+8YMEbUrUq5ofSjt/PenhGr0utu8V70VkIOUhMyazjP832YhrWHb Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10400"; a="267083979" X-IronPort-AV: E=Sophos;i="5.92,253,1650956400"; d="scan'208";a="267083979" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jul 2022 08:39:23 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,253,1650956400"; d="scan'208";a="920633476" Received: from silpixa00401384.ir.intel.com ([10.243.22.75]) by fmsmga005.fm.intel.com with ESMTP; 07 Jul 2022 08:39:21 -0700 From: Emma Finn To: dev@openvswitch.org, echaudro@redhat.com, harry.van.haaren@intel.com, kumar.amber@intel.com Date: Thu, 7 Jul 2022 15:38:57 +0000 Message-Id: <20220707153900.3147694-8-emma.finn@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220707153900.3147694-1-emma.finn@intel.com> References: <20220614115743.1143341-2-emma.finn@intel.com> <20220707153900.3147694-1-emma.finn@intel.com> MIME-Version: 1.0 Cc: i.maximets@ovn.org Subject: [ovs-dev] [v8 07/10] odp-execute: Add ISA implementation of pop_vlan action. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This commit adds the AVX512 implementation of the pop_vlan action. Signed-off-by: Emma Finn --- lib/odp-execute-avx512.c | 153 +++++++++++++++++++++++++++++++++++++- lib/odp-execute-private.h | 2 +- 2 files changed, 153 insertions(+), 2 deletions(-) diff --git a/lib/odp-execute-avx512.c b/lib/odp-execute-avx512.c index 33c9078cf..8e42a2595 100644 --- a/lib/odp-execute-avx512.c +++ b/lib/odp-execute-avx512.c @@ -14,6 +14,11 @@ * limitations under the License. */ +#ifdef __x86_64__ +/* Sparse cannot handle the AVX512 instructions. */ +#if !defined(__CHECKER__) + + #include #include @@ -23,10 +28,156 @@ #include "odp-netlink.h" #include "openvswitch/vlog.h" +VLOG_DEFINE_THIS_MODULE(odp_execute_avx512); + +/* The below three build asserts make sure that l2_5_ofs, l3_ofs, and l4_ofs + * fields remain in the same order and offset to l2_padd_size. This is needed + * as the avx512_dp_packet_resize_l2() function will manipulate those fields at + * a fixed memory index based on the l2_padd_size offset. */ +BUILD_ASSERT_DECL(offsetof(struct dp_packet, l2_pad_size) + + MEMBER_SIZEOF(struct dp_packet, l2_pad_size) == + offsetof(struct dp_packet, l2_5_ofs)); + +BUILD_ASSERT_DECL(offsetof(struct dp_packet, l2_5_ofs) + + MEMBER_SIZEOF(struct dp_packet, l2_5_ofs) == + offsetof(struct dp_packet, l3_ofs)); + +BUILD_ASSERT_DECL(offsetof(struct dp_packet, l3_ofs) + + MEMBER_SIZEOF(struct dp_packet, l3_ofs) == + offsetof(struct dp_packet, l4_ofs)); + +/* The below build assert makes sure it's safe to read/write 128-bits starting + * at the l2_pad_size location. */ +BUILD_ASSERT_DECL(sizeof(struct dp_packet) - + offsetof(struct dp_packet, l2_pad_size) >= sizeof(__m128i)); + +static inline void ALWAYS_INLINE +avx512_dp_packet_resize_l2(struct dp_packet *b, int resize_by_bytes) +{ + /* Update packet size/data pointers, same as the scalar implementation. */ + if (resize_by_bytes >= 0) { + dp_packet_push_uninit(b, resize_by_bytes); + } else { + dp_packet_pull(b, -resize_by_bytes); + } + + /* The next step is to update the l2_5_ofs, l3_ofs and l4_ofs fields which + * the scalar implementation does with the dp_packet_adjust_layer_offset() + * function. */ + + /* Set the v_zero register to all zero's. */ + const __m128i v_zeros = _mm_setzero_si128(); + + /* Set the v_u16_max register to all one's. */ + const __m128i v_u16_max = _mm_cmpeq_epi16(v_zeros, v_zeros); + + /* Each lane represents 16 bits in a 12-bit register. In this case the + * first three 16-bit values, which will map to the l2_5_ofs, l3_ofs and + * l4_ofs fields. */ + const uint8_t k_lanes = 0b1110; + + /* Set all 16-bit words in the 128-bits v_offset register to the value we + * need to add/substract from the l2_5_ofs, l3_ofs, and l4_ofs fields. */ + __m128i v_offset = _mm_set1_epi16(abs(resize_by_bytes)); + + /* Load 128 bits from the dp_packet structure starting at the l2_pad_size + * offset. */ + void *adjust_ptr = &b->l2_pad_size; + __m128i v_adjust_src = _mm_loadu_si128(adjust_ptr); + + /* Here is the tricky part, we only need to update the value of the three + * fields if they are not UINT16_MAX. The following function will return + * a mask of lanes (read fields) that are not UINT16_MAX. It will do this + * by comparing only the lanes we requested, k_lanes, and if they match + * v_u16_max, the bit will be set. */ + __mmask8 k_cmp = _mm_mask_cmpneq_epu16_mask(k_lanes, v_adjust_src, + v_u16_max); + + /* Based on the bytes adjust (positive, or negative) it will do the actual + * add or subtraction. These functions will only operate on the lanes + * (fields) requested based on k_cmp, i.e: + * k_cmp = [l2_5_ofs, l3_ofs, l4_ofs] + * for field in kcmp + * v_adjust_src[field] = v_adjust_src[field] + v_offset + */ + __m128i v_adjust_wip; + + if (resize_by_bytes >= 0) { + v_adjust_wip = _mm_mask_add_epi16(v_adjust_src, k_cmp, + v_adjust_src, v_offset); + } else { + v_adjust_wip = _mm_mask_sub_epi16(v_adjust_src, k_cmp, + v_adjust_src, v_offset); + } + + /* Here we write back the full 128-bits. */ + _mm_storeu_si128(adjust_ptr, v_adjust_wip); +} + +/* This function performs the same operation on each packet in the batch as + * the scalar eth_pop_vlan() function. */ +static void +action_avx512_pop_vlan(struct dp_packet_batch *batch, + const struct nlattr *a OVS_UNUSED) +{ + struct dp_packet *packet; + + /* Set the v_zero register to all zero's. */ + const __m128i v_zeros = _mm_setzero_si128(); + + DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { + struct vlan_eth_header *veh = dp_packet_eth(packet); + + if (veh && dp_packet_size(packet) >= sizeof *veh && + eth_type_vlan(veh->veth_type)) { + + /* Load the first 128-bits of l2 header into the v_ether register. + * This result in the veth_dst/src and veth_type/tci of the + * vlan_eth_header structure to be loaded. */ + __m128i v_ether = _mm_loadu_si128((void *) veh); + + /* This creates a 256-bit value containing the first four fields + * of the vlan_eth_header plus 128 zero-bit. The result will be the + * lowest 128-bits after the right shift, hence we shift the data + * 128(zero)-bits minus the VLAN_HEADER_LEN, so we are left with + * only the veth_dst and veth_src fields. */ + __m128i v_realign = _mm_alignr_epi8(v_ether, v_zeros, + sizeof(__m128i) - + VLAN_HEADER_LEN); + + /* Write back the modified ethernet header. */ + _mm_storeu_si128((void *) veh, v_realign); + + /* As we removed the VLAN_HEADER we now need to adjust all the + * offsets. */ + avx512_dp_packet_resize_l2(packet, -VLAN_HEADER_LEN); + } + } +} + int -action_avx512_init(struct odp_execute_action_impl *self OVS_UNUSED) +action_avx512_init(struct odp_execute_action_impl *self) { /* Set function pointers for actions that can be applied directly, these * are identified by OVS_ACTION_ATTR_*. */ + self->funcs[OVS_ACTION_ATTR_POP_VLAN] = action_avx512_pop_vlan; return 0; } + +#endif /* Sparse */ + +#else /* __x86_64__ */ + +#include +#include "odp-execute-private.h" +/* Function itself is required to be called, even in e.g. 32-bit builds. + * This dummy init function ensures 32-bit builds succeed too. + */ + +int +action_avx512_init(struct odp_execute_action_impl *self OVS_UNUSED) +{ + return 0; +} + +#endif diff --git a/lib/odp-execute-private.h b/lib/odp-execute-private.h index 3ece71e7b..b3a10cd82 100644 --- a/lib/odp-execute-private.h +++ b/lib/odp-execute-private.h @@ -90,7 +90,7 @@ struct odp_execute_action_impl * odp_execute_action_set(const char *name); int action_autoval_init(struct odp_execute_action_impl *self); -int action_avx512_init(struct odp_execute_action_impl *self OVS_UNUSED); +int action_avx512_init(struct odp_execute_action_impl *self); void odp_execute_action_get_info(struct ds *name); From patchwork Thu Jul 7 15:38:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Finn, Emma" X-Patchwork-Id: 1653680 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=ChKUJd6/; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lf0w70vjBz9s0w for ; Fri, 8 Jul 2022 01:39:51 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 6465D41186; Thu, 7 Jul 2022 15:39:48 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 6465D41186 Authentication-Results: smtp2.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=ChKUJd6/ X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xpCKNUprQvIm; Thu, 7 Jul 2022 15:39:45 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id D25894115D; Thu, 7 Jul 2022 15:39:42 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org D25894115D Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 4E556C007E; Thu, 7 Jul 2022 15:39:42 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1B0A4C007D for ; Thu, 7 Jul 2022 15:39:41 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id BF823410C1 for ; Thu, 7 Jul 2022 15:39:27 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org BF823410C1 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ojVG_IwMtxLI for ; Thu, 7 Jul 2022 15:39:26 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org EB72F410BB Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by smtp2.osuosl.org (Postfix) with ESMTPS id EB72F410BB for ; Thu, 7 Jul 2022 15:39:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657208366; x=1688744366; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=sEX/bwPk1dLY/wPykIWKhErBhB4Dk4G/FzeBW2MJv38=; b=ChKUJd6/BpVpdV+vwOkmorkSlyHZ0fbI2QISS7CIlWEoPNedCAXLAqWo wqFwqZAMwjGj+viELg0OLKdEZ6kUTGChl0QCacpww05QFD+tLi+hfjlEq h3cg6HOedctZWQoZ5ii0KMKyvT2rzjZt6OQ48IfrA2gxuplsXw9lhIh1x wsRYwFG8aN8YXzeHOdyUu74/6MlueiOlxjH488qxpLAHyAR6utqaBUOql 477iAc8MqBrNQhZpR9T1pslQ+tdEOK0GsmCwr9T7YdKIs5mjYXjkk1JzM Myjg4c/7Dwmvtug04nkyNZjsTvL5gh0sCwzUQcx07ATGS1hsqloGiltgB A==; X-IronPort-AV: E=McAfee;i="6400,9594,10400"; a="267083983" X-IronPort-AV: E=Sophos;i="5.92,253,1650956400"; d="scan'208";a="267083983" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jul 2022 08:39:25 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,253,1650956400"; d="scan'208";a="920633486" Received: from silpixa00401384.ir.intel.com ([10.243.22.75]) by fmsmga005.fm.intel.com with ESMTP; 07 Jul 2022 08:39:23 -0700 From: Emma Finn To: dev@openvswitch.org, echaudro@redhat.com, harry.van.haaren@intel.com, kumar.amber@intel.com Date: Thu, 7 Jul 2022 15:38:58 +0000 Message-Id: <20220707153900.3147694-9-emma.finn@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220707153900.3147694-1-emma.finn@intel.com> References: <20220614115743.1143341-2-emma.finn@intel.com> <20220707153900.3147694-1-emma.finn@intel.com> MIME-Version: 1.0 Cc: i.maximets@ovn.org Subject: [ovs-dev] [v8 08/10] odp-execute: Add ISA implementation of push_vlan action. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This commit adds the AVX512 implementation of the push_vlan action. Signed-off-by: Emma Finn --- lib/odp-execute-avx512.c | 55 ++++++++++++++++++++++++++++++++++++++++ lib/odp-execute.c | 22 +++++++++------- 2 files changed, 68 insertions(+), 9 deletions(-) diff --git a/lib/odp-execute-avx512.c b/lib/odp-execute-avx512.c index 8e42a2595..6252cd3d0 100644 --- a/lib/odp-execute-avx512.c +++ b/lib/odp-execute-avx512.c @@ -155,12 +155,67 @@ action_avx512_pop_vlan(struct dp_packet_batch *batch, } } +/* This function performs the same operation on each packet in the batch as + * the scalar eth_push_vlan() function. */ +static void +action_avx512_push_vlan(struct dp_packet_batch *batch, const struct nlattr *a) +{ + struct dp_packet *packet; + const struct ovs_action_push_vlan *vlan = nl_attr_get(a); + ovs_be16 tpid, tci; + + /* This shuffle mask is used below, and each position tells where to + * move the bytes to. So here, the fourth byte in v_ether is moved to + * byte location 0 in v_shift. The fifth is moved to 1, etc., etc. + * The 0xFF is special it tells to fill that position with 0. + */ + static const uint8_t vlan_push_shuffle_mask[16] = { + 4, 5, 6, 7, 8, 9, 10, 11, + 12, 13, 14, 15, 0xFF, 0xFF, 0xFF, 0xFF + }; + + /* Load the shuffle mask in v_index. */ + __m128i v_index = _mm_loadu_si128((void *) vlan_push_shuffle_mask); + + DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { + tpid = vlan->vlan_tpid; + tci = vlan->vlan_tci; + + /* As we are about to insert the VLAN_HEADER we now need to adjust all + * the offsets. */ + avx512_dp_packet_resize_l2(packet, VLAN_HEADER_LEN); + + char *pkt_data = (char *) dp_packet_data(packet); + + /* Build up the VLAN TCI/TPID in a single uint32_t. */ + const uint16_t tci_proc = tci & htons(~VLAN_CFI); + const uint32_t tpid_tci = (tci_proc << 16) | tpid; + + /* Load the first 128-bits of the packet into the v_ether register. + * Note that this includes the 4 unused bytes (VLAN_HEADER_LEN). */ + __m128i v_ether = _mm_loadu_si128((void *) pkt_data); + + /* Move(shuffle) the veth_dst and veth_src data to create room for + * the vlan header. */ + __m128i v_shift = _mm_shuffle_epi8(v_ether, v_index); + + /* Copy(insert) the 32-bit VLAN header, tpid_tci, at the 3rd 32-bit + * word offset, i.e. ofssetof(vlan_eth_header, veth_type) */ + __m128i v_vlan_hdr = _mm_insert_epi32(v_shift, tpid_tci, 3); + + /* Write back the modified ethernet header. */ + _mm_storeu_si128((void *) pkt_data, v_vlan_hdr); + } +} + int action_avx512_init(struct odp_execute_action_impl *self) { /* Set function pointers for actions that can be applied directly, these * are identified by OVS_ACTION_ATTR_*. */ self->funcs[OVS_ACTION_ATTR_POP_VLAN] = action_avx512_pop_vlan; + self->funcs[OVS_ACTION_ATTR_PUSH_VLAN] = action_avx512_push_vlan; + return 0; } diff --git a/lib/odp-execute.c b/lib/odp-execute.c index 49d92f012..67d1e914b 100644 --- a/lib/odp-execute.c +++ b/lib/odp-execute.c @@ -846,6 +846,17 @@ action_pop_vlan(struct dp_packet_batch *batch, } } +static void +action_push_vlan(struct dp_packet_batch *batch, const struct nlattr *a) +{ + struct dp_packet *packet; + const struct ovs_action_push_vlan *vlan = nl_attr_get(a); + + DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { + eth_push_vlan(packet, vlan->vlan_tpid, vlan->vlan_tci); + } +} + /* Implementation of the scalar actions impl init function. Build up the * array of func ptrs here. */ @@ -855,6 +866,7 @@ odp_action_scalar_init(struct odp_execute_action_impl *self) /* Set function pointers for actions that can be applied directly, these * are identified by OVS_ACTION_ATTR_*. */ self->funcs[OVS_ACTION_ATTR_POP_VLAN] = action_pop_vlan; + self->funcs[OVS_ACTION_ATTR_PUSH_VLAN] = action_push_vlan; return 0; } @@ -1045,15 +1057,6 @@ odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, break; } - case OVS_ACTION_ATTR_PUSH_VLAN: { - const struct ovs_action_push_vlan *vlan = nl_attr_get(a); - - DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { - eth_push_vlan(packet, vlan->vlan_tpid, vlan->vlan_tci); - } - break; - } - case OVS_ACTION_ATTR_PUSH_MPLS: { const struct ovs_action_push_mpls *mpls = nl_attr_get(a); @@ -1206,6 +1209,7 @@ odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, case __OVS_ACTION_ATTR_MAX: /* The following actions are handled by the scalar implementation. */ case OVS_ACTION_ATTR_POP_VLAN: + case OVS_ACTION_ATTR_PUSH_VLAN: OVS_NOT_REACHED(); } From patchwork Thu Jul 7 15:38:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Finn, Emma" X-Patchwork-Id: 1653681 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=c+lKqXOr; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lf0wC345gz9s0w for ; Fri, 8 Jul 2022 01:39:55 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 3B0E24116C; Thu, 7 Jul 2022 15:39:53 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 3B0E24116C Authentication-Results: smtp2.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=c+lKqXOr X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wCtfHmQEHWng; Thu, 7 Jul 2022 15:39:50 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id 3388941169; Thu, 7 Jul 2022 15:39:45 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 3388941169 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 10845C0070; Thu, 7 Jul 2022 15:39:44 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 52040C0080 for ; Thu, 7 Jul 2022 15:39:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 54FB4410CC for ; Thu, 7 Jul 2022 15:39:30 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 54FB4410CC X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id uR6k6bYqhSzO for ; Thu, 7 Jul 2022 15:39:27 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 9C152410AE Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by smtp2.osuosl.org (Postfix) with ESMTPS id 9C152410AE for ; Thu, 7 Jul 2022 15:39:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657208367; x=1688744367; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=PYDXkg/enDlUr1oEwGYtrQgW9jpYwLyS9vLNN5PhUps=; b=c+lKqXOr+E6Wr27CruNS34tM3wt0MvpgAFXj3AHEt4GhS25wEUH0SgiC 2AN7H7MZyBT7wYIl+X2saIP5ZsgIvMlfQyRQNndyTIsiyTnt2XIMW2e3S HrdJcL/3PhRQFpEo2kNPB+JKH05dhORGR2Ds5tXJuFIe+0tg68i6hiu/f prUJ7g0Xzz53ni1x+/29RlCZT7zUzoU7WmOynbm7xJtWLvve9tuG/omC+ ur7IJAQ/E/SC7w7qd6al0Jis0ZOojL4c1YwFSSbdj4Az9SA0QIquNt1q4 JzMkCJttUMDWDG1BcWcSktEtlnmkMyNXHv5lXb4HyVgAxU2cUrAEhgGZH A==; X-IronPort-AV: E=McAfee;i="6400,9594,10400"; a="267083995" X-IronPort-AV: E=Sophos;i="5.92,253,1650956400"; d="scan'208";a="267083995" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jul 2022 08:39:27 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,253,1650956400"; d="scan'208";a="920633496" Received: from silpixa00401384.ir.intel.com ([10.243.22.75]) by fmsmga005.fm.intel.com with ESMTP; 07 Jul 2022 08:39:25 -0700 From: Emma Finn To: dev@openvswitch.org, echaudro@redhat.com, harry.van.haaren@intel.com, kumar.amber@intel.com Date: Thu, 7 Jul 2022 15:38:59 +0000 Message-Id: <20220707153900.3147694-10-emma.finn@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220707153900.3147694-1-emma.finn@intel.com> References: <20220614115743.1143341-2-emma.finn@intel.com> <20220707153900.3147694-1-emma.finn@intel.com> MIME-Version: 1.0 Cc: i.maximets@ovn.org Subject: [ovs-dev] [v8 09/10] odp-execute: Add ISA implementation of set_masked ETH X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This commit includes infrastructure changes for enabling set_masked_X actions and also adds support for the AVX512 implementation of the eth_set_addrs action. Signed-off-by: Emma Finn --- lib/odp-execute-avx512.c | 90 +++++++++++++++++++++++++++++++++++++++ lib/odp-execute-private.c | 14 ++++++ lib/odp-execute-private.h | 3 ++ lib/odp-execute.c | 49 +++++++++++---------- lib/odp-execute.h | 3 ++ 5 files changed, 137 insertions(+), 22 deletions(-) diff --git a/lib/odp-execute-avx512.c b/lib/odp-execute-avx512.c index 6252cd3d0..12eafc35e 100644 --- a/lib/odp-execute-avx512.c +++ b/lib/odp-execute-avx512.c @@ -24,6 +24,7 @@ #include "dp-packet.h" #include "immintrin.h" +#include "odp-execute.h" #include "odp-execute-private.h" #include "odp-netlink.h" #include "openvswitch/vlog.h" @@ -51,6 +52,16 @@ BUILD_ASSERT_DECL(offsetof(struct dp_packet, l3_ofs) + BUILD_ASSERT_DECL(sizeof(struct dp_packet) - offsetof(struct dp_packet, l2_pad_size) >= sizeof(__m128i)); +/* The below build assert makes sure the order of the fields needed by + * the set masked functions shuffle operations do not change. This should not + * happen as these are defined under the Linux uapi. */ +BUILD_ASSERT_DECL(offsetof(struct ovs_key_ethernet, eth_src) + + MEMBER_SIZEOF(struct ovs_key_ethernet, eth_src) == + offsetof(struct ovs_key_ethernet, eth_dst)); + +/* Array of callback functions, one for each masked operation. */ +odp_execute_action_cb impl_set_masked_funcs[__OVS_KEY_ATTR_MAX]; + static inline void ALWAYS_INLINE avx512_dp_packet_resize_l2(struct dp_packet *b, int resize_by_bytes) { @@ -208,6 +219,80 @@ action_avx512_push_vlan(struct dp_packet_batch *batch, const struct nlattr *a) } } +/* This function performs the same operation on each packet in the batch as + * the scalar odp_eth_set_addrs() function. */ +static void +action_avx512_eth_set_addrs(struct dp_packet_batch *batch, + const struct nlattr *a) +{ + const struct ovs_key_ethernet *key, *mask; + struct dp_packet *packet; + + a = nl_attr_get(a); + key = nl_attr_get(a); + mask = odp_get_key_mask(a, struct ovs_key_ethernet); + + /* Read the content of the key(src) and mask in the respective registers. + * We only load the src and dest addresses, which is only 96-bits and not + * 128-bits. */ + __m128i v_src = _mm_maskz_loadu_epi32(0x7,(void *) key); + __m128i v_mask = _mm_maskz_loadu_epi32(0x7, (void *) mask); + + + /* These shuffle masks are used below, and each position tells where to + * move the bytes to. So here, the fourth sixth byte in + * ovs_key_ethernet is moved to byte location 0 in v_src/v_mask. + * The seventh is moved to 1, etc., etc. + * This swap is needed to move the src and dest MAC addresses in the + * same order as in the ethernet packet. */ + static const uint8_t eth_shuffle[16] = { + 6, 7, 8, 9, 10, 11, 0, 1, + 2, 3, 4, 5, 0xFF, 0xFF, 0xFF, 0xFF + }; + + /* Load the shuffle mask in v_shuf. */ + __m128i v_shuf = _mm_loadu_si128((void *) eth_shuffle); + + /* Swap the key/mask src and dest addresses to the ethernet order. */ + v_src = _mm_shuffle_epi8(v_src, v_shuf); + v_mask = _mm_shuffle_epi8(v_mask, v_shuf); + + DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { + + struct eth_header *eh = dp_packet_eth(packet); + + if (!eh) { + continue; + } + + /* Load the first 128-bits of the packet into the v_ether register. */ + __m128i v_dst = _mm_loadu_si128((void *) eh); + + /* AND the v_mask to the packet data (v_dst). */ + __m128i dst_masked = _mm_andnot_si128(v_mask, v_dst); + + /* OR the new addresses (v_src) with the masked packet addresses + * (dst_masked). */ + __m128i res = _mm_or_si128(v_src, dst_masked); + + /* Write back the modified ethernet addresses. */ + _mm_storeu_si128((void *) eh, res); + } +} + +static void +action_avx512_set_masked(struct dp_packet_batch *batch, const struct nlattr *a) +{ + const struct nlattr *mask = nl_attr_get(a); + enum ovs_key_attr attr_type = nl_attr_type(mask); + + if (attr_type <= OVS_KEY_ATTR_MAX && impl_set_masked_funcs[attr_type]) { + impl_set_masked_funcs[attr_type](batch, a); + } else { + odp_execute_scalar_action(batch, a); + } +} + int action_avx512_init(struct odp_execute_action_impl *self) { @@ -215,6 +300,11 @@ action_avx512_init(struct odp_execute_action_impl *self) * are identified by OVS_ACTION_ATTR_*. */ self->funcs[OVS_ACTION_ATTR_POP_VLAN] = action_avx512_pop_vlan; self->funcs[OVS_ACTION_ATTR_PUSH_VLAN] = action_avx512_push_vlan; + self->funcs[OVS_ACTION_ATTR_SET_MASKED] = action_avx512_set_masked; + + /* Set function pointers for the individual operations supported by the + * SET_MASKED action. */ + impl_set_masked_funcs[OVS_KEY_ATTR_ETHERNET] = action_avx512_eth_set_addrs; return 0; } diff --git a/lib/odp-execute-private.c b/lib/odp-execute-private.c index d99a94a93..a6df84c90 100644 --- a/lib/odp-execute-private.c +++ b/lib/odp-execute-private.c @@ -22,6 +22,7 @@ #include "cpu.h" #include "dpdk.h" #include "dp-packet.h" +#include "odp-execute.h" #include "odp-execute-private.h" #include "odp-netlink.h" #include "odp-util.h" @@ -246,6 +247,19 @@ action_autoval_generic(struct dp_packet_batch *batch, const struct nlattr *a) dp_packet_delete_batch(&original_batch, true); } +void +odp_execute_scalar_action(struct dp_packet_batch *batch, + const struct nlattr *action) +{ + enum ovs_action_attr type = nl_attr_type(action); + + if (action_impls[ACTION_IMPL_SCALAR].funcs[type] && + type <= OVS_ACTION_ATTR_MAX) { + + action_impls[ACTION_IMPL_SCALAR].funcs[type](batch, action); + } +} + int action_autoval_init(struct odp_execute_action_impl *self) { diff --git a/lib/odp-execute-private.h b/lib/odp-execute-private.h index b3a10cd82..3fdcdc95b 100644 --- a/lib/odp-execute-private.h +++ b/lib/odp-execute-private.h @@ -94,4 +94,7 @@ int action_avx512_init(struct odp_execute_action_impl *self); void odp_execute_action_get_info(struct ds *name); +void odp_execute_scalar_action(struct dp_packet_batch *batch, + const struct nlattr *action); + #endif /* ODP_EXTRACT_PRIVATE */ diff --git a/lib/odp-execute.c b/lib/odp-execute.c index 67d1e914b..09c8e4401 100644 --- a/lib/odp-execute.c +++ b/lib/odp-execute.c @@ -562,8 +562,6 @@ odp_execute_set_action(struct dp_packet *packet, const struct nlattr *a) } } -#define get_mask(a, type) ((const type *)(const void *)(a + 1) + 1) - static void odp_execute_masked_set_action(struct dp_packet *packet, const struct nlattr *a) @@ -575,17 +573,17 @@ odp_execute_masked_set_action(struct dp_packet *packet, switch (type) { case OVS_KEY_ATTR_PRIORITY: md->skb_priority = nl_attr_get_u32(a) - | (md->skb_priority & ~*get_mask(a, uint32_t)); + | (md->skb_priority & ~*odp_get_key_mask(a, uint32_t)); break; case OVS_KEY_ATTR_SKB_MARK: md->pkt_mark = nl_attr_get_u32(a) - | (md->pkt_mark & ~*get_mask(a, uint32_t)); + | (md->pkt_mark & ~*odp_get_key_mask(a, uint32_t)); break; case OVS_KEY_ATTR_ETHERNET: odp_eth_set_addrs(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_ethernet)); + odp_get_key_mask(a, struct ovs_key_ethernet)); break; case OVS_KEY_ATTR_NSH: { @@ -595,27 +593,27 @@ odp_execute_masked_set_action(struct dp_packet *packet, case OVS_KEY_ATTR_IPV4: odp_set_ipv4(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_ipv4)); + odp_get_key_mask(a, struct ovs_key_ipv4)); break; case OVS_KEY_ATTR_IPV6: odp_set_ipv6(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_ipv6)); + odp_get_key_mask(a, struct ovs_key_ipv6)); break; case OVS_KEY_ATTR_TCP: odp_set_tcp(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_tcp)); + odp_get_key_mask(a, struct ovs_key_tcp)); break; case OVS_KEY_ATTR_UDP: odp_set_udp(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_udp)); + odp_get_key_mask(a, struct ovs_key_udp)); break; case OVS_KEY_ATTR_SCTP: odp_set_sctp(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_sctp)); + odp_get_key_mask(a, struct ovs_key_sctp)); break; case OVS_KEY_ATTR_MPLS: @@ -623,33 +621,33 @@ odp_execute_masked_set_action(struct dp_packet *packet, if (mh) { put_16aligned_be32(&mh->mpls_lse, nl_attr_get_be32(a) | (get_16aligned_be32(&mh->mpls_lse) - & ~*get_mask(a, ovs_be32))); + & ~*odp_get_key_mask(a, ovs_be32))); } break; case OVS_KEY_ATTR_ARP: set_arp(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_arp)); + odp_get_key_mask(a, struct ovs_key_arp)); break; case OVS_KEY_ATTR_ND: odp_set_nd(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_nd)); + odp_get_key_mask(a, struct ovs_key_nd)); break; case OVS_KEY_ATTR_ND_EXTENSIONS: odp_set_nd_ext(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_nd_extensions)); + odp_get_key_mask(a, struct ovs_key_nd_extensions)); break; case OVS_KEY_ATTR_DP_HASH: md->dp_hash = nl_attr_get_u32(a) - | (md->dp_hash & ~*get_mask(a, uint32_t)); + | (md->dp_hash & ~*odp_get_key_mask(a, uint32_t)); break; case OVS_KEY_ATTR_RECIRC_ID: md->recirc_id = nl_attr_get_u32(a) - | (md->recirc_id & ~*get_mask(a, uint32_t)); + | (md->recirc_id & ~*odp_get_key_mask(a, uint32_t)); break; case OVS_KEY_ATTR_TUNNEL: /* Masked data not supported for tunnel. */ @@ -857,6 +855,17 @@ action_push_vlan(struct dp_packet_batch *batch, const struct nlattr *a) } } +static void +action_set_masked(struct dp_packet_batch *batch, const struct nlattr *a) +{ + const struct nlattr *key = nl_attr_get(a); + struct dp_packet *packet; + + DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { + odp_execute_masked_set_action(packet, key); + } +} + /* Implementation of the scalar actions impl init function. Build up the * array of func ptrs here. */ @@ -867,6 +876,7 @@ odp_action_scalar_init(struct odp_execute_action_impl *self) * are identified by OVS_ACTION_ATTR_*. */ self->funcs[OVS_ACTION_ATTR_POP_VLAN] = action_pop_vlan; self->funcs[OVS_ACTION_ATTR_PUSH_VLAN] = action_push_vlan; + self->funcs[OVS_ACTION_ATTR_SET_MASKED] = action_set_masked; return 0; } @@ -1078,12 +1088,6 @@ odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, } break; - case OVS_ACTION_ATTR_SET_MASKED: - DP_PACKET_BATCH_FOR_EACH(i, packet, batch) { - odp_execute_masked_set_action(packet, nl_attr_get(a)); - } - break; - case OVS_ACTION_ATTR_SAMPLE: DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { odp_execute_sample(dp, packet, steal && last_action, a, @@ -1210,6 +1214,7 @@ odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, /* The following actions are handled by the scalar implementation. */ case OVS_ACTION_ATTR_POP_VLAN: case OVS_ACTION_ATTR_PUSH_VLAN: + case OVS_ACTION_ATTR_SET_MASKED: OVS_NOT_REACHED(); } diff --git a/lib/odp-execute.h b/lib/odp-execute.h index 0921ee924..2ba1ec5d2 100644 --- a/lib/odp-execute.h +++ b/lib/odp-execute.h @@ -46,4 +46,7 @@ void odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, const struct nlattr *actions, size_t actions_len, odp_execute_cb dp_execute_action); + +#define odp_get_key_mask(a, type) ((const type *)(const void *)(a + 1) + 1) + #endif From patchwork Thu Jul 7 15:39:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Finn, Emma" X-Patchwork-Id: 1653682 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=GyAM/VUZ; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lf0wS5Hg2z9s0w for ; Fri, 8 Jul 2022 01:40:08 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 6F9F7410CF; Thu, 7 Jul 2022 15:40:06 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 6F9F7410CF Authentication-Results: smtp2.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=GyAM/VUZ X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TUaCoDR1zg70; Thu, 7 Jul 2022 15:40:03 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id B20594115E; Thu, 7 Jul 2022 15:40:00 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org B20594115E Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 5AD23C0033; Thu, 7 Jul 2022 15:40:00 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 18617C007D for ; Thu, 7 Jul 2022 15:39:59 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 9571741135 for ; Thu, 7 Jul 2022 15:39:33 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 9571741135 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CKeg9MtFumjo for ; Thu, 7 Jul 2022 15:39:31 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org B5C52410CB Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by smtp2.osuosl.org (Postfix) with ESMTPS id B5C52410CB for ; Thu, 7 Jul 2022 15:39:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657208369; x=1688744369; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=zn1vzIzCYnZyFGsi5sM8urB5IzwLdt45A/UAkaz3I4Y=; b=GyAM/VUZcvpOnLUsHodG0duwSMA2h+8OAnvU6hy0i7pGGgUnzYOPeZlw kKuyvFA286kYU7j0U3grx0euVtbHHjDQaXnqVZsawH9WKalNmrXcVUFHS O4ge7arIBPKFrMVHWHyKSRI2LfwBlzJ0Zb9eEnMZ3+RpfQcWNUfiPq3Pq q0QwMMTwnqaYLCR/sNK8MxSHmwmp0AhA2Ob6jSnuWrL3iU8prSBfnEdWg eC0ljapjLAsHVoo+nSQnRtBRGw3mSoSU6WUtK1tqWgoolVg52uoL3B65k i7YGk+w+MHR7TJjbKWUViM9Nc0NMVdiPSnWvSqyQYR/o2Okzj1cndpwGL A==; X-IronPort-AV: E=McAfee;i="6400,9594,10400"; a="267084005" X-IronPort-AV: E=Sophos;i="5.92,253,1650956400"; d="scan'208";a="267084005" Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 07 Jul 2022 08:39:29 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,253,1650956400"; d="scan'208";a="920633504" Received: from silpixa00401384.ir.intel.com ([10.243.22.75]) by fmsmga005.fm.intel.com with ESMTP; 07 Jul 2022 08:39:27 -0700 From: Emma Finn To: dev@openvswitch.org, echaudro@redhat.com, harry.van.haaren@intel.com, kumar.amber@intel.com Date: Thu, 7 Jul 2022 15:39:00 +0000 Message-Id: <20220707153900.3147694-11-emma.finn@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220707153900.3147694-1-emma.finn@intel.com> References: <20220614115743.1143341-2-emma.finn@intel.com> <20220707153900.3147694-1-emma.finn@intel.com> MIME-Version: 1.0 Cc: i.maximets@ovn.org Subject: [ovs-dev] [v8 10/10] odp-execute: Add ISA implementation of set_masked IPv4 action X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This commit adds support for the AVX512 implementation of the ipv4_set_addrs action as well as an AVX512 implementation of updating the checksums. Signed-off-by: Emma Finn --- lib/odp-execute-avx512.c | 208 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 208 insertions(+) diff --git a/lib/odp-execute-avx512.c b/lib/odp-execute-avx512.c index 12eafc35e..b21464d19 100644 --- a/lib/odp-execute-avx512.c +++ b/lib/odp-execute-avx512.c @@ -22,6 +22,7 @@ #include #include +#include "csum.h" #include "dp-packet.h" #include "immintrin.h" #include "odp-execute.h" @@ -59,6 +60,22 @@ BUILD_ASSERT_DECL(offsetof(struct ovs_key_ethernet, eth_src) + MEMBER_SIZEOF(struct ovs_key_ethernet, eth_src) == offsetof(struct ovs_key_ethernet, eth_dst)); +BUILD_ASSERT_DECL(offsetof(struct ovs_key_ipv4, ipv4_src) + + MEMBER_SIZEOF(struct ovs_key_ipv4, ipv4_src) == + offsetof(struct ovs_key_ipv4, ipv4_dst)); + +BUILD_ASSERT_DECL(offsetof(struct ovs_key_ipv4, ipv4_dst) + + MEMBER_SIZEOF(struct ovs_key_ipv4, ipv4_dst) == + offsetof(struct ovs_key_ipv4, ipv4_proto)); + +BUILD_ASSERT_DECL(offsetof(struct ovs_key_ipv4, ipv4_proto) + + MEMBER_SIZEOF(struct ovs_key_ipv4, ipv4_proto) == + offsetof(struct ovs_key_ipv4, ipv4_tos)); + +BUILD_ASSERT_DECL(offsetof(struct ovs_key_ipv4, ipv4_tos) + + MEMBER_SIZEOF(struct ovs_key_ipv4, ipv4_tos) == + offsetof(struct ovs_key_ipv4, ipv4_ttl)); + /* Array of callback functions, one for each masked operation. */ odp_execute_action_cb impl_set_masked_funcs[__OVS_KEY_ATTR_MAX]; @@ -280,6 +297,196 @@ action_avx512_eth_set_addrs(struct dp_packet_batch *batch, } } +static inline uint16_t ALWAYS_INLINE +avx512_get_delta(__m256i old_header, __m256i res) +{ + __m256i v_zeros = _mm256_setzero_si256(); + uint16_t delta; + + /* These two shuffle masks, v_swap16a and v_swap16b, are to shuffle the + * old and new header to add padding after each 16-bit value for the + * following carry over addition. */ + __m256i v_swap16a = _mm256_setr_epi16(0x0100, 0xFFFF, 0x0302, 0xFFFF, + 0x0504, 0xFFFF, 0x0706, 0xFFFF, + 0x0100, 0xFFFF, 0x0302, 0xFFFF, + 0xFFFF, 0xFFFF, 0xFFFF, 0xFFFF); + __m256i v_swap16b = _mm256_setr_epi16(0x0908, 0xFFFF, 0x0B0A, 0xFFFF, + 0x0D0C, 0xFFFF, 0x0F0E, 0xFFFF, + 0xFFFF, 0xFFFF, 0xFFFF, 0xFFFF, + 0xFFFF, 0xFFFF, 0xFFFF, 0xFFFF); + __m256i v_shuf_old1 = _mm256_shuffle_epi8(old_header, v_swap16a); + __m256i v_shuf_old2 = _mm256_shuffle_epi8(old_header, v_swap16b); + __m256i v_shuf_new1 = _mm256_shuffle_epi8(res, v_swap16a); + __m256i v_shuf_new2 = _mm256_shuffle_epi8(res, v_swap16b); + + /* Add each part of the old and new headers together. */ + __m256i v_delta1 = _mm256_add_epi32(v_shuf_old1, v_shuf_new1); + __m256i v_delta2 = _mm256_add_epi32(v_shuf_old2, v_shuf_new2); + + /* Add old and new header. */ + __m256i v_delta = _mm256_add_epi32(v_delta1, v_delta2); + + /* Perform horizontal add to go from 8x32-bits to 2x32-bits. */ + v_delta = _mm256_hadd_epi32(v_delta, v_zeros); + v_delta = _mm256_hadd_epi32(v_delta, v_zeros); + + /* Shuffle 32-bit value from 3rd lane into first lane for final + * horizontal add. */ + __m256i v_swap32a = _mm256_setr_epi32(0x0, 0x4, 0xF, 0xF, + 0xF, 0xF, 0xF, 0xF); + v_delta = _mm256_permutexvar_epi32(v_swap32a, v_delta); + + v_delta = _mm256_hadd_epi32(v_delta, v_zeros); + v_delta = _mm256_hadd_epi16(v_delta, v_zeros); + + /* Extract delta value. */ + delta = _mm256_extract_epi16(v_delta, 0); + + return delta; +} + +static inline uint16_t ALWAYS_INLINE +avx512_l4_update_csum(__m256i old_header, __m256i res) +{ + __m256i v_zeros = _mm256_setzero_si256(); + uint16_t delta; + + /* Set the v_ones register to all one's. */ + __m256i v_ones = _mm256_cmpeq_epi16(v_zeros, v_zeros); + + /* Combine the old and new header, i.e. adding in the new IP addresses + * in the old header (oh). This is done by using the 0x03C 16-bit mask, + * picking 16-bit word 7 till 10. */ + __m256i v_blend_new = _mm256_mask_blend_epi16(0x03C0, old_header, res); + + /* Invert the old_header register. */ + old_header =_mm256_andnot_si256(old_header, v_ones); + + /* Calculate the delta between the old and new header. */ + delta = avx512_get_delta(old_header, v_blend_new); + + return delta; + +} + +static inline uint16_t ALWAYS_INLINE +avx512_ipv4_update_csum(__m256i res, __m256i old_header) +{ + __m256i v_zeros = _mm256_setzero_si256(); + uint16_t delta; + + /* Set the v_ones register to all one's. */ + __m256i v_ones = _mm256_cmpeq_epi16(v_zeros, v_zeros); + + /* Invert the old_header register. */ + old_header =_mm256_andnot_si256(old_header, v_ones); + + /* Calculate the delta between the old and new header. */ + delta = avx512_get_delta(old_header, res); + + return delta; +} + +/* This function performs the same operation on each packet in the batch as + * the scalar odp_set_ipv4() function. */ +static void +action_avx512_ipv4_set_addrs(struct dp_packet_batch *batch, + const struct nlattr *a) +{ + const struct ovs_key_ipv4 *key, *mask; + struct dp_packet *packet; + a = nl_attr_get(a); + key = nl_attr_get(a); + mask = odp_get_key_mask(a, struct ovs_key_ipv4); + + /* Read the content of the key(src) and mask in the respective registers. + * We only load the size of the actual structure, which is only 96-bits. */ + __m256i v_key = _mm256_maskz_loadu_epi32(0x7, (void *) key); + __m256i v_mask = _mm256_maskz_loadu_epi32(0x7, (void *) mask); + + /* This two shuffle masks, v_shuf32, v_shuffle, are to shuffle key and + * mask to match the ip_header structure layout. */ + static const uint8_t ip_shuffle_mask[32] = { + 0xFF, 0x05, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0x06, 0xFF, 0xFF, 0xFF, 0x00, 0x01, 0x02, 0x03, + 0x00, 0x01, 0x02, 0x03, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF}; + + __m256i v_shuf32 = _mm256_setr_epi32(0x0, 0x2, 0xF, 0xF, + 0x1, 0xF, 0xF, 0xF); + + __m256i v_shuffle = _mm256_loadu_si256((void *) ip_shuffle_mask); + + /* Two shuffles are required for key and mask to match the layout of + * the ip_header struct. The _shuffle_epi8 only works within 128-bit + * lanes, so a permute is required to move src and dst into the correct + * lanes. And then a shuffle is used to move the fields into the right + * order. + */ + __m256i v_key_shuf = _mm256_permutexvar_epi32(v_shuf32, v_key); + v_key_shuf = _mm256_shuffle_epi8(v_key_shuf, v_shuffle); + + __m256i v_mask_shuf = _mm256_permutexvar_epi32(v_shuf32, v_mask); + v_mask_shuf = _mm256_shuffle_epi8(v_mask_shuf, v_shuffle); + + DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { + struct ip_header *nh = dp_packet_l3(packet); + ovs_be16 old_csum = ~nh->ip_csum; + + /* Load the first 32 bytes of the IPv4 header. Without options, which + * is the most common case it's 20 bytes, but can be up to 60 bytes. */ + __m256i v_packet = _mm256_loadu_si256((void *) nh); + + /* AND the v_pkt_mask to the packet data (v_packet). */ + __m256i v_pkt_masked = _mm256_andnot_si256(v_mask_shuf, v_packet); + + /* OR the new addresses (v_key_shuf) with the masked packet addresses + * (v_pkt_masked). */ + __m256i v_res = _mm256_or_si256(v_key_shuf, v_pkt_masked); + + /* Update the IP checksum based on updated IP values. */ + uint16_t delta = avx512_ipv4_update_csum(v_res, v_packet); + uint32_t new_csum = old_csum + delta; + delta = csum_finish(new_csum); + + /* Insert new checksum. */ + v_res = _mm256_insert_epi16(v_res, delta, 5); + + /* If ip_src or ip_dst has been modified, L4 checksum needs to + * be updated too. */ + if (mask->ipv4_src || mask->ipv4_dst) { + + uint16_t delta_checksum = avx512_l4_update_csum(v_packet, v_res); + + if (nh->ip_proto == IPPROTO_UDP) { + /* New UDP checksum. */ + struct udp_header *uh = dp_packet_l4(packet); + if (uh->udp_csum) { + uint16_t old_udp_checksum = ~uh->udp_csum; + uint32_t udp_checksum = old_udp_checksum + delta_checksum; + udp_checksum = csum_finish(udp_checksum); + + if (!udp_checksum) { + udp_checksum = htons(0xffff); + } + /* Insert new udp checksum. */ + v_res = _mm256_insert_epi16(v_res, udp_checksum, 13); + } + } else if (nh->ip_proto == IPPROTO_TCP) { + /* New TCP checksum. */ + struct tcp_header *th = dp_packet_l4(packet); + uint16_t old_tcp_checksum = ~th->tcp_csum; + uint32_t tcp_checksum = old_tcp_checksum + delta_checksum; + tcp_checksum = csum_finish(tcp_checksum); + + th->tcp_csum = tcp_checksum; + } + } + /* Write back the modified IPv4 addresses. */ + _mm256_storeu_si256((void *) nh, v_res); + } +} + static void action_avx512_set_masked(struct dp_packet_batch *batch, const struct nlattr *a) { @@ -305,6 +512,7 @@ action_avx512_init(struct odp_execute_action_impl *self) /* Set function pointers for the individual operations supported by the * SET_MASKED action. */ impl_set_masked_funcs[OVS_KEY_ATTR_ETHERNET] = action_avx512_eth_set_addrs; + impl_set_masked_funcs[OVS_KEY_ATTR_IPV4] = action_avx512_ipv4_set_addrs; return 0; }