From patchwork Tue Jul 12 17:44:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Van Haaren, Harry" X-Patchwork-Id: 1655542 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=H7ATm3Wm; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lj7T94Xsrz9s07 for ; Wed, 13 Jul 2022 03:45:49 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 1E127611F0; Tue, 12 Jul 2022 17:45:47 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 1E127611F0 Authentication-Results: smtp3.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=H7ATm3Wm X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id b3q7roATzekP; Tue, 12 Jul 2022 17:45:45 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id 260D161187; Tue, 12 Jul 2022 17:45:44 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 260D161187 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 11C96C0035; Tue, 12 Jul 2022 17:45:43 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 90102C002D for ; Tue, 12 Jul 2022 17:45:40 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 678C660B11 for ; Tue, 12 Jul 2022 17:45:39 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 678C660B11 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4wZ2hQFzFSiS for ; Tue, 12 Jul 2022 17:45:38 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 13D7860EC8 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by smtp3.osuosl.org (Postfix) with ESMTPS id 13D7860EC8 for ; Tue, 12 Jul 2022 17:45:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657647938; x=1689183938; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=zmTjzbDbF67ROz/yqF3ks6wjP1UMMXTurtItRs019ds=; b=H7ATm3Wm0TWvlBndb/HI0eNKDBz3Vehe0AnIMtRDNjwSieJ0UYt5eIRv oW4l5/yDrooYzY9HEQLUnk1mUAF7ERt6iH+65t5PntwwCIez4xajmdgjV B3YW+BAa3kFpC15/yAD9GXcK8HUr/Vdo37WkX4DCNgDbS5JZXeSlzc7JI rO/h4bsVJjdkuj6cEL6SspF4g7z2g3veZh5kagFC8FYk7iBF72r+5DUPp re+u8kSrsa+/HBFz+8n5JTT29VmH5kJaJL6kJLgrs2faVpd5fpBR4fjns y7/UUYFtjxSDO1EfhhuHLGOfY6mYCoa5uH07eeo08DnnsNaQCFQD1RFMo A==; X-IronPort-AV: E=McAfee;i="6400,9594,10406"; a="286130840" X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="286130840" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jul 2022 10:45:37 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="627979729" Received: from silpixa00401454.ir.intel.com ([10.55.128.122]) by orsmga001.jf.intel.com with ESMTP; 12 Jul 2022 10:45:35 -0700 From: Harry van Haaren To: dev@openvswitch.org Date: Tue, 12 Jul 2022 17:44:47 +0000 Message-Id: <20220712174456.2682549-2-harry.van.haaren@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220712174456.2682549-1-harry.van.haaren@intel.com> References: <20220707153900.3147694-1-emma.finn@intel.com> <20220712174456.2682549-1-harry.van.haaren@intel.com> MIME-Version: 1.0 Cc: i.maximets@ovn.org, kumar.amber@intel.com Subject: [ovs-dev] [PATCH v9 01/10] odp-execute: Add function pointers to odp-execute for different action implementations. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" From: Emma Finn This commit introduces the initial infrastructure required to allow different implementations for OvS actions. The patch introduces action function pointers which allows user to switch between different action implementations available. This will allow for more performance and flexibility so the user can choose the action implementation to best suite their use case. Signed-off-by: Emma Finn Acked-by: Harry van Haaren --- v9: - rebase conflicts on NEWS - As Actions is initialized in vswitchd/bridge.c now, logs are output on startup, which have been added to the tests/ofproto-macros. --- lib/automake.mk | 2 + lib/odp-execute-private.c | 92 +++++++++++++++++++++++++++++++++++++++ lib/odp-execute-private.h | 76 ++++++++++++++++++++++++++++++++ lib/odp-execute.c | 51 +++++++++++++++++++++- lib/odp-execute.h | 7 +++ tests/ofproto-macros.at | 1 + vswitchd/bridge.c | 3 ++ 7 files changed, 231 insertions(+), 1 deletion(-) create mode 100644 lib/odp-execute-private.c create mode 100644 lib/odp-execute-private.h diff --git a/lib/automake.mk b/lib/automake.mk index 1d00cfa20..23ba4fab0 100644 --- a/lib/automake.mk +++ b/lib/automake.mk @@ -216,6 +216,8 @@ lib_libopenvswitch_la_SOURCES = \ lib/object-collection.h \ lib/odp-execute.c \ lib/odp-execute.h \ + lib/odp-execute-private.c \ + lib/odp-execute-private.h \ lib/odp-util.c \ lib/odp-util.h \ lib/ofp-actions.c \ diff --git a/lib/odp-execute-private.c b/lib/odp-execute-private.c new file mode 100644 index 000000000..2c30ed05b --- /dev/null +++ b/lib/odp-execute-private.c @@ -0,0 +1,92 @@ +/* + * Copyright (c) 2022 Intel. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include +#include +#include +#include + +#include "dpdk.h" +#include "dp-packet.h" +#include "odp-execute-private.h" +#include "odp-netlink.h" +#include "odp-util.h" +#include "openvswitch/vlog.h" + +VLOG_DEFINE_THIS_MODULE(odp_execute_impl); +static int active_action_impl_index; + +static struct odp_execute_action_impl action_impls[] = { + [ACTION_IMPL_SCALAR] = { + .available = false, + .name = "scalar", + .init_func = NULL, + }, +}; + +static void +action_impl_copy_funcs(struct odp_execute_action_impl *dest, + const struct odp_execute_action_impl *src) +{ + for (int i = 0; i < __OVS_ACTION_ATTR_MAX; i++) { + atomic_store_relaxed(&dest->funcs[i], src->funcs[i]); + } +} + +struct odp_execute_action_impl * +odp_execute_action_set(const char *name) +{ + for (int i = 0; i < ACTION_IMPL_MAX; i++) { + /* String compare, and set ptrs atomically. */ + if (!strcmp(action_impls[i].name, name)) { + active_action_impl_index = i; + + VLOG_INFO("Action implementation set to %s", name); + return &action_impls[i]; + } + } + return NULL; +} + +void +odp_execute_action_init(void) +{ + /* Each impl's function array is initialized to reflect the scalar + * implementation. This simplifies adding optimized implementations, + * as the autovalidator can always compare all actions. + * + * Below will check if impl is available and copies the scalar functions + * to all other implementations. + */ + for (int i = 0; i < ACTION_IMPL_MAX; i++) { + bool avail = true; + + if (i != ACTION_IMPL_SCALAR) { + action_impl_copy_funcs(&action_impls[i], + &action_impls[ACTION_IMPL_SCALAR]); + } + + if (action_impls[i].init_func) { + /* Return zero is success, non-zero means error. */ + avail = (action_impls[i].init_func(&action_impls[i]) == 0); + } + + action_impls[i].available = avail; + + VLOG_INFO("Action implementation %s (available: %s)", + action_impls[i].name, avail ? "Yes" : "No"); + } +} diff --git a/lib/odp-execute-private.h b/lib/odp-execute-private.h new file mode 100644 index 000000000..24126cdca --- /dev/null +++ b/lib/odp-execute-private.h @@ -0,0 +1,76 @@ +/* + * Copyright (c) 2022 Intel. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef ODP_EXTRACT_PRIVATE +#define ODP_EXTRACT_PRIVATE 1 + +#include "dp-packet.h" +#include "odp-execute.h" +#include "odp-netlink.h" +#include "ovs-atomic.h" + +/* Forward declaration for typedef. */ +struct odp_execute_action_impl; + +/* Typedef for an initialization function that can initialize each + * implementation, checking requirements such as CPU ISA. + */ +typedef int (*odp_execute_action_init_func) + (struct odp_execute_action_impl *self); + +/* Structure represents an implementation of the odp actions. */ +struct odp_execute_action_impl { + /* When set, the CPU ISA required for this implementation is available + * and the implementation can be used. + */ + bool available; + + /* Name of the implementation. */ + const char *name; + + /* Function is used to detect if this CPU has the ISA required + * to run the optimized action implementation and if available, initializes + * the implementation for use. + */ + odp_execute_action_init_func init_func; + + /* An array of callback functions, one for each action. */ + ATOMIC(odp_execute_action_cb) funcs[__OVS_ACTION_ATTR_MAX]; +}; + +/* Order of Actions implementations. */ +enum odp_execute_action_impl_idx { + ACTION_IMPL_SCALAR, + /* See ACTION_IMPL_BEGIN below, for "first to-be-validated" impl. + * Do not change the autovalidator position in this list without updating + * the define below. + */ + + ACTION_IMPL_MAX, +}; + +/* Index to start verifying implementations from. */ +BUILD_ASSERT_DECL(ACTION_IMPL_SCALAR == 0); + +/* Odp execute init handles setting up the state of the actions functions at + * initialization time. It cannot return errors, as it must always succeed in + * initializing the scalar/generic codepath. + */ +void odp_execute_action_init(void); + +struct odp_execute_action_impl * odp_execute_action_set(const char *name); + +#endif /* ODP_EXTRACT_PRIVATE */ diff --git a/lib/odp-execute.c b/lib/odp-execute.c index 7da56793d..bf8223634 100644 --- a/lib/odp-execute.c +++ b/lib/odp-execute.c @@ -17,6 +17,7 @@ #include #include "odp-execute.h" +#include "odp-execute-private.h" #include #include #include @@ -833,6 +834,38 @@ requires_datapath_assistance(const struct nlattr *a) return false; } +/* The active function pointers on the datapath. ISA optimized implementations + * are enabled by plugging them into this static arary, which is consulted when + * applying actions on the datapath. + */ +static struct odp_execute_action_impl *actions_active_impl; + +static int +odp_actions_impl_set(const char *name) +{ + struct odp_execute_action_impl *active; + active = odp_execute_action_set(name); + if (!active) { + VLOG_ERR("Failed setting action implementation to %s", name); + return 1; + } + + actions_active_impl = active; + return 0; + +} + +void +odp_execute_init(void) +{ + static struct ovsthread_once once = OVSTHREAD_ONCE_INITIALIZER; + if (ovsthread_once_start(&once)) { + odp_execute_action_init(); + odp_actions_impl_set("scalar"); + ovsthread_once_done(&once); + } +} + /* Executes all of the 'actions_len' bytes of datapath actions in 'actions' on * the packets in 'batch'. If 'steal' is true, possibly modifies and * definitely free the packets in 'batch', otherwise leaves 'batch' unchanged. @@ -857,6 +890,7 @@ odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, NL_ATTR_FOR_EACH_UNSAFE (a, left, actions, actions_len) { int type = nl_attr_type(a); + enum ovs_action_attr attr_type = (enum ovs_action_attr) type; bool last_action = (left <= NLA_ALIGN(a->nla_len)); if (requires_datapath_assistance(a)) { @@ -879,8 +913,20 @@ odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, continue; } - switch ((enum ovs_action_attr) type) { + /* If type is set in the active actions implementation, call the + * function-pointer and continue to the next action. + */ + if (actions_active_impl->funcs[attr_type] && + attr_type < __OVS_ACTION_ATTR_MAX) { + actions_active_impl->funcs[attr_type](batch, a); + continue; + } + /* If the action was not handled by the active function pointers above, + * process them by switching on the type below. + */ + + switch (attr_type) { case OVS_ACTION_ATTR_HASH: { const struct ovs_action_hash *hash_act = nl_attr_get(a); @@ -1094,6 +1140,9 @@ odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, case __OVS_ACTION_ATTR_MAX: OVS_NOT_REACHED(); } + + /* Do not add any generic processing here, as it won't be executed when + * an ISA-specific action implementation exists. */ } dp_packet_delete_batch(batch, steal); diff --git a/lib/odp-execute.h b/lib/odp-execute.h index a3578a575..0921ee924 100644 --- a/lib/odp-execute.h +++ b/lib/odp-execute.h @@ -28,6 +28,13 @@ struct dp_packet; struct pkt_metadata; struct dp_packet_batch; + +/* Called once at initialization time. */ +void odp_execute_init(void); + +typedef void (*odp_execute_action_cb)(struct dp_packet_batch *batch, + const struct nlattr *action); + typedef void (*odp_execute_cb)(void *dp, struct dp_packet_batch *batch, const struct nlattr *action, bool should_steal); diff --git a/tests/ofproto-macros.at b/tests/ofproto-macros.at index b18f0fbc1..b0862a549 100644 --- a/tests/ofproto-macros.at +++ b/tests/ofproto-macros.at @@ -180,6 +180,7 @@ m4_define([_OVS_VSWITCHD_START], on_exit "kill_ovs_vswitchd `cat ovs-vswitchd.pid`" AT_CHECK([[sed < stderr ' /ovs_numa|INFO|Discovered /d +/odp_execute_impl|INFO|Action implementation /d /vlog|INFO|opened log file/d /vswitchd|INFO|ovs-vswitchd (Open vSwitch)/d /reconnect|INFO|/d diff --git a/vswitchd/bridge.c b/vswitchd/bridge.c index e328d8ead..b704410d3 100644 --- a/vswitchd/bridge.c +++ b/vswitchd/bridge.c @@ -40,6 +40,7 @@ #include "netdev.h" #include "netdev-offload.h" #include "nx-match.h" +#include "odp-execute.h" #include "ofproto/bond.h" #include "ofproto/ofproto.h" #include "openvswitch/dynamic-string.h" @@ -530,6 +531,8 @@ bridge_init(const char *remote) stp_init(); lldp_init(); rstp_init(); + odp_execute_init(); + ifaces_changed = seq_create(); last_ifaces_changed = seq_read(ifaces_changed); ifnotifier = if_notifier_create(if_change_cb, NULL); From patchwork Tue Jul 12 17:44:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Van Haaren, Harry" X-Patchwork-Id: 1655543 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=UViCFTjO; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lj7TC75YDz9s07 for ; Wed, 13 Jul 2022 03:45:51 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id D1A7F4099A; Tue, 12 Jul 2022 17:45:48 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org D1A7F4099A Authentication-Results: smtp4.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=UViCFTjO X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ppXNhtoAlT2F; Tue, 12 Jul 2022 17:45:47 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id D4AE141A67; Tue, 12 Jul 2022 17:45:45 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org D4AE141A67 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 01E4CC0082; Tue, 12 Jul 2022 17:45:44 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id D468FC002D for ; Tue, 12 Jul 2022 17:45:41 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 7754860FFF for ; Tue, 12 Jul 2022 17:45:41 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 7754860FFF Authentication-Results: smtp3.osuosl.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=UViCFTjO X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UxbuHc4byFXU for ; Tue, 12 Jul 2022 17:45:40 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org B55D360B11 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by smtp3.osuosl.org (Postfix) with ESMTPS id B55D360B11 for ; Tue, 12 Jul 2022 17:45:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657647940; x=1689183940; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=4eCgS72R30yBuIWryotfe8y3TxxRXLyovhzacS/zgHI=; b=UViCFTjO19odiwVEbkoPc+PXYV7NECcdI7MQGev9OVQTIFAEpsCaRE3N rozxPumiE4ku6viApPWEdo93B8557QVMnkbo5JH4DqXqtI4VXUZot5BZO m5gYFkyyu0LbfSrls2jcuVP+NbQY1tFKK/jFw96zfaQIfPRvZvspRE8G7 bkK8e5HLOcYN4z7CtronrYfhlCD5v3TGdDHa0xG3SH1im+uo9E0B8/XMe IjECxwzn1ygJr40hs4TlyJVODVvKUXcf37t0wDDP0niQYyMv3xty6tamb kSkQ8mfCVzaan9Bm3SfMU5tjdMLffCbvZ0/0Dhc/VGUekmvjmo+x25+z4 w==; X-IronPort-AV: E=McAfee;i="6400,9594,10406"; a="286130843" X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="286130843" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jul 2022 10:45:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="627979740" Received: from silpixa00401454.ir.intel.com ([10.55.128.122]) by orsmga001.jf.intel.com with ESMTP; 12 Jul 2022 10:45:38 -0700 From: Harry van Haaren To: dev@openvswitch.org Date: Tue, 12 Jul 2022 17:44:48 +0000 Message-Id: <20220712174456.2682549-3-harry.van.haaren@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220712174456.2682549-1-harry.van.haaren@intel.com> References: <20220707153900.3147694-1-emma.finn@intel.com> <20220712174456.2682549-1-harry.van.haaren@intel.com> MIME-Version: 1.0 Cc: i.maximets@ovn.org, kumar.amber@intel.com Subject: [ovs-dev] [PATCH v9 02/10] odp-execute: Add function pointer for pop_vlan action. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" From: Emma Finn This commit removes the pop_vlan action from the large switch and creates a separate function for batched processing. A function pointer is also added to call the new batched function for the pop_vlan action. Signed-off-by: Emma Finn Acked-by: Harry van Haaren --- lib/odp-execute-private.c | 16 +++++++++++++++- lib/odp-execute-private.h | 5 +++++ lib/odp-execute.c | 32 ++++++++++++++++++++++++++------ 3 files changed, 46 insertions(+), 7 deletions(-) diff --git a/lib/odp-execute-private.c b/lib/odp-execute-private.c index 2c30ed05b..c1d153c6e 100644 --- a/lib/odp-execute-private.c +++ b/lib/odp-execute-private.c @@ -33,7 +33,7 @@ static struct odp_execute_action_impl action_impls[] = { [ACTION_IMPL_SCALAR] = { .available = false, .name = "scalar", - .init_func = NULL, + .init_func = odp_action_scalar_init, }, }; @@ -88,5 +88,19 @@ odp_execute_action_init(void) VLOG_INFO("Action implementation %s (available: %s)", action_impls[i].name, avail ? "Yes" : "No"); + + /* The following is a run-time check to make sure a scalar + * implementation exists for the given ISA implementation. This is to + * make sure the autovalidator works as expected. */ + if (avail && i != ACTION_IMPL_SCALAR) { + for (int j = 0; j < __OVS_ACTION_ATTR_MAX; j++) { + /* No ovs_assert(), as it can be compiled out. */ + if (action_impls[ACTION_IMPL_SCALAR].funcs[j] == NULL + && action_impls[i].funcs[j] != NULL) { + ovs_assert_failure(OVS_SOURCE_LOCATOR, __func__, + "Missing scalar action function!"); + } + } + } } } diff --git a/lib/odp-execute-private.h b/lib/odp-execute-private.h index 24126cdca..ae06fbc09 100644 --- a/lib/odp-execute-private.h +++ b/lib/odp-execute-private.h @@ -71,6 +71,11 @@ BUILD_ASSERT_DECL(ACTION_IMPL_SCALAR == 0); */ void odp_execute_action_init(void); +/* Init functions for the action implementations. Initializes the function + * pointers for optimized action types. + */ +int odp_action_scalar_init(struct odp_execute_action_impl *self); + struct odp_execute_action_impl * odp_execute_action_set(const char *name); #endif /* ODP_EXTRACT_PRIVATE */ diff --git a/lib/odp-execute.c b/lib/odp-execute.c index bf8223634..aa51476f4 100644 --- a/lib/odp-execute.c +++ b/lib/odp-execute.c @@ -834,6 +834,30 @@ requires_datapath_assistance(const struct nlattr *a) return false; } +static void +action_pop_vlan(struct dp_packet_batch *batch, + const struct nlattr *a OVS_UNUSED) +{ + struct dp_packet *packet; + + DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { + eth_pop_vlan(packet); + } +} + +/* Implementation of the scalar actions impl init function. Build up the + * array of func ptrs here. + */ +int +odp_action_scalar_init(struct odp_execute_action_impl *self) +{ + /* Set function pointers for actions that can be applied directly, these + * are identified by OVS_ACTION_ATTR_*. */ + self->funcs[OVS_ACTION_ATTR_POP_VLAN] = action_pop_vlan; + + return 0; +} + /* The active function pointers on the datapath. ISA optimized implementations * are enabled by plugging them into this static arary, which is consulted when * applying actions on the datapath. @@ -982,12 +1006,6 @@ odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, break; } - case OVS_ACTION_ATTR_POP_VLAN: - DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { - eth_pop_vlan(packet); - } - break; - case OVS_ACTION_ATTR_PUSH_MPLS: { const struct ovs_action_push_mpls *mpls = nl_attr_get(a); @@ -1138,6 +1156,8 @@ odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, case OVS_ACTION_ATTR_CT: case OVS_ACTION_ATTR_UNSPEC: case __OVS_ACTION_ATTR_MAX: + /* The following actions are handled by the scalar implementation. */ + case OVS_ACTION_ATTR_POP_VLAN: OVS_NOT_REACHED(); } From patchwork Tue Jul 12 17:44:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Van Haaren, Harry" X-Patchwork-Id: 1655545 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=CwOsKv9l; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lj7TJ3ZCpz9sFk for ; Wed, 13 Jul 2022 03:45:56 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id A20414074E; Tue, 12 Jul 2022 17:45:52 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org A20414074E Authentication-Results: smtp2.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=CwOsKv9l X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1ybVEEfwaCxO; Tue, 12 Jul 2022 17:45:50 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id 0AE4F40FC4; Tue, 12 Jul 2022 17:45:49 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 0AE4F40FC4 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id B35B7C0035; Tue, 12 Jul 2022 17:45:48 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2C6DAC0084 for ; Tue, 12 Jul 2022 17:45:44 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 01E32610E6 for ; Tue, 12 Jul 2022 17:45:44 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 01E32610E6 Authentication-Results: smtp3.osuosl.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=CwOsKv9l X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mlPOY3AVGrM0 for ; Tue, 12 Jul 2022 17:45:43 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org DB25D61187 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by smtp3.osuosl.org (Postfix) with ESMTPS id DB25D61187 for ; Tue, 12 Jul 2022 17:45:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657647942; x=1689183942; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=tT1ol1JqEQq+6VcZf9KElsjZZgywpZoGK8wm4gzVhV4=; b=CwOsKv9lnHhW58MP7BUpcKaNdU5L5rDoso55XDeQfKadroLdjA/sq4Sc qMcaFFmrKNJovOPBtC+HwI8we/DoQ4hgyaHoI3KclnVoXkq1odl3zScV/ 286wO0kSgAyRhqlWw2xHhfihZXQUcO9FwKIEpmLh11YsRiNaeXSZCb89i oDLtawWp0IJ4OxkQXCzWM5Y8j0Stxx9hwiEO8U7IXoZ6eBERyDcTDYrj4 87bFD1j9b+W8f+6fKU0InizjGe3p9zibH/A4z4YRYD1wIMjH3XlnVof0l 1WRWLvOWiqwl2a5PdUq+gCWhXU/PljXd4OfwSbSCcWnE0FzBtMb7kgXS7 Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10406"; a="286130854" X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="286130854" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jul 2022 10:45:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="627979748" Received: from silpixa00401454.ir.intel.com ([10.55.128.122]) by orsmga001.jf.intel.com with ESMTP; 12 Jul 2022 10:45:40 -0700 From: Harry van Haaren To: dev@openvswitch.org Date: Tue, 12 Jul 2022 17:44:49 +0000 Message-Id: <20220712174456.2682549-4-harry.van.haaren@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220712174456.2682549-1-harry.van.haaren@intel.com> References: <20220707153900.3147694-1-emma.finn@intel.com> <20220712174456.2682549-1-harry.van.haaren@intel.com> MIME-Version: 1.0 Cc: i.maximets@ovn.org, kumar.amber@intel.com Subject: [ovs-dev] [PATCH v9 03/10] odp-execute: Add auto validation function for actions. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" From: Emma Finn This commit introduced the auto-validation function which allows users to compare the batch of packets obtained from different action implementations against the linear action implementation. The autovalidator function can be triggered at runtime using the following command: $ ovs-appctl odp-execute/action-impl-set autovalidator Signed-off-by: Emma Finn Acked-by: Harry van Haaren --- NEWS | 2 + lib/dp-packet.c | 24 ++++++++++ lib/dp-packet.h | 4 ++ lib/odp-execute-private.c | 99 +++++++++++++++++++++++++++++++++++++++ lib/odp-execute-private.h | 6 +++ 5 files changed, 135 insertions(+) diff --git a/NEWS b/NEWS index 433bb1654..aeba359e5 100644 --- a/NEWS +++ b/NEWS @@ -49,6 +49,8 @@ Post-v2.17.0 * 'dpif-netdev/subtable-lookup-prio-get' appctl command renamed to 'dpif-netdev/subtable-lookup-info-get' to better reflect its purpose. The old variant is kept for backward compatibility. + * Add actions auto-validator function to compare different actions + implementations against default implementation. - Linux datapath: * Add offloading meter tc police. diff --git a/lib/dp-packet.c b/lib/dp-packet.c index 35c72542a..4538d2a61 100644 --- a/lib/dp-packet.c +++ b/lib/dp-packet.c @@ -506,3 +506,27 @@ dp_packet_resize_l2(struct dp_packet *b, int increment) dp_packet_adjust_layer_offset(&b->l2_5_ofs, increment); return dp_packet_data(b); } + +bool +dp_packet_compare_offsets(struct dp_packet *b1, struct dp_packet *b2, + struct ds *err_str) +{ + if ((b1->l2_pad_size != b2->l2_pad_size) || + (b1->l2_5_ofs != b2->l2_5_ofs) || + (b1->l3_ofs != b2->l3_ofs) || + (b1->l4_ofs != b2->l4_ofs)) { + if (err_str) { + ds_put_format(err_str, "Packet offset comparison failed\n"); + ds_put_format(err_str, "Buffer 1 offsets: l2_pad_size %u," + " l2_5_ofs : %u l3_ofs %u, l4_ofs %u\n", + b1->l2_pad_size, b1->l2_5_ofs, + b1->l3_ofs, b1->l4_ofs); + ds_put_format(err_str, "Buffer 2 offsets: l2_pad_size %u," + " l2_5_ofs : %u l3_ofs %u, l4_ofs %u\n", + b2->l2_pad_size, b2->l2_5_ofs, + b2->l3_ofs, b2->l4_ofs); + } + return false; + } + return true; +} diff --git a/lib/dp-packet.h b/lib/dp-packet.h index eea5a9215..55eeaab2c 100644 --- a/lib/dp-packet.h +++ b/lib/dp-packet.h @@ -236,6 +236,10 @@ void *dp_packet_steal_data(struct dp_packet *); static inline bool dp_packet_equal(const struct dp_packet *, const struct dp_packet *); +bool dp_packet_compare_offsets(struct dp_packet *good, + struct dp_packet *test, + struct ds *err_str); + /* Frees memory that 'b' points to, as well as 'b' itself. */ static inline void diff --git a/lib/odp-execute-private.c b/lib/odp-execute-private.c index c1d153c6e..780d6d289 100644 --- a/lib/odp-execute-private.c +++ b/lib/odp-execute-private.c @@ -30,6 +30,12 @@ VLOG_DEFINE_THIS_MODULE(odp_execute_impl); static int active_action_impl_index; static struct odp_execute_action_impl action_impls[] = { + [ACTION_IMPL_AUTOVALIDATOR] = { + .available = false, + .name = "autovalidator", + .init_func = action_autoval_init, + }, + [ACTION_IMPL_SCALAR] = { .available = false, .name = "scalar", @@ -104,3 +110,96 @@ odp_execute_action_init(void) } } } + +/* Init sequence required to be scalar first to pick up the default scalar +* implementations, allowing over-riding of the optimized functions later. +*/ +BUILD_ASSERT_DECL(ACTION_IMPL_SCALAR == 0); +BUILD_ASSERT_DECL(ACTION_IMPL_AUTOVALIDATOR == 1); + +/* Loop over packets, and validate each one for the given action. */ +static void +action_autoval_generic(struct dp_packet_batch *batch, const struct nlattr *a) +{ + struct odp_execute_action_impl *scalar = &action_impls[ACTION_IMPL_SCALAR]; + enum ovs_action_attr attr_type = nl_attr_type(a); + struct dp_packet_batch original_batch; + bool failed = false; + + dp_packet_batch_clone(&original_batch, batch); + + scalar->funcs[attr_type](batch, a); + + for (int impl = ACTION_IMPL_BEGIN; impl < ACTION_IMPL_MAX; impl++) { + /* Clone original batch and execute implementation under test. */ + struct dp_packet_batch test_batch; + + dp_packet_batch_clone(&test_batch, &original_batch); + action_impls[impl].funcs[attr_type](&test_batch, a); + + /* Loop over implementations, checking each one. */ + for (int pidx = 0; pidx < original_batch.count; pidx++) { + struct dp_packet *good_pkt = batch->packets[pidx]; + struct dp_packet *test_pkt = test_batch.packets[pidx]; + + struct ds log_msg = DS_EMPTY_INITIALIZER; + + /* Compare packet length and payload contents. */ + bool eq = dp_packet_equal(good_pkt, test_pkt); + + if (!eq) { + ds_put_format(&log_msg, "Packet: %d\nAction : ", pidx); + format_odp_actions(&log_msg, a, a->nla_len, NULL); + ds_put_format(&log_msg, "\nGood hex:\n"); + ds_put_hex_dump(&log_msg, dp_packet_data(good_pkt), + dp_packet_size(good_pkt), 0, false); + ds_put_format(&log_msg, "Test hex:\n"); + ds_put_hex_dump(&log_msg, dp_packet_data(test_pkt), + dp_packet_size(test_pkt), 0, false); + + failed = true; + } + + /* Compare offsets and RSS */ + if (!dp_packet_compare_offsets(good_pkt, test_pkt, &log_msg)) { + failed = true; + } + + if (dp_packet_rss_valid(good_pkt)) { + uint32_t good_hash = dp_packet_get_rss_hash(good_pkt); + uint32_t test_hash = dp_packet_get_rss_hash(test_pkt); + + if (good_hash != test_hash) { + ds_put_format(&log_msg, + "Autovalidation rss hash failed\n"); + ds_put_format(&log_msg, "Good RSS hash : %u\n", good_hash); + ds_put_format(&log_msg, "Test RSS hash : %u\n", test_hash); + + failed = true; + } + } + + if (failed) { + VLOG_ERR("Autovalidation of %s failed. Details:\n%s", + action_impls[impl].name, ds_cstr(&log_msg)); + ds_destroy(&log_msg); + failed = false; + } + } + dp_packet_delete_batch(&test_batch, true); + } + dp_packet_delete_batch(&original_batch, true); +} + +int +action_autoval_init(struct odp_execute_action_impl *self) +{ + /* Set function pointers for actions that can be applied directly, these + * are identified by OVS_ACTION_ATTR_*. */ + for (int i = 0; i < __OVS_ACTION_ATTR_MAX; i++) { + if (action_impls[ACTION_IMPL_SCALAR].funcs[i]) { + self->funcs[i] = action_autoval_generic; + } + } + return 0; +} diff --git a/lib/odp-execute-private.h b/lib/odp-execute-private.h index ae06fbc09..074a8d67e 100644 --- a/lib/odp-execute-private.h +++ b/lib/odp-execute-private.h @@ -54,6 +54,7 @@ struct odp_execute_action_impl { /* Order of Actions implementations. */ enum odp_execute_action_impl_idx { ACTION_IMPL_SCALAR, + ACTION_IMPL_AUTOVALIDATOR, /* See ACTION_IMPL_BEGIN below, for "first to-be-validated" impl. * Do not change the autovalidator position in this list without updating * the define below. @@ -64,6 +65,9 @@ enum odp_execute_action_impl_idx { /* Index to start verifying implementations from. */ BUILD_ASSERT_DECL(ACTION_IMPL_SCALAR == 0); +BUILD_ASSERT_DECL(ACTION_IMPL_AUTOVALIDATOR == 1); + +#define ACTION_IMPL_BEGIN (ACTION_IMPL_AUTOVALIDATOR + 1) /* Odp execute init handles setting up the state of the actions functions at * initialization time. It cannot return errors, as it must always succeed in @@ -78,4 +82,6 @@ int odp_action_scalar_init(struct odp_execute_action_impl *self); struct odp_execute_action_impl * odp_execute_action_set(const char *name); +int action_autoval_init(struct odp_execute_action_impl *self); + #endif /* ODP_EXTRACT_PRIVATE */ From patchwork Tue Jul 12 17:44:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Van Haaren, Harry" X-Patchwork-Id: 1655544 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=Q3CyZuhQ; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lj7TH6Z3Cz9s07 for ; Wed, 13 Jul 2022 03:45:55 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id F00E260B11; Tue, 12 Jul 2022 17:45:53 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org F00E260B11 Authentication-Results: smtp3.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=Q3CyZuhQ X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sqz7uu0bRx5v; Tue, 12 Jul 2022 17:45:52 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id 87B49611D9; Tue, 12 Jul 2022 17:45:51 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 87B49611D9 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1C75EC0082; Tue, 12 Jul 2022 17:45:51 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id D52FDC0071 for ; Tue, 12 Jul 2022 17:45:46 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id ACB1A611F0 for ; Tue, 12 Jul 2022 17:45:46 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org ACB1A611F0 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id r8wL7t5CKhl1 for ; Tue, 12 Jul 2022 17:45:45 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 68CC3611D9 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by smtp3.osuosl.org (Postfix) with ESMTPS id 68CC3611D9 for ; Tue, 12 Jul 2022 17:45:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657647945; x=1689183945; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Cdq/9N3gdF8cdiCZ0NBzW75ovGsEbLy/F/s+KtUEGhQ=; b=Q3CyZuhQdMGq42EPRSTLJN5BoqgLfXX8rDfSGrAV4RYZsvkWnnTMhTh5 UsHJgCl3UBnl+a09oXdCSp7yyhQ5PZgEmAydyaD1i/YGgWT1r9RHNAngC 299BGKmYAVSqbcizxWFbdagflhixVz+W4OKqvc1pcq4v5/OB5BZlXDAhA 3n2fVeEJndxBPwbaG20E/GhFTo1P22agFjk9l0DPDrC6QJjY/oW/kWXVP RB8kYP3+R3EzVV+Qg+0TRJpIhLfSPEzHYM3HgkZandp06MrVhfj0gV9A2 Jdq6mvxv2U8nKlxTkyBq74tJJtoioWc5BbNSPvd02JuQelvvK5x6QYXf1 w==; X-IronPort-AV: E=McAfee;i="6400,9594,10406"; a="286130863" X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="286130863" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jul 2022 10:45:45 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="627979771" Received: from silpixa00401454.ir.intel.com ([10.55.128.122]) by orsmga001.jf.intel.com with ESMTP; 12 Jul 2022 10:45:42 -0700 From: Harry van Haaren To: dev@openvswitch.org Date: Tue, 12 Jul 2022 17:44:50 +0000 Message-Id: <20220712174456.2682549-5-harry.van.haaren@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220712174456.2682549-1-harry.van.haaren@intel.com> References: <20220707153900.3147694-1-emma.finn@intel.com> <20220712174456.2682549-1-harry.van.haaren@intel.com> MIME-Version: 1.0 Cc: i.maximets@ovn.org, kumar.amber@intel.com Subject: [ovs-dev] [PATCH v9 04/10] odp-execute: Add command to switch action implementation. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" From: Emma Finn This commit adds a new command to allow the user to switch the active action implementation at runtime. Usage: $ ovs-appctl odp-execute/action-impl-set scalar This commit also adds a new command to retrieve the list of available action implementations. This can be used by to check what implementations of actions are available and what implementation is active during runtime. Usage: $ ovs-appctl odp-execute/action-impl-show Added separate test-case for ovs-actions show/set commands: PMD - ovs-actions configuration Signed-off-by: Emma Finn Signed-off-by: Kumar Amber Signed-off-by: Sunil Pai G Co-authored-by: Kumar Amber Co-authored-by: Sunil Pai G Acked-by: Harry van Haaren --- v9: rebase conflict on NEWS --- NEWS | 2 ++ lib/automake.mk | 1 + lib/odp-execute-private.c | 12 ++++++++++ lib/odp-execute-private.h | 2 ++ lib/odp-execute-unixctl.man | 10 +++++++++ lib/odp-execute.c | 44 +++++++++++++++++++++++++++++++++++++ tests/pmd.at | 39 ++++++++++++++++++++++++++++++++ vswitchd/ovs-vswitchd.8.in | 1 + 8 files changed, 111 insertions(+) create mode 100644 lib/odp-execute-unixctl.man diff --git a/NEWS b/NEWS index aeba359e5..cf35f4ae4 100644 --- a/NEWS +++ b/NEWS @@ -51,6 +51,8 @@ Post-v2.17.0 The old variant is kept for backward compatibility. * Add actions auto-validator function to compare different actions implementations against default implementation. + * Add command line option to switch between different actions + implementations available at run time. - Linux datapath: * Add offloading meter tc police. diff --git a/lib/automake.mk b/lib/automake.mk index 23ba4fab0..5c3b05f6b 100644 --- a/lib/automake.mk +++ b/lib/automake.mk @@ -584,6 +584,7 @@ MAN_FRAGMENTS += \ lib/netdev-dpdk-unixctl.man \ lib/dpif-netdev-unixctl.man \ lib/dpif-netlink-unixctl.man \ + lib/odp-execute-unixctl.man \ lib/ofp-version.man \ lib/ovs.tmac \ lib/ovs-replay.man \ diff --git a/lib/odp-execute-private.c b/lib/odp-execute-private.c index 780d6d289..38be22ec9 100644 --- a/lib/odp-execute-private.c +++ b/lib/odp-execute-private.c @@ -67,6 +67,18 @@ odp_execute_action_set(const char *name) return NULL; } +void +odp_execute_action_get_info(struct ds *string) +{ + ds_put_cstr(string, "Available Actions implementations:\n"); + for (int i = 0; i < ACTION_IMPL_MAX; i++) { + ds_put_format(string, " %s (available: %s, active: %s)\n", + action_impls[i].name, + action_impls[i].available ? "Yes" : "No", + i == active_action_impl_index ? "Yes" : "No"); + } +} + void odp_execute_action_init(void) { diff --git a/lib/odp-execute-private.h b/lib/odp-execute-private.h index 074a8d67e..d6eebbf37 100644 --- a/lib/odp-execute-private.h +++ b/lib/odp-execute-private.h @@ -84,4 +84,6 @@ struct odp_execute_action_impl * odp_execute_action_set(const char *name); int action_autoval_init(struct odp_execute_action_impl *self); +void odp_execute_action_get_info(struct ds *name); + #endif /* ODP_EXTRACT_PRIVATE */ diff --git a/lib/odp-execute-unixctl.man b/lib/odp-execute-unixctl.man new file mode 100644 index 000000000..82d51e1d3 --- /dev/null +++ b/lib/odp-execute-unixctl.man @@ -0,0 +1,10 @@ +.SS "ODP-EXECUTE COMMANDS" +These commands manage the "odp-execute" component. + +.IP "\fBodp-execute/action-impl-show\fR +Lists the actions implementations that are available and highlights the +currently enabled one. +. +.IP "\fBodp-execute/action-impl-set\fR \fIaction_impl\fR" +Sets the action implementation to any available implementation. By default +"scalar" is used. diff --git a/lib/odp-execute.c b/lib/odp-execute.c index aa51476f4..64c058a75 100644 --- a/lib/odp-execute.c +++ b/lib/odp-execute.c @@ -39,6 +39,7 @@ #include "csum.h" #include "conntrack.h" #include "openvswitch/vlog.h" +#include "unixctl.h" VLOG_DEFINE_THIS_MODULE(odp_execute); COVERAGE_DEFINE(datapath_drop_sample_error); @@ -879,6 +880,48 @@ odp_actions_impl_set(const char *name) } +static void +action_impl_set(struct unixctl_conn *conn, int argc OVS_UNUSED, + const char *argv[] OVS_UNUSED, void *aux OVS_UNUSED) +{ + struct ds reply = DS_EMPTY_INITIALIZER; + + int err = odp_actions_impl_set(argv[1]); + if (err) { + ds_put_format(&reply, + "Error: unknown action implementation, %s, specified!\n", + argv[1]); + unixctl_command_reply_error(conn, ds_cstr(&reply)); + } else { + ds_put_format(&reply, "Action implementation set to %s.\n", argv[1]); + unixctl_command_reply(conn, ds_cstr(&reply)); + } + + ds_destroy(&reply); +} + +static void +action_impl_show(struct unixctl_conn *conn, int argc OVS_UNUSED, + const char *argv[] OVS_UNUSED, void *aux OVS_UNUSED) +{ + struct ds reply = DS_EMPTY_INITIALIZER; + + odp_execute_action_get_info(&reply); + unixctl_command_reply(conn, ds_cstr(&reply)); + ds_destroy(&reply); +} + +static void +odp_execute_unixctl_init(void) +{ + unixctl_command_register("odp-execute/action-impl-set", "name", + 1, 1, action_impl_set, + NULL); + unixctl_command_register("odp-execute/action-impl-show", "", + 0, 0, action_impl_show, + NULL); +} + void odp_execute_init(void) { @@ -886,6 +929,7 @@ odp_execute_init(void) if (ovsthread_once_start(&once)) { odp_execute_action_init(); odp_actions_impl_set("scalar"); + odp_execute_unixctl_init(); ovsthread_once_done(&once); } } diff --git a/tests/pmd.at b/tests/pmd.at index 4342c50e0..4ca926581 100644 --- a/tests/pmd.at +++ b/tests/pmd.at @@ -1192,3 +1192,42 @@ ovs-appctl: ovs-vswitchd: server returned an error OVS_VSWITCHD_STOP AT_CLEANUP + +AT_SETUP([PMD - ovs-actions configuration]) +OVS_VSWITCHD_START([], [], [], [--dummy-numa 0,0]) +AT_CHECK([ovs-vsctl add-port br0 p1 -- set Interface p1 type=dummy-pmd]) + +AT_CHECK([ovs-vsctl show], [], [stdout]) + +dnl Set the scalar first, so we always have the scalar impl as Active. +AT_CHECK([ovs-appctl odp-execute/action-impl-set scalar], [0], [dnl +Action implementation set to scalar. +]) +AT_CHECK([ovs-appctl odp-execute/action-impl-show | grep "scalar"], [], [dnl + scalar (available: Yes, active: Yes) +]) + +AT_CHECK([ovs-appctl odp-execute/action-impl-show | grep "autovalidator"], [], [dnl + autovalidator (available: Yes, active: No) +]) + +dnl Set the autovalidator impl to active. +AT_CHECK([ovs-appctl odp-execute/action-impl-set autovalidator], [0], [dnl +Action implementation set to autovalidator. +]) + +AT_CHECK([ovs-appctl odp-execute/action-impl-show | grep "scalar"], [], [dnl + scalar (available: Yes, active: No) +]) + +AT_CHECK([ovs-appctl odp-execute/action-impl-show | grep "autovalidator"], [], [dnl + autovalidator (available: Yes, active: Yes) +]) + +AT_CHECK([ovs-appctl odp-execute/action-impl-set invalid_implementation], [2], [], [dnl +Error: unknown action implementation, invalid_implementation, specified! +ovs-appctl: ovs-vswitchd: server returned an error +]) + +OVS_VSWITCHD_STOP(["/Failed setting action implementation to invalid_implementation/d"]) +AT_CLEANUP diff --git a/vswitchd/ovs-vswitchd.8.in b/vswitchd/ovs-vswitchd.8.in index 1a32402be..9569265fc 100644 --- a/vswitchd/ovs-vswitchd.8.in +++ b/vswitchd/ovs-vswitchd.8.in @@ -282,6 +282,7 @@ type). .so lib/dpif-netdev-unixctl.man .so lib/dpif-netlink-unixctl.man .so lib/netdev-dpdk-unixctl.man +.so lib/odp-execute-unixctl.man .so ofproto/ofproto-dpif-unixctl.man .so ofproto/ofproto-unixctl.man .so lib/vlog-unixctl.man From patchwork Tue Jul 12 17:44:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Van Haaren, Harry" X-Patchwork-Id: 1655546 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=I9Hfj6Az; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lj7TQ5sdVz9s07 for ; Wed, 13 Jul 2022 03:46:02 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 449588405C; Tue, 12 Jul 2022 17:46:00 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 449588405C Authentication-Results: smtp1.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=I9Hfj6Az X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id eoYrGTaZRUQE; Tue, 12 Jul 2022 17:45:58 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTPS id 280B283FF5; Tue, 12 Jul 2022 17:45:57 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 280B283FF5 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 0CE36C002D; Tue, 12 Jul 2022 17:45:56 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 17565C0035 for ; Tue, 12 Jul 2022 17:45:55 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 4A8D841016 for ; Tue, 12 Jul 2022 17:45:52 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 4A8D841016 Authentication-Results: smtp2.osuosl.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=I9Hfj6Az X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Q5GVDKHaJDlE for ; Tue, 12 Jul 2022 17:45:48 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 0E2A841002 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by smtp2.osuosl.org (Postfix) with ESMTPS id 0E2A841002 for ; Tue, 12 Jul 2022 17:45:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657647948; x=1689183948; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=V+Ud41ctCAiYiQeJVsOe00GfmxrADFKsb3MF17tVj2c=; b=I9Hfj6Azor479F1AL0Z+kDCVK39jprCkKl1CsMaJW4EfU47iyjS0DCdN XsQwwcZtJyorkrOdG/oeQhUPo2WeZBCSW8IgjeNdu6/ewCRu4tt4cIrk+ f4U++4QZR9oF32wUSDbDqn38g3i/J0MyA1ro7F3lMnz63vSaVeil+adr2 xf9Zg+4ebTmfqp0tGoUiib4P2XoVhTDinM+LL0Gx4fl3pbcRGZG4CM1n9 ncEdeswIauZx3FlWyIpRzuYpsmgG5wpsItktB07mIOUJlvRaBBfAtHIvh cmNk9J1a5hfYDL9McUZgEMpxdHoyXJjGxQBPy6dSrSCj4OUrlG+p0j8XE w==; X-IronPort-AV: E=McAfee;i="6400,9594,10406"; a="286130875" X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="286130875" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jul 2022 10:45:47 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="627979782" Received: from silpixa00401454.ir.intel.com ([10.55.128.122]) by orsmga001.jf.intel.com with ESMTP; 12 Jul 2022 10:45:45 -0700 From: Harry van Haaren To: dev@openvswitch.org Date: Tue, 12 Jul 2022 17:44:51 +0000 Message-Id: <20220712174456.2682549-6-harry.van.haaren@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220712174456.2682549-1-harry.van.haaren@intel.com> References: <20220707153900.3147694-1-emma.finn@intel.com> <20220712174456.2682549-1-harry.van.haaren@intel.com> MIME-Version: 1.0 Cc: i.maximets@ovn.org, kumar.amber@intel.com Subject: [ovs-dev] [PATCH v9 05/10] acinclude: Add configure option to enable actions autovalidator at build time. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" From: Kumar Amber This commit adds a new command to allow the user to enable the actions autovalidator by default at build time thus allowing for running unit test by default. $ ./configure --enable-actions-default-autovalidator Signed-off-by: Kumar Amber Acked-by: Harry van Haaren --- v9: - rebase conflict on NEWS - fixup missing "dnl" in comment introduced by previous line-wrapping --- NEWS | 2 ++ acinclude.m4 | 20 ++++++++++++++++++++ configure.ac | 1 + lib/odp-execute.c | 4 ++++ 4 files changed, 27 insertions(+) diff --git a/NEWS b/NEWS index cf35f4ae4..2359b6bcf 100644 --- a/NEWS +++ b/NEWS @@ -53,6 +53,8 @@ Post-v2.17.0 implementations against default implementation. * Add command line option to switch between different actions implementations available at run time. + * Add build time configure command to enable auto-validator as default + actions implementation at build time. - Linux datapath: * Add offloading meter tc police. diff --git a/acinclude.m4 b/acinclude.m4 index d15f11a4e..84fad425c 100644 --- a/acinclude.m4 +++ b/acinclude.m4 @@ -14,6 +14,26 @@ # See the License for the specific language governing permissions and # limitations under the License. +dnl Set OVS Actions Autovalidator as the default action implementation +dnl at compile time This enables automatically running all unit tests +dnl with all actions implementations. +AC_DEFUN([OVS_CHECK_ACTIONS_AUTOVALIDATOR], [ + AC_ARG_ENABLE([actions-default-autovalidator], + [AC_HELP_STRING([--enable-actions-default-autovalidator], + [Enable actions autovalidator as default + ovs actions implementation.])], + [autovalidator=yes],[autovalidator=no]) + AC_MSG_CHECKING([whether actions Autovalidator is default implementation]) + if test "$autovalidator" != yes; then + AC_MSG_RESULT([no]) + else + AC_DEFINE([ACTIONS_AUTOVALIDATOR_DEFAULT], [1], + [Autovalidator for actions is a default implementation.]) + AC_MSG_RESULT([yes]) + fi +]) + + dnl Set OVS MFEX Autovalidator as default miniflow extract at compile time? dnl This enables automatically running all unit tests with all MFEX dnl implementations. diff --git a/configure.ac b/configure.ac index 6f8679d7c..6c51e48ce 100644 --- a/configure.ac +++ b/configure.ac @@ -184,6 +184,7 @@ OVS_CONDITIONAL_CC_OPTION([-Wno-unused-parameter], [HAVE_WNO_UNUSED_PARAMETER]) OVS_ENABLE_WERROR_TOP OVS_ENABLE_SPARSE OVS_CTAGS_IDENTIFIERS +OVS_CHECK_ACTIONS_AUTOVALIDATOR OVS_CHECK_DPCLS_AUTOVALIDATOR OVS_CHECK_DPIF_AVX512_DEFAULT OVS_CHECK_MFEX_AUTOVALIDATOR diff --git a/lib/odp-execute.c b/lib/odp-execute.c index 64c058a75..f713acabe 100644 --- a/lib/odp-execute.c +++ b/lib/odp-execute.c @@ -928,7 +928,11 @@ odp_execute_init(void) static struct ovsthread_once once = OVSTHREAD_ONCE_INITIALIZER; if (ovsthread_once_start(&once)) { odp_execute_action_init(); +#ifdef ACTIONS_AUTOVALIDATOR_DEFAULT + odp_actions_impl_set("autovalidator"); +#else odp_actions_impl_set("scalar"); +#endif odp_execute_unixctl_init(); ovsthread_once_done(&once); } From patchwork Tue Jul 12 17:44:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Van Haaren, Harry" X-Patchwork-Id: 1655547 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=T4IPxPJ5; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lj7TT5vYNz9s07 for ; Wed, 13 Jul 2022 03:46:05 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 3725F421C9; Tue, 12 Jul 2022 17:46:03 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 3725F421C9 Authentication-Results: smtp4.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=T4IPxPJ5 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1EgZqzO6DS8H; Tue, 12 Jul 2022 17:46:01 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTPS id 29E0341B99; Tue, 12 Jul 2022 17:46:00 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 29E0341B99 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id CC843C0071; Tue, 12 Jul 2022 17:45:59 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 455A6C0071 for ; Tue, 12 Jul 2022 17:45:58 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 6CEFA40FDE for ; Tue, 12 Jul 2022 17:45:53 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 6CEFA40FDE Authentication-Results: smtp2.osuosl.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=T4IPxPJ5 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cXYwoj11XcFL for ; Tue, 12 Jul 2022 17:45:51 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 7A53340FCE Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by smtp2.osuosl.org (Postfix) with ESMTPS id 7A53340FCE for ; Tue, 12 Jul 2022 17:45:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657647951; x=1689183951; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=lYfuOPhGQD3Egmn63enJ6/Dky5Lp+KONZ+4zYrhUOxk=; b=T4IPxPJ53+WOD3FtbdWq1t4YM76q47sJji7Tpe183Hldk9cCSFQ9BZER EYxuH6d/6K7gRkDFywYga0bpusggzcJ0qwCljypbuwe5ZoK/GtTYwW6Zv okWjkc2sE2WZ9EgK3O3WLCwuLeXelItnphrj2Xm+ChyEWCdywLL+x6Y4+ pdbqpzzZc42XgsfBkn4XSO5YPnA4Y36BAHB9GAhhxv2nyU6zmf4Llfjgu T6eszT02QzeSzTQrq3bdCFzMtXCHq/4LTzT7fH1iSWbC8SBWNBepfqR2h a9lLzEtFmqkuYHzEZnGQj1jCccp0rRtK4qh4Nff1z0BBwB23ejVnnPFTP g==; X-IronPort-AV: E=McAfee;i="6400,9594,10406"; a="286130882" X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="286130882" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jul 2022 10:45:49 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="627979792" Received: from silpixa00401454.ir.intel.com ([10.55.128.122]) by orsmga001.jf.intel.com with ESMTP; 12 Jul 2022 10:45:47 -0700 From: Harry van Haaren To: dev@openvswitch.org Date: Tue, 12 Jul 2022 17:44:52 +0000 Message-Id: <20220712174456.2682549-7-harry.van.haaren@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220712174456.2682549-1-harry.van.haaren@intel.com> References: <20220707153900.3147694-1-emma.finn@intel.com> <20220712174456.2682549-1-harry.van.haaren@intel.com> MIME-Version: 1.0 Cc: i.maximets@ovn.org, kumar.amber@intel.com Subject: [ovs-dev] [PATCH v9 06/10] odp-execute: Add ISA implementation of actions. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" From: Emma Finn This commit adds the AVX512 implementation of the action functionality. Usage: $ ovs-appctl odp-execute/action-impl-set avx512 Signed-off-by: Emma Finn Acked-by: Harry van Haaren --- v9: rebase conflict on NEWS --- Documentation/ref/ovs-actions.7.rst | 26 ++++++++++++++++++++++++++ Documentation/topics/testing.rst | 24 ++++++++++++++++-------- NEWS | 1 + lib/cpu.c | 1 + lib/cpu.h | 1 + lib/odp-execute-private.c | 8 ++++++++ lib/odp-execute-private.h | 6 ++++++ 7 files changed, 59 insertions(+), 8 deletions(-) diff --git a/Documentation/ref/ovs-actions.7.rst b/Documentation/ref/ovs-actions.7.rst index b59b7634f..2410acc4a 100644 --- a/Documentation/ref/ovs-actions.7.rst +++ b/Documentation/ref/ovs-actions.7.rst @@ -125,6 +125,32 @@ the one added to the set later replaces the earlier action: An action set may only contain the actions listed above. +Actions Implementations (Experimental) +-------------------------------------- + +Actions are used in OpenFlow flows to describe what to do when the flow +matches a packet. Just like with the datapath interface, SIMD instructions +with the userspace datapath can be applied to the action implementation to +improve performance. + +OVS provides multiple implementations of the actions. +Available implementations can be listed with the following command:: + + $ ovs-appctl odp-execute/action-impl-show + Available Actions implementations: + scalar (available: Yes, active: Yes) + autovalidator (available: Yes, active: No) + avx512 (available: Yes, active: No) + +By default, ``scalar`` is used. Implementations can be selected by +name:: + + $ ovs-appctl odp-execute/action-impl-set avx512 + Action implementation set to avx512. + + $ ovs-appctl odp-execute/action-impl-set scalar + Action implementation set to scalar. + Error Handling -------------- diff --git a/Documentation/topics/testing.rst b/Documentation/topics/testing.rst index c15d5b38f..a6c747b18 100644 --- a/Documentation/topics/testing.rst +++ b/Documentation/topics/testing.rst @@ -361,12 +361,12 @@ testsuite. Userspace datapath: Testing and Validation of CPU-specific Optimizations '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' -As multiple versions of the datapath classifier and packet parsing functions -can co-exist, each with different CPU ISA optimizations, it is important to -validate that they all give the exact same results. To easily test all the -implementations, an ``autovalidator`` implementation of them exists. This -implementation runs all other available implementations, and verifies that the -results are identical. +As multiple versions of the datapath classifier, packet parsing functions and +actions can co-exist, each with different CPU ISA optimizations, it is +important to validate that they all give the exact same results. To easily +test all the implementations, an ``autovalidator`` implementation of them +exists. This implementation runs all other available implementations, and +verifies that the results are identical. Running the OVS unit tests with the autovalidator enabled ensures all implementations provide the same results. Note that the performance of the @@ -382,18 +382,26 @@ To set the autovalidator for the packet parser, use this command:: $ ovs-appctl dpif-netdev/miniflow-parser-set autovalidator +To set the autovalidator for actions, use this command:: + + $ ovs-appctl odp-execute/action-impl-set autovalidator + To run the OVS unit test suite with the autovalidator as the default implementation, it is required to recompile OVS. During the recompilation, the default priority of the `autovalidator` implementation is set to the -maximum priority, ensuring every test will be run with every implementation:: +maximum priority, ensuring every test will be run with every implementation. +Priority is only related to mfex autovalidator and not the actions +autovalidator.:: - $ ./configure --enable-autovalidator --enable-mfex-default-autovalidator + $ ./configure --enable-autovalidator --enable-mfex-default-autovalidator \ + --enable-actions-default-autovalidator The following line should be seen in the configuration log when the above options are used:: checking whether DPCLS Autovalidator is default implementation... yes checking whether MFEX Autovalidator is default implementation... yes + checking whether actions Autovalidator is default implementation... yes Compile OVS in debug mode to have `ovs_assert` statements error out if there is a mis-match in the datapath classifier lookup or packet parser diff --git a/NEWS b/NEWS index 2359b6bcf..fa2f7d535 100644 --- a/NEWS +++ b/NEWS @@ -55,6 +55,7 @@ Post-v2.17.0 implementations available at run time. * Add build time configure command to enable auto-validator as default actions implementation at build time. + * Add AVX512 implementation of actions. - Linux datapath: * Add offloading meter tc police. diff --git a/lib/cpu.c b/lib/cpu.c index 2df003c51..0292f715e 100644 --- a/lib/cpu.c +++ b/lib/cpu.c @@ -53,6 +53,7 @@ X86_ISA(X86_EXT_FEATURES_LEAF, EBX, 16, OVS_CPU_ISA_X86_AVX512F) X86_ISA(X86_EXT_FEATURES_LEAF, EBX, 30, OVS_CPU_ISA_X86_AVX512BW) X86_ISA(X86_EXT_FEATURES_LEAF, ECX, 1, OVS_CPU_ISA_X86_AVX512VBMI) X86_ISA(X86_EXT_FEATURES_LEAF, ECX, 14, OVS_CPU_ISA_X86_VPOPCNTDQ) +X86_ISA(X86_EXT_FEATURES_LEAF, EBX, 31, OVS_CPU_ISA_X86_AVX512VL) #endif bool diff --git a/lib/cpu.h b/lib/cpu.h index 92897bb71..3215229bc 100644 --- a/lib/cpu.h +++ b/lib/cpu.h @@ -25,6 +25,7 @@ enum ovs_cpu_isa { OVS_CPU_ISA_X86_AVX512F, OVS_CPU_ISA_X86_AVX512BW, OVS_CPU_ISA_X86_AVX512VBMI, + OVS_CPU_ISA_X86_AVX512VL, OVS_CPU_ISA_X86_VPOPCNTDQ, OVS_CPU_ISA_X86_LAST = OVS_CPU_ISA_X86_VPOPCNTDQ, }; diff --git a/lib/odp-execute-private.c b/lib/odp-execute-private.c index 38be22ec9..f7fb60467 100644 --- a/lib/odp-execute-private.c +++ b/lib/odp-execute-private.c @@ -41,6 +41,14 @@ static struct odp_execute_action_impl action_impls[] = { .name = "scalar", .init_func = odp_action_scalar_init, }, + +#ifdef ACTION_IMPL_AVX512_CHECK + [ACTION_IMPL_AVX512] = { + .available = false, + .name = "avx512", + .init_func = NULL, + }, +#endif }; static void diff --git a/lib/odp-execute-private.h b/lib/odp-execute-private.h index d6eebbf37..1c636faeb 100644 --- a/lib/odp-execute-private.h +++ b/lib/odp-execute-private.h @@ -22,6 +22,9 @@ #include "odp-netlink.h" #include "ovs-atomic.h" +#define ACTION_IMPL_AVX512_CHECK (__x86_64__ && HAVE_AVX512F \ + && HAVE_LD_AVX512_GOOD && __SSE4_2__) + /* Forward declaration for typedef. */ struct odp_execute_action_impl; @@ -59,6 +62,9 @@ enum odp_execute_action_impl_idx { * Do not change the autovalidator position in this list without updating * the define below. */ +#ifdef ACTION_IMPL_AVX512_CHECK + ACTION_IMPL_AVX512, +#endif ACTION_IMPL_MAX, }; From patchwork Tue Jul 12 17:44:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Van Haaren, Harry" X-Patchwork-Id: 1655549 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=YR/1E1gW; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lj7Tq0sDBz9s07 for ; Wed, 13 Jul 2022 03:46:23 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 0F84D422EF; Tue, 12 Jul 2022 17:46:19 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 0F84D422EF Authentication-Results: smtp4.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=YR/1E1gW X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oylsvaiSR7qF; Tue, 12 Jul 2022 17:46:16 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTPS id D76A9422A5; Tue, 12 Jul 2022 17:46:14 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org D76A9422A5 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2D5A7C0083; Tue, 12 Jul 2022 17:46:12 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 52232C002D for ; Tue, 12 Jul 2022 17:46:10 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id E622741041 for ; Tue, 12 Jul 2022 17:45:55 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org E622741041 Authentication-Results: smtp2.osuosl.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=YR/1E1gW X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RT4gMbEG0PZn for ; Tue, 12 Jul 2022 17:45:53 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org AD2F340FE2 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by smtp2.osuosl.org (Postfix) with ESMTPS id AD2F340FE2 for ; Tue, 12 Jul 2022 17:45:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657647952; x=1689183952; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=kX1fk+eLl4ZtbSKHIjluCi9N2EyYSEUJ/I0JI2z+Prw=; b=YR/1E1gW7SaalNHfT1YqB7OAuAXAEiFr8mzTxobrJWYQSNREyxAUdGIg /nR8EaKhsZ+8hNWFiQDgKPnBy04jvY8ulJfU7WX4Bd7hPG9oLizQMoaUb dlh728MA11kBli1prxQgjMLhM3yvLceC20kOVzYsF/6MJpvr6O7smfXcv kHceMgxjrcf/yHixiS+xVlXFOE8Fr5q4YSJtb9J4qSYLRAdAUXfu3p7Ha HnGn9r3PaRP128VXpb9hc0e4bUGHF2Ixovpm4xtVk1EICoKd0Qsfi2n5j z6dbuzAKn3ieOgIiRDLBuOl1Emmpm6/YoIowS50bfJmuvneJP3Tz9cYRk g==; X-IronPort-AV: E=McAfee;i="6400,9594,10406"; a="286130890" X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="286130890" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jul 2022 10:45:52 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="627979808" Received: from silpixa00401454.ir.intel.com ([10.55.128.122]) by orsmga001.jf.intel.com with ESMTP; 12 Jul 2022 10:45:50 -0700 From: Harry van Haaren To: dev@openvswitch.org Date: Tue, 12 Jul 2022 17:44:53 +0000 Message-Id: <20220712174456.2682549-8-harry.van.haaren@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220712174456.2682549-1-harry.van.haaren@intel.com> References: <20220707153900.3147694-1-emma.finn@intel.com> <20220712174456.2682549-1-harry.van.haaren@intel.com> MIME-Version: 1.0 Cc: i.maximets@ovn.org, kumar.amber@intel.com Subject: [ovs-dev] [PATCH v9 07/10] odp-execute: Add ISA implementation of pop_vlan action. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" From: Emma Finn This commit adds the AVX512 implementation of the pop_vlan action. Signed-off-by: Emma Finn --- lib/automake.mk | 3 +- lib/odp-execute-avx512.c | 182 ++++++++++++++++++++++++++++++++++++++ lib/odp-execute-private.c | 33 ++++++- lib/odp-execute-private.h | 2 + 4 files changed, 218 insertions(+), 2 deletions(-) create mode 100644 lib/odp-execute-avx512.c diff --git a/lib/automake.mk b/lib/automake.mk index 5c3b05f6b..4ce5cc1ff 100644 --- a/lib/automake.mk +++ b/lib/automake.mk @@ -44,7 +44,8 @@ lib_libopenvswitchavx512_la_CFLAGS += \ -mavx512vl lib_libopenvswitchavx512_la_SOURCES += \ lib/dpif-netdev-extract-avx512.c \ - lib/dpif-netdev-lookup-avx512-gather.c + lib/dpif-netdev-lookup-avx512-gather.c \ + lib/odp-execute-avx512.c endif # HAVE_AVX512VL endif # HAVE_AVX512BW lib_libopenvswitchavx512_la_LDFLAGS = \ diff --git a/lib/odp-execute-avx512.c b/lib/odp-execute-avx512.c new file mode 100644 index 000000000..fd10f7f5c --- /dev/null +++ b/lib/odp-execute-avx512.c @@ -0,0 +1,182 @@ +/* + * Copyright (c) 2022 Intel. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifdef __x86_64__ +/* Sparse cannot handle the AVX512 instructions. */ +#if !defined(__CHECKER__) + +#include +#include + +#include "dp-packet.h" +#include "immintrin.h" +#include "odp-execute-private.h" +#include "odp-netlink.h" +#include "openvswitch/vlog.h" + +VLOG_DEFINE_THIS_MODULE(odp_execute_avx512); + +/* The below three build asserts make sure that l2_5_ofs, l3_ofs, and l4_ofs + * fields remain in the same order and offset to l2_padd_size. This is needed + * as the avx512_dp_packet_resize_l2() function will manipulate those fields at + * a fixed memory index based on the l2_padd_size offset. */ +BUILD_ASSERT_DECL(offsetof(struct dp_packet, l2_pad_size) + + MEMBER_SIZEOF(struct dp_packet, l2_pad_size) == + offsetof(struct dp_packet, l2_5_ofs)); + +BUILD_ASSERT_DECL(offsetof(struct dp_packet, l2_5_ofs) + + MEMBER_SIZEOF(struct dp_packet, l2_5_ofs) == + offsetof(struct dp_packet, l3_ofs)); + +BUILD_ASSERT_DECL(offsetof(struct dp_packet, l3_ofs) + + MEMBER_SIZEOF(struct dp_packet, l3_ofs) == + offsetof(struct dp_packet, l4_ofs)); + +/* The below build assert makes sure it's safe to read/write 128-bits starting + * at the l2_pad_size location. */ +BUILD_ASSERT_DECL(sizeof(struct dp_packet) - + offsetof(struct dp_packet, l2_pad_size) >= sizeof(__m128i)); + +static inline void ALWAYS_INLINE +avx512_dp_packet_resize_l2(struct dp_packet *b, int resize_by_bytes) +{ + /* Update packet size/data pointers, same as the scalar implementation. */ + if (resize_by_bytes >= 0) { + dp_packet_push_uninit(b, resize_by_bytes); + } else { + dp_packet_pull(b, -resize_by_bytes); + } + + /* The next step is to update the l2_5_ofs, l3_ofs and l4_ofs fields which + * the scalar implementation does with the dp_packet_adjust_layer_offset() + * function. */ + + /* Set the v_zero register to all zero's. */ + const __m128i v_zeros = _mm_setzero_si128(); + + /* Set the v_u16_max register to all one's. */ + const __m128i v_u16_max = _mm_cmpeq_epi16(v_zeros, v_zeros); + + /* Each lane represents 16 bits in a 12-bit register. In this case the + * first three 16-bit values, which will map to the l2_5_ofs, l3_ofs and + * l4_ofs fields. */ + const uint8_t k_lanes = 0b1110; + + /* Set all 16-bit words in the 128-bits v_offset register to the value we + * need to add/substract from the l2_5_ofs, l3_ofs, and l4_ofs fields. */ + __m128i v_offset = _mm_set1_epi16(abs(resize_by_bytes)); + + /* Load 128 bits from the dp_packet structure starting at the l2_pad_size + * offset. */ + void *adjust_ptr = &b->l2_pad_size; + __m128i v_adjust_src = _mm_loadu_si128(adjust_ptr); + + /* Here is the tricky part, we only need to update the value of the three + * fields if they are not UINT16_MAX. The following function will return + * a mask of lanes (read fields) that are not UINT16_MAX. It will do this + * by comparing only the lanes we requested, k_lanes, and if they match + * v_u16_max, the bit will be set. */ + __mmask8 k_cmp = _mm_mask_cmpneq_epu16_mask(k_lanes, v_adjust_src, + v_u16_max); + + /* Based on the bytes adjust (positive, or negative) it will do the actual + * add or subtraction. These functions will only operate on the lanes + * (fields) requested based on k_cmp, i.e: + * k_cmp = [l2_5_ofs, l3_ofs, l4_ofs] + * for field in kcmp + * v_adjust_src[field] = v_adjust_src[field] + v_offset + */ + __m128i v_adjust_wip; + + if (resize_by_bytes >= 0) { + v_adjust_wip = _mm_mask_add_epi16(v_adjust_src, k_cmp, + v_adjust_src, v_offset); + } else { + v_adjust_wip = _mm_mask_sub_epi16(v_adjust_src, k_cmp, + v_adjust_src, v_offset); + } + + /* Here we write back the full 128-bits. */ + _mm_storeu_si128(adjust_ptr, v_adjust_wip); +} + +/* This function performs the same operation on each packet in the batch as + * the scalar eth_pop_vlan() function. */ +static void +action_avx512_pop_vlan(struct dp_packet_batch *batch, + const struct nlattr *a OVS_UNUSED) +{ + struct dp_packet *packet; + + /* Set the v_zero register to all zero's. */ + const __m128i v_zeros = _mm_setzero_si128(); + + DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { + struct vlan_eth_header *veh = dp_packet_eth(packet); + + if (veh && dp_packet_size(packet) >= sizeof *veh && + eth_type_vlan(veh->veth_type)) { + + /* Load the first 128-bits of l2 header into the v_ether register. + * This result in the veth_dst/src and veth_type/tci of the + * vlan_eth_header structure to be loaded. */ + __m128i v_ether = _mm_loadu_si128((void *) veh); + + /* This creates a 256-bit value containing the first four fields + * of the vlan_eth_header plus 128 zero-bit. The result will be the + * lowest 128-bits after the right shift, hence we shift the data + * 128(zero)-bits minus the VLAN_HEADER_LEN, so we are left with + * only the veth_dst and veth_src fields. */ + __m128i v_realign = _mm_alignr_epi8(v_ether, v_zeros, + sizeof(__m128i) - + VLAN_HEADER_LEN); + + /* Write back the modified ethernet header. */ + _mm_storeu_si128((void *) veh, v_realign); + + /* As we removed the VLAN_HEADER we now need to adjust all the + * offsets. */ + avx512_dp_packet_resize_l2(packet, -VLAN_HEADER_LEN); + } + } +} + +int +action_avx512_init(struct odp_execute_action_impl *self OVS_UNUSED) +{ + /* Set function pointers for actions that can be applied directly, these + * are identified by OVS_ACTION_ATTR_*. */ + self->funcs[OVS_ACTION_ATTR_POP_VLAN] = action_avx512_pop_vlan; + return 0; +} + +#endif /* Sparse */ + +#else /* __x86_64__ */ + +#include +#include "odp-execute-private.h" +/* Function itself is required to be called, even in e.g. 32-bit builds. + * This dummy init function ensures 32-bit builds succeed too. + */ + +int +action_avx512_init(struct odp_execute_action_impl *self OVS_UNUSED) +{ + return 0; +} + +#endif diff --git a/lib/odp-execute-private.c b/lib/odp-execute-private.c index f7fb60467..ad736523b 100644 --- a/lib/odp-execute-private.c +++ b/lib/odp-execute-private.c @@ -19,6 +19,7 @@ #include #include +#include "cpu.h" #include "dpdk.h" #include "dp-packet.h" #include "odp-execute-private.h" @@ -29,6 +30,36 @@ VLOG_DEFINE_THIS_MODULE(odp_execute_impl); static int active_action_impl_index; +#ifdef ACTION_IMPL_AVX512_CHECK +/* Probe functions to check ISA requirements. */ +static bool +action_avx512_isa_probe(void) +{ + static enum ovs_cpu_isa isa_required[] = { + OVS_CPU_ISA_X86_AVX512F, + OVS_CPU_ISA_X86_AVX512BW, + OVS_CPU_ISA_X86_BMI2, + OVS_CPU_ISA_X86_AVX512VL, + }; + for (int i = 0; i < ARRAY_SIZE(isa_required); i++) { + if (!cpu_has_isa(isa_required[i])) { + return false; + } + } + return true; +} +static int +action_avx512_probe(struct odp_execute_action_impl *self) +{ + if (!action_avx512_isa_probe()) { + return -ENOTSUP; + } else { + action_avx512_init(self); + } + return 0; +} +#endif + static struct odp_execute_action_impl action_impls[] = { [ACTION_IMPL_AUTOVALIDATOR] = { .available = false, @@ -46,7 +77,7 @@ static struct odp_execute_action_impl action_impls[] = { [ACTION_IMPL_AVX512] = { .available = false, .name = "avx512", - .init_func = NULL, + .init_func = action_avx512_probe, }, #endif }; diff --git a/lib/odp-execute-private.h b/lib/odp-execute-private.h index 1c636faeb..b3a10cd82 100644 --- a/lib/odp-execute-private.h +++ b/lib/odp-execute-private.h @@ -90,6 +90,8 @@ struct odp_execute_action_impl * odp_execute_action_set(const char *name); int action_autoval_init(struct odp_execute_action_impl *self); +int action_avx512_init(struct odp_execute_action_impl *self); + void odp_execute_action_get_info(struct ds *name); #endif /* ODP_EXTRACT_PRIVATE */ From patchwork Tue Jul 12 17:44:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Van Haaren, Harry" X-Patchwork-Id: 1655548 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=ZpTjC09d; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lj7Tg4VbTz9s07 for ; Wed, 13 Jul 2022 03:46:15 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id B322942296; Tue, 12 Jul 2022 17:46:13 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org B322942296 Authentication-Results: smtp4.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=ZpTjC09d X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id uvCNxKUTciG3; Tue, 12 Jul 2022 17:46:12 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id 455CB41BAF; Tue, 12 Jul 2022 17:46:11 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 455CB41BAF Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2216AC007D; Tue, 12 Jul 2022 17:46:11 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1556DC002D for ; Tue, 12 Jul 2022 17:46:10 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 25E4341BAF for ; Tue, 12 Jul 2022 17:45:56 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 25E4341BAF X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id azPuza7zXLip for ; Tue, 12 Jul 2022 17:45:55 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 1DC6241BE2 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by smtp4.osuosl.org (Postfix) with ESMTPS id 1DC6241BE2 for ; Tue, 12 Jul 2022 17:45:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657647955; x=1689183955; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=QnY9YYOfxDip6m5srAAEeKp140v0TZiPMO7Mvqv9aq8=; b=ZpTjC09deeL99FzHNlbvsTDykg5chXY17aH/MH6rYlhtNz80LtE4revi xpEGKh6Q93q4bLogeRjqxGZ6w2rS94P/0F07MaFuUZy9aU+9l9I4mDdsF 3KE1z8BqT0YMgWk2RpjcMx5ZhyBlRBdbe1UnPqRPTXRqQma5jS24xmWEp UY2dDexp+31lo3vELfVKdj21YGDMKPmCCeQzW0FwHXW0PB5ZQhwf8O+jx aONUn3oF9ILfK8VF+s/jxSeej1neP7hREqGzD5gHxVGRf5rvMOInuL1Ht 0d3LXjO/YKDDX/p4bEajyHZ6CsEvDlJEJwi0LP1bhr28MnkkJj27QpEpi Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10406"; a="286130896" X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="286130896" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jul 2022 10:45:54 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="627979814" Received: from silpixa00401454.ir.intel.com ([10.55.128.122]) by orsmga001.jf.intel.com with ESMTP; 12 Jul 2022 10:45:52 -0700 From: Harry van Haaren To: dev@openvswitch.org Date: Tue, 12 Jul 2022 17:44:54 +0000 Message-Id: <20220712174456.2682549-9-harry.van.haaren@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220712174456.2682549-1-harry.van.haaren@intel.com> References: <20220707153900.3147694-1-emma.finn@intel.com> <20220712174456.2682549-1-harry.van.haaren@intel.com> MIME-Version: 1.0 Cc: i.maximets@ovn.org, kumar.amber@intel.com Subject: [ovs-dev] [PATCH v9 08/10] odp-execute: Add ISA implementation of push_vlan action. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" From: Emma Finn This commit adds the AVX512 implementation of the push_vlan action. Signed-off-by: Emma Finn --- lib/odp-execute-avx512.c | 55 ++++++++++++++++++++++++++++++++++++++++ lib/odp-execute.c | 22 +++++++++------- 2 files changed, 68 insertions(+), 9 deletions(-) diff --git a/lib/odp-execute-avx512.c b/lib/odp-execute-avx512.c index fd10f7f5c..3449acff7 100644 --- a/lib/odp-execute-avx512.c +++ b/lib/odp-execute-avx512.c @@ -154,12 +154,67 @@ action_avx512_pop_vlan(struct dp_packet_batch *batch, } } +/* This function performs the same operation on each packet in the batch as + * the scalar eth_push_vlan() function. */ +static void +action_avx512_push_vlan(struct dp_packet_batch *batch, const struct nlattr *a) +{ + struct dp_packet *packet; + const struct ovs_action_push_vlan *vlan = nl_attr_get(a); + ovs_be16 tpid, tci; + + /* This shuffle mask is used below, and each position tells where to + * move the bytes to. So here, the fourth byte in v_ether is moved to + * byte location 0 in v_shift. The fifth is moved to 1, etc., etc. + * The 0xFF is special it tells to fill that position with 0. + */ + static const uint8_t vlan_push_shuffle_mask[16] = { + 4, 5, 6, 7, 8, 9, 10, 11, + 12, 13, 14, 15, 0xFF, 0xFF, 0xFF, 0xFF + }; + + /* Load the shuffle mask in v_index. */ + __m128i v_index = _mm_loadu_si128((void *) vlan_push_shuffle_mask); + + DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { + tpid = vlan->vlan_tpid; + tci = vlan->vlan_tci; + + /* As we are about to insert the VLAN_HEADER we now need to adjust all + * the offsets. */ + avx512_dp_packet_resize_l2(packet, VLAN_HEADER_LEN); + + char *pkt_data = (char *) dp_packet_data(packet); + + /* Build up the VLAN TCI/TPID in a single uint32_t. */ + const uint32_t tci_proc = tci & htons(~VLAN_CFI); + const uint32_t tpid_tci = (tci_proc << 16) | tpid; + + /* Load the first 128-bits of the packet into the v_ether register. + * Note that this includes the 4 unused bytes (VLAN_HEADER_LEN). */ + __m128i v_ether = _mm_loadu_si128((void *) pkt_data); + + /* Move(shuffle) the veth_dst and veth_src data to create room for + * the vlan header. */ + __m128i v_shift = _mm_shuffle_epi8(v_ether, v_index); + + /* Copy(insert) the 32-bit VLAN header, tpid_tci, at the 3rd 32-bit + * word offset, i.e. ofssetof(vlan_eth_header, veth_type) */ + __m128i v_vlan_hdr = _mm_insert_epi32(v_shift, tpid_tci, 3); + + /* Write back the modified ethernet header. */ + _mm_storeu_si128((void *) pkt_data, v_vlan_hdr); + } +} + int action_avx512_init(struct odp_execute_action_impl *self OVS_UNUSED) { /* Set function pointers for actions that can be applied directly, these * are identified by OVS_ACTION_ATTR_*. */ self->funcs[OVS_ACTION_ATTR_POP_VLAN] = action_avx512_pop_vlan; + self->funcs[OVS_ACTION_ATTR_PUSH_VLAN] = action_avx512_push_vlan; + return 0; } diff --git a/lib/odp-execute.c b/lib/odp-execute.c index f713acabe..3f562eb34 100644 --- a/lib/odp-execute.c +++ b/lib/odp-execute.c @@ -846,6 +846,17 @@ action_pop_vlan(struct dp_packet_batch *batch, } } +static void +action_push_vlan(struct dp_packet_batch *batch, const struct nlattr *a) +{ + struct dp_packet *packet; + const struct ovs_action_push_vlan *vlan = nl_attr_get(a); + + DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { + eth_push_vlan(packet, vlan->vlan_tpid, vlan->vlan_tci); + } +} + /* Implementation of the scalar actions impl init function. Build up the * array of func ptrs here. */ @@ -855,6 +866,7 @@ odp_action_scalar_init(struct odp_execute_action_impl *self) /* Set function pointers for actions that can be applied directly, these * are identified by OVS_ACTION_ATTR_*. */ self->funcs[OVS_ACTION_ATTR_POP_VLAN] = action_pop_vlan; + self->funcs[OVS_ACTION_ATTR_PUSH_VLAN] = action_push_vlan; return 0; } @@ -1045,15 +1057,6 @@ odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, break; } - case OVS_ACTION_ATTR_PUSH_VLAN: { - const struct ovs_action_push_vlan *vlan = nl_attr_get(a); - - DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { - eth_push_vlan(packet, vlan->vlan_tpid, vlan->vlan_tci); - } - break; - } - case OVS_ACTION_ATTR_PUSH_MPLS: { const struct ovs_action_push_mpls *mpls = nl_attr_get(a); @@ -1206,6 +1209,7 @@ odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, case __OVS_ACTION_ATTR_MAX: /* The following actions are handled by the scalar implementation. */ case OVS_ACTION_ATTR_POP_VLAN: + case OVS_ACTION_ATTR_PUSH_VLAN: OVS_NOT_REACHED(); } From patchwork Tue Jul 12 17:44:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Van Haaren, Harry" X-Patchwork-Id: 1655550 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=Ka0PqF2W; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lj7Ty6p0lz9s07 for ; Wed, 13 Jul 2022 03:46:30 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 2DD0A41023; Tue, 12 Jul 2022 17:46:28 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 2DD0A41023 Authentication-Results: smtp2.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=Ka0PqF2W X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DgIB5QwI4SGm; Tue, 12 Jul 2022 17:46:24 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id AF5D941053; Tue, 12 Jul 2022 17:46:21 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org AF5D941053 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 67BF5C0071; Tue, 12 Jul 2022 17:46:21 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1AE9FC0035 for ; Tue, 12 Jul 2022 17:46:20 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 2F2E441B80 for ; Tue, 12 Jul 2022 17:45:59 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 2F2E441B80 Authentication-Results: smtp4.osuosl.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=Ka0PqF2W X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xHt6kmVLU-aw for ; Tue, 12 Jul 2022 17:45:57 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 4286341B77 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by smtp4.osuosl.org (Postfix) with ESMTPS id 4286341B77 for ; Tue, 12 Jul 2022 17:45:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657647957; x=1689183957; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=RhixqbxIxd/koe10BghIisYyniluIJgcFaijHQAt1TI=; b=Ka0PqF2WNK8YpLr2xa6ylHov715fDfR6bAvqW+MUIjqZ7LwD0JWwYZ04 8AHQ9hQogS7TUQpwmZ97Yz9oq5u7kmBNsDhwFzLrbsDlDQxHqmddVp2bS cpZyuwbVAZsA4e17TIsDh1mBM4xvizkGZH07MlzE5+u5WFKqchm1ZP5+2 2LHlHaGvi24jxG8+8s7K5bFP4gAU20zSzjaA70bIIxvETB6ntAVNxrhst 9QswbBlik/W09Xwd5VyflM7U8+XWH3QrNVmuDywmTVXfiqqDMTYSYugJP rOxmCwDXQKqd8eC6lzxsEFlvJeLgTjgt2FmlL44+dOmWEquKuEV1qAN4f g==; X-IronPort-AV: E=McAfee;i="6400,9594,10406"; a="286130901" X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="286130901" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jul 2022 10:45:56 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="627979830" Received: from silpixa00401454.ir.intel.com ([10.55.128.122]) by orsmga001.jf.intel.com with ESMTP; 12 Jul 2022 10:45:54 -0700 From: Harry van Haaren To: dev@openvswitch.org Date: Tue, 12 Jul 2022 17:44:55 +0000 Message-Id: <20220712174456.2682549-10-harry.van.haaren@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220712174456.2682549-1-harry.van.haaren@intel.com> References: <20220707153900.3147694-1-emma.finn@intel.com> <20220712174456.2682549-1-harry.van.haaren@intel.com> MIME-Version: 1.0 Cc: i.maximets@ovn.org, kumar.amber@intel.com Subject: [ovs-dev] [PATCH v9 09/10] odp-execute: Add ISA implementation of set_masked ETH X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" From: Emma Finn This commit includes infrastructure changes for enabling set_masked_X actions and also adds support for the AVX512 implementation of the eth_set_addrs action. Signed-off-by: Emma Finn --- lib/odp-execute-avx512.c | 90 +++++++++++++++++++++++++++++++++++++++ lib/odp-execute-private.c | 14 ++++++ lib/odp-execute-private.h | 3 ++ lib/odp-execute.c | 49 +++++++++++---------- lib/odp-execute.h | 3 ++ 5 files changed, 137 insertions(+), 22 deletions(-) diff --git a/lib/odp-execute-avx512.c b/lib/odp-execute-avx512.c index 3449acff7..8ecdaecf6 100644 --- a/lib/odp-execute-avx512.c +++ b/lib/odp-execute-avx512.c @@ -23,6 +23,7 @@ #include "dp-packet.h" #include "immintrin.h" +#include "odp-execute.h" #include "odp-execute-private.h" #include "odp-netlink.h" #include "openvswitch/vlog.h" @@ -50,6 +51,16 @@ BUILD_ASSERT_DECL(offsetof(struct dp_packet, l3_ofs) + BUILD_ASSERT_DECL(sizeof(struct dp_packet) - offsetof(struct dp_packet, l2_pad_size) >= sizeof(__m128i)); +/* The below build assert makes sure the order of the fields needed by + * the set masked functions shuffle operations do not change. This should not + * happen as these are defined under the Linux uapi. */ +BUILD_ASSERT_DECL(offsetof(struct ovs_key_ethernet, eth_src) + + MEMBER_SIZEOF(struct ovs_key_ethernet, eth_src) == + offsetof(struct ovs_key_ethernet, eth_dst)); + +/* Array of callback functions, one for each masked operation. */ +odp_execute_action_cb impl_set_masked_funcs[__OVS_KEY_ATTR_MAX]; + static inline void ALWAYS_INLINE avx512_dp_packet_resize_l2(struct dp_packet *b, int resize_by_bytes) { @@ -207,6 +218,80 @@ action_avx512_push_vlan(struct dp_packet_batch *batch, const struct nlattr *a) } } +/* This function performs the same operation on each packet in the batch as + * the scalar odp_eth_set_addrs() function. */ +static void +action_avx512_eth_set_addrs(struct dp_packet_batch *batch, + const struct nlattr *a) +{ + const struct ovs_key_ethernet *key, *mask; + struct dp_packet *packet; + + a = nl_attr_get(a); + key = nl_attr_get(a); + mask = odp_get_key_mask(a, struct ovs_key_ethernet); + + /* Read the content of the key(src) and mask in the respective registers. + * We only load the src and dest addresses, which is only 96-bits and not + * 128-bits. */ + __m128i v_src = _mm_maskz_loadu_epi32(0x7,(void *) key); + __m128i v_mask = _mm_maskz_loadu_epi32(0x7, (void *) mask); + + + /* These shuffle masks are used below, and each position tells where to + * move the bytes to. So here, the fourth sixth byte in + * ovs_key_ethernet is moved to byte location 0 in v_src/v_mask. + * The seventh is moved to 1, etc., etc. + * This swap is needed to move the src and dest MAC addresses in the + * same order as in the ethernet packet. */ + static const uint8_t eth_shuffle[16] = { + 6, 7, 8, 9, 10, 11, 0, 1, + 2, 3, 4, 5, 0xFF, 0xFF, 0xFF, 0xFF + }; + + /* Load the shuffle mask in v_shuf. */ + __m128i v_shuf = _mm_loadu_si128((void *) eth_shuffle); + + /* Swap the key/mask src and dest addresses to the ethernet order. */ + v_src = _mm_shuffle_epi8(v_src, v_shuf); + v_mask = _mm_shuffle_epi8(v_mask, v_shuf); + + DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { + + struct eth_header *eh = dp_packet_eth(packet); + + if (!eh) { + continue; + } + + /* Load the first 128-bits of the packet into the v_ether register. */ + __m128i v_dst = _mm_loadu_si128((void *) eh); + + /* AND the v_mask to the packet data (v_dst). */ + __m128i dst_masked = _mm_andnot_si128(v_mask, v_dst); + + /* OR the new addresses (v_src) with the masked packet addresses + * (dst_masked). */ + __m128i res = _mm_or_si128(v_src, dst_masked); + + /* Write back the modified ethernet addresses. */ + _mm_storeu_si128((void *) eh, res); + } +} + +static void +action_avx512_set_masked(struct dp_packet_batch *batch, const struct nlattr *a) +{ + const struct nlattr *mask = nl_attr_get(a); + enum ovs_key_attr attr_type = nl_attr_type(mask); + + if (attr_type <= OVS_KEY_ATTR_MAX && impl_set_masked_funcs[attr_type]) { + impl_set_masked_funcs[attr_type](batch, a); + } else { + odp_execute_scalar_action(batch, a); + } +} + int action_avx512_init(struct odp_execute_action_impl *self OVS_UNUSED) { @@ -214,6 +299,11 @@ action_avx512_init(struct odp_execute_action_impl *self OVS_UNUSED) * are identified by OVS_ACTION_ATTR_*. */ self->funcs[OVS_ACTION_ATTR_POP_VLAN] = action_avx512_pop_vlan; self->funcs[OVS_ACTION_ATTR_PUSH_VLAN] = action_avx512_push_vlan; + self->funcs[OVS_ACTION_ATTR_SET_MASKED] = action_avx512_set_masked; + + /* Set function pointers for the individual operations supported by the + * SET_MASKED action. */ + impl_set_masked_funcs[OVS_KEY_ATTR_ETHERNET] = action_avx512_eth_set_addrs; return 0; } diff --git a/lib/odp-execute-private.c b/lib/odp-execute-private.c index ad736523b..475ed55da 100644 --- a/lib/odp-execute-private.c +++ b/lib/odp-execute-private.c @@ -22,6 +22,7 @@ #include "cpu.h" #include "dpdk.h" #include "dp-packet.h" +#include "odp-execute.h" #include "odp-execute-private.h" #include "odp-netlink.h" #include "odp-util.h" @@ -242,6 +243,19 @@ action_autoval_generic(struct dp_packet_batch *batch, const struct nlattr *a) dp_packet_delete_batch(&original_batch, true); } +void +odp_execute_scalar_action(struct dp_packet_batch *batch, + const struct nlattr *action) +{ + enum ovs_action_attr type = nl_attr_type(action); + + if (action_impls[ACTION_IMPL_SCALAR].funcs[type] && + type <= OVS_ACTION_ATTR_MAX) { + + action_impls[ACTION_IMPL_SCALAR].funcs[type](batch, action); + } +} + int action_autoval_init(struct odp_execute_action_impl *self) { diff --git a/lib/odp-execute-private.h b/lib/odp-execute-private.h index b3a10cd82..3fdcdc95b 100644 --- a/lib/odp-execute-private.h +++ b/lib/odp-execute-private.h @@ -94,4 +94,7 @@ int action_avx512_init(struct odp_execute_action_impl *self); void odp_execute_action_get_info(struct ds *name); +void odp_execute_scalar_action(struct dp_packet_batch *batch, + const struct nlattr *action); + #endif /* ODP_EXTRACT_PRIVATE */ diff --git a/lib/odp-execute.c b/lib/odp-execute.c index 3f562eb34..fe5790c8a 100644 --- a/lib/odp-execute.c +++ b/lib/odp-execute.c @@ -562,8 +562,6 @@ odp_execute_set_action(struct dp_packet *packet, const struct nlattr *a) } } -#define get_mask(a, type) ((const type *)(const void *)(a + 1) + 1) - static void odp_execute_masked_set_action(struct dp_packet *packet, const struct nlattr *a) @@ -575,17 +573,17 @@ odp_execute_masked_set_action(struct dp_packet *packet, switch (type) { case OVS_KEY_ATTR_PRIORITY: md->skb_priority = nl_attr_get_u32(a) - | (md->skb_priority & ~*get_mask(a, uint32_t)); + | (md->skb_priority & ~*odp_get_key_mask(a, uint32_t)); break; case OVS_KEY_ATTR_SKB_MARK: md->pkt_mark = nl_attr_get_u32(a) - | (md->pkt_mark & ~*get_mask(a, uint32_t)); + | (md->pkt_mark & ~*odp_get_key_mask(a, uint32_t)); break; case OVS_KEY_ATTR_ETHERNET: odp_eth_set_addrs(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_ethernet)); + odp_get_key_mask(a, struct ovs_key_ethernet)); break; case OVS_KEY_ATTR_NSH: { @@ -595,27 +593,27 @@ odp_execute_masked_set_action(struct dp_packet *packet, case OVS_KEY_ATTR_IPV4: odp_set_ipv4(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_ipv4)); + odp_get_key_mask(a, struct ovs_key_ipv4)); break; case OVS_KEY_ATTR_IPV6: odp_set_ipv6(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_ipv6)); + odp_get_key_mask(a, struct ovs_key_ipv6)); break; case OVS_KEY_ATTR_TCP: odp_set_tcp(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_tcp)); + odp_get_key_mask(a, struct ovs_key_tcp)); break; case OVS_KEY_ATTR_UDP: odp_set_udp(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_udp)); + odp_get_key_mask(a, struct ovs_key_udp)); break; case OVS_KEY_ATTR_SCTP: odp_set_sctp(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_sctp)); + odp_get_key_mask(a, struct ovs_key_sctp)); break; case OVS_KEY_ATTR_MPLS: @@ -623,33 +621,33 @@ odp_execute_masked_set_action(struct dp_packet *packet, if (mh) { put_16aligned_be32(&mh->mpls_lse, nl_attr_get_be32(a) | (get_16aligned_be32(&mh->mpls_lse) - & ~*get_mask(a, ovs_be32))); + & ~*odp_get_key_mask(a, ovs_be32))); } break; case OVS_KEY_ATTR_ARP: set_arp(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_arp)); + odp_get_key_mask(a, struct ovs_key_arp)); break; case OVS_KEY_ATTR_ND: odp_set_nd(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_nd)); + odp_get_key_mask(a, struct ovs_key_nd)); break; case OVS_KEY_ATTR_ND_EXTENSIONS: odp_set_nd_ext(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_nd_extensions)); + odp_get_key_mask(a, struct ovs_key_nd_extensions)); break; case OVS_KEY_ATTR_DP_HASH: md->dp_hash = nl_attr_get_u32(a) - | (md->dp_hash & ~*get_mask(a, uint32_t)); + | (md->dp_hash & ~*odp_get_key_mask(a, uint32_t)); break; case OVS_KEY_ATTR_RECIRC_ID: md->recirc_id = nl_attr_get_u32(a) - | (md->recirc_id & ~*get_mask(a, uint32_t)); + | (md->recirc_id & ~*odp_get_key_mask(a, uint32_t)); break; case OVS_KEY_ATTR_TUNNEL: /* Masked data not supported for tunnel. */ @@ -857,6 +855,17 @@ action_push_vlan(struct dp_packet_batch *batch, const struct nlattr *a) } } +static void +action_set_masked(struct dp_packet_batch *batch, const struct nlattr *a) +{ + const struct nlattr *key = nl_attr_get(a); + struct dp_packet *packet; + + DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { + odp_execute_masked_set_action(packet, key); + } +} + /* Implementation of the scalar actions impl init function. Build up the * array of func ptrs here. */ @@ -867,6 +876,7 @@ odp_action_scalar_init(struct odp_execute_action_impl *self) * are identified by OVS_ACTION_ATTR_*. */ self->funcs[OVS_ACTION_ATTR_POP_VLAN] = action_pop_vlan; self->funcs[OVS_ACTION_ATTR_PUSH_VLAN] = action_push_vlan; + self->funcs[OVS_ACTION_ATTR_SET_MASKED] = action_set_masked; return 0; } @@ -1078,12 +1088,6 @@ odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, } break; - case OVS_ACTION_ATTR_SET_MASKED: - DP_PACKET_BATCH_FOR_EACH(i, packet, batch) { - odp_execute_masked_set_action(packet, nl_attr_get(a)); - } - break; - case OVS_ACTION_ATTR_SAMPLE: DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { odp_execute_sample(dp, packet, steal && last_action, a, @@ -1210,6 +1214,7 @@ odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, /* The following actions are handled by the scalar implementation. */ case OVS_ACTION_ATTR_POP_VLAN: case OVS_ACTION_ATTR_PUSH_VLAN: + case OVS_ACTION_ATTR_SET_MASKED: OVS_NOT_REACHED(); } diff --git a/lib/odp-execute.h b/lib/odp-execute.h index 0921ee924..2ba1ec5d2 100644 --- a/lib/odp-execute.h +++ b/lib/odp-execute.h @@ -46,4 +46,7 @@ void odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, const struct nlattr *actions, size_t actions_len, odp_execute_cb dp_execute_action); + +#define odp_get_key_mask(a, type) ((const type *)(const void *)(a + 1) + 1) + #endif From patchwork Tue Jul 12 17:44:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Van Haaren, Harry" X-Patchwork-Id: 1655551 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=JHrFpZj0; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lj7V434fMz9s07 for ; Wed, 13 Jul 2022 03:46:36 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 416BD42313; Tue, 12 Jul 2022 17:46:34 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 416BD42313 Authentication-Results: smtp4.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=JHrFpZj0 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CreQUheqtjRX; Tue, 12 Jul 2022 17:46:32 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTPS id F292B4231E; Tue, 12 Jul 2022 17:46:30 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org F292B4231E Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id B8973C0035; Tue, 12 Jul 2022 17:46:30 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 03FCAC002D for ; Tue, 12 Jul 2022 17:46:29 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id BAC7B4217B for ; Tue, 12 Jul 2022 17:46:01 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org BAC7B4217B X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id c3dUtTp2y8d6 for ; Tue, 12 Jul 2022 17:45:59 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 9A2DA41A5A Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by smtp4.osuosl.org (Postfix) with ESMTPS id 9A2DA41A5A for ; Tue, 12 Jul 2022 17:45:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1657647959; x=1689183959; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=NSe/91uVhPqVbrnCDjYHwC+ehYqyTTksG7pFReFtBXA=; b=JHrFpZj0dlF3WE0bVitai+B9coDCzxNe1T3h9+xd40pw3tGsjcViRpah 4xaWhfEiXsR5a7hcuIrke38IRtQK6AYityfl+UE9ZyuudagV3VLYD6aUv rYj8GETzAeJgkEOgyT2yISbr3MAwaanv+UxP+6BB8K7QLyMZETa6jsCyR XmC53BY4vL0GKZFnCAqUhxHrf8BxTLGxV7mtN+/jMc8UhkVHlxVmvzVzd hCq6rb6sjf+fSYeCiHxbTMJqw8IQ/NUKy7r14Q9AFi+4IpNM8X8BO3v6T IlxypmdXMg+qvJIYXf/1jzskl6sRKzSF0UeYiKXN15SbxEchvTCeXsbDW Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10406"; a="286130910" X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="286130910" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 12 Jul 2022 10:45:59 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,266,1650956400"; d="scan'208";a="627979841" Received: from silpixa00401454.ir.intel.com ([10.55.128.122]) by orsmga001.jf.intel.com with ESMTP; 12 Jul 2022 10:45:57 -0700 From: Harry van Haaren To: dev@openvswitch.org Date: Tue, 12 Jul 2022 17:44:56 +0000 Message-Id: <20220712174456.2682549-11-harry.van.haaren@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220712174456.2682549-1-harry.van.haaren@intel.com> References: <20220707153900.3147694-1-emma.finn@intel.com> <20220712174456.2682549-1-harry.van.haaren@intel.com> MIME-Version: 1.0 Cc: i.maximets@ovn.org, kumar.amber@intel.com Subject: [ovs-dev] [PATCH v9 10/10] odp-execute: Add ISA implementation of set_masked IPv4 action X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" From: Emma Finn This commit adds support for the AVX512 implementation of the ipv4_set_addrs action as well as an AVX512 implementation of updating the checksums. Signed-off-by: Emma Finn --- lib/odp-execute-avx512.c | 208 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 208 insertions(+) diff --git a/lib/odp-execute-avx512.c b/lib/odp-execute-avx512.c index 8ecdaecf6..a0c97f312 100644 --- a/lib/odp-execute-avx512.c +++ b/lib/odp-execute-avx512.c @@ -21,6 +21,7 @@ #include #include +#include "csum.h" #include "dp-packet.h" #include "immintrin.h" #include "odp-execute.h" @@ -58,6 +59,22 @@ BUILD_ASSERT_DECL(offsetof(struct ovs_key_ethernet, eth_src) + MEMBER_SIZEOF(struct ovs_key_ethernet, eth_src) == offsetof(struct ovs_key_ethernet, eth_dst)); +BUILD_ASSERT_DECL(offsetof(struct ovs_key_ipv4, ipv4_src) + + MEMBER_SIZEOF(struct ovs_key_ipv4, ipv4_src) == + offsetof(struct ovs_key_ipv4, ipv4_dst)); + +BUILD_ASSERT_DECL(offsetof(struct ovs_key_ipv4, ipv4_dst) + + MEMBER_SIZEOF(struct ovs_key_ipv4, ipv4_dst) == + offsetof(struct ovs_key_ipv4, ipv4_proto)); + +BUILD_ASSERT_DECL(offsetof(struct ovs_key_ipv4, ipv4_proto) + + MEMBER_SIZEOF(struct ovs_key_ipv4, ipv4_proto) == + offsetof(struct ovs_key_ipv4, ipv4_tos)); + +BUILD_ASSERT_DECL(offsetof(struct ovs_key_ipv4, ipv4_tos) + + MEMBER_SIZEOF(struct ovs_key_ipv4, ipv4_tos) == + offsetof(struct ovs_key_ipv4, ipv4_ttl)); + /* Array of callback functions, one for each masked operation. */ odp_execute_action_cb impl_set_masked_funcs[__OVS_KEY_ATTR_MAX]; @@ -279,6 +296,196 @@ action_avx512_eth_set_addrs(struct dp_packet_batch *batch, } } +static inline uint16_t ALWAYS_INLINE +avx512_get_delta(__m256i old_header, __m256i res) +{ + __m256i v_zeros = _mm256_setzero_si256(); + uint16_t delta; + + /* These two shuffle masks, v_swap16a and v_swap16b, are to shuffle the + * old and new header to add padding after each 16-bit value for the + * following carry over addition. */ + __m256i v_swap16a = _mm256_setr_epi16(0x0100, 0xFFFF, 0x0302, 0xFFFF, + 0x0504, 0xFFFF, 0x0706, 0xFFFF, + 0x0100, 0xFFFF, 0x0302, 0xFFFF, + 0xFFFF, 0xFFFF, 0xFFFF, 0xFFFF); + __m256i v_swap16b = _mm256_setr_epi16(0x0908, 0xFFFF, 0x0B0A, 0xFFFF, + 0x0D0C, 0xFFFF, 0x0F0E, 0xFFFF, + 0xFFFF, 0xFFFF, 0xFFFF, 0xFFFF, + 0xFFFF, 0xFFFF, 0xFFFF, 0xFFFF); + __m256i v_shuf_old1 = _mm256_shuffle_epi8(old_header, v_swap16a); + __m256i v_shuf_old2 = _mm256_shuffle_epi8(old_header, v_swap16b); + __m256i v_shuf_new1 = _mm256_shuffle_epi8(res, v_swap16a); + __m256i v_shuf_new2 = _mm256_shuffle_epi8(res, v_swap16b); + + /* Add each part of the old and new headers together. */ + __m256i v_delta1 = _mm256_add_epi32(v_shuf_old1, v_shuf_new1); + __m256i v_delta2 = _mm256_add_epi32(v_shuf_old2, v_shuf_new2); + + /* Add old and new header. */ + __m256i v_delta = _mm256_add_epi32(v_delta1, v_delta2); + + /* Perform horizontal add to go from 8x32-bits to 2x32-bits. */ + v_delta = _mm256_hadd_epi32(v_delta, v_zeros); + v_delta = _mm256_hadd_epi32(v_delta, v_zeros); + + /* Shuffle 32-bit value from 3rd lane into first lane for final + * horizontal add. */ + __m256i v_swap32a = _mm256_setr_epi32(0x0, 0x4, 0xF, 0xF, + 0xF, 0xF, 0xF, 0xF); + v_delta = _mm256_permutexvar_epi32(v_swap32a, v_delta); + + v_delta = _mm256_hadd_epi32(v_delta, v_zeros); + v_delta = _mm256_hadd_epi16(v_delta, v_zeros); + + /* Extract delta value. */ + delta = _mm256_extract_epi16(v_delta, 0); + + return delta; +} + +static inline uint16_t ALWAYS_INLINE +avx512_l4_update_csum(__m256i old_header, __m256i res) +{ + __m256i v_zeros = _mm256_setzero_si256(); + uint16_t delta; + + /* Set the v_ones register to all one's. */ + __m256i v_ones = _mm256_cmpeq_epi16(v_zeros, v_zeros); + + /* Combine the old and new header, i.e. adding in the new IP addresses + * in the old header (oh). This is done by using the 0x03C 16-bit mask, + * picking 16-bit word 7 till 10. */ + __m256i v_blend_new = _mm256_mask_blend_epi16(0x03C0, old_header, res); + + /* Invert the old_header register. */ + old_header =_mm256_andnot_si256(old_header, v_ones); + + /* Calculate the delta between the old and new header. */ + delta = avx512_get_delta(old_header, v_blend_new); + + return delta; + +} + +static inline uint16_t ALWAYS_INLINE +avx512_ipv4_update_csum(__m256i res, __m256i old_header) +{ + __m256i v_zeros = _mm256_setzero_si256(); + uint16_t delta; + + /* Set the v_ones register to all one's. */ + __m256i v_ones = _mm256_cmpeq_epi16(v_zeros, v_zeros); + + /* Invert the old_header register. */ + old_header =_mm256_andnot_si256(old_header, v_ones); + + /* Calculate the delta between the old and new header. */ + delta = avx512_get_delta(old_header, res); + + return delta; +} + +/* This function performs the same operation on each packet in the batch as + * the scalar odp_set_ipv4() function. */ +static void +action_avx512_ipv4_set_addrs(struct dp_packet_batch *batch, + const struct nlattr *a) +{ + const struct ovs_key_ipv4 *key, *mask; + struct dp_packet *packet; + a = nl_attr_get(a); + key = nl_attr_get(a); + mask = odp_get_key_mask(a, struct ovs_key_ipv4); + + /* Read the content of the key(src) and mask in the respective registers. + * We only load the size of the actual structure, which is only 96-bits. */ + __m256i v_key = _mm256_maskz_loadu_epi32(0x7, (void *) key); + __m256i v_mask = _mm256_maskz_loadu_epi32(0x7, (void *) mask); + + /* This two shuffle masks, v_shuf32, v_shuffle, are to shuffle key and + * mask to match the ip_header structure layout. */ + static const uint8_t ip_shuffle_mask[32] = { + 0xFF, 0x05, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0x06, 0xFF, 0xFF, 0xFF, 0x00, 0x01, 0x02, 0x03, + 0x00, 0x01, 0x02, 0x03, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF}; + + __m256i v_shuf32 = _mm256_setr_epi32(0x0, 0x2, 0xF, 0xF, + 0x1, 0xF, 0xF, 0xF); + + __m256i v_shuffle = _mm256_loadu_si256((void *) ip_shuffle_mask); + + /* Two shuffles are required for key and mask to match the layout of + * the ip_header struct. The _shuffle_epi8 only works within 128-bit + * lanes, so a permute is required to move src and dst into the correct + * lanes. And then a shuffle is used to move the fields into the right + * order. + */ + __m256i v_key_shuf = _mm256_permutexvar_epi32(v_shuf32, v_key); + v_key_shuf = _mm256_shuffle_epi8(v_key_shuf, v_shuffle); + + __m256i v_mask_shuf = _mm256_permutexvar_epi32(v_shuf32, v_mask); + v_mask_shuf = _mm256_shuffle_epi8(v_mask_shuf, v_shuffle); + + DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { + struct ip_header *nh = dp_packet_l3(packet); + ovs_be16 old_csum = ~nh->ip_csum; + + /* Load the 20 bytes of the IPv4 header. Without options, which is the + * most common case it's 20 bytes, but can be up to 60 bytes. */ + __m256i v_packet = _mm256_maskz_loadu_epi32(0x1F, (void *) nh); + + /* AND the v_pkt_mask to the packet data (v_packet). */ + __m256i v_pkt_masked = _mm256_andnot_si256(v_mask_shuf, v_packet); + + /* OR the new addresses (v_key_shuf) with the masked packet addresses + * (v_pkt_masked). */ + __m256i v_res = _mm256_or_si256(v_key_shuf, v_pkt_masked); + + /* Update the IP checksum based on updated IP values. */ + uint16_t delta = avx512_ipv4_update_csum(v_res, v_packet); + uint32_t new_csum = old_csum + delta; + delta = csum_finish(new_csum); + + /* Insert new checksum. */ + v_res = _mm256_insert_epi16(v_res, delta, 5); + + /* If ip_src or ip_dst has been modified, L4 checksum needs to + * be updated too. */ + if (mask->ipv4_src || mask->ipv4_dst) { + + uint16_t delta_checksum = avx512_l4_update_csum(v_packet, v_res); + + if (nh->ip_proto == IPPROTO_UDP) { + /* New UDP checksum. */ + struct udp_header *uh = dp_packet_l4(packet); + if (uh->udp_csum) { + uint16_t old_udp_checksum = ~uh->udp_csum; + uint32_t udp_checksum = old_udp_checksum + delta_checksum; + udp_checksum = csum_finish(udp_checksum); + + if (!udp_checksum) { + udp_checksum = htons(0xffff); + } + /* Insert new udp checksum. */ + uh->udp_csum = udp_checksum; + } + } else if (nh->ip_proto == IPPROTO_TCP) { + /* New TCP checksum. */ + struct tcp_header *th = dp_packet_l4(packet); + uint16_t old_tcp_checksum = ~th->tcp_csum; + uint32_t tcp_checksum = old_tcp_checksum + delta_checksum; + tcp_checksum = csum_finish(tcp_checksum); + + th->tcp_csum = tcp_checksum; + } + } + /* Write back the modified IPv4 addresses. */ + _mm256_mask_storeu_epi32((void *) nh, 0x1F, v_res); + } +} + static void action_avx512_set_masked(struct dp_packet_batch *batch, const struct nlattr *a) { @@ -304,6 +511,7 @@ action_avx512_init(struct odp_execute_action_impl *self OVS_UNUSED) /* Set function pointers for the individual operations supported by the * SET_MASKED action. */ impl_set_masked_funcs[OVS_KEY_ATTR_ETHERNET] = action_avx512_eth_set_addrs; + impl_set_masked_funcs[OVS_KEY_ATTR_IPV4] = action_avx512_ipv4_set_addrs; return 0; }