From patchwork Tue Jun 14 11:54:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Finn, Emma" X-Patchwork-Id: 1643250 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=mru8ticK; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LMn0c0sczz9sG2 for ; Tue, 14 Jun 2022 21:54:24 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 3F34682BF4; Tue, 14 Jun 2022 11:54:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yGaFlErVBQXQ; Tue, 14 Jun 2022 11:54:21 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTPS id 6A139818E6; Tue, 14 Jun 2022 11:54:20 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 318FDC0032; Tue, 14 Jun 2022 11:54:20 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id B980DC002D for ; Tue, 14 Jun 2022 11:54:18 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id B672081926 for ; Tue, 14 Jun 2022 11:54:18 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RtKMDn5y07Vo for ; Tue, 14 Jun 2022 11:54:18 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by smtp1.osuosl.org (Postfix) with ESMTPS id 017E1818E6 for ; Tue, 14 Jun 2022 11:54:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655207657; x=1686743657; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=lOhGyJwI2De5TQI6L5whqIN07FtFbQY7VGBWPgH7+ds=; b=mru8ticKeXI+39vpV/Ja5oryaPpNyi8ceW1kbCApEowxKybhiYncRUim ha0rn0MNKdVSIaU0WAOCbwh8qiJF8nie+xcadOsbRWv/vViV8nJq8Jm+Q 8BfJvTfghP3e4c4Tz97AW6qoXXKBDfMearSlszgujDinVe5zc+wzJ6PZ2 KOT7vU8V+ZuavYC1IX14iGhWfrB+zn1aIqttHiC4jIci7rv70I53Rwxhj fNHxlEXu1pJyZettzlUhnWnQrvAJ3hDIjhIgEhsTOv+dPIYg5lhmOsksZ RGA5GtfFyp4jlBxbfecXxPVQxefU/ApPBju+8KfjI165y1BKZW9sLghHu A==; X-IronPort-AV: E=McAfee;i="6400,9594,10377"; a="278634004" X-IronPort-AV: E=Sophos;i="5.91,300,1647327600"; d="scan'208";a="278634004" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jun 2022 04:54:17 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,300,1647327600"; d="scan'208";a="830382521" Received: from silpixa00401384.ir.intel.com ([10.243.23.50]) by fmsmga006.fm.intel.com with ESMTP; 14 Jun 2022 04:54:15 -0700 From: Emma Finn To: ian.stokes@intel.com, echaudro@redhat.com, harry.van.haaren@intel.com, dev@openvswitch.org Date: Tue, 14 Jun 2022 11:54:01 +0000 Message-Id: <20220614115409.1143269-1-emma.finn@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220510142202.1087967-1-emma.finn@intel.com> References: <20220510142202.1087967-1-emma.finn@intel.com> MIME-Version: 1.0 Subject: [ovs-dev] [PATCH v7 03/11] odp-execute: Add function pointer for pop_vlan action. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This commit removes the pop_vlan action from the large switch and creates a separate function for batched processing. A function pointer is also added to call the new batched function for the pop_vlan action. Signed-off-by: Emma Finn Acked-by: Harry van Haaren --- lib/odp-execute-private.c | 18 +++++++++++++++- lib/odp-execute-private.h | 8 +++++++ lib/odp-execute.c | 44 +++++++++++++++++++++++++++++++++------ lib/odp-execute.h | 2 ++ 4 files changed, 65 insertions(+), 7 deletions(-) diff --git a/lib/odp-execute-private.c b/lib/odp-execute-private.c index 92db2386c..25dbbfefc 100644 --- a/lib/odp-execute-private.c +++ b/lib/odp-execute-private.c @@ -27,12 +27,13 @@ #include "openvswitch/vlog.h" VLOG_DEFINE_THIS_MODULE(odp_execute_impl); +static int active_action_impl_index; static struct odp_execute_action_impl action_impls[] = { [ACTION_IMPL_SCALAR] = { .available = false, .name = "scalar", - .init_func = NULL, + .init_func = odp_action_scalar_init, }, }; @@ -45,6 +46,21 @@ action_impl_copy_funcs(struct odp_execute_action_impl *src, } } +int +odp_execute_action_set(const char *name, + struct odp_execute_action_impl *active) +{ + for (int i = 0; i < ACTION_IMPL_MAX; i++) { + /* String compare, and set ptrs atomically. */ + if (!strcmp(action_impls[i].name, name)) { + action_impl_copy_funcs(active, &action_impls[i]);; + active_action_impl_index = i; + return 0; + } + } + return -EINVAL; +} + void odp_execute_action_init(void) { diff --git a/lib/odp-execute-private.h b/lib/odp-execute-private.h index 08d1faf38..c5ab00b07 100644 --- a/lib/odp-execute-private.h +++ b/lib/odp-execute-private.h @@ -71,4 +71,12 @@ BUILD_ASSERT_DECL(ACTION_IMPL_SCALAR == 0); */ void odp_execute_action_init(void); +/* Init functions for the action implementations. Initializes the function + * pointers for optimized action types. + */ +int odp_action_scalar_init(struct odp_execute_action_impl *self); + +int odp_execute_action_set(const char *name, + struct odp_execute_action_impl *active); + #endif /* ODP_EXTRACT_PRIVATE */ diff --git a/lib/odp-execute.c b/lib/odp-execute.c index 67ebbe195..eff80d93f 100644 --- a/lib/odp-execute.c +++ b/lib/odp-execute.c @@ -834,6 +834,30 @@ requires_datapath_assistance(const struct nlattr *a) return false; } +static void +action_pop_vlan(struct dp_packet_batch *batch, + const struct nlattr *a OVS_UNUSED) +{ + struct dp_packet *packet; + + DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { + eth_pop_vlan(packet); + } +} + +/* Implementation of the scalar actions impl init function. Build up the + * array of func ptrs here. + */ +int +odp_action_scalar_init(struct odp_execute_action_impl *self) +{ + /* Set function pointers for actions that can be applied directly, these + * are identified by OVS_ACTION_ATTR_*. */ + self->funcs[OVS_ACTION_ATTR_POP_VLAN] = action_pop_vlan; + + return 0; +} + /* The active function pointers on the datapath. ISA optimized implementations * are enabled by plugging them into this static arary, which is consulted when * applying actions on the datapath. @@ -846,10 +870,22 @@ odp_execute_init(void) static struct ovsthread_once once = OVSTHREAD_ONCE_INITIALIZER; if (ovsthread_once_start(&once)) { odp_execute_action_init(); + odp_actions_impl_set("scalar"); ovsthread_once_done(&once); } } +int +odp_actions_impl_set(const char *name) +{ + + int err = odp_execute_action_set(name, &actions_active_impl); + if (err) { + VLOG_ERR("Failed setting action implementation to %s, error %d", + name, err); + } + return err; +} /* Executes all of the 'actions_len' bytes of datapath actions in 'actions' on * the packets in 'batch'. If 'steal' is true, possibly modifies and @@ -964,12 +1000,6 @@ odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, break; } - case OVS_ACTION_ATTR_POP_VLAN: - DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { - eth_pop_vlan(packet); - } - break; - case OVS_ACTION_ATTR_PUSH_MPLS: { const struct ovs_action_push_mpls *mpls = nl_attr_get(a); @@ -1120,6 +1150,8 @@ odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, case OVS_ACTION_ATTR_CT: case OVS_ACTION_ATTR_UNSPEC: case __OVS_ACTION_ATTR_MAX: + /* The following actions are handled by the scalar implementation. */ + case OVS_ACTION_ATTR_POP_VLAN: OVS_NOT_REACHED(); } diff --git a/lib/odp-execute.h b/lib/odp-execute.h index 0921ee924..50d47b716 100644 --- a/lib/odp-execute.h +++ b/lib/odp-execute.h @@ -35,6 +35,8 @@ void odp_execute_init(void); typedef void (*odp_execute_action_cb)(struct dp_packet_batch *batch, const struct nlattr *action); +int odp_actions_impl_set(const char *name); + typedef void (*odp_execute_cb)(void *dp, struct dp_packet_batch *batch, const struct nlattr *action, bool should_steal); From patchwork Tue Jun 14 11:54:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Finn, Emma" X-Patchwork-Id: 1643251 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=W8+o2HHa; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LMn0n0m6Yz9sG2 for ; Tue, 14 Jun 2022 21:54:33 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id D417E60F45; Tue, 14 Jun 2022 11:54:30 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HrbC-5gnDGf0; Tue, 14 Jun 2022 11:54:28 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id 614B360B18; Tue, 14 Jun 2022 11:54:27 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2E897C0082; Tue, 14 Jun 2022 11:54:27 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 0BEA1C002D for ; Tue, 14 Jun 2022 11:54:26 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 55F3760A6C for ; Tue, 14 Jun 2022 11:54:25 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id suRAJLj8AIo7 for ; Tue, 14 Jun 2022 11:54:24 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by smtp3.osuosl.org (Postfix) with ESMTPS id 718F360F36 for ; Tue, 14 Jun 2022 11:54:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655207664; x=1686743664; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=YtYf1XhgAyvAxwMYQTjCJHcyBUnPqNdpkPRKTh+QNpc=; b=W8+o2HHaFVT8et9sIeEtZFY/+5msA9koYHm8Pivk0d16HCnsqpBcIC1M R+uvzh/nk3w5RwFZBykfrRX36Xj1/k52GWT7AQcWgdG5hBaRFGp7chNlL sAD6rScpXRaTO2e9YGZjVxFXyNgH2HbZ3FuC/7d1lt8xOHpfCeAcBqg5a MGuLNHYLXcAYL0IT77OS+ZI6ZpqDpKrqW1PebIvY8lDDT3Q2+G2n7E7BT T07i020NgI4wpxZxHOIWOzYcKV2Dhw+yQAo2itnZnCiBAQAMZpBZByjEr kt3wAwyqhfllQuMRHbLET4icj6MCucSAYR6+SsjLJTr+exb1RceAM695A g==; X-IronPort-AV: E=McAfee;i="6400,9594,10377"; a="342557563" X-IronPort-AV: E=Sophos;i="5.91,300,1647327600"; d="scan'208";a="342557563" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jun 2022 04:54:23 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,300,1647327600"; d="scan'208";a="830382533" Received: from silpixa00401384.ir.intel.com ([10.243.23.50]) by fmsmga006.fm.intel.com with ESMTP; 14 Jun 2022 04:54:21 -0700 From: Emma Finn To: ian.stokes@intel.com, echaudro@redhat.com, harry.van.haaren@intel.com, dev@openvswitch.org Date: Tue, 14 Jun 2022 11:54:02 +0000 Message-Id: <20220614115409.1143269-2-emma.finn@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220614115409.1143269-1-emma.finn@intel.com> References: <20220510142202.1087967-1-emma.finn@intel.com> <20220614115409.1143269-1-emma.finn@intel.com> MIME-Version: 1.0 Subject: [ovs-dev] [PATCH v7 04/11] odp-execute: Add auto validation function for actions. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This commit introduced the auto-validation function which allows users to compare the batch of packets obtained from different action implementations against the linear action implementation. The autovalidator function can be triggered at runtime using the following command: $ ovs-appctl dpif-netdev/action-impl-set autovalidator Signed-off-by: Emma Finn Acked-by: Harry van Haaren --- NEWS | 3 ++ lib/dp-packet.c | 23 +++++++++ lib/dp-packet.h | 4 ++ lib/odp-execute-private.c | 98 +++++++++++++++++++++++++++++++++++++++ lib/odp-execute-private.h | 6 +++ 5 files changed, 134 insertions(+) diff --git a/NEWS b/NEWS index 9fe3f44f4..3a25f3035 100644 --- a/NEWS +++ b/NEWS @@ -32,6 +32,9 @@ Post-v2.17.0 - DPDK: * OVS validated with DPDK 21.11.1. It is recommended to use this version until further releases. + - Userspace datapath: + * Add actions auto-validator function to compare different actions + implementations against default implementation. v2.17.0 - 17 Feb 2022 diff --git a/lib/dp-packet.c b/lib/dp-packet.c index 35c72542a..237dcf19e 100644 --- a/lib/dp-packet.c +++ b/lib/dp-packet.c @@ -506,3 +506,26 @@ dp_packet_resize_l2(struct dp_packet *b, int increment) dp_packet_adjust_layer_offset(&b->l2_5_ofs, increment); return dp_packet_data(b); } + +bool +dp_packet_compare_offsets(struct dp_packet *b1, struct dp_packet *b2, + struct ds *err_str) +{ + if ((b1->l2_pad_size != b2->l2_pad_size) || + (b1->l2_5_ofs != b2->l2_5_ofs) || + (b1->l3_ofs != b2->l3_ofs) || + (b1->l4_ofs != b2->l4_ofs)) { + ds_put_format(err_str, "Packet offset comparison failed" + "\n"); + ds_put_format(err_str, "Buffer 1 offsets: l2_pad_size %u," + " l2_5_ofs : %u l3_ofs %u, l4_ofs %u\n", + b1->l2_pad_size, b1->l2_5_ofs, + b1->l3_ofs, b1->l4_ofs); + ds_put_format(err_str, "Buffer 2 offsets: l2_pad_size %u," + " l2_5_ofs : %u l3_ofs %u, l4_ofs %u\n", + b2->l2_pad_size, b2->l2_5_ofs, + b2->l3_ofs, b2->l4_ofs); + return false; + } + return true; +} diff --git a/lib/dp-packet.h b/lib/dp-packet.h index bddaa2b5d..1776c3bfe 100644 --- a/lib/dp-packet.h +++ b/lib/dp-packet.h @@ -236,6 +236,10 @@ void *dp_packet_steal_data(struct dp_packet *); static inline bool dp_packet_equal(const struct dp_packet *, const struct dp_packet *); +bool dp_packet_compare_offsets(struct dp_packet *good, + struct dp_packet *test, + struct ds *err_str); + /* Frees memory that 'b' points to, as well as 'b' itself. */ static inline void diff --git a/lib/odp-execute-private.c b/lib/odp-execute-private.c index 25dbbfefc..267f32c3e 100644 --- a/lib/odp-execute-private.c +++ b/lib/odp-execute-private.c @@ -28,8 +28,15 @@ VLOG_DEFINE_THIS_MODULE(odp_execute_impl); static int active_action_impl_index; +static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1); static struct odp_execute_action_impl action_impls[] = { + [ACTION_IMPL_AUTOVALIDATOR] = { + .available = false, + .name = "autovalidator", + .init_func = action_autoval_init, + }, + [ACTION_IMPL_SCALAR] = { .available = false, .name = "scalar", @@ -94,3 +101,94 @@ odp_execute_action_init(void) action_impls[i].name, avail ? "available" : "not available"); } } + +/* Init sequence required to be scalar first to pick up the default scalar +* implementations, allowing over-riding of the optimized functions later. +*/ +BUILD_ASSERT_DECL(ACTION_IMPL_SCALAR == 0); +BUILD_ASSERT_DECL(ACTION_IMPL_AUTOVALIDATOR == 1); + +/* Loop over packets, and validate each one for the given action. */ +static void +action_autoval_generic(struct dp_packet_batch *batch, const struct nlattr *a) +{ + bool failed = false; + int type = nl_attr_type(a); + enum ovs_action_attr attr_type = (enum ovs_action_attr) type; + struct odp_execute_action_impl *scalar = &action_impls[ACTION_IMPL_SCALAR]; + struct dp_packet_batch good_batch; + + dp_packet_batch_clone(&good_batch, batch); + + scalar->funcs[attr_type](&good_batch, a); + + for (int impl = ACTION_IMPL_BEGIN; impl < ACTION_IMPL_MAX; impl++) { + /* Clone original batch and execute implementation under test. */ + struct dp_packet_batch test_batch; + + dp_packet_batch_clone(&test_batch, batch); + action_impls[impl].funcs[attr_type](&test_batch, a); + + /* Loop over implementations, checking each one. */ + for (int pidx = 0; pidx < batch->count; pidx++) { + struct dp_packet *good_pkt = good_batch.packets[pidx]; + struct dp_packet *test_pkt = test_batch.packets[pidx]; + + struct ds log_msg = DS_EMPTY_INITIALIZER; + + /* Compare packet length and payload contents. */ + bool eq = dp_packet_equal(good_pkt, test_pkt); + + if (!eq) { + ds_put_format(&log_msg, "Packet: %d\nAction : ", pidx); + format_odp_actions(&log_msg, a, a->nla_len, NULL); + ds_put_format(&log_msg, "\nGood hex:\n"); + ds_put_hex_dump(&log_msg, dp_packet_data(good_pkt), + dp_packet_size(good_pkt), 0, false); + ds_put_format(&log_msg, "Test hex:\n"); + ds_put_hex_dump(&log_msg, dp_packet_data(test_pkt), + dp_packet_size(test_pkt), 0, false); + + failed = true; + } + + /* Compare offsets and RSS */ + if (!dp_packet_compare_offsets(good_pkt, test_pkt, &log_msg)) { + failed = true; + } + + uint32_t good_hash = dp_packet_get_rss_hash(good_pkt); + uint32_t test_hash = dp_packet_get_rss_hash(test_pkt); + + if (good_hash != test_hash) { + ds_put_format(&log_msg, "Autovalidation rss hash failed\n"); + ds_put_format(&log_msg, "Good RSS hash : %u\n", good_hash); + ds_put_format(&log_msg, "Test RSS hash : %u\n", test_hash); + + failed = true; + } + + if (failed) { + VLOG_ERR_RL(&rl, "Autovalidation of %s failed. Details:\n%s", + action_impls[impl].name, ds_cstr(&log_msg)); + ds_destroy(&log_msg); + failed = false; + } + } + dp_packet_delete_batch(&test_batch, 1); + } + dp_packet_delete_batch(&good_batch, 1); + + /* Apply the action to the original batch for continued processing. */ + scalar->funcs[attr_type](batch, a); +} + +int +action_autoval_init(struct odp_execute_action_impl *self) +{ + /* Set function pointers for actions that can be applied directly, these + * are identified by OVS_ACTION_ATTR_*. */ + self->funcs[OVS_ACTION_ATTR_POP_VLAN] = action_autoval_generic; + + return 0; +} diff --git a/lib/odp-execute-private.h b/lib/odp-execute-private.h index c5ab00b07..d3dc669d1 100644 --- a/lib/odp-execute-private.h +++ b/lib/odp-execute-private.h @@ -54,6 +54,7 @@ struct odp_execute_action_impl { /* Order of Actions implementations. */ enum odp_execute_action_impl_idx { ACTION_IMPL_SCALAR, + ACTION_IMPL_AUTOVALIDATOR, /* See ACTION_IMPL_BEGIN below, for "first to-be-validated" impl. * Do not change the autovalidator position in this list without updating * the define below. @@ -64,6 +65,9 @@ enum odp_execute_action_impl_idx { /* Index to start verifying implementations from. */ BUILD_ASSERT_DECL(ACTION_IMPL_SCALAR == 0); +BUILD_ASSERT_DECL(ACTION_IMPL_AUTOVALIDATOR == 1); + +#define ACTION_IMPL_BEGIN (ACTION_IMPL_AUTOVALIDATOR + 1) /* Odp execute init handles setting up the state of the actions functions at * initialization time. It cannot return errors, as it must always succeed in @@ -76,6 +80,8 @@ void odp_execute_action_init(void); */ int odp_action_scalar_init(struct odp_execute_action_impl *self); +int action_autoval_init(struct odp_execute_action_impl *self); + int odp_execute_action_set(const char *name, struct odp_execute_action_impl *active); From patchwork Tue Jun 14 11:54:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Finn, Emma" X-Patchwork-Id: 1643252 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=R2I8HG/B; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LMn0v1XcWz9sG2 for ; Tue, 14 Jun 2022 21:54:39 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 4F6ED60F6F; Tue, 14 Jun 2022 11:54:37 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id DX93mDe1kPGd; Tue, 14 Jun 2022 11:54:36 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 567B660F77; Tue, 14 Jun 2022 11:54:35 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2BCB2C007A; Tue, 14 Jun 2022 11:54:35 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 214BEC0032 for ; Tue, 14 Jun 2022 11:54:34 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 72E0F60C15 for ; Tue, 14 Jun 2022 11:54:29 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zivGHMrf-nh2 for ; Tue, 14 Jun 2022 11:54:28 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by smtp3.osuosl.org (Postfix) with ESMTPS id 7674A60F44 for ; Tue, 14 Jun 2022 11:54:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655207668; x=1686743668; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=NiWcoNbfv8TWchEUNWZXLKkSlqMGZOfHLucm40RbNHg=; b=R2I8HG/BwN8NNkw6Y9lJFoSdCjQW7hrSJ1KWAul42OCY4IEHJhS2WSfX BLhYnD8k4x5C/2UBET9nf/XIxvBLDokpIPhxxasIinSp8HW2kGbnNN+Mn gxsYKWdTF8heZcpRScg2ZVFScuGBwApI2YVzgBRUL0V+5njXx8NskYlbz AqyfQ+9wP3owjR0s1TTLSE39bniAl16L6K6VShFqMpXyjri09WPOEhqT3 bFysqp8wGxMA6XTbLXtUwLsKhYXAbTOwxsaGwCUMhXAQJ1h7S439b/HAJ 94X1S7tMqKMMpSFvvSri57lKAZH6UZQ7l715kUp3xWE8ewq5kJKzbWPTC g==; X-IronPort-AV: E=McAfee;i="6400,9594,10377"; a="342557571" X-IronPort-AV: E=Sophos;i="5.91,300,1647327600"; d="scan'208";a="342557571" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jun 2022 04:54:28 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,300,1647327600"; d="scan'208";a="830382550" Received: from silpixa00401384.ir.intel.com ([10.243.23.50]) by fmsmga006.fm.intel.com with ESMTP; 14 Jun 2022 04:54:26 -0700 From: Emma Finn To: ian.stokes@intel.com, echaudro@redhat.com, harry.van.haaren@intel.com, dev@openvswitch.org Date: Tue, 14 Jun 2022 11:54:03 +0000 Message-Id: <20220614115409.1143269-3-emma.finn@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220614115409.1143269-1-emma.finn@intel.com> References: <20220510142202.1087967-1-emma.finn@intel.com> <20220614115409.1143269-1-emma.finn@intel.com> MIME-Version: 1.0 Cc: Kumar Amber Subject: [ovs-dev] [PATCH v7 05/11] odp-execute: Add command to switch action implementation. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This commit adds a new command to allow the user to switch the active action implementation at runtime. A probe function is executed before switching the implementation, to ensure the CPU is capable of running the ISA required. Usage: $ ovs-appctl dpif-netdev/action-impl-set scalar This commit also adds a new command to retrieve the list of available action implementations. This can be used by to check what implementations of actions are available and what implementation is active during runtime. Usage: $ ovs-appctl dpif-netdev/action-impl-show Added separate test-case for ovs-actions show/set commands: 1023: PMD - ovs-actions configuration Signed-off-by: Emma Finn Co-authored-by: Kumar Amber Signed-off-by: Kumar Amber Acked-by: Harry van Haaren --- NEWS | 3 +++ lib/dpif-netdev-unixctl.man | 8 ++++++++ lib/dpif-netdev.c | 38 +++++++++++++++++++++++++++++++++++++ lib/odp-execute-private.c | 12 ++++++++++++ lib/odp-execute-private.h | 3 +++ lib/odp-execute.h | 2 ++ tests/pmd.at | 30 +++++++++++++++++++++++++++++ 7 files changed, 96 insertions(+) diff --git a/NEWS b/NEWS index 3a25f3035..90ceabd63 100644 --- a/NEWS +++ b/NEWS @@ -35,6 +35,9 @@ Post-v2.17.0 - Userspace datapath: * Add actions auto-validator function to compare different actions implementations against default implementation. + * Add command line option to switch between different actions + implementations available at run time. + v2.17.0 - 17 Feb 2022 diff --git a/lib/dpif-netdev-unixctl.man b/lib/dpif-netdev-unixctl.man index 8cd847416..81ef7d856 100644 --- a/lib/dpif-netdev-unixctl.man +++ b/lib/dpif-netdev-unixctl.man @@ -262,3 +262,11 @@ PMDs in the case where no value is specified. By default "scalar" is used. \fIstudy_cnt\fR defaults to 128 and indicates the number of packets that the "study" miniflow implementation must parse before choosing an optimal implementation. + +.IP "\fBdpif-netdev/action-impl-show\fR +Lists the actions implementations that are available and highlights the +currently enabled one. +. +.IP "\fBdpif-netdev/action-impl-set\fR \fIaction_impl\fR" +Sets the action implementation to any available implementation. By default +"scalar" is used. diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index 47dd7a1a6..5a35c7ce5 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -63,6 +63,7 @@ #include "netdev-vport.h" #include "netlink.h" #include "odp-execute.h" +#include "odp-execute-private.h" #include "odp-util.h" #include "openvswitch/dynamic-string.h" #include "openvswitch/list.h" @@ -1387,6 +1388,37 @@ error: ds_destroy(&reply); } +static void +action_impl_set(struct unixctl_conn *conn, int argc OVS_UNUSED, + const char *argv[] OVS_UNUSED, void *aux OVS_UNUSED) +{ + struct ds reply = DS_EMPTY_INITIALIZER; + + int err = odp_actions_impl_set(argv[1]); + if (err) { + ds_put_format(&reply, + "Error: unknown action implementation, %s, specified!\n", + argv[1]); + unixctl_command_reply_error(conn, ds_cstr(&reply)); + } else { + ds_put_format(&reply, "Action implementation set to %s.\n", argv[1]); + unixctl_command_reply(conn, ds_cstr(&reply)); + } + + ds_destroy(&reply); +} + +static void +action_impl_show(struct unixctl_conn *conn, int argc OVS_UNUSED, + const char *argv[] OVS_UNUSED, void *aux OVS_UNUSED) +{ + struct ds reply = DS_EMPTY_INITIALIZER; + + odp_execute_action_get_info(&reply); + unixctl_command_reply(conn, ds_cstr(&reply)); + ds_destroy(&reply); +} + static void dpif_netdev_pmd_rebalance(struct unixctl_conn *conn, int argc, const char *argv[], void *aux OVS_UNUSED) @@ -1624,6 +1656,12 @@ dpif_netdev_init(void) unixctl_command_register("dpif-netdev/miniflow-parser-get", "", 0, 0, dpif_miniflow_extract_impl_get, NULL); + unixctl_command_register("dpif-netdev/action-impl-set", "name", + 1, 1, action_impl_set, + NULL); + unixctl_command_register("dpif-netdev/action-impl-show", "", + 0, 0, action_impl_show, + NULL); return 0; } diff --git a/lib/odp-execute-private.c b/lib/odp-execute-private.c index 267f32c3e..f8d0896b5 100644 --- a/lib/odp-execute-private.c +++ b/lib/odp-execute-private.c @@ -68,6 +68,18 @@ odp_execute_action_set(const char *name, return -EINVAL; } +void +odp_execute_action_get_info(struct ds *string) +{ + ds_put_cstr(string, "Available Actions implementations:\n"); + for (int i = 0; i < ACTION_IMPL_MAX; i++) { + ds_put_format(string, " %s (available: %s, active: %s)\n", + action_impls[i].name, + action_impls[i].available ? "Yes" : "No", + i == active_action_impl_index ? "Yes" : "No"); + } +} + void odp_execute_action_init(void) { diff --git a/lib/odp-execute-private.h b/lib/odp-execute-private.h index d3dc669d1..5322eb8df 100644 --- a/lib/odp-execute-private.h +++ b/lib/odp-execute-private.h @@ -85,4 +85,7 @@ int action_autoval_init(struct odp_execute_action_impl *self); int odp_execute_action_set(const char *name, struct odp_execute_action_impl *active); +void odp_execute_action_get_info(struct ds *name); + + #endif /* ODP_EXTRACT_PRIVATE */ diff --git a/lib/odp-execute.h b/lib/odp-execute.h index 50d47b716..8668ab73f 100644 --- a/lib/odp-execute.h +++ b/lib/odp-execute.h @@ -23,6 +23,7 @@ #include #include "openvswitch/types.h" +struct ds; struct nlattr; struct dp_packet; struct pkt_metadata; @@ -36,6 +37,7 @@ typedef void (*odp_execute_action_cb)(struct dp_packet_batch *batch, const struct nlattr *action); int odp_actions_impl_set(const char *name); +int odp_actions_impl_get(struct ds *name); typedef void (*odp_execute_cb)(void *dp, struct dp_packet_batch *batch, const struct nlattr *action, bool should_steal); diff --git a/tests/pmd.at b/tests/pmd.at index e6b173dab..ac05f5f7d 100644 --- a/tests/pmd.at +++ b/tests/pmd.at @@ -1200,3 +1200,33 @@ ovs-appctl: ovs-vswitchd: server returned an error OVS_VSWITCHD_STOP AT_CLEANUP + +AT_SETUP([PMD - ovs-actions configuration]) +OVS_VSWITCHD_START([], [], [], [--dummy-numa 0,0]) +AT_CHECK([ovs-vsctl add-port br0 p1 -- set Interface p1 type=dummy-pmd]) + +dnl Scalar impl is set by default. +AT_CHECK([ovs-vsctl show], [], [stdout]) +AT_CHECK([ovs-appctl dpif-netdev/action-impl-show | grep "scalar"], [], [dnl + scalar (available: Yes, active: Yes) +]) + +AT_CHECK([ovs-appctl dpif-netdev/action-impl-show | grep "autovalidator"], [], [dnl + autovalidator (available: Yes, active: No) +]) + +dnl Set the autovalidator impl to active. +AT_CHECK([ovs-appctl dpif-netdev/action-impl-set autovalidator], [0], [dnl +Action implementation set to autovalidator. +]) + +AT_CHECK([ovs-appctl dpif-netdev/action-impl-show | grep "scalar"], [], [dnl + scalar (available: Yes, active: No) +]) + +AT_CHECK([ovs-appctl dpif-netdev/action-impl-show | grep "autovalidator"], [], [dnl + autovalidator (available: Yes, active: Yes) +]) + +OVS_VSWITCHD_STOP +AT_CLEANUP From patchwork Tue Jun 14 11:54:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Finn, Emma" X-Patchwork-Id: 1643253 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=NGqwRiVD; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LMn0z34pRz9sG2 for ; Tue, 14 Jun 2022 21:54:43 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 36BD960F97; Tue, 14 Jun 2022 11:54:40 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wKCI2qmxTeA3; Tue, 14 Jun 2022 11:54:39 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 1CCFB60F94; Tue, 14 Jun 2022 11:54:38 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id F188EC007A; Tue, 14 Jun 2022 11:54:37 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 32DF4C0032 for ; Tue, 14 Jun 2022 11:54:37 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 8E71C40ADB for ; Tue, 14 Jun 2022 11:54:35 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=intel.com Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id idgCqsDxEQFi for ; Tue, 14 Jun 2022 11:54:35 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by smtp2.osuosl.org (Postfix) with ESMTPS id F1A4640AE2 for ; Tue, 14 Jun 2022 11:54:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655207675; x=1686743675; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=DnEln5PjcCVGRwt5ign+3sz/luOIjjwBQUsfyZOy+Ic=; b=NGqwRiVDmzn0vcoMhhjvCRwof8fBMO2kIhMNdqjdLM7WvQY5Zr8iztab lFHZ5uds8CHb6hiZpu7w7KYiDvK28pPDtX76+BbTuncND/qMqj5MTJuVp B9ozYYte1syh5VCFltiPpHxzfXE+h3QCqLy3KTU832VYZI5dOphCjZhfL ymn1Jgt8GCMLhGRODxiSfDHrniOzPPtJmuIzMxft07MCo+z4lGMxSb9wv Z1w+UtGjjXw9SBN8vm5rGHmfaw97WM9RqLZQJa6cQ/dUyOk8ZQ6u0gZso 1viXaAwayjPciEPV5QnlAkMmSdkMIzcT4t/Se2fKB7m+vVq55oGl3vJHV g==; X-IronPort-AV: E=McAfee;i="6400,9594,10377"; a="277378298" X-IronPort-AV: E=Sophos;i="5.91,300,1647327600"; d="scan'208";a="277378298" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jun 2022 04:54:34 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,300,1647327600"; d="scan'208";a="830382588" Received: from silpixa00401384.ir.intel.com ([10.243.23.50]) by fmsmga006.fm.intel.com with ESMTP; 14 Jun 2022 04:54:32 -0700 From: Emma Finn To: ian.stokes@intel.com, echaudro@redhat.com, harry.van.haaren@intel.com, dev@openvswitch.org Date: Tue, 14 Jun 2022 11:54:04 +0000 Message-Id: <20220614115409.1143269-4-emma.finn@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220614115409.1143269-1-emma.finn@intel.com> References: <20220510142202.1087967-1-emma.finn@intel.com> <20220614115409.1143269-1-emma.finn@intel.com> MIME-Version: 1.0 Cc: Kumar Amber Subject: [ovs-dev] [PATCH v7 06/11] dpif-netdev: Add configure option to enable actions autovalidator at build time. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" From: Kumar Amber This commit adds a new command to allow the user to enable the actions autovalidator by default at build time thus allowing for running unit test by default. $ ./configure --enable-actions-default-autovalidator Signed-off-by: Kumar Amber Acked-by: Harry van Haaren --- NEWS | 3 +++ acinclude.m4 | 21 +++++++++++++++++++++ configure.ac | 1 + lib/odp-execute.c | 4 ++++ 4 files changed, 29 insertions(+) diff --git a/NEWS b/NEWS index 90ceabd63..7ae4d87bb 100644 --- a/NEWS +++ b/NEWS @@ -37,6 +37,9 @@ Post-v2.17.0 implementations against default implementation. * Add command line option to switch between different actions implementations available at run time. + * Add build time configure command to enable auto-validator as default + actions implementation at build time. + diff --git a/acinclude.m4 b/acinclude.m4 index 7b2889a40..98f4599b1 100644 --- a/acinclude.m4 +++ b/acinclude.m4 @@ -14,6 +14,27 @@ # See the License for the specific language governing permissions and # limitations under the License. +dnl Set OVS Actions Autovalidator as the default action implementation +at compile time? +dnl This enables automatically running all unit tests with all actions +dnl implementations. +AC_DEFUN([OVS_CHECK_ACTIONS_AUTOVALIDATOR], [ + AC_ARG_ENABLE([actions-default-autovalidator], + [AC_HELP_STRING([--enable-actions-default-autovalidator], + [Enable actions autovalidator as default + ovs actions implementation.])], + [autovalidator=yes],[autovalidator=no]) + AC_MSG_CHECKING([whether actions Autovalidator is default implementation]) + if test "$autovalidator" != yes; then + AC_MSG_RESULT([no]) + else + AC_DEFINE([MFEX_AUTOVALIDATOR_DEFAULT], [1], + [Autovalidator for actions is a default implementation.]) + AC_MSG_RESULT([yes]) + fi +]) + + dnl Set OVS MFEX Autovalidator as default miniflow extract at compile time? dnl This enables automatically running all unit tests with all MFEX dnl implementations. diff --git a/configure.ac b/configure.ac index 59ea0a281..ab8e1bd12 100644 --- a/configure.ac +++ b/configure.ac @@ -184,6 +184,7 @@ OVS_CONDITIONAL_CC_OPTION([-Wno-unused-parameter], [HAVE_WNO_UNUSED_PARAMETER]) OVS_ENABLE_WERROR OVS_ENABLE_SPARSE OVS_CTAGS_IDENTIFIERS +OVS_CHECK_ACTIONS_AUTOVALIDATOR OVS_CHECK_DPCLS_AUTOVALIDATOR OVS_CHECK_DPIF_AVX512_DEFAULT OVS_CHECK_MFEX_AUTOVALIDATOR diff --git a/lib/odp-execute.c b/lib/odp-execute.c index eff80d93f..a49b331ef 100644 --- a/lib/odp-execute.c +++ b/lib/odp-execute.c @@ -870,7 +870,11 @@ odp_execute_init(void) static struct ovsthread_once once = OVSTHREAD_ONCE_INITIALIZER; if (ovsthread_once_start(&once)) { odp_execute_action_init(); +#ifdef ACTIONS_AUTOVALIDATOR_DEFAULT + odp_actions_impl_set("autovalidator"); +#else odp_actions_impl_set("scalar"); +#endif ovsthread_once_done(&once); } } From patchwork Tue Jun 14 11:54:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Finn, Emma" X-Patchwork-Id: 1643254 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=j8PJTT8d; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LMn140byvz9sG2 for ; Tue, 14 Jun 2022 21:54:48 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 97B2D40B35; Tue, 14 Jun 2022 11:54:46 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id xGmsGZeYeGug; Tue, 14 Jun 2022 11:54:45 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id 1D29740B41; Tue, 14 Jun 2022 11:54:44 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id E9897C0032; Tue, 14 Jun 2022 11:54:43 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 0258AC002D for ; Tue, 14 Jun 2022 11:54:43 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id C94FB40B15 for ; Tue, 14 Jun 2022 11:54:39 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bqa62dGoiDh9 for ; Tue, 14 Jun 2022 11:54:38 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by smtp2.osuosl.org (Postfix) with ESMTPS id F1C1140ADB for ; Tue, 14 Jun 2022 11:54:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655207678; x=1686743678; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=MXJRlN+y4FmeFiZE0gVnYwejuuAiyMSov9pb9/tI9ws=; b=j8PJTT8d2uJ78jRCNk1fSfgOsH5J0grhbSUPg46c2fOc9UmXu9QeRl5O u6YxERlbE8fuSAK8B/CpoCt6aZym+SnH+7S4WGO4Sv+TlngL8wK5isERJ UGqYaHr7hkQY7Pt5shRjOdXdo8Hf78sPSFiq6X3XHc2GMhYBzlQmG0NA2 vh37Kl9Bf2DYbVDOcMvZ2p0bW6wYw6BGe1A9haAnqiVrDbTB4Wmi+DAUe 3Yz+FsVKF503WzrQdbQ9IXjCMRYotgh5AggkawxzOb8rkJw0OdLqmXAnk PUZoYQEUxOMiZcAt9snwKPeolLv+6foBTF8+bIbwc4DZmPFm/zq2ZoZ1A Q==; X-IronPort-AV: E=McAfee;i="6400,9594,10377"; a="277378308" X-IronPort-AV: E=Sophos;i="5.91,300,1647327600"; d="scan'208";a="277378308" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jun 2022 04:54:37 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,300,1647327600"; d="scan'208";a="830382602" Received: from silpixa00401384.ir.intel.com ([10.243.23.50]) by fmsmga006.fm.intel.com with ESMTP; 14 Jun 2022 04:54:35 -0700 From: Emma Finn To: ian.stokes@intel.com, echaudro@redhat.com, harry.van.haaren@intel.com, dev@openvswitch.org Date: Tue, 14 Jun 2022 11:54:05 +0000 Message-Id: <20220614115409.1143269-5-emma.finn@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220614115409.1143269-1-emma.finn@intel.com> References: <20220510142202.1087967-1-emma.finn@intel.com> <20220614115409.1143269-1-emma.finn@intel.com> MIME-Version: 1.0 Subject: [ovs-dev] [PATCH v7 07/11] odp-execute: Add ISA implementation of actions. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This commit adds the AVX512 implementation of the action functionality. Usage: $ ovs-appctl dpif-netdev/action-impl-set avx512 Signed-off-by: Emma Finn Acked-by: Harry van Haaren --- Documentation/ref/ovs-actions.7.rst | 26 ++++++++++++++ Documentation/topics/testing.rst | 24 ++++++++----- NEWS | 2 ++ lib/automake.mk | 6 +++- lib/cpu.c | 1 + lib/cpu.h | 1 + lib/odp-execute-avx512.c | 56 +++++++++++++++++++++++++++++ lib/odp-execute-private.c | 8 +++++ lib/odp-execute-private.h | 6 +++- 9 files changed, 120 insertions(+), 10 deletions(-) create mode 100644 lib/odp-execute-avx512.c diff --git a/Documentation/ref/ovs-actions.7.rst b/Documentation/ref/ovs-actions.7.rst index b59b7634f..a363d1571 100644 --- a/Documentation/ref/ovs-actions.7.rst +++ b/Documentation/ref/ovs-actions.7.rst @@ -2354,3 +2354,29 @@ never leads to recursion. The ``resubmit`` extension action is more flexible. **Conformance** OpenFlow 1.1 introduced ``goto_table``. Open vSwitch 2.1 added support for ``goto_table``. + +Actions Implementations +----------------------- + +Actions are used in OpenFlow flows to describe what to do when the flow +matches a packet. Just like with the datapath interface, SIMD instructions +with the userspace datapath can be applied to the action implementation to +improve performance. + +OVS provides multiple implementations of the actions. +Available implementations can be listed with the following command:: + + $ ovs-appctl dpif-netdev/action-impl-show + Available Actions implementations: + scalar (available: Yes, active: Yes) + autovalidator (available: Yes, active: No) + avx512 (available: Yes, active: No) + +By default, ``scalar`` is used. Implementations can be selected by +name:: + + $ ovs-appctl dpif-netdev/action-impl-set avx512 + Action implementation set to avx512. + + $ ovs-appctl dpif-netdev/action-impl-set scalar + Action implementation set to scalar. diff --git a/Documentation/topics/testing.rst b/Documentation/topics/testing.rst index c15d5b38f..d279131b8 100644 --- a/Documentation/topics/testing.rst +++ b/Documentation/topics/testing.rst @@ -361,12 +361,12 @@ testsuite. Userspace datapath: Testing and Validation of CPU-specific Optimizations '''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' -As multiple versions of the datapath classifier and packet parsing functions -can co-exist, each with different CPU ISA optimizations, it is important to -validate that they all give the exact same results. To easily test all the -implementations, an ``autovalidator`` implementation of them exists. This -implementation runs all other available implementations, and verifies that the -results are identical. +As multiple versions of the datapath classifier, packet parsing functions and +actions can co-exist, each with different CPU ISA optimizations, it is +important to validate that they all give the exact same results. To easily +test all the implementations, an ``autovalidator`` implementation of them +exists. This implementation runs all other available implementations, and +verifies that the results are identical. Running the OVS unit tests with the autovalidator enabled ensures all implementations provide the same results. Note that the performance of the @@ -382,18 +382,26 @@ To set the autovalidator for the packet parser, use this command:: $ ovs-appctl dpif-netdev/miniflow-parser-set autovalidator +To set the autovalidator for actions, use this command:: + + $ ovs-appctl dpif-netdev/action-impl-set autovalidator + To run the OVS unit test suite with the autovalidator as the default implementation, it is required to recompile OVS. During the recompilation, the default priority of the `autovalidator` implementation is set to the -maximum priority, ensuring every test will be run with every implementation:: +maximum priority, ensuring every test will be run with every implementation. +Priority is only related to mfex autovalidator and not the actions +autovalidator.:: - $ ./configure --enable-autovalidator --enable-mfex-default-autovalidator + $ ./configure --enable-autovalidator --enable-mfex-default-autovalidator \ + --enable-actions-default-autovalidator The following line should be seen in the configuration log when the above options are used:: checking whether DPCLS Autovalidator is default implementation... yes checking whether MFEX Autovalidator is default implementation... yes + checking whether actions Autovalidator is default implementation... yes Compile OVS in debug mode to have `ovs_assert` statements error out if there is a mis-match in the datapath classifier lookup or packet parser diff --git a/NEWS b/NEWS index 7ae4d87bb..927b7eed8 100644 --- a/NEWS +++ b/NEWS @@ -39,6 +39,8 @@ Post-v2.17.0 implementations available at run time. * Add build time configure command to enable auto-validator as default actions implementation at build time. + * Add AVX512 implementation of actions. + diff --git a/lib/automake.mk b/lib/automake.mk index 1a49dd30b..ad3ea3fb1 100644 --- a/lib/automake.mk +++ b/lib/automake.mk @@ -31,6 +31,9 @@ lib_LTLIBRARIES += lib/libopenvswitchavx512.la lib_libopenvswitch_la_LIBADD += lib/libopenvswitchavx512.la lib_libopenvswitchavx512_la_CFLAGS = \ -mavx512f \ + -mavx512bw \ + -mavx512vl \ + -mavx512dq \ -mbmi \ -mbmi2 \ -fPIC \ @@ -44,7 +47,8 @@ lib_libopenvswitchavx512_la_CFLAGS += \ -mavx512bw lib_libopenvswitchavx512_la_SOURCES += \ lib/dpif-netdev-extract-avx512.c \ - lib/dpif-netdev-lookup-avx512-gather.c + lib/dpif-netdev-lookup-avx512-gather.c \ + lib/odp-execute-avx512.c endif # HAVE_AVX512BW lib_libopenvswitchavx512_la_LDFLAGS = \ -static diff --git a/lib/cpu.c b/lib/cpu.c index 2df003c51..0292f715e 100644 --- a/lib/cpu.c +++ b/lib/cpu.c @@ -53,6 +53,7 @@ X86_ISA(X86_EXT_FEATURES_LEAF, EBX, 16, OVS_CPU_ISA_X86_AVX512F) X86_ISA(X86_EXT_FEATURES_LEAF, EBX, 30, OVS_CPU_ISA_X86_AVX512BW) X86_ISA(X86_EXT_FEATURES_LEAF, ECX, 1, OVS_CPU_ISA_X86_AVX512VBMI) X86_ISA(X86_EXT_FEATURES_LEAF, ECX, 14, OVS_CPU_ISA_X86_VPOPCNTDQ) +X86_ISA(X86_EXT_FEATURES_LEAF, EBX, 31, OVS_CPU_ISA_X86_AVX512VL) #endif bool diff --git a/lib/cpu.h b/lib/cpu.h index 92897bb71..3215229bc 100644 --- a/lib/cpu.h +++ b/lib/cpu.h @@ -25,6 +25,7 @@ enum ovs_cpu_isa { OVS_CPU_ISA_X86_AVX512F, OVS_CPU_ISA_X86_AVX512BW, OVS_CPU_ISA_X86_AVX512VBMI, + OVS_CPU_ISA_X86_AVX512VL, OVS_CPU_ISA_X86_VPOPCNTDQ, OVS_CPU_ISA_X86_LAST = OVS_CPU_ISA_X86_VPOPCNTDQ, }; diff --git a/lib/odp-execute-avx512.c b/lib/odp-execute-avx512.c new file mode 100644 index 000000000..1fb334689 --- /dev/null +++ b/lib/odp-execute-avx512.c @@ -0,0 +1,56 @@ +/* + * Copyright (c) 2022 Intel. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include +#include + +#include "cpu.h" +#include "dp-packet.h" +#include "immintrin.h" +#include "odp-execute-private.h" +#include "odp-netlink.h" +#include "openvswitch/vlog.h" + +/* Probe functions to check ISA requirements. */ +static bool +avx512_isa_probe(void) +{ + static enum ovs_cpu_isa isa_required[] = { + OVS_CPU_ISA_X86_AVX512F, + OVS_CPU_ISA_X86_AVX512BW, + OVS_CPU_ISA_X86_BMI2, + OVS_CPU_ISA_X86_AVX512VL, + }; + + bool ret = true; + for (int i = 0; i < ARRAY_SIZE(isa_required); i++) { + if (!cpu_has_isa(isa_required[i])) { + ret = -ENOTSUP; + } + } + + return ret; +} + +int +action_avx512_init(struct odp_execute_action_impl *self) +{ + if (!avx512_isa_probe()) { + return -ENOTSUP; + } + + return 0; +} diff --git a/lib/odp-execute-private.c b/lib/odp-execute-private.c index f8d0896b5..de2e4dfc4 100644 --- a/lib/odp-execute-private.c +++ b/lib/odp-execute-private.c @@ -42,6 +42,14 @@ static struct odp_execute_action_impl action_impls[] = { .name = "scalar", .init_func = odp_action_scalar_init, }, + + #if (__x86_64__ && HAVE_AVX512F && HAVE_LD_AVX512_GOOD && __SSE4_2__) + [ACTION_IMPL_AVX512] = { + .available = false, + .name = "avx512", + .init_func = action_avx512_init, + }, + #endif }; static void diff --git a/lib/odp-execute-private.h b/lib/odp-execute-private.h index 5322eb8df..e4724b8b2 100644 --- a/lib/odp-execute-private.h +++ b/lib/odp-execute-private.h @@ -59,6 +59,9 @@ enum odp_execute_action_impl_idx { * Do not change the autovalidator position in this list without updating * the define below. */ + #if (__x86_64__ && HAVE_AVX512F && HAVE_LD_AVX512_GOOD && __SSE4_2__) + ACTION_IMPL_AVX512, + #endif ACTION_IMPL_MAX, }; @@ -82,10 +85,11 @@ int odp_action_scalar_init(struct odp_execute_action_impl *self); int action_autoval_init(struct odp_execute_action_impl *self); +int action_avx512_init(struct odp_execute_action_impl *self); + int odp_execute_action_set(const char *name, struct odp_execute_action_impl *active); void odp_execute_action_get_info(struct ds *name); - #endif /* ODP_EXTRACT_PRIVATE */ From patchwork Tue Jun 14 11:54:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Finn, Emma" X-Patchwork-Id: 1643255 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=Xb1p/a5o; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LMn1D1zGYz9sG2 for ; Tue, 14 Jun 2022 21:54:56 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 1D66E61002; Tue, 14 Jun 2022 11:54:54 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Q91Nb5NlqWWv; Tue, 14 Jun 2022 11:54:52 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id 7CE0660C15; Tue, 14 Jun 2022 11:54:50 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 25B09C0081; Tue, 14 Jun 2022 11:54:50 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 369BEC007A for ; Tue, 14 Jun 2022 11:54:49 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 291ED40AE0 for ; Tue, 14 Jun 2022 11:54:42 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=intel.com Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Q9Jwh2JuSLEn for ; Tue, 14 Jun 2022 11:54:40 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by smtp2.osuosl.org (Postfix) with ESMTPS id AD35140A87 for ; Tue, 14 Jun 2022 11:54:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655207680; x=1686743680; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=d1qIzC67UCjH9kSBiWaTQx32rp2M1c6Y7e/Ld5qE8u4=; b=Xb1p/a5oh6Eejr9wMxkWIjPAXZhpsAxVegHEvN5uhUZU0eowvvXMFa8L R5TSyXGkaEpOf3qpuMu/zol7T60r5rfjxpHMuJXUxxAwC2H+OsESm7xHN MhvC1XQUp0sfllygjlaNg6/697fZFYMYDK+XEUUxALoJX/5HgGykn4uKy jIZCYurxU7aAALvVsW/T8HLfd0HxheLBeYmYKWg0TKSiv0TQyI39A/tlX AodAU1wAI11gZ++wykKeCko2wmlwybSiZUufjykohqMvqvqQzEnSl/+HH jT2bd9oVqlYb5/Cx09z/9fTvlOqVGTt30bj4bIcREZJ/e+VHAG3GzueJl w==; X-IronPort-AV: E=McAfee;i="6400,9594,10377"; a="277378316" X-IronPort-AV: E=Sophos;i="5.91,300,1647327600"; d="scan'208";a="277378316" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jun 2022 04:54:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,300,1647327600"; d="scan'208";a="830382622" Received: from silpixa00401384.ir.intel.com ([10.243.23.50]) by fmsmga006.fm.intel.com with ESMTP; 14 Jun 2022 04:54:39 -0700 From: Emma Finn To: ian.stokes@intel.com, echaudro@redhat.com, harry.van.haaren@intel.com, dev@openvswitch.org Date: Tue, 14 Jun 2022 11:54:06 +0000 Message-Id: <20220614115409.1143269-6-emma.finn@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220614115409.1143269-1-emma.finn@intel.com> References: <20220510142202.1087967-1-emma.finn@intel.com> <20220614115409.1143269-1-emma.finn@intel.com> MIME-Version: 1.0 Subject: [ovs-dev] [PATCH v7 08/11] odp-execute: Add ISA implementation of pop_vlan action. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This commit adds the AVX512 implementation of the pop_vlan action. Signed-off-by: Emma Finn --- lib/odp-execute-avx512.c | 91 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 91 insertions(+) diff --git a/lib/odp-execute-avx512.c b/lib/odp-execute-avx512.c index 1fb334689..f9e2b1727 100644 --- a/lib/odp-execute-avx512.c +++ b/lib/odp-execute-avx512.c @@ -14,6 +14,11 @@ * limitations under the License. */ +#ifdef __x86_64__ +/* Sparse cannot handle the AVX512 instructions. */ +#if !defined(__CHECKER__) + + #include #include @@ -24,6 +29,86 @@ #include "odp-netlink.h" #include "openvswitch/vlog.h" +VLOG_DEFINE_THIS_MODULE(odp_execute_avx512); +BUILD_ASSERT_DECL(offsetof(struct dp_packet, l2_5_ofs) + + MEMBER_SIZEOF(struct dp_packet, l2_5_ofs) == + offsetof(struct dp_packet, l3_ofs)); + +BUILD_ASSERT_DECL(offsetof(struct dp_packet, l3_ofs) + + MEMBER_SIZEOF(struct dp_packet, l3_ofs) == + offsetof(struct dp_packet, l4_ofs)); + +/* Adjust the size of the l2 portion of the dp_packet, updating the l2 + * pointer and the layer offsets. The function will broadcast resize_by_bytes + * across a register and uses a kmask to identify which lanes should be + * incremented/decremented. Either an add or subtract will be performed + * and the result is stored back to the original packet. */ +static inline void ALWAYS_INLINE +avx512_dp_packet_resize_l2(struct dp_packet *b, int resize_by_bytes) +{ + /* Update packet size/data pointers */ + if (resize_by_bytes >= 0) { + dp_packet_prealloc_headroom(b, resize_by_bytes); + } else { + ovs_assert(dp_packet_size(b) - dp_packet_l2_pad_size(b) >= + -resize_by_bytes); + } + + dp_packet_set_data(b, (char *) dp_packet_data(b) - resize_by_bytes); + dp_packet_set_size(b, dp_packet_size(b) + resize_by_bytes); + + const __m128i v_zeros = _mm_setzero_si128(); + const __m128i v_u16_max = _mm_cmpeq_epi16(v_zeros, v_zeros); + + const uint8_t k_lanes = 0b1110; + __m128i v_offset = _mm_set1_epi16(abs(resize_by_bytes)); + + /* Load 128 bits from the dp_packet structure starting at the l2_pad_size + * offset. */ + void *adjust_ptr = &b->l2_pad_size; + __m128i v_adjust_src = _mm_loadu_si128(adjust_ptr); + + __mmask8 k_cmp = _mm_mask_cmpneq_epu16_mask(k_lanes, v_adjust_src, + v_u16_max); + + __m128i v_adjust_wip; + + if (resize_by_bytes >= 0) { + v_adjust_wip = _mm_mask_add_epi16(v_adjust_src, k_cmp, + v_adjust_src, v_offset); + } else { + v_adjust_wip = _mm_mask_sub_epi16(v_adjust_src, k_cmp, + v_adjust_src, v_offset); + } + + _mm_storeu_si128(adjust_ptr, v_adjust_wip); +} + +/* This function will load the entire vlan_eth_header into a 128-bit wide + * register. Then use an 8-byte realign to shift the header right by 12 bytes + * to remove the vlan header and store the results back to the orginal header. + */ +static void +action_avx512_pop_vlan(struct dp_packet_batch *batch, + const struct nlattr *a OVS_UNUSED) +{ + struct dp_packet *packet; + + DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { + struct vlan_eth_header *veh = dp_packet_eth(packet); + + if (veh && dp_packet_size(packet) >= sizeof *veh && + eth_type_vlan(veh->veth_type)) { + + __m128i v_ether = _mm_loadu_si128((void *) veh); + __m128i v_realign = _mm_alignr_epi8(v_ether, _mm_setzero_si128(), + 16 - VLAN_HEADER_LEN); + _mm_storeu_si128((void *) veh, v_realign); + avx512_dp_packet_resize_l2(packet, -VLAN_HEADER_LEN); + } + } +} + /* Probe functions to check ISA requirements. */ static bool avx512_isa_probe(void) @@ -52,5 +137,11 @@ action_avx512_init(struct odp_execute_action_impl *self) return -ENOTSUP; } + /* Set function pointers for actions that can be applied directly, these + * are identified by OVS_ACTION_ATTR_*. */ + self->funcs[OVS_ACTION_ATTR_POP_VLAN] = action_avx512_pop_vlan; return 0; } + +#endif +#endif From patchwork Tue Jun 14 11:54:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Finn, Emma" X-Patchwork-Id: 1643256 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=JEVK1utI; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LMn1R1DHpz9sG2 for ; Tue, 14 Jun 2022 21:55:07 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 289D74189B; Tue, 14 Jun 2022 11:55:05 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 48Ms1zUkcdCO; Tue, 14 Jun 2022 11:55:03 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id 74DB8418C6; Tue, 14 Jun 2022 11:55:02 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 51F77C0081; Tue, 14 Jun 2022 11:55:02 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 7F8C1C0032 for ; Tue, 14 Jun 2022 11:55:01 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 147F840B29 for ; Tue, 14 Jun 2022 11:54:44 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=intel.com Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3sbDqWWUkiLz for ; Tue, 14 Jun 2022 11:54:43 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by smtp2.osuosl.org (Postfix) with ESMTPS id 48C2840B41 for ; Tue, 14 Jun 2022 11:54:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655207683; x=1686743683; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=nwuiMUI4AwYsoWJ8tNty9haSYOXesi1EVKye8j0ojaI=; b=JEVK1utIFTrFuI557Btcdx57/m4dDdLLkcUZaPj8/FED8WFFRA6SfZps QvqIxllvZXeyR4Xvg924aHn5EFECinjxIHslL65a91DnL06fHIHHVcyuE nHxIxv5+28rGRZ9W8rWs76gEgyivufooAM7wM3nn0r5NZfb7nFkkglTJt 1Fg1GoQaF5GcrquPznCZm6yulmMO+GOksDttVfUdQmVtDBJ3L4Ms3dBgB h/3JJYK6LD+LGsdp5dKd2vZR5HiSE34LSF3mvJ6Cq76/ViqLQ3UfImo2S 1V1BgzBDlaDXrFxCgnxxr7smXKPeMAe/EF1MdaTenWMsb62LLejyBM47v g==; X-IronPort-AV: E=McAfee;i="6400,9594,10377"; a="277378323" X-IronPort-AV: E=Sophos;i="5.91,300,1647327600"; d="scan'208";a="277378323" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jun 2022 04:54:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,300,1647327600"; d="scan'208";a="830382637" Received: from silpixa00401384.ir.intel.com ([10.243.23.50]) by fmsmga006.fm.intel.com with ESMTP; 14 Jun 2022 04:54:41 -0700 From: Emma Finn To: ian.stokes@intel.com, echaudro@redhat.com, harry.van.haaren@intel.com, dev@openvswitch.org Date: Tue, 14 Jun 2022 11:54:07 +0000 Message-Id: <20220614115409.1143269-7-emma.finn@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220614115409.1143269-1-emma.finn@intel.com> References: <20220510142202.1087967-1-emma.finn@intel.com> <20220614115409.1143269-1-emma.finn@intel.com> MIME-Version: 1.0 Subject: [ovs-dev] [PATCH v7 09/11] odp-execute: Add ISA implementation of push_vlan action. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This commit adds the AVX512 implementation of the push_vlan action. Signed-off-by: Emma Finn --- lib/odp-execute-avx512.c | 37 +++++++++++++++++++++++++++++++++++++ lib/odp-execute-private.c | 1 + lib/odp-execute.c | 22 +++++++++++++--------- 3 files changed, 51 insertions(+), 9 deletions(-) diff --git a/lib/odp-execute-avx512.c b/lib/odp-execute-avx512.c index f9e2b1727..bb178cbac 100644 --- a/lib/odp-execute-avx512.c +++ b/lib/odp-execute-avx512.c @@ -109,6 +109,41 @@ action_avx512_pop_vlan(struct dp_packet_batch *batch, } } +/* This function will load the entire eth_header into a 128-bit wide register. + * Then use an 8-byte shuffle to shift the data left to make room for + * the vlan header. Insert the new vlan header and then store back to the + * original packet. */ +static void +action_avx512_push_vlan(struct dp_packet_batch *batch, const struct nlattr *a) +{ + struct dp_packet *packet; + const struct ovs_action_push_vlan *vlan = nl_attr_get(a); + ovs_be16 tpid, tci; + + DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { + tpid = vlan->vlan_tpid; + tci = vlan->vlan_tci; + + avx512_dp_packet_resize_l2(packet, VLAN_HEADER_LEN); + + /* Build up the VLAN TCI/TPID, and merge with the moving of Ether. */ + char *pkt_data = (char *) dp_packet_data(packet); + const uint16_t tci_proc = tci & htons(~VLAN_CFI); + const uint32_t tpid_tci = (tci_proc << 16) | tpid; + + static const uint8_t vlan_push_shuffle_mask[16] = { + 4, 5, 6, 7, 8, 9, 10, 11, + 12, 13, 14, 15, 0xFF, 0xFF, 0xFF, 0xFF + }; + + __m128i v_ether = _mm_loadu_si128((void *) pkt_data); + __m128i v_index = _mm_loadu_si128((void *) vlan_push_shuffle_mask); + __m128i v_shift = _mm_shuffle_epi8(v_ether, v_index); + __m128i v_vlan_hdr = _mm_insert_epi32(v_shift, tpid_tci, 3); + _mm_storeu_si128((void *) pkt_data, v_vlan_hdr); + } +} + /* Probe functions to check ISA requirements. */ static bool avx512_isa_probe(void) @@ -140,6 +175,8 @@ action_avx512_init(struct odp_execute_action_impl *self) /* Set function pointers for actions that can be applied directly, these * are identified by OVS_ACTION_ATTR_*. */ self->funcs[OVS_ACTION_ATTR_POP_VLAN] = action_avx512_pop_vlan; + self->funcs[OVS_ACTION_ATTR_PUSH_VLAN] = action_avx512_push_vlan; + return 0; } diff --git a/lib/odp-execute-private.c b/lib/odp-execute-private.c index de2e4dfc4..751a68fe3 100644 --- a/lib/odp-execute-private.c +++ b/lib/odp-execute-private.c @@ -209,6 +209,7 @@ action_autoval_init(struct odp_execute_action_impl *self) /* Set function pointers for actions that can be applied directly, these * are identified by OVS_ACTION_ATTR_*. */ self->funcs[OVS_ACTION_ATTR_POP_VLAN] = action_autoval_generic; + self->funcs[OVS_ACTION_ATTR_PUSH_VLAN] = action_autoval_generic; return 0; } diff --git a/lib/odp-execute.c b/lib/odp-execute.c index a49b331ef..59f6bdc64 100644 --- a/lib/odp-execute.c +++ b/lib/odp-execute.c @@ -845,6 +845,17 @@ action_pop_vlan(struct dp_packet_batch *batch, } } +static void +action_push_vlan(struct dp_packet_batch *batch, const struct nlattr *a) +{ + struct dp_packet *packet; + const struct ovs_action_push_vlan *vlan = nl_attr_get(a); + + DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { + eth_push_vlan(packet, vlan->vlan_tpid, vlan->vlan_tci); + } +} + /* Implementation of the scalar actions impl init function. Build up the * array of func ptrs here. */ @@ -854,6 +865,7 @@ odp_action_scalar_init(struct odp_execute_action_impl *self) /* Set function pointers for actions that can be applied directly, these * are identified by OVS_ACTION_ATTR_*. */ self->funcs[OVS_ACTION_ATTR_POP_VLAN] = action_pop_vlan; + self->funcs[OVS_ACTION_ATTR_PUSH_VLAN] = action_push_vlan; return 0; } @@ -995,15 +1007,6 @@ odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, break; } - case OVS_ACTION_ATTR_PUSH_VLAN: { - const struct ovs_action_push_vlan *vlan = nl_attr_get(a); - - DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { - eth_push_vlan(packet, vlan->vlan_tpid, vlan->vlan_tci); - } - break; - } - case OVS_ACTION_ATTR_PUSH_MPLS: { const struct ovs_action_push_mpls *mpls = nl_attr_get(a); @@ -1156,6 +1159,7 @@ odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, case __OVS_ACTION_ATTR_MAX: /* The following actions are handled by the scalar implementation. */ case OVS_ACTION_ATTR_POP_VLAN: + case OVS_ACTION_ATTR_PUSH_VLAN: OVS_NOT_REACHED(); } From patchwork Tue Jun 14 11:54:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Finn, Emma" X-Patchwork-Id: 1643258 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=CzsfkI1w; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LMn1n1MTxz9sG2 for ; Tue, 14 Jun 2022 21:55:25 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 6329A8317B; Tue, 14 Jun 2022 11:55:23 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Geg5dthMHbDc; Tue, 14 Jun 2022 11:55:22 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTPS id 3AFEF83180; Tue, 14 Jun 2022 11:55:21 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 10959C0032; Tue, 14 Jun 2022 11:55:21 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 5C874C002D for ; Tue, 14 Jun 2022 11:55:20 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id EB35D40B2F for ; Tue, 14 Jun 2022 11:54:47 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=intel.com Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3MdTB4PlRABj for ; Tue, 14 Jun 2022 11:54:46 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by smtp2.osuosl.org (Postfix) with ESMTPS id 5A0F740B06 for ; Tue, 14 Jun 2022 11:54:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655207686; x=1686743686; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=yFNh4cSzPLRFmus4NGFo8SqUv+WkUj67ECYJKzWFt5s=; b=CzsfkI1wCfE7r2q7eaaWLY12dKeaLzxwwEbu/F3xWyfbDueOB4xUX8nF 5MuZOp8tFMH/nT5sEWdE90PW4KLsxnct1l8cs65jUVpKnTfSTpIkmen/A YK0fAKTphT8yDEKvU6Zh7hLxHxzFEQjcOc3zjOIh6LIS1pd/RkAPFjc4w Gz1J+/HdOMDYm/zxOUcVO/USU9eMBWIQ+jivcjmCyzBfoLg8Hiw6W3TId jWTJa9Ro17QywWv0JIC3vzUUbsO0oevbvb2ljXQAr+oTBCmBmX6S2K/zo pRD+M7akfTqc+9z40XhTApmuN9Lk/ezOOdAEKgzb3lqH1ytRlAqG6hCPI g==; X-IronPort-AV: E=McAfee;i="6400,9594,10377"; a="277378327" X-IronPort-AV: E=Sophos;i="5.91,300,1647327600"; d="scan'208";a="277378327" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jun 2022 04:54:46 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,300,1647327600"; d="scan'208";a="830382657" Received: from silpixa00401384.ir.intel.com ([10.243.23.50]) by fmsmga006.fm.intel.com with ESMTP; 14 Jun 2022 04:54:44 -0700 From: Emma Finn To: ian.stokes@intel.com, echaudro@redhat.com, harry.van.haaren@intel.com, dev@openvswitch.org Date: Tue, 14 Jun 2022 11:54:08 +0000 Message-Id: <20220614115409.1143269-8-emma.finn@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220614115409.1143269-1-emma.finn@intel.com> References: <20220510142202.1087967-1-emma.finn@intel.com> <20220614115409.1143269-1-emma.finn@intel.com> MIME-Version: 1.0 Subject: [ovs-dev] [PATCH v7 10/11] odp-execute: Add ISA implementation of set_masked ETH X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This commit includes infrastructure changes for enabling set_masked_X actions and also adds support for the AVX512 implementation of the eth_set_addrs action. Signed-off-by: Emma Finn --- lib/odp-execute-avx512.c | 69 +++++++++++++++++++++++++++++++++++++++ lib/odp-execute-private.c | 56 +++++++++++++++++++++++++++++-- lib/odp-execute-private.h | 4 +++ lib/odp-execute.c | 65 +++++++++++++++++++++++++----------- lib/odp-execute.h | 3 ++ 5 files changed, 175 insertions(+), 22 deletions(-) diff --git a/lib/odp-execute-avx512.c b/lib/odp-execute-avx512.c index bb178cbac..ffe25b41d 100644 --- a/lib/odp-execute-avx512.c +++ b/lib/odp-execute-avx512.c @@ -38,6 +38,12 @@ BUILD_ASSERT_DECL(offsetof(struct dp_packet, l3_ofs) + MEMBER_SIZEOF(struct dp_packet, l3_ofs) == offsetof(struct dp_packet, l4_ofs)); +BUILD_ASSERT_DECL(offsetof(struct ovs_key_ethernet, eth_src) + + MEMBER_SIZEOF(struct ovs_key_ethernet, eth_src) == + offsetof(struct ovs_key_ethernet, eth_dst)); + +static struct odp_execute_action_impl avx512_impl; + /* Adjust the size of the l2 portion of the dp_packet, updating the l2 * pointer and the layer offsets. The function will broadcast resize_by_bytes * across a register and uses a kmask to identify which lanes should be @@ -144,6 +150,61 @@ action_avx512_push_vlan(struct dp_packet_batch *batch, const struct nlattr *a) } } +/* This function will load the contents of eth_header into a 128-bit wide + * register. Then an 8-byte shuffle is required to shuffle both key and + * mask to match the layout of the eth_header struct. A bitwise ANDNOT and OR + * is performed on the entire header and results are stored back. */ +static void +action_avx512_eth_set_addrs(struct dp_packet_batch *batch, + const struct nlattr *a) +{ + a = nl_attr_get(a); + const struct ovs_key_ethernet *key = nl_attr_get(a); + const struct ovs_key_ethernet *mask = get_mask(a, struct ovs_key_ethernet); + struct dp_packet *packet; + + __m128i v_src = _mm_loadu_si128((void *) key); + __m128i v_mask = _mm_loadu_si128((void *) mask); + + DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { + + struct eth_header *eh = dp_packet_eth(packet); + + if (!eh) { + continue; + } + + static const uint8_t eth_shuffle[16] = { + 6, 7, 8, 9, 10, 11, 0, 1, + 2, 3, 4, 5, 12, 13, 14, 15 + }; + + __m128i v_dst = _mm_loadu_si128((void *) eh); + __m128i v_shuf = _mm_loadu_si128((void *) eth_shuffle); + + v_src = _mm_shuffle_epi8(v_src, v_shuf); + v_mask = _mm_shuffle_epi8(v_mask, v_shuf); + + __m128i dst_masked = _mm_andnot_si128(v_mask, v_dst); + __m128i res = _mm_or_si128(v_src, dst_masked); + + __m128i res_blend = _mm_blend_epi16(v_dst, res, 0x3F); + _mm_storeu_si128((void *) eh, res_blend); + } +} + +static void +action_avx512_set_masked(struct dp_packet_batch *batch OVS_UNUSED, + const struct nlattr *a) +{ + a = nl_attr_get(a); + enum ovs_key_attr attr_type = nl_attr_type(a); + + if (avx512_impl.set_masked_funcs[attr_type]) { + avx512_impl.set_masked_funcs[attr_type](batch, a); + } +} + /* Probe functions to check ISA requirements. */ static bool avx512_isa_probe(void) @@ -176,6 +237,14 @@ action_avx512_init(struct odp_execute_action_impl *self) * are identified by OVS_ACTION_ATTR_*. */ self->funcs[OVS_ACTION_ATTR_POP_VLAN] = action_avx512_pop_vlan; self->funcs[OVS_ACTION_ATTR_PUSH_VLAN] = action_avx512_push_vlan; + self->funcs[OVS_ACTION_ATTR_SET_MASKED] = action_avx512_set_masked; + + /* Set function pointers that need a 2nd-level function. SET_MASKED action + * requires further processing for action type. Note that 2nd level items + * are identified by OVS_KEY_ATTR_*. */ + self->set_masked_funcs[OVS_KEY_ATTR_ETHERNET] = + action_avx512_eth_set_addrs; + avx512_impl = *self; return 0; } diff --git a/lib/odp-execute-private.c b/lib/odp-execute-private.c index 751a68fe3..e2d650779 100644 --- a/lib/odp-execute-private.c +++ b/lib/odp-execute-private.c @@ -29,6 +29,8 @@ VLOG_DEFINE_THIS_MODULE(odp_execute_impl); static int active_action_impl_index; static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1); +static struct odp_execute_action_impl autoval_impl; +static bool set_masked = false; static struct odp_execute_action_impl action_impls[] = { [ACTION_IMPL_AUTOVALIDATOR] = { @@ -59,6 +61,11 @@ action_impl_copy_funcs(struct odp_execute_action_impl *src, for (int i = 0; i < __OVS_ACTION_ATTR_MAX; i++) { atomic_store_relaxed(&src->funcs[i], dst->funcs[i]); } + + for (uint32_t i = 0; i < __OVS_KEY_ATTR_MAX; i++) { + atomic_store_relaxed(&src->set_masked_funcs[i], + dst->set_masked_funcs[i]); + } } int @@ -135,19 +142,36 @@ action_autoval_generic(struct dp_packet_batch *batch, const struct nlattr *a) bool failed = false; int type = nl_attr_type(a); enum ovs_action_attr attr_type = (enum ovs_action_attr) type; + enum ovs_key_attr key_attr_type = (enum ovs_key_attr) type; + + if (attr_type == OVS_ACTION_ATTR_SET_MASKED) { + set_masked = true; + const struct nlattr *key = nl_attr_get(a); + key_attr_type = nl_attr_type(key); + } + struct odp_execute_action_impl *scalar = &action_impls[ACTION_IMPL_SCALAR]; struct dp_packet_batch good_batch; dp_packet_batch_clone(&good_batch, batch); - scalar->funcs[attr_type](&good_batch, a); + if (!set_masked) { + scalar->funcs[attr_type](&good_batch, a); + } else { + scalar->set_masked_funcs[key_attr_type](&good_batch, a); + } for (int impl = ACTION_IMPL_BEGIN; impl < ACTION_IMPL_MAX; impl++) { /* Clone original batch and execute implementation under test. */ struct dp_packet_batch test_batch; dp_packet_batch_clone(&test_batch, batch); - action_impls[impl].funcs[attr_type](&test_batch, a); + + if (!set_masked) { + action_impls[impl].funcs[attr_type](&test_batch, a); + } else { + action_impls[impl].set_masked_funcs[key_attr_type](&test_batch, a); + } /* Loop over implementations, checking each one. */ for (int pidx = 0; pidx < batch->count; pidx++) { @@ -200,7 +224,26 @@ action_autoval_generic(struct dp_packet_batch *batch, const struct nlattr *a) dp_packet_delete_batch(&good_batch, 1); /* Apply the action to the original batch for continued processing. */ - scalar->funcs[attr_type](batch, a); + if (!set_masked) { + scalar->funcs[attr_type](batch, a); + } else { + scalar->set_masked_funcs[key_attr_type](batch, a); + } + + set_masked = false; +} + +static void +action_set_masked_init(struct dp_packet_batch *batch OVS_UNUSED, + const struct nlattr *a) +{ + const struct nlattr *type = nl_attr_get(a); + enum ovs_key_attr attr_type = nl_attr_type(type); + + if (autoval_impl.set_masked_funcs[attr_type]) { + set_masked = true; + autoval_impl.set_masked_funcs[attr_type](batch, a); + } } int @@ -210,6 +253,13 @@ action_autoval_init(struct odp_execute_action_impl *self) * are identified by OVS_ACTION_ATTR_*. */ self->funcs[OVS_ACTION_ATTR_POP_VLAN] = action_autoval_generic; self->funcs[OVS_ACTION_ATTR_PUSH_VLAN] = action_autoval_generic; + self->funcs[OVS_ACTION_ATTR_SET_MASKED] = action_set_masked_init; + + /* Set function pointers that need a 2nd-level function. SET_MASKED action + * requires further processing for action type. Note that 2nd level items + * are identified by OVS_KEY_ATTR_*. */ + self->set_masked_funcs[OVS_KEY_ATTR_ETHERNET] = action_autoval_generic; + autoval_impl = *self; return 0; } diff --git a/lib/odp-execute-private.h b/lib/odp-execute-private.h index e4724b8b2..1f4d614ca 100644 --- a/lib/odp-execute-private.h +++ b/lib/odp-execute-private.h @@ -49,6 +49,10 @@ struct odp_execute_action_impl { /* An array of callback functions, one for each action. */ ATOMIC(odp_execute_action_cb) funcs[__OVS_ACTION_ATTR_MAX]; + + /* An array of callback functions, one for each action type. */ + ATOMIC(odp_execute_action_cb) set_masked_funcs[__OVS_KEY_ATTR_MAX]; + }; /* Order of Actions implementations. */ diff --git a/lib/odp-execute.c b/lib/odp-execute.c index 59f6bdc64..db6e1ec03 100644 --- a/lib/odp-execute.c +++ b/lib/odp-execute.c @@ -561,8 +561,6 @@ odp_execute_set_action(struct dp_packet *packet, const struct nlattr *a) } } -#define get_mask(a, type) ((const type *)(const void *)(a + 1) + 1) - static void odp_execute_masked_set_action(struct dp_packet *packet, const struct nlattr *a) @@ -582,11 +580,6 @@ odp_execute_masked_set_action(struct dp_packet *packet, | (md->pkt_mark & ~*get_mask(a, uint32_t)); break; - case OVS_KEY_ATTR_ETHERNET: - odp_eth_set_addrs(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_ethernet)); - break; - case OVS_KEY_ATTR_NSH: { odp_set_nsh(packet, a, true); break; @@ -669,6 +662,8 @@ odp_execute_masked_set_action(struct dp_packet *packet, case OVS_KEY_ATTR_TCP_FLAGS: case OVS_KEY_ATTR_TUNNEL_INFO: case __OVS_KEY_ATTR_MAX: + /* The following action types are handled by the scalar implementation. */ + case OVS_KEY_ATTR_ETHERNET: default: OVS_NOT_REACHED(); } @@ -834,6 +829,12 @@ requires_datapath_assistance(const struct nlattr *a) return false; } +/* The active function pointers on the datapath. ISA optimized implementations + * are enabled by plugging them into this static arary, which is consulted when + * applying actions on the datapath. + */ +static struct odp_execute_action_impl actions_active_impl; + static void action_pop_vlan(struct dp_packet_batch *batch, const struct nlattr *a OVS_UNUSED) @@ -856,6 +857,36 @@ action_push_vlan(struct dp_packet_batch *batch, const struct nlattr *a) } } +static void +action_set_masked(struct dp_packet_batch *batch, const struct nlattr *a) +{ + struct dp_packet *packet; + + const struct nlattr *key = nl_attr_get(a); + enum ovs_key_attr key_type = nl_attr_type(key); + + if (actions_active_impl.set_masked_funcs[key_type]) { + actions_active_impl.set_masked_funcs[key_type](batch, a); + } else { + a = nl_attr_get(a); + DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { + odp_execute_masked_set_action(packet, a); + } + } +} + +static void +action_mod_eth(struct dp_packet_batch *batch, const struct nlattr *a) +{ + a = nl_attr_get(a); + struct dp_packet *packet; + + DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { + odp_eth_set_addrs(packet, nl_attr_get(a), + get_mask(a, struct ovs_key_ethernet)); + } +} + /* Implementation of the scalar actions impl init function. Build up the * array of func ptrs here. */ @@ -866,16 +897,17 @@ odp_action_scalar_init(struct odp_execute_action_impl *self) * are identified by OVS_ACTION_ATTR_*. */ self->funcs[OVS_ACTION_ATTR_POP_VLAN] = action_pop_vlan; self->funcs[OVS_ACTION_ATTR_PUSH_VLAN] = action_push_vlan; + self->funcs[OVS_ACTION_ATTR_SET_MASKED] = action_set_masked; + + /* Set function pointers that need a 2nd-level function. SET_MASKED action + * requires further processing for action type. Note that 2nd level items + * are identified by OVS_KEY_ATTR_*. */ + self->set_masked_funcs[OVS_KEY_ATTR_ETHERNET] = action_mod_eth; + actions_active_impl = *self; return 0; } -/* The active function pointers on the datapath. ISA optimized implementations - * are enabled by plugging them into this static arary, which is consulted when - * applying actions on the datapath. - */ -static struct odp_execute_action_impl actions_active_impl; - void odp_execute_init(void) { @@ -1028,12 +1060,6 @@ odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, } break; - case OVS_ACTION_ATTR_SET_MASKED: - DP_PACKET_BATCH_FOR_EACH(i, packet, batch) { - odp_execute_masked_set_action(packet, nl_attr_get(a)); - } - break; - case OVS_ACTION_ATTR_SAMPLE: DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { odp_execute_sample(dp, packet, steal && last_action, a, @@ -1160,6 +1186,7 @@ odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, /* The following actions are handled by the scalar implementation. */ case OVS_ACTION_ATTR_POP_VLAN: case OVS_ACTION_ATTR_PUSH_VLAN: + case OVS_ACTION_ATTR_SET_MASKED: OVS_NOT_REACHED(); } diff --git a/lib/odp-execute.h b/lib/odp-execute.h index 8668ab73f..762b99473 100644 --- a/lib/odp-execute.h +++ b/lib/odp-execute.h @@ -50,4 +50,7 @@ void odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, const struct nlattr *actions, size_t actions_len, odp_execute_cb dp_execute_action); + +#define get_mask(a, type) ((const type *)(const void *)(a + 1) + 1) + #endif From patchwork Tue Jun 14 11:54:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Finn, Emma" X-Patchwork-Id: 1643259 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=bjjx4UUN; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LMn1z1RjGz9sG2 for ; Tue, 14 Jun 2022 21:55:35 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 1B6B441907; Tue, 14 Jun 2022 11:55:33 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sWujz3PAQFpp; Tue, 14 Jun 2022 11:55:30 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTPS id 735EE418C4; Tue, 14 Jun 2022 11:55:29 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 45180C007A; Tue, 14 Jun 2022 11:55:29 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 13E47C0032 for ; Tue, 14 Jun 2022 11:55:28 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 9DCDC40B60 for ; Tue, 14 Jun 2022 11:54:51 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=intel.com Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6IemGljNi8Hd for ; Tue, 14 Jun 2022 11:54:50 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by smtp2.osuosl.org (Postfix) with ESMTPS id 47D7640B59 for ; Tue, 14 Jun 2022 11:54:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655207690; x=1686743690; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=kcGTVmrDCadXPfeF00LJllOLj2bTBhGeVFFrFkZREeU=; b=bjjx4UUNZIDRj2sA0AMl4+MSNId5Ls0Zivxpicn9Qb9GN5oGDKRsqC9F Fp8p1P0mNPG6OjBoDySUHL+AoRDbMyCuARzK9+5+AgH0IL8K2Hlu/7g4C jWNahXkBbWYEiLTuqAFAnZ97gMIalI8MgnX++CMC+AeQVOVgCjaxI0nOI vCE1SPX9rK2JT7rbU5UoIPkENTxJQaO8VWAmd0J4OBDJghvYlvwx1AMY+ M6a9dLvWH+2yg4VCAvwL/hlLPbnNzLxwPA6usO/+RzfIj0+KuNpJ6ILJV Sp17VychPxsV7ftOLkaVqgdncoUHze2g9HXtCd52C/FOhLdvchdJuCmHN g==; X-IronPort-AV: E=McAfee;i="6400,9594,10377"; a="277378333" X-IronPort-AV: E=Sophos;i="5.91,300,1647327600"; d="scan'208";a="277378333" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jun 2022 04:54:50 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,300,1647327600"; d="scan'208";a="830382686" Received: from silpixa00401384.ir.intel.com ([10.243.23.50]) by fmsmga006.fm.intel.com with ESMTP; 14 Jun 2022 04:54:48 -0700 From: Emma Finn To: ian.stokes@intel.com, echaudro@redhat.com, harry.van.haaren@intel.com, dev@openvswitch.org Date: Tue, 14 Jun 2022 11:54:09 +0000 Message-Id: <20220614115409.1143269-9-emma.finn@intel.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20220614115409.1143269-1-emma.finn@intel.com> References: <20220510142202.1087967-1-emma.finn@intel.com> <20220614115409.1143269-1-emma.finn@intel.com> MIME-Version: 1.0 Subject: [ovs-dev] [PATCH v7 11/11] odp-execute: Add ISA implementation of set_masked IPv4 action X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This commit adds support for the AVX512 implementation of the ipv4_set_addrs action as well as an AVX512 implementation of updating the checksums. Signed-off-by: Emma Finn --- lib/odp-execute-avx512.c | 210 ++++++++++++++++++++++++++++++++++++++ lib/odp-execute-private.c | 1 + lib/odp-execute.c | 19 +++- 3 files changed, 225 insertions(+), 5 deletions(-) diff --git a/lib/odp-execute-avx512.c b/lib/odp-execute-avx512.c index ffe25b41d..5cba14b92 100644 --- a/lib/odp-execute-avx512.c +++ b/lib/odp-execute-avx512.c @@ -22,6 +22,7 @@ #include #include +#include "csum.h" #include "cpu.h" #include "dp-packet.h" #include "immintrin.h" @@ -193,6 +194,213 @@ action_avx512_eth_set_addrs(struct dp_packet_batch *batch, } } +/* Calculate delta checksum by summing only ip_src and ip_dst fields of + * ip_header. Resulting checksum will be used for updating L4 checksum */ +static inline uint16_t ALWAYS_INLINE +avx512_l4_update_csum(struct ip_header *old_header, __m256i res) +{ + uint16_t tmp_checksum; + __m256i v_zeros = _mm256_setzero_si256(); + + /* Each field needs to be shuffle into 16- bit granularity and across + * lanes. */ + __m256i v_swap16a = _mm256_setr_epi16(0x0100, 0xffff, 0x0302, 0xffff, + 0x0504, 0xffff, 0x0706, 0xffff, + 0x0100, 0xffff, 0x0302, 0xffff, + 0xffff, 0xffff, 0xffff, 0xffff); + __m256i v_swap16b = _mm256_setr_epi16(0x0908, 0xffff, 0xffff, 0xffff, + 0x0d0c, 0xffff, 0x0f0e, 0xffff, + 0xffff, 0xffff, 0xffff, 0xffff, + 0xffff, 0xffff, 0xffff, 0xffff); + __m256i v_swap32a = _mm256_setr_epi32(0x0, 0x4, 0xF, 0xF, + 0xF, 0xF, 0xF, 0xF); + + __m256i oh = _mm256_loadu_si256((void *) old_header); + oh = _mm256_mask_blend_epi16(0x3C0, oh, res); + __m256i v_shuf1 = _mm256_shuffle_epi8(oh, v_swap16a); + __m256i v_shuf2 = _mm256_shuffle_epi8(oh, v_swap16b); + + /* Add field values. */ + __m256i v_sum = _mm256_add_epi32(v_shuf1, v_shuf2); + + /* Perform horizontal add to go from 8x32-bits to 2x32-bits. */ + v_sum = _mm256_hadd_epi32(v_sum, v_zeros); + v_sum = _mm256_hadd_epi32(v_sum, v_zeros); + + /* Shuffle 32-bit value from 3rd lane into first lane for final hadd. */ + v_sum = _mm256_permutexvar_epi32(v_swap32a, v_sum); + v_sum = _mm256_hadd_epi32(v_sum, v_zeros); + v_sum = _mm256_hadd_epi16(v_sum, v_zeros); + + /* Extract checksum value. */ + tmp_checksum = _mm256_extract_epi16(v_sum, 0); + + return ~tmp_checksum; +} + +/* Calculate checksum by summing entire contents of ip_header leaving out + * current checksum field. */ +static inline uint16_t ALWAYS_INLINE +avx512_ipv4_recalc_csum(__m256i res) +{ + uint32_t new_checksum; + __m256i v_zeros = _mm256_setzero_si256(); + + /* Each field needs to be shuffle into 16-bit granularity and across + * lanes. */ + __m256i v_swap16a = _mm256_setr_epi16(0x0100, 0xffff, 0x0302, 0xffff, + 0x0504, 0xffff, 0x0706, 0xffff, + 0x0100, 0xffff, 0x0302, 0xffff, + 0xffff, 0xffff, 0xffff, 0xffff); + + __m256i v_swap16b = _mm256_setr_epi16(0x0908, 0xffff, 0xffff, 0xffff, + 0x0d0c, 0xffff, 0x0f0e, 0xffff, + 0xffff, 0xffff, 0xffff, 0xffff, + 0xffff, 0xffff, 0xffff, 0xffff); + + __m256i v_swap32a = _mm256_setr_epi32(0x0, 0x4, 0xF, 0xF, + 0xF, 0xF, 0xF, 0xF); + + __m256i v_shuf1 = _mm256_shuffle_epi8(res, v_swap16a); + __m256i v_shuf2 = _mm256_shuffle_epi8(res, v_swap16b); + + /* Add field values. */ + __m256i v_sum = _mm256_add_epi32(v_shuf1, v_shuf2); + + /* Perform horizontal add to go from 8x32-bits to 2x32-bits. */ + v_sum = _mm256_hadd_epi32(v_sum, v_zeros); + v_sum = _mm256_hadd_epi32(v_sum, v_zeros); + + /* Shuffle 32-bit value from 3rd lane into first lane for final hadd. */ + v_sum = _mm256_permutexvar_epi32(v_swap32a, v_sum); + v_sum = _mm256_hadd_epi32(v_sum, v_zeros); + v_sum = _mm256_hadd_epi16(v_sum, v_zeros); + + /* Extract new checksum value. */ + new_checksum = _mm256_extract_epi16(v_sum, 0); + + return ~new_checksum; +} + +/* The shuffles used in action_avx512_ipv4_set_addrs() require the ovs_key_ipv4 + * struct to be in this layout. If struct changes, shuffle mask also needs to + * be updated. */ +BUILD_ASSERT_DECL(offsetof(struct ovs_key_ipv4, ipv4_src) + + MEMBER_SIZEOF(struct ovs_key_ipv4, ipv4_src) == + offsetof(struct ovs_key_ipv4, ipv4_dst)); + +BUILD_ASSERT_DECL(offsetof(struct ovs_key_ipv4, ipv4_dst) + + MEMBER_SIZEOF(struct ovs_key_ipv4, ipv4_dst) == + offsetof(struct ovs_key_ipv4, ipv4_proto)); + +BUILD_ASSERT_DECL(offsetof(struct ovs_key_ipv4, ipv4_proto) + + MEMBER_SIZEOF(struct ovs_key_ipv4, ipv4_proto) == + offsetof(struct ovs_key_ipv4, ipv4_tos)); + +BUILD_ASSERT_DECL(offsetof(struct ovs_key_ipv4, ipv4_tos) + + MEMBER_SIZEOF(struct ovs_key_ipv4, ipv4_tos) == + offsetof(struct ovs_key_ipv4, ipv4_ttl)); + +static void +action_avx512_ipv4_set_addrs(struct dp_packet_batch *batch, + const struct nlattr *a) +{ + a = nl_attr_get(a); + const struct ovs_key_ipv4 *key = nl_attr_get(a); + const struct ovs_key_ipv4 *mask = get_mask(a, struct ovs_key_ipv4); + struct dp_packet *packet; + ovs_be16 old_csum; + + __m256i v_key = _mm256_loadu_si256((void *) key); + __m256i v_mask = _mm256_loadu_si256((void *) mask); + + DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { + struct ip_header *nh = dp_packet_l3(packet); + old_csum = nh->ip_csum; + + __m256i v_packet = _mm256_loadu_si256((void *) nh); + + /* Shuffle key and mask to match ip_header struct layout. */ + static const uint8_t ip_shuffle_mask[32] = { + 0xFF, 5, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 6, 0xFF, 0xFF, 0xFF, 0, 1, 2, 3, + 0, 1, 2, 3, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF}; + __m256i v_shuf32 = _mm256_setr_epi32(0x0, 0x2, 0xF, 0xF, + 0x1, 0xF, 0xF, 0xF); + + __m256i v_shuffle = _mm256_loadu_si256((void *) ip_shuffle_mask); + + /* Two shuffles are required for key and mask to match the layout of + * the ip_header struct. The _shuffle_epi8 only works within 128-bit + * lanes, so a permute is required to move src and dst into the correct + * lanes. And then a shuffle is used to move the fields into the right + * order. + */ + __m256i v_key_shuf = _mm256_permutexvar_epi32(v_shuf32, v_key); + v_key_shuf = _mm256_shuffle_epi8(v_key_shuf, v_shuffle); + + __m256i v_mask_shuf = _mm256_permutexvar_epi32(v_shuf32, v_mask); + v_mask_shuf = _mm256_shuffle_epi8(v_mask_shuf, v_shuffle); + + __m256i v_pkt_masked = _mm256_andnot_si256(v_mask_shuf, v_packet); + __m256i v_res = _mm256_or_si256(v_key_shuf, v_pkt_masked); + + /* Recalculate the ip_csum based on updated values. */ + uint16_t checksum = avx512_ipv4_recalc_csum(v_res); + + /* Insert new checksum. */ + v_res = _mm256_insert_epi16(v_res, checksum, 5); + + /* If ip_src or ip_dst has been modified, L4 checksum needs to + * be updated too. */ + int update_mask = _mm256_movemask_epi8(v_mask); + if (update_mask & 0xFF) { + + uint16_t tmp_checksum = avx512_l4_update_csum(nh, v_res); + tmp_checksum = ~tmp_checksum; + uint16_t csum; + + if (nh->ip_proto == IPPROTO_UDP) { + /* New UDP checksum. */ + struct udp_header *uh = dp_packet_l4(packet); + if (!uh->udp_csum) { + uh->udp_csum = htons(0xffff); + } else { + uint16_t old_udp_checksum = ~uh->udp_csum; + + uint32_t udp_checksum = old_csum + tmp_checksum; + udp_checksum = csum_finish(udp_checksum); + uint16_t udp_csum = ~udp_checksum; + + uint32_t nw_udp_checksum = udp_csum + old_udp_checksum; + + csum = csum_finish(nw_udp_checksum); + + /* Insert new udp checksum. */ + v_res = _mm256_insert_epi16(v_res, csum, 13); + } + } else if (nh->ip_proto == IPPROTO_TCP) { + /* New TCP checksum. */ + struct tcp_header *th = dp_packet_l4(packet); + uint16_t old_tcp_checksum = ~th->tcp_csum; + + uint32_t tcp_checksum = old_csum + tmp_checksum; + tcp_checksum = csum_finish(tcp_checksum); + uint16_t tcp_csum = ~tcp_checksum; + + uint32_t nw_tcp_checksum = tcp_csum + old_tcp_checksum; + + csum = csum_finish(nw_tcp_checksum); + + th->tcp_csum = csum; + } + } + /* Store new IP header. */ + _mm256_storeu_si256((void *) nh, v_res); + } +} + static void action_avx512_set_masked(struct dp_packet_batch *batch OVS_UNUSED, const struct nlattr *a) @@ -244,6 +452,8 @@ action_avx512_init(struct odp_execute_action_impl *self) * are identified by OVS_KEY_ATTR_*. */ self->set_masked_funcs[OVS_KEY_ATTR_ETHERNET] = action_avx512_eth_set_addrs; + self->set_masked_funcs[OVS_KEY_ATTR_IPV4] = + action_avx512_ipv4_set_addrs; avx512_impl = *self; return 0; diff --git a/lib/odp-execute-private.c b/lib/odp-execute-private.c index e2d650779..763c8afc7 100644 --- a/lib/odp-execute-private.c +++ b/lib/odp-execute-private.c @@ -259,6 +259,7 @@ action_autoval_init(struct odp_execute_action_impl *self) * requires further processing for action type. Note that 2nd level items * are identified by OVS_KEY_ATTR_*. */ self->set_masked_funcs[OVS_KEY_ATTR_ETHERNET] = action_autoval_generic; + self->set_masked_funcs[OVS_KEY_ATTR_IPV4] = action_autoval_generic; autoval_impl = *self; return 0; diff --git a/lib/odp-execute.c b/lib/odp-execute.c index db6e1ec03..8da009ce9 100644 --- a/lib/odp-execute.c +++ b/lib/odp-execute.c @@ -585,11 +585,6 @@ odp_execute_masked_set_action(struct dp_packet *packet, break; } - case OVS_KEY_ATTR_IPV4: - odp_set_ipv4(packet, nl_attr_get(a), - get_mask(a, struct ovs_key_ipv4)); - break; - case OVS_KEY_ATTR_IPV6: odp_set_ipv6(packet, nl_attr_get(a), get_mask(a, struct ovs_key_ipv6)); @@ -664,6 +659,7 @@ odp_execute_masked_set_action(struct dp_packet *packet, case __OVS_KEY_ATTR_MAX: /* The following action types are handled by the scalar implementation. */ case OVS_KEY_ATTR_ETHERNET: + case OVS_KEY_ATTR_IPV4: default: OVS_NOT_REACHED(); } @@ -887,6 +883,18 @@ action_mod_eth(struct dp_packet_batch *batch, const struct nlattr *a) } } +static void +action_mod_ipv4(struct dp_packet_batch *batch, const struct nlattr *a) +{ + a = nl_attr_get(a); + struct dp_packet *packet; + + DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { + odp_set_ipv4(packet, nl_attr_get(a), + get_mask(a, struct ovs_key_ipv4)); + } +} + /* Implementation of the scalar actions impl init function. Build up the * array of func ptrs here. */ @@ -903,6 +911,7 @@ odp_action_scalar_init(struct odp_execute_action_impl *self) * requires further processing for action type. Note that 2nd level items * are identified by OVS_KEY_ATTR_*. */ self->set_masked_funcs[OVS_KEY_ATTR_ETHERNET] = action_mod_eth; + self->set_masked_funcs[OVS_KEY_ATTR_IPV4] = action_mod_ipv4; actions_active_impl = *self; return 0;