From patchwork Wed Jan 5 16:53:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Finn, Emma" X-Patchwork-Id: 1575771 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=JEwMFZ01; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4JTbDt61tlz9sSs for ; Thu, 6 Jan 2022 03:54:38 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 9D0836FB97; Wed, 5 Jan 2022 16:54:36 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pesTFvwACNZl; Wed, 5 Jan 2022 16:54:35 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id CDFDA6FBEC; Wed, 5 Jan 2022 16:54:28 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 80F7BC002F; Wed, 5 Jan 2022 16:54:28 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 244E3C0077 for ; Wed, 5 Jan 2022 16:54:27 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id A0A5782D14 for ; Wed, 5 Jan 2022 16:54:16 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=intel.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OO0jTnWe0pn0 for ; Wed, 5 Jan 2022 16:54:15 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by smtp1.osuosl.org (Postfix) with ESMTPS id 4CACD83112 for ; Wed, 5 Jan 2022 16:54:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1641401655; x=1672937655; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=uFKJR6ZpiU7HNdQTqlvh+CorKnwQoslZbytmVl50YEU=; b=JEwMFZ01YvTchGw7wu2RA9wDVL3yDzHRIu1dq+m4H+X1K5lCCBBf/gLE w181ofG/rlQCUsQT8CkAc32/3pFUmkFFRAfnhS5GGYXLaT6jxwNaBezZQ Xiz/lvhSBTeVx4i/jyBbTREYUGEEcceyik1d3VCqsnJJavox2uImDrvg1 K+IIVCQZ6pU8wU8PD4dUiyK6kESyxYJ6RyuQ0haKXojvFWR+gW60Hm64v Kc07SSI1ed3bUfkcIExo2iQsF8v8NnXFayaeZmQ8+NMDFCM5E5pcOwoFx 2UvXwoaEB4JcmAw7gnBrvDA9WhqWhoEvjWVvkZI4pcMAfLhIP0WBCrYNy g==; X-IronPort-AV: E=McAfee;i="6200,9189,10217"; a="242688544" X-IronPort-AV: E=Sophos;i="5.88,264,1635231600"; d="scan'208";a="242688544" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga103.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Jan 2022 08:54:15 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,264,1635231600"; d="scan'208";a="611525921" Received: from silpixa00400899.ir.intel.com ([10.243.23.110]) by FMSMGA003.fm.intel.com with ESMTP; 05 Jan 2022 08:54:13 -0800 From: Emma Finn To: dev@openvswitch.org, harry.van.haaren@intel.com, kumar.amber@intel.com Date: Wed, 5 Jan 2022 16:53:49 +0000 Message-Id: <20220105165349.3447695-10-emma.finn@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220105165349.3447695-1-emma.finn@intel.com> References: <20211203153301.37692-1-emma.finn@intel.com> <20220105165349.3447695-1-emma.finn@intel.com> MIME-Version: 1.0 Subject: [ovs-dev] [PATCH v4 9/9] odp-execute: Add ISA implementation of push_vlan action. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This commit adds the AVX512 implementation of the push_vlan action. The implementation here is auto-validated by the miniflow extract autovalidator, hence its correctness can be easily tested and verified. Signed-off-by: Emma Finn --- NEWS | 1 + lib/odp-execute-avx512.c | 74 ++++++++++++++++++++++++++++++++++----- lib/odp-execute-private.c | 1 + lib/odp-execute.c | 24 ++++++++----- 4 files changed, 83 insertions(+), 17 deletions(-) diff --git a/NEWS b/NEWS index f5032bdd0..af1759685 100644 --- a/NEWS +++ b/NEWS @@ -22,6 +22,7 @@ Post-v2.16.0 actions implementation at build time. * Add AVX512 implementation of actions. * Add support for an AVX512 optimized version of pop_vlan action. + * Add support for an AVX512 optimized version of push_vlan action. - Python: * For SSL support, the use of the pyOpenSSL library has been replaced with the native 'ssl' module. diff --git a/lib/odp-execute-avx512.c b/lib/odp-execute-avx512.c index 8bbfd5203..3d850026a 100644 --- a/lib/odp-execute-avx512.c +++ b/lib/odp-execute-avx512.c @@ -39,10 +39,22 @@ BUILD_ASSERT_DECL(offsetof(struct dp_packet, l3_ofs) + MEMBER_SIZEOF(struct dp_packet, l3_ofs) == offsetof(struct dp_packet, l4_ofs)); +enum OPERATION { + OP_ADD, + OP_SUB, +}; + static inline void ALWAYS_INLINE -avx512_dp_packet_resize_l2(struct dp_packet *b, int increment) +avx512_dp_packet_resize_l2(struct dp_packet *b, int increment, + enum OPERATION op) { /* update packet size/data pointers */ + if (increment >= 0) { + dp_packet_prealloc_headroom(b, increment); + } else { + ovs_assert(dp_packet_size(b) - dp_packet_l2_pad_size(b) >= -increment); + } + dp_packet_set_data(b, (char *) dp_packet_data(b) - increment); dp_packet_set_size(b, dp_packet_size(b) + increment); @@ -50,9 +62,9 @@ avx512_dp_packet_resize_l2(struct dp_packet *b, int increment) const __m128i v_zeros = _mm_setzero_si128(); const __m128i v_u16_max = _mm_cmpeq_epi16(v_zeros, v_zeros); - /* Only these lanes can be incremented for push-VLAN action. */ + /* Only these lanes can be incremented/decremented for L2. */ const uint8_t k_lanes = 0b1110; - __m128i v_offset = _mm_set1_epi16(VLAN_HEADER_LEN); + __m128i v_offset = _mm_set1_epi16(abs(increment)); /* Load packet and compare with UINT16_MAX */ void *adjust_ptr = &b->l2_pad_size; @@ -60,9 +72,19 @@ avx512_dp_packet_resize_l2(struct dp_packet *b, int increment) __mmask8 k_cmp = _mm_mask_cmpneq_epu16_mask(k_lanes, v_adjust_src, v_u16_max); - /* Add VLAN_HEADER_LEN using compare mask, store results. */ - __m128i v_adjust_wip = _mm_mask_sub_epi16(v_adjust_src, k_cmp, - v_adjust_src, v_offset); + /* Update VLAN_HEADER_LEN using compare mask, store results. */ + __m128i v_adjust_wip; + switch (op) { + case OP_ADD: + v_adjust_wip = _mm_mask_add_epi16(v_adjust_src, k_cmp, + v_adjust_src, v_offset); + break; + case OP_SUB: + v_adjust_wip = _mm_mask_sub_epi16(v_adjust_src, k_cmp, + v_adjust_src, v_offset); + break; + } + _mm_storeu_si128(adjust_ptr, v_adjust_wip); } @@ -79,8 +101,7 @@ avx512_eth_pop_vlan(struct dp_packet *packet) __m128i v_realign = _mm_alignr_epi8(v_ether, _mm_setzero_si128(), 16 - VLAN_HEADER_LEN); _mm_storeu_si128((void *) veh, v_realign); - avx512_dp_packet_resize_l2(packet, -VLAN_HEADER_LEN); - + avx512_dp_packet_resize_l2(packet, -VLAN_HEADER_LEN, OP_SUB); } } @@ -96,6 +117,42 @@ action_avx512_pop_vlan(void *dp OVS_UNUSED, struct dp_packet_batch *batch, } } +static inline void ALWAYS_INLINE +avx512_eth_push_vlan(struct dp_packet *packet, ovs_be16 tpid, ovs_be16 tci) +{ + avx512_dp_packet_resize_l2(packet, VLAN_HEADER_LEN, OP_ADD); + + /* Build up the VLAN TCI/TPID, and merge with the moving of Ether. */ + char *new_pkt_data = (char *) dp_packet_data(packet); + + const uint16_t tci_proc = tci & htons(~VLAN_CFI); + const uint32_t tpid_tci = (tci_proc << 16) | tpid; + __m128i v_ether = _mm_loadu_si128((void *) new_pkt_data); + + static const uint8_t vlan_push_shuffle_mask[16] = { + 4, 5, 6, 7, 8, 9, 10, 11, + 12, 13, 14, 15, 0xFF, 0xFF, 0xFF, 0xFF + }; + + __m128i v_index = _mm_loadu_si128((void *) vlan_push_shuffle_mask); + __m128i v_shift = _mm_shuffle_epi8(v_ether, v_index); + __m128i v_vlan_hdr = _mm_insert_epi32(v_shift, tpid_tci, 3); + _mm_storeu_si128((void *) new_pkt_data, v_vlan_hdr); +} + +static void +action_avx512_push_vlan(void *dp OVS_UNUSED, struct dp_packet_batch *batch, + const struct nlattr *a, + bool should_steal OVS_UNUSED) +{ + struct dp_packet *packet; + const struct ovs_action_push_vlan *vlan = nl_attr_get(a); + + DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { + avx512_eth_push_vlan(packet, vlan->vlan_tpid, vlan->vlan_tci); + } +} + /* Probe functions to check ISA requirements. */ static int32_t avx512_isa_probe(uint32_t needs_vbmi) @@ -136,6 +193,7 @@ action_avx512_init(struct odp_execute_action_impl *self) { avx512_isa_probe(0); self->funcs[OVS_ACTION_ATTR_POP_VLAN] = action_avx512_pop_vlan; + self->funcs[OVS_ACTION_ATTR_PUSH_VLAN] = action_avx512_push_vlan; return 0; } diff --git a/lib/odp-execute-private.c b/lib/odp-execute-private.c index 7c58d90d2..0bff66b50 100644 --- a/lib/odp-execute-private.c +++ b/lib/odp-execute-private.c @@ -218,6 +218,7 @@ int32_t action_autoval_init(struct odp_execute_action_impl *self) { self->funcs[OVS_ACTION_ATTR_POP_VLAN] = action_autoval_generic; + self->funcs[OVS_ACTION_ATTR_PUSH_VLAN] = action_autoval_generic; return 0; } diff --git a/lib/odp-execute.c b/lib/odp-execute.c index 1bc9fae09..40f71fa96 100644 --- a/lib/odp-execute.c +++ b/lib/odp-execute.c @@ -842,6 +842,19 @@ action_pop_vlan(void *dp OVS_UNUSED, struct dp_packet_batch *batch, } } +static void +action_push_vlan(void *dp OVS_UNUSED, struct dp_packet_batch *batch, + const struct nlattr *a OVS_UNUSED, + bool should_steal OVS_UNUSED) +{ + struct dp_packet *packet; + const struct ovs_action_push_vlan *vlan = nl_attr_get(a); + + DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { + eth_push_vlan(packet, vlan->vlan_tpid, vlan->vlan_tci); + } +} + /* Implementation of the scalar actions impl init function. Build up the * array of func ptrs here. */ @@ -849,6 +862,7 @@ int32_t odp_action_scalar_init(struct odp_execute_action_impl *self) { self->funcs[OVS_ACTION_ATTR_POP_VLAN] = action_pop_vlan; + self->funcs[OVS_ACTION_ATTR_PUSH_VLAN] = action_push_vlan; return 0; } @@ -991,15 +1005,6 @@ odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, break; } - case OVS_ACTION_ATTR_PUSH_VLAN: { - const struct ovs_action_push_vlan *vlan = nl_attr_get(a); - - DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { - eth_push_vlan(packet, vlan->vlan_tpid, vlan->vlan_tci); - } - break; - } - case OVS_ACTION_ATTR_PUSH_MPLS: { const struct ovs_action_push_mpls *mpls = nl_attr_get(a); @@ -1133,6 +1138,7 @@ odp_execute_actions(void *dp, struct dp_packet_batch *batch, bool steal, case OVS_ACTION_ATTR_OUTPUT: case OVS_ACTION_ATTR_LB_OUTPUT: case OVS_ACTION_ATTR_POP_VLAN: + case OVS_ACTION_ATTR_PUSH_VLAN: case OVS_ACTION_ATTR_TUNNEL_PUSH: case OVS_ACTION_ATTR_TUNNEL_POP: case OVS_ACTION_ATTR_USERSPACE: