From patchwork Tue May 10 14:21:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Finn, Emma" X-Patchwork-Id: 1629193 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=bmg3dMmQ; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KyKzf1s2Gz9sCq for ; Wed, 11 May 2022 00:24:14 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 7AA9F82F57; Tue, 10 May 2022 14:24:12 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FlVWC_EWcvxu; Tue, 10 May 2022 14:24:11 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTPS id 3EFF282EDB; Tue, 10 May 2022 14:24:10 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id EF660C0032; Tue, 10 May 2022 14:24:09 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 09544C0032 for ; Tue, 10 May 2022 14:24:09 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id A55FC416F4 for ; Tue, 10 May 2022 14:23:26 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=intel.com Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CQReeh_bRzrO for ; Tue, 10 May 2022 14:23:25 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by smtp4.osuosl.org (Postfix) with ESMTPS id 0A70141762 for ; Tue, 10 May 2022 14:23:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1652192604; x=1683728604; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=maBAFBzIttNNAIrY0tnFvrNhGBPwSvqvf2O44nZEZXA=; b=bmg3dMmQT7Wq35x9t7B5w0QYLlfMtLrLISqOaaTyyjb7OOLfbDv+W4la ygKJuR8AGgZivUCBe1GqnlDLvtcFGWXam00qLqd0AD7tnpggnNR4l83rm 0jO4Pv9rFK7UuFdLmZqmxiK7UtMRTEP/KtEp7QudYUJ5gXv9EjyRnT24p oqZk2Nf+yFyFs69oEEeSOxWEaYbJaQLQKH7kAt9DJn1548OzRn+bCvNwF mv4WWxy5NkzQYk22w86FJ5XximVdOxtQOqjYWGyXEMmpaUC1EH+05NURx 6h1SU15wdEYYUvmMz3W8dQbCm1JJX6ZPZwJm36OZHzcbTC4GGvI0OPYdf w==; X-IronPort-AV: E=McAfee;i="6400,9594,10342"; a="266975939" X-IronPort-AV: E=Sophos;i="5.91,214,1647327600"; d="scan'208";a="266975939" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 May 2022 07:23:23 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,214,1647327600"; d="scan'208";a="541797025" Received: from silpixa00400899.ir.intel.com ([10.243.22.42]) by orsmga006.jf.intel.com with ESMTP; 10 May 2022 07:23:21 -0700 From: Emma Finn To: echaudro@redhat.com, harry.van.haaren@intel.com, kumar.amber@intel.com, ian.stokes@intel.com, dev@openvswitch.org Date: Tue, 10 May 2022 14:21:59 +0000 Message-Id: <20220510142202.1087967-9-emma.finn@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220510142202.1087967-1-emma.finn@intel.com> References: <20220112094244.81402-1-emma.finn@intel.com> <20220510142202.1087967-1-emma.finn@intel.com> MIME-Version: 1.0 Subject: [ovs-dev] [v6 08/11] odp-execute: Add ISA implementation of pop_vlan action. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This commit adds the AVX512 implementation of the pop_vlan action. Signed-off-by: Emma Finn --- lib/odp-execute-avx512.c | 73 ++++++++++++++++++++++++++++++++++++++- lib/odp-execute-private.c | 2 +- lib/odp-execute-private.h | 2 +- 3 files changed, 74 insertions(+), 3 deletions(-) diff --git a/lib/odp-execute-avx512.c b/lib/odp-execute-avx512.c index 84f68d378..637956236 100644 --- a/lib/odp-execute-avx512.c +++ b/lib/odp-execute-avx512.c @@ -14,6 +14,11 @@ * limitations under the License. */ +#ifdef __x86_64__ +/* Sparse cannot handle the AVX512 instructions. */ +#if !defined(__CHECKER__) + + #include #include @@ -24,6 +29,67 @@ #include "odp-netlink.h" #include "openvswitch/vlog.h" +VLOG_DEFINE_THIS_MODULE(odp_execute_avx512); +BUILD_ASSERT_DECL(offsetof(struct dp_packet, l2_5_ofs) + + MEMBER_SIZEOF(struct dp_packet, l2_5_ofs) == + offsetof(struct dp_packet, l3_ofs)); + +BUILD_ASSERT_DECL(offsetof(struct dp_packet, l3_ofs) + + MEMBER_SIZEOF(struct dp_packet, l3_ofs) == + offsetof(struct dp_packet, l4_ofs)); + +static inline void ALWAYS_INLINE +avx512_dp_packet_resize_l2(struct dp_packet *b, int resize_by_bytes) +{ + /* Update packet size/data pointers */ + dp_packet_set_data(b, (char *) dp_packet_data(b) - resize_by_bytes); + dp_packet_set_size(b, dp_packet_size(b) + resize_by_bytes); + + /* Increment u16 packet offset values */ + const __m128i v_zeros = _mm_setzero_si128(); + const __m128i v_u16_max = _mm_cmpeq_epi16(v_zeros, v_zeros); + + /* Only these lanes can be incremented/decremented for L2. */ + const uint8_t k_lanes = 0b1110; + __m128i v_offset = _mm_set1_epi16(VLAN_HEADER_LEN); + + /* Load packet and compare with UINT16_MAX */ + void *adjust_ptr = &b->l2_pad_size; + __m128i v_adjust_src = _mm_loadu_si128(adjust_ptr); + + /* Generate K mask to use for updating offset values of + * the packet buffer. */ + __mmask8 k_cmp = _mm_mask_cmpneq_epu16_mask(k_lanes, v_adjust_src, + v_u16_max); + + /* Update VLAN_HEADER_LEN using compare mask, store results. */ + __m128i v_adjust_wip = _mm_mask_sub_epi16(v_adjust_src, k_cmp, + v_adjust_src, v_offset); + _mm_storeu_si128(adjust_ptr, v_adjust_wip); +} + +static void +action_avx512_pop_vlan(void *dp OVS_UNUSED, struct dp_packet_batch *batch, + const struct nlattr *a OVS_UNUSED, + bool should_steal OVS_UNUSED) +{ + struct dp_packet *packet; + + DP_PACKET_BATCH_FOR_EACH (i, packet, batch) { + struct vlan_eth_header *veh = dp_packet_eth(packet); + + if (veh && dp_packet_size(packet) >= sizeof *veh && + eth_type_vlan(veh->veth_type)) { + + __m128i v_ether = _mm_loadu_si128((void *) veh); + __m128i v_realign = _mm_alignr_epi8(v_ether, _mm_setzero_si128(), + 16 - VLAN_HEADER_LEN); + _mm_storeu_si128((void *) veh, v_realign); + avx512_dp_packet_resize_l2(packet, -VLAN_HEADER_LEN); + } + } +} + /* Probe functions to check ISA requirements. */ static int32_t avx512_isa_probe(uint32_t needs_vbmi) @@ -60,8 +126,13 @@ action_avx512_probe(void) int32_t -action_avx512_init(void) +action_avx512_init(struct odp_execute_action_impl *self) { avx512_isa_probe(0); + self->funcs[OVS_ACTION_ATTR_POP_VLAN] = action_avx512_pop_vlan; + return 0; } + +#endif +#endif diff --git a/lib/odp-execute-private.c b/lib/odp-execute-private.c index 2bfa84152..8257bba80 100644 --- a/lib/odp-execute-private.c +++ b/lib/odp-execute-private.c @@ -52,7 +52,7 @@ static struct odp_execute_action_impl action_impls[] = { .available = 1, .name = "avx512", .probe = action_avx512_probe, - .init_func = NULL, + .init_func = action_avx512_init, }, #endif }; diff --git a/lib/odp-execute-private.h b/lib/odp-execute-private.h index 13fc74e52..231d72492 100644 --- a/lib/odp-execute-private.h +++ b/lib/odp-execute-private.h @@ -100,7 +100,7 @@ int32_t odp_execute_action_set(const char *name, int32_t odp_action_scalar_init(struct odp_execute_action_impl *self); /* Init function for the optimized with AVX512 actions. */ -int32_t action_avx512_init(void); +int32_t action_avx512_init(struct odp_execute_action_impl *self); /* Probe function to check ISA requirements. */ int32_t action_avx512_probe(void);