From patchwork Mon Apr 19 16:30:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Noah Goldstein X-Patchwork-Id: 1468001 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=sourceware.org; envelope-from=libc-alpha-bounces@sourceware.org; receiver=) Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; secure) header.d=sourceware.org header.i=@sourceware.org header.a=rsa-sha256 header.s=default header.b=tGyr2JAG; dkim-atps=neutral Received: from sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FPC424xJdz9vFK for ; Tue, 20 Apr 2021 02:30:58 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 75D553959E55; Mon, 19 Apr 2021 16:30:56 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 75D553959E55 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1618849856; bh=oGzWgiWHuB6IuWratrSG4i/wvJc806FHajQ5J2E4f3E=; h=To:Subject:Date:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=tGyr2JAGaQToNaInSx8oqIuSEn6OfsuOb6sUrGU0stW22NDbmcFqef69qfQQ7E9Zp XBt2uCFGTAkSvJBRe9Fyf1QZxiRVtJwrHx6xdJiMgT/icEuoXfHg8We9bPGHdCHSoS MomnnDUWJEIiIwy1kqSSr88GdRPfvefDlUFqrpoE= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-qt1-x82a.google.com (mail-qt1-x82a.google.com [IPv6:2607:f8b0:4864:20::82a]) by sourceware.org (Postfix) with ESMTPS id F237C3959CB8 for ; Mon, 19 Apr 2021 16:30:44 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org F237C3959CB8 Received: by mail-qt1-x82a.google.com with SMTP id 18so13112130qtz.6 for ; Mon, 19 Apr 2021 09:30:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=oGzWgiWHuB6IuWratrSG4i/wvJc806FHajQ5J2E4f3E=; b=kY+uuO7WMnH7NyxZtg5XGig6Sy3TCZokUBw3F8b4zsPkWRXNy3DjBZb3dKj6K/cFzz e19D+E4AEnsJM7j0zRKYh4BQY1ZDPIdcB/3ub3ha6RjXR3JN7WDttQ16VmT5qxQ+dPl3 LkIWYuHCyuE6+7+DzRn7ZD4/51EDw774Qln/TOYdpGY1AftQ9os810/Wi2wrqaLAB4QB u6qwfR7a5lRZw0HkEWDx1OFLNNcT0C/dXpnFx3tkHZnB6P3JlrU7HpMHUMoux0QDMvdb kzuE7yHu/6KpMXtPS3U7sW+1+d4mAV4+QwwOf6WCav4A0VGowGUqut3ureqd3mVzq0Qf BbyA== X-Gm-Message-State: AOAM533rssg6sILjq4/ECKtzJSisUax6Dm6ZsJe5SeBwTDZH8GdrjnwE bJ81bLcWcTvs8QaGeirV9GdQUujmsDs= X-Google-Smtp-Source: ABdhPJxPOoRWB3vM+EWvypCfefhzBEEw1f1MhT9t/Fnwj1VCwUG7JuRG25J8/1/MZJRfH40HPNI2+g== X-Received: by 2002:ac8:44d9:: with SMTP id b25mr12116242qto.28.1618849844272; Mon, 19 Apr 2021 09:30:44 -0700 (PDT) Received: from localhost.localdomain (pool-71-245-178-39.pitbpa.fios.verizon.net. [71.245.178.39]) by smtp.googlemail.com with ESMTPSA id m2sm10115505qkc.14.2021.04.19.09.30.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Apr 2021 09:30:44 -0700 (PDT) To: libc-alpha@sourceware.org Subject: [PATCH v2 1/2] x86: Optimize less_vec evex and avx512 memset-vec-unaligned-erms.S Date: Mon, 19 Apr 2021 12:30:24 -0400 Message-Id: <20210419163025.2285675-1-goldstein.w.n@gmail.com> X-Mailer: git-send-email 2.29.2 MIME-Version: 1.0 X-Spam-Status: No, score=-12.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Noah Goldstein via Libc-alpha From: Noah Goldstein Reply-To: Noah Goldstein Errors-To: libc-alpha-bounces@sourceware.org Sender: "Libc-alpha" No bug. This commit adds optimized cased for less_vec memset case that uses the avx512vl/avx512bw mask store avoiding the excessive branches. test-memset and test-wmemset are passing. Signed-off-by: Noah Goldstein --- sysdeps/x86_64/multiarch/ifunc-memset.h | 6 ++- .../multiarch/memset-avx512-unaligned-erms.S | 2 +- .../multiarch/memset-evex-unaligned-erms.S | 2 +- .../multiarch/memset-vec-unaligned-erms.S | 52 +++++++++++++++---- 4 files changed, 47 insertions(+), 15 deletions(-) diff --git a/sysdeps/x86_64/multiarch/ifunc-memset.h b/sysdeps/x86_64/multiarch/ifunc-memset.h index 502f946a84..eda5640541 100644 --- a/sysdeps/x86_64/multiarch/ifunc-memset.h +++ b/sysdeps/x86_64/multiarch/ifunc-memset.h @@ -54,7 +54,8 @@ IFUNC_SELECTOR (void) && !CPU_FEATURES_ARCH_P (cpu_features, Prefer_No_AVX512)) { if (CPU_FEATURE_USABLE_P (cpu_features, AVX512VL) - && CPU_FEATURE_USABLE_P (cpu_features, AVX512BW)) + && CPU_FEATURE_USABLE_P (cpu_features, AVX512BW) + && CPU_FEATURE_USABLE_P (cpu_features, BMI2)) { if (CPU_FEATURE_USABLE_P (cpu_features, ERMS)) return OPTIMIZE (avx512_unaligned_erms); @@ -68,7 +69,8 @@ IFUNC_SELECTOR (void) if (CPU_FEATURE_USABLE_P (cpu_features, AVX2)) { if (CPU_FEATURE_USABLE_P (cpu_features, AVX512VL) - && CPU_FEATURE_USABLE_P (cpu_features, AVX512BW)) + && CPU_FEATURE_USABLE_P (cpu_features, AVX512BW) + && CPU_FEATURE_USABLE_P (cpu_features, BMI2)) { if (CPU_FEATURE_USABLE_P (cpu_features, ERMS)) return OPTIMIZE (evex_unaligned_erms); diff --git a/sysdeps/x86_64/multiarch/memset-avx512-unaligned-erms.S b/sysdeps/x86_64/multiarch/memset-avx512-unaligned-erms.S index 22e7b187c8..d03460be93 100644 --- a/sysdeps/x86_64/multiarch/memset-avx512-unaligned-erms.S +++ b/sysdeps/x86_64/multiarch/memset-avx512-unaligned-erms.S @@ -19,6 +19,6 @@ # define SECTION(p) p##.evex512 # define MEMSET_SYMBOL(p,s) p##_avx512_##s # define WMEMSET_SYMBOL(p,s) p##_avx512_##s - +# define USE_LESS_VEC_MASKMOV 1 # include "memset-vec-unaligned-erms.S" #endif diff --git a/sysdeps/x86_64/multiarch/memset-evex-unaligned-erms.S b/sysdeps/x86_64/multiarch/memset-evex-unaligned-erms.S index ae0a4d6e46..eb3541ef60 100644 --- a/sysdeps/x86_64/multiarch/memset-evex-unaligned-erms.S +++ b/sysdeps/x86_64/multiarch/memset-evex-unaligned-erms.S @@ -19,6 +19,6 @@ # define SECTION(p) p##.evex # define MEMSET_SYMBOL(p,s) p##_evex_##s # define WMEMSET_SYMBOL(p,s) p##_evex_##s - +# define USE_LESS_VEC_MASKMOV 1 # include "memset-vec-unaligned-erms.S" #endif diff --git a/sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S b/sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S index 584747f1a1..6b02e87f48 100644 --- a/sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S +++ b/sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S @@ -63,6 +63,9 @@ # endif #endif +#define PAGE_SIZE 4096 +#define LOG_PAGE_SIZE 12 + #ifndef SECTION # error SECTION is not defined! #endif @@ -213,11 +216,38 @@ L(loop): cmpq %rcx, %rdx jne L(loop) VZEROUPPER_SHORT_RETURN + + .p2align 4 L(less_vec): /* Less than 1 VEC. */ # if VEC_SIZE != 16 && VEC_SIZE != 32 && VEC_SIZE != 64 # error Unsupported VEC_SIZE! # endif +# ifdef USE_LESS_VEC_MASKMOV + /* Clear high bits from edi. Only keeping bits relevant to page + cross check. Using sall instead of andl saves 3 bytes. Note + that we are using rax which is set in + MEMSET_VDUP_TO_VEC0_AND_SET_RETURN as ptr from here on out. */ + sall $(32 - LOG_PAGE_SIZE), %edi + /* Check if VEC_SIZE load cross page. Mask loads suffer serious + performance degradation when it has to fault supress. */ + cmpl $((PAGE_SIZE - VEC_SIZE) << (32 - LOG_PAGE_SIZE)), %edi + ja L(cross_page) +# if VEC_SIZE > 32 + movq $-1, %rcx + bzhiq %rdx, %rcx, %rcx + kmovq %rcx, %k1 +# else + movl $-1, %ecx + bzhil %edx, %ecx, %ecx + kmovd %ecx, %k1 +# endif + vmovdqu8 %VEC(0), (%rax) {%k1} + VZEROUPPER_RETURN + + .p2align 4 +L(cross_page): +# endif # if VEC_SIZE > 32 cmpb $32, %dl jae L(between_32_63) @@ -234,36 +264,36 @@ L(less_vec): cmpb $1, %dl ja L(between_2_3) jb 1f - movb %cl, (%rdi) + movb %cl, (%rax) 1: VZEROUPPER_RETURN # if VEC_SIZE > 32 /* From 32 to 63. No branch when size == 32. */ L(between_32_63): - VMOVU %YMM0, -32(%rdi,%rdx) - VMOVU %YMM0, (%rdi) + VMOVU %YMM0, -32(%rax,%rdx) + VMOVU %YMM0, (%rax) VZEROUPPER_RETURN # endif # if VEC_SIZE > 16 /* From 16 to 31. No branch when size == 16. */ L(between_16_31): - VMOVU %XMM0, -16(%rdi,%rdx) - VMOVU %XMM0, (%rdi) + VMOVU %XMM0, -16(%rax,%rdx) + VMOVU %XMM0, (%rax) VZEROUPPER_RETURN # endif /* From 8 to 15. No branch when size == 8. */ L(between_8_15): - movq %rcx, -8(%rdi,%rdx) - movq %rcx, (%rdi) + movq %rcx, -8(%rax,%rdx) + movq %rcx, (%rax) VZEROUPPER_RETURN L(between_4_7): /* From 4 to 7. No branch when size == 4. */ - movl %ecx, -4(%rdi,%rdx) - movl %ecx, (%rdi) + movl %ecx, -4(%rax,%rdx) + movl %ecx, (%rax) VZEROUPPER_RETURN L(between_2_3): /* From 2 to 3. No branch when size == 2. */ - movw %cx, -2(%rdi,%rdx) - movw %cx, (%rdi) + movw %cx, -2(%rax,%rdx) + movw %cx, (%rax) VZEROUPPER_RETURN END (MEMSET_SYMBOL (__memset, unaligned_erms)) From patchwork Mon Apr 19 16:30:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Noah Goldstein X-Patchwork-Id: 1468000 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=8.43.85.97; helo=sourceware.org; envelope-from=libc-alpha-bounces@sourceware.org; receiver=) Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; secure) header.d=sourceware.org header.i=@sourceware.org header.a=rsa-sha256 header.s=default header.b=YRDjSwR3; dkim-atps=neutral Received: from sourceware.org (ip-8-43-85-97.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4FPC3w61zhz9vDx for ; Tue, 20 Apr 2021 02:30:52 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id E158F3959C2F; Mon, 19 Apr 2021 16:30:50 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org E158F3959C2F DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1618849850; bh=l8KD1Dg/i15zQMa9KC/JlAIgnqMBAQMlbAeTENg6zUg=; h=To:Subject:Date:In-Reply-To:References:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=YRDjSwR3TF8sFRHrM59aKfASUV0ck8bBRN3SSjZOJj61WXGgFZSrNshDFQ1hNThkh VQT/eUsk0a5IfBEtc/A/XSvjbpa+0nVqs8Hyg17DiB5+2RkVjo7TnzirnjRqwP48DB FWiXal3h4NMuwPCHMVRG9mJha+96ptcCiSFHjvDE= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-qk1-x732.google.com (mail-qk1-x732.google.com [IPv6:2607:f8b0:4864:20::732]) by sourceware.org (Postfix) with ESMTPS id 408453959CB9 for ; Mon, 19 Apr 2021 16:30:46 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org 408453959CB9 Received: by mail-qk1-x732.google.com with SMTP id y136so4288824qkb.1 for ; Mon, 19 Apr 2021 09:30:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=l8KD1Dg/i15zQMa9KC/JlAIgnqMBAQMlbAeTENg6zUg=; b=rXL7IGn6JwS/9AFmd77hy1mIShf3riSmcMhfxXFnSBLmksEkFYwPUelmYGFER83oK6 K3hAtBzxRSbp9NViKOi+vufLf8a7M9n9jo4ilQkDuCwATD2HPLHxmVC+p+IzFo4kdusa kVrr90ahVUUdkIwlY0KF+2DZ0kcSARORz8AchxHTMIcWT6yjMA9AZPcr3p8OSBePkod/ dagx444s3pFW0Lc/frF6pyEmICBLkjfMWfOZfv8ltzDDzAlLW6P2Mh8kVPJ2kmN5TcwS 6GT3F6stBlrmDpV3r+85XdL8zDG/D76kanvvL7BQkn/A/I24u22KHdc3IYKN52AethRe Faqw== X-Gm-Message-State: AOAM5309dsdgCdcrWQe/x4aA6dPdCgKYg+MH2qzS2t7v0JeUkIrm3KJ3 Trl4EBCzIRli5G2Rxx5m6Y1k+s8+kbE= X-Google-Smtp-Source: ABdhPJy96oKz9mepozEGXnDreC5AUDATqW/HUDvylOMP+myiQ1te7K6WZPe1JAEqx2FFyHqIFQb31g== X-Received: by 2002:a05:620a:7d1:: with SMTP id 17mr12872341qkb.403.1618849845691; Mon, 19 Apr 2021 09:30:45 -0700 (PDT) Received: from localhost.localdomain (pool-71-245-178-39.pitbpa.fios.verizon.net. [71.245.178.39]) by smtp.googlemail.com with ESMTPSA id m2sm10115505qkc.14.2021.04.19.09.30.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Apr 2021 09:30:45 -0700 (PDT) To: libc-alpha@sourceware.org Subject: [PATCH v2 2/2] x86: Expand test-memset.c and bench-memset.c Date: Mon, 19 Apr 2021 12:30:25 -0400 Message-Id: <20210419163025.2285675-2-goldstein.w.n@gmail.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210419163025.2285675-1-goldstein.w.n@gmail.com> References: <20210419163025.2285675-1-goldstein.w.n@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-12.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Noah Goldstein via Libc-alpha From: Noah Goldstein Reply-To: Noah Goldstein Errors-To: libc-alpha-bounces@sourceware.org Sender: "Libc-alpha" No bug. This commit adds tests cases and benchmarks for page cross and for memset to the end of the page without crossing. As well in test-memset.c this commit adds sentinel on start/end of tstbuf to test for overwrites Signed-off-by: Noah Goldstein --- benchtests/bench-memset.c | 6 ++++-- string/test-memset.c | 20 +++++++++++++++----- 2 files changed, 19 insertions(+), 7 deletions(-) diff --git a/benchtests/bench-memset.c b/benchtests/bench-memset.c index 1174900e88..d6619b4836 100644 --- a/benchtests/bench-memset.c +++ b/benchtests/bench-memset.c @@ -61,7 +61,7 @@ do_one_test (json_ctx_t *json_ctx, impl_t *impl, CHAR *s, static void do_test (json_ctx_t *json_ctx, size_t align, int c, size_t len) { - align &= 63; + align &= 4095; if ((align + len) * sizeof (CHAR) > page_size) return; @@ -110,9 +110,11 @@ test_main (void) { for (i = 0; i < 18; ++i) do_test (&json_ctx, 0, c, 1 << i); - for (i = 1; i < 32; ++i) + for (i = 1; i < 64; ++i) { do_test (&json_ctx, i, c, i); + do_test (&json_ctx, 4096 - i, c, i); + do_test (&json_ctx, 4095, c, i); if (i & (i - 1)) do_test (&json_ctx, 0, c, i); } diff --git a/string/test-memset.c b/string/test-memset.c index eb71517390..82bfcd6ad4 100644 --- a/string/test-memset.c +++ b/string/test-memset.c @@ -109,16 +109,24 @@ SIMPLE_MEMSET (CHAR *s, int c, size_t n) static void do_one_test (impl_t *impl, CHAR *s, int c __attribute ((unused)), size_t n) { - CHAR tstbuf[n]; + CHAR buf[n + 2]; + CHAR *tstbuf = buf + 1; + CHAR sentinel = c - 1; + buf[0] = sentinel; + buf[n + 1] = sentinel; #ifdef TEST_BZERO simple_bzero (tstbuf, n); CALL (impl, s, n); - if (memcmp (s, tstbuf, n) != 0) + if (memcmp (s, tstbuf, n) != 0 + || buf[0] != sentinel + || buf[n + 1] != sentinel) #else CHAR *res = CALL (impl, s, c, n); if (res != s || SIMPLE_MEMSET (tstbuf, c, n) != tstbuf - || MEMCMP (s, tstbuf, n) != 0) + || MEMCMP (s, tstbuf, n) != 0 + || buf[0] != sentinel + || buf[n + 1] != sentinel) #endif /* !TEST_BZERO */ { error (0, 0, "Wrong result in function %s", impl->name); @@ -130,7 +138,7 @@ do_one_test (impl_t *impl, CHAR *s, int c __attribute ((unused)), size_t n) static void do_test (size_t align, int c, size_t len) { - align &= 7; + align &= 4095; if ((align + len) * sizeof (CHAR) > page_size) return; @@ -245,9 +253,11 @@ test_main (void) { for (i = 0; i < 18; ++i) do_test (0, c, 1 << i); - for (i = 1; i < 32; ++i) + for (i = 1; i < 64; ++i) { do_test (i, c, i); + do_test (4096 - i, c, i); + do_test (4095, c, i); if (i & (i - 1)) do_test (0, c, i); }