From patchwork Tue Jun 14 17:57:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Noah Goldstein X-Patchwork-Id: 1643377 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (1024-bit key; secure) header.d=sourceware.org header.i=@sourceware.org header.a=rsa-sha256 header.s=default header.b=LnGmujw5; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=8.43.85.97; helo=sourceware.org; envelope-from=libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org; receiver=) Received: from sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LMx4B5hdPz9tkj for ; Wed, 15 Jun 2022 03:58:02 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 00241385276A for ; Tue, 14 Jun 2022 17:57:57 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 00241385276A DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1655229478; bh=LkC/PlJPU4O6GhRDcXsZvsj5s5bDZMv2yPkknXR8rLM=; h=To:Subject:Date:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=LnGmujw5ouC6/PiWJWWP91ZnccRlNHEBnJr71ch5Qzt9f5g1heA6k1VdMcgbtm/gS ca1mZC3t/e7AU8TffQfD2oCBbxhQAWqoeToJhWMWUiSdEUK706cIsVRQBH0Qg8bkE3 Ub5YRwr+gwcpW9bt6hT4FE6z+YewV8iAYoFZUhzE= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pj1-x1035.google.com (mail-pj1-x1035.google.com [IPv6:2607:f8b0:4864:20::1035]) by sourceware.org (Postfix) with ESMTPS id 307F53856DC7 for ; Tue, 14 Jun 2022 17:57:41 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 307F53856DC7 Received: by mail-pj1-x1035.google.com with SMTP id cx11so9127309pjb.1 for ; Tue, 14 Jun 2022 10:57:41 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=LkC/PlJPU4O6GhRDcXsZvsj5s5bDZMv2yPkknXR8rLM=; b=ue9MaWhhFHuUAMpG9ijXKwC1mOr+IHXT8owRBjgL2li8JdiIzmZSnab3iEJoRgKQQx 8BgRBpJsKKVkXowH84IZ73LRmZrCb3bqYtKYzf1FMzrY+e8414KaBT9eZdps7fDRZN8s SV8O1ex1FtEm3pgvsTF7umqOGo/Zf6tOKaKuKT2aQkIXhjeE8kPUhZUZClRUTU2D87Az 6wLLg882KNyCpqJRpGiJTpNH6N+rjXuZAn609QrvEU989HnYnRNhpZcmAgwG0xmBdrEt lEN1ECvxoVgzF3YTK+c6yokKyl3w9k5+1sXD2i620DhBqyiD94ypLPRRsy4vHdINKjqM iSkw== X-Gm-Message-State: AJIora/aikQpElMB8v0wx3nVxt5VEUEsc0oYYvFIPg7fyUHPAATNhC+E NTVm/eHEJzmjb4LvEVxoIMuyPhHDSME= X-Google-Smtp-Source: AGRyM1u9i4VI0znXHXtpaw1prlYDsXMPO217Ssf140mrGGcFGVvp9ilrd8baJsb/ocqvPiQv4/YgMg== X-Received: by 2002:a17:90b:4b88:b0:1e2:d504:fb22 with SMTP id lr8-20020a17090b4b8800b001e2d504fb22mr5757294pjb.97.1655229459373; Tue, 14 Jun 2022 10:57:39 -0700 (PDT) Received: from noah-tgl.. ([192.55.60.47]) by smtp.gmail.com with ESMTPSA id cd17-20020a056a00421100b0051c72bb78d7sm7799091pfb.26.2022.06.14.10.57.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 Jun 2022 10:57:39 -0700 (PDT) To: libc-alpha@sourceware.org Subject: [PATCH v1] x86: Fix incorrect tunable logic for memmove-vec-unaligned-erms. Date: Tue, 14 Jun 2022 10:57:36 -0700 Message-Id: <20220614175737.2616508-1-goldstein.w.n@gmail.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Spam-Status: No, score=-12.0 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Noah Goldstein via Libc-alpha From: Noah Goldstein Reply-To: Noah Goldstein Errors-To: libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org Sender: "Libc-alpha" Move the setting of `rep_movsb_stop_threshold` to after the tunables have been collected so that the `rep_movsb_stop_threshold` (which is used to redirect control flow to the non_temporal case) will use any user value for `non_temporal_threshold` (from glibc.cpu.x86_non_temporal_threshold) As well, use the proper bound in non_tempral case in 'memmove-vec-unaligned-erms' when entering from size being above `rep_movsb_stop_threshold`. --- sysdeps/x86/dl-cacheinfo.h | 24 +++++++++---------- .../multiarch/memmove-vec-unaligned-erms.S | 2 +- 2 files changed, 13 insertions(+), 13 deletions(-) diff --git a/sysdeps/x86/dl-cacheinfo.h b/sysdeps/x86/dl-cacheinfo.h index f64a2fb0ba..cc3b840f9c 100644 --- a/sysdeps/x86/dl-cacheinfo.h +++ b/sysdeps/x86/dl-cacheinfo.h @@ -898,18 +898,6 @@ dl_init_cacheinfo (struct cpu_features *cpu_features) if (CPU_FEATURE_USABLE_P (cpu_features, FSRM)) rep_movsb_threshold = 2112; - unsigned long int rep_movsb_stop_threshold; - /* ERMS feature is implemented from AMD Zen3 architecture and it is - performing poorly for data above L2 cache size. Henceforth, adding - an upper bound threshold parameter to limit the usage of Enhanced - REP MOVSB operations and setting its value to L2 cache size. */ - if (cpu_features->basic.kind == arch_kind_amd) - rep_movsb_stop_threshold = core; - /* Setting the upper bound of ERMS to the computed value of - non-temporal threshold for architectures other than AMD. */ - else - rep_movsb_stop_threshold = non_temporal_threshold; - /* The default threshold to use Enhanced REP STOSB. */ unsigned long int rep_stosb_threshold = 2048; @@ -951,6 +939,18 @@ dl_init_cacheinfo (struct cpu_features *cpu_features) SIZE_MAX); #endif + unsigned long int rep_movsb_stop_threshold; + /* ERMS feature is implemented from AMD Zen3 architecture and it is + performing poorly for data above L2 cache size. Henceforth, adding + an upper bound threshold parameter to limit the usage of Enhanced + REP MOVSB operations and setting its value to L2 cache size. */ + if (cpu_features->basic.kind == arch_kind_amd) + rep_movsb_stop_threshold = core; + /* Setting the upper bound of ERMS to the computed value of + non-temporal threshold for architectures other than AMD. */ + else + rep_movsb_stop_threshold = non_temporal_threshold; + cpu_features->data_cache_size = data; cpu_features->shared_cache_size = shared; cpu_features->non_temporal_threshold = non_temporal_threshold; diff --git a/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S b/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S index af51177d5d..6d93b7c690 100644 --- a/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S +++ b/sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S @@ -724,7 +724,7 @@ L(skip_short_movsb_check): .p2align 4,, 10 #if (defined USE_MULTIARCH || VEC_SIZE == 16) && IS_IN (libc) L(large_memcpy_2x_check): - cmp __x86_rep_movsb_threshold(%rip), %RDX_LP + cmp __x86_shared_non_temporal_threshold(%rip), %RDX_LP jb L(more_8x_vec_check) L(large_memcpy_2x): /* To reach this point it is impossible for dst > src and