From patchwork Wed Jun 29 23:07:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Noah Goldstein X-Patchwork-Id: 1650413 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (1024-bit key; secure) header.d=sourceware.org header.i=@sourceware.org header.a=rsa-sha256 header.s=default header.b=oPjt+anN; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=8.43.85.97; helo=sourceware.org; envelope-from=libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org; receiver=) Received: from sourceware.org (ip-8-43-85-97.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LYHDS3tMHz9s0w for ; Thu, 30 Jun 2022 09:07:36 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id D37123846473 for ; Wed, 29 Jun 2022 23:07:30 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org D37123846473 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1656544050; bh=wKyqMoXrZHrM8JPSiLwLmSc/EK9L3oeg9pCFQtYJu0M=; h=To:Subject:Date:In-Reply-To:References:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=oPjt+anNS56tXM5bYcSSPeThjtaGZ06IwtX13DDgwSQvn3n1hOhctKDwYt2uAwNQg MszGsVSCBh1Q2MKjfplOTW8bw6dnFd1c9r5jDUTQvCjrUDc8ABLvnh9QsIpnUKZ8Sq k944081YXpiqz7QfcK3hTbtDtx/dBHDO1MbwB7TA= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pl1-x62c.google.com (mail-pl1-x62c.google.com [IPv6:2607:f8b0:4864:20::62c]) by sourceware.org (Postfix) with ESMTPS id 6A5AD38485AE for ; Wed, 29 Jun 2022 23:07:15 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 6A5AD38485AE Received: by mail-pl1-x62c.google.com with SMTP id k14so15450227plh.4 for ; Wed, 29 Jun 2022 16:07:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wKyqMoXrZHrM8JPSiLwLmSc/EK9L3oeg9pCFQtYJu0M=; b=xPLTZjmkPaO3S9/9rQSlu4p5izeCVN5dpBpTcqliVwQGsvCG5cn+01TC7asN9SOpG9 9qcTYJzheDY3lFElwvJf84dchWWNEjttfh09NJszCeWuXbnLU3X+GLnLhzij5VOp0rsS 5YG8bYd/jyYwMKZ9jDp5C7rcQwAGPEWygj7HwpAA8Kcd2+1sil06wuey/RqqREXJ5yRt oUfjPRuUJcHHwQ4d1lCi/eXMXk93BzxRhT+Sdh9UBhi9qjpThN5PX1q77NHPmhISMn9k 5QTSeiogfzADWOugB+9DR9vxtTWsIWz2L3mg3xNYShuq1fjYdjLqymw+lyc1mbwAhrAg NZsA== X-Gm-Message-State: AJIora85TBovTw6Sz2GYHrYCZfHqcaXkMnnFN6uuqtRc6VyNee69Y6se g0pYdCR7Hud4/GxY6XSYLyBlPJc4880= X-Google-Smtp-Source: AGRyM1uL7MXlwaR8bbrX1KAUi9O/HIlcOFyvBWXBnZ1ml0Fa+/NLw+9V/nqE7W8OdUcMv8Kwn54HtA== X-Received: by 2002:a17:902:f683:b0:16a:17d0:8a6a with SMTP id l3-20020a170902f68300b0016a17d08a6amr12803122plg.60.1656544034203; Wed, 29 Jun 2022 16:07:14 -0700 (PDT) Received: from noah-tgl.. ([192.55.60.43]) by smtp.gmail.com with ESMTPSA id h13-20020aa79f4d000000b00525601ded18sm12370101pfr.92.2022.06.29.16.07.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 Jun 2022 16:07:13 -0700 (PDT) To: libc-alpha@sourceware.org Subject: [PATCH v3 2/3] x86: Move and slightly improve memset_erms Date: Wed, 29 Jun 2022 16:07:05 -0700 Message-Id: <20220629230706.1264225-2-goldstein.w.n@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20220629230706.1264225-1-goldstein.w.n@gmail.com> References: <20220628152735.17863-1-goldstein.w.n@gmail.com> <20220629230706.1264225-1-goldstein.w.n@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-12.1 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Noah Goldstein via Libc-alpha From: Noah Goldstein Reply-To: Noah Goldstein Errors-To: libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org Sender: "Libc-alpha" Implementation wise: 1. Remove the VZEROUPPER as memset_{impl}_unaligned_erms does not use the L(stosb) label that was previously defined. 2. Don't give the hotpath (fallthrough) to zero size. Code positioning wise: Move memset_{chk}_erms to its own file. Leaving it in between the memset_{impl}_unaligned both adds unnecessary complexity to the file and wastes space in a relatively hot cache section. --- sysdeps/x86_64/multiarch/Makefile | 1 + sysdeps/x86_64/multiarch/memset-erms.S | 44 +++++++++++++++++++ .../multiarch/memset-vec-unaligned-erms.S | 31 ------------- 3 files changed, 45 insertions(+), 31 deletions(-) create mode 100644 sysdeps/x86_64/multiarch/memset-erms.S diff --git a/sysdeps/x86_64/multiarch/Makefile b/sysdeps/x86_64/multiarch/Makefile index 62a4d96fb8..18cea04423 100644 --- a/sysdeps/x86_64/multiarch/Makefile +++ b/sysdeps/x86_64/multiarch/Makefile @@ -30,6 +30,7 @@ sysdep_routines += \ memset-avx2-unaligned-erms-rtm \ memset-avx512-no-vzeroupper \ memset-avx512-unaligned-erms \ + memset-erms \ memset-evex-unaligned-erms \ memset-sse2-unaligned-erms \ rawmemchr-avx2 \ diff --git a/sysdeps/x86_64/multiarch/memset-erms.S b/sysdeps/x86_64/multiarch/memset-erms.S new file mode 100644 index 0000000000..e83cccc731 --- /dev/null +++ b/sysdeps/x86_64/multiarch/memset-erms.S @@ -0,0 +1,44 @@ +/* memset implement with rep stosb + Copyright (C) 2022 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + + +#include + +#if defined USE_MULTIARCH && IS_IN (libc) + .text +ENTRY (__memset_chk_erms) + cmp %RDX_LP, %RCX_LP + jb HIDDEN_JUMPTARGET (__chk_fail) +END (__memset_chk_erms) + +/* Only used to measure performance of REP STOSB. */ +ENTRY (__memset_erms) + /* Skip zero length. */ + test %RDX_LP, %RDX_LP + jz L(stosb_return_zero) + mov %RDX_LP, %RCX_LP + movzbl %sil, %eax + mov %RDI_LP, %RDX_LP + rep stosb + mov %RDX_LP, %RAX_LP + ret +L(stosb_return_zero): + movq %rdi, %rax + ret +END (__memset_erms) +#endif diff --git a/sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S b/sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S index abc12d9cda..905d0fa464 100644 --- a/sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S +++ b/sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S @@ -156,37 +156,6 @@ L(entry_from_wmemset): #if defined USE_MULTIARCH && IS_IN (libc) END (MEMSET_SYMBOL (__memset, unaligned)) -# if VEC_SIZE == 16 -ENTRY (__memset_chk_erms) - cmp %RDX_LP, %RCX_LP - jb HIDDEN_JUMPTARGET (__chk_fail) -END (__memset_chk_erms) - -/* Only used to measure performance of REP STOSB. */ -ENTRY (__memset_erms) - /* Skip zero length. */ - test %RDX_LP, %RDX_LP - jnz L(stosb) - movq %rdi, %rax - ret -# else -/* Provide a hidden symbol to debugger. */ - .hidden MEMSET_SYMBOL (__memset, erms) -ENTRY (MEMSET_SYMBOL (__memset, erms)) -# endif -L(stosb): - mov %RDX_LP, %RCX_LP - movzbl %sil, %eax - mov %RDI_LP, %RDX_LP - rep stosb - mov %RDX_LP, %RAX_LP - VZEROUPPER_RETURN -# if VEC_SIZE == 16 -END (__memset_erms) -# else -END (MEMSET_SYMBOL (__memset, erms)) -# endif - # if defined SHARED && IS_IN (libc) ENTRY_CHK (MEMSET_CHK_SYMBOL (__memset_chk, unaligned_erms)) cmp %RDX_LP, %RCX_LP