From patchwork Wed Jun 12 22:53:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Pinski X-Patchwork-Id: 1947149 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=quicinc.com header.i=@quicinc.com header.a=rsa-sha256 header.s=qcppdkim1 header.b=Bw/lsS/A; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4W017l2Gybz1ydW for ; Thu, 13 Jun 2024 08:54:27 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 98DA43882061 for ; Wed, 12 Jun 2024 22:54:25 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mx0a-0031df01.pphosted.com (mx0a-0031df01.pphosted.com [205.220.168.131]) by sourceware.org (Postfix) with ESMTPS id DE0AA3882052 for ; Wed, 12 Jun 2024 22:54:03 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org DE0AA3882052 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=quicinc.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=quicinc.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org DE0AA3882052 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=205.220.168.131 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1718232846; cv=none; b=rnae0h4Qvu8P3SAVqtQncKCQx45vxBuvOrL8bjs7/Kg9TEqcoArujwTOSR3ToFajlIdeXMVUwec2skPqoIvPqiuF3d/Kak4k/dAGzchEKX90C5Qsru7+fTUvwcq1ji75r1L0ZURasaEDYohzc3M/B76Jaac0huAplz6UI1VtUgQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1718232846; c=relaxed/simple; bh=ETZ9BZ5v6HN6dwBNJOveeA7utNiL8iIrRxrSffzVh+o=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=Gx+VJy3odRI0F+YOt0TibXmyfBNKxfISTYWBOfG/eGJ09SZ5EycMblDfqq+GjBJBUcXAEhqRNDBet45mP8idWyTwRTFgIeQdIq3cWlQBFGWfdHGh/hbAmVNcp3krtwxQXXO2p942iQcD4NNCYioFCdnIi4CnZ+i3gTMGuJcravo= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from pps.filterd (m0279865.ppops.net [127.0.0.1]) by mx0a-0031df01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 45CKn8xH022314 for ; Wed, 12 Jun 2024 22:54:03 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=quicinc.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=qcppdkim1; bh= XfXoiz5X1r+svxmqjz5ZQtKXafK5waobotIx0xEcgI0=; b=Bw/lsS/A6/GDf9OZ THzWj1k5KQBNc7zr0QH17+vxaM36/6hbxV2Xq847RwmxZSVR1lflXTv+cRBhMHSH dlIH3wYQN9p1QpTFuAtChMgWw1TliyGYI6UJNm9bd9EakvLcWMxlNeiQ/MijRE7V RQPBQmCIVlKAU3aZsachzTbGZiSgKfUISvkYBGylIxYU3mNJSi47HF4FLv+YUTVt DjanxFdjOEad7vGtlMFEqn2Fg8tTHyofI3puBdpNkCg64z6J0jxb80Oi1dPuxqRM 56/QjokS4T2lpVk0ZpCEqfbV1yIPQvQ5patF7vFlWuxGPtswKx5TCA/aNsP3eKx4 PFBLLg== Received: from nasanppmta05.qualcomm.com (i-global254.qualcomm.com [199.106.103.254]) by mx0a-0031df01.pphosted.com (PPS) with ESMTPS id 3yq4s8jqbn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 12 Jun 2024 22:54:02 +0000 (GMT) Received: from nasanex01c.na.qualcomm.com (nasanex01c.na.qualcomm.com [10.45.79.139]) by NASANPPMTA05.qualcomm.com (8.17.1.19/8.17.1.19) with ESMTPS id 45CMs2Bl004652 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 12 Jun 2024 22:54:02 GMT Received: from hu-apinski-lv.qualcomm.com (10.49.16.6) by nasanex01c.na.qualcomm.com (10.45.79.139) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.9; Wed, 12 Jun 2024 15:54:01 -0700 From: Andrew Pinski To: CC: Andrew Pinski Subject: [PATCH v3 2/2] Aarch64: Add new memset for Qualcomm's oryon-1 core Date: Wed, 12 Jun 2024 15:53:36 -0700 Message-ID: <20240612225336.1531323-2-quic_apinski@quicinc.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240612225336.1531323-1-quic_apinski@quicinc.com> References: <20240612225336.1531323-1-quic_apinski@quicinc.com> MIME-Version: 1.0 X-Originating-IP: [10.49.16.6] X-ClientProxiedBy: nalasex01b.na.qualcomm.com (10.47.209.197) To nasanex01c.na.qualcomm.com (10.45.79.139) X-QCInternal: smtphost X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=5800 signatures=585085 X-Proofpoint-GUID: QJ3AHCAMZOqodBgNFq7Ac9iE9CfJf7FP X-Proofpoint-ORIG-GUID: QJ3AHCAMZOqodBgNFq7Ac9iE9CfJf7FP X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-06-12_10,2024-06-12_02,2024-05-17_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 bulkscore=0 spamscore=0 mlxscore=0 malwarescore=0 phishscore=0 lowpriorityscore=0 priorityscore=1501 clxscore=1015 mlxlogscore=999 suspectscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2405170001 definitions=main-2406120163 X-Spam-Status: No, score=-12.9 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, RCVD_IN_MSPIKE_H2, SCC_5_SHORT_WORD_LINES, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org Qualcom's new core, oryon-1, has a different characteristics for memset than the current versions of memset. For non-zero, larger sizes, using GPRs rather than the SIMD stores is ~30% faster. For even larger sizes, using the nontemporal stores is needed not to polute the L1/L2 caches. For zero values, using `dc zva` should be used. Since we know the size will always be 64 bytes, we don't need to figure out the size there. I started with the emag memset and added back the `dc zva` code. Changes since v1: * v3: Fix comment formating Signed-off-by: Andrew Pinski Reviewed-by: Adhemerval Zanella --- sysdeps/aarch64/multiarch/Makefile | 1 + sysdeps/aarch64/multiarch/ifunc-impl-list.c | 1 + sysdeps/aarch64/multiarch/memset.c | 5 + sysdeps/aarch64/multiarch/memset_oryon1.S | 169 ++++++++++++++++++++ 4 files changed, 176 insertions(+) create mode 100644 sysdeps/aarch64/multiarch/memset_oryon1.S diff --git a/sysdeps/aarch64/multiarch/Makefile b/sysdeps/aarch64/multiarch/Makefile index ef5ea9ab8c..3e251cc234 100644 --- a/sysdeps/aarch64/multiarch/Makefile +++ b/sysdeps/aarch64/multiarch/Makefile @@ -15,6 +15,7 @@ sysdep_routines += \ memset_generic \ memset_kunpeng \ memset_mops \ + memset_oryon1 \ memset_zva64 \ strlen_asimd \ strlen_generic \ diff --git a/sysdeps/aarch64/multiarch/ifunc-impl-list.c b/sysdeps/aarch64/multiarch/ifunc-impl-list.c index 65c56b9b41..b2fda541f9 100644 --- a/sysdeps/aarch64/multiarch/ifunc-impl-list.c +++ b/sysdeps/aarch64/multiarch/ifunc-impl-list.c @@ -56,6 +56,7 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, IFUNC_IMPL_ADD (array, i, memmove, 1, __memmove_generic)) IFUNC_IMPL (i, name, memset, IFUNC_IMPL_ADD (array, i, memset, (zva_size == 64), __memset_zva64) + IFUNC_IMPL_ADD (array, i, memset, (zva_size == 64), __memset_oryon1) IFUNC_IMPL_ADD (array, i, memset, 1, __memset_emag) IFUNC_IMPL_ADD (array, i, memset, 1, __memset_kunpeng) #if HAVE_AARCH64_SVE_ASM diff --git a/sysdeps/aarch64/multiarch/memset.c b/sysdeps/aarch64/multiarch/memset.c index 34bce045dd..bd063c16c9 100644 --- a/sysdeps/aarch64/multiarch/memset.c +++ b/sysdeps/aarch64/multiarch/memset.c @@ -1,5 +1,6 @@ /* Multiple versions of memset. AARCH64 version. Copyright (C) 2017-2024 Free Software Foundation, Inc. + Copyright The GNU Toolchain Authors. This file is part of the GNU C Library. The GNU C Library is free software; you can redistribute it and/or @@ -34,6 +35,7 @@ extern __typeof (__redirect_memset) __memset_kunpeng attribute_hidden; extern __typeof (__redirect_memset) __memset_a64fx attribute_hidden; extern __typeof (__redirect_memset) __memset_generic attribute_hidden; extern __typeof (__redirect_memset) __memset_mops attribute_hidden; +extern __typeof (__redirect_memset) __memset_oryon1 attribute_hidden; static inline __typeof (__redirect_memset) * select_memset_ifunc (void) @@ -49,6 +51,9 @@ select_memset_ifunc (void) return __memset_a64fx; } + if (IS_ORYON1 (midr) && zva_size == 64) + return __memset_oryon1; + if (IS_KUNPENG920 (midr)) return __memset_kunpeng; diff --git a/sysdeps/aarch64/multiarch/memset_oryon1.S b/sysdeps/aarch64/multiarch/memset_oryon1.S new file mode 100644 index 0000000000..b43a43b54e --- /dev/null +++ b/sysdeps/aarch64/multiarch/memset_oryon1.S @@ -0,0 +1,169 @@ +/* Optimized memset for Qualcomm's oyron-1 core. + Copyright (C) 2018-2024 Free Software Foundation, Inc. + Copyright The GNU Toolchain Authors. + + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +#include +#include "memset-reg.h" + +/* Assumptions: + ARMv8-a, AArch64, unaligned accesses + */ + +ENTRY (__memset_oryon1) + + PTR_ARG (0) + SIZE_ARG (2) + + bfi valw, valw, 8, 8 + bfi valw, valw, 16, 16 + bfi val, val, 32, 32 + + add dstend, dstin, count + + cmp count, 96 + b.hi L(set_long) + cmp count, 16 + b.hs L(set_medium) + + /* Set 0..15 bytes. */ + tbz count, 3, 1f + str val, [dstin] + str val, [dstend, -8] + ret + + .p2align 3 +1: tbz count, 2, 2f + str valw, [dstin] + str valw, [dstend, -4] + ret +2: cbz count, 3f + strb valw, [dstin] + tbz count, 1, 3f + strh valw, [dstend, -2] +3: ret + + .p2align 3 + /* Set 16..96 bytes. */ +L(set_medium): + stp val, val, [dstin] + tbnz count, 6, L(set96) + stp val, val, [dstend, -16] + tbz count, 5, 1f + stp val, val, [dstin, 16] + stp val, val, [dstend, -32] +1: ret + + .p2align 6 + /* Set 64..96 bytes. Write 64 bytes from the start and + 32 bytes from the end. */ +L(set96): + stp val, val, [dstin, 16] + stp val, val, [dstin, 32] + stp val, val, [dstin, 48] + stp val, val, [dstend, -32] + stp val, val, [dstend, -16] + ret + + .p2align 6 +L(set_long): + stp val, val, [dstin] + bic dst, dstin, 15 + cmp count, 256 + ccmp valw, 0, 0, cs + b.eq L(try_zva) + cmp count, #32768 + b.hi L(set_long_with_nontemp) + /* Small-size or non-zero memset does not use DC ZVA. */ + sub count, dstend, dst + + /* Adjust count and bias for loop. By subtracting extra 1 from count, + it is easy to use tbz instruction to check whether loop tailing + count is less than 33 bytes, so as to bypass 2 unnecessary stps. */ + sub count, count, 64+16+1 + +1: stp val, val, [dst, 16] + stp val, val, [dst, 32] + stp val, val, [dst, 48] + stp val, val, [dst, 64]! + subs count, count, 64 + b.hs 1b + + tbz count, 5, 1f /* Remaining count is less than 33 bytes? */ + stp val, val, [dst, 16] + stp val, val, [dst, 32] +1: stp val, val, [dstend, -32] + stp val, val, [dstend, -16] + ret + +L(set_long_with_nontemp): + /* Small-size or non-zero memset does not use DC ZVA. */ + sub count, dstend, dst + + /* Adjust count and bias for loop. By subtracting extra 1 from count, + it is easy to use tbz instruction to check whether loop tailing + count is less than 33 bytes, so as to bypass 2 unnecessary stps. */ + sub count, count, 64+16+1 + +1: stnp val, val, [dst, 16] + stnp val, val, [dst, 32] + stnp val, val, [dst, 48] + stnp val, val, [dst, 64] + add dst, dst, #64 + subs count, count, 64 + b.hs 1b + + tbz count, 5, 1f /* Remaining count is less than 33 bytes? */ + stnp val, val, [dst, 16] + stnp val, val, [dst, 32] +1: stnp val, val, [dstend, -32] + stnp val, val, [dstend, -16] + ret + +L(try_zva): + /* Write the first and last 64 byte aligned block using stp rather + than using DC ZVA as it is faster. */ + .p2align 6 +L(zva_64): + stp val, val, [dst, 16] + stp val, val, [dst, 32] + stp val, val, [dst, 48] + bic dst, dst, 63 + stp val, val, [dst, 64] + stp val, val, [dst, 64+16] + stp val, val, [dst, 96] + stp val, val, [dst, 96+16] + sub count, dstend, dst /* Count is now 128 too large. */ + sub count, count, 128+64+64 /* Adjust count and bias for loop. */ + add dst, dst, 128 +1: dc zva, dst + add dst, dst, 64 + subs count, count, 64 + b.hi 1b + stp val, val, [dst, 0] + stp val, val, [dst, 16] + stp val, val, [dst, 32] + stp val, val, [dst, 48] + + stp val, val, [dstend, -64] + stp val, val, [dstend, -64+16] + stp val, val, [dstend, -32] + stp val, val, [dstend, -16] + ret + +END (__memset_oryon1)