From patchwork Tue Feb 7 00:16:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Christoph_M=C3=BCllner?= X-Patchwork-Id: 1738548 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=sourceware.org; envelope-from=libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=vrull.eu header.i=@vrull.eu header.a=rsa-sha256 header.s=google header.b=jtfz4UJ5; dkim-atps=neutral Received: from sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4P9kLg4sdDz23y5 for ; Tue, 7 Feb 2023 11:20:59 +1100 (AEDT) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id A34423881D35 for ; Tue, 7 Feb 2023 00:20:57 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by sourceware.org (Postfix) with ESMTPS id 634A43858431 for ; Tue, 7 Feb 2023 00:16:52 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 634A43858431 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=vrull.eu Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=vrull.eu Received: by mail-wm1-x32b.google.com with SMTP id j32-20020a05600c1c2000b003dc4fd6e61dso12016350wms.5 for ; Mon, 06 Feb 2023 16:16:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vrull.eu; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sWR8SAAC+fazj6mFfkJTHA8fGla9wAUW7XCvoAqOVLY=; b=jtfz4UJ5UsidoXCX/E+pFOkRcalu2DZZp+N3CY+2EC2uD+rSuv0MAdcu4D8g5j1xTT 2PnXCNARMaKP6Te20FahFbiE52lxE87E4F/Az5fLoaOl8m2qWvzNXdfEvOQv/cs7jq1I xYzJss98oUGAOg5jNw11PsfCjqNC9SLg8tj/q8IIrn+oQPHOOClKlxitduV0x/I5hCuM ip/M3Hw9MCg5O0Zh1Vm0I1Qvz3W+hiPMIDfFCIsdY8rqEy7rWE7BcM0875bXemUHgEmp Wseg0QhSLPf+ipuTSuIEEV67UvOJ2Vfg2v+i6Z3hz+x/pTUduMwp+gtn/qCJ3VdjBvuz UCuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sWR8SAAC+fazj6mFfkJTHA8fGla9wAUW7XCvoAqOVLY=; b=vQlTcqvK34lJDS8oRuuKBITnhDBA58upFgeLcMyYhYxG1yG3H0DKdJ3yGL60PPMjy/ THioLqCufXN1H3NQyD/Vg9jYwzD2dcyhlo6sjr6OZO/mIgNY9OFELlKovHCjxoqw2doz p8C83VMFDG5+Cm6tLjZ32ynOh+lgeKD1GJHrwTKMlJ57Pb2Dc08oDdVw81DpyKZiRPXW dUpio5rIQjLOK+frTn5jBfyqfTwm1qew4HRbxsjGHokDCFEhhbuETUnCq0Z2Huif2qi/ Oi1AeZMqsBh+ulDRVBgBKL+T6atnqNAn3yTDLh4ejSiEPFe57yywnWye9Z5eLDX1XrLg NOww== X-Gm-Message-State: AO0yUKUZgZ9e0uFzMHG6WEf3nHxmGZVJPYuub+VPR+hPXV4OQpCgy9sH t7/HUz4SP7xseWZfQW1CKDj9jHYd9HHoyWo1 X-Google-Smtp-Source: AK7set/y4Z2jqM6MPq7WC6HZWk3w7pDbcsWLSPmUXslekK/p7Bmmw5q4h7d8UMOa1UfuEhzEcaL7KQ== X-Received: by 2002:a05:600c:3486:b0:3df:9858:c03d with SMTP id a6-20020a05600c348600b003df9858c03dmr12443128wmq.18.1675729010895; Mon, 06 Feb 2023 16:16:50 -0800 (PST) Received: from beast.fritz.box (62-178-148-172.cable.dynamic.surfer.at. [62.178.148.172]) by smtp.gmail.com with ESMTPSA id f1-20020a1cc901000000b003df14531724sm16862050wmb.21.2023.02.06.16.16.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 06 Feb 2023 16:16:50 -0800 (PST) From: Christoph Muellner To: libc-alpha@sourceware.org, Palmer Dabbelt , Darius Rad , Andrew Waterman , DJ Delorie , Vineet Gupta , Kito Cheng , Jeff Law , Philipp Tomsich , Heiko Stuebner Cc: =?utf-8?q?Christoph_M=C3=BCllner?= Subject: [RFC PATCH 14/19] riscv: Add accelerated strlen routine Date: Tue, 7 Feb 2023 01:16:13 +0100 Message-Id: <20230207001618.458947-15-christoph.muellner@vrull.eu> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230207001618.458947-1-christoph.muellner@vrull.eu> References: <20230207001618.458947-1-christoph.muellner@vrull.eu> MIME-Version: 1.0 X-Spam-Status: No, score=-12.7 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, JMQ_SPF_NEUTRAL, KAM_MANYTO, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org Sender: "Libc-alpha" From: Christoph Müllner The implementation of strlen() can be accelerated using Zbb's orc.b instruction. Let's add an implementation that provides that. The implementation is part of the Bitmanip specification. Signed-off-by: Christoph Müllner --- sysdeps/riscv/multiarch/Makefile | 3 +- sysdeps/riscv/multiarch/ifunc-impl-list.c | 1 + sysdeps/riscv/multiarch/strlen.c | 6 +- sysdeps/riscv/multiarch/strlen_zbb.S | 105 ++++++++++++++++++++++ 4 files changed, 113 insertions(+), 2 deletions(-) create mode 100644 sysdeps/riscv/multiarch/strlen_zbb.S diff --git a/sysdeps/riscv/multiarch/Makefile b/sysdeps/riscv/multiarch/Makefile index 8e2b020233..b2247b7326 100644 --- a/sysdeps/riscv/multiarch/Makefile +++ b/sysdeps/riscv/multiarch/Makefile @@ -8,5 +8,6 @@ sysdep_routines += \ memset_rv64_unaligned \ memset_rv64_unaligned_cboz64 \ \ - strlen_generic + strlen_generic \ + strlen_zbb endif diff --git a/sysdeps/riscv/multiarch/ifunc-impl-list.c b/sysdeps/riscv/multiarch/ifunc-impl-list.c index f848fc8401..2b4d2e1c17 100644 --- a/sysdeps/riscv/multiarch/ifunc-impl-list.c +++ b/sysdeps/riscv/multiarch/ifunc-impl-list.c @@ -55,6 +55,7 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, IFUNC_IMPL_ADD (array, i, memset, 1, __memset_generic)) IFUNC_IMPL (i, name, strlen, + IFUNC_IMPL_ADD (array, i, strlen, 1, __strlen_zbb) IFUNC_IMPL_ADD (array, i, strlen, 1, __strlen_generic)) diff --git a/sysdeps/riscv/multiarch/strlen.c b/sysdeps/riscv/multiarch/strlen.c index 85f7a91c9f..8b2f4d94b2 100644 --- a/sysdeps/riscv/multiarch/strlen.c +++ b/sysdeps/riscv/multiarch/strlen.c @@ -30,8 +30,12 @@ extern __typeof (__redirect_strlen) __libc_strlen; extern __typeof (__redirect_strlen) __strlen_generic attribute_hidden; +extern __typeof (__redirect_strlen) __strlen_zbb attribute_hidden; -libc_ifunc (__libc_strlen, __strlen_generic); +libc_ifunc (__libc_strlen, + HAVE_RV(zbb) + ? __strlen_zbb + : __strlen_generic); # undef strlen strong_alias (__libc_strlen, strlen); diff --git a/sysdeps/riscv/multiarch/strlen_zbb.S b/sysdeps/riscv/multiarch/strlen_zbb.S new file mode 100644 index 0000000000..a0ca599c8e --- /dev/null +++ b/sysdeps/riscv/multiarch/strlen_zbb.S @@ -0,0 +1,105 @@ +/* Copyright (C) 2022 Free Software Foundation, Inc. + + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +#include +#include + +/* Assumptions: rvi_zbb. */ +/* Implementation from the Bitmanip specification. */ + +#define src a0 +#define result a0 +#define addr a1 +#define data a2 +#define offset a3 +#define offset_bits a3 +#define valid_bytes a4 +#define m1 a4 + +#if __riscv_xlen == 64 +# define REG_L ld +# define SZREG 8 +#else +# define REG_L lw +# define SZREG 4 +#endif + +#define BITSPERBYTELOG 3 + +#if __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__ +# define CZ clz +# define SHIFT sll +#else +# define CZ ctz +# define SHIFT srl +#endif + +#ifndef STRLEN +# define STRLEN __strlen_zbb +#endif + +.option push +.option arch,+zbb + +ENTRY_ALIGN (STRLEN, 6) + andi offset, src, SZREG-1 + andi addr, src, -SZREG + + li valid_bytes, SZREG + sub valid_bytes, valid_bytes, offset + slli offset_bits, offset, BITSPERBYTELOG + REG_L data, 0(addr) + /* Shift the partial/unaligned chunk we loaded to remove the bytes + * from before the start of the string, adding NUL bytes at the end. */ + SHIFT data, data, offset_bits + orc.b data, data + not data, data + /* Non-NUL bytes in the string have been expanded to 0x00, while + * NUL bytes have become 0xff. Search for the first set bit + * (corresponding to a NUL byte in the original chunk). */ + CZ data, data + /* The first chunk is special: compare against the number of valid + * bytes in this chunk. */ + srli result, data, 3 + bgtu valid_bytes, result, L(done) + addi offset, addr, SZREG + li m1, -1 + + /* Our critical loop is 4 instructions and processes data in 4 byte + * or 8 byte chunks. */ + .p2align 2 +L(loop): + REG_L data, SZREG(addr) + addi addr, addr, SZREG + orc.b data, data + beq data, m1, L(loop) + +L(epilogue): + not data, data + CZ data, data + sub offset, addr, offset + add result, result, offset + srli data, data, 3 + add result, result, data +L(done): + ret + +.option pop + +END (STRLEN) +libc_hidden_builtin_def (STRLEN)