From patchwork Wed Aug 23 22:31:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Evan Green X-Patchwork-Id: 1825008 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=rivosinc-com.20221208.gappssmtp.com header.i=@rivosinc-com.20221208.gappssmtp.com header.a=rsa-sha256 header.s=20221208 header.b=sWq4oexG; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4RWLYG6k5pz1yZs for ; Thu, 24 Aug 2023 08:31:46 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 6D7DC385084A for ; Wed, 23 Aug 2023 22:31:44 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pl1-x62c.google.com (mail-pl1-x62c.google.com [IPv6:2607:f8b0:4864:20::62c]) by sourceware.org (Postfix) with ESMTPS id 379E73857704 for ; Wed, 23 Aug 2023 22:31:31 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 379E73857704 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivosinc.com Received: by mail-pl1-x62c.google.com with SMTP id d9443c01a7336-1bf1935f6c2so2404255ad.1 for ; Wed, 23 Aug 2023 15:31:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1692829890; x=1693434690; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=b3ot02C/T3SRHN9du7YJu0kdE25vZs3A6/ZgJgKPk2Q=; b=sWq4oexGeTPUJwFR0ldJb9Bc9I+gytBVJlsMzEQfS3YrOKOBwtWDTQddUsn01o+DkV rUalpeTd2Rg5zewMaYLm4yBZFZUbpIRVjts4AZ/Fl5qESWbXUOyxQkdleWck2mcjxOsZ LWUpQjC+LVm2vpt3ftbP0vPo6mymAbr4sJZSEzbp4b6awGexE7GYv0shDhl8FI2kMDeX zGyvfkHHOae2VportyePOr0bRBnJ3GhMz5Hm0+T+Oa0eFOtLCN3dznHuCovSgKasD1gV eH+hZUuKTWiQQzaG3Lj7iMLF+mn33H8Ks/PDBnKuiPI8X1BEwfNFgC1Go0uvflFQnLR4 L8VQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692829890; x=1693434690; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=b3ot02C/T3SRHN9du7YJu0kdE25vZs3A6/ZgJgKPk2Q=; b=GEwWgpVYxgh9SGOvKmbGdiQoQrcbnNYpTahb8/F/hG6RbFNkxOAW5bvjCCvIs8Bqqz iqEa/FhURdDYenULlJ0WT41T4B/X7zfRvvNb8bEzzb+XRY3KL6T23/HE08xiXWylhAvr OurQg2KlcpXXLnonxBjflrIry3scFtOXGuYVQhlXnH3jEtHyuWbEkVhrL+kIDO2zHcs7 V/yKSTDEtlAAoi8iwGnsFl3l4kBeBLe0jMT1CiF5qk8SPaOwP2LGa0cnN8JC3o7BgSuY iFr6ib5VfCz8+iKkhufc/gluibZJM/QBqtJHKkaHca2h08CmC+oh4y7wsb0q1COaexrS esBQ== X-Gm-Message-State: AOJu0YzrdOI3sKRGTotNN14iR3BCnjA0qRclVnFOx5A+u6tzYhyr62TA M4MC6f4HwKv5lO8N6l6qMgAHR4gfGOtuWiOJQws= X-Google-Smtp-Source: AGHT+IHiui0y0IdnSpN88DjxW0/C04MvB4PFbSQ8a6WFocIdVXSgX5VPt+4ROE1cjnR/w4flHOeYdQ== X-Received: by 2002:a17:902:e550:b0:1b9:d38d:f08d with SMTP id n16-20020a170902e55000b001b9d38df08dmr24248091plf.18.1692829889744; Wed, 23 Aug 2023 15:31:29 -0700 (PDT) Received: from evan.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id i1-20020a17090332c100b001bbf7fd354csm11335150plr.213.2023.08.23.15.31.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Aug 2023 15:31:29 -0700 (PDT) From: Evan Green To: libc-alpha@sourceware.org Cc: vineetg@rivosinc.com, palmer@rivosinc.com, Florian Weimer , slewis@rivosinc.com, Evan Green Subject: [PATCH v7 1/6] riscv: Add Linux hwprobe syscall support Date: Wed, 23 Aug 2023 15:31:10 -0700 Message-Id: <20230823223116.645225-2-evan@rivosinc.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823223116.645225-1-evan@rivosinc.com> References: <20230823223116.645225-1-evan@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-11.6 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org Sender: "Libc-alpha" Add awareness and a thin wrapper function around a new Linux system call that allows callers to get architecture and microarchitecture information about the CPUs from the kernel. This can be used to do things like dynamically choose a memcpy implementation. Signed-off-by: Evan Green Reviewed-by: Palmer Dabbelt --- Changes in v7: - Bumped Versions up to 2.39 (Joseph) - Used INTERNAL_SYSCALL_CALL, and return positive errno to match pthreads API (Florian). - Remove __THROW since it creates a warning in combination with the fortified access decorators. Changes in v6: - Prefixed __riscv_hwprobe() parameters names with __ to avoid user macro namespace pollution (Joseph) Changes in v4: - Remove __USE_GNU (Florian) - __nonnull, __wur, __THROW, and __fortified_attr_access decorations (Florian) - change long to long int (Florian) - Fix comment formatting (Florian) - Update backup kernel header content copy. - Fix function declaration formatting (Florian) - Changed export versions to 2.38 Changes in v3: - Update argument types to match v4 kernel interface Changes in v2: - hwprobe.h: Use __has_include and duplicate Linux content to make compilation work when Linux headers are absent (Adhemerval) - hwprobe.h: Put declaration under __USE_GNU (Adhemerval) - Use INLINE_SYSCALL_CALL (Adhemerval) - Update versions - Update UNALIGNED_MASK to match kernel v3 series. sysdeps/unix/sysv/linux/riscv/Makefile | 4 +- sysdeps/unix/sysv/linux/riscv/Versions | 3 + sysdeps/unix/sysv/linux/riscv/hwprobe.c | 39 ++++++++++ .../unix/sysv/linux/riscv/rv32/libc.abilist | 1 + .../unix/sysv/linux/riscv/rv64/libc.abilist | 1 + sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h | 72 +++++++++++++++++++ 6 files changed, 118 insertions(+), 2 deletions(-) create mode 100644 sysdeps/unix/sysv/linux/riscv/hwprobe.c create mode 100644 sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h diff --git a/sysdeps/unix/sysv/linux/riscv/Makefile b/sysdeps/unix/sysv/linux/riscv/Makefile index 4b6eacb32f..45cc29e40d 100644 --- a/sysdeps/unix/sysv/linux/riscv/Makefile +++ b/sysdeps/unix/sysv/linux/riscv/Makefile @@ -1,6 +1,6 @@ ifeq ($(subdir),misc) -sysdep_headers += sys/cachectl.h -sysdep_routines += flush-icache +sysdep_headers += sys/cachectl.h sys/hwprobe.h +sysdep_routines += flush-icache hwprobe endif ifeq ($(subdir),stdlib) diff --git a/sysdeps/unix/sysv/linux/riscv/Versions b/sysdeps/unix/sysv/linux/riscv/Versions index 5625d2a0b8..8717b62a4a 100644 --- a/sysdeps/unix/sysv/linux/riscv/Versions +++ b/sysdeps/unix/sysv/linux/riscv/Versions @@ -8,4 +8,7 @@ libc { GLIBC_2.27 { __riscv_flush_icache; } + GLIBC_2.39 { + __riscv_hwprobe; + } } diff --git a/sysdeps/unix/sysv/linux/riscv/hwprobe.c b/sysdeps/unix/sysv/linux/riscv/hwprobe.c new file mode 100644 index 0000000000..f2dc46cfd7 --- /dev/null +++ b/sysdeps/unix/sysv/linux/riscv/hwprobe.c @@ -0,0 +1,39 @@ +/* RISC-V hardware feature probing support on Linux + Copyright (C) 2023 Free Software Foundation, Inc. + + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public License as + published by the Free Software Foundation; either version 2.1 of the + License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include +#include +#include + +int __riscv_hwprobe (struct riscv_hwprobe *__pairs, size_t __pair_count, + size_t __cpu_count, unsigned long int *__cpus, + unsigned int __flags) +{ + int r; + + r = INTERNAL_SYSCALL_CALL (riscv_hwprobe, 5, __pairs, __pair_count, + __cpu_count, __cpus, __flags); + + /* Negate negative errno values to match pthreads API. */ + if (r < 0) + r = -r; + + return r; +} diff --git a/sysdeps/unix/sysv/linux/riscv/rv32/libc.abilist b/sysdeps/unix/sysv/linux/riscv/rv32/libc.abilist index b9740a1afc..8fab4a606f 100644 --- a/sysdeps/unix/sysv/linux/riscv/rv32/libc.abilist +++ b/sysdeps/unix/sysv/linux/riscv/rv32/libc.abilist @@ -2436,3 +2436,4 @@ GLIBC_2.38 strlcat F GLIBC_2.38 strlcpy F GLIBC_2.38 wcslcat F GLIBC_2.38 wcslcpy F +GLIBC_2.38 __riscv_hwprobe F diff --git a/sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist b/sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist index e3b4656aa2..1ebb91deed 100644 --- a/sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist +++ b/sysdeps/unix/sysv/linux/riscv/rv64/libc.abilist @@ -2636,3 +2636,4 @@ GLIBC_2.38 strlcat F GLIBC_2.38 strlcpy F GLIBC_2.38 wcslcat F GLIBC_2.38 wcslcpy F +GLIBC_2.38 __riscv_hwprobe F diff --git a/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h b/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h new file mode 100644 index 0000000000..aa189a4818 --- /dev/null +++ b/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h @@ -0,0 +1,72 @@ +/* RISC-V architecture probe interface + Copyright (C) 2023 Free Software Foundation, Inc. + + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +#ifndef _SYS_HWPROBE_H +#define _SYS_HWPROBE_H 1 + +#include +#include +#ifdef __has_include +# if __has_include () +# include +# endif +#endif + +/* Define a (probably stale) version of the interface if the Linux headers + aren't present. */ +#ifndef RISCV_HWPROBE_KEY_MVENDORID +struct riscv_hwprobe { + signed long long int key; + unsigned long long int value; +}; + +#define RISCV_HWPROBE_KEY_MVENDORID 0 +#define RISCV_HWPROBE_KEY_MARCHID 1 +#define RISCV_HWPROBE_KEY_MIMPID 2 +#define RISCV_HWPROBE_KEY_BASE_BEHAVIOR 3 +#define RISCV_HWPROBE_BASE_BEHAVIOR_IMA (1 << 0) +#define RISCV_HWPROBE_KEY_IMA_EXT_0 4 +#define RISCV_HWPROBE_IMA_FD (1 << 0) +#define RISCV_HWPROBE_IMA_C (1 << 1) +#define RISCV_HWPROBE_IMA_V (1 << 2) +#define RISCV_HWPROBE_EXT_ZBA (1 << 3) +#define RISCV_HWPROBE_EXT_ZBB (1 << 4) +#define RISCV_HWPROBE_EXT_ZBS (1 << 5) +#define RISCV_HWPROBE_KEY_CPUPERF_0 5 +#define RISCV_HWPROBE_MISALIGNED_UNKNOWN (0 << 0) +#define RISCV_HWPROBE_MISALIGNED_EMULATED (1 << 0) +#define RISCV_HWPROBE_MISALIGNED_SLOW (2 << 0) +#define RISCV_HWPROBE_MISALIGNED_FAST (3 << 0) +#define RISCV_HWPROBE_MISALIGNED_UNSUPPORTED (4 << 0) +#define RISCV_HWPROBE_MISALIGNED_MASK (7 << 0) + +#endif /* RISCV_HWPROBE_KEY_MVENDORID */ + +__BEGIN_DECLS + +extern int __riscv_hwprobe (struct riscv_hwprobe *__pairs, size_t __pair_count, + size_t __cpu_count, unsigned long int *__cpus, + unsigned int __flags) + __nonnull ((1)) __wur + __fortified_attr_access (__read_write__, 1, 2) + __fortified_attr_access (__read_only__, 4, 3); + +__END_DECLS + +#endif /* sys/hwprobe.h */ From patchwork Wed Aug 23 22:31:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Evan Green X-Patchwork-Id: 1825010 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=rivosinc-com.20221208.gappssmtp.com header.i=@rivosinc-com.20221208.gappssmtp.com header.a=rsa-sha256 header.s=20221208 header.b=iKBYXtv7; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4RWLYP5p55z1yNm for ; Thu, 24 Aug 2023 08:31:53 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id D3E183870C16 for ; Wed, 23 Aug 2023 22:31:51 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pl1-x62d.google.com (mail-pl1-x62d.google.com [IPv6:2607:f8b0:4864:20::62d]) by sourceware.org (Postfix) with ESMTPS id E9B2B385DC3A for ; Wed, 23 Aug 2023 22:31:34 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org E9B2B385DC3A Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivosinc.com Received: by mail-pl1-x62d.google.com with SMTP id d9443c01a7336-1bf6ea270b2so27965295ad.0 for ; Wed, 23 Aug 2023 15:31:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1692829893; x=1693434693; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qRJ39EhVTuQjul/N5iZq/AvOhgSvsw+6qVl245uVW4k=; b=iKBYXtv7b5cyGhKa3RA/XxVjnSsPJ70wCfvlooMMWLzzYvGcLrFpAKnSxq+J1xZkJU 83h+FaxHVQnfFKU6k2onKs4X+voaJgtwAsikavCEZslIpPnlkSHhNbNRgwXjSMBcHzGk g6RgHOhVpDX5+RiohllUGk+2k8GH/FPUZG6RgMtmxAQKVqjHwi0cqz0922XGu0FCn/jm OGdecTGVrDMd+rnFW6qjQSJOMTso2Vus3MVt4lGA/Ap2f04PnFsDOMtLwfdsnZjeeobv VNx0c6GN6EZfZucjDulZVtAuelEtQwXoo1lBX5Dtwx8aZCLaSCFamj780Un68veietqy k6Hw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692829893; x=1693434693; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qRJ39EhVTuQjul/N5iZq/AvOhgSvsw+6qVl245uVW4k=; b=OI78Y5fBV8BYeKaItoijrjfWcMvN7t7BSi0iKOF8fDe47OqdqCgn6aw0H+Pfj5bH6K i8mXjWPDeOInpaGRaeVXYYXmQ1VW7KdoSKRvVC4dfIy/JtTfRVNOfIXH0ooh0WTehA8h 0/L9jJgz6LA5v4lrCDnmhmbQ8OXyvV3xPWMVIpznbhZSP3S5+6oX9FpFpoT7I5uv7158 YMYDS0NUK6Lb1kx7KIJFl74N1Mt3WnW7Qk4aBo/durQ7x8VCnjAoyuzqU4iqTizW1s2w Dj/+sAhIkKnr0+/Pu58LN6wXlN78PIQmO/+mdo4akKQCLqW3oaQj7CxDkhJSKI8/BIY/ Azsg== X-Gm-Message-State: AOJu0YxbQZeHdIr61AIigP9KsB8yc9ZFMIiV5fWdzASUxO3nxrgjoU4n SKhp7PT/u3EKgiDx1bZ5ZbEXTIhL/Us6kbk7ugM= X-Google-Smtp-Source: AGHT+IHHjXkr/xXdN2QFT+j/4HQ0pSVC7/M5Z0WrHUYhoOzREShaYLksvjf5dh/BIzKe7RFWjZmB5Q== X-Received: by 2002:a17:902:c101:b0:1bf:6cbc:6ead with SMTP id 1-20020a170902c10100b001bf6cbc6eadmr9783397pli.22.1692829893591; Wed, 23 Aug 2023 15:31:33 -0700 (PDT) Received: from evan.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id i1-20020a17090332c100b001bbf7fd354csm11335150plr.213.2023.08.23.15.31.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Aug 2023 15:31:33 -0700 (PDT) From: Evan Green To: libc-alpha@sourceware.org Cc: vineetg@rivosinc.com, palmer@rivosinc.com, Florian Weimer , slewis@rivosinc.com, Evan Green Subject: [PATCH v7 2/6] riscv: Add hwprobe vdso call support Date: Wed, 23 Aug 2023 15:31:11 -0700 Message-Id: <20230823223116.645225-3-evan@rivosinc.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823223116.645225-1-evan@rivosinc.com> References: <20230823223116.645225-1-evan@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-11.7 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org Sender: "Libc-alpha" The new riscv_hwprobe syscall also comes with a vDSO for faster answers to your most common questions. Call in today to speak with a kernel representative near you! Signed-off-by: Evan Green Reviewed-by: Palmer Dabbelt --- Changes in v7: - Use INTERNAL_VSYSCALL_CALL (Florian) Changes in v3: - Add the "return" to the vsyscall - Fix up vdso arg types to match kernel v4 version - Remove ifdef around INLINE_VSYSCALL (Adhemerval) Changes in v2: - Add vDSO interface sysdeps/unix/sysv/linux/dl-vdso-setup.c | 10 ++++++++++ sysdeps/unix/sysv/linux/dl-vdso-setup.h | 3 +++ sysdeps/unix/sysv/linux/riscv/hwprobe.c | 17 ++++++++++++++--- sysdeps/unix/sysv/linux/riscv/sysdep.h | 1 + 4 files changed, 28 insertions(+), 3 deletions(-) diff --git a/sysdeps/unix/sysv/linux/dl-vdso-setup.c b/sysdeps/unix/sysv/linux/dl-vdso-setup.c index 97eaaeac37..ed8b1ef426 100644 --- a/sysdeps/unix/sysv/linux/dl-vdso-setup.c +++ b/sysdeps/unix/sysv/linux/dl-vdso-setup.c @@ -71,6 +71,16 @@ PROCINFO_CLASS int (*_dl_vdso_clock_getres_time64) (clockid_t, # ifdef HAVE_GET_TBFREQ PROCINFO_CLASS uint64_t (*_dl_vdso_get_tbfreq)(void) RELRO; # endif + +/* RISC-V specific ones. */ +# ifdef HAVE_RISCV_HWPROBE +PROCINFO_CLASS int (*_dl_vdso_riscv_hwprobe)(void *, + size_t, + size_t, + unsigned long *, + unsigned int) RELRO; +# endif + #endif #undef RELRO diff --git a/sysdeps/unix/sysv/linux/dl-vdso-setup.h b/sysdeps/unix/sysv/linux/dl-vdso-setup.h index 867072b897..39eafd5316 100644 --- a/sysdeps/unix/sysv/linux/dl-vdso-setup.h +++ b/sysdeps/unix/sysv/linux/dl-vdso-setup.h @@ -47,6 +47,9 @@ setup_vdso_pointers (void) #ifdef HAVE_GET_TBFREQ GLRO(dl_vdso_get_tbfreq) = dl_vdso_vsym (HAVE_GET_TBFREQ); #endif +#ifdef HAVE_RISCV_HWPROBE + GLRO(dl_vdso_riscv_hwprobe) = dl_vdso_vsym (HAVE_RISCV_HWPROBE); +#endif } #endif diff --git a/sysdeps/unix/sysv/linux/riscv/hwprobe.c b/sysdeps/unix/sysv/linux/riscv/hwprobe.c index f2dc46cfd7..57a4b8a0a8 100644 --- a/sysdeps/unix/sysv/linux/riscv/hwprobe.c +++ b/sysdeps/unix/sysv/linux/riscv/hwprobe.c @@ -27,9 +27,20 @@ int __riscv_hwprobe (struct riscv_hwprobe *__pairs, size_t __pair_count, unsigned int __flags) { int r; - - r = INTERNAL_SYSCALL_CALL (riscv_hwprobe, 5, __pairs, __pair_count, - __cpu_count, __cpus, __flags); + __riscv_hwprobe_t vdso_hwprobe = + (__riscv_hwprobe_t)GLRO(dl_vdso_riscv_hwprobe); + + if (vdso_hwprobe != NULL) + { + r = INTERNAL_VSYSCALL_CALL (vdso_hwprobe, 5, __pairs, __pair_count, + __cpu_count, __cpus, __flags); + } + else + { + r = INTERNAL_SYSCALL_CALL (riscv_hwprobe, 5, __pairs, __pair_count, + __cpu_count, __cpus, __flags); + + } /* Negate negative errno values to match pthreads API. */ if (r < 0) diff --git a/sysdeps/unix/sysv/linux/riscv/sysdep.h b/sysdeps/unix/sysv/linux/riscv/sysdep.h index 5583b96d23..ee015dfeb6 100644 --- a/sysdeps/unix/sysv/linux/riscv/sysdep.h +++ b/sysdeps/unix/sysv/linux/riscv/sysdep.h @@ -156,6 +156,7 @@ /* List of system calls which are supported as vsyscalls (for RV32 and RV64). */ # define HAVE_GETCPU_VSYSCALL "__vdso_getcpu" +# define HAVE_RISCV_HWPROBE "__vdso_riscv_hwprobe" # undef HAVE_INTERNAL_BRK_ADDR_SYMBOL # define HAVE_INTERNAL_BRK_ADDR_SYMBOL 1 From patchwork Wed Aug 23 22:31:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Evan Green X-Patchwork-Id: 1825011 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=rivosinc-com.20221208.gappssmtp.com header.i=@rivosinc-com.20221208.gappssmtp.com header.a=rsa-sha256 header.s=20221208 header.b=bFqpsVPp; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4RWLYk1Ny6z1yNm for ; Thu, 24 Aug 2023 08:32:10 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 3ED2B3851145 for ; Wed, 23 Aug 2023 22:32:08 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pl1-x633.google.com (mail-pl1-x633.google.com [IPv6:2607:f8b0:4864:20::633]) by sourceware.org (Postfix) with ESMTPS id 7784E3853D3C for ; Wed, 23 Aug 2023 22:31:38 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 7784E3853D3C Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivosinc.com Received: by mail-pl1-x633.google.com with SMTP id d9443c01a7336-1c09673b006so15326655ad.1 for ; Wed, 23 Aug 2023 15:31:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1692829897; x=1693434697; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tcFpzT8/0+zuHQr8jRQPh8DN1RQyeYpMkBX8bsq0IHs=; b=bFqpsVPpR0cR1a8lB1qiKiEuWhyrStgvhzFlTVEZBHvDWYhWwa9cTwtaC9X5b7Dd2V ioVBEeTe3nVO0wWrsVp8mrV1pFIIYFlgfwlXEkwPb0L+gh9WnvOD6UNiIHZWOe0ck47Z xKVt0G/Vc82xfDvWZIoBcfHcjjl0gsBCjFzsfMpVmwRYJs2wgawrfyzguY5bRoEZNs8O ysqhNGrgqYXf0jN9Tac0nIor448d9QjIBFfagINmDE9BTKfCdnXCP5M9lzgbztjrl8sZ f9pCpT5JjKR5VLIuhjQVUrdgB+MPSrj61fWpjuSoIW15UKh4iCG7tqIZaGGRMJyu5Uvp 35BA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692829897; x=1693434697; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tcFpzT8/0+zuHQr8jRQPh8DN1RQyeYpMkBX8bsq0IHs=; b=aOuWZrm+9JvqKp3hhFFKW5+hbuQ6AiDG8byWS1AkRq+dYtAFrRz752Sf1s4WZdZpgy vdxDlY2Quz03QaIxN0O8bhAnSlPoIUMiOsxT4f0l/391uQdu0El0YZxYcqFuV2jUKb80 G0oDnKonsi4AepVYtWriQWlq3WmDGbLb3Qya+/UpDwW5bQxRwWoGYqy9JUJB3fhfgueC lWLpir++LiqM+rUSUmD2D+ka/FQiEc7lW75E6GuEoVWj/+N262dxVlDnT6mCDD0/deeX h+tj+7r/FEVCueNn7QQ75fxFEy5jWcnE4vY51keOfsrR6dZzPZzzbOlS+b4C7EI113+Y OlRw== X-Gm-Message-State: AOJu0YydfHDQoYE56G4z7qxLMWEuLy0Is9oy8mitFqtafYnN/VtWTKOR 5C494Y0p5aOjf553cOiClfwSK3pYYFz3dG9/BKw= X-Google-Smtp-Source: AGHT+IHRoFaCnuXTOcsKb1cfXdeZCVgT5C6d0Q6kaqRqE1MAlVmBjOW1bY0yFnIo9wAovgOEdGH9Bw== X-Received: by 2002:a17:903:120b:b0:1b8:6850:c3c4 with SMTP id l11-20020a170903120b00b001b86850c3c4mr13661010plh.22.1692829896879; Wed, 23 Aug 2023 15:31:36 -0700 (PDT) Received: from evan.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id i1-20020a17090332c100b001bbf7fd354csm11335150plr.213.2023.08.23.15.31.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Aug 2023 15:31:36 -0700 (PDT) From: Evan Green To: libc-alpha@sourceware.org Cc: vineetg@rivosinc.com, palmer@rivosinc.com, Florian Weimer , slewis@rivosinc.com, Evan Green Subject: [PATCH v7 3/6] riscv: Add __riscv_hwprobe pointer to ifunc calls Date: Wed, 23 Aug 2023 15:31:12 -0700 Message-Id: <20230823223116.645225-4-evan@rivosinc.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823223116.645225-1-evan@rivosinc.com> References: <20230823223116.645225-1-evan@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-11.7 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org Sender: "Libc-alpha" The new __riscv_hwprobe() function is designed to be used by ifunc selector functions. This presents a challenge for applications and libraries, as ifunc selectors are invoked before all relocations have been performed, so an external call to __riscv_hwprobe() from an ifunc selector won't work. To address this, pass a pointer to the __riscv_hwprobe() vDSO function into ifunc selectors as the second argument (alongside dl_hwcap, which was already being passed). Include a typedef as well for convenience, so that ifunc users don't have to go through contortions to call this routine. Users will need to remember to check the second argument for NULL, both to account for older glibcs that don't pass the function, and older kernels that don't have the vDSO pointer. Signed-off-by: Evan Green --- Changes in v7: - Remove __THROW from function pointer type, as it creates warnings together with __fortified_attr_access. sysdeps/riscv/dl-irel.h | 8 ++++---- sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h | 10 ++++++++++ 2 files changed, 14 insertions(+), 4 deletions(-) diff --git a/sysdeps/riscv/dl-irel.h b/sysdeps/riscv/dl-irel.h index eaeec5467c..2147504458 100644 --- a/sysdeps/riscv/dl-irel.h +++ b/sysdeps/riscv/dl-irel.h @@ -31,10 +31,10 @@ static inline ElfW(Addr) __attribute ((always_inline)) elf_ifunc_invoke (ElfW(Addr) addr) { - /* The second argument is a void pointer to preserve the extension - fexibility. */ - return ((ElfW(Addr) (*) (uint64_t, void *)) (addr)) - (GLRO(dl_hwcap), NULL); + /* The third argument is a void pointer to preserve the extension + flexibility. */ + return ((ElfW(Addr) (*) (uint64_t, void *, void *)) (addr)) + (GLRO(dl_hwcap), GLRO(dl_vdso_riscv_hwprobe), NULL); } static inline void diff --git a/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h b/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h index aa189a4818..fd3be5a411 100644 --- a/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h +++ b/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h @@ -67,6 +67,16 @@ extern int __riscv_hwprobe (struct riscv_hwprobe *__pairs, size_t __pair_count, __fortified_attr_access (__read_write__, 1, 2) __fortified_attr_access (__read_only__, 4, 3); +/* A pointer to the __riscv_hwprobe vDSO function is passed as the second + argument to ifunc selector routines. Include a function pointer type for + convenience in calling the function in those settings. */ +typedef int (*__riscv_hwprobe_t) (struct riscv_hwprobe *__pairs, size_t __pair_count, + size_t __cpu_count, unsigned long int *__cpus, + unsigned int __flags) + __nonnull ((1)) __wur + __fortified_attr_access (__read_write__, 1, 2) + __fortified_attr_access (__read_only__, 4, 3); + __END_DECLS #endif /* sys/hwprobe.h */ From patchwork Wed Aug 23 22:31:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Evan Green X-Patchwork-Id: 1825014 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=rivosinc-com.20221208.gappssmtp.com header.i=@rivosinc-com.20221208.gappssmtp.com header.a=rsa-sha256 header.s=20221208 header.b=XH4YWhrN; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4RWLZC67sCz1yNm for ; Thu, 24 Aug 2023 08:32:35 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 7C978382E6B6 for ; Wed, 23 Aug 2023 22:32:33 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pl1-x630.google.com (mail-pl1-x630.google.com [IPv6:2607:f8b0:4864:20::630]) by sourceware.org (Postfix) with ESMTPS id B4FE03853D1C for ; Wed, 23 Aug 2023 22:31:41 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org B4FE03853D1C Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivosinc.com Received: by mail-pl1-x630.google.com with SMTP id d9443c01a7336-1bdbbede5d4so48942115ad.2 for ; Wed, 23 Aug 2023 15:31:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1692829900; x=1693434700; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ztinmuxaNjH/z1RZboH05tpq5nAt8cssy5r2/1B6f28=; b=XH4YWhrN2sprLvHZoxKJX7rrpYYqqkH8O5/vTp6QmpjlDK+6CR63z2ESimlDviUpqS twjWb/wPsJmLC/BteuRwbmfp9o5DSgRsN7v8pxOSdglqUVbenqR9yuS4LmM9M3M56yBN dBkTylQ50RPrP/s4Kl5yq7sLHeC7N6JvgyN8XH4HQ24HRK3KbZydtdJitFbiJjHbOU3h 2wsnYEY/cUXiHKGCXcNCDkpmQ79MlTr1v563sZNszMSwpBEJsZdw9nvmB5DMPD5iuzg2 OMW5bJer9W7d4/7Z2yoZbQMQofHajUW3QV1h/ql+I9eICNhBPZ4HMPCiM+G4wE+QIF6G Nx7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692829900; x=1693434700; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ztinmuxaNjH/z1RZboH05tpq5nAt8cssy5r2/1B6f28=; b=JbNHE//2WRGzoXGDaEJ9/mUID1JE29sow2b8/vUErfYtxGI1gcbnatd1AI0b3Gl9KP SPI9I5759ttMwxH/Q+2ZoEjdaeQn64dQMKTd2ai2tBhBUY0PdtGU8ivjHVhVpDwe7H+S ShOo1JwhHyPAl1SEAABIpitjBipg9Cda0FpXu2VaCBlv7qd4OGSbE49KcjN4Z7k2yOrK mqAb1Gj7XdN7ySm2NaT/2yqweZ9bjpmZbY2+gTi28Y8okJR3FW/2l/OIMcRcxc1sM/EJ SZQlu+Ji9UM/HU0JhUw20F7mFkEBCOtmo0KljhBgbppTsiSLuPZ0dmH4AmvzG3JCdiy0 /+yw== X-Gm-Message-State: AOJu0YwpyrMOTK4Jwr4im6BH0GUTrLjTNoeGHe1nfVirWhBk76mgtLQ3 vlww8fvz0at69u6vHX0+a/k4dnaNWY2/D57Y33A= X-Google-Smtp-Source: AGHT+IFkHzobO4TjGAbyJxkiBzCN0RsKMM+7YCxkj+1Pxgbkm5UiK21vlw6Jv/F8bzWD02Dlr0Zsbw== X-Received: by 2002:a17:903:41c4:b0:1b8:6a09:9cf9 with SMTP id u4-20020a17090341c400b001b86a099cf9mr18116621ple.26.1692829900389; Wed, 23 Aug 2023 15:31:40 -0700 (PDT) Received: from evan.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id i1-20020a17090332c100b001bbf7fd354csm11335150plr.213.2023.08.23.15.31.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Aug 2023 15:31:40 -0700 (PDT) From: Evan Green To: libc-alpha@sourceware.org Cc: vineetg@rivosinc.com, palmer@rivosinc.com, Florian Weimer , slewis@rivosinc.com, Evan Green Subject: [PATCH v7 4/6] riscv: Enable multi-arg ifunc resolvers Date: Wed, 23 Aug 2023 15:31:13 -0700 Message-Id: <20230823223116.645225-5-evan@rivosinc.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823223116.645225-1-evan@rivosinc.com> References: <20230823223116.645225-1-evan@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-11.0 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, URIBL_BLACK autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org Sender: "Libc-alpha" RISC-V is apparently the first architecture to pass more than one argument to ifunc resolvers. The helper macros in libc-symbols.h, __ifunc_resolver(), __ifunc(), and __ifunc_hidden(), are incompatible with this. These macros have an "arg" (non-final) parameter that represents the parameter signature of the ifunc resolver. The result is an inability to pass the required comma through in a single preprocessor argument. Rearrange the __ifunc_resolver() macro to be variadic, and pass the types as those variable parameters. Move the guts of __ifunc() and __ifunc_hidden() into new macros, __ifunc_args(), and __ifunc_args_hidden(), that pass the variable arguments down through to __ifunc_resolver(). Then redefine __ifunc() and __ifunc_hidden(), which are used in a bunch of places, to simply shuffle the arguments down into __ifunc_args[_hidden]. Finally, define a riscv-ifunc.h header, which provides convenience macros to those looking to write ifunc selectors that use both arguments. Signed-off-by: Evan Green --- (no changes since v6) Changes in v6: - Introduced riscv-ifunc.h for multi-arg ifunc selectors. Note: I opted to create another layer of macros (__ifunc_args()) rather than doing the treewide change to rearrange the signature of __ifunc() and __ifunc_hidden(). If folks like the overall approach but would prefer the treewide change, I can do that too. --- include/libc-symbols.h | 28 +++++++++++++++++----------- sysdeps/riscv/riscv-ifunc.h | 27 +++++++++++++++++++++++++++ 2 files changed, 44 insertions(+), 11 deletions(-) create mode 100644 sysdeps/riscv/riscv-ifunc.h diff --git a/include/libc-symbols.h b/include/libc-symbols.h index 5794614488..36b92039c5 100644 --- a/include/libc-symbols.h +++ b/include/libc-symbols.h @@ -665,9 +665,9 @@ for linking") #endif /* Helper / base macros for indirect function symbols. */ -#define __ifunc_resolver(type_name, name, expr, arg, init, classifier) \ +#define __ifunc_resolver(type_name, name, expr, init, classifier, ...) \ classifier inhibit_stack_protector \ - __typeof (type_name) *name##_ifunc (arg) \ + __typeof (type_name) *name##_ifunc (__VA_ARGS__) \ { \ init (); \ __typeof (type_name) *res = expr; \ @@ -675,13 +675,13 @@ for linking") } #ifdef HAVE_GCC_IFUNC -# define __ifunc(type_name, name, expr, arg, init) \ +# define __ifunc_args(type_name, name, expr, init, ...) \ extern __typeof (type_name) name __attribute__ \ ((ifunc (#name "_ifunc"))); \ - __ifunc_resolver (type_name, name, expr, arg, init, static) + __ifunc_resolver (type_name, name, expr, init, static, __VA_ARGS__) -# define __ifunc_hidden(type_name, name, expr, arg, init) \ - __ifunc (type_name, name, expr, arg, init) +# define __ifunc_args_hidden(type_name, name, expr, init, ...) \ + __ifunc (type_name, name, expr, init, __VA_ARGS__) #else /* Gcc does not support __attribute__ ((ifunc (...))). Use the old behaviour as fallback. But keep in mind that the debug information for the ifunc @@ -692,18 +692,24 @@ for linking") different signatures. (Gcc support is disabled at least on a ppc64le Ubuntu 14.04 system.) */ -# define __ifunc(type_name, name, expr, arg, init) \ +# define __ifunc_args(type_name, name, expr, init, ...) \ extern __typeof (type_name) name; \ - __typeof (type_name) *name##_ifunc (arg) __asm__ (#name); \ - __ifunc_resolver (type_name, name, expr, arg, init,) \ + __typeof (type_name) *name##_ifunc (__VA_ARGS__) __asm__ (#name); \ + __ifunc_resolver (type_name, name, expr, init, , __VA_ARGS__) \ __asm__ (".type " #name ", %gnu_indirect_function"); -# define __ifunc_hidden(type_name, name, expr, arg, init) \ +# define __ifunc_args_hidden(type_name, name, expr, init, ...) \ extern __typeof (type_name) __libc_##name; \ - __ifunc (type_name, __libc_##name, expr, arg, init) \ + __ifunc (type_name, __libc_##name, expr, __VA_INIT__, init) \ strong_alias (__libc_##name, name); #endif /* !HAVE_GCC_IFUNC */ +#define __ifunc(type_name, name, expr, arg, init) \ + __ifunc_args (type_name, name, expr, init, arg) + +#define __ifunc_hidden(type_name, name, expr, arg, init) \ + __ifunc_args_hidden (type_name, expr, init, arg) + /* The following macros are used for indirect function symbols in libc.so. First of all, you need to have the function prototyped somewhere, say in foo.h: diff --git a/sysdeps/riscv/riscv-ifunc.h b/sysdeps/riscv/riscv-ifunc.h new file mode 100644 index 0000000000..7bff591d1e --- /dev/null +++ b/sysdeps/riscv/riscv-ifunc.h @@ -0,0 +1,27 @@ +/* Common definition for ifunc resolvers. Linux/RISC-V version. + This file is part of the GNU C Library. + Copyright (C) 2023 Free Software Foundation, Inc. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include +#include +#include + +#define INIT_ARCH() + +#define riscv_libc_ifunc(name, expr) \ + __ifunc_args (name, name, expr(hwcap, hwprobe), INIT_ARCH, \ + uint64_t hwcap, __riscv_hwprobe_t hwprobe) From patchwork Wed Aug 23 22:31:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Evan Green X-Patchwork-Id: 1825013 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=rivosinc-com.20221208.gappssmtp.com header.i=@rivosinc-com.20221208.gappssmtp.com header.a=rsa-sha256 header.s=20221208 header.b=GTnL/5uA; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4RWLZ40vyTz1yNm for ; Thu, 24 Aug 2023 08:32:28 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 33E723882AD0 for ; Wed, 23 Aug 2023 22:32:26 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pl1-x635.google.com (mail-pl1-x635.google.com [IPv6:2607:f8b0:4864:20::635]) by sourceware.org (Postfix) with ESMTPS id D4ABF3850404 for ; Wed, 23 Aug 2023 22:31:44 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org D4ABF3850404 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivosinc.com Received: by mail-pl1-x635.google.com with SMTP id d9443c01a7336-1bf092a16c9so48249885ad.0 for ; Wed, 23 Aug 2023 15:31:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1692829903; x=1693434703; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=V0QqkAAWiEyNKG93cR9qzCkSsz8yqQX1gfSOrM8MtYM=; b=GTnL/5uAIUg0PDTbZ04nQ9Art/ZO632VYAG4Gi9IhcbZ2D3B8jGePTm5LRMyxvnqtA JkYSoBUBDcqvz+3TuFM1k9vZKan/lKC7qYsnCEBMkBIVBdtjgO/8nMJlY/TecwqScc7F XkPYg8NFulCxQyGCC2iGI0NRtHlvVf38zpmKPbQD+WuQcJObwwTO7r5rvf7h6rqMwldm YngTp6D5G6tH6aKoYWfru6pz2AQFMKaH0HHn8NOYBmXdx8Shiojngk/ZqJf4H7Q0/mgi uTudUZCrt3EloTAqkTOMvQ9TMdRvfUoysiR+R1YKIarBLpfvU64ybBehde+HaB7CkAGN D62A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692829903; x=1693434703; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=V0QqkAAWiEyNKG93cR9qzCkSsz8yqQX1gfSOrM8MtYM=; b=KgwFc6Z72zHMsjc/XrM9l7xGdPWsHSjgHByRicx7+/8GJZZFstfaBKZi7UzCAyjBhh oljxtrFYJLSu2DErijVR3Ts78FtdqjMy7jGDo8SFQqJbJPvFa5zb+MzNFH62Iv/O6MST bJs9g0UFvuJ5CzUifOOh65Fz1cxORmsRPYhGRinISlxD1vM7g54lKgPQIooYSsMNmox6 mSUbvnP413xrp0eYHX/b+u4SofjFTv4zHju0dCqOWMF8Ea83lJCfrMWqvob5bSYfIAnM KdMgT13n0oeywMxQjIZm5d03+DJay5Dm4ctlSBqJw50n4SC52AhmV4gu/dxzb1tW6JA+ Xa7A== X-Gm-Message-State: AOJu0YwiRcqbFXF9FadbEmstG98TLAoyN8Ss4p01DBwtsWKPJd8MwnfP pRrmK08KFZrF1waVnGWJShyqWWeMm9dfguKzZpA= X-Google-Smtp-Source: AGHT+IH+k0cYEgAEKWc2slsS4yjieErRWRhvsHFZFoLKxiA2egjJwVyO1F348EDgL+dcPH8B9SSECw== X-Received: by 2002:a17:902:d2c4:b0:1bd:e64c:5c7e with SMTP id n4-20020a170902d2c400b001bde64c5c7emr15573822plc.61.1692829903539; Wed, 23 Aug 2023 15:31:43 -0700 (PDT) Received: from evan.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id i1-20020a17090332c100b001bbf7fd354csm11335150plr.213.2023.08.23.15.31.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Aug 2023 15:31:43 -0700 (PDT) From: Evan Green To: libc-alpha@sourceware.org Cc: vineetg@rivosinc.com, palmer@rivosinc.com, Florian Weimer , slewis@rivosinc.com, Evan Green Subject: [PATCH v7 5/6] riscv: Add ifunc helper method to hwprobe.h Date: Wed, 23 Aug 2023 15:31:14 -0700 Message-Id: <20230823223116.645225-6-evan@rivosinc.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823223116.645225-1-evan@rivosinc.com> References: <20230823223116.645225-1-evan@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-11.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org Sender: "Libc-alpha" Add a little helper method so it's easier to fetch a single value from the hwprobe function when used within an ifunc selector. Signed-off-by: Evan Green --- Changes in v7: - Introduced static inline helper (Richard) sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h | 24 +++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h b/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h index fd3be5a411..7378b1baa2 100644 --- a/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h +++ b/sysdeps/unix/sysv/linux/riscv/sys/hwprobe.h @@ -22,6 +22,7 @@ #include #include +#include #ifdef __has_include # if __has_include () # include @@ -79,4 +80,27 @@ typedef int (*__riscv_hwprobe_t) (struct riscv_hwprobe *__pairs, size_t __pair_c __END_DECLS +/* Helper function usable from ifunc selectors that probes a single key. */ +static inline int __riscv_hwprobe_one(__riscv_hwprobe_t hwprobe_func, + signed long long int key, + unsigned long long int *value) +{ + struct riscv_hwprobe pair; + int rc; + + if (!hwprobe_func) + return ENOSYS; + + pair.key = key; + rc = hwprobe_func(&pair, 1, 0, NULL, 0); + if (rc) + return rc; + + if (pair.key < 0) + return ENOENT; + + *value = pair.value; + return 0; +} + #endif /* sys/hwprobe.h */ From patchwork Wed Aug 23 22:31:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Evan Green X-Patchwork-Id: 1825012 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=rivosinc-com.20221208.gappssmtp.com header.i=@rivosinc-com.20221208.gappssmtp.com header.a=rsa-sha256 header.s=20221208 header.b=XlzQkYw6; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=8.43.85.97; helo=server2.sourceware.org; envelope-from=libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (ip-8-43-85-97.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4RWLYv46Fdz1yNm for ; Thu, 24 Aug 2023 08:32:19 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 9F9BC3882061 for ; Wed, 23 Aug 2023 22:32:17 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-pl1-x631.google.com (mail-pl1-x631.google.com [IPv6:2607:f8b0:4864:20::631]) by sourceware.org (Postfix) with ESMTPS id DEF1C38768BB for ; Wed, 23 Aug 2023 22:31:47 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org DEF1C38768BB Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=rivosinc.com Received: by mail-pl1-x631.google.com with SMTP id d9443c01a7336-1bbff6b2679so40939605ad.1 for ; Wed, 23 Aug 2023 15:31:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1692829906; x=1693434706; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Pd51pKyOvtqx3uDsEId4w38od4XJbOAao9Ay/wUcVHU=; b=XlzQkYw62e3eKIUKVBMY2X7IdoWnnullVkfqGGlAHJEQ5qIogwvcaxQeR90BGoYV8I DrnEfMFkdeNCnkEJGxiwgqIyZ/PkP6cOi44PvnFkI1UUlNwFghAAGR8BRkTyNwLywwyL +X8UGXxvAPU/0LoWaZmRAcuEnfEt9wQz924DUWSLt7CIxDa0A4+9ROboTSdrqRpSik9S FElyR2zfZDl9XKOUZ4xu+GrxScMOHwFHSazyYhSBOJB/D0rxJsWzkm7QmIL/dUDWNWY7 tdc+BRE5LxXESQv1v6z8ym1c/atgfoWCx0gV+lgMWgC5EsuFMyhDD2m9+k2D6khQjaNZ UHCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692829906; x=1693434706; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Pd51pKyOvtqx3uDsEId4w38od4XJbOAao9Ay/wUcVHU=; b=Rh9p5trpl0cfSlHKQs5KX8cCG4n/9Lwk8tQFvKvYvnIHvzhXGL8v4Fvr535eUpbkVh xbdh3of7gDMU7g4Edqwy2/CEBxBLxxs3V/fz0ZH1svUEgI9CdqRsjdP0asg6WhTg+dHV 5BjHP8kZ+bLSOk/cXBhJaeXWUgZkPlY7kKc4U+HFPlST8E1lI7FO4iPacVKs+OX3zcZl 7Nmz+qa23V9nRrR4z3vF+OznDlIuhewhf/6kDrH58ZvcoGjGsuFYLs+Dh7OzgfSQjURn 1iCRqLE9H/+bVPtOZwLTGsuqeNOUOrAyeK/KLkPYp+C+o1QcgKXCEuTz2HI/H5LD/UzK sClw== X-Gm-Message-State: AOJu0Yy9wWVWhct1ka8WLIZhl9qFpzr3rlApJnEpGvieYE23U5fP5JtO 09+DYULAZ4glNpZRhgfdEX8bqiWgmkFd5kVDPng= X-Google-Smtp-Source: AGHT+IGJ7g80hb6qVFnK1LdNLOHxMkEaeRSOdGA1+tiOdKOqAFvwjad7HzuGWxlTyjjerNBD8w5ahQ== X-Received: by 2002:a17:903:1d2:b0:1b6:80f0:d969 with SMTP id e18-20020a17090301d200b001b680f0d969mr13159850plh.11.1692829906477; Wed, 23 Aug 2023 15:31:46 -0700 (PDT) Received: from evan.ba.rivosinc.com ([66.220.2.162]) by smtp.gmail.com with ESMTPSA id i1-20020a17090332c100b001bbf7fd354csm11335150plr.213.2023.08.23.15.31.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Aug 2023 15:31:46 -0700 (PDT) From: Evan Green To: libc-alpha@sourceware.org Cc: vineetg@rivosinc.com, palmer@rivosinc.com, Florian Weimer , slewis@rivosinc.com, Evan Green Subject: [PATCH v7 6/6] riscv: Add and use alignment-ignorant memcpy Date: Wed, 23 Aug 2023 15:31:15 -0700 Message-Id: <20230823223116.645225-7-evan@rivosinc.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230823223116.645225-1-evan@rivosinc.com> References: <20230823223116.645225-1-evan@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-11.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org Sender: "Libc-alpha" For CPU implementations that can perform unaligned accesses with little or no performance penalty, create a memcpy implementation that does not bother aligning buffers. It will use a block of integer registers, a single integer register, and fall back to bytewise copy for the remainder. Signed-off-by: Evan Green Reviewed-by: Palmer Dabbelt --- Changes in v7: - Use new helper function in memcpy ifunc selector (Richard) Changes in v6: - Fix a couple regressions in the assembly from v5 :/ - Use passed hwprobe pointer in memcpy ifunc selector. Changes in v5: - Do unaligned word access for final trailing bytes (Richard) Changes in v4: - Fixed comment style (Florian) Changes in v3: - Word align dest for large memcpy()s. - Add tags - Remove spurious blank line from sysdeps/riscv/memcpy.c Changes in v2: - Used _MASK instead of _FAST value itself. --- sysdeps/riscv/memcopy.h | 26 ++++ sysdeps/riscv/memcpy.c | 63 ++++++++ sysdeps/riscv/memcpy_noalignment.S | 138 ++++++++++++++++++ sysdeps/unix/sysv/linux/riscv/Makefile | 4 + .../unix/sysv/linux/riscv/memcpy-generic.c | 24 +++ 5 files changed, 255 insertions(+) create mode 100644 sysdeps/riscv/memcopy.h create mode 100644 sysdeps/riscv/memcpy.c create mode 100644 sysdeps/riscv/memcpy_noalignment.S create mode 100644 sysdeps/unix/sysv/linux/riscv/memcpy-generic.c diff --git a/sysdeps/riscv/memcopy.h b/sysdeps/riscv/memcopy.h new file mode 100644 index 0000000000..2b685c8aa0 --- /dev/null +++ b/sysdeps/riscv/memcopy.h @@ -0,0 +1,26 @@ +/* memcopy.h -- definitions for memory copy functions. RISC-V version. + Copyright (C) 2023 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +/* Redefine the generic memcpy implementation to __memcpy_generic, so + the memcpy ifunc can select between generic and special versions. + In rtld, don't bother with all the ifunciness. */ +#if IS_IN (libc) +#define MEMCPY __memcpy_generic +#endif diff --git a/sysdeps/riscv/memcpy.c b/sysdeps/riscv/memcpy.c new file mode 100644 index 0000000000..285ca85f8e --- /dev/null +++ b/sysdeps/riscv/memcpy.c @@ -0,0 +1,63 @@ +/* Multiple versions of memcpy. + All versions must be listed in ifunc-impl-list.c. + Copyright (C) 2017-2023 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#if IS_IN (libc) +/* Redefine memcpy so that the compiler won't complain about the type + mismatch with the IFUNC selector in strong_alias, below. */ +# undef memcpy +# define memcpy __redirect_memcpy +# include +# include +# include +# include +# include + +# define INIT_ARCH() + +extern __typeof (__redirect_memcpy) __libc_memcpy; + +extern __typeof (__redirect_memcpy) __memcpy_generic attribute_hidden; +extern __typeof (__redirect_memcpy) __memcpy_noalignment attribute_hidden; + +static inline __typeof (__redirect_memcpy) * +select_memcpy_ifunc (uint64_t dl_hwcap, __riscv_hwprobe_t hwprobe_func) +{ + unsigned long long int value; + + INIT_ARCH (); + + if (__riscv_hwprobe_one(hwprobe_func, RISCV_HWPROBE_KEY_CPUPERF_0, &value)) + return __memcpy_generic; + + if ((value & RISCV_HWPROBE_MISALIGNED_MASK) == RISCV_HWPROBE_MISALIGNED_FAST) + return __memcpy_noalignment; + + return __memcpy_generic; +} + +riscv_libc_ifunc (__libc_memcpy, select_memcpy_ifunc); + +# undef memcpy +strong_alias (__libc_memcpy, memcpy); +# ifdef SHARED +__hidden_ver1 (memcpy, __GI_memcpy, __redirect_memcpy) + __attribute__ ((visibility ("hidden"))) __attribute_copy__ (memcpy); +# endif + +#endif diff --git a/sysdeps/riscv/memcpy_noalignment.S b/sysdeps/riscv/memcpy_noalignment.S new file mode 100644 index 0000000000..f3bf8e5867 --- /dev/null +++ b/sysdeps/riscv/memcpy_noalignment.S @@ -0,0 +1,138 @@ +/* memcpy for RISC-V, ignoring buffer alignment + Copyright (C) 2023 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +#include +#include + +/* void *memcpy(void *, const void *, size_t) */ +ENTRY (__memcpy_noalignment) + move t6, a0 /* Preserve return value */ + + /* Bail if 0 */ + beqz a2, 7f + + /* Jump to byte copy if size < SZREG */ + li a4, SZREG + bltu a2, a4, 5f + + /* Round down to the nearest "page" size */ + andi a4, a2, ~((16*SZREG)-1) + beqz a4, 2f + add a3, a1, a4 + + /* Copy the first word to get dest word aligned */ + andi a5, t6, SZREG-1 + beqz a5, 1f + REG_L a6, (a1) + REG_S a6, (t6) + + /* Align dst up to a word, move src and size as well. */ + addi t6, t6, SZREG-1 + andi t6, t6, ~(SZREG-1) + sub a5, t6, a0 + add a1, a1, a5 + sub a2, a2, a5 + + /* Recompute page count */ + andi a4, a2, ~((16*SZREG)-1) + beqz a4, 2f + +1: + /* Copy "pages" (chunks of 16 registers) */ + REG_L a4, 0(a1) + REG_L a5, SZREG(a1) + REG_L a6, 2*SZREG(a1) + REG_L a7, 3*SZREG(a1) + REG_L t0, 4*SZREG(a1) + REG_L t1, 5*SZREG(a1) + REG_L t2, 6*SZREG(a1) + REG_L t3, 7*SZREG(a1) + REG_L t4, 8*SZREG(a1) + REG_L t5, 9*SZREG(a1) + REG_S a4, 0(t6) + REG_S a5, SZREG(t6) + REG_S a6, 2*SZREG(t6) + REG_S a7, 3*SZREG(t6) + REG_S t0, 4*SZREG(t6) + REG_S t1, 5*SZREG(t6) + REG_S t2, 6*SZREG(t6) + REG_S t3, 7*SZREG(t6) + REG_S t4, 8*SZREG(t6) + REG_S t5, 9*SZREG(t6) + REG_L a4, 10*SZREG(a1) + REG_L a5, 11*SZREG(a1) + REG_L a6, 12*SZREG(a1) + REG_L a7, 13*SZREG(a1) + REG_L t0, 14*SZREG(a1) + REG_L t1, 15*SZREG(a1) + addi a1, a1, 16*SZREG + REG_S a4, 10*SZREG(t6) + REG_S a5, 11*SZREG(t6) + REG_S a6, 12*SZREG(t6) + REG_S a7, 13*SZREG(t6) + REG_S t0, 14*SZREG(t6) + REG_S t1, 15*SZREG(t6) + addi t6, t6, 16*SZREG + bltu a1, a3, 1b + andi a2, a2, (16*SZREG)-1 /* Update count */ + +2: + /* Remainder is smaller than a page, compute native word count */ + beqz a2, 7f + andi a5, a2, ~(SZREG-1) + andi a2, a2, (SZREG-1) + add a3, a1, a5 + /* Jump directly to last word if no words. */ + beqz a5, 4f + +3: + /* Use single native register copy */ + REG_L a4, 0(a1) + addi a1, a1, SZREG + REG_S a4, 0(t6) + addi t6, t6, SZREG + bltu a1, a3, 3b + + /* Jump directly out if no more bytes */ + beqz a2, 7f + +4: + /* Copy the last word unaligned */ + add a3, a1, a2 + add a4, t6, a2 + REG_L a5, -SZREG(a3) + REG_S a5, -SZREG(a4) + ret + +5: + /* Copy bytes when the total copy is . */ + +#include + +extern __typeof (memcpy) __memcpy_generic; +hidden_proto(__memcpy_generic) + +#include