From patchwork Fri Aug 11 07:14:14 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Siddhesh Poyarekar X-Patchwork-Id: 800434 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=sourceware.org (client-ip=209.132.180.131; helo=sourceware.org; envelope-from=libc-alpha-return-83004-incoming=patchwork.ozlabs.org@sourceware.org; receiver=) Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; secure) header.d=sourceware.org header.i=@sourceware.org header.b="YpKhArLn"; dkim-atps=neutral Received: from sourceware.org (server1.sourceware.org [209.132.180.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3xTGVY0Nl0z9t33 for ; Fri, 11 Aug 2017 17:15:16 +1000 (AEST) DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:from:to:subject:date:message-id:in-reply-to :references; q=dns; s=default; b=G6ALVfTnT1fozTyk40XSZioBLixZ8O4 RQ2RZ7Ma0WnBa5FYFrgBS0IuwJZVIoOT/6bW1QyltUaoRBFBIr9c7QxmCfpGE6hv MiypvEdic5EvS5mc14AuVXR99A0oaTMoMOdctFM4N1IVfQLSDHbR2SzR1plkkBmL eli9R9uhV/Yo= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:from:to:subject:date:message-id:in-reply-to :references; s=default; bh=SutrjhaZxE9JpLt4k7Zq0HSbKCI=; b=YpKhA rLn+aQQKNVEmR+7CenMADFclMpS/CCsNMILfpb83Tju1ujiShwD3HBllVgdXVjMX ISUb4ipBteDiG6ZP5cfdLUpC67wOW1XdOzHUFYaUgnt/KvTWs3YvAkwaodi58+vd 9GTq//U47xG0O56aRl6SFpUYVMNN4YSF40U7CE= Received: (qmail 48915 invoked by alias); 11 Aug 2017 07:14:46 -0000 Mailing-List: contact libc-alpha-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: libc-alpha-owner@sourceware.org Delivered-To: mailing list libc-alpha@sourceware.org Received: (qmail 37582 invoked by uid 89); 11 Aug 2017 07:14:31 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-26.1 required=5.0 tests=BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, RCVD_IN_DNSWL_NONE, SPF_NEUTRAL autolearn=ham version=3.3.2 spammy=increasingly X-HELO: homiemail-a119.g.dreamhost.com From: Siddhesh Poyarekar To: libc-alpha@sourceware.org Subject: [PATCH 2/2] Call the correct memcpy function through mempcpy Date: Fri, 11 Aug 2017 12:44:14 +0530 Message-Id: <1502435654-18032-3-git-send-email-siddhesh@sourceware.org> In-Reply-To: <1502435654-18032-1-git-send-email-siddhesh@sourceware.org> References: <1502435654-18032-1-git-send-email-siddhesh@sourceware.org> The generic mempcpy ends up calling only the generic memcpy and as a result, is unable to return the most optimal results. Add ifunc resolver and entry points for mempcpy to jump into the correct memcpy implementation for the target. * sysdeps/aarch64/multiarch/Makefile * sysdeps/aarch64/multiarch/ifunc-impl-list.c * sysdeps/aarch64/multiarch/mempcpy.c * sysdeps/aarch64/multiarch/mempcpy_chk-nonshared.S * sysdeps/aarch64/multiarch/mempcpy_chk.c * sysdeps/aarch64/multiarch/mempcpy_falkor.S * sysdeps/aarch64/multiarch/mempcpy_generic.S * sysdeps/aarch64/multiarch/mempcpy_thunderx.S --- sysdeps/aarch64/multiarch/Makefile | 5 ++- sysdeps/aarch64/multiarch/ifunc-impl-list.c | 8 ++++ sysdeps/aarch64/multiarch/mempcpy.c | 47 +++++++++++++++++++ sysdeps/aarch64/multiarch/mempcpy_chk-nonshared.S | 28 ++++++++++++ sysdeps/aarch64/multiarch/mempcpy_chk.c | 35 +++++++++++++++ sysdeps/aarch64/multiarch/mempcpy_falkor.S | 23 ++++++++++ sysdeps/aarch64/multiarch/mempcpy_generic.S | 55 +++++++++++++++++++++++ sysdeps/aarch64/multiarch/mempcpy_thunderx.S | 23 ++++++++++ 8 files changed, 222 insertions(+), 2 deletions(-) create mode 100644 sysdeps/aarch64/multiarch/mempcpy.c create mode 100644 sysdeps/aarch64/multiarch/mempcpy_chk-nonshared.S create mode 100644 sysdeps/aarch64/multiarch/mempcpy_chk.c create mode 100644 sysdeps/aarch64/multiarch/mempcpy_falkor.S create mode 100644 sysdeps/aarch64/multiarch/mempcpy_generic.S create mode 100644 sysdeps/aarch64/multiarch/mempcpy_thunderx.S diff --git a/sysdeps/aarch64/multiarch/Makefile b/sysdeps/aarch64/multiarch/Makefile index 8189eb4..5ba6962 100644 --- a/sysdeps/aarch64/multiarch/Makefile +++ b/sysdeps/aarch64/multiarch/Makefile @@ -1,7 +1,8 @@ ifeq ($(subdir),string) -sysdep_routines += memcpy_generic memcpy_thunderx memcpy_falkor +sysdep_routines += memcpy_generic memcpy_thunderx memcpy_falkor \ + mempcpy_generic mempcpy_thunderx mempcpy_falkor endif ifeq ($(subdir),debug) -sysdep_routines += memcpy_chk-nonshared +sysdep_routines += memcpy_chk-nonshared mempcpy_chk-nonshared endif diff --git a/sysdeps/aarch64/multiarch/ifunc-impl-list.c b/sysdeps/aarch64/multiarch/ifunc-impl-list.c index d534818..fd3b305 100644 --- a/sysdeps/aarch64/multiarch/ifunc-impl-list.c +++ b/sysdeps/aarch64/multiarch/ifunc-impl-list.c @@ -46,6 +46,14 @@ __libc_ifunc_impl_list (const char *name, struct libc_ifunc_impl *array, IFUNC_IMPL_ADD (array, i, __memcpy_chk, 1, __memcpy_chk_thunderx) IFUNC_IMPL_ADD (array, i, __memcpy_chk, 1, __memcpy_chk_falkor) IFUNC_IMPL_ADD (array, i, __memcpy_chk, 1, __memcpy_chk_generic)) + IFUNC_IMPL (i, name, mempcpy, + IFUNC_IMPL_ADD (array, i, mempcpy, 1, __mempcpy_thunderx) + IFUNC_IMPL_ADD (array, i, mempcpy, 1, __mempcpy_falkor) + IFUNC_IMPL_ADD (array, i, mempcpy, 1, __mempcpy_generic)) + IFUNC_IMPL (i, name, __mempcpy_chk, + IFUNC_IMPL_ADD (array, i, __mempcpy_chk, 1, __mempcpy_chk_thunderx) + IFUNC_IMPL_ADD (array, i, __mempcpy_chk, 1, __mempcpy_chk_falkor) + IFUNC_IMPL_ADD (array, i, __mempcpy_chk, 1, __mempcpy_chk_generic)) IFUNC_IMPL (i, name, memmove, IFUNC_IMPL_ADD (array, i, memmove, 1, __memmove_thunderx) IFUNC_IMPL_ADD (array, i, memmove, 1, __memmove_generic)) diff --git a/sysdeps/aarch64/multiarch/mempcpy.c b/sysdeps/aarch64/multiarch/mempcpy.c new file mode 100644 index 0000000..3492f59 --- /dev/null +++ b/sysdeps/aarch64/multiarch/mempcpy.c @@ -0,0 +1,47 @@ +/* Multiple versions of mempcpy. AARCH64 version. + Copyright (C) 2017 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +/* Define multiple versions only for the definition in libc. */ + +#if IS_IN (libc) +/* Redefine mempcpy so that the compiler won't complain about the type + mismatch with the IFUNC selector in strong_alias, below. */ +# undef mempcpy +# undef __mempcpy +# define mempcpy __redirect_mempcpy +# include +# include + +extern __typeof (__redirect_mempcpy) __mempcpy; + +extern __typeof (__redirect_mempcpy) __mempcpy_generic attribute_hidden; +extern __typeof (__redirect_mempcpy) __mempcpy_thunderx attribute_hidden; +extern __typeof (__redirect_mempcpy) __mempcpy_falkor attribute_hidden; + +libc_ifunc (__mempcpy, + (IS_THUNDERX (midr) + ? __mempcpy_thunderx + : (IS_FALKOR (midr) + ? __mempcpy_falkor + : __mempcpy_generic))); + +# undef mempcpy +strong_alias (__mempcpy, mempcpy); +#else +# include +#endif diff --git a/sysdeps/aarch64/multiarch/mempcpy_chk-nonshared.S b/sysdeps/aarch64/multiarch/mempcpy_chk-nonshared.S new file mode 100644 index 0000000..5a6afdb --- /dev/null +++ b/sysdeps/aarch64/multiarch/mempcpy_chk-nonshared.S @@ -0,0 +1,28 @@ +/* Non-shared version of mempcpy_chk. + Copyright (C) 2017 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +#include + +#if IS_IN (libc) && !defined SHARED +ENTRY_ALIGN (__mempcpy_chk, 6) + cmp x3, x2 + b.lo __chk_fail + b mempcpy + nop +END (__mempcpy_chk) +#endif diff --git a/sysdeps/aarch64/multiarch/mempcpy_chk.c b/sysdeps/aarch64/multiarch/mempcpy_chk.c new file mode 100644 index 0000000..1b4bf15 --- /dev/null +++ b/sysdeps/aarch64/multiarch/mempcpy_chk.c @@ -0,0 +1,35 @@ +/* Multiple versions of mempcpy. AARCH64 version. + Copyright (C) 2017 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, see + . */ + +/* Define multiple versions only for the definition in libc. */ + +#if IS_IN (libc) && defined SHARED +# include +# include + +extern __typeof (__mempcpy_chk) __mempcpy_chk_generic attribute_hidden; +extern __typeof (__mempcpy_chk) __mempcpy_chk_thunderx attribute_hidden; +extern __typeof (__mempcpy_chk) __mempcpy_chk_falkor attribute_hidden; + +libc_ifunc (__mempcpy_chk, + (IS_THUNDERX (midr) + ? __mempcpy_chk_thunderx + : (IS_FALKOR (midr) + ? __mempcpy_chk_falkor + : __mempcpy_chk_generic))); +#endif diff --git a/sysdeps/aarch64/multiarch/mempcpy_falkor.S b/sysdeps/aarch64/multiarch/mempcpy_falkor.S new file mode 100644 index 0000000..4cf822d --- /dev/null +++ b/sysdeps/aarch64/multiarch/mempcpy_falkor.S @@ -0,0 +1,23 @@ +/* Optimized mempcpy for Qualcomm Falkor processor. + Copyright (C) 2017 Free Software Foundation, Inc. + + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +#define MEMPCPY __mempcpy_falkor +# define MEMPCPY_CHK __mempcpy_chk_falkor +#define MEMCPY __memcpy_falkor +#include diff --git a/sysdeps/aarch64/multiarch/mempcpy_generic.S b/sysdeps/aarch64/multiarch/mempcpy_generic.S new file mode 100644 index 0000000..2f4e8ad --- /dev/null +++ b/sysdeps/aarch64/multiarch/mempcpy_generic.S @@ -0,0 +1,55 @@ +/* Generic mempcpy to select via IFUNC. + Copyright (C) 2017 Free Software Foundation, Inc. + + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +#include + +#if IS_IN (libc) && !defined MEMPCPY +# define MEMPCPY __mempcpy_generic +# define MEMPCPY_CHK __mempcpy_chk_generic +# define MEMCPY __memcpy_generic + +# undef libc_hidden_builtin_def +# define libc_hidden_builtin_def(name) + +# ifdef SHARED + .globl __GI_mempcpy; __GI_mempcpy = __mempcpy_generic +# endif + +#endif + +/* Push the sum of the destination and count on to stack and return it after + memcpy is done. We could avoid the register spill by ensuring that the + memcpy does not use one of the scratch registers, but that could get + increasingly error-prone when we add new implementations for memcpy. */ + +#ifdef SHARED +ENTRY_ALIGN (MEMPCPY_CHK, 4) + cmp x3, x2 + b.lo __chk_fail +END (MEMPCPY_CHK) +#endif + +ENTRY_ALIGN (MEMPCPY, 4) + add x15, x0, x2 + stp x15, x30, [sp, -16] + bl MEMCPY + ldp x0, x30, [sp, -16] + ret +END (MEMPCPY) +libc_hidden_builtin_def (MEMPCPY) diff --git a/sysdeps/aarch64/multiarch/mempcpy_thunderx.S b/sysdeps/aarch64/multiarch/mempcpy_thunderx.S new file mode 100644 index 0000000..41f1b2f --- /dev/null +++ b/sysdeps/aarch64/multiarch/mempcpy_thunderx.S @@ -0,0 +1,23 @@ +/* A Thunderx Optimized mempcpy implementation for AARCH64. + Copyright (C) 2017 Free Software Foundation, Inc. + + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library. If not, see + . */ + +#define MEMPCPY __mempcpy_thunderx +# define MEMPCPY_CHK __mempcpy_chk_thunderx +#define MEMCPY __memcpy_thunderx +#include