From patchwork Wed Jun 14 18:30:45 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Palmer Dabbelt X-Patchwork-Id: 775923 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from sourceware.org (server1.sourceware.org [209.132.180.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3wnwKG2skSz9s3s for ; Thu, 15 Jun 2017 04:34:42 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; secure) header.d=sourceware.org header.i=@sourceware.org header.b="avIidRnr"; dkim-atps=neutral DomainKey-Signature: a=rsa-sha1; c=nofws; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:from:to:cc:cc:cc:cc:subject:date:message-id :in-reply-to:references; q=dns; s=default; b=rl/qt2/oPFTq9YqeBcV OqT71wULL2T6MQ72gxWlfJgjptWctukyMF7hqpalbVJV7rDslh3eOZZt5OxfXzYS 0HAb2BoJX4Qhcqcn5OFxUhyYGeC+GDvwUWJgZMOZ1hzQ8deeeVsQqXqe7c6aibJo QaI5oeSmYvmLXBhy35LY8GfM= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sourceware.org; h=list-id :list-unsubscribe:list-subscribe:list-archive:list-post :list-help:sender:from:to:cc:cc:cc:cc:subject:date:message-id :in-reply-to:references; s=default; bh=Xv4W3VyKMXiawupykYCEZDn/m sU=; b=avIidRnr5bzd5VXhBfQn173pb/UYaPoY4pY094dLdujngFozPDR40xPim pvOii94f8amFuNcE6RmgmNhp/YR8UkKOycJ3mT+BCiAlb5B2G3cGs3GxiODc08kS 000aJG1Id7IxXlJJRkUNJBVMSBl6Wn525Y5+f9WkugdCceZikM= Received: (qmail 98902 invoked by alias); 14 Jun 2017 18:33:21 -0000 Mailing-List: contact libc-alpha-help@sourceware.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Subscribe: List-Archive: List-Post: List-Help: , Sender: libc-alpha-owner@sourceware.org Delivered-To: mailing list libc-alpha@sourceware.org Received: (qmail 98633 invoked by uid 89); 14 Jun 2017 18:33:19 -0000 Authentication-Results: sourceware.org; auth=none X-Spam-SWARE-Status: No, score=-24.2 required=5.0 tests=AWL, BAYES_00, GIT_PATCH_0, GIT_PATCH_1, GIT_PATCH_2, GIT_PATCH_3, RCVD_IN_DNSWL_NONE, SPF_PASS, UNSUBSCRIBE_BODY autolearn=ham version=3.3.2 spammy= X-HELO: mail-pf0-f196.google.com X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:cc:cc:cc:subject:date:message-id :in-reply-to:references; bh=+z9OFq+8/woEUyhJ4yhe9qC5kvEOkYufCvLp+7wxHDY=; b=YaAmzdA/v/DEjNZiORtj7mCtMfWaiuBGXtrdD0D1jDL6KqXHbuqs28ykzyYZWTtd6j gAIW2ZIGnoZGPMlwYHlxl1g80qimLH7bejamNkGPMH5b/80iG2hv27NgjKHVTo2q930g pIn6SioPFtoBjfw2MCj75zWDqI/8LGWhWvF6b/ivhLk6IwfOztUJ8z8Ad7AvJqS+Ud5Z oMPFYwnw7hi+cwUInsyx94VQBRDXV5EHyPLoVenefN3JjoSqhIF39+/uUKKwBqvvt04L IwOQfBisz/AZl/i1LGJiug2n1sVH+EacL4UKLCDlbal9uq+ttpgjdYxiYxox6H+1jj5U nD1g== X-Gm-Message-State: AKS2vOzCamYumR5BLzYoLu1F/ZlgNRjAX8zLd/e/Fv7o4hVHyNhCjVAp lUHb1UxaHN6BVG7UInel5g== X-Received: by 10.84.229.143 with SMTP id c15mr1545123plk.27.1497465195605; Wed, 14 Jun 2017 11:33:15 -0700 (PDT) From: Palmer Dabbelt To: libc-alpha@sourceware.org Cc: Andrew Waterman Cc: patches@groups.riscv.org Cc: Darius Rad Cc: Palmer Dabbelt Subject: [PATCH 09/12] RISC-V: Atomic and Locking Routines Date: Wed, 14 Jun 2017 11:30:45 -0700 Message-Id: <20170614183048.11040-10-palmer@dabbelt.com> In-Reply-To: <20170614183048.11040-1-palmer@dabbelt.com> References: <20170614183048.11040-1-palmer@dabbelt.com> This patch implements various atomic and locking routines on RISC-V, either via the A extension (when present) or via a Linux system call that does a compare-and-exchange. This contains both the library routines and the syscall wrapper. --- sysdeps/riscv/atomic-machine.h | 132 +++++++++++++++++++++++++ sysdeps/riscv/nptl/bits/pthreadtypes-arch.h | 98 ++++++++++++++++++ sysdeps/riscv/nptl/bits/semaphore.h | 33 +++++++ sysdeps/riscv/nptl/libc-lowlevellock.c | 8 ++ sysdeps/riscv/nptl/pthread_spin_lock.c | 43 ++++++++ sysdeps/riscv/nptl/pthread_spin_trylock.c | 43 ++++++++ sysdeps/unix/sysv/linux/riscv/atomic-machine.h | 53 ++++++++++ 7 files changed, 410 insertions(+) create mode 100644 sysdeps/riscv/atomic-machine.h create mode 100644 sysdeps/riscv/nptl/bits/pthreadtypes-arch.h create mode 100644 sysdeps/riscv/nptl/bits/semaphore.h create mode 100644 sysdeps/riscv/nptl/libc-lowlevellock.c create mode 100644 sysdeps/riscv/nptl/pthread_spin_lock.c create mode 100644 sysdeps/riscv/nptl/pthread_spin_trylock.c create mode 100644 sysdeps/unix/sysv/linux/riscv/atomic-machine.h diff --git a/sysdeps/riscv/atomic-machine.h b/sysdeps/riscv/atomic-machine.h new file mode 100644 index 0000000000..c88dc1ce33 --- /dev/null +++ b/sysdeps/riscv/atomic-machine.h @@ -0,0 +1,132 @@ +/* Low-level functions for atomic operations. RISC-V version. + Copyright (C) 2011-2016 Free Software Foundation, Inc. + + Contributed by Andrew Waterman (andrew@sifive.com). + + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, write to the Free + Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA + 02111-1307 USA. */ + +#ifndef _RISCV_BITS_ATOMIC_H +#define _RISCV_BITS_ATOMIC_H 1 + +#include + +typedef int32_t atomic32_t; +typedef uint32_t uatomic32_t; +typedef int_fast32_t atomic_fast32_t; +typedef uint_fast32_t uatomic_fast32_t; + +typedef int64_t atomic64_t; +typedef uint64_t uatomic64_t; +typedef int_fast64_t atomic_fast64_t; +typedef uint_fast64_t uatomic_fast64_t; + +typedef intptr_t atomicptr_t; +typedef uintptr_t uatomicptr_t; +typedef intmax_t atomic_max_t; +typedef uintmax_t uatomic_max_t; + +#define atomic_full_barrier() __sync_synchronize() + +#ifdef __riscv_atomic + +#define __HAVE_64B_ATOMICS (__riscv_xlen >= 64) +#define USE_ATOMIC_COMPILER_BUILTINS 1 + +/* MIPS uses swap on machines that have it, and uses CAS on machines that + * don't. This, we use amoswap when the A extension is enabled, and fall back + * to the atomic system call when the A extension is disabled. */ +#ifdef __riscv_atomic +# define ATOMIC_EXCHANGE_USES_CAS 0 +#else +# define ATOMIC_EXCHANGE_USES_CAS 1 +#endif + +#define asm_amo(which, ordering, mem, value) ({ \ + typeof(*mem) __tmp; \ + if (sizeof(__tmp) == 4) \ + asm volatile (which ".w" ordering "\t%0, %z2, %1" \ + : "=r"(__tmp), "+A"(*(mem)) \ + : "rJ"(value)); \ + else if (sizeof(__tmp) == 8) \ + asm volatile (which ".d" ordering "\t%0, %z2, %1" \ + : "=r"(__tmp), "+A"(*(mem)) \ + : "rJ"(value)); \ + else \ + abort(); \ + __tmp; }) + +/* Atomic compare and exchange. */ + +#define atomic_cas(ordering, mem, newval, oldval) ({ \ + typeof(*mem) __tmp; \ + int __tmp2; \ + if (sizeof(__tmp) == 4) \ + asm volatile ("1: lr.w" ordering "\t%0, %2\n" \ + "bne\t%0, %z4, 1f\n" \ + "sc.w" ordering "\t%1, %z3, %2\n" \ + "bnez\t%1, 1b\n" \ + "1:" \ + : "=&r"(__tmp), "=&r"(__tmp2), "+A"(*(mem)) \ + : "rJ"(newval), "rJ"(oldval)); \ + else if (sizeof(__tmp) == 8) \ + asm volatile ("1: lr.d" ordering "\t%0, %2\n" \ + "bne\t%0, %z4, 1f\n" \ + "sc.d" ordering "\t%1, %z3, %2\n" \ + "bnez\t%1, 1b\n" \ + "1:" \ + : "=&r"(__tmp), "=&r"(__tmp2), "+A"(*(mem)) \ + : "rJ"(newval), "rJ"(oldval)); \ + else \ + abort(); \ + __tmp; }) + +#define atomic_compare_and_exchange_val_acq(mem, newval, oldval) \ + atomic_cas(".aq", mem, newval, oldval) + +#define atomic_compare_and_exchange_val_rel(mem, newval, oldval) \ + atomic_cas(".rl", mem, newval, oldval) + +/* Atomic exchange (without compare). */ + +#define atomic_exchange_acq(mem, value) asm_amo("amoswap", ".aq", mem, value) +#define atomic_exchange_rel(mem, value) asm_amo("amoswap", ".rl", mem, value) + + +/* Atomically add value and return the previous (unincremented) value. */ + +#define atomic_exchange_and_add(mem, value) asm_amo("amoadd", "", mem, value) + +#define atomic_max(mem, value) asm_amo("amomaxu", "", mem, value) +#define atomic_min(mem, value) asm_amo("amominu", "", mem, value) + +#define atomic_bit_test_set(mem, bit) \ + ({ typeof(*mem) __mask = (typeof(*mem))1 << (bit); \ + asm_amo("amoor", "", mem, __mask) & __mask; }) + +#define catomic_exchange_and_add(mem, value) \ + atomic_exchange_and_add(mem, value) +#define catomic_max(mem, value) atomic_max(mem, value) + +#else /* __riscv_atomic */ + +#define __HAVE_64B_ATOMICS 0 +#define USE_ATOMIC_COMPILER_BUILTINS 0 + +#endif /* !__riscv_atomic */ + +#endif /* bits/atomic.h */ diff --git a/sysdeps/riscv/nptl/bits/pthreadtypes-arch.h b/sysdeps/riscv/nptl/bits/pthreadtypes-arch.h new file mode 100644 index 0000000000..4473b0891a --- /dev/null +++ b/sysdeps/riscv/nptl/bits/pthreadtypes-arch.h @@ -0,0 +1,98 @@ +/* Machine-specific pthread type layouts. RISC-V version. + Copyright (C) 2011-2014, 2017 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, write to the Free + Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA + 02111-1307 USA. */ + +#ifndef _BITS_PTHREADTYPES_ARCH_H +#define _BITS_PTHREADTYPES_ARCH_H 1 + +#include + +#if __riscv_xlen == 64 +# define __SIZEOF_PTHREAD_ATTR_T 56 +# define __SIZEOF_PTHREAD_MUTEX_T 40 +# define __SIZEOF_PTHREAD_MUTEXATTR_T 4 +# define __SIZEOF_PTHREAD_COND_T 48 +# define __SIZEOF_PTHREAD_CONDATTR_T 4 +# define __SIZEOF_PTHREAD_RWLOCK_T 56 +# define __SIZEOF_PTHREAD_RWLOCKATTR_T 8 +# define __SIZEOF_PTHREAD_BARRIER_T 32 +# define __SIZEOF_PTHREAD_BARRIERATTR_T 4 +#else +# define __SIZEOF_PTHREAD_ATTR_T 36 +# define __SIZEOF_PTHREAD_MUTEX_T 24 +# define __SIZEOF_PTHREAD_MUTEXATTR_T 4 +# define __SIZEOF_PTHREAD_COND_T 48 +# define __SIZEOF_PTHREAD_CONDATTR_T 4 +# define __SIZEOF_PTHREAD_RWLOCK_T 32 +# define __SIZEOF_PTHREAD_RWLOCKATTR_T 8 +# define __SIZEOF_PTHREAD_BARRIER_T 20 +# define __SIZEOF_PTHREAD_BARRIERATTR_T 4 +#endif + +#define __PTHREAD_COMPAT_PADDING_MID +#define __PTHREAD_COMPAT_PADDING_END +#define __PTHREAD_MUTEX_LOCK_ELISION 0 + +#define __LOCK_ALIGNMENT +#define __ONCE_ALIGNMENT + +struct __pthread_rwlock_arch_t +{ +# if __riscv_xlen == 64 + unsigned int __readers; + unsigned int __writers; + unsigned int __wrphase_futex; + unsigned int __writers_futex; + int __cur_writer; + int __shared; + int __pad3; + int __pad4; + unsigned long int __pad1; + unsigned long int __pad2; + /* FLAGS must stay at this position in the structure to maintain + binary compatibility. */ + unsigned int __flags; +# else + unsigned int __readers; + unsigned int __writers; + unsigned int __wrphase_futex; + unsigned int __writers_futex; + int __cur_writer; + int __pad3; +#if __BYTE_ORDER == __BIG_ENDIAN + unsigned char __pad1; + unsigned char __pad2; + unsigned char __shared; + /* FLAGS must stay at this position in the structure to maintain + binary compatibility. */ + unsigned char __flags; +#else + /* FLAGS must stay at this position in the structure to maintain + binary compatibility. */ + unsigned char __flags; + unsigned char __shared; + unsigned char __pad1; + unsigned char __pad2; +#endif + int __writer; +# endif +}; + +#define __PTHREAD_RWLOCK_ELISION_EXTRA 0 + +#endif /* bits/pthreadtypes.h */ diff --git a/sysdeps/riscv/nptl/bits/semaphore.h b/sysdeps/riscv/nptl/bits/semaphore.h new file mode 100644 index 0000000000..003c9c2bcc --- /dev/null +++ b/sysdeps/riscv/nptl/bits/semaphore.h @@ -0,0 +1,33 @@ +/* Copyright (C) 2002, 2005, 2007, 2017 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, write to the Free + Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA + 02111-1307 USA. */ + +#ifndef _SEMAPHORE_H +# error "Never use directly; include instead." +#endif + +#define __SIZEOF_SEM_T (4 * __SIZEOF_POINTER__) + +/* Value returned if `sem_open' failed. */ +#define SEM_FAILED ((sem_t *) 0) + + +typedef union +{ + char __size[__SIZEOF_SEM_T]; + long int __align; +} sem_t; diff --git a/sysdeps/riscv/nptl/libc-lowlevellock.c b/sysdeps/riscv/nptl/libc-lowlevellock.c new file mode 100644 index 0000000000..0ecd41e630 --- /dev/null +++ b/sysdeps/riscv/nptl/libc-lowlevellock.c @@ -0,0 +1,8 @@ +/* This kludge works around a libpthread static linking problem: + https://sourceware.org/bugzilla/show_bug.cgi?id=15648 */ + +#ifndef SHARED +# define __lll_lock_wait_private weak_function __lll_lock_wait_private +#endif + +#include diff --git a/sysdeps/riscv/nptl/pthread_spin_lock.c b/sysdeps/riscv/nptl/pthread_spin_lock.c new file mode 100644 index 0000000000..dd5ffae97f --- /dev/null +++ b/sysdeps/riscv/nptl/pthread_spin_lock.c @@ -0,0 +1,43 @@ +/* Copyright (C) 2005 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, write to the Free + Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA + 02111-1307 USA. */ + +#ifdef __riscv_atomic + +#include + +int pthread_spin_lock(pthread_spinlock_t* lock) +{ + int tmp1, tmp2; + + asm volatile ("\n\ + 1:lw %0, 0(%2)\n\ + li %1, %3\n\ + bnez %0, 1b\n\ + amoswap.w.aq %0, %1, 0(%2)\n\ + bnez %0, 1b" + : "=&r"(tmp1), "=&r"(tmp2) : "r"(lock), "i"(EBUSY) + ); + + return tmp1; +} + +#else /* __riscv_atomic */ + +#include + +#endif /* !__riscv_atomic */ diff --git a/sysdeps/riscv/nptl/pthread_spin_trylock.c b/sysdeps/riscv/nptl/pthread_spin_trylock.c new file mode 100644 index 0000000000..9bd5d80fd6 --- /dev/null +++ b/sysdeps/riscv/nptl/pthread_spin_trylock.c @@ -0,0 +1,43 @@ +/* Copyright (C) 2005 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, write to the Free + Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA + 02111-1307 USA. */ + +#ifdef __riscv_atomic + +#include + +int pthread_spin_trylock(pthread_spinlock_t* lock) +{ + int tmp1, tmp2; + + asm volatile ("\n\ + lw %0, 0(%2)\n\ + li %1, %3\n\ + bnez %0, 1f\n\ + amoswap.w.aq %0, %1, 0(%2)\n\ + 1:" + : "=&r"(tmp1), "=&r"(tmp2) : "r"(lock), "i"(EBUSY) + ); + + return tmp1; +} + +#else /* __riscv_atomic */ + +#include + +#endif /* !__riscv_atomic */ diff --git a/sysdeps/unix/sysv/linux/riscv/atomic-machine.h b/sysdeps/unix/sysv/linux/riscv/atomic-machine.h new file mode 100644 index 0000000000..5ce6f8da40 --- /dev/null +++ b/sysdeps/unix/sysv/linux/riscv/atomic-machine.h @@ -0,0 +1,53 @@ +/* Low-level functions for atomic operations. RISC-V version. + Copyright (C) 2014-2017 Free Software Foundation, Inc. + This file is part of the GNU C Library. + + The GNU C Library is free software; you can redistribute it and/or + modify it under the terms of the GNU Lesser General Public + License as published by the Free Software Foundation; either + version 2.1 of the License, or (at your option) any later version. + + The GNU C Library is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + Lesser General Public License for more details. + + You should have received a copy of the GNU Lesser General Public + License along with the GNU C Library; if not, write to the Free + Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA + 02111-1307 USA. */ + +#ifndef _LINUX_RISCV_BITS_ATOMIC_H +#define _LINUX_RISCV_BITS_ATOMIC_H 1 + +#include_next + +#ifndef __riscv_atomic + +#include +#include + +#define __arch_compare_and_exchange_val_8_acq(mem, newval, oldval) \ + (abort (), (__typeof (*mem)) 0) + +#define __arch_compare_and_exchange_val_16_acq(mem, newval, oldval) \ + (abort (), (__typeof (*mem)) 0) + +/* The only basic operation needed is compare and exchange. */ +#define __arch_compare_and_exchange_val_32_acq(mem, newval, oldval) \ + ({ \ + INTERNAL_SYSCALL_DECL (__err); \ + (__typeof (*mem)) INTERNAL_SYSCALL (sysriscv, __err, 4, \ + RISCV_ATOMIC_CMPXCHG, mem, oldval, newval); \ + }) + +#define __arch_compare_and_exchange_val_64_acq(mem, newval, oldval) \ + ({ \ + INTERNAL_SYSCALL_DECL (__err); \ + (__typeof (*mem)) INTERNAL_SYSCALL (sysriscv, __err, 4, \ + RISCV_ATOMIC_CMPXCHG64, mem, oldval, newval); \ + }) + +#endif /* __riscv_atomic */ + +#endif /* bits/atomic.h */