From patchwork Mon Jul 11 03:04:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1654615 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=OWg56A2F; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lh80S4B7Hz9ryY for ; Mon, 11 Jul 2022 13:06:00 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Lh80S3LNwz3c1m for ; Mon, 11 Jul 2022 13:06:00 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=OWg56A2F; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::430; helo=mail-pf1-x430.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=OWg56A2F; dkim-atps=neutral Received: from mail-pf1-x430.google.com (mail-pf1-x430.google.com [IPv6:2607:f8b0:4864:20::430]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Lh7zP64DMz3bv2 for ; Mon, 11 Jul 2022 13:05:05 +1000 (AEST) Received: by mail-pf1-x430.google.com with SMTP id e16so3685383pfm.11 for ; Sun, 10 Jul 2022 20:05:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7nFI4sp9mFY/FJDB0cZhkvVcIpsx+sF5+0w/jBHOM+4=; b=OWg56A2FHrOuDWlneASsVMJVtcsHCDHWUkbgzX4O5vDjcYpJJLRkucB2dn/PQ6Mhhb iKx1Q7eMq3h27EBFKHg94ivSiWurhTRLbKs6QdNbmtTfQCb9ZK6AfPCAz67sZ/mQs74c fz03TsIQq4aEHIDvfKdO/o7r10jdUyU0w3qDI6XYM0oS7znDV6CySsQET9udeqFICoXG dCVHQG5EItblRSwuVHn77HM9xCSClwV5muc+kGYSix1IBqmq93PAvSeEkzI9uDSP5n6V qKuTaCji43aft7I3Y81rR/r6YtF6g8k7fnc/gbKdZ7QofiAXmt7UIV15AFkWZoRSeA68 68Eg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7nFI4sp9mFY/FJDB0cZhkvVcIpsx+sF5+0w/jBHOM+4=; b=vfiqwiiaaSUg1N+eiDJsBn2NM/fxoX7Bvsokh2RdSm8wgUKejIf8U/p7gJuwZdQ0B/ 4hD9F/mvrjLueYOt1j2zATqjADwkMYKBJKDudF+MkU7vGcZzdyU9w8ih93R0vZ2LlRPT /RRPI2bI3QhSIfekQI0xZ2QzyEQo2UVVwYQd6QH+5Xq9+QngU+MLFG/UgeM8Ghgg+SGs xJKR1YnToTrbKyXGvqNdOV0bVGpSFbZ3R56/iCMWcPyZngUDp5s3+y5g/Lmbflr2CXqa o5BI1qPJnBqtGObLDXkyxv/vvZ2oeMslUu8GV5wpdQPBQBjGiAaTNDpnrcwd1TsHunDm MgxQ== X-Gm-Message-State: AJIora+o+CASGW/Lxsqsfph86bE6mpMUoYTnlYnIjwzeCKrQA0iUtDJK 4M2rjv6DaPOMETUJuAVPsji1AWw8Xlc= X-Google-Smtp-Source: AGRyM1tzVV3oSRYBVrOGt5GXaoazJ6m5nvIiTNnnCTDeRtYgcynOjblUmnPBaPKzREnCX01nHr/07Q== X-Received: by 2002:a63:164d:0:b0:416:4bc:1c28 with SMTP id 13-20020a63164d000000b0041604bc1c28mr2281668pgw.302.1657508703206; Sun, 10 Jul 2022 20:05:03 -0700 (PDT) Received: from bobo.ibm.com ([203.220.77.143]) by smtp.gmail.com with ESMTPSA id o4-20020a17090a5b0400b001ef8c81959dsm3562232pji.16.2022.07.10.20.05.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Jul 2022 20:05:02 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 01/14] powerpc/qspinlock: powerpc qspinlock implementation Date: Mon, 11 Jul 2022 13:04:40 +1000 Message-Id: <20220711030453.150644-2-npiggin@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220711030453.150644-1-npiggin@gmail.com> References: <20220711030453.150644-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Add a powerpc specific implementation of queued spinlocks. This is the build framework with a very simple (non-queued) spinlock implementation to begin with. Later changes add queueing, and other features and optimisations one-at-a-time. It is done this way to more easily see how the queued spinlocks are built, and to make performance and correctness bisects more useful. Generic PV qspinlock code is causing latency / starvation regressions on large systems that are resulting in hard lockups reported (mostly in pathoogical cases). The generic qspinlock code has a number of issues important for powerpc hardware and hypervisors that aren't easily solved without changing code that would impact other architectures. Follow s390's lead and implement our own for now. Issues for powerpc using generic qspinlocks: - The previous lock value should not be loaded with simple loads, and need not be passed around from previous loads or cmpxchg results, because powerpc uses ll/sc-style atomics which can perform more complex operations that do not require this. powerpc implementations tend to prefer loads use larx for improved coherency performance. - The queueing process should absolutely minimise the number of stores to the lock word to reduce exclusive coherency probes, important for large system scalability. The pending logic is counter productive here. - Non-atomic unlock for paravirt locks is important (atomic instructions tend to still be more expensive than x86 CPUs). - Yielding to the lock owner is important in the oversubscribed paravirt case, which requires storing the owner CPU in the lock word. - More control of lock stealing for the paravirt case is important to keep latency down on large systems. - The lock acquisition operation should always be made with a special variant of atomic instructions with the lock hint bit set, including (especially) in the queueing paths. This is more a matter of adding more arch lock helpers so not an insurmountable problem for generic code. Testing a paravirt (KVM guest) system with 512 vCPUs and 488 hardware threads on a 4-socket POWER10. The purpose of these tests is not to show throughput advantage because they are not representative of a real workload and quite spinlock bound, but rather to show some latency and fairness examples in pathological cases for undersubscribed and oversubscribed guest situations: will-it-scale/open1_threads -t64 min max total simple 4897 6066 344628 vanilla 3821 4100 251680 patched 5974 6995 415986 will-it-scale/open1_threads -t512 min max total simple 109 477 205022 vanilla 2 14675 571786 patched 227 711 267419 lockstorm is a simple in-kernel test that just has threads hammering on a lock and collecting statistics (wait: arrival->acquire time, starve: number of times the lock was acquired while we waited, sequential: number of times we got the lock in a row): lockstorm cpus=0-63 min max total max wait max starve max sequential simple 124831 155101 8993621 1866us 16714 500 vanilla 81768 90426 5524856 1167us 8134 1087 patched 134411 161234 9622482 1996us 19023 437 lockstorm lock stress test cpus=0-511 min max total max wait max starve max sequential simple 1442 3127 1434772 60922us 65320 590 vanilla 6 202616 7090954 580014us 3829074 282107 patched 1801 6296 2292002 53165us 103015 430 dbench 64 tput (MB/s) max-lat (ms) simple 6224.33 24.76 vanilla 6951.91 4.73 patched 8416.77 18.59 dbench 512 tput (MB/s) max-lat (ms) simple 4015.59 503.19 vanilla 4339.17 461.06 patched 5396.53 677.11 - qspin-try.patch - powerpc-qspl.patch - qspl-owner-cpu.patch - owner-cpu-cleanup.patch - qspl-owner-spin-queueud.patch - yield-prev-mcs.patch - qspl-steal.patch - qspl-clear-mustq.patch - fixes.patch - qspl-stealer-no-mustq.patch - fixes2 - fixes3 - rewrite - tuning.patch - qspl-tuning.patch - yield-clear-mustq.patch - mcs-no-yield-owner.patch - propagate-yield-cpu.patch - pv-tune-spins.patch - set-must-q-cleanup.patch - cleanup-publish.patch - fix - cleanup.patch - more-tunables.patch - tuning-and-propagate.patch - tail-mask-remove.patch - debug-qspl.patch - asm-fixes.patch - fix-asm2.patch --- arch/powerpc/Kconfig | 1 - arch/powerpc/include/asm/qspinlock.h | 78 ++++++++++------------ arch/powerpc/include/asm/qspinlock_types.h | 13 ++++ arch/powerpc/include/asm/spinlock_types.h | 2 +- arch/powerpc/lib/Makefile | 4 +- arch/powerpc/lib/qspinlock.c | 18 +++++ 6 files changed, 69 insertions(+), 47 deletions(-) create mode 100644 arch/powerpc/include/asm/qspinlock_types.h create mode 100644 arch/powerpc/lib/qspinlock.c diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 7aa12e88c580..4838e6c96b20 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -154,7 +154,6 @@ config PPC select ARCH_USE_CMPXCHG_LOCKREF if PPC64 select ARCH_USE_MEMTEST select ARCH_USE_QUEUED_RWLOCKS if PPC_QUEUED_SPINLOCKS - select ARCH_USE_QUEUED_SPINLOCKS if PPC_QUEUED_SPINLOCKS select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT select ARCH_WANT_IPC_PARSE_VERSION select ARCH_WANT_IRQS_OFF_ACTIVATE_MM diff --git a/arch/powerpc/include/asm/qspinlock.h b/arch/powerpc/include/asm/qspinlock.h index 39c1c7f80579..cb2b4f91e976 100644 --- a/arch/powerpc/include/asm/qspinlock.h +++ b/arch/powerpc/include/asm/qspinlock.h @@ -2,66 +2,56 @@ #ifndef _ASM_POWERPC_QSPINLOCK_H #define _ASM_POWERPC_QSPINLOCK_H -#include -#include +#include +#include +#include -#define _Q_PENDING_LOOPS (1 << 9) /* not tuned */ - -void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val); -void __pv_queued_spin_lock_slowpath(struct qspinlock *lock, u32 val); -void __pv_queued_spin_unlock(struct qspinlock *lock); - -static __always_inline void queued_spin_lock(struct qspinlock *lock) +static __always_inline int queued_spin_is_locked(struct qspinlock *lock) { - u32 val = 0; + return atomic_read(&lock->val); +} - if (likely(arch_atomic_try_cmpxchg_lock(&lock->val, &val, _Q_LOCKED_VAL))) - return; +static __always_inline int queued_spin_value_unlocked(struct qspinlock lock) +{ + return !atomic_read(&lock.val); +} - if (!IS_ENABLED(CONFIG_PARAVIRT_SPINLOCKS) || !is_shared_processor()) - queued_spin_lock_slowpath(lock, val); - else - __pv_queued_spin_lock_slowpath(lock, val); +static __always_inline int queued_spin_is_contended(struct qspinlock *lock) +{ + return 0; } -#define queued_spin_lock queued_spin_lock -static inline void queued_spin_unlock(struct qspinlock *lock) +static __always_inline int queued_spin_trylock(struct qspinlock *lock) { - if (!IS_ENABLED(CONFIG_PARAVIRT_SPINLOCKS) || !is_shared_processor()) - smp_store_release(&lock->locked, 0); - else - __pv_queued_spin_unlock(lock); + if (atomic_cmpxchg_acquire(&lock->val, 0, 1) == 0) + return 1; + return 0; } -#define queued_spin_unlock queued_spin_unlock -#ifdef CONFIG_PARAVIRT_SPINLOCKS -#define SPIN_THRESHOLD (1<<15) /* not tuned */ +void queued_spin_lock_slowpath(struct qspinlock *lock); -static __always_inline void pv_wait(u8 *ptr, u8 val) +static __always_inline void queued_spin_lock(struct qspinlock *lock) { - if (*ptr != val) - return; - yield_to_any(); - /* - * We could pass in a CPU here if waiting in the queue and yield to - * the previous CPU in the queue. - */ + if (!queued_spin_trylock(lock)) + queued_spin_lock_slowpath(lock); } -static __always_inline void pv_kick(int cpu) +static inline void queued_spin_unlock(struct qspinlock *lock) { - prod_cpu(cpu); + atomic_set_release(&lock->val, 0); } -#endif +#define arch_spin_is_locked(l) queued_spin_is_locked(l) +#define arch_spin_is_contended(l) queued_spin_is_contended(l) +#define arch_spin_value_unlocked(l) queued_spin_value_unlocked(l) +#define arch_spin_lock(l) queued_spin_lock(l) +#define arch_spin_trylock(l) queued_spin_trylock(l) +#define arch_spin_unlock(l) queued_spin_unlock(l) -/* - * Queued spinlocks rely heavily on smp_cond_load_relaxed() to busy-wait, - * which was found to have performance problems if implemented with - * the preferred spin_begin()/spin_end() SMT priority pattern. Use the - * generic version instead. - */ - -#include +#ifdef CONFIG_PARAVIRT_SPINLOCKS +void pv_spinlocks_init(void); +#else +static inline void pv_spinlocks_init(void) { } +#endif #endif /* _ASM_POWERPC_QSPINLOCK_H */ diff --git a/arch/powerpc/include/asm/qspinlock_types.h b/arch/powerpc/include/asm/qspinlock_types.h new file mode 100644 index 000000000000..59606bc0c774 --- /dev/null +++ b/arch/powerpc/include/asm/qspinlock_types.h @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef _ASM_POWERPC_QSPINLOCK_TYPES_H +#define _ASM_POWERPC_QSPINLOCK_TYPES_H + +#include + +typedef struct qspinlock { + atomic_t val; +} arch_spinlock_t; + +#define __ARCH_SPIN_LOCK_UNLOCKED { .val = ATOMIC_INIT(0) } + +#endif /* _ASM_POWERPC_QSPINLOCK_TYPES_H */ diff --git a/arch/powerpc/include/asm/spinlock_types.h b/arch/powerpc/include/asm/spinlock_types.h index d5f8a74ed2e8..40b01446cf75 100644 --- a/arch/powerpc/include/asm/spinlock_types.h +++ b/arch/powerpc/include/asm/spinlock_types.h @@ -7,7 +7,7 @@ #endif #ifdef CONFIG_PPC_QUEUED_SPINLOCKS -#include +#include #include #else #include diff --git a/arch/powerpc/lib/Makefile b/arch/powerpc/lib/Makefile index 8560c912186d..b895cbf6a709 100644 --- a/arch/powerpc/lib/Makefile +++ b/arch/powerpc/lib/Makefile @@ -52,7 +52,9 @@ obj-$(CONFIG_PPC_BOOK3S_64) += copyuser_power7.o copypage_power7.o \ obj64-y += copypage_64.o copyuser_64.o mem_64.o hweight_64.o \ memcpy_64.o copy_mc_64.o -ifndef CONFIG_PPC_QUEUED_SPINLOCKS +ifdef CONFIG_PPC_QUEUED_SPINLOCKS +obj64-$(CONFIG_SMP) += qspinlock.o +else obj64-$(CONFIG_SMP) += locks.o endif diff --git a/arch/powerpc/lib/qspinlock.c b/arch/powerpc/lib/qspinlock.c new file mode 100644 index 000000000000..8dbce99a373c --- /dev/null +++ b/arch/powerpc/lib/qspinlock.c @@ -0,0 +1,18 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +#include +#include +#include + +void queued_spin_lock_slowpath(struct qspinlock *lock) +{ + while (!queued_spin_trylock(lock)) + cpu_relax(); +} +EXPORT_SYMBOL(queued_spin_lock_slowpath); + +#ifdef CONFIG_PARAVIRT_SPINLOCKS +void pv_spinlocks_init(void) +{ +} +#endif + From patchwork Mon Jul 11 03:04:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1654616 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=XuwK9ZCM; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lh8181rcQz9ryY for ; Mon, 11 Jul 2022 13:06:36 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Lh8180Y5Hz3c5L for ; Mon, 11 Jul 2022 13:06:36 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=XuwK9ZCM; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::42e; helo=mail-pf1-x42e.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=XuwK9ZCM; dkim-atps=neutral Received: from mail-pf1-x42e.google.com (mail-pf1-x42e.google.com [IPv6:2607:f8b0:4864:20::42e]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Lh7zR1HTmz3bxt for ; Mon, 11 Jul 2022 13:05:07 +1000 (AEST) Received: by mail-pf1-x42e.google.com with SMTP id b9so3681928pfp.10 for ; Sun, 10 Jul 2022 20:05:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lSou1iY8NAwrGFoX+ejzklzdAJG+1HYgkHfIjoYr2Jo=; b=XuwK9ZCMnPPFmy5mZ1agOWvmD/xBkYY45RvPF3GB6QsKoImE/iAduBfhEmdDkh/AdE zbdSNnV8IUaYi6blr7Px4qnlyokKEjlfnDAOgsZXrxMLbh1dXxz17uvmlGBb0VgjdPyl tp+NN6P5MooeGqYd7sfPB7mfQD6ipK65GKy1EXRx0dEfTS9RD6ZDK109nHZ+B+2AslhQ Njog9xj1Th+5DmkePa7jBjU6iPJ15kNycixDSWzMtSHSX1AtnNXvKWtKgflH77P49kcL EtNKj8MarVlsgMXgjK7Rv+n78Qhk92oqaCtOsS1e0g0oDyjxrigQ47qnv4MEjoDUU5Vs NIRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lSou1iY8NAwrGFoX+ejzklzdAJG+1HYgkHfIjoYr2Jo=; b=p+zoXahnMZYN3yhiRBqsNMymK/HqG/wx+f4jFiJZudYsFkvtSdbrAUcTOTkPgbK3GL uzI4079cRckDHgDWPjgHuY/RZ4r0G7/ghyRWBgRjym/K2F0beKvcyZ/+uscsZCIS+3QX VFL8pZog1Q8/NhKRlWNrYYKOayYNfkIgCxmp7BqJ4CG30KC0GWMDkjWQrEAoBECRtlug JlPMRt5RGHXAPKeliX+mSPZld9eeC5c2KlteB9+B/yXyXh+DnkAdk+N3TcdI8QlLPfyg VWJy/GZ264b2g4yrPqhy2H41jh9ypxUaG8aQrQQZWTqSnVHfCcBohRELEr8M9PGpRWah D4Sg== X-Gm-Message-State: AJIora/5Afjiha2Fop8cR6PztqNU4eiy9G7iMkmBojU5+ipKZAtDCl4g LNYKb9RkEGkm5J/rOkNI7fu2rj2mZzQ= X-Google-Smtp-Source: AGRyM1su6piqagYVg+WwinCJbEwNy5ztxNEL4tPJICfy1E7gs2qo7qGXjcFqFjSBxfdq56U8iYUPpQ== X-Received: by 2002:a63:5441:0:b0:412:64de:5f35 with SMTP id e1-20020a635441000000b0041264de5f35mr14084406pgm.321.1657508705234; Sun, 10 Jul 2022 20:05:05 -0700 (PDT) Received: from bobo.ibm.com ([203.220.77.143]) by smtp.gmail.com with ESMTPSA id o4-20020a17090a5b0400b001ef8c81959dsm3562232pji.16.2022.07.10.20.05.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Jul 2022 20:05:05 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 02/14] powerpc/qspinlock: add mcs queueing for contended waiters Date: Mon, 11 Jul 2022 13:04:41 +1000 Message-Id: <20220711030453.150644-3-npiggin@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220711030453.150644-1-npiggin@gmail.com> References: <20220711030453.150644-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" This forms the basis of the qspinlock slow path. Like generic qspinlocks and unlike the vanilla MCS algorithm, the lock owner does not participate in the queue, only waiters. The first waiter spins on the lock word, then when the lock is released it takes ownership and unqueues the next waiter. This is how qspinlocks can be implemented with the spinlock API -- lock owners don't need a node, only waiters do. --- arch/powerpc/include/asm/qspinlock.h | 10 +- arch/powerpc/include/asm/qspinlock_types.h | 21 +++ arch/powerpc/lib/qspinlock.c | 166 ++++++++++++++++++++- 3 files changed, 191 insertions(+), 6 deletions(-) diff --git a/arch/powerpc/include/asm/qspinlock.h b/arch/powerpc/include/asm/qspinlock.h index cb2b4f91e976..f06117aa60e1 100644 --- a/arch/powerpc/include/asm/qspinlock.h +++ b/arch/powerpc/include/asm/qspinlock.h @@ -18,12 +18,12 @@ static __always_inline int queued_spin_value_unlocked(struct qspinlock lock) static __always_inline int queued_spin_is_contended(struct qspinlock *lock) { - return 0; + return !!(atomic_read(&lock->val) & _Q_TAIL_CPU_MASK); } static __always_inline int queued_spin_trylock(struct qspinlock *lock) { - if (atomic_cmpxchg_acquire(&lock->val, 0, 1) == 0) + if (atomic_cmpxchg_acquire(&lock->val, 0, _Q_LOCKED_VAL) == 0) return 1; return 0; } @@ -38,7 +38,11 @@ static __always_inline void queued_spin_lock(struct qspinlock *lock) static inline void queued_spin_unlock(struct qspinlock *lock) { - atomic_set_release(&lock->val, 0); + for (;;) { + int val = atomic_read(&lock->val); + if (atomic_cmpxchg_release(&lock->val, val, val & ~_Q_LOCKED_VAL) == val) + return; + } } #define arch_spin_is_locked(l) queued_spin_is_locked(l) diff --git a/arch/powerpc/include/asm/qspinlock_types.h b/arch/powerpc/include/asm/qspinlock_types.h index 59606bc0c774..9630e714c70d 100644 --- a/arch/powerpc/include/asm/qspinlock_types.h +++ b/arch/powerpc/include/asm/qspinlock_types.h @@ -10,4 +10,25 @@ typedef struct qspinlock { #define __ARCH_SPIN_LOCK_UNLOCKED { .val = ATOMIC_INIT(0) } +/* + * Bitfields in the atomic value: + * + * 0: locked bit + * 16-31: tail cpu (+1) + */ +#define _Q_SET_MASK(type) (((1U << _Q_ ## type ## _BITS) - 1)\ + << _Q_ ## type ## _OFFSET) +#define _Q_LOCKED_OFFSET 0 +#define _Q_LOCKED_BITS 1 +#define _Q_LOCKED_MASK _Q_SET_MASK(LOCKED) +#define _Q_LOCKED_VAL (1U << _Q_LOCKED_OFFSET) + +#define _Q_TAIL_CPU_OFFSET 16 +#define _Q_TAIL_CPU_BITS (32 - _Q_TAIL_CPU_OFFSET) +#define _Q_TAIL_CPU_MASK _Q_SET_MASK(TAIL_CPU) + +#if CONFIG_NR_CPUS >= (1U << _Q_TAIL_CPU_BITS) +#error "qspinlock does not support such large CONFIG_NR_CPUS" +#endif + #endif /* _ASM_POWERPC_QSPINLOCK_TYPES_H */ diff --git a/arch/powerpc/lib/qspinlock.c b/arch/powerpc/lib/qspinlock.c index 8dbce99a373c..c65f7719a188 100644 --- a/arch/powerpc/lib/qspinlock.c +++ b/arch/powerpc/lib/qspinlock.c @@ -1,12 +1,172 @@ // SPDX-License-Identifier: GPL-2.0-or-later +#include +#include +#include #include -#include +#include +#include #include -void queued_spin_lock_slowpath(struct qspinlock *lock) +#define MAX_NODES 4 + +struct qnode { + struct qnode *next; + struct qspinlock *lock; + u8 locked; /* 1 if lock acquired */ +}; + +struct qnodes { + int count; + struct qnode nodes[MAX_NODES]; +}; + +static DEFINE_PER_CPU_ALIGNED(struct qnodes, qnodes); + +static inline int encode_tail_cpu(void) +{ + return (smp_processor_id() + 1) << _Q_TAIL_CPU_OFFSET; +} + +static inline int get_tail_cpu(int val) +{ + return (val >> _Q_TAIL_CPU_OFFSET) - 1; +} + +/* Take the lock by setting the bit, no other CPUs may concurrently lock it. */ +static __always_inline void lock_set_locked(struct qspinlock *lock) +{ + atomic_or(_Q_LOCKED_VAL, &lock->val); + __atomic_acquire_fence(); +} + +/* Take lock, clearing tail, cmpxchg with val (which must not be locked) */ +static __always_inline int trylock_clear_tail_cpu(struct qspinlock *lock, int val) +{ + int newval = _Q_LOCKED_VAL; + + if (atomic_cmpxchg_acquire(&lock->val, val, newval) == val) + return 1; + else + return 0; +} + +/* + * Publish our tail, replacing previous tail. Return previous value. + * + * This provides a release barrier for publishing node, and an acquire barrier + * for getting the old node. + */ +static __always_inline int publish_tail_cpu(struct qspinlock *lock, int tail) { - while (!queued_spin_trylock(lock)) + for (;;) { + int val = atomic_read(&lock->val); + int newval = (val & ~_Q_TAIL_CPU_MASK) | tail; + int old; + + old = atomic_cmpxchg(&lock->val, val, newval); + if (old == val) + return old; + } +} + +static inline struct qnode *get_tail_qnode(struct qspinlock *lock, int val) +{ + int cpu = get_tail_cpu(val); + struct qnodes *qnodesp = per_cpu_ptr(&qnodes, cpu); + int idx; + + for (idx = 0; idx < MAX_NODES; idx++) { + struct qnode *qnode = &qnodesp->nodes[idx]; + if (qnode->lock == lock) + return qnode; + } + + BUG(); +} + +static inline void queued_spin_lock_mcs_queue(struct qspinlock *lock) +{ + struct qnodes *qnodesp; + struct qnode *next, *node; + int val, old, tail; + int idx; + + BUILD_BUG_ON(CONFIG_NR_CPUS >= (1U << _Q_TAIL_CPU_BITS)); + + qnodesp = this_cpu_ptr(&qnodes); + if (unlikely(qnodesp->count == MAX_NODES)) { + while (!queued_spin_trylock(lock)) + cpu_relax(); + return; + } + + idx = qnodesp->count++; + /* + * Ensure that we increment the head node->count before initialising + * the actual node. If the compiler is kind enough to reorder these + * stores, then an IRQ could overwrite our assignments. + */ + barrier(); + node = &qnodesp->nodes[idx]; + node->next = NULL; + node->lock = lock; + node->locked = 0; + + tail = encode_tail_cpu(); + + old = publish_tail_cpu(lock, tail); + + /* + * If there was a previous node; link it and wait until reaching the + * head of the waitqueue. + */ + if (old & _Q_TAIL_CPU_MASK) { + struct qnode *prev = get_tail_qnode(lock, old); + + /* Link @node into the waitqueue. */ + WRITE_ONCE(prev->next, node); + + /* Wait for mcs node lock to be released */ + while (!node->locked) + cpu_relax(); + + smp_rmb(); /* acquire barrier for the mcs lock */ + } + + /* We're at the head of the waitqueue, wait for the lock. */ + while ((val = atomic_read(&lock->val)) & _Q_LOCKED_VAL) + cpu_relax(); + + /* If we're the last queued, must clean up the tail. */ + if ((val & _Q_TAIL_CPU_MASK) == tail) { + if (trylock_clear_tail_cpu(lock, val)) + goto release; + /* Another waiter must have enqueued */ + } + + /* We must be the owner, just set the lock bit and acquire */ + lock_set_locked(lock); + + /* contended path; must wait for next != NULL (MCS protocol) */ + while (!(next = READ_ONCE(node->next))) cpu_relax(); + + /* + * Unlock the next mcs waiter node. Release barrier is not required + * here because the acquirer is only accessing the lock word, and + * the acquire barrier we took the lock with orders that update vs + * this store to locked. The corresponding barrier is the smp_rmb() + * acquire barrier for mcs lock, above. + */ + WRITE_ONCE(next->locked, 1); + +release: + qnodesp->count--; /* release the node */ +} + +void queued_spin_lock_slowpath(struct qspinlock *lock) +{ + queued_spin_lock_mcs_queue(lock); } EXPORT_SYMBOL(queued_spin_lock_slowpath); From patchwork Mon Jul 11 03:04:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1654617 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=jzOB2NZM; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lh81p3yzkz9ryY for ; Mon, 11 Jul 2022 13:07:10 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Lh81p2rWxz3f3Y for ; Mon, 11 Jul 2022 13:07:10 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=jzOB2NZM; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::52e; helo=mail-pg1-x52e.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=jzOB2NZM; dkim-atps=neutral Received: from mail-pg1-x52e.google.com (mail-pg1-x52e.google.com [IPv6:2607:f8b0:4864:20::52e]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Lh7zV0Wpqz3c5h for ; Mon, 11 Jul 2022 13:05:09 +1000 (AEST) Received: by mail-pg1-x52e.google.com with SMTP id g4so3671724pgc.1 for ; Sun, 10 Jul 2022 20:05:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=3ZcC2mDAll+4KfiH8LhisJmeCP80W0HEH3VgJ6JyROI=; b=jzOB2NZMmvYJg6N1fAxTVsHBmE/TwoDzJ+AWvirEQRyiNJtMVQjl1YDejD+bDFw8Ry NWT89K9folFZFuJLVuetSEooD5wLUDMxwYF92QNEFTgLw3UP+Aka6XgYMnv4aOa6sGl/ AbnzvdV7guOecjglk+vJkRbGpRuoYrKPuKqkrWQgBWAfviFWifIWbwvBtCPpuE0Dy8ha CppPud/RQP4107FW3mPN/Xdfd4aD6NKghEyMioV7snt5aK+l2lstZeSNxomwQv+TWl7A ETJmcRuSGvjnhBrBbnztHJktm7sGSo1M6aWSt4C1LX9fCH5pZwIDWioUHwGLne2opzA4 jS1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=3ZcC2mDAll+4KfiH8LhisJmeCP80W0HEH3VgJ6JyROI=; b=Rr1dQfbiqVwD3AqG5K4lTN03ni6cmVZg7C/QXwPXclGKU5YirhclRCbtNrvW/ZBbsD /3ALk7tJxYx4++WdfpRQ6lKyMqs7cVzwW8HTW4drC5ANdwLUEyLLew6tAQbp+g+rdGli J3SlfiQkzJXgVd1/2UEZKBCL4pS1mLchoacEXoUppJv/aOIOX2BLSxla+D51vkIwCiWp W4KgHlAD442IDcg5skeU5/54i1VVdgZgHk18TAFpl0R6FBvZxkxaI/nbljuejKfuKiLc S0VwAskxrIlU9RV2P6dJojch74DL3O1c6XOk7uGJW/cjSer+mmpDBVIy49rk0eXcAqyO m8LQ== X-Gm-Message-State: AJIora/3ZeD5JrKqeiNPCylZrxXOWZ0mxoxzwaj/lVlTGBXNt/Y7RC35 5mfIf/JYp4uC3GSnbGn2b/XtWLw0G9M= X-Google-Smtp-Source: AGRyM1uLY8WS1UHQKLhR/Glbj/w6aWggV9L8SRhnK1WpN+LUQa6/7TFDwlhapsAX5sdvD2aYkDiBcQ== X-Received: by 2002:a65:6b8a:0:b0:3fc:4c06:8a8d with SMTP id d10-20020a656b8a000000b003fc4c068a8dmr13841063pgw.83.1657508707461; Sun, 10 Jul 2022 20:05:07 -0700 (PDT) Received: from bobo.ibm.com ([203.220.77.143]) by smtp.gmail.com with ESMTPSA id o4-20020a17090a5b0400b001ef8c81959dsm3562232pji.16.2022.07.10.20.05.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Jul 2022 20:05:07 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 03/14] powerpc/qspinlock: use a half-word store to unlock to avoid larx/stcx. Date: Mon, 11 Jul 2022 13:04:42 +1000 Message-Id: <20220711030453.150644-4-npiggin@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220711030453.150644-1-npiggin@gmail.com> References: <20220711030453.150644-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" The first 16 bits of the lock are only modified by the owner, and other modifications always use atomic operations on the entire 32 bits, so unlocks can use plain stores on the 16 bits. This is the same kind of optimisation done by core qspinlock code. --- arch/powerpc/include/asm/qspinlock.h | 6 +----- arch/powerpc/include/asm/qspinlock_types.h | 19 +++++++++++++++++-- 2 files changed, 18 insertions(+), 7 deletions(-) diff --git a/arch/powerpc/include/asm/qspinlock.h b/arch/powerpc/include/asm/qspinlock.h index f06117aa60e1..79a1936fb68d 100644 --- a/arch/powerpc/include/asm/qspinlock.h +++ b/arch/powerpc/include/asm/qspinlock.h @@ -38,11 +38,7 @@ static __always_inline void queued_spin_lock(struct qspinlock *lock) static inline void queued_spin_unlock(struct qspinlock *lock) { - for (;;) { - int val = atomic_read(&lock->val); - if (atomic_cmpxchg_release(&lock->val, val, val & ~_Q_LOCKED_VAL) == val) - return; - } + smp_store_release(&lock->locked, 0); } #define arch_spin_is_locked(l) queued_spin_is_locked(l) diff --git a/arch/powerpc/include/asm/qspinlock_types.h b/arch/powerpc/include/asm/qspinlock_types.h index 9630e714c70d..3425dab42576 100644 --- a/arch/powerpc/include/asm/qspinlock_types.h +++ b/arch/powerpc/include/asm/qspinlock_types.h @@ -3,12 +3,27 @@ #define _ASM_POWERPC_QSPINLOCK_TYPES_H #include +#include typedef struct qspinlock { - atomic_t val; + union { + atomic_t val; + +#ifdef __LITTLE_ENDIAN + struct { + u16 locked; + u8 reserved[2]; + }; +#else + struct { + u8 reserved[2]; + u16 locked; + }; +#endif + }; } arch_spinlock_t; -#define __ARCH_SPIN_LOCK_UNLOCKED { .val = ATOMIC_INIT(0) } +#define __ARCH_SPIN_LOCK_UNLOCKED { { .val = ATOMIC_INIT(0) } } /* * Bitfields in the atomic value: From patchwork Mon Jul 11 03:04:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1654618 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=OAw6TMVc; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lh82T2Q5Hz9ryY for ; Mon, 11 Jul 2022 13:07:45 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Lh82T1PGpz3c2N for ; Mon, 11 Jul 2022 13:07:45 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=OAw6TMVc; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::534; helo=mail-pg1-x534.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=OAw6TMVc; dkim-atps=neutral Received: from mail-pg1-x534.google.com (mail-pg1-x534.google.com [IPv6:2607:f8b0:4864:20::534]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Lh7zX21Z0z3c2B for ; Mon, 11 Jul 2022 13:05:11 +1000 (AEST) Received: by mail-pg1-x534.google.com with SMTP id f11so2769463pgj.7 for ; Sun, 10 Jul 2022 20:05:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ABhXW4KmxbtqDSaBLpPXD0viVy0GV1a5k2HrFVFAIYQ=; b=OAw6TMVcXUyU7AVDsfkm97wnmr2hVLUM7sZP0uYpwGugTJLfxIgP8Pv8/jvlvUQeYK F3y3KjuHZ+jLy4irDCsB2ZQGSSV6NGZbljCegNlV1ysGhfV9ibesoKJ5qwZKtbMgyznV 25lxc8wRAp8FeRJTuTTNU5oQ8PyvmB4U8S1rL/3ZAvGn87ZU46VV3egiWwqf0c74hoAT /F/0njBt/RL6VppusiycqpF40wznUqHSWCwpLRJpgQhWjLQzyncHgspTiYLwMyt6PdOF eszFdou1EVIrE3k4cjaiSdWjvSLBrDsDTS0Ecm9nnAH3Vc8zWRWZVaw1eZNcvnhTs605 hjbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ABhXW4KmxbtqDSaBLpPXD0viVy0GV1a5k2HrFVFAIYQ=; b=oQD6ky81E3wdhSshZnhJ4y5o35WuRBiwuclHeDEh0s5Ga0XCSysbJTOkuCPsKwOG2U i/XKtfcdQrHcUKJeB54v+peHPgAiSBxbFNCwHnZr9qaFwETxtRe2XY3/FU58dDU3ZtjH J9aCSuNYXSdCXSY6mqsbAjH4vmer8d22ncAHw27B/eLRAi4GKzI6arWx5NZ/9wKhyVYt Zls75FUlZc8JOlXRXbofsB3afOoZMxMeJUmXwyGvr6cUts1ysJT3g1Yl40+fQfvK2512 7ZSD8wkMJKBXett+AMBF+mw1Aq/i5w/Tn40hnAgj5GnwsxDUiRMbWBiFnWsOp+TZfPUH aozw== X-Gm-Message-State: AJIora8Iniwj2gNGzcrywVmYoUy6OOfQ4/iWS+Nn3DwZhur/dKnc/0RI ujUdQg9ToitVBSlqmMmsEEnEW5tyWec= X-Google-Smtp-Source: AGRyM1tp8yu8oJMJvLqh2bBfEjQorPl8DBUxs08MoTlX2vaAyiLxAcFPTEgqpXWkB0RXrNWr2BC+rg== X-Received: by 2002:a63:db02:0:b0:412:1688:9ab7 with SMTP id e2-20020a63db02000000b0041216889ab7mr13687425pgg.563.1657508709517; Sun, 10 Jul 2022 20:05:09 -0700 (PDT) Received: from bobo.ibm.com ([203.220.77.143]) by smtp.gmail.com with ESMTPSA id o4-20020a17090a5b0400b001ef8c81959dsm3562232pji.16.2022.07.10.20.05.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Jul 2022 20:05:09 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 04/14] powerpc/qspinlock: convert atomic operations to assembly Date: Mon, 11 Jul 2022 13:04:43 +1000 Message-Id: <20220711030453.150644-5-npiggin@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220711030453.150644-1-npiggin@gmail.com> References: <20220711030453.150644-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" This uses more optimal ll/sc style access patterns (rather than cmpxchg), and also sets the EH=1 lock hint on those operations which acquire ownership of the lock. --- arch/powerpc/include/asm/qspinlock.h | 25 +++++-- arch/powerpc/include/asm/qspinlock_types.h | 6 +- arch/powerpc/lib/qspinlock.c | 81 +++++++++++++++------- 3 files changed, 79 insertions(+), 33 deletions(-) diff --git a/arch/powerpc/include/asm/qspinlock.h b/arch/powerpc/include/asm/qspinlock.h index 79a1936fb68d..3ab354159e5e 100644 --- a/arch/powerpc/include/asm/qspinlock.h +++ b/arch/powerpc/include/asm/qspinlock.h @@ -2,28 +2,43 @@ #ifndef _ASM_POWERPC_QSPINLOCK_H #define _ASM_POWERPC_QSPINLOCK_H -#include #include #include static __always_inline int queued_spin_is_locked(struct qspinlock *lock) { - return atomic_read(&lock->val); + return READ_ONCE(lock->val); } static __always_inline int queued_spin_value_unlocked(struct qspinlock lock) { - return !atomic_read(&lock.val); + return !lock.val; } static __always_inline int queued_spin_is_contended(struct qspinlock *lock) { - return !!(atomic_read(&lock->val) & _Q_TAIL_CPU_MASK); + return !!(READ_ONCE(lock->val) & _Q_TAIL_CPU_MASK); } static __always_inline int queued_spin_trylock(struct qspinlock *lock) { - if (atomic_cmpxchg_acquire(&lock->val, 0, _Q_LOCKED_VAL) == 0) + u32 new = _Q_LOCKED_VAL; + u32 prev; + + asm volatile( +"1: lwarx %0,0,%1,%3 # queued_spin_trylock \n" +" cmpwi 0,%0,0 \n" +" bne- 2f \n" +" stwcx. %2,0,%1 \n" +" bne- 1b \n" +"\t" PPC_ACQUIRE_BARRIER " \n" +"2: \n" + : "=&r" (prev) + : "r" (&lock->val), "r" (new), + "i" (IS_ENABLED(CONFIG_PPC64) ? 1 : 0) + : "cr0", "memory"); + + if (likely(prev == 0)) return 1; return 0; } diff --git a/arch/powerpc/include/asm/qspinlock_types.h b/arch/powerpc/include/asm/qspinlock_types.h index 3425dab42576..210adf05b235 100644 --- a/arch/powerpc/include/asm/qspinlock_types.h +++ b/arch/powerpc/include/asm/qspinlock_types.h @@ -7,7 +7,7 @@ typedef struct qspinlock { union { - atomic_t val; + u32 val; #ifdef __LITTLE_ENDIAN struct { @@ -23,10 +23,10 @@ typedef struct qspinlock { }; } arch_spinlock_t; -#define __ARCH_SPIN_LOCK_UNLOCKED { { .val = ATOMIC_INIT(0) } } +#define __ARCH_SPIN_LOCK_UNLOCKED { { .val = 0 } } /* - * Bitfields in the atomic value: + * Bitfields in the lock word: * * 0: locked bit * 16-31: tail cpu (+1) diff --git a/arch/powerpc/lib/qspinlock.c b/arch/powerpc/lib/qspinlock.c index c65f7719a188..76dca922ba71 100644 --- a/arch/powerpc/lib/qspinlock.c +++ b/arch/powerpc/lib/qspinlock.c @@ -1,5 +1,4 @@ // SPDX-License-Identifier: GPL-2.0-or-later -#include #include #include #include @@ -22,32 +21,59 @@ struct qnodes { static DEFINE_PER_CPU_ALIGNED(struct qnodes, qnodes); -static inline int encode_tail_cpu(void) +static inline u32 encode_tail_cpu(void) { return (smp_processor_id() + 1) << _Q_TAIL_CPU_OFFSET; } -static inline int get_tail_cpu(int val) +static inline int get_tail_cpu(u32 val) { return (val >> _Q_TAIL_CPU_OFFSET) - 1; } /* Take the lock by setting the bit, no other CPUs may concurrently lock it. */ +/* Take the lock by setting the lock bit, no other CPUs will touch it. */ static __always_inline void lock_set_locked(struct qspinlock *lock) { - atomic_or(_Q_LOCKED_VAL, &lock->val); - __atomic_acquire_fence(); + u32 new = _Q_LOCKED_VAL; + u32 prev; + + asm volatile( +"1: lwarx %0,0,%1,%3 # lock_set_locked \n" +" or %0,%0,%2 \n" +" stwcx. %0,0,%1 \n" +" bne- 1b \n" +"\t" PPC_ACQUIRE_BARRIER " \n" + : "=&r" (prev) + : "r" (&lock->val), "r" (new), + "i" (IS_ENABLED(CONFIG_PPC64) ? 1 : 0) + : "cr0", "memory"); } -/* Take lock, clearing tail, cmpxchg with val (which must not be locked) */ -static __always_inline int trylock_clear_tail_cpu(struct qspinlock *lock, int val) +/* Take lock, clearing tail, cmpxchg with old (which must not be locked) */ +static __always_inline int trylock_clear_tail_cpu(struct qspinlock *lock, u32 old) { - int newval = _Q_LOCKED_VAL; - - if (atomic_cmpxchg_acquire(&lock->val, val, newval) == val) + u32 new = _Q_LOCKED_VAL; + u32 prev; + + BUG_ON(old & _Q_LOCKED_VAL); + + asm volatile( +"1: lwarx %0,0,%1,%4 # trylock_clear_tail_cpu \n" +" cmpw 0,%0,%2 \n" +" bne- 2f \n" +" stwcx. %3,0,%1 \n" +" bne- 1b \n" +"\t" PPC_ACQUIRE_BARRIER " \n" +"2: \n" + : "=&r" (prev) + : "r" (&lock->val), "r"(old), "r" (new), + "i" (IS_ENABLED(CONFIG_PPC64) ? 1 : 0) + : "cr0", "memory"); + + if (likely(prev == old)) return 1; - else - return 0; + return 0; } /* @@ -56,20 +82,25 @@ static __always_inline int trylock_clear_tail_cpu(struct qspinlock *lock, int va * This provides a release barrier for publishing node, and an acquire barrier * for getting the old node. */ -static __always_inline int publish_tail_cpu(struct qspinlock *lock, int tail) +static __always_inline u32 publish_tail_cpu(struct qspinlock *lock, u32 tail) { - for (;;) { - int val = atomic_read(&lock->val); - int newval = (val & ~_Q_TAIL_CPU_MASK) | tail; - int old; - - old = atomic_cmpxchg(&lock->val, val, newval); - if (old == val) - return old; - } + u32 prev, tmp; + + asm volatile( +"\t" PPC_RELEASE_BARRIER " \n" +"1: lwarx %0,0,%2 # publish_tail_cpu \n" +" andc %1,%0,%4 \n" +" or %1,%1,%3 \n" +" stwcx. %1,0,%2 \n" +" bne- 1b \n" + : "=&r" (prev), "=&r"(tmp) + : "r" (&lock->val), "r" (tail), "r"(_Q_TAIL_CPU_MASK) + : "cr0", "memory"); + + return prev; } -static inline struct qnode *get_tail_qnode(struct qspinlock *lock, int val) +static inline struct qnode *get_tail_qnode(struct qspinlock *lock, u32 val) { int cpu = get_tail_cpu(val); struct qnodes *qnodesp = per_cpu_ptr(&qnodes, cpu); @@ -88,7 +119,7 @@ static inline void queued_spin_lock_mcs_queue(struct qspinlock *lock) { struct qnodes *qnodesp; struct qnode *next, *node; - int val, old, tail; + u32 val, old, tail; int idx; BUILD_BUG_ON(CONFIG_NR_CPUS >= (1U << _Q_TAIL_CPU_BITS)); @@ -134,7 +165,7 @@ static inline void queued_spin_lock_mcs_queue(struct qspinlock *lock) } /* We're at the head of the waitqueue, wait for the lock. */ - while ((val = atomic_read(&lock->val)) & _Q_LOCKED_VAL) + while ((val = READ_ONCE(lock->val)) & _Q_LOCKED_VAL) cpu_relax(); /* If we're the last queued, must clean up the tail. */ From patchwork Mon Jul 11 03:04:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1654619 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=pnVQMq0w; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lh83G3WTLz9ryY for ; Mon, 11 Jul 2022 13:08:26 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Lh83G2lgHz3f8j for ; Mon, 11 Jul 2022 13:08:26 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=pnVQMq0w; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::630; helo=mail-pl1-x630.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=pnVQMq0w; dkim-atps=neutral Received: from mail-pl1-x630.google.com (mail-pl1-x630.google.com [IPv6:2607:f8b0:4864:20::630]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Lh7zZ3n17z3bsf for ; Mon, 11 Jul 2022 13:05:14 +1000 (AEST) Received: by mail-pl1-x630.google.com with SMTP id c13so3348335pla.6 for ; Sun, 10 Jul 2022 20:05:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ro2Y1lzmOoBreukK8lR2Me+tjUvCAWszXf9DWX+bUCQ=; b=pnVQMq0wilk0FDqO/ca9Di3vT478K02np6X8dB/Te3nKn6D/ktCFl96Btf/7gfJpni e2X7XRAa/80r9AhEpoSyUaYcqnUaPSrpZ1r0nUndqlB7qnqGzPERIFrrnTJYc1aplK+3 Y1TKQTz9ADpBuolBYm5qOnhBg6dvT+7mppCcOKDRLYfQ0SoakpdvjXTLSxxxd5CsO5RM lf6QP7SpTcxGJe6pPNp/HkcYTRfd9kB6IJdI8moK5mgamrSTHs2jTwBRplbUQJ00enA1 LnqrBoMx1cvy0UYJmF+wgx9w0MMrX33ZsXFBJGDCmi+QMSctaGqBsEDg8l+s9Ytp/LJm dJzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ro2Y1lzmOoBreukK8lR2Me+tjUvCAWszXf9DWX+bUCQ=; b=2pnBxHNbhzShmOTFT9OsnNk22tHxcSV0vVRFXC14JGV1hpM4wjU35x+UUqWSxxYvHK a7F23LP3wg4Fkg/We1NVPlkuWaAQqh2J1TG8mvpId2aZcsNJIsaG5c2e81zZg6CqQhhD Jm/USWJ0vyMWwCRGVLMHRttwKmSBpiqFZAsUT5U/pMobxZQ7OTqK8nx01PiC1xNzMoBX r/XIOtP++e9pOQvH12GfmJOHDWa5EFlZU2sIXT3hJnTQ9J4OBpTGyZQqwaPedZYPP5o5 FzKB8GUalm9Yp/yU1C7ae9fUeOoSDHAPWJ1KfK/Rgh9FO0p6GSzr8wP6JuX17E46raF6 FZyQ== X-Gm-Message-State: AJIora8a4uecoY5pGvB3XqsdJkvpxu17XzmfEcdZyFBLYMZ3iYgkiyCD kuP+DczcK4XMhJNAGyCFD1skiMrsikU= X-Google-Smtp-Source: AGRyM1s88pzQzU/Wp5+dbM5q/CfCpUw3Tw9WSuyi7afOd0MoFwBdfFRDDdZnj97hnhsPhKFji6AQPA== X-Received: by 2002:a17:90b:3e8a:b0:1f0:4157:daf8 with SMTP id rj10-20020a17090b3e8a00b001f04157daf8mr1948161pjb.222.1657508711798; Sun, 10 Jul 2022 20:05:11 -0700 (PDT) Received: from bobo.ibm.com ([203.220.77.143]) by smtp.gmail.com with ESMTPSA id o4-20020a17090a5b0400b001ef8c81959dsm3562232pji.16.2022.07.10.20.05.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Jul 2022 20:05:11 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 05/14] powerpc/qspinlock: allow new waiters to steal the lock before queueing Date: Mon, 11 Jul 2022 13:04:44 +1000 Message-Id: <20220711030453.150644-6-npiggin@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220711030453.150644-1-npiggin@gmail.com> References: <20220711030453.150644-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Allow new waiters a number of spins on the lock word before queueing, which particularly helps paravirt performance when physical CPUs are oversubscribed. --- arch/powerpc/lib/qspinlock.c | 143 ++++++++++++++++++++++++++++++++--- 1 file changed, 132 insertions(+), 11 deletions(-) diff --git a/arch/powerpc/lib/qspinlock.c b/arch/powerpc/lib/qspinlock.c index 76dca922ba71..cb87991602ff 100644 --- a/arch/powerpc/lib/qspinlock.c +++ b/arch/powerpc/lib/qspinlock.c @@ -19,6 +19,10 @@ struct qnodes { struct qnode nodes[MAX_NODES]; }; +/* Tuning parameters */ +static int STEAL_SPINS __read_mostly = (1<<5); +static bool MAYBE_STEALERS __read_mostly = true; + static DEFINE_PER_CPU_ALIGNED(struct qnodes, qnodes); static inline u32 encode_tail_cpu(void) @@ -76,6 +80,39 @@ static __always_inline int trylock_clear_tail_cpu(struct qspinlock *lock, u32 ol return 0; } +static __always_inline u32 __trylock_cmpxchg(struct qspinlock *lock, u32 old, u32 new) +{ + u32 prev; + + BUG_ON(old & _Q_LOCKED_VAL); + + asm volatile( +"1: lwarx %0,0,%1,%4 # queued_spin_trylock_cmpxchg \n" +" cmpw 0,%0,%2 \n" +" bne- 2f \n" +" stwcx. %3,0,%1 \n" +" bne- 1b \n" +"\t" PPC_ACQUIRE_BARRIER " \n" +"2: \n" + : "=&r" (prev) + : "r" (&lock->val), "r"(old), "r" (new), + "i" (IS_ENABLED(CONFIG_PPC64) ? 1 : 0) + : "cr0", "memory"); + + return prev; +} + +/* Take lock, preserving tail, cmpxchg with val (which must not be locked) */ +static __always_inline int trylock_with_tail_cpu(struct qspinlock *lock, u32 val) +{ + u32 newval = _Q_LOCKED_VAL | (val & _Q_TAIL_CPU_MASK); + + if (__trylock_cmpxchg(lock, val, newval) == val) + return 1; + else + return 0; +} + /* * Publish our tail, replacing previous tail. Return previous value. * @@ -115,6 +152,26 @@ static inline struct qnode *get_tail_qnode(struct qspinlock *lock, u32 val) BUG(); } +static inline bool try_to_steal_lock(struct qspinlock *lock) +{ + int iters; + + /* Attempt to steal the lock */ + for (iters = 0; iters < STEAL_SPINS; iters++) { + u32 val = READ_ONCE(lock->val); + + if (val & _Q_LOCKED_VAL) { + cpu_relax(); + continue; + } + + if (trylock_with_tail_cpu(lock, val)) + return true; + } + + return false; +} + static inline void queued_spin_lock_mcs_queue(struct qspinlock *lock) { struct qnodes *qnodesp; @@ -164,20 +221,39 @@ static inline void queued_spin_lock_mcs_queue(struct qspinlock *lock) smp_rmb(); /* acquire barrier for the mcs lock */ } - /* We're at the head of the waitqueue, wait for the lock. */ - while ((val = READ_ONCE(lock->val)) & _Q_LOCKED_VAL) - cpu_relax(); + if (!MAYBE_STEALERS) { + /* We're at the head of the waitqueue, wait for the lock. */ + while ((val = READ_ONCE(lock->val)) & _Q_LOCKED_VAL) + cpu_relax(); - /* If we're the last queued, must clean up the tail. */ - if ((val & _Q_TAIL_CPU_MASK) == tail) { - if (trylock_clear_tail_cpu(lock, val)) - goto release; - /* Another waiter must have enqueued */ - } + /* If we're the last queued, must clean up the tail. */ + if ((val & _Q_TAIL_CPU_MASK) == tail) { + if (trylock_clear_tail_cpu(lock, val)) + goto release; + /* Another waiter must have enqueued. */ + } + + /* We must be the owner, just set the lock bit and acquire */ + lock_set_locked(lock); + } else { +again: + /* We're at the head of the waitqueue, wait for the lock. */ + while ((val = READ_ONCE(lock->val)) & _Q_LOCKED_VAL) + cpu_relax(); - /* We must be the owner, just set the lock bit and acquire */ - lock_set_locked(lock); + /* If we're the last queued, must clean up the tail. */ + if ((val & _Q_TAIL_CPU_MASK) == tail) { + if (trylock_clear_tail_cpu(lock, val)) + goto release; + /* Another waiter must have enqueued, or lock stolen. */ + } else { + if (trylock_with_tail_cpu(lock, val)) + goto unlock_next; + } + goto again; + } +unlock_next: /* contended path; must wait for next != NULL (MCS protocol) */ while (!(next = READ_ONCE(node->next))) cpu_relax(); @@ -197,6 +273,9 @@ static inline void queued_spin_lock_mcs_queue(struct qspinlock *lock) void queued_spin_lock_slowpath(struct qspinlock *lock) { + if (try_to_steal_lock(lock)) + return; + queued_spin_lock_mcs_queue(lock); } EXPORT_SYMBOL(queued_spin_lock_slowpath); @@ -204,6 +283,48 @@ EXPORT_SYMBOL(queued_spin_lock_slowpath); #ifdef CONFIG_PARAVIRT_SPINLOCKS void pv_spinlocks_init(void) { + STEAL_SPINS = (1<<15); } #endif +#include +static int steal_spins_set(void *data, u64 val) +{ + static DEFINE_MUTEX(lock); + + mutex_lock(&lock); + if (val && !STEAL_SPINS) { + MAYBE_STEALERS = true; + /* wait for waiter to go away */ + synchronize_rcu(); + STEAL_SPINS = val; + } else if (!val && STEAL_SPINS) { + STEAL_SPINS = val; + /* wait for all possible stealers to go away */ + synchronize_rcu(); + MAYBE_STEALERS = false; + } else { + STEAL_SPINS = val; + } + mutex_unlock(&lock); + + return 0; +} + +static int steal_spins_get(void *data, u64 *val) +{ + *val = STEAL_SPINS; + + return 0; +} + +DEFINE_SIMPLE_ATTRIBUTE(fops_steal_spins, steal_spins_get, steal_spins_set, "%llu\n"); + +static __init int spinlock_debugfs_init(void) +{ + debugfs_create_file("qspl_steal_spins", 0600, arch_debugfs_dir, NULL, &fops_steal_spins); + + return 0; +} +device_initcall(spinlock_debugfs_init); + From patchwork Mon Jul 11 03:04:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1654620 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=FhJfYL91; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lh83y1JL5z9ryY for ; Mon, 11 Jul 2022 13:09:02 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Lh83y0HMvz3fCX for ; Mon, 11 Jul 2022 13:09:02 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=FhJfYL91; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::102a; helo=mail-pj1-x102a.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=FhJfYL91; dkim-atps=neutral Received: from mail-pj1-x102a.google.com (mail-pj1-x102a.google.com [IPv6:2607:f8b0:4864:20::102a]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Lh7zc4dKhz3c4k for ; Mon, 11 Jul 2022 13:05:16 +1000 (AEST) Received: by mail-pj1-x102a.google.com with SMTP id i8-20020a17090a4b8800b001ef8a65bfbdso3727184pjh.1 for ; Sun, 10 Jul 2022 20:05:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0QWMyXxzMyRq6hI/XusD/3eoPZD0vCnM3p0Mn37XFoM=; b=FhJfYL91QF7b4j07ySQLqtjj1dPA+LBMfdj20MxmBNEsrxRlZ9YJtOwSUznFUgoavW nPzLy0Hy1TPpt3Feqiw0wqF9JJ1m17wFfYiFspD7F97Wl6wt7rQ+3lOVHV/1ZYhMeBj2 oOSd8YZ6DwK4dzWZZdyYQsfzNDJBwxDfMQabGuO6HljiHNOOtTiVOEU3O27U1xFXrjGC CfefkPU2qrS6KP5VSV34SD1OWxQDQt5ed6VKPkxRBlO+gkbJUjCbz/jVprmcUro5m4sZ fjXww4YTe8LOVVjSImBXxCLbo15KMtmtDTNAqHstNFiqIEv6w4sxJxEqWK9oEpXD7gIm 3b6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0QWMyXxzMyRq6hI/XusD/3eoPZD0vCnM3p0Mn37XFoM=; b=m0xlla+1m0xD1LdkTSZXbOl/WD3hhxoCzCPHmuQfiOJuXUAVTG627RNziWNmj8PpZf +3vL8A0w01tfyRXdUXEMoSA/W57Y0BdKJ30UE361F0McbJ0czHObkoMagi79WImcOZAW 417WRFBpsRDQcJWFxaISZxB20JLNkTX/a6IKxZ7UIqtyFkst9JQR+boHF+cW/B2DEBjK 1RMbR5ilk78GIaTV3ZoVBB7nxjRHqmBVD1Qb9IRDfjJpAWetkT+96OTQBfvzzf2V09pf eMzEU/B8kg6OOv1ezo7ReehgWl5SEq6w0Ljy0Wv+Cqvd1FVSQCOk9JA2cCTpP06etEtd gy4g== X-Gm-Message-State: AJIora+/cmQ6PZKD9Mc/U4a+O83cVZIF31a43ucayxjgGDg1FnCPM77x vfw87IJ7HVPKVa9TqN56/iQVdSc5CHM= X-Google-Smtp-Source: AGRyM1udV095lkqg/qHzJ8lwlMt+v0nwhl7Ng3HYnucDXUj8bvk/3lSQIqD97X3N8KSXA8ca9aobdA== X-Received: by 2002:a17:90b:4b41:b0:1ec:cb06:2fa3 with SMTP id mi1-20020a17090b4b4100b001eccb062fa3mr14851179pjb.55.1657508714009; Sun, 10 Jul 2022 20:05:14 -0700 (PDT) Received: from bobo.ibm.com ([203.220.77.143]) by smtp.gmail.com with ESMTPSA id o4-20020a17090a5b0400b001ef8c81959dsm3562232pji.16.2022.07.10.20.05.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Jul 2022 20:05:13 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 06/14] powerpc/qspinlock: theft prevention to control latency Date: Mon, 11 Jul 2022 13:04:45 +1000 Message-Id: <20220711030453.150644-7-npiggin@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220711030453.150644-1-npiggin@gmail.com> References: <20220711030453.150644-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Give the queue head the ability to stop stealers. After a number of spins without sucessfully acquiring the lock, the queue head employs this, which will assure it is the next owner. --- arch/powerpc/include/asm/qspinlock_types.h | 10 ++++- arch/powerpc/lib/qspinlock.c | 45 +++++++++++++++++++++- 2 files changed, 52 insertions(+), 3 deletions(-) diff --git a/arch/powerpc/include/asm/qspinlock_types.h b/arch/powerpc/include/asm/qspinlock_types.h index 210adf05b235..8b20f5e22bba 100644 --- a/arch/powerpc/include/asm/qspinlock_types.h +++ b/arch/powerpc/include/asm/qspinlock_types.h @@ -29,7 +29,8 @@ typedef struct qspinlock { * Bitfields in the lock word: * * 0: locked bit - * 16-31: tail cpu (+1) + * 16: must queue bit + * 17-31: tail cpu (+1) */ #define _Q_SET_MASK(type) (((1U << _Q_ ## type ## _BITS) - 1)\ << _Q_ ## type ## _OFFSET) @@ -38,7 +39,12 @@ typedef struct qspinlock { #define _Q_LOCKED_MASK _Q_SET_MASK(LOCKED) #define _Q_LOCKED_VAL (1U << _Q_LOCKED_OFFSET) -#define _Q_TAIL_CPU_OFFSET 16 +#define _Q_MUST_Q_OFFSET 16 +#define _Q_MUST_Q_BITS 1 +#define _Q_MUST_Q_MASK _Q_SET_MASK(MUST_Q) +#define _Q_MUST_Q_VAL (1U << _Q_MUST_Q_OFFSET) + +#define _Q_TAIL_CPU_OFFSET 17 #define _Q_TAIL_CPU_BITS (32 - _Q_TAIL_CPU_OFFSET) #define _Q_TAIL_CPU_MASK _Q_SET_MASK(TAIL_CPU) diff --git a/arch/powerpc/lib/qspinlock.c b/arch/powerpc/lib/qspinlock.c index cb87991602ff..662a744fa1ee 100644 --- a/arch/powerpc/lib/qspinlock.c +++ b/arch/powerpc/lib/qspinlock.c @@ -22,6 +22,7 @@ struct qnodes { /* Tuning parameters */ static int STEAL_SPINS __read_mostly = (1<<5); static bool MAYBE_STEALERS __read_mostly = true; +static int HEAD_SPINS __read_mostly = (1<<13); static DEFINE_PER_CPU_ALIGNED(struct qnodes, qnodes); @@ -137,6 +138,23 @@ static __always_inline u32 publish_tail_cpu(struct qspinlock *lock, u32 tail) return prev; } +static __always_inline u32 lock_set_mustq(struct qspinlock *lock) +{ + u32 new = _Q_MUST_Q_VAL; + u32 prev; + + asm volatile( +"1: lwarx %0,0,%1 # queued_spin_set_mustq \n" +" or %0,%0,%2 \n" +" stwcx. %0,0,%1 \n" +" bne- 1b \n" + : "=&r" (prev) + : "r" (&lock->val), "r" (new) + : "cr0", "memory"); + + return prev; +} + static inline struct qnode *get_tail_qnode(struct qspinlock *lock, u32 val) { int cpu = get_tail_cpu(val); @@ -160,6 +178,9 @@ static inline bool try_to_steal_lock(struct qspinlock *lock) for (iters = 0; iters < STEAL_SPINS; iters++) { u32 val = READ_ONCE(lock->val); + if (val & _Q_MUST_Q_VAL) + break; + if (val & _Q_LOCKED_VAL) { cpu_relax(); continue; @@ -236,10 +257,14 @@ static inline void queued_spin_lock_mcs_queue(struct qspinlock *lock) /* We must be the owner, just set the lock bit and acquire */ lock_set_locked(lock); } else { + int iters = 0; again: /* We're at the head of the waitqueue, wait for the lock. */ - while ((val = READ_ONCE(lock->val)) & _Q_LOCKED_VAL) + while ((val = READ_ONCE(lock->val)) & _Q_LOCKED_VAL) { + if (iters++ == HEAD_SPINS) + lock_set_mustq(lock); cpu_relax(); + } /* If we're the last queued, must clean up the tail. */ if ((val & _Q_TAIL_CPU_MASK) == tail) { @@ -284,6 +309,7 @@ EXPORT_SYMBOL(queued_spin_lock_slowpath); void pv_spinlocks_init(void) { STEAL_SPINS = (1<<15); + HEAD_SPINS = (1<<13); } #endif @@ -320,9 +346,26 @@ static int steal_spins_get(void *data, u64 *val) DEFINE_SIMPLE_ATTRIBUTE(fops_steal_spins, steal_spins_get, steal_spins_set, "%llu\n"); +static int head_spins_set(void *data, u64 val) +{ + HEAD_SPINS = val; + + return 0; +} + +static int head_spins_get(void *data, u64 *val) +{ + *val = HEAD_SPINS; + + return 0; +} + +DEFINE_SIMPLE_ATTRIBUTE(fops_head_spins, head_spins_get, head_spins_set, "%llu\n"); + static __init int spinlock_debugfs_init(void) { debugfs_create_file("qspl_steal_spins", 0600, arch_debugfs_dir, NULL, &fops_steal_spins); + debugfs_create_file("qspl_head_spins", 0600, arch_debugfs_dir, NULL, &fops_head_spins); return 0; } From patchwork Mon Jul 11 03:04:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1654621 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=gGogE3iH; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lh84c5r82z9ryY for ; Mon, 11 Jul 2022 13:09:36 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Lh84c4mr1z3fFp for ; Mon, 11 Jul 2022 13:09:36 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=gGogE3iH; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::629; helo=mail-pl1-x629.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=gGogE3iH; dkim-atps=neutral Received: from mail-pl1-x629.google.com (mail-pl1-x629.google.com [IPv6:2607:f8b0:4864:20::629]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Lh7zf4RJ7z3c1M for ; Mon, 11 Jul 2022 13:05:18 +1000 (AEST) Received: by mail-pl1-x629.google.com with SMTP id f11so3356325plr.4 for ; Sun, 10 Jul 2022 20:05:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=OKjdm3ih4ly1bz9fxkTZSfyX0ow4MRNkgFKhqnzspLA=; b=gGogE3iH2dEQe+djIir7vkwwgbyKq4K8qhdrNLFXXnmFk0RyAZZxx0CISB3ZppamwJ D7RnEKhFa0EpDRUSkwOnxqBN4AQAz3ZdHosX9PN28Axa76b0xHuMkrsjOLPPFH6PuzUl 4IJOBIieQCvg1wpsAKydqsaHCxhqHj7cc+/4QwUIxxoBIsd+8ivJNFVwDOu2XpKX1vw0 i+215VrJITtW4hy8aC4GKjvFxz8fa4HYesDfWVt1krqz9TdkjPXayW5kZLtEgImc4OJi 46b6yPlzY8GI8syvt+hrgyT6tvVdr8F8A4wyoeDgHieBi4B7vJ6+Y8LJlx0ZoosnH6iL FQjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OKjdm3ih4ly1bz9fxkTZSfyX0ow4MRNkgFKhqnzspLA=; b=4h9N3QRsiEN35KiYZyEgzKl1TgHeERnKgDZUZxBbfnLR4zRydGOcDgQoXg8fthBGN2 zBgXL+yNZAdRQy2OvrFisNCLEIIba62Iz5OsHepWIeGS+aW2jMw0E7hVquf5G6fxJj8d 5mA3/BaoXtMYRpWGpykNToFr7Hm3/DThYBBJ9xLES9DuFEM4kXcLrCWYz6D3Ujm24w7p sLHWBiTBAkyxWA4ATeUPN/Pns13P1TqDwC6RbeoBUgzttBZ0oH5iDXKLBscSoJE8xkwD wNEdXp6mG7hJ7MlwMByu2aa/GrAU6ihDXvQKhBormOgWNhN2FgZaW6tPGYoD672T5GU8 Amyw== X-Gm-Message-State: AJIora9aRR3M9qcXWxAlOdofU3B364n1a9CNRY52nCHima9hRPQda2EA GVk0RK/9hUMj/6uNa1+Pjc7L5IAycyA= X-Google-Smtp-Source: AGRyM1tJPZwr8kK8L54yx3FamXAKqOQjW+mmkyAm2Zq628MA+s9KYPjRtmdxKkEi4joo+MHy7AMEgg== X-Received: by 2002:a17:90a:6741:b0:1ef:7f62:6cd1 with SMTP id c1-20020a17090a674100b001ef7f626cd1mr14841906pjm.89.1657508716073; Sun, 10 Jul 2022 20:05:16 -0700 (PDT) Received: from bobo.ibm.com ([203.220.77.143]) by smtp.gmail.com with ESMTPSA id o4-20020a17090a5b0400b001ef8c81959dsm3562232pji.16.2022.07.10.20.05.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Jul 2022 20:05:15 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 07/14] powerpc/qspinlock: store owner CPU in lock word Date: Mon, 11 Jul 2022 13:04:46 +1000 Message-Id: <20220711030453.150644-8-npiggin@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220711030453.150644-1-npiggin@gmail.com> References: <20220711030453.150644-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Store the owner CPU number in the lock word so it may be yielded to, as powerpc's paravirtualised simple spinlocks do. --- arch/powerpc/include/asm/qspinlock.h | 8 +++++++- arch/powerpc/include/asm/qspinlock_types.h | 10 ++++++++++ arch/powerpc/lib/qspinlock.c | 6 +++--- 3 files changed, 20 insertions(+), 4 deletions(-) diff --git a/arch/powerpc/include/asm/qspinlock.h b/arch/powerpc/include/asm/qspinlock.h index 3ab354159e5e..44601b261e08 100644 --- a/arch/powerpc/include/asm/qspinlock.h +++ b/arch/powerpc/include/asm/qspinlock.h @@ -20,9 +20,15 @@ static __always_inline int queued_spin_is_contended(struct qspinlock *lock) return !!(READ_ONCE(lock->val) & _Q_TAIL_CPU_MASK); } +static __always_inline u32 queued_spin_get_locked_val(void) +{ + /* XXX: make this use lock value in paca like simple spinlocks? */ + return _Q_LOCKED_VAL | (smp_processor_id() << _Q_OWNER_CPU_OFFSET); +} + static __always_inline int queued_spin_trylock(struct qspinlock *lock) { - u32 new = _Q_LOCKED_VAL; + u32 new = queued_spin_get_locked_val(); u32 prev; asm volatile( diff --git a/arch/powerpc/include/asm/qspinlock_types.h b/arch/powerpc/include/asm/qspinlock_types.h index 8b20f5e22bba..35f9525381e6 100644 --- a/arch/powerpc/include/asm/qspinlock_types.h +++ b/arch/powerpc/include/asm/qspinlock_types.h @@ -29,6 +29,8 @@ typedef struct qspinlock { * Bitfields in the lock word: * * 0: locked bit + * 1-14: lock holder cpu + * 15: unused bit * 16: must queue bit * 17-31: tail cpu (+1) */ @@ -39,6 +41,14 @@ typedef struct qspinlock { #define _Q_LOCKED_MASK _Q_SET_MASK(LOCKED) #define _Q_LOCKED_VAL (1U << _Q_LOCKED_OFFSET) +#define _Q_OWNER_CPU_OFFSET 1 +#define _Q_OWNER_CPU_BITS 14 +#define _Q_OWNER_CPU_MASK _Q_SET_MASK(OWNER_CPU) + +#if CONFIG_NR_CPUS > (1U << _Q_OWNER_CPU_BITS) +#error "qspinlock does not support such large CONFIG_NR_CPUS" +#endif + #define _Q_MUST_Q_OFFSET 16 #define _Q_MUST_Q_BITS 1 #define _Q_MUST_Q_MASK _Q_SET_MASK(MUST_Q) diff --git a/arch/powerpc/lib/qspinlock.c b/arch/powerpc/lib/qspinlock.c index 662a744fa1ee..3c6554a02de7 100644 --- a/arch/powerpc/lib/qspinlock.c +++ b/arch/powerpc/lib/qspinlock.c @@ -40,7 +40,7 @@ static inline int get_tail_cpu(u32 val) /* Take the lock by setting the lock bit, no other CPUs will touch it. */ static __always_inline void lock_set_locked(struct qspinlock *lock) { - u32 new = _Q_LOCKED_VAL; + u32 new = queued_spin_get_locked_val(); u32 prev; asm volatile( @@ -58,7 +58,7 @@ static __always_inline void lock_set_locked(struct qspinlock *lock) /* Take lock, clearing tail, cmpxchg with old (which must not be locked) */ static __always_inline int trylock_clear_tail_cpu(struct qspinlock *lock, u32 old) { - u32 new = _Q_LOCKED_VAL; + u32 new = queued_spin_get_locked_val(); u32 prev; BUG_ON(old & _Q_LOCKED_VAL); @@ -106,7 +106,7 @@ static __always_inline u32 __trylock_cmpxchg(struct qspinlock *lock, u32 old, u3 /* Take lock, preserving tail, cmpxchg with val (which must not be locked) */ static __always_inline int trylock_with_tail_cpu(struct qspinlock *lock, u32 val) { - u32 newval = _Q_LOCKED_VAL | (val & _Q_TAIL_CPU_MASK); + u32 newval = queued_spin_get_locked_val() | (val & _Q_TAIL_CPU_MASK); if (__trylock_cmpxchg(lock, val, newval) == val) return 1; From patchwork Mon Jul 11 03:04:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1654622 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=YP9q5qHt; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lh85H2n8gz9ryY for ; Mon, 11 Jul 2022 13:10:11 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Lh85H0BWyz3f3R for ; Mon, 11 Jul 2022 13:10:11 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=YP9q5qHt; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::42a; helo=mail-pf1-x42a.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=YP9q5qHt; dkim-atps=neutral Received: from mail-pf1-x42a.google.com (mail-pf1-x42a.google.com [IPv6:2607:f8b0:4864:20::42a]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Lh7zh5c9zz3c1S for ; Mon, 11 Jul 2022 13:05:20 +1000 (AEST) Received: by mail-pf1-x42a.google.com with SMTP id 70so3713704pfx.1 for ; Sun, 10 Jul 2022 20:05:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=DK0VgMuz0XNDMCQTRnkUinv7v4fb+yrzyA/f87s3QSU=; b=YP9q5qHtvJ1ykzEel7epeenEm9HHkB1ixeZ3cMVJAjvyGE6pnDc26R6axhvpikNpS9 ZHYY9gl/nPzJLGMfrAO+H9PPNH0nXJ/jpeiPcBe16Z4mg0cE9WxyYZGrBbwSNpAN1m/W 0gk4sTLthrQWb2AavxT8QbKXk/1Zdr2vY9CiivldTyr/Af+NCaLP8hWfwO7sljWu4ngk 1NXFLuvLYGGg5qH4tpMq84vY6sLnH4nO8fWGWnIld/BK6wzE75zvjr5Ktct350Bfkk7g 8kxRM873Kk/ftcsu3qxKsZgLdtmpdlTLLYcWnXnGiUD7Z3h/GuVm4mXxg/4cvIsexbRN THBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DK0VgMuz0XNDMCQTRnkUinv7v4fb+yrzyA/f87s3QSU=; b=rAT19eJra+6/aZi+0KZdNW9r0EYNGI61ylqVBKfLMOL6Zwh0neFjEDLVFS0JBShQet 8b0WR40yyj0DYQ6XXEshTtr6j1fwrTN0ZFbIPDi4L23L5FKc2wcogq50LQ3C7Ur6C6aG Lr1kUJgvVejm7izMG0X17CiLcRO2LQoYDRrLjDGVcR3cOKEqmGJrURhr4EBx56G+mTD5 UmJpu1ORmWZWb6wGiSmLfC+gAru4wkAPM0fidwP0wBSxwSgiJSIaD9JDIsc9/jDuK/Dg tIujkAiYKTY5TUv0PLiltcu3s/7CDsuqHP0HbdM+0lxNAOD4FPDl2aoNaZYFmhjpWuux ZiNw== X-Gm-Message-State: AJIora8Jiz89bhZjihe7fW1TmldHOLfjOm19yfrvsuaGx+JEEbnSSdFo mMNehrCLpd0jOsv/HQ6DAiJMczQ5TOM= X-Google-Smtp-Source: AGRyM1vKCVQQK9MpuKVkaO9K1nwrAS6aq/zXn9ErMaTAhqojH8z1QXtDL2wKJJjv0jDnqZ0beK0kXg== X-Received: by 2002:a05:6a00:168a:b0:4f7:e161:83cd with SMTP id k10-20020a056a00168a00b004f7e16183cdmr16318361pfc.56.1657508718155; Sun, 10 Jul 2022 20:05:18 -0700 (PDT) Received: from bobo.ibm.com ([203.220.77.143]) by smtp.gmail.com with ESMTPSA id o4-20020a17090a5b0400b001ef8c81959dsm3562232pji.16.2022.07.10.20.05.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Jul 2022 20:05:17 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 08/14] powerpc/qspinlock: paravirt yield to lock owner Date: Mon, 11 Jul 2022 13:04:47 +1000 Message-Id: <20220711030453.150644-9-npiggin@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220711030453.150644-1-npiggin@gmail.com> References: <20220711030453.150644-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Waiters spinning on the lock word should yield to the lock owner if the vCPU is preempted. This improves performance when the hypervisor has oversubscribed physical CPUs. --- arch/powerpc/lib/qspinlock.c | 93 +++++++++++++++++++++++++++++++----- 1 file changed, 82 insertions(+), 11 deletions(-) diff --git a/arch/powerpc/lib/qspinlock.c b/arch/powerpc/lib/qspinlock.c index 3c6554a02de7..ee85198d5e28 100644 --- a/arch/powerpc/lib/qspinlock.c +++ b/arch/powerpc/lib/qspinlock.c @@ -5,6 +5,7 @@ #include #include #include +#include #define MAX_NODES 4 @@ -24,6 +25,8 @@ static int STEAL_SPINS __read_mostly = (1<<5); static bool MAYBE_STEALERS __read_mostly = true; static int HEAD_SPINS __read_mostly = (1<<13); +static bool pv_yield_owner __read_mostly = true; + static DEFINE_PER_CPU_ALIGNED(struct qnodes, qnodes); static inline u32 encode_tail_cpu(void) @@ -36,7 +39,11 @@ static inline int get_tail_cpu(u32 val) return (val >> _Q_TAIL_CPU_OFFSET) - 1; } -/* Take the lock by setting the bit, no other CPUs may concurrently lock it. */ +static inline int get_owner_cpu(u32 val) +{ + return (val & _Q_OWNER_CPU_MASK) >> _Q_OWNER_CPU_OFFSET; +} + /* Take the lock by setting the lock bit, no other CPUs will touch it. */ static __always_inline void lock_set_locked(struct qspinlock *lock) { @@ -170,7 +177,45 @@ static inline struct qnode *get_tail_qnode(struct qspinlock *lock, u32 val) BUG(); } -static inline bool try_to_steal_lock(struct qspinlock *lock) +static void yield_to_locked_owner(struct qspinlock *lock, u32 val, bool paravirt) +{ + int owner; + u32 yield_count; + + BUG_ON(!(val & _Q_LOCKED_VAL)); + + if (!paravirt) + goto relax; + + if (!pv_yield_owner) + goto relax; + + owner = get_owner_cpu(val); + yield_count = yield_count_of(owner); + + if ((yield_count & 1) == 0) + goto relax; /* owner vcpu is running */ + + /* + * Read the lock word after sampling the yield count. On the other side + * there may a wmb because the yield count update is done by the + * hypervisor preemption and the value update by the OS, however this + * ordering might reduce the chance of out of order accesses and + * improve the heuristic. + */ + smp_rmb(); + + if (READ_ONCE(lock->val) == val) { + yield_to_preempted(owner, yield_count); + /* Don't relax if we yielded. Maybe we should? */ + return; + } +relax: + cpu_relax(); +} + + +static __always_inline bool try_to_steal_lock(struct qspinlock *lock, bool paravirt) { int iters; @@ -182,7 +227,7 @@ static inline bool try_to_steal_lock(struct qspinlock *lock) break; if (val & _Q_LOCKED_VAL) { - cpu_relax(); + yield_to_locked_owner(lock, val, paravirt); continue; } @@ -193,7 +238,7 @@ static inline bool try_to_steal_lock(struct qspinlock *lock) return false; } -static inline void queued_spin_lock_mcs_queue(struct qspinlock *lock) +static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, bool paravirt) { struct qnodes *qnodesp; struct qnode *next, *node; @@ -245,7 +290,7 @@ static inline void queued_spin_lock_mcs_queue(struct qspinlock *lock) if (!MAYBE_STEALERS) { /* We're at the head of the waitqueue, wait for the lock. */ while ((val = READ_ONCE(lock->val)) & _Q_LOCKED_VAL) - cpu_relax(); + yield_to_locked_owner(lock, val, paravirt); /* If we're the last queued, must clean up the tail. */ if ((val & _Q_TAIL_CPU_MASK) == tail) { @@ -261,9 +306,11 @@ static inline void queued_spin_lock_mcs_queue(struct qspinlock *lock) again: /* We're at the head of the waitqueue, wait for the lock. */ while ((val = READ_ONCE(lock->val)) & _Q_LOCKED_VAL) { - if (iters++ == HEAD_SPINS) + if (iters++ == HEAD_SPINS) { lock_set_mustq(lock); - cpu_relax(); + val |= _Q_MUST_Q_VAL; + } + yield_to_locked_owner(lock, val, paravirt); } /* If we're the last queued, must clean up the tail. */ @@ -298,10 +345,15 @@ static inline void queued_spin_lock_mcs_queue(struct qspinlock *lock) void queued_spin_lock_slowpath(struct qspinlock *lock) { - if (try_to_steal_lock(lock)) - return; - - queued_spin_lock_mcs_queue(lock); + if (IS_ENABLED(CONFIG_PARAVIRT_SPINLOCKS) && is_shared_processor()) { + if (try_to_steal_lock(lock, true)) + return; + queued_spin_lock_mcs_queue(lock, true); + } else { + if (try_to_steal_lock(lock, false)) + return; + queued_spin_lock_mcs_queue(lock, false); + } } EXPORT_SYMBOL(queued_spin_lock_slowpath); @@ -362,10 +414,29 @@ static int head_spins_get(void *data, u64 *val) DEFINE_SIMPLE_ATTRIBUTE(fops_head_spins, head_spins_get, head_spins_set, "%llu\n"); +static int pv_yield_owner_set(void *data, u64 val) +{ + pv_yield_owner = !!val; + + return 0; +} + +static int pv_yield_owner_get(void *data, u64 *val) +{ + *val = pv_yield_owner; + + return 0; +} + +DEFINE_SIMPLE_ATTRIBUTE(fops_pv_yield_owner, pv_yield_owner_get, pv_yield_owner_set, "%llu\n"); + static __init int spinlock_debugfs_init(void) { debugfs_create_file("qspl_steal_spins", 0600, arch_debugfs_dir, NULL, &fops_steal_spins); debugfs_create_file("qspl_head_spins", 0600, arch_debugfs_dir, NULL, &fops_head_spins); + if (is_shared_processor()) { + debugfs_create_file("qspl_pv_yield_owner", 0600, arch_debugfs_dir, NULL, &fops_pv_yield_owner); + } return 0; } From patchwork Mon Jul 11 03:04:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1654623 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=TlbVcN9J; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lh8645HtGz9ryY for ; Mon, 11 Jul 2022 13:10:52 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Lh8641jdFz3fMd for ; Mon, 11 Jul 2022 13:10:52 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=TlbVcN9J; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::102a; helo=mail-pj1-x102a.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=TlbVcN9J; dkim-atps=neutral Received: from mail-pj1-x102a.google.com (mail-pj1-x102a.google.com [IPv6:2607:f8b0:4864:20::102a]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Lh7zk38xHz3cFW for ; Mon, 11 Jul 2022 13:05:22 +1000 (AEST) Received: by mail-pj1-x102a.google.com with SMTP id o31-20020a17090a0a2200b001ef7bd037bbso3744643pjo.0 for ; Sun, 10 Jul 2022 20:05:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pcEsjqfe9z82LNi/h168uZWmWZKxEa/6hhUbZNwifdA=; b=TlbVcN9JpFi1Scas0JcVJTQmKbwvCPDxP3CW0QSas0BLH7TkwZ9i8rIdlc3c6XwEj8 wJaCOqnPEUEJjg5SbF/8eB2FODt7iVvr62uZX+uklOWp6TStoAEb6KywOSMFsKyyTewf bYlko2/dYkaFc4CFo+DO4YXg5cu4yNzwP/KCgVbRnrJs+6XUrcZVAIjoArbv+Br0Bx+U 38UB6IKBPPHC/y2oGFumAHLc1Bf4n3yJdkZFqN42kqUf8YE1knNQfpoHfpAHC/f0CTRM Iqu0gRbcrK86b+XJNksTdZUb06KiN5Hvz20gXnZXdtsrd+J/cv+e2Vu7vRo/vEmopAYd 7Jgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pcEsjqfe9z82LNi/h168uZWmWZKxEa/6hhUbZNwifdA=; b=EEc0Gu2IaO2P7k3W+Ig4WNVlK0P/ZWIZ0KG4TV2DhDSP54qt2cphco+eOArbC4LQ46 wmjIfhIgW5965B5f/ROjyASv1k7d2TEeDnUISz9ZJrosDjp5VjbrC+Q60QLW5DCLKawW 05aYGPo5VYTT5adyWHgg/w+Uh1lpZNvIjn3QCgFu91WQDD31upHZUqtOWkD5cQEpZBuX oZjZ8ISyT9sNX7P+/p4uVdpw+XQKBXFRiysWRNLGOg6HCrlLLBvKUTnxxSYrSt1a7niY l/0zzXojDD+X3KbzxqeVDNJe4J7R7H4W61fdmxiyh9/kDacHLQHveCkuxJJA0kB96PJV pXeg== X-Gm-Message-State: AJIora83/ARid7+UHrewYBQwAly4FnMdwqy20gqtW9igcLMVzg3CayX0 mlIkjhWl3kbqUw1+CWf9BDEQH33sFrc= X-Google-Smtp-Source: AGRyM1uSIyAdkqPciYVSXTeuFtM9nmAOLUbAjuZ8cd2foE/Fr+Ov/y1/Z1qXR6CULEoaNoba2/dfCA== X-Received: by 2002:a17:903:3011:b0:16c:3d34:fde4 with SMTP id o17-20020a170903301100b0016c3d34fde4mr7183128pla.79.1657508720361; Sun, 10 Jul 2022 20:05:20 -0700 (PDT) Received: from bobo.ibm.com ([203.220.77.143]) by smtp.gmail.com with ESMTPSA id o4-20020a17090a5b0400b001ef8c81959dsm3562232pji.16.2022.07.10.20.05.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Jul 2022 20:05:20 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 09/14] powerpc/qspinlock: implement option to yield to previous node Date: Mon, 11 Jul 2022 13:04:48 +1000 Message-Id: <20220711030453.150644-10-npiggin@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220711030453.150644-1-npiggin@gmail.com> References: <20220711030453.150644-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Queued waiters which are not at the head of the queue don't spin on the lock word but their qnode lock word, waiting for the previous queued CPU to release them. Add an option which allows these waiters to yield to the previous CPU if its vCPU is preempted. Disable this option by default for now, i.e., no logical change. --- arch/powerpc/lib/qspinlock.c | 46 +++++++++++++++++++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-) diff --git a/arch/powerpc/lib/qspinlock.c b/arch/powerpc/lib/qspinlock.c index ee85198d5e28..1a58ed51c060 100644 --- a/arch/powerpc/lib/qspinlock.c +++ b/arch/powerpc/lib/qspinlock.c @@ -26,6 +26,7 @@ static bool MAYBE_STEALERS __read_mostly = true; static int HEAD_SPINS __read_mostly = (1<<13); static bool pv_yield_owner __read_mostly = true; +static bool pv_yield_prev __read_mostly = false; static DEFINE_PER_CPU_ALIGNED(struct qnodes, qnodes); @@ -214,6 +215,31 @@ static void yield_to_locked_owner(struct qspinlock *lock, u32 val, bool paravirt cpu_relax(); } +static void yield_to_prev(struct qspinlock *lock, struct qnode *node, int prev_cpu, bool paravirt) +{ + u32 yield_count; + + if (!paravirt) + goto relax; + + if (!pv_yield_prev) + goto relax; + + yield_count = yield_count_of(prev_cpu); + if ((yield_count & 1) == 0) + goto relax; /* owner vcpu is running */ + + smp_rmb(); /* See yield_to_locked_owner comment */ + + if (!node->locked) { + yield_to_preempted(prev_cpu, yield_count); + return; + } + +relax: + cpu_relax(); +} + static __always_inline bool try_to_steal_lock(struct qspinlock *lock, bool paravirt) { @@ -276,13 +302,14 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b */ if (old & _Q_TAIL_CPU_MASK) { struct qnode *prev = get_tail_qnode(lock, old); + int prev_cpu = get_tail_cpu(old); /* Link @node into the waitqueue. */ WRITE_ONCE(prev->next, node); /* Wait for mcs node lock to be released */ while (!node->locked) - cpu_relax(); + yield_to_prev(lock, node, prev_cpu, paravirt); smp_rmb(); /* acquire barrier for the mcs lock */ } @@ -430,12 +457,29 @@ static int pv_yield_owner_get(void *data, u64 *val) DEFINE_SIMPLE_ATTRIBUTE(fops_pv_yield_owner, pv_yield_owner_get, pv_yield_owner_set, "%llu\n"); +static int pv_yield_prev_set(void *data, u64 val) +{ + pv_yield_prev = !!val; + + return 0; +} + +static int pv_yield_prev_get(void *data, u64 *val) +{ + *val = pv_yield_prev; + + return 0; +} + +DEFINE_SIMPLE_ATTRIBUTE(fops_pv_yield_prev, pv_yield_prev_get, pv_yield_prev_set, "%llu\n"); + static __init int spinlock_debugfs_init(void) { debugfs_create_file("qspl_steal_spins", 0600, arch_debugfs_dir, NULL, &fops_steal_spins); debugfs_create_file("qspl_head_spins", 0600, arch_debugfs_dir, NULL, &fops_head_spins); if (is_shared_processor()) { debugfs_create_file("qspl_pv_yield_owner", 0600, arch_debugfs_dir, NULL, &fops_pv_yield_owner); + debugfs_create_file("qspl_pv_yield_prev", 0600, arch_debugfs_dir, NULL, &fops_pv_yield_prev); } return 0; From patchwork Mon Jul 11 03:04:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1654624 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=h5GXpzY1; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lh86l4ySwz9ryY for ; Mon, 11 Jul 2022 13:11:27 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Lh86l3Nndz3fQw for ; Mon, 11 Jul 2022 13:11:27 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=h5GXpzY1; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::430; helo=mail-pf1-x430.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=h5GXpzY1; dkim-atps=neutral Received: from mail-pf1-x430.google.com (mail-pf1-x430.google.com [IPv6:2607:f8b0:4864:20::430]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Lh7zm6XJxz3cS8 for ; Mon, 11 Jul 2022 13:05:24 +1000 (AEST) Received: by mail-pf1-x430.google.com with SMTP id v7so1638311pfb.0 for ; Sun, 10 Jul 2022 20:05:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=AwQ1wrsYWuJZ0vQCidAPq6J1aA8T0687rJ/fusLTP1A=; b=h5GXpzY1uxRfskLMYw7M4H9NrxdwSWwDt2QAoyYdg26omsVL0zR1knvI78bwinZv1p xpwLG63sgCeIT+CiJzuRkJlktPZOMmZwbVDWH4GrD7uu5rcnOji0QAMS4F5d5wHUI737 fLaT5TXOcfAay/+dO62t2mXOcB8+wcHrSxIjAhhE4scC2MulInL1EREr1tPUYIyLVquH 1ieFfIW2oU42ULQ0416fq8n2uy/iBPm31VfWxKuNK5c+mGd2aMJFA8BIjqaLwMH+p44K 9KrIsTA3BAovnfvvr/8VZMRwcIfSvj7IzkyvXY+qBIdb9uVXHRN//RrEDCfCptyK+Tlr SNyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=AwQ1wrsYWuJZ0vQCidAPq6J1aA8T0687rJ/fusLTP1A=; b=R17+fRh/+YSeCJWuPglQDE1mmvvEMV4jxrZO75qmEgiZbZl91ZdHrLNgXhmdmocxsK 80l+tnwA3ccZKCy8QeV30ox4E4/LaU79oLe9uEPBXLMz06Z1xOJ760zzLMRMQWrdLuZu tgzGqSJz37uGjuUF8FwmpVETvvRLb36lsfrv2sUQdWG+WXh4XaMGt06epj3qGVc7pld8 aXGo8vPqZONmBaNd0AfRETWsDs8o8oPGREXhTpAUb00ZBsXDp7jph5NCFHYvuHWjO3T+ SmM71LBeMVGt835ZtxS8tR2xcOp9sMVMpXJgsKFOc59pcz2LNgQj0gSddj/QbttE3Uo4 HoXg== X-Gm-Message-State: AJIora8OzPLWNiz/S8bb85g7MWzkhwoXWyplmJL75VHOKaiLRMhjNK9J AAv+eu7cuvRaL9M4JjAWrguQfkw/xrE= X-Google-Smtp-Source: AGRyM1tDmIGNBCpKhTVcSQoQYftuMRwA04hOFHDlt2xzrpXRJAXhENs/M2LTH673bDUXWZvPlU9BDg== X-Received: by 2002:a05:6a00:80d:b0:525:520a:1736 with SMTP id m13-20020a056a00080d00b00525520a1736mr16227733pfk.36.1657508722445; Sun, 10 Jul 2022 20:05:22 -0700 (PDT) Received: from bobo.ibm.com ([203.220.77.143]) by smtp.gmail.com with ESMTPSA id o4-20020a17090a5b0400b001ef8c81959dsm3562232pji.16.2022.07.10.20.05.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Jul 2022 20:05:22 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 10/14] powerpc/qspinlock: allow stealing when head of queue yields Date: Mon, 11 Jul 2022 13:04:49 +1000 Message-Id: <20220711030453.150644-11-npiggin@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220711030453.150644-1-npiggin@gmail.com> References: <20220711030453.150644-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" If the head of queue is preventing stealing but it finds the owner vCPU is preempted, it will yield its cycles to the owner which could cause it to become preempted. Add an option to re-allow stealers before yielding, and disallow them again after returning from the yield. Disable this option by default for now, i.e., no logical change. --- arch/powerpc/lib/qspinlock.c | 49 +++++++++++++++++++++++++++++++++--- 1 file changed, 45 insertions(+), 4 deletions(-) diff --git a/arch/powerpc/lib/qspinlock.c b/arch/powerpc/lib/qspinlock.c index 1a58ed51c060..4f1dc3322485 100644 --- a/arch/powerpc/lib/qspinlock.c +++ b/arch/powerpc/lib/qspinlock.c @@ -26,6 +26,7 @@ static bool MAYBE_STEALERS __read_mostly = true; static int HEAD_SPINS __read_mostly = (1<<13); static bool pv_yield_owner __read_mostly = true; +static bool pv_yield_allow_steal __read_mostly = false; static bool pv_yield_prev __read_mostly = false; static DEFINE_PER_CPU_ALIGNED(struct qnodes, qnodes); @@ -163,6 +164,24 @@ static __always_inline u32 lock_set_mustq(struct qspinlock *lock) return prev; } +static __always_inline u32 lock_clear_mustq(struct qspinlock *lock) +{ + u32 new = _Q_MUST_Q_VAL; + u32 prev; + + asm volatile( +"1: lwarx %0,0,%1 # queued_spin_set_mustq \n" +" andc %0,%0,%2 \n" +" stwcx. %0,0,%1 \n" +" bne- 1b \n" + : "=&r" (prev) + : "r" (&lock->val), "r" (new) + : "cr0", "memory"); + + return prev; +} + + static inline struct qnode *get_tail_qnode(struct qspinlock *lock, u32 val) { int cpu = get_tail_cpu(val); @@ -178,7 +197,7 @@ static inline struct qnode *get_tail_qnode(struct qspinlock *lock, u32 val) BUG(); } -static void yield_to_locked_owner(struct qspinlock *lock, u32 val, bool paravirt) +static void yield_to_locked_owner(struct qspinlock *lock, u32 val, bool paravirt, bool clear_mustq) { int owner; u32 yield_count; @@ -207,7 +226,11 @@ static void yield_to_locked_owner(struct qspinlock *lock, u32 val, bool paravirt smp_rmb(); if (READ_ONCE(lock->val) == val) { + if (clear_mustq) + lock_clear_mustq(lock); yield_to_preempted(owner, yield_count); + if (clear_mustq) + lock_set_mustq(lock); /* Don't relax if we yielded. Maybe we should? */ return; } @@ -253,7 +276,7 @@ static __always_inline bool try_to_steal_lock(struct qspinlock *lock, bool parav break; if (val & _Q_LOCKED_VAL) { - yield_to_locked_owner(lock, val, paravirt); + yield_to_locked_owner(lock, val, paravirt, false); continue; } @@ -317,7 +340,7 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b if (!MAYBE_STEALERS) { /* We're at the head of the waitqueue, wait for the lock. */ while ((val = READ_ONCE(lock->val)) & _Q_LOCKED_VAL) - yield_to_locked_owner(lock, val, paravirt); + yield_to_locked_owner(lock, val, paravirt, false); /* If we're the last queued, must clean up the tail. */ if ((val & _Q_TAIL_CPU_MASK) == tail) { @@ -337,7 +360,8 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b lock_set_mustq(lock); val |= _Q_MUST_Q_VAL; } - yield_to_locked_owner(lock, val, paravirt); + yield_to_locked_owner(lock, val, paravirt, + pv_yield_allow_steal && (iters > HEAD_SPINS)); } /* If we're the last queued, must clean up the tail. */ @@ -457,6 +481,22 @@ static int pv_yield_owner_get(void *data, u64 *val) DEFINE_SIMPLE_ATTRIBUTE(fops_pv_yield_owner, pv_yield_owner_get, pv_yield_owner_set, "%llu\n"); +static int pv_yield_allow_steal_set(void *data, u64 val) +{ + pv_yield_allow_steal = !!val; + + return 0; +} + +static int pv_yield_allow_steal_get(void *data, u64 *val) +{ + *val = pv_yield_allow_steal; + + return 0; +} + +DEFINE_SIMPLE_ATTRIBUTE(fops_pv_yield_allow_steal, pv_yield_allow_steal_get, pv_yield_allow_steal_set, "%llu\n"); + static int pv_yield_prev_set(void *data, u64 val) { pv_yield_prev = !!val; @@ -479,6 +519,7 @@ static __init int spinlock_debugfs_init(void) debugfs_create_file("qspl_head_spins", 0600, arch_debugfs_dir, NULL, &fops_head_spins); if (is_shared_processor()) { debugfs_create_file("qspl_pv_yield_owner", 0600, arch_debugfs_dir, NULL, &fops_pv_yield_owner); + debugfs_create_file("qspl_pv_yield_allow_steal", 0600, arch_debugfs_dir, NULL, &fops_pv_yield_allow_steal); debugfs_create_file("qspl_pv_yield_prev", 0600, arch_debugfs_dir, NULL, &fops_pv_yield_prev); } From patchwork Mon Jul 11 03:04:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1654625 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=ZT5DoGnl; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lh87X6Dclz9ryY for ; Mon, 11 Jul 2022 13:12:08 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Lh87X4k9Yz3fTW for ; Mon, 11 Jul 2022 13:12:08 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=ZT5DoGnl; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::1035; helo=mail-pj1-x1035.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=ZT5DoGnl; dkim-atps=neutral Received: from mail-pj1-x1035.google.com (mail-pj1-x1035.google.com [IPv6:2607:f8b0:4864:20::1035]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Lh7zq1WFHz3cdl for ; Mon, 11 Jul 2022 13:05:26 +1000 (AEST) Received: by mail-pj1-x1035.google.com with SMTP id v4-20020a17090abb8400b001ef966652a3so7041651pjr.4 for ; Sun, 10 Jul 2022 20:05:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=IE8MQEsQ355Hs7Q4bU0ECDFtQ5VB9dWEJo0U5GfufEQ=; b=ZT5DoGnlL+jxedt+pb3QhEkvXWywZF6Ioi24nUuzfrnw3+pv1hcuboGi7vcZXdVm/E Zh2Scb3z8AqWHf8yh7firXLKAjeeYG54s76MLEWiwi2boDFPH4RyjTCafRjeXf/jQ3fy fgFSuriZoB/Ah66dSfvexnLqqZuM8osxEyPAcqEIICEWTbHT9nyeUjle/67EAlAuuBXs PwjOP7vyMS4aBOYH1J9EXhedqLnNnlsWfcwne9VbyTijyyBoRvMesKurDGaLj5FZXOPe wQatZGVIJXQiUZEq+wkJR9LakhGETf0fj9PmD5A/k+ER2ScxpbvFvEnwuZndahReKVnB wNcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=IE8MQEsQ355Hs7Q4bU0ECDFtQ5VB9dWEJo0U5GfufEQ=; b=I4TpF+3fPGafQXgJaTh0nM4BezXr63ZO8i6N+uzMvgaF4bo+WF0zCim2/btRmB40Ls ZZsxw8CArpgbHlIoB2i5z6ZceZeSynfF7gdHBIPDGG5Onxzgw3bdM2GljwNWxdnXikIL f/PQZRDGEItxBfVFZqEXT/6pA++E7iQJEjKUy035MIXfVlgAtmj6+Nr4nT8LAVSf5U3z Ie7kXRMkyZtzauHgatC3k19qNh73WkgrD0FquGzmWKqMsFecAbt1HVYtZN6jI5dXbFgy 3Yu3dPsnK3gPqf1jwmPwlqjo+6N3Q/bDlJjBlsCPKtPkVF+Gd+KBEquFuVAZuDjO+oDF VaPw== X-Gm-Message-State: AJIora/HWtRwYUZISb8T6h9OQwd9Nlgfvgfzg1TJGXg4Lf8N0pmjn5qG haXQ5MA5tExIbtU4Fx6uOKoLN+Jk974= X-Google-Smtp-Source: AGRyM1vv+EdkZwrfHFXk351DOzZS4FqZfWqJvD77K6tsIyTkueCcxMYC40wajuPpqI5zWaYQJUE6kA== X-Received: by 2002:a17:902:e84f:b0:16b:8a74:d572 with SMTP id t15-20020a170902e84f00b0016b8a74d572mr16401951plg.146.1657508724688; Sun, 10 Jul 2022 20:05:24 -0700 (PDT) Received: from bobo.ibm.com ([203.220.77.143]) by smtp.gmail.com with ESMTPSA id o4-20020a17090a5b0400b001ef8c81959dsm3562232pji.16.2022.07.10.20.05.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Jul 2022 20:05:24 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 11/14] powerpc/qspinlock: allow propagation of yield CPU down the queue Date: Mon, 11 Jul 2022 13:04:50 +1000 Message-Id: <20220711030453.150644-12-npiggin@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220711030453.150644-1-npiggin@gmail.com> References: <20220711030453.150644-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Having all CPUs poll the lock word for the owner CPU that should be yielded to defeats most of the purpose of using MCS queueing for scalability. Yet it may be desirable for queued waiters to to yield to a preempted owner. s390 addreses this problem by having queued waiters sample the lock word to find the owner much less frequently. In this approach, the waiters never sample it directly, but the queue head propagates the owner CPU back to the next waiter if it ever finds the owner has been preempted. Queued waiters then subsequently propagate the owner CPU back to the next waiter, and so on. Disable this option by default for now, i.e., no logical change. --- arch/powerpc/lib/qspinlock.c | 103 +++++++++++++++++++++++++++++++++-- 1 file changed, 98 insertions(+), 5 deletions(-) diff --git a/arch/powerpc/lib/qspinlock.c b/arch/powerpc/lib/qspinlock.c index 4f1dc3322485..c39af19f006e 100644 --- a/arch/powerpc/lib/qspinlock.c +++ b/arch/powerpc/lib/qspinlock.c @@ -12,6 +12,7 @@ struct qnode { struct qnode *next; struct qspinlock *lock; + int yield_cpu; u8 locked; /* 1 if lock acquired */ }; @@ -28,6 +29,7 @@ static int HEAD_SPINS __read_mostly = (1<<13); static bool pv_yield_owner __read_mostly = true; static bool pv_yield_allow_steal __read_mostly = false; static bool pv_yield_prev __read_mostly = false; +static bool pv_yield_propagate_owner __read_mostly = false; static DEFINE_PER_CPU_ALIGNED(struct qnodes, qnodes); @@ -197,7 +199,7 @@ static inline struct qnode *get_tail_qnode(struct qspinlock *lock, u32 val) BUG(); } -static void yield_to_locked_owner(struct qspinlock *lock, u32 val, bool paravirt, bool clear_mustq) +static void __yield_to_locked_owner(struct qspinlock *lock, u32 val, bool paravirt, bool clear_mustq) { int owner; u32 yield_count; @@ -238,13 +240,76 @@ static void yield_to_locked_owner(struct qspinlock *lock, u32 val, bool paravirt cpu_relax(); } +static void yield_to_locked_owner(struct qspinlock *lock, u32 val, bool paravirt) +{ + __yield_to_locked_owner(lock, val, paravirt, false); +} + +static void yield_head_to_locked_owner(struct qspinlock *lock, u32 val, bool paravirt, bool clear_mustq) +{ + __yield_to_locked_owner(lock, val, paravirt, clear_mustq); +} + +static void propagate_yield_cpu(struct qnode *node, u32 val, int *set_yield_cpu, bool paravirt) +{ + struct qnode *next; + int owner; + + if (!paravirt) + return; + if (!pv_yield_propagate_owner) + return; + + owner = get_owner_cpu(val); + if (*set_yield_cpu == owner) + return; + + next = READ_ONCE(node->next); + if (!next) + return; + + if (vcpu_is_preempted(owner)) { + next->yield_cpu = owner; + *set_yield_cpu = owner; + } else if (*set_yield_cpu != -1) { + next->yield_cpu = owner; + *set_yield_cpu = owner; + } +} + static void yield_to_prev(struct qspinlock *lock, struct qnode *node, int prev_cpu, bool paravirt) { u32 yield_count; + int yield_cpu; if (!paravirt) goto relax; + if (!pv_yield_propagate_owner) + goto yield_prev; + + yield_cpu = READ_ONCE(node->yield_cpu); + if (yield_cpu == -1) { + /* Propagate back the -1 CPU */ + if (node->next && node->next->yield_cpu != -1) + node->next->yield_cpu = yield_cpu; + goto yield_prev; + } + + yield_count = yield_count_of(yield_cpu); + if ((yield_count & 1) == 0) + goto yield_prev; /* owner vcpu is running */ + + smp_rmb(); + + if (yield_cpu == node->yield_cpu) { + if (node->next && node->next->yield_cpu != yield_cpu) + node->next->yield_cpu = yield_cpu; + yield_to_preempted(yield_cpu, yield_count); + return; + } + +yield_prev: if (!pv_yield_prev) goto relax; @@ -276,7 +341,7 @@ static __always_inline bool try_to_steal_lock(struct qspinlock *lock, bool parav break; if (val & _Q_LOCKED_VAL) { - yield_to_locked_owner(lock, val, paravirt, false); + yield_to_locked_owner(lock, val, paravirt); continue; } @@ -313,6 +378,7 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b node = &qnodesp->nodes[idx]; node->next = NULL; node->lock = lock; + node->yield_cpu = -1; node->locked = 0; tail = encode_tail_cpu(); @@ -334,13 +400,21 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b while (!node->locked) yield_to_prev(lock, node, prev_cpu, paravirt); + /* Clear out stale propagated yield_cpu */ + if (paravirt && pv_yield_propagate_owner && node->yield_cpu != -1) + node->yield_cpu = -1; + smp_rmb(); /* acquire barrier for the mcs lock */ } if (!MAYBE_STEALERS) { + int set_yield_cpu = -1; + /* We're at the head of the waitqueue, wait for the lock. */ - while ((val = READ_ONCE(lock->val)) & _Q_LOCKED_VAL) - yield_to_locked_owner(lock, val, paravirt, false); + while ((val = READ_ONCE(lock->val)) & _Q_LOCKED_VAL) { + propagate_yield_cpu(node, val, &set_yield_cpu, paravirt); + yield_head_to_locked_owner(lock, val, paravirt, false); + } /* If we're the last queued, must clean up the tail. */ if ((val & _Q_TAIL_CPU_MASK) == tail) { @@ -352,6 +426,7 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b /* We must be the owner, just set the lock bit and acquire */ lock_set_locked(lock); } else { + int set_yield_cpu = -1; int iters = 0; again: /* We're at the head of the waitqueue, wait for the lock. */ @@ -360,7 +435,8 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b lock_set_mustq(lock); val |= _Q_MUST_Q_VAL; } - yield_to_locked_owner(lock, val, paravirt, + propagate_yield_cpu(node, val, &set_yield_cpu, paravirt); + yield_head_to_locked_owner(lock, val, paravirt, pv_yield_allow_steal && (iters > HEAD_SPINS)); } @@ -513,6 +589,22 @@ static int pv_yield_prev_get(void *data, u64 *val) DEFINE_SIMPLE_ATTRIBUTE(fops_pv_yield_prev, pv_yield_prev_get, pv_yield_prev_set, "%llu\n"); +static int pv_yield_propagate_owner_set(void *data, u64 val) +{ + pv_yield_propagate_owner = !!val; + + return 0; +} + +static int pv_yield_propagate_owner_get(void *data, u64 *val) +{ + *val = pv_yield_propagate_owner; + + return 0; +} + +DEFINE_SIMPLE_ATTRIBUTE(fops_pv_yield_propagate_owner, pv_yield_propagate_owner_get, pv_yield_propagate_owner_set, "%llu\n"); + static __init int spinlock_debugfs_init(void) { debugfs_create_file("qspl_steal_spins", 0600, arch_debugfs_dir, NULL, &fops_steal_spins); @@ -521,6 +613,7 @@ static __init int spinlock_debugfs_init(void) debugfs_create_file("qspl_pv_yield_owner", 0600, arch_debugfs_dir, NULL, &fops_pv_yield_owner); debugfs_create_file("qspl_pv_yield_allow_steal", 0600, arch_debugfs_dir, NULL, &fops_pv_yield_allow_steal); debugfs_create_file("qspl_pv_yield_prev", 0600, arch_debugfs_dir, NULL, &fops_pv_yield_prev); + debugfs_create_file("qspl_pv_yield_propagate_owner", 0600, arch_debugfs_dir, NULL, &fops_pv_yield_propagate_owner); } return 0; From patchwork Mon Jul 11 03:04:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1654626 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=fgFAZ4a3; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lh88D1Ggvz9ryY for ; Mon, 11 Jul 2022 13:12:44 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Lh88C52xTz3f9S for ; Mon, 11 Jul 2022 13:12:43 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=fgFAZ4a3; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::102a; helo=mail-pj1-x102a.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=fgFAZ4a3; dkim-atps=neutral Received: from mail-pj1-x102a.google.com (mail-pj1-x102a.google.com [IPv6:2607:f8b0:4864:20::102a]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Lh7zr5Sw9z3c1T for ; Mon, 11 Jul 2022 13:05:28 +1000 (AEST) Received: by mail-pj1-x102a.google.com with SMTP id z12-20020a17090a7b8c00b001ef84000b8bso7040391pjc.1 for ; Sun, 10 Jul 2022 20:05:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=L7MAN4mRWMKVF7KUEsEUoGY4r81MiARcRETbL/y8vEc=; b=fgFAZ4a3PQB7uZjlKeK2/hh9rxMm0ip1g8R3RqeIytPW37MzO6fO9jqtwDVpzWUMkE yJMEzYKfTISk+qxgxAFYmhm95icH5AhzV18Xa46CIIWjKlOXfec9TioPvMsE4Y61U7qC VIHiQUioCZ7axt/77q7T37opavme4nhdgq2Hx0YBnbVlk33BYR58FQ66MhOuSEU7mi4z 2hAKorfcPywh9XpmodOWVK7JuCp4DsBLS1qdRLSN1HDebdASC9Y090tiBjfnm7M21Ghl zZRHxlamw+kM7/T9O/PkkLDeLcxJVnGHb9VJG22GpSgHSNnpkp2igxBB9lWr1WrHDa/E WI6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=L7MAN4mRWMKVF7KUEsEUoGY4r81MiARcRETbL/y8vEc=; b=6evJnYXHCZViOo7SfkOyXiLEkw9sUFkjF6mZoJBPKCLmwk89fqubW7udktoeVRwULs CkbxCD0XINxCBZ4+iSgE4CexQavdspLjHWgj4pYYgpdP4oRAazDHMhYl1rG1nIOm4EwI OiRUMiq/11gGuYOITNy2rHOD0y/yOrJyJ3wCqWMIauiOk4PJsu2Caeza6Fy6J6Vxmo0y yCI9LxGLE9/u6sWIwaQI3QO7JqZAuhDTBLXrBC35FzBaV5w3RGmJLS1SIAfTDzCh9lmZ cu97MAv04sQ2PYNmGmwh5KrU10Ds3OCMMhzCfgCqUaLVUibAPoXDSn7EFHDpElTgGeiE +76Q== X-Gm-Message-State: AJIora8KrsZnlU4OEc+RTuL0JQf0/UHeTHDicQ0NUSuBJFuQhzGwJTWu 2hLU9GBeo0wta6hCJPnmJ9SuHMaGT90= X-Google-Smtp-Source: AGRyM1t+zut1plVylBNllSbrBelRluNWA4pp9vG6GMe/CeeVIwFi9vd8SOVtvgKJ84x0Zgrlkg+k4A== X-Received: by 2002:a17:90b:1c90:b0:1ee:d804:d2c1 with SMTP id oo16-20020a17090b1c9000b001eed804d2c1mr15303856pjb.92.1657508726861; Sun, 10 Jul 2022 20:05:26 -0700 (PDT) Received: from bobo.ibm.com ([203.220.77.143]) by smtp.gmail.com with ESMTPSA id o4-20020a17090a5b0400b001ef8c81959dsm3562232pji.16.2022.07.10.20.05.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Jul 2022 20:05:26 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 12/14] powerpc/qspinlock: add ability to prod new queue head CPU Date: Mon, 11 Jul 2022 13:04:51 +1000 Message-Id: <20220711030453.150644-13-npiggin@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220711030453.150644-1-npiggin@gmail.com> References: <20220711030453.150644-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" After the head of the queue acquires the lock, it releases the next waiter in the queue to become the new head. Add an option to prod the new head if its vCPU was preempted. This may only have an effect if queue waiters are yielding. Disable this option by default for now, i.e., no logical change. --- arch/powerpc/lib/qspinlock.c | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/arch/powerpc/lib/qspinlock.c b/arch/powerpc/lib/qspinlock.c index c39af19f006e..ce0563c56915 100644 --- a/arch/powerpc/lib/qspinlock.c +++ b/arch/powerpc/lib/qspinlock.c @@ -12,6 +12,7 @@ struct qnode { struct qnode *next; struct qspinlock *lock; + int cpu; int yield_cpu; u8 locked; /* 1 if lock acquired */ }; @@ -30,6 +31,7 @@ static bool pv_yield_owner __read_mostly = true; static bool pv_yield_allow_steal __read_mostly = false; static bool pv_yield_prev __read_mostly = false; static bool pv_yield_propagate_owner __read_mostly = false; +static bool pv_prod_head __read_mostly = false; static DEFINE_PER_CPU_ALIGNED(struct qnodes, qnodes); @@ -378,6 +380,7 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b node = &qnodesp->nodes[idx]; node->next = NULL; node->lock = lock; + node->cpu = smp_processor_id(); node->yield_cpu = -1; node->locked = 0; @@ -464,7 +467,14 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b * this store to locked. The corresponding barrier is the smp_rmb() * acquire barrier for mcs lock, above. */ - WRITE_ONCE(next->locked, 1); + if (paravirt && pv_prod_head) { + int next_cpu = next->cpu; + WRITE_ONCE(next->locked, 1); + if (vcpu_is_preempted(next_cpu)) + prod_cpu(next_cpu); + } else { + WRITE_ONCE(next->locked, 1); + } release: qnodesp->count--; /* release the node */ @@ -605,6 +615,22 @@ static int pv_yield_propagate_owner_get(void *data, u64 *val) DEFINE_SIMPLE_ATTRIBUTE(fops_pv_yield_propagate_owner, pv_yield_propagate_owner_get, pv_yield_propagate_owner_set, "%llu\n"); +static int pv_prod_head_set(void *data, u64 val) +{ + pv_prod_head = !!val; + + return 0; +} + +static int pv_prod_head_get(void *data, u64 *val) +{ + *val = pv_prod_head; + + return 0; +} + +DEFINE_SIMPLE_ATTRIBUTE(fops_pv_prod_head, pv_prod_head_get, pv_prod_head_set, "%llu\n"); + static __init int spinlock_debugfs_init(void) { debugfs_create_file("qspl_steal_spins", 0600, arch_debugfs_dir, NULL, &fops_steal_spins); @@ -614,6 +640,7 @@ static __init int spinlock_debugfs_init(void) debugfs_create_file("qspl_pv_yield_allow_steal", 0600, arch_debugfs_dir, NULL, &fops_pv_yield_allow_steal); debugfs_create_file("qspl_pv_yield_prev", 0600, arch_debugfs_dir, NULL, &fops_pv_yield_prev); debugfs_create_file("qspl_pv_yield_propagate_owner", 0600, arch_debugfs_dir, NULL, &fops_pv_yield_propagate_owner); + debugfs_create_file("qspl_pv_prod_head", 0600, arch_debugfs_dir, NULL, &fops_pv_prod_head); } return 0; From patchwork Mon Jul 11 03:04:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1654627 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=UqTq3xQA; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lh88v1Yddz9ryY for ; Mon, 11 Jul 2022 13:13:19 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Lh88v03z7z3cLZ for ; Mon, 11 Jul 2022 13:13:19 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=UqTq3xQA; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::633; helo=mail-pl1-x633.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=UqTq3xQA; dkim-atps=neutral Received: from mail-pl1-x633.google.com (mail-pl1-x633.google.com [IPv6:2607:f8b0:4864:20::633]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Lh7zv5rNfz3cFL for ; Mon, 11 Jul 2022 13:05:31 +1000 (AEST) Received: by mail-pl1-x633.google.com with SMTP id j12so3343883plj.8 for ; Sun, 10 Jul 2022 20:05:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MVkkB6BjjOTsJFW9Qxm3jwsKHsI36EMDreg19DDa4Nc=; b=UqTq3xQAnyRWKy8SEf2KA33BsY+MfbXH9fxyCFkBWbZwObQpDMj2tupi8LyeQkl5D+ NZ4uYaD3S63twfAtrTAnOYCgEZf9ko3Oy9pGGwPs6mhRte9oinhkQA6g4Hzm8/AliFc2 nRkgBtyVA7T1h7lWfU84MSeHSKFzbsx56KkdKytuqYdCv8zGZkRbQIDMjLBAlU4KfDdS CaEhw4ylp8gxoYAUmXEMjJnitq9QjAIL5yqLJKl8KzsJNSIEyPM715dLOJDJIfnuV+W8 +OuhDzaW52qMHYDV0D6xBrygNu4o4Bnz0Se0hEan8JXTH8/0oxPek7FzXoPB6BkV9ju4 tJbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=MVkkB6BjjOTsJFW9Qxm3jwsKHsI36EMDreg19DDa4Nc=; b=isHYEyHYf4kiSnXyExb4KPMewVvVHedKXQNfnZdGxbkLRRKuwAKIvMGqDdH52fuGRL 5jnK6jv9UjmcN3jXfAUrpLsBGaSA4n+2HxfIwXCxNPdrNqhofD5vF+GQ7Ik6clCtxMYI rufAzprndGGscghv0EGyQAQBjFNe78CfbBRF3SHOLNA32BYETx77rwMlXpjn55sxuSKz HHnoxpsjOZMlK0d/f9UVgt+O6asSWzK06sS+7m9Sgk96guelhXuZ9hkf+WZESNHEvCFr TDYDn3wvWnm8aQYXE9rr29/8FeazA2Q4IcupPjM09QAgXo1JSIMHamP6TveZOWGe64u5 NqrQ== X-Gm-Message-State: AJIora8/KsmV1APF/g/9g8a86qgPU7Nas8scMe26V1DYdpBtnCpups0P 1d5mbL4fY8MchVC0xVRrwASDMgtW7qU= X-Google-Smtp-Source: AGRyM1tj8cKhtgKxYhsTXcLzZKln3POQeD6Dnc5rvNPGYKXZwDPzExSeYvrXfDUb8oJ1tTpXLw00UA== X-Received: by 2002:a17:902:b947:b0:16a:79b7:21f2 with SMTP id h7-20020a170902b94700b0016a79b721f2mr16370048pls.130.1657508729152; Sun, 10 Jul 2022 20:05:29 -0700 (PDT) Received: from bobo.ibm.com ([203.220.77.143]) by smtp.gmail.com with ESMTPSA id o4-20020a17090a5b0400b001ef8c81959dsm3562232pji.16.2022.07.10.20.05.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Jul 2022 20:05:28 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 13/14] powerpc/qspinlock: trylock and initial lock attempt may steal Date: Mon, 11 Jul 2022 13:04:52 +1000 Message-Id: <20220711030453.150644-14-npiggin@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220711030453.150644-1-npiggin@gmail.com> References: <20220711030453.150644-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" This gives trylock slightly more strength, and it also gives most of the benefit of passing 'val' back through the slowpath without the complexity. --- arch/powerpc/include/asm/qspinlock.h | 39 +++++++++++++++++++++++++++- arch/powerpc/lib/qspinlock.c | 9 +++++++ 2 files changed, 47 insertions(+), 1 deletion(-) diff --git a/arch/powerpc/include/asm/qspinlock.h b/arch/powerpc/include/asm/qspinlock.h index 44601b261e08..d3d2039237b2 100644 --- a/arch/powerpc/include/asm/qspinlock.h +++ b/arch/powerpc/include/asm/qspinlock.h @@ -5,6 +5,8 @@ #include #include +#define _Q_SPIN_TRY_LOCK_STEAL 1 + static __always_inline int queued_spin_is_locked(struct qspinlock *lock) { return READ_ONCE(lock->val); @@ -26,11 +28,12 @@ static __always_inline u32 queued_spin_get_locked_val(void) return _Q_LOCKED_VAL | (smp_processor_id() << _Q_OWNER_CPU_OFFSET); } -static __always_inline int queued_spin_trylock(struct qspinlock *lock) +static __always_inline int __queued_spin_trylock_nosteal(struct qspinlock *lock) { u32 new = queued_spin_get_locked_val(); u32 prev; + /* Trylock succeeds only when unlocked and no queued nodes */ asm volatile( "1: lwarx %0,0,%1,%3 # queued_spin_trylock \n" " cmpwi 0,%0,0 \n" @@ -49,6 +52,40 @@ static __always_inline int queued_spin_trylock(struct qspinlock *lock) return 0; } +static __always_inline int __queued_spin_trylock_steal(struct qspinlock *lock) +{ + u32 new = queued_spin_get_locked_val(); + u32 prev, tmp; + + /* Trylock may get ahead of queued nodes if it finds unlocked */ + asm volatile( +"1: lwarx %0,0,%2,%5 # queued_spin_trylock \n" +" andc. %1,%0,%4 \n" +" bne- 2f \n" +" and %1,%0,%4 \n" +" or %1,%1,%3 \n" +" stwcx. %1,0,%2 \n" +" bne- 1b \n" +"\t" PPC_ACQUIRE_BARRIER " \n" +"2: \n" + : "=&r" (prev), "=&r" (tmp) + : "r" (&lock->val), "r" (new), "r" (_Q_TAIL_CPU_MASK), + "i" (IS_ENABLED(CONFIG_PPC64) ? 1 : 0) + : "cr0", "memory"); + + if (likely(!(prev & ~_Q_TAIL_CPU_MASK))) + return 1; + return 0; +} + +static __always_inline int queued_spin_trylock(struct qspinlock *lock) +{ + if (!_Q_SPIN_TRY_LOCK_STEAL) + return __queued_spin_trylock_nosteal(lock); + else + return __queued_spin_trylock_steal(lock); +} + void queued_spin_lock_slowpath(struct qspinlock *lock); static __always_inline void queued_spin_lock(struct qspinlock *lock) diff --git a/arch/powerpc/lib/qspinlock.c b/arch/powerpc/lib/qspinlock.c index ce0563c56915..d67b923e4f98 100644 --- a/arch/powerpc/lib/qspinlock.c +++ b/arch/powerpc/lib/qspinlock.c @@ -24,7 +24,11 @@ struct qnodes { /* Tuning parameters */ static int STEAL_SPINS __read_mostly = (1<<5); +#if _Q_SPIN_TRY_LOCK_STEAL == 1 +static const bool MAYBE_STEALERS = true; +#else static bool MAYBE_STEALERS __read_mostly = true; +#endif static int HEAD_SPINS __read_mostly = (1<<13); static bool pv_yield_owner __read_mostly = true; @@ -505,6 +509,10 @@ void pv_spinlocks_init(void) #include static int steal_spins_set(void *data, u64 val) { +#if _Q_SPIN_TRY_LOCK_STEAL == 1 + /* MAYBE_STEAL remains true */ + STEAL_SPINS = val; +#else static DEFINE_MUTEX(lock); mutex_lock(&lock); @@ -522,6 +530,7 @@ static int steal_spins_set(void *data, u64 val) STEAL_SPINS = val; } mutex_unlock(&lock); +#endif return 0; } From patchwork Mon Jul 11 03:04:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1654628 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=RH+f7myL; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4Lh89Z74MTz9ryY for ; Mon, 11 Jul 2022 13:13:54 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Lh89Z2PJxz3dqX for ; Mon, 11 Jul 2022 13:13:54 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=RH+f7myL; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::529; helo=mail-pg1-x529.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=RH+f7myL; dkim-atps=neutral Received: from mail-pg1-x529.google.com (mail-pg1-x529.google.com [IPv6:2607:f8b0:4864:20::529]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Lh7zx5jk3z3cLZ for ; Mon, 11 Jul 2022 13:05:33 +1000 (AEST) Received: by mail-pg1-x529.google.com with SMTP id 23so3628455pgc.8 for ; Sun, 10 Jul 2022 20:05:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=keeY9AKTFr1mGjJUi/T0jROWikDzUuoWADIBOKMQN+8=; b=RH+f7myLkUmcTfCHwvyS8MlE4kkBI8E4cRVQYcfbAmSkEFUwFxjzPdPgAXsJ4NOTtn lt2rv9/00CLbSjJ4bOjAWAUJYDKFdKe19MTNXwl59nLnYtjvt4l/dXclmYkZLTQgE5Ps od8aZTH2L8CF+o9Y9rtmTwg+Lpd8iYNmQ/5zfVHTHZ87V0gh2Nj8i9SSMCLBaeH1wKWg 1LtTHN8oF2c8PfWK2K3rkOHp8SEbtlhvjTgWr601SpeKZs+9pPYVHIDtezcJksCeZxEu MyEXhh+Mt//4gXjJx6NcjMa8Nu/SG+ne5JDxPOKGnPL7xs+Os0lQTYj0R6FdKMvm7vdT kliA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=keeY9AKTFr1mGjJUi/T0jROWikDzUuoWADIBOKMQN+8=; b=KaIH7HKs1e3vu7i76kG1y6qpkjP6R4YtU3zlNxrEDF6pwp437qCRwdBb/AY9Z0tJnD iM2fO8siDkwxUpqv9g9ZpLh8vEjECfLLm3OR6nWNn3JHac5Zs1c87iguLDowmao8JidW TDbrYZlmvCA2qxVsTvB9McSANtjG2khlIWEPJg78NKhAzvsxfLa5SczaCeU+WQdnvN+N zfgc0nioTdI8gYm8VMogSbaS5StiFlh0JLctg5RKUIMrVebpJrF03CAz7BqiNLcVmeML OdzdCyxfvxeHEMJog8r3biggEIOJSlKPK+/x3lR4mEjomrZBzYInwh8HHMuc4/nrZn4z JG/w== X-Gm-Message-State: AJIora98hTaDoriqJtwvCTn2e1FvBLPAPbhX4qBgFaNCpc2OFbqyXm0d N5iLPp8iSqQTAxHG5XCgJnqZdmcWpO4= X-Google-Smtp-Source: AGRyM1tZ/aBrm3V6j7hMvE5/SaNqfCOx+mvEGPXgc1H0GYkPW4eDJxKj5bElcNxUzSKe7te+NTVBQQ== X-Received: by 2002:a63:1921:0:b0:412:407f:f012 with SMTP id z33-20020a631921000000b00412407ff012mr13642964pgl.125.1657508731227; Sun, 10 Jul 2022 20:05:31 -0700 (PDT) Received: from bobo.ibm.com ([203.220.77.143]) by smtp.gmail.com with ESMTPSA id o4-20020a17090a5b0400b001ef8c81959dsm3562232pji.16.2022.07.10.20.05.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 10 Jul 2022 20:05:30 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 14/14] powerpc/qspinlock: use spin_begin/end API Date: Mon, 11 Jul 2022 13:04:53 +1000 Message-Id: <20220711030453.150644-15-npiggin@gmail.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20220711030453.150644-1-npiggin@gmail.com> References: <20220711030453.150644-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Use the spin_begin/spin_cpu_relax/spin_end APIs in qspinlock, which helps to prevent threads issuing a lot of expensive priority nops which may not have much effect due to immediately executing low then medium priority. --- arch/powerpc/lib/qspinlock.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/arch/powerpc/lib/qspinlock.c b/arch/powerpc/lib/qspinlock.c index d67b923e4f98..486423d566d3 100644 --- a/arch/powerpc/lib/qspinlock.c +++ b/arch/powerpc/lib/qspinlock.c @@ -243,7 +243,7 @@ static void __yield_to_locked_owner(struct qspinlock *lock, u32 val, bool paravi return; } relax: - cpu_relax(); + spin_cpu_relax(); } static void yield_to_locked_owner(struct qspinlock *lock, u32 val, bool paravirt) @@ -331,7 +331,7 @@ static void yield_to_prev(struct qspinlock *lock, struct qnode *node, int prev_c } relax: - cpu_relax(); + spin_cpu_relax(); } @@ -340,6 +340,7 @@ static __always_inline bool try_to_steal_lock(struct qspinlock *lock, bool parav int iters; /* Attempt to steal the lock */ + spin_begin(); for (iters = 0; iters < STEAL_SPINS; iters++) { u32 val = READ_ONCE(lock->val); @@ -354,6 +355,7 @@ static __always_inline bool try_to_steal_lock(struct qspinlock *lock, bool parav if (trylock_with_tail_cpu(lock, val)) return true; } + spin_end(); return false; } @@ -404,8 +406,10 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b WRITE_ONCE(prev->next, node); /* Wait for mcs node lock to be released */ + spin_begin(); while (!node->locked) yield_to_prev(lock, node, prev_cpu, paravirt); + spin_end(); /* Clear out stale propagated yield_cpu */ if (paravirt && pv_yield_propagate_owner && node->yield_cpu != -1) @@ -418,10 +422,12 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b int set_yield_cpu = -1; /* We're at the head of the waitqueue, wait for the lock. */ + spin_begin(); while ((val = READ_ONCE(lock->val)) & _Q_LOCKED_VAL) { propagate_yield_cpu(node, val, &set_yield_cpu, paravirt); yield_head_to_locked_owner(lock, val, paravirt, false); } + spin_end(); /* If we're the last queued, must clean up the tail. */ if ((val & _Q_TAIL_CPU_MASK) == tail) { @@ -436,6 +442,7 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b int set_yield_cpu = -1; int iters = 0; again: + spin_begin(); /* We're at the head of the waitqueue, wait for the lock. */ while ((val = READ_ONCE(lock->val)) & _Q_LOCKED_VAL) { if (iters++ == HEAD_SPINS) { @@ -446,6 +453,7 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b yield_head_to_locked_owner(lock, val, paravirt, pv_yield_allow_steal && (iters > HEAD_SPINS)); } + spin_end(); /* If we're the last queued, must clean up the tail. */ if ((val & _Q_TAIL_CPU_MASK) == tail) { @@ -461,8 +469,13 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b unlock_next: /* contended path; must wait for next != NULL (MCS protocol) */ - while (!(next = READ_ONCE(node->next))) - cpu_relax(); + next = READ_ONCE(node->next); + if (!next) { + spin_begin(); + while (!(next = READ_ONCE(node->next))) + cpu_relax(); + spin_end(); + } /* * Unlock the next mcs waiter node. Release barrier is not required