From patchwork Mon Nov 14 16:11:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1703656 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=GUx7Fxoh; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4N9vV30n80z23lt for ; Tue, 15 Nov 2022 03:12:43 +1100 (AEDT) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4N9vV26lJWz3cjY for ; Tue, 15 Nov 2022 03:12:42 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=GUx7Fxoh; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::1034; helo=mail-pj1-x1034.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=GUx7Fxoh; dkim-atps=neutral Received: from mail-pj1-x1034.google.com (mail-pj1-x1034.google.com [IPv6:2607:f8b0:4864:20::1034]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4N9vSn2qbCz3cJF for ; Tue, 15 Nov 2022 03:11:37 +1100 (AEDT) Received: by mail-pj1-x1034.google.com with SMTP id v4-20020a17090a088400b00212cb0ed97eso11087003pjc.5 for ; Mon, 14 Nov 2022 08:11:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tYVe9VYwgB4+meWyJBvYBlpNZu3XcDPYt6dNK5s35qE=; b=GUx7Fxoh+2cDZBTWeSpfzyWogR0GeDTRRVL5fvFGY0v37gpP6Z8wiNa0MEfcpwU8Tw pw76BmQOwZ1uBy90dKdwQmnotaj73hOLuasmlyjvO/oTDDExrv7cPSat7byuq+OpBRDS wTBAC+4Y1MJ4S2nywd+p7ShtvnqJxpreuHbJMNQZj69Veq3PLnx4FOdxH8rEIad201ku asI75t56fYwtHToNofCkukqKGRaGo49SJVhdTA6j2IJFbB7TJiFHp+rN16dKKZwTw3Qj EMo6Vgk1NeJJyVorndh1DmZpJ1ivFhRZlimDU9yvJC0jIflajqCJwRAz3mrGQQLh7EIr aekQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tYVe9VYwgB4+meWyJBvYBlpNZu3XcDPYt6dNK5s35qE=; b=Svdo3L73w0SLQGBoRraSK8A8jYOmrwDcyXOp02FGJgWR8uwZ44bfz+uvbK24p8ByyR cOlCi6ZeBRTgEnlB40qXobq5wOde3vO1K2m2RUy/PNjTWRaaUZdJD4EBtysUeqbablQx w476Q32+mq22cxUY+hRdDmLh5ensr2PSgp3YZ20AOJ+nvXY5xoINM4gnrum9jevvvs7x tk59/BBljZT63N+0S7rH8Fdwd2EYtiSSp5cE8OV5LVqAZOi2WMjL8i/YFtafRwuCImzF 71u/2AfrrVYVYtCqHlNWsnqnPrzmw4bbz1RyVlbgpffNMcs1DzDIrArUOeJOFdh8oc4C Cixg== X-Gm-Message-State: ANoB5pmhikSdLN//O0MQD1Si1Q2vtRwXn8ydSJW+EnISmiIBouNHZimV V/l58xFWiftblY9J27llX5WerJmN75hwtw== X-Google-Smtp-Source: AA0mqf6hbEZPA9K7mUHkmTnvUBXG+phz6pYXnX15wbPlQSuOD7FHnVS4jyrXZK2bSRAg5t0h2qMyfQ== X-Received: by 2002:a17:902:7895:b0:182:fce5:fde0 with SMTP id q21-20020a170902789500b00182fce5fde0mr14551551pll.132.1668442292892; Mon, 14 Nov 2022 08:11:32 -0800 (PST) Received: from bobo.ozlabs.ibm.com (27-32-22-7.tpgi.com.au. [27.32.22.7]) by smtp.gmail.com with ESMTPSA id e2-20020a170902d38200b00186e2b3e12fsm7635883pld.261.2022.11.14.08.11.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Nov 2022 08:11:32 -0800 (PST) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 1/4] powerpc/qspinlock: Avoid cmpxchg pattern in lock stealing Date: Tue, 15 Nov 2022 02:11:16 +1000 Message-Id: <20221114161119.2883620-2-npiggin@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20221114161119.2883620-1-npiggin@gmail.com> References: <20221114161119.2883620-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jordan Niethe , Laurent Dufour , Nicholas Piggin Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Using a cmpxchg-style trylock in the lock stealing code opens the possibility for failures due to the lock word changing, even if it could have been stolen. Instead, use the stealing trylock which will do the right thing and succeed unless the lwarx finds it locked or mustq. Signed-off-by: Nicholas Piggin --- arch/powerpc/lib/qspinlock.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/powerpc/lib/qspinlock.c b/arch/powerpc/lib/qspinlock.c index 36afdfde41aa..ff718f27cbc9 100644 --- a/arch/powerpc/lib/qspinlock.c +++ b/arch/powerpc/lib/qspinlock.c @@ -497,7 +497,7 @@ static __always_inline bool try_to_steal_lock(struct qspinlock *lock, bool parav if (unlikely(!(val & _Q_LOCKED_VAL))) { spin_end(); - if (trylock_with_tail_cpu(lock, val)) + if (__queued_spin_trylock_steal(lock)) return true; spin_begin(); } else { From patchwork Mon Nov 14 16:11:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1703657 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=Wu+hoYCT; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4N9vW36xkLz23lt for ; Tue, 15 Nov 2022 03:13:35 +1100 (AEDT) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4N9vW34hjqz3f43 for ; Tue, 15 Nov 2022 03:13:35 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=Wu+hoYCT; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::42c; helo=mail-pf1-x42c.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=Wu+hoYCT; dkim-atps=neutral Received: from mail-pf1-x42c.google.com (mail-pf1-x42c.google.com [IPv6:2607:f8b0:4864:20::42c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4N9vSq3YDbz3cKq for ; Tue, 15 Nov 2022 03:11:39 +1100 (AEDT) Received: by mail-pf1-x42c.google.com with SMTP id b185so11410465pfb.9 for ; Mon, 14 Nov 2022 08:11:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mZoC8j/PDFEvbBag+O0cH2CpPqg5IP+JIm/6F1DBzfo=; b=Wu+hoYCT48OktT5yGr4oHazRXw9QkrOgEII0Eg8/aVYCtJMCHxSFQbI6rBZqfP7Pnq Q0Y1rzlhLr+6rvKevnoDxD741vtu3jwGm9IZUOLlPbUyNSMc5wd7pxUi8ID6yE/h4NDE /AMc+hj58WQV/eyW+Vp/3R0zXk0l7KaJCdjR+kvDT4jWvkip4k49kFoCgLacm0U2260P n3rZUFy5mKOm6UF0Yqn3nrQ9RmouTj6YVR4Q1wyFRicxfhLcjZcou6YYYacW0ASYngcE lAeTu7HsH9B8Xsj46wrKlDehnHTslzobMXMZvR9iKiE14ofbzn+3KIrd3i011sUtduL9 8NYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mZoC8j/PDFEvbBag+O0cH2CpPqg5IP+JIm/6F1DBzfo=; b=epYj8FytV0FengnjCsGbIm3d3E2C0mEwZjsrFHeJXnwulLWUzrgvPg3cp3Sk0timIZ Cy132+Usad41g3uBjIGMePY44MWpAeTGdyWU48weJgSMM8evkjE7PuftIXsu/LjSryRK YUqzeuGqfAK6dpd0zKWFQxWbz/lP51YUpnLXXDUaLA6PjeQE1xsdnjfpsZpfgQEcLmu1 LGF7U5H1ChX5FzvAOjvv+d28ATVZa0/VaqemnvoB1qdAbsmZmo0JH2KiD0YStXEtF1Hs CSong9gmwHuCLozp0Z5WOggad8GXy4KovjzW+6d27IqCcHnsHMqKXyuraNSThlyvEG5E YI5A== X-Gm-Message-State: ANoB5plDWiPG6VfussiWSq2V0RqGBvCJLqJ7aRwqyoF4WDNG1yeQ/ygS h6j856khnUpzvKPGb4k2pYhqba4L0PRNLw== X-Google-Smtp-Source: AA0mqf77Dhnqpf3IlgMJkfKHpwalMNU4yyJdMX+CdFzH34xCgKJGl5FcvpXZbQaVa+zY3dj4QqpYew== X-Received: by 2002:a05:6a00:3022:b0:560:e4d1:8df5 with SMTP id ay34-20020a056a00302200b00560e4d18df5mr14753405pfb.39.1668442296803; Mon, 14 Nov 2022 08:11:36 -0800 (PST) Received: from bobo.ozlabs.ibm.com (27-32-22-7.tpgi.com.au. [27.32.22.7]) by smtp.gmail.com with ESMTPSA id e2-20020a170902d38200b00186e2b3e12fsm7635883pld.261.2022.11.14.08.11.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Nov 2022 08:11:35 -0800 (PST) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 2/4] powerpc/qspinlock: Avoid cmpxchg style patterns in queue head locking Date: Tue, 15 Nov 2022 02:11:17 +1000 Message-Id: <20221114161119.2883620-3-npiggin@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20221114161119.2883620-1-npiggin@gmail.com> References: <20221114161119.2883620-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jordan Niethe , Laurent Dufour , Nicholas Piggin Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Locking by the MCS queue head must clear the tail CPU if there are no more queue entries left, and it has to deal with concurrent lock stealing. Implementing these with cmpxchg style updates leaves the possibility for unnecessary failure when the lock word changes. Implement this instead within one larx/stcx. critical section that tests the value and takes the appropriate action: bail out if it was locked, otherwise lock and clear the tail if we are the tail, else lock and leave the tail. With this primitive, there is no longer a significant reason to keep the large !maybe_stealers special case, so remove it. Signed-off-by: Nicholas Piggin --- arch/powerpc/lib/qspinlock.c | 92 ++++++++++++------------------------ 1 file changed, 29 insertions(+), 63 deletions(-) diff --git a/arch/powerpc/lib/qspinlock.c b/arch/powerpc/lib/qspinlock.c index ff718f27cbc9..79793b3209ea 100644 --- a/arch/powerpc/lib/qspinlock.c +++ b/arch/powerpc/lib/qspinlock.c @@ -104,62 +104,36 @@ static inline int get_owner_cpu(u32 val) return (val & _Q_OWNER_CPU_MASK) >> _Q_OWNER_CPU_OFFSET; } -/* Take the lock by setting the lock bit, no other CPUs will touch it. */ -static __always_inline void set_locked(struct qspinlock *lock) +static __always_inline u32 trylock_clear_my_tail(struct qspinlock *lock, u32 mytail) { - u32 new = queued_spin_encode_locked_val(); + u32 newval = queued_spin_encode_locked_val(); u32 prev, tmp; asm volatile( -"1: lwarx %0,0,%2,%4 # set_locked \n" -" or %1,%0,%3 \n" -" stwcx. %1,0,%2 \n" +"1: lwarx %0,0,%2,%7 # trylock_clear_my_tail \n" + /* This test is necessary if there could be stealers */ +" andi. %1,%0,%5 \n" +" bne 3f \n" + /* Test whether the lock tail == mytail */ +" and %1,%0,%6 \n" +" cmpw 0,%1,%3 \n" + /* Merge the new locked value */ +" or %1,%1,%4 \n" +" bne 2f \n" + /* If the lock tail matched, then clear it, otherwise leave it. */ +" andc %1,%1,%6 \n" +"2: stwcx. %1,0,%2 \n" " bne- 1b \n" "\t" PPC_ACQUIRE_BARRIER " \n" +"3: \n" : "=&r" (prev), "=&r" (tmp) - : "r" (&lock->val), "r" (new), + : "r" (&lock->val), "r"(mytail), "r" (newval), + "i" (_Q_LOCKED_VAL), + "r" (_Q_TAIL_CPU_MASK), "i" (IS_ENABLED(CONFIG_PPC64)) : "cr0", "memory"); - BUG_ON(prev & _Q_LOCKED_VAL); -} - -static __always_inline u32 __trylock_cmpxchg(struct qspinlock *lock, u32 old, u32 new) -{ - u32 prev; - - BUG_ON(old & _Q_LOCKED_VAL); - - asm volatile( -"1: lwarx %0,0,%1,%4 # __trylock_cmpxchg \n" -" cmpw 0,%0,%2 \n" -" bne- 2f \n" -" stwcx. %3,0,%1 \n" -" bne- 1b \n" -"\t" PPC_ACQUIRE_BARRIER " \n" -"2: \n" - : "=&r" (prev) - : "r" (&lock->val), "r"(old), "r" (new), - "i" (IS_ENABLED(CONFIG_PPC64)) - : "cr0", "memory"); - - return likely(prev == old); -} - -/* Take lock, clearing tail, cmpxchg with old (which must not be locked) */ -static __always_inline int trylock_clear_tail_cpu(struct qspinlock *lock, u32 val) -{ - u32 newval = queued_spin_encode_locked_val(); - - return __trylock_cmpxchg(lock, val, newval); -} - -/* Take lock, preserving tail, cmpxchg with val (which must not be locked) */ -static __always_inline int trylock_with_tail_cpu(struct qspinlock *lock, u32 val) -{ - u32 newval = queued_spin_encode_locked_val() | (val & _Q_TAIL_CPU_MASK); - - return __trylock_cmpxchg(lock, val, newval); + return prev; } /* @@ -620,14 +594,11 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b spin_end(); /* If we're the last queued, must clean up the tail. */ - if ((val & _Q_TAIL_CPU_MASK) == tail) { - if (trylock_clear_tail_cpu(lock, val)) - goto release; - /* Another waiter must have enqueued. */ - } + old = trylock_clear_my_tail(lock, tail); + BUG_ON(old & _Q_LOCKED_VAL); + if ((old & _Q_TAIL_CPU_MASK) == tail) + goto release; - /* We must be the owner, just set the lock bit and acquire */ - set_locked(lock); } else { int set_yield_cpu = -1; int iters = 0; @@ -682,18 +653,13 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b spin_end(); /* If we're the last queued, must clean up the tail. */ - if ((val & _Q_TAIL_CPU_MASK) == tail) { - if (trylock_clear_tail_cpu(lock, val)) - goto release; - /* Another waiter must have enqueued, or lock stolen. */ - } else { - if (trylock_with_tail_cpu(lock, val)) - goto unlock_next; - } - goto again; + old = trylock_clear_my_tail(lock, tail); + if (unlikely(old & _Q_LOCKED_VAL)) + goto again; + if ((old & _Q_TAIL_CPU_MASK) == tail) + goto release; } -unlock_next: /* contended path; must wait for next != NULL (MCS protocol) */ next = READ_ONCE(node->next); if (!next) { From patchwork Mon Nov 14 16:11:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1703658 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=QjH6cidG; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4N9vX42mVkz23lt for ; Tue, 15 Nov 2022 03:14:28 +1100 (AEDT) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4N9vX41Z1Lz3cNb for ; Tue, 15 Nov 2022 03:14:28 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=QjH6cidG; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::62b; helo=mail-pl1-x62b.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=QjH6cidG; dkim-atps=neutral Received: from mail-pl1-x62b.google.com (mail-pl1-x62b.google.com [IPv6:2607:f8b0:4864:20::62b]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4N9vSt3r8Hz3cHm for ; Tue, 15 Nov 2022 03:11:42 +1100 (AEDT) Received: by mail-pl1-x62b.google.com with SMTP id c2so10447994plz.11 for ; Mon, 14 Nov 2022 08:11:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FNdYY8itr/zxiwuI2RKozXsUxE2hCXY6c7m7M08B9gs=; b=QjH6cidGuZnPVECFv4B1tE28r/0NNTOiDzRwfh9E+OM10G/5x6r7UxVasZdNzxGBLf MND7mfxhxWYHB5K0vT5tC+VY8CP9LVXwpwFjjWHl1CYjPZEpGwaC0dckKHzaAUCQgdE3 U9r+u2nDDsY+Ml6B69jfN28SrlZCQSn7A9E4OWrjqCUUEg0IugTBj4hpnShGYextwr35 uPFWyuVb1eiINuOI2bGJv1DX4Xl6vm+cFzwYvLTH8abHlbmfbngqqKgbkGKh0bmG5x1R 0ujhQvYYJp3XCPpeAvHwzzlDN3JiX0CHW2SceWIAaY2g4KfVLBo9EP8BQPWuZplulOhp QLmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FNdYY8itr/zxiwuI2RKozXsUxE2hCXY6c7m7M08B9gs=; b=i532LOY90N19fn6cUFiUEI/j59VHIrUtaY5OPAFbOXjgKt9h/jYcf43h6VkbmbZIP6 0aFP+BLqnD7m6Q7KV9JO5WxITnZl9Bz43BbPcjeTYrQPa3KO9y7VSt9JprRzXeEC+hCw 5YVvnhjNE0rUzLzK0TYJ5Ma5xnyimwEVBGysi47jJkWYuCnc75nGq5N9FKGKyp2YtKyZ 1ezGeglwPdYbQOcXT2KHbwlfic8zVO4ug+BTPVCbyV8smxFO3WyW/HdIlGE95I/hAryg 0T2lgR989ucnDHCroaBlg0Fu8bYc+CtEnqsxDtmvcLs8TVdURtCilbg3cxxMAb477ygS dJmg== X-Gm-Message-State: ANoB5pmQcCmCDDXfRXOsHYTr5nbz1/JsUu6VOvZDid8DZTgHvVgjyzss JRaY2VlKWiSExYCqclWEMeCIVrIck9JE4Q== X-Google-Smtp-Source: AA0mqf6JI2FOgrBCHfUFqHUrBl3yfkANrzn+Z7tCxYwvns+GN2+KLPx0KBEhuAuZYWbIM/De2OKIJw== X-Received: by 2002:a17:902:e890:b0:186:a22a:177e with SMTP id w16-20020a170902e89000b00186a22a177emr14149002plg.163.1668442299892; Mon, 14 Nov 2022 08:11:39 -0800 (PST) Received: from bobo.ozlabs.ibm.com (27-32-22-7.tpgi.com.au. [27.32.22.7]) by smtp.gmail.com with ESMTPSA id e2-20020a170902d38200b00186e2b3e12fsm7635883pld.261.2022.11.14.08.11.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Nov 2022 08:11:39 -0800 (PST) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 3/4] powerpc/qspinlock: Remove !maybe_waiters special case queue head locking Date: Tue, 15 Nov 2022 02:11:18 +1000 Message-Id: <20221114161119.2883620-4-npiggin@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20221114161119.2883620-1-npiggin@gmail.com> References: <20221114161119.2883620-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jordan Niethe , Laurent Dufour , Nicholas Piggin Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" With the update primitive that clears the tail if it matches, and is tolerant of other queueing activity on the lock, there is no longer a significant reason to keep the large !maybe_stealers special case, so remove it. Signed-off-by: Nicholas Piggin --- arch/powerpc/lib/qspinlock.c | 124 +++++++++++++++-------------------- 1 file changed, 53 insertions(+), 71 deletions(-) diff --git a/arch/powerpc/lib/qspinlock.c b/arch/powerpc/lib/qspinlock.c index 79793b3209ea..457e748b0078 100644 --- a/arch/powerpc/lib/qspinlock.c +++ b/arch/powerpc/lib/qspinlock.c @@ -523,7 +523,11 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b struct qnode *next, *node; u32 val, old, tail; bool seen_preempted = false; + bool sleepy = false; + bool mustq = false; int idx; + int set_yield_cpu = -1; + int iters = 0; BUILD_BUG_ON(CONFIG_NR_CPUS >= (1U << _Q_TAIL_CPU_BITS)); @@ -577,90 +581,68 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b smp_rmb(); /* acquire barrier for the mcs lock */ } - if (!maybe_stealers) { - int set_yield_cpu = -1; - - /* We're at the head of the waitqueue, wait for the lock. */ - spin_begin(); - for (;;) { - val = READ_ONCE(lock->val); - if (!(val & _Q_LOCKED_VAL)) - break; - - propagate_yield_cpu(node, val, &set_yield_cpu, paravirt); - if (yield_head_to_locked_owner(lock, val, paravirt)) - seen_preempted = true; - } - spin_end(); - - /* If we're the last queued, must clean up the tail. */ - old = trylock_clear_my_tail(lock, tail); - BUG_ON(old & _Q_LOCKED_VAL); - if ((old & _Q_TAIL_CPU_MASK) == tail) - goto release; - - } else { - int set_yield_cpu = -1; - int iters = 0; - bool sleepy = false; - bool mustq = false; + /* We're at the head of the waitqueue, wait for the lock. */ +again: + spin_begin(); + for (;;) { bool preempted; -again: - /* We're at the head of the waitqueue, wait for the lock. */ - spin_begin(); - for (;;) { - val = READ_ONCE(lock->val); - if (!(val & _Q_LOCKED_VAL)) - break; - - if (paravirt && pv_sleepy_lock) { - if (!sleepy) { - if (val & _Q_SLEEPY_VAL) { - seen_sleepy_lock(); - sleepy = true; - } else if (recently_sleepy()) { - sleepy = true; - } - } - if (pv_sleepy_lock_sticky && seen_preempted && - !(val & _Q_SLEEPY_VAL)) { - if (try_set_sleepy(lock, val)) - val |= _Q_SLEEPY_VAL; + val = READ_ONCE(lock->val); + if (!(val & _Q_LOCKED_VAL)) + break; + + if (paravirt && pv_sleepy_lock && maybe_stealers) { + if (!sleepy) { + if (val & _Q_SLEEPY_VAL) { + seen_sleepy_lock(); + sleepy = true; + } else if (recently_sleepy()) { + sleepy = true; } } + if (pv_sleepy_lock_sticky && seen_preempted && + !(val & _Q_SLEEPY_VAL)) { + if (try_set_sleepy(lock, val)) + val |= _Q_SLEEPY_VAL; + } + } - propagate_yield_cpu(node, val, &set_yield_cpu, paravirt); - preempted = yield_head_to_locked_owner(lock, val, paravirt); - if (preempted) - seen_preempted = true; + propagate_yield_cpu(node, val, &set_yield_cpu, paravirt); + preempted = yield_head_to_locked_owner(lock, val, paravirt); + if (!maybe_stealers) + continue; + + if (preempted) + seen_preempted = true; - if (paravirt && preempted) { - sleepy = true; + if (paravirt && preempted) { + sleepy = true; - if (!pv_spin_on_preempted_owner) - iters++; - } else { + if (!pv_spin_on_preempted_owner) iters++; - } + } else { + iters++; + } - if (!mustq && iters >= get_head_spins(paravirt, sleepy)) { - mustq = true; - set_mustq(lock); - val |= _Q_MUST_Q_VAL; - } + if (!mustq && iters >= get_head_spins(paravirt, sleepy)) { + mustq = true; + set_mustq(lock); + val |= _Q_MUST_Q_VAL; } - spin_end(); + } + spin_end(); - /* If we're the last queued, must clean up the tail. */ - old = trylock_clear_my_tail(lock, tail); - if (unlikely(old & _Q_LOCKED_VAL)) - goto again; - if ((old & _Q_TAIL_CPU_MASK) == tail) - goto release; + /* If we're the last queued, must clean up the tail. */ + old = trylock_clear_my_tail(lock, tail); + if (unlikely(old & _Q_LOCKED_VAL)) { + BUG_ON(!maybe_stealers); + goto again; /* Can only be true if maybe_stealers. */ } - /* contended path; must wait for next != NULL (MCS protocol) */ + if ((old & _Q_TAIL_CPU_MASK) == tail) + goto release; /* We were the tail, no next. */ + + /* There is a next, must wait for node->next != NULL (MCS protocol) */ next = READ_ONCE(node->next); if (!next) { spin_begin(); From patchwork Mon Nov 14 16:11:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1703659 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=PwkTK6Vy; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4N9vYD4RFJz1yqS for ; Tue, 15 Nov 2022 03:15:28 +1100 (AEDT) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4N9vYD1X9hz3dvY for ; Tue, 15 Nov 2022 03:15:28 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=PwkTK6Vy; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::42c; helo=mail-pf1-x42c.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=PwkTK6Vy; dkim-atps=neutral Received: from mail-pf1-x42c.google.com (mail-pf1-x42c.google.com [IPv6:2607:f8b0:4864:20::42c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4N9vSx66J5z3cLC for ; Tue, 15 Nov 2022 03:11:45 +1100 (AEDT) Received: by mail-pf1-x42c.google.com with SMTP id v28so11412343pfi.12 for ; Mon, 14 Nov 2022 08:11:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RZGBrKdRsGyUTA5xkITQ1xeRBZZEgNSXBjqGlOM+4LY=; b=PwkTK6Vya2lwSyoOTTazwZQUeaSSIweVLQXNsRbFrQrX9bXDuYy+Mb6UWL2WM1+4qI Q7f5FqDzYcW6kQAcSgBOjwanAIaeY6tePBFTFfklnoqi3Wg7xcCk4XKL4BNzvMsFK8e8 NlEnl2Mrpwu3OKRoDynry9L++tUURLVnDkjpjZQnq13erKWH9n30EDAn3P1B4DusKhQt k8D9OJK9Y8VCD+ntWPh1RAU986v1IFlNyJXo+a0Nn2hrTH/bXfX6Dlfd0DuvKGF4bdrf 7Gmtjchdyk2IJ0mWaWyaBDoAteeN4s/ssENzrcAv3VEWeic89MBrwktIQXcFfNRlziyH tNqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RZGBrKdRsGyUTA5xkITQ1xeRBZZEgNSXBjqGlOM+4LY=; b=GsquRaymMu37RoQtG+bCoplcK4eH0+zycFqFoA1bvr5ij8hRGPaKfJMhQJxWntUobF wCMMwEwm0zNjMkKORZZclh8ukgItD/9hUg1ubrIFotxGnz5HRFU0pGFkYa8WYvy+DS9V RGMMYENTabWmOAYi4dvsQzRBvT3dnQ9dUIrv5/GzFadjz1OgJ7iFolRFBMcromGQg//z OxJZJlj9L7hxiU/hrCd8hY9gA7ZyWkthFVoTOZyIvRAvyZKDwpbvf+xf202h5LbPaq4E NZr1lpfPXhA/Otu3KbLS2PfHIDT6OzLlzvk6dDG0poRVjW19RKBE+H05fq247QfG52kO xwJA== X-Gm-Message-State: ANoB5pmLoPRTgtaGCt677Hxz7votKxB4U+0tb5wVVlvyYy30Bv+FF3Mr yrj/Hdc3cyAp6ILw0FtoKpVctnFbk3Dfkw== X-Google-Smtp-Source: AA0mqf5YBsBn4azGG8nxqCGy3rejIBWhVoEMbfA6ib5RAwwgElgqg+neDoKBSCDo0uUN8x7NfIK1Ag== X-Received: by 2002:a63:560e:0:b0:476:837c:e0a with SMTP id k14-20020a63560e000000b00476837c0e0amr4652585pgb.411.1668442303379; Mon, 14 Nov 2022 08:11:43 -0800 (PST) Received: from bobo.ozlabs.ibm.com (27-32-22-7.tpgi.com.au. [27.32.22.7]) by smtp.gmail.com with ESMTPSA id e2-20020a170902d38200b00186e2b3e12fsm7635883pld.261.2022.11.14.08.11.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Nov 2022 08:11:42 -0800 (PST) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [RFC PATCH 4/4] powerpc/qspinlock: add compile-time tuning adjustments Date: Tue, 15 Nov 2022 02:11:19 +1000 Message-Id: <20221114161119.2883620-5-npiggin@gmail.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20221114161119.2883620-1-npiggin@gmail.com> References: <20221114161119.2883620-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jordan Niethe , Laurent Dufour , Nicholas Piggin Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" This adds compile-time options that allow the EH lock hint bit to be enabled or disabled, and adds some new options that may or may not help matters. To help with experimentation and tuning. Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/qspinlock.h | 61 ++++++++++++++++++++++++++-- arch/powerpc/lib/qspinlock.c | 39 ++++++++++++++++-- 2 files changed, 94 insertions(+), 6 deletions(-) diff --git a/arch/powerpc/include/asm/qspinlock.h b/arch/powerpc/include/asm/qspinlock.h index 56638175e49b..5efd66aeb907 100644 --- a/arch/powerpc/include/asm/qspinlock.h +++ b/arch/powerpc/include/asm/qspinlock.h @@ -5,15 +5,68 @@ #include #include +#ifdef CONFIG_PPC64 +/* + * Use the EH=1 hint for accesses that result in the lock being acquired. + * The hardware is supposed to optimise this pattern by holding the lock + * cacheline longer, and releasing when a store to the same memory (the + * unlock) is performed. + */ +#define _Q_SPIN_EH_HINT 1 +#else +#define _Q_SPIN_EH_HINT 0 +#endif + /* * The trylock itself may steal. This makes trylocks slightly stronger, and - * might make spin locks slightly more efficient when stealing. + * makes locks slightly more efficient when stealing. * * This is compile-time, so if true then there may always be stealers, so the * nosteal paths become unused. */ #define _Q_SPIN_TRY_LOCK_STEAL 1 +/* + * Put a speculation barrier after testing the lock/node and finding it + * busy. Try to prevent pointless speculation in slow paths. + * + * Slows down the lockstorm microbenchmark with no stealing, where locking + * is purely FIFO through the queue. May have more benefit in real workload + * where speculating into the wrong place could have a greater cost. + */ +#define _Q_SPIN_SPEC_BARRIER 0 + +#ifdef CONFIG_PPC64 +/* + * Execute a miso instruction after passing the MCS lock ownership to the + * queue head. Miso is intended to make stores visible to other CPUs sooner. + * + * This seems to make the lockstorm microbenchmark nospin test go slightly + * faster on POWER10, but disable for now. + */ +#define _Q_SPIN_MISO 0 +#else +#define _Q_SPIN_MISO 0 +#endif + +#ifdef CONFIG_PPC64 +/* + * This executes miso after an unlock of the lock word, having ownership + * pass to the next CPU sooner. This will slow the uncontended path to some + * degree. Not evidence it helps yet. + */ +#define _Q_SPIN_MISO_UNLOCK 0 +#else +#define _Q_SPIN_MISO_UNLOCK 0 +#endif + +/* + * Seems to slow down lockstorm microbenchmark, suspect queue node just + * has to become shared again right afterwards when its waiter spins on + * the lock field. + */ +#define _Q_SPIN_PREFETCH_NEXT 0 + static __always_inline int queued_spin_is_locked(struct qspinlock *lock) { return READ_ONCE(lock->val); @@ -51,7 +104,7 @@ static __always_inline int __queued_spin_trylock_nosteal(struct qspinlock *lock) "2: \n" : "=&r" (prev) : "r" (&lock->val), "r" (new), - "i" (IS_ENABLED(CONFIG_PPC64)) + "i" (_Q_SPIN_EH_HINT) : "cr0", "memory"); return likely(prev == 0); @@ -75,7 +128,7 @@ static __always_inline int __queued_spin_trylock_steal(struct qspinlock *lock) "2: \n" : "=&r" (prev), "=&r" (tmp) : "r" (&lock->val), "r" (new), "r" (_Q_TAIL_CPU_MASK), - "i" (IS_ENABLED(CONFIG_PPC64)) + "i" (_Q_SPIN_EH_HINT) : "cr0", "memory"); return likely(!(prev & ~_Q_TAIL_CPU_MASK)); @@ -100,6 +153,8 @@ static __always_inline void queued_spin_lock(struct qspinlock *lock) static inline void queued_spin_unlock(struct qspinlock *lock) { smp_store_release(&lock->locked, 0); + if (_Q_SPIN_MISO_UNLOCK) + asm volatile("miso" ::: "memory"); } #define arch_spin_is_locked(l) queued_spin_is_locked(l) diff --git a/arch/powerpc/lib/qspinlock.c b/arch/powerpc/lib/qspinlock.c index 457e748b0078..01fbb4209367 100644 --- a/arch/powerpc/lib/qspinlock.c +++ b/arch/powerpc/lib/qspinlock.c @@ -48,6 +48,12 @@ static bool pv_prod_head __read_mostly = false; static DEFINE_PER_CPU_ALIGNED(struct qnodes, qnodes); static DEFINE_PER_CPU_ALIGNED(u64, sleepy_lock_seen_clock); +#if _Q_SPIN_SPEC_BARRIER == 1 +#define spec_barrier() do { asm volatile("ori 31,31,0" ::: "memory"); } while (0) +#else +#define spec_barrier() do { } while (0) +#endif + static __always_inline bool recently_sleepy(void) { /* pv_sleepy_lock is true when this is called */ @@ -130,7 +136,7 @@ static __always_inline u32 trylock_clear_my_tail(struct qspinlock *lock, u32 myt : "r" (&lock->val), "r"(mytail), "r" (newval), "i" (_Q_LOCKED_VAL), "r" (_Q_TAIL_CPU_MASK), - "i" (IS_ENABLED(CONFIG_PPC64)) + "i" (_Q_SPIN_EH_HINT) : "cr0", "memory"); return prev; @@ -468,6 +474,7 @@ static __always_inline bool try_to_steal_lock(struct qspinlock *lock, bool parav val = READ_ONCE(lock->val); if (val & _Q_MUST_Q_VAL) break; + spec_barrier(); if (unlikely(!(val & _Q_LOCKED_VAL))) { spin_end(); @@ -533,6 +540,7 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b qnodesp = this_cpu_ptr(&qnodes); if (unlikely(qnodesp->count >= MAX_NODES)) { + spec_barrier(); while (!queued_spin_trylock(lock)) cpu_relax(); return; @@ -569,9 +577,12 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b /* Wait for mcs node lock to be released */ spin_begin(); while (!node->locked) { + spec_barrier(); + if (yield_to_prev(lock, node, old, paravirt)) seen_preempted = true; } + spec_barrier(); spin_end(); /* Clear out stale propagated yield_cpu */ @@ -579,6 +590,17 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b node->yield_cpu = -1; smp_rmb(); /* acquire barrier for the mcs lock */ + + /* + * Generic qspinlocks have this prefetch here, but it seems + * like it could cause additional line transitions because + * the waiter will keep loading from it. + */ + if (_Q_SPIN_PREFETCH_NEXT) { + next = READ_ONCE(node->next); + if (next) + prefetchw(next); + } } /* We're at the head of the waitqueue, wait for the lock. */ @@ -590,6 +612,7 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b val = READ_ONCE(lock->val); if (!(val & _Q_LOCKED_VAL)) break; + spec_barrier(); if (paravirt && pv_sleepy_lock && maybe_stealers) { if (!sleepy) { @@ -630,6 +653,7 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b val |= _Q_MUST_Q_VAL; } } + spec_barrier(); spin_end(); /* If we're the last queued, must clean up the tail. */ @@ -650,6 +674,7 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b cpu_relax(); spin_end(); } + spec_barrier(); /* * Unlock the next mcs waiter node. Release barrier is not required @@ -661,10 +686,14 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b if (paravirt && pv_prod_head) { int next_cpu = next->cpu; WRITE_ONCE(next->locked, 1); + if (_Q_SPIN_MISO) + asm volatile("miso" ::: "memory"); if (vcpu_is_preempted(next_cpu)) prod_cpu(next_cpu); } else { WRITE_ONCE(next->locked, 1); + if (_Q_SPIN_MISO) + asm volatile("miso" ::: "memory"); } release: @@ -679,12 +708,16 @@ void queued_spin_lock_slowpath(struct qspinlock *lock) * is passed as the paravirt argument to the functions. */ if (IS_ENABLED(CONFIG_PARAVIRT_SPINLOCKS) && is_shared_processor()) { - if (try_to_steal_lock(lock, true)) + if (try_to_steal_lock(lock, true)) { + spec_barrier(); return; + } queued_spin_lock_mcs_queue(lock, true); } else { - if (try_to_steal_lock(lock, false)) + if (try_to_steal_lock(lock, false)) { + spec_barrier(); return; + } queued_spin_lock_mcs_queue(lock, false); } }