From patchwork Mon Oct 16 12:43:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1849343 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=bpDEZnHH; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4S8Gyg0nnJz20Zj for ; Mon, 16 Oct 2023 23:44:27 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=bpDEZnHH; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4S8Gyf6jllz3dh2 for ; Mon, 16 Oct 2023 23:44:26 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=bpDEZnHH; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::42c; helo=mail-pf1-x42c.google.com; envelope-from=npiggin@gmail.com; receiver=lists.ozlabs.org) Received: from mail-pf1-x42c.google.com (mail-pf1-x42c.google.com [IPv6:2607:f8b0:4864:20::42c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4S8GxL5Lccz30Q4 for ; Mon, 16 Oct 2023 23:43:18 +1100 (AEDT) Received: by mail-pf1-x42c.google.com with SMTP id d2e1a72fcca58-6be0277c05bso684320b3a.0 for ; Mon, 16 Oct 2023 05:43:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697460195; x=1698064995; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FHFNauDcUWKtivz1H+j8KgAD3PFnM59+M+NZETg+vSA=; b=bpDEZnHH9uqfOMGAogcrbqAtoJLKir68q2Mk6Hhiw+6YTKKvAVfUDiYXVjaPPpzrz0 iu/cKgXs9VaEjxe9SGELwLjyVDPewzNMWEqzOInL9TGzS4Y2wfFzo3JdK/AsEuH7dP0u pOPpjdlF6ZzW9X/5WWsGMBKphGbnNC3QsACDKHbZXNnimmZNqvsYcJ8SmtLOGwr0Y2Hv me4Tr22xZy2kTLDYiueuv+B90EzZCyd9/FFGJZFQoWMBP8htltplKj+JtecPLrDGn6tT pZEjDdDM9ehMSA1a/YxkOae4+TELWU3YWhtefGaj5AoLSGvpVL0kCkf0MZjf0odkULTr AzSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697460195; x=1698064995; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FHFNauDcUWKtivz1H+j8KgAD3PFnM59+M+NZETg+vSA=; b=uqhoxnPPCks7NE7Xoa9qBQnThnHK0Idtf7lJlsf7h6OB65Bs0LSSPIALzfYOTW9XpX J7V99vRy9CyAHmUrU42dBSk/pzOZgTaUasoasLBM2h3BP9DrKRuwABslEytsuQQujheg dvkqRzk4ZU1RZC578EbZ8ObBNHy8CNjmdGd5rNuhwaZlA6qeNBSrlrqUFhW4iBtxIM1k vxPwjjjyq4mcOoDqG7nHVCAZiS1GqAQnzdWBjyZeju9ilrz681Y6OHQCgb1q660DZGVm leaE/BDpMVhPVOeQN7+pJRdqyqcNjucby5rNOyb3OBgWJWSVUv37FO0VEo7qV5BTezY5 Fzhw== X-Gm-Message-State: AOJu0Yxlynzl4yhxCeOEk81JXqWyAnbvIUizi+enZIm9UEolDDz8tx6P 2V2knKOEbBFQ3rm7rrXP0CgjCS4SuOQ= X-Google-Smtp-Source: AGHT+IHeRC2fJ0RrClgasakjX/wQYTAjd6pHck3fEjqsV2hNaPJQJNZZV6GLrfPTSBzZ/eCp+fURYQ== X-Received: by 2002:a05:6a00:2d1b:b0:6bd:7cbd:15a2 with SMTP id fa27-20020a056a002d1b00b006bd7cbd15a2mr2707554pfb.26.1697460195529; Mon, 16 Oct 2023 05:43:15 -0700 (PDT) Received: from wheely.local0.net (193-116-200-79.tpgi.com.au. [193.116.200.79]) by smtp.gmail.com with ESMTPSA id t123-20020a625f81000000b0068c10187dc3sm6298850pfb.168.2023.10.16.05.43.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Oct 2023 05:43:15 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH 1/6] powerpc/qspinlock: Fix stale propagated yield_cpu Date: Mon, 16 Oct 2023 22:43:00 +1000 Message-ID: <20231016124305.139923-2-npiggin@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231016124305.139923-1-npiggin@gmail.com> References: <20231016124305.139923-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Shrikanth Hegde , Laurent Dufour , Srikar Dronamraju , Nicholas Piggin , "Nysal Jan K . A" Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" yield_cpu is a sample of a preempted lock holder that gets propagated back through the queue. Queued waiters use this to yield to the preempted lock holder without continually sampling the lock word (which would defeat the purpose of MCS queueing by bouncing the cache line). The problem is that yield_cpu can become stale. It can take some time to be passed down the chain, and if any queued waiter gets preempted then it will cease to propagate the yield_cpu to later waiters. This can result in yielding to a CPU that no longer holds the lock, which is bad, but particularly if it is currently in H_CEDE (idle), then it appears to be preempted and some hypervisors (PowerVM) can cause very long H_CONFER latencies waiting for H_CEDE wakeup. This results in latency spikes and hard lockups on oversubscribed partitions with lock contention. This is a minimal fix. Before yielding to yield_cpu, sample the lock word to confirm yield_cpu is still the owner, and bail out of it is not. Thanks to a bunch of people who reported this and tracked down the exact problem using tracepoints and dispatch trace logs. Fixes: 28db61e207ea ("powerpc/qspinlock: allow propagation of yield CPU down the queue") Reported-by: Srikar Dronamraju Reported-by: Laurent Dufour Reported-by: Shrikanth Hegde Debugged-by: Nysal Jan K.A Signed-off-by: Nicholas Piggin --- arch/powerpc/lib/qspinlock.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/arch/powerpc/lib/qspinlock.c b/arch/powerpc/lib/qspinlock.c index 253620979d0c..6dd2f46bd3ef 100644 --- a/arch/powerpc/lib/qspinlock.c +++ b/arch/powerpc/lib/qspinlock.c @@ -406,6 +406,9 @@ static __always_inline bool yield_to_prev(struct qspinlock *lock, struct qnode * if ((yield_count & 1) == 0) goto yield_prev; /* owner vcpu is running */ + if (get_owner_cpu(READ_ONCE(lock->val)) != yield_cpu) + goto yield_prev; /* re-sample lock owner */ + spin_end(); preempted = true; From patchwork Mon Oct 16 12:43:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1849344 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=Lot0eneY; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4S8Gzc0hScz20Vq for ; Mon, 16 Oct 2023 23:45:16 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=Lot0eneY; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4S8Gzb5ZnPz3vZy for ; Mon, 16 Oct 2023 23:45:15 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=Lot0eneY; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::c2c; helo=mail-oo1-xc2c.google.com; envelope-from=npiggin@gmail.com; receiver=lists.ozlabs.org) Received: from mail-oo1-xc2c.google.com (mail-oo1-xc2c.google.com [IPv6:2607:f8b0:4864:20::c2c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4S8Gyb0zdqz3dDg for ; Mon, 16 Oct 2023 23:44:22 +1100 (AEDT) Received: by mail-oo1-xc2c.google.com with SMTP id 006d021491bc7-57ba5f05395so2760013eaf.1 for ; Mon, 16 Oct 2023 05:44:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697460259; x=1698065059; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VOg8GvzntlYfAxH5Ok/9uHc4ELZsVCGk5wGPn/0WVBo=; b=Lot0eneYmZF8ZUGglYfe/xNcqKn8XQe7TDqe8+sPMMcHIl+UQ+mLlu9J1kUsFlZJVX ZxZSByXbIfuj3dNBejTJ464KHAW6abhLDmdco8ngN4r9pV9oRS8WFmNciaz+ta8XsRPK Wrpm2eG0z11KhvRudRMDIclWA6dupa2RDt32QzmBBfl6cZfEgHOcwYWi97oS9m7sYx4c exOcGrSOSNGurftuD+fccq0s5tlngu0xuOvMgFAwCuCvDm5zEiE2K0Ft7mqnpjde9VQg TKvjxydwhfvTvA3zclg6hg8t/gXWG1yQ7M+FXQZ1x8ekGS/k7sWzZMK6aCseSxDv/SXd JEHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697460259; x=1698065059; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VOg8GvzntlYfAxH5Ok/9uHc4ELZsVCGk5wGPn/0WVBo=; b=rsts6sNFk3258h0jTxPipf3QKj6B12hWpm/sldtB7BXUM1e+96WVGUcHObdD+X/On6 BBysBV4bJ4w8gRBooGMlEu8zVCANYyKF3uuQPDWDcHVY5wUnp+54p5yEwOM8V4HiYG/l /y7NYGQJQYicaO9j3WnR6TDzhWCtYhDnRwsihz7i9glFtQfYzzCo4r0ZehN8XiOB4la8 rKg44v5VPLojrJCNkTMegV7KhfBpx9NsaBcpBImTqFeB18l+gnMXuGywyaHjr+P/ODdO 1Eam622uBxdC6XDP4Zisw2ZMUr+pQzLXsAxIWIoLVB319h9rcAhndqseIG/1N5yUs3Pr TLGg== X-Gm-Message-State: AOJu0YxCrBdhAQezxr+nPrVL8ywp9kUB9AdGjAKpmaMQ097fZXD9VLCG v1NTB+XG/9QiR/nyfcME513yMfdvHvc= X-Google-Smtp-Source: AGHT+IHrB8rNshNZHt1Er0FXkXepehBglxF1+oi/OH9JdEvAUp9j748KmwPB9DoxvSP38Y/d4A2GlQ== X-Received: by 2002:a05:6358:7296:b0:164:953b:35a9 with SMTP id w22-20020a056358729600b00164953b35a9mr21927028rwf.23.1697460259352; Mon, 16 Oct 2023 05:44:19 -0700 (PDT) Received: from wheely.local0.net (193-116-200-79.tpgi.com.au. [193.116.200.79]) by smtp.gmail.com with ESMTPSA id t123-20020a625f81000000b0068c10187dc3sm6298850pfb.168.2023.10.16.05.44.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Oct 2023 05:44:19 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH 2/6] powerpc/qspinlock: stop queued waiters trying to set lock sleepy Date: Mon, 16 Oct 2023 22:43:01 +1000 Message-ID: <20231016124305.139923-3-npiggin@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231016124305.139923-1-npiggin@gmail.com> References: <20231016124305.139923-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Shrikanth Hegde , Srikar Dronamraju , Nicholas Piggin , "Nysal Jan K . A" Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" If a queued waiter notices the lock owner or the previous waiter has been preempted, it attempts to mark the lock sleepy, but it does this as a try-set operation using the original lock value it got when queueing, which will become stale as the queue progresses, and the try-set will fail. Drop this and just set the sleepy seen clock. Signed-off-by: Nicholas Piggin --- arch/powerpc/lib/qspinlock.c | 24 ++++++++++-------------- 1 file changed, 10 insertions(+), 14 deletions(-) diff --git a/arch/powerpc/lib/qspinlock.c b/arch/powerpc/lib/qspinlock.c index 6dd2f46bd3ef..75608ced14c2 100644 --- a/arch/powerpc/lib/qspinlock.c +++ b/arch/powerpc/lib/qspinlock.c @@ -247,22 +247,18 @@ static __always_inline void seen_sleepy_lock(void) this_cpu_write(sleepy_lock_seen_clock, sched_clock()); } -static __always_inline void seen_sleepy_node(struct qspinlock *lock, u32 val) +static __always_inline void seen_sleepy_node(void) { if (pv_sleepy_lock) { if (pv_sleepy_lock_interval_ns) this_cpu_write(sleepy_lock_seen_clock, sched_clock()); - if (val & _Q_LOCKED_VAL) { - if (!(val & _Q_SLEEPY_VAL)) - try_set_sleepy(lock, val); - } + /* Don't set sleepy because we likely have a stale val */ } } -static struct qnode *get_tail_qnode(struct qspinlock *lock, u32 val) +static struct qnode *get_tail_qnode(struct qspinlock *lock, int prev_cpu) { - int cpu = decode_tail_cpu(val); - struct qnodes *qnodesp = per_cpu_ptr(&qnodes, cpu); + struct qnodes *qnodesp = per_cpu_ptr(&qnodes, prev_cpu); int idx; /* @@ -381,9 +377,8 @@ static __always_inline void propagate_yield_cpu(struct qnode *node, u32 val, int } /* Called inside spin_begin() */ -static __always_inline bool yield_to_prev(struct qspinlock *lock, struct qnode *node, u32 val, bool paravirt) +static __always_inline bool yield_to_prev(struct qspinlock *lock, struct qnode *node, int prev_cpu, bool paravirt) { - int prev_cpu = decode_tail_cpu(val); u32 yield_count; int yield_cpu; bool preempted = false; @@ -412,7 +407,7 @@ static __always_inline bool yield_to_prev(struct qspinlock *lock, struct qnode * spin_end(); preempted = true; - seen_sleepy_node(lock, val); + seen_sleepy_node(); smp_rmb(); @@ -436,7 +431,7 @@ static __always_inline bool yield_to_prev(struct qspinlock *lock, struct qnode * spin_end(); preempted = true; - seen_sleepy_node(lock, val); + seen_sleepy_node(); smp_rmb(); /* See __yield_to_locked_owner comment */ @@ -587,7 +582,8 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b * head of the waitqueue. */ if (old & _Q_TAIL_CPU_MASK) { - struct qnode *prev = get_tail_qnode(lock, old); + int prev_cpu = decode_tail_cpu(old); + struct qnode *prev = get_tail_qnode(lock, prev_cpu); /* Link @node into the waitqueue. */ WRITE_ONCE(prev->next, node); @@ -597,7 +593,7 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b while (!READ_ONCE(node->locked)) { spec_barrier(); - if (yield_to_prev(lock, node, old, paravirt)) + if (yield_to_prev(lock, node, prev_cpu, paravirt)) seen_preempted = true; } spec_barrier(); From patchwork Mon Oct 16 12:43:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1849345 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=l/yjKNgs; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4S8H0Y0FYwz20Vq for ; Mon, 16 Oct 2023 23:46:05 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=l/yjKNgs; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4S8H0X6JRTz3vb2 for ; Mon, 16 Oct 2023 23:46:04 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=l/yjKNgs; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::431; helo=mail-pf1-x431.google.com; envelope-from=npiggin@gmail.com; receiver=lists.ozlabs.org) Received: from mail-pf1-x431.google.com (mail-pf1-x431.google.com [IPv6:2607:f8b0:4864:20::431]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4S8Gyd3jGdz3cDS for ; Mon, 16 Oct 2023 23:44:25 +1100 (AEDT) Received: by mail-pf1-x431.google.com with SMTP id d2e1a72fcca58-6bd73395bceso954830b3a.0 for ; Mon, 16 Oct 2023 05:44:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697460262; x=1698065062; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=UupOmSkZAxcmy+M8Pv7epk4VfkElc+gXtgy2XI98HsA=; b=l/yjKNgsjVmWQhNepMpzcoAOevu5htIh6dqx4ZClKu4qGL2Vf/kFlVv2FAQ8de+m0X EjacaM49BpWRD0prjUsREFEOnod9RGOhnDPh4w7uOKXxXVfy7pQc9JaQ1cOnxfGW2QEc TeBoLuim1sWoG04P8b8vMFYZ1SrDJ0czG+bI7pm88YsHvRaYMZl3hb3kZ+ukpcN5xA2y BDa+DLQ1BZ8ARHyYv0VcLwvJcnW05u78ZDU2Brk3HNya+bMfQrQqgDCY720551iGmBkW iwCY9chPehYU/qfcX7pEo4EGXOPM4ivvR++kQasSsw09CixWC7CqZqiwdfV9SGngm/o7 iHfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697460262; x=1698065062; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UupOmSkZAxcmy+M8Pv7epk4VfkElc+gXtgy2XI98HsA=; b=Vx5XSctMPuHIIvkOipFfy4DapYetbL1s2pGbKhUr1h6gGxVO7M04T2t/eC2MX/+jcO iFTyjxRM2sOu/tkn3wfPnWLlmVeU1zt0GVKGzWDPLqwla7BknNvYAM5fJU3ymviA+2KV DXINHc3rSqP9ulwzPR3Qa77VwFI4i1+HuIzoTMdZnthuVx51hNTnikxHysxu+u1xXLew IzRE8uL/A3xcWHVZGz8YIFCJ/QgKdTPfm3BsLfQ8nonMIYXefq3+Y77sdYkIKoMrzokL hCQJ4cEayj1v/d2ySSA33wmRskYqKB0mBPft88rwgOCLqLb0LMLgycRGvbfrAZWmSD9j yptQ== X-Gm-Message-State: AOJu0YzuJbvq8bQ9WAXxoiFZ0+ymiAoRb4FH5aBy4rSXZvZdMwfyHhPT kf86X/CwcLgiF3RsAnFOXVpYPraSA58= X-Google-Smtp-Source: AGHT+IEHcz830r6swNYvqq/XSGCUiGUkS2DyHW6Vl6PCIBiAW2rqe0GsFzat2nZHMFWjJGXft/DLwg== X-Received: by 2002:a05:6a00:1516:b0:68f:b015:ea99 with SMTP id q22-20020a056a00151600b0068fb015ea99mr11295141pfu.10.1697460262529; Mon, 16 Oct 2023 05:44:22 -0700 (PDT) Received: from wheely.local0.net (193-116-200-79.tpgi.com.au. [193.116.200.79]) by smtp.gmail.com with ESMTPSA id t123-20020a625f81000000b0068c10187dc3sm6298850pfb.168.2023.10.16.05.44.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Oct 2023 05:44:22 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH 3/6] powerpc/qspinlock: propagate owner preemptedness rather than CPU number Date: Mon, 16 Oct 2023 22:43:02 +1000 Message-ID: <20231016124305.139923-4-npiggin@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231016124305.139923-1-npiggin@gmail.com> References: <20231016124305.139923-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Shrikanth Hegde , Srikar Dronamraju , Nicholas Piggin , "Nysal Jan K . A" Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Rather than propagating the CPU number of the preempted lock owner, just propagate whether the owner was preempted. Waiters must read the lock value when yielding to it to prevent races anyway, so might as well always load the owner CPU from the lock. To further simplify the code, also don't propagate the -1 (or sleepy=false in the new scheme) down the queue. Instead, have the waiters clear it themselves when finding the lock owner is not preempted. Signed-off-by: Nicholas Piggin --- arch/powerpc/lib/qspinlock.c | 80 ++++++++++++++++-------------------- 1 file changed, 36 insertions(+), 44 deletions(-) diff --git a/arch/powerpc/lib/qspinlock.c b/arch/powerpc/lib/qspinlock.c index 75608ced14c2..0932d24a6b07 100644 --- a/arch/powerpc/lib/qspinlock.c +++ b/arch/powerpc/lib/qspinlock.c @@ -16,7 +16,8 @@ struct qnode { struct qnode *next; struct qspinlock *lock; int cpu; - int yield_cpu; + u8 sleepy; /* 1 if the previous vCPU was preempted or + * if the previous node was sleepy */ u8 locked; /* 1 if lock acquired */ }; @@ -349,7 +350,7 @@ static __always_inline bool yield_head_to_locked_owner(struct qspinlock *lock, u return __yield_to_locked_owner(lock, val, paravirt, mustq); } -static __always_inline void propagate_yield_cpu(struct qnode *node, u32 val, int *set_yield_cpu, bool paravirt) +static __always_inline void propagate_sleepy(struct qnode *node, u32 val, bool *set_sleepy, bool paravirt) { struct qnode *next; int owner; @@ -358,21 +359,17 @@ static __always_inline void propagate_yield_cpu(struct qnode *node, u32 val, int return; if (!pv_yield_propagate_owner) return; - - owner = get_owner_cpu(val); - if (*set_yield_cpu == owner) + if (*set_sleepy) return; next = READ_ONCE(node->next); if (!next) return; + owner = get_owner_cpu(val); if (vcpu_is_preempted(owner)) { - next->yield_cpu = owner; - *set_yield_cpu = owner; - } else if (*set_yield_cpu != -1) { - next->yield_cpu = owner; - *set_yield_cpu = owner; + next->sleepy = 1; + *set_sleepy = true; } } @@ -380,7 +377,6 @@ static __always_inline void propagate_yield_cpu(struct qnode *node, u32 val, int static __always_inline bool yield_to_prev(struct qspinlock *lock, struct qnode *node, int prev_cpu, bool paravirt) { u32 yield_count; - int yield_cpu; bool preempted = false; if (!paravirt) @@ -389,36 +385,32 @@ static __always_inline bool yield_to_prev(struct qspinlock *lock, struct qnode * if (!pv_yield_propagate_owner) goto yield_prev; - yield_cpu = READ_ONCE(node->yield_cpu); - if (yield_cpu == -1) { - /* Propagate back the -1 CPU */ - if (node->next && node->next->yield_cpu != -1) - node->next->yield_cpu = yield_cpu; + if (!READ_ONCE(node->sleepy)) { + /* Propagate back sleepy==false */ + if (node->next && node->next->sleepy) + node->next->sleepy = 0; goto yield_prev; - } - - yield_count = yield_count_of(yield_cpu); - if ((yield_count & 1) == 0) - goto yield_prev; /* owner vcpu is running */ - - if (get_owner_cpu(READ_ONCE(lock->val)) != yield_cpu) - goto yield_prev; /* re-sample lock owner */ - - spin_end(); - - preempted = true; - seen_sleepy_node(); - - smp_rmb(); + } else { + u32 val = READ_ONCE(lock->val); + + if (val & _Q_LOCKED_VAL) { + if (node->next && !node->next->sleepy) { + /* + * Propagate sleepy to next waiter. Only if + * owner is preempted, which allows the queue + * to become "non-sleepy" if vCPU preemption + * ceases to occur, even if the lock remains + * highly contended. + */ + if (vcpu_is_preempted(get_owner_cpu(val))) + node->next->sleepy = 1; + } - if (yield_cpu == node->yield_cpu) { - if (node->next && node->next->yield_cpu != yield_cpu) - node->next->yield_cpu = yield_cpu; - yield_to_preempted(yield_cpu, yield_count); - spin_begin(); - return preempted; + preempted = yield_to_locked_owner(lock, val, paravirt); + if (preempted) + return preempted; + } } - spin_begin(); yield_prev: if (!pv_yield_prev) @@ -541,7 +533,7 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b bool sleepy = false; bool mustq = false; int idx; - int set_yield_cpu = -1; + bool set_sleepy = false; int iters = 0; BUILD_BUG_ON(CONFIG_NR_CPUS >= (1U << _Q_TAIL_CPU_BITS)); @@ -565,7 +557,7 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b node->next = NULL; node->lock = lock; node->cpu = smp_processor_id(); - node->yield_cpu = -1; + node->sleepy = 0; node->locked = 0; tail = encode_tail_cpu(node->cpu); @@ -599,9 +591,9 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b spec_barrier(); spin_end(); - /* Clear out stale propagated yield_cpu */ - if (paravirt && pv_yield_propagate_owner && node->yield_cpu != -1) - node->yield_cpu = -1; + /* Clear out stale propagated sleepy */ + if (paravirt && pv_yield_propagate_owner && node->sleepy) + node->sleepy = 0; smp_rmb(); /* acquire barrier for the mcs lock */ @@ -644,7 +636,7 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b } } - propagate_yield_cpu(node, val, &set_yield_cpu, paravirt); + propagate_sleepy(node, val, &set_sleepy, paravirt); preempted = yield_head_to_locked_owner(lock, val, paravirt); if (!maybe_stealers) continue; From patchwork Mon Oct 16 12:43:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1849346 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=YMBRFNrF; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4S8H1V71s0z20Vq for ; Mon, 16 Oct 2023 23:46:54 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=YMBRFNrF; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4S8H1V4WCZz3vjn for ; Mon, 16 Oct 2023 23:46:54 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=YMBRFNrF; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::42f; helo=mail-pf1-x42f.google.com; envelope-from=npiggin@gmail.com; receiver=lists.ozlabs.org) Received: from mail-pf1-x42f.google.com (mail-pf1-x42f.google.com [IPv6:2607:f8b0:4864:20::42f]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4S8Gyh2hJ6z3ck9 for ; Mon, 16 Oct 2023 23:44:28 +1100 (AEDT) Received: by mail-pf1-x42f.google.com with SMTP id d2e1a72fcca58-6b36e1fcea0so1918189b3a.1 for ; Mon, 16 Oct 2023 05:44:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697460265; x=1698065065; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0B101eYTsYrUl4oIUxvtFA1Uy7uNQ2+OB4reEXGoeJM=; b=YMBRFNrFJWlfBwB1iuOFFQ1tcE8qSJc24Y9ZrzzvdWpzQMuLSlGlx7FjrsP8stHpFD iwjjos5hf3OFGzdcOY+x30lSKzTW8AwlH5JuN/zePqEL1Gy9rKcaPomVRB8RQcB8AAm+ 5S2H/QtigowC2Fh+bZXw5yEE9STL0hPtonnCmML6WNXMg38ki4HV3+OE0QVNOcZme+Qs a9UaiMZc6DNjWDg2HzBB+LB8kT082mGLvDU4nYaFzGi8jqFhko93oQxaaZkIGmmv/AHG d2RsC4KDV4D7KFYA8jdd1S21+NiH6VKBrzDqgoPNZ8MTLHDwvl1RZmyr9Snz1em1pIbf Wl/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697460265; x=1698065065; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0B101eYTsYrUl4oIUxvtFA1Uy7uNQ2+OB4reEXGoeJM=; b=JnZYxh1mga9sC2Tp0MqGmXcgCvh1zMw1zj3U81NN9MfYPo5h3JqFOm6nVbrD+IKmk0 Co3jCH6APt5wSII5/tUxleUgwMLcRhYLjyHxTpcyXD42XXaiqqCi2aS/rzOM1V+uNKxY nzy9ktUHv01b3ApjHoHyfxZnuI3Ao0U+0/MjWYcK9Fow2Jaws5wvIuqOVTdgsm6Y2qu/ cxVLABU+5RE5OWkd9MuccwprInnGrXU/5CzIkE/H3liOU3kEr0NXDZNr+j6gh0QLQrg1 CzPTXyZ9Hol4iCjpiApQoQyeP/u9swL6C9zJ8BW6ULslnYMlnTQ7qIy4bPgekRaK0w/Y RTaQ== X-Gm-Message-State: AOJu0YwNqhfDPIG7CtZv0YDJZ+nkuUD3KGZ+gePHt/fE7N74ax9+GZuE fPJ8ON4dHXHVqzM/l0NYS9ZXFBPT4nU= X-Google-Smtp-Source: AGHT+IFR980nu2DFOp1oTGwPHNBPnDZK7N9EQy9TifMYkOvbxNEsiBGI/ry0YUCZnLSeiVos896qzQ== X-Received: by 2002:a05:6a00:1745:b0:6be:367a:2a71 with SMTP id j5-20020a056a00174500b006be367a2a71mr1467449pfc.16.1697460265547; Mon, 16 Oct 2023 05:44:25 -0700 (PDT) Received: from wheely.local0.net (193-116-200-79.tpgi.com.au. [193.116.200.79]) by smtp.gmail.com with ESMTPSA id t123-20020a625f81000000b0068c10187dc3sm6298850pfb.168.2023.10.16.05.44.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Oct 2023 05:44:25 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH 4/6] powerpc/qspinlock: don't propagate the not-sleepy state Date: Mon, 16 Oct 2023 22:43:03 +1000 Message-ID: <20231016124305.139923-5-npiggin@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231016124305.139923-1-npiggin@gmail.com> References: <20231016124305.139923-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Shrikanth Hegde , Srikar Dronamraju , Nicholas Piggin , "Nysal Jan K . A" Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" To simplify things, don't propagate the not-sleepy condition back down the queue. Instead, have the waiters clear their own node->sleepy when finding the lock owner is not preempted. Signed-off-by: Nicholas Piggin --- arch/powerpc/lib/qspinlock.c | 26 ++++++++------------------ 1 file changed, 8 insertions(+), 18 deletions(-) diff --git a/arch/powerpc/lib/qspinlock.c b/arch/powerpc/lib/qspinlock.c index 0932d24a6b07..6bb627e90a32 100644 --- a/arch/powerpc/lib/qspinlock.c +++ b/arch/powerpc/lib/qspinlock.c @@ -350,7 +350,7 @@ static __always_inline bool yield_head_to_locked_owner(struct qspinlock *lock, u return __yield_to_locked_owner(lock, val, paravirt, mustq); } -static __always_inline void propagate_sleepy(struct qnode *node, u32 val, bool *set_sleepy, bool paravirt) +static __always_inline void propagate_sleepy(struct qnode *node, u32 val, bool paravirt) { struct qnode *next; int owner; @@ -359,18 +359,17 @@ static __always_inline void propagate_sleepy(struct qnode *node, u32 val, bool * return; if (!pv_yield_propagate_owner) return; - if (*set_sleepy) - return; next = READ_ONCE(node->next); if (!next) return; + if (next->sleepy) + return; + owner = get_owner_cpu(val); - if (vcpu_is_preempted(owner)) { + if (vcpu_is_preempted(owner)) next->sleepy = 1; - *set_sleepy = true; - } } /* Called inside spin_begin() */ @@ -385,12 +384,7 @@ static __always_inline bool yield_to_prev(struct qspinlock *lock, struct qnode * if (!pv_yield_propagate_owner) goto yield_prev; - if (!READ_ONCE(node->sleepy)) { - /* Propagate back sleepy==false */ - if (node->next && node->next->sleepy) - node->next->sleepy = 0; - goto yield_prev; - } else { + if (node->sleepy) { u32 val = READ_ONCE(lock->val); if (val & _Q_LOCKED_VAL) { @@ -410,6 +404,7 @@ static __always_inline bool yield_to_prev(struct qspinlock *lock, struct qnode * if (preempted) return preempted; } + node->sleepy = false; } yield_prev: @@ -533,7 +528,6 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b bool sleepy = false; bool mustq = false; int idx; - bool set_sleepy = false; int iters = 0; BUILD_BUG_ON(CONFIG_NR_CPUS >= (1U << _Q_TAIL_CPU_BITS)); @@ -591,10 +585,6 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b spec_barrier(); spin_end(); - /* Clear out stale propagated sleepy */ - if (paravirt && pv_yield_propagate_owner && node->sleepy) - node->sleepy = 0; - smp_rmb(); /* acquire barrier for the mcs lock */ /* @@ -636,7 +626,7 @@ static __always_inline void queued_spin_lock_mcs_queue(struct qspinlock *lock, b } } - propagate_sleepy(node, val, &set_sleepy, paravirt); + propagate_sleepy(node, val, paravirt); preempted = yield_head_to_locked_owner(lock, val, paravirt); if (!maybe_stealers) continue; From patchwork Mon Oct 16 12:43:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1849349 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=Im9GQTEd; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4S8H2R2n97z20Vq for ; Mon, 16 Oct 2023 23:47:43 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=Im9GQTEd; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4S8H2R1wQyz3vfr for ; Mon, 16 Oct 2023 23:47:43 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=Im9GQTEd; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::c36; helo=mail-oo1-xc36.google.com; envelope-from=npiggin@gmail.com; receiver=lists.ozlabs.org) Received: from mail-oo1-xc36.google.com (mail-oo1-xc36.google.com [IPv6:2607:f8b0:4864:20::c36]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4S8Gym17ZKz3fCg for ; Mon, 16 Oct 2023 23:44:32 +1100 (AEDT) Received: by mail-oo1-xc36.google.com with SMTP id 006d021491bc7-57b68556d6dso2499151eaf.1 for ; Mon, 16 Oct 2023 05:44:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697460269; x=1698065069; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LrpQ/NYc53eOlcMLNjra1LpVolMH1LrpZK5UvQKufpY=; b=Im9GQTEdXUNn4e6eKCmXEQ+MGrW3OgnspxahqG5TA+PRLWt4u/qF1HvaG+TxzT1+7Y Zqqaex/GPZqRcUaluDgxYhSCLCTc6bDqJ00uZJHgPGfkvTSP5JkGPWjZK5iKoiaE/mrF qLx+qoQuP/TlaQYIIJ2MsEzIfPGfoTg3sAKgl07+z4xMVcCd7SZD8R7FXe7Z4qJpUD6l IK1tQ3F4P9G5qGNFWyXCK8kFYG9NXR5xVJBUK/gYApnO08eutk50ziGbrOt2jmZm/Awy P4a3MuRdZKMd8lseqa4Q8FI3YZfH8ZqFB/6WtXoQ04/VIG4rELHsp4B5/05nLQ/cqqP7 yJrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697460269; x=1698065069; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LrpQ/NYc53eOlcMLNjra1LpVolMH1LrpZK5UvQKufpY=; b=aBk6Syy3q0xAeFiIpwe6/P3nsSfXGHVEeG167J0Ks3RU1wL1rJ4TjhDdIWsejDR2ws vS1fhy2Y+mBV7r5EB0o9yWO8Yx1Gu0U/XgKA4uq1hNBABU1ibvlud25eQz+/NVwpcDc4 y1QfzoxlOjiQ81ZaTjAVKg5O2Q+JJnrKA84SzIKbFLamV1MPj3eJMns0616QNybpuprw Arb9d6o4E8HRDPEb8o/9Bl1Fl0HjddZg7s2ekLt5hT3RVOSwiR0GyI3FmUSB9s0cMrhS ja21pZGs/RHg7o7MlwAp8+w9PIVg54qvaj9BGH3ImqEQzdQu1ahIK5SvE8EIERu0OiIe n5ww== X-Gm-Message-State: AOJu0YxSeHP0KeZrt09+ya8u/xvSjcmEUv1m1TGEQWnwtzg0XOi+MvHa VG58B5aVgSclq8NMvs9y0yJ/olqsQXo= X-Google-Smtp-Source: AGHT+IHnQE07nBCjGhyVsdRjIZ9XFOH87Yb/VB2M7YLZ8+lIBaUK1ojAoWkT5MyCPjJEVV+9kznBhg== X-Received: by 2002:a05:6358:7e49:b0:145:707f:ebe1 with SMTP id p9-20020a0563587e4900b00145707febe1mr39459416rwm.12.1697460268803; Mon, 16 Oct 2023 05:44:28 -0700 (PDT) Received: from wheely.local0.net (193-116-200-79.tpgi.com.au. [193.116.200.79]) by smtp.gmail.com with ESMTPSA id t123-20020a625f81000000b0068c10187dc3sm6298850pfb.168.2023.10.16.05.44.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Oct 2023 05:44:28 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH 5/6] powerpc/qspinlock: Propagate sleepy if previous waiter is preempted Date: Mon, 16 Oct 2023 22:43:04 +1000 Message-ID: <20231016124305.139923-6-npiggin@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231016124305.139923-1-npiggin@gmail.com> References: <20231016124305.139923-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Shrikanth Hegde , Srikar Dronamraju , Nicholas Piggin , "Nysal Jan K . A" Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" The sleepy (aka lock-owner-is-preempted) condition is propagated down the queue by each waiter. If a waiter becomes preempted, it can no longer propagate sleepy. To allow subsequent waiters to yield to the lock owner, also check the lock owner in this case. Signed-off-by: Nicholas Piggin --- arch/powerpc/lib/qspinlock.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/arch/powerpc/lib/qspinlock.c b/arch/powerpc/lib/qspinlock.c index 6bb627e90a32..c68c2bd7b853 100644 --- a/arch/powerpc/lib/qspinlock.c +++ b/arch/powerpc/lib/qspinlock.c @@ -384,7 +384,11 @@ static __always_inline bool yield_to_prev(struct qspinlock *lock, struct qnode * if (!pv_yield_propagate_owner) goto yield_prev; - if (node->sleepy) { + /* + * If the previous waiter was preempted it might not be able to + * propagate sleepy to us, so check the lock in that case too. + */ + if (node->sleepy || vcpu_is_preempted(prev_cpu)) { u32 val = READ_ONCE(lock->val); if (val & _Q_LOCKED_VAL) { From patchwork Mon Oct 16 12:43:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1849350 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=Q4MChWYC; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=patchwork.ozlabs.org) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4S8H3X31l0z20Vq for ; Mon, 16 Oct 2023 23:48:40 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=Q4MChWYC; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4S8H3X1tD0z3vkK for ; Mon, 16 Oct 2023 23:48:40 +1100 (AEDT) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=Q4MChWYC; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::433; helo=mail-pf1-x433.google.com; envelope-from=npiggin@gmail.com; receiver=lists.ozlabs.org) Received: from mail-pf1-x433.google.com (mail-pf1-x433.google.com [IPv6:2607:f8b0:4864:20::433]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4S8Gyp4Wvnz3c9c for ; Mon, 16 Oct 2023 23:44:34 +1100 (AEDT) Received: by mail-pf1-x433.google.com with SMTP id d2e1a72fcca58-6b77ab73c6fso1686542b3a.1 for ; Mon, 16 Oct 2023 05:44:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697460272; x=1698065072; darn=lists.ozlabs.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OlLNqSZupppKfiY8WdnaqnqKZN1zYh1Is8CUScrbS0g=; b=Q4MChWYCABFcPVj61rzCp6tjBVNZK2ZZfMrxzNB7FMqU5j41QxYX4qXAO0Gok1gws8 btJHhz1quMNEsSZpLZur6O5ObEO8DtQ86cYXvhZpqeahxyCAlGr0lCxcQBD1fDBS/c7h By38sMwqwa2ov2ApaHcZcKnjy09a42qOOAxV2gSVTsN+KgH15EaMhV5CSsPJsfGQ6dVR PPeq0AdP14qDSn+BHeY2jVwW6nctez1CVo4SdRgwW7Z6iJIVA1wCPRDlyTUXRqCbqUQM lhmg/hJUam8IDEx3w4pReeeFtyl9tNWjzm2Iq+ZblG9ESGUXqMtLfzWDI6rFVPxPNSSI hV9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697460272; x=1698065072; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OlLNqSZupppKfiY8WdnaqnqKZN1zYh1Is8CUScrbS0g=; b=HdLpvgeXdybWkI38USswg45IwX75JCf2B31dTwpdMIiHn42YGjYeFyPpl3AsembDaB gLwX6iBPPUtjwRhSHYSBOn1+9C2o8xv25n7nZGJZV7MsyyvnI14DCFpBsJP9A/yJ9py6 Sv2vHTmsiZSzla9qDhmIzh82hkUjZoxyIYJ3nJfEvoBrfj7+Jw/Ims9nJt4S2Y4dHsQw jPCxXAuxQPZbUqj3cblDLHr1jktB2ytMez+526VsrYTlXub0karprbhHc4FJna2bjvVy ri5DXFp4cigQaQEhWga6bZsnNkDU1X7uFdy370nPHiTEn9u3KYV6LXC1Ms7EzvQ4XTOA ks2w== X-Gm-Message-State: AOJu0YyOUsLTrFnMDp5CzfZSrhj5fCYtgNBebr2XXU1+ZQQHeRL4yCZp EcO3Pksti9vt1hSk21gp2mQNyHXY/WY= X-Google-Smtp-Source: AGHT+IERBnj5zsGzEZkix0sE/TWaVH5BYZGfi99cuFbjPA0IPmXJiY7PI/oLhNh4fJyvmug79oAKZw== X-Received: by 2002:a05:6a00:1399:b0:68e:351b:15b9 with SMTP id t25-20020a056a00139900b0068e351b15b9mr9497802pfg.4.1697460271799; Mon, 16 Oct 2023 05:44:31 -0700 (PDT) Received: from wheely.local0.net (193-116-200-79.tpgi.com.au. [193.116.200.79]) by smtp.gmail.com with ESMTPSA id t123-20020a625f81000000b0068c10187dc3sm6298850pfb.168.2023.10.16.05.44.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Oct 2023 05:44:31 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH 6/6] powerpc/qspinlock: Rename yield_propagate_owner tunable Date: Mon, 16 Oct 2023 22:43:05 +1000 Message-ID: <20231016124305.139923-7-npiggin@gmail.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20231016124305.139923-1-npiggin@gmail.com> References: <20231016124305.139923-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Shrikanth Hegde , Srikar Dronamraju , Nicholas Piggin , "Nysal Jan K . A" Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Rename yield_propagate_owner to yield_sleepy_owner, which better describes what it does (what, not how). Signed-off-by: Nicholas Piggin --- arch/powerpc/lib/qspinlock.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/arch/powerpc/lib/qspinlock.c b/arch/powerpc/lib/qspinlock.c index c68c2bd7b853..5de4dd549f6e 100644 --- a/arch/powerpc/lib/qspinlock.c +++ b/arch/powerpc/lib/qspinlock.c @@ -44,7 +44,7 @@ static bool pv_sleepy_lock_sticky __read_mostly = false; static u64 pv_sleepy_lock_interval_ns __read_mostly = 0; static int pv_sleepy_lock_factor __read_mostly = 256; static bool pv_yield_prev __read_mostly = true; -static bool pv_yield_propagate_owner __read_mostly = true; +static bool pv_yield_sleepy_owner __read_mostly = true; static bool pv_prod_head __read_mostly = false; static DEFINE_PER_CPU_ALIGNED(struct qnodes, qnodes); @@ -357,7 +357,7 @@ static __always_inline void propagate_sleepy(struct qnode *node, u32 val, bool p if (!paravirt) return; - if (!pv_yield_propagate_owner) + if (!pv_yield_sleepy_owner) return; next = READ_ONCE(node->next); @@ -381,7 +381,7 @@ static __always_inline bool yield_to_prev(struct qspinlock *lock, struct qnode * if (!paravirt) goto relax; - if (!pv_yield_propagate_owner) + if (!pv_yield_sleepy_owner) goto yield_prev; /* @@ -934,21 +934,21 @@ static int pv_yield_prev_get(void *data, u64 *val) DEFINE_SIMPLE_ATTRIBUTE(fops_pv_yield_prev, pv_yield_prev_get, pv_yield_prev_set, "%llu\n"); -static int pv_yield_propagate_owner_set(void *data, u64 val) +static int pv_yield_sleepy_owner_set(void *data, u64 val) { - pv_yield_propagate_owner = !!val; + pv_yield_sleepy_owner = !!val; return 0; } -static int pv_yield_propagate_owner_get(void *data, u64 *val) +static int pv_yield_sleepy_owner_get(void *data, u64 *val) { - *val = pv_yield_propagate_owner; + *val = pv_yield_sleepy_owner; return 0; } -DEFINE_SIMPLE_ATTRIBUTE(fops_pv_yield_propagate_owner, pv_yield_propagate_owner_get, pv_yield_propagate_owner_set, "%llu\n"); +DEFINE_SIMPLE_ATTRIBUTE(fops_pv_yield_sleepy_owner, pv_yield_sleepy_owner_get, pv_yield_sleepy_owner_set, "%llu\n"); static int pv_prod_head_set(void *data, u64 val) { @@ -980,7 +980,7 @@ static __init int spinlock_debugfs_init(void) debugfs_create_file("qspl_pv_sleepy_lock_interval_ns", 0600, arch_debugfs_dir, NULL, &fops_pv_sleepy_lock_interval_ns); debugfs_create_file("qspl_pv_sleepy_lock_factor", 0600, arch_debugfs_dir, NULL, &fops_pv_sleepy_lock_factor); debugfs_create_file("qspl_pv_yield_prev", 0600, arch_debugfs_dir, NULL, &fops_pv_yield_prev); - debugfs_create_file("qspl_pv_yield_propagate_owner", 0600, arch_debugfs_dir, NULL, &fops_pv_yield_propagate_owner); + debugfs_create_file("qspl_pv_yield_sleepy_owner", 0600, arch_debugfs_dir, NULL, &fops_pv_yield_sleepy_owner); debugfs_create_file("qspl_pv_prod_head", 0600, arch_debugfs_dir, NULL, &fops_pv_prod_head); }