From patchwork Wed Apr 19 01:17:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Noah Goldstein X-Patchwork-Id: 1770469 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=sourceware.org; envelope-from=libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; secure) header.d=sourceware.org header.i=@sourceware.org header.a=rsa-sha256 header.s=default header.b=bZ5m6HDL; dkim-atps=neutral Received: from sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Q1NFS1DKQz1yZk for ; Wed, 19 Apr 2023 11:17:48 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 096E23857703 for ; Wed, 19 Apr 2023 01:17:46 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 096E23857703 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1681867066; bh=wlsnY6I6iXbtZs/rnMrhVJlEOpEOn/hglRk+4OB3BKs=; h=To:Cc:Subject:Date:List-Id:List-Unsubscribe:List-Archive: List-Post:List-Help:List-Subscribe:From:Reply-To:From; b=bZ5m6HDLE5qxyMdDROpJfRJ/IUlRR0l+aZlobRIYeeJXLDYMAacghxs1hslUNH76E f61oXPLj1aoC0jorSvnBFD6tapuxL61VQ4jZlI5EotfS7ur4WT/Ifh3u3/rdw7Aqpo HT/FV/Necgad5GFgyIRSusLkJJQCa1nPAKdtHU1Y= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-ej1-x630.google.com (mail-ej1-x630.google.com [IPv6:2a00:1450:4864:20::630]) by sourceware.org (Postfix) with ESMTPS id B5C313858D1E for ; Wed, 19 Apr 2023 01:17:30 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org B5C313858D1E Received: by mail-ej1-x630.google.com with SMTP id fw30so24928172ejc.5 for ; Tue, 18 Apr 2023 18:17:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681867048; x=1684459048; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=wlsnY6I6iXbtZs/rnMrhVJlEOpEOn/hglRk+4OB3BKs=; b=OtLgCvuS+hLALrcaz6rGgynHKS4SmXjx/2oq9949/b6wM8EDrCBrdsryMcIyl2qM4a tfYwpBhc3p3Be/88Ql+Ekhqg6xEn7/7NCZamPCUQgZyf29v2LCpGmHVnB+Di77EiLvHZ zgzKcea7HxE38vH4Mt9PDyWCNigWfi1pM8K2I2hrt8UBQmqynEqmAX3qX0vpWLF8tPUz gA2nNE5GenhziKQxvhib/KgmTUfJ44F0+eoJu2md7fI1OKHrw8OjerdKWgqmUtM2nFWm fyMlFrn6MBEjr2dCHc61K9RFJwaRgbCLh4gizNszpW0frM3dmq3lWW4eV1osvIHezVpK H7Sw== X-Gm-Message-State: AAQBX9eWU4+C1xYvxsIf9U8yLukCnj6BW/s8JGke1uZJCtwX4NAp7nTL Jg6pfzlDKZPLI/oe0rrf/kzkxQlWdsk= X-Google-Smtp-Source: AKy350ZRoXP99eqmmXArLlCOj0+5qJlEZUKFVUo4yksjn/NaOQfjLUJrcRRn/EwuIE7Xc8B/Gbp5QA== X-Received: by 2002:a17:906:3454:b0:94e:e859:7b07 with SMTP id d20-20020a170906345400b0094ee8597b07mr15380473ejb.32.1681867048339; Tue, 18 Apr 2023 18:17:28 -0700 (PDT) Received: from noahgold-desk.intel.com ([2600:1010:b044:5553:786d:a5bb:b72b:86f8]) by smtp.gmail.com with ESMTPSA id q17-20020a1709066ad100b0094f6f45b2c1sm3924713ejs.156.2023.04.18.18.17.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Apr 2023 18:17:28 -0700 (PDT) To: libc-alpha@sourceware.org Cc: goldstein.w.n@gmail.com, hjl.tools@gmail.com, carlos@systemhalted.org Subject: [PATCH v1] nptl: Add single-threaded optimization to ADAPTIVE_NP mutex Date: Tue, 18 Apr 2023 20:17:22 -0500 Message-Id: <20230419011722.1154501-1-goldstein.w.n@gmail.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Spam-Status: No, score=-12.1 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Noah Goldstein via Libc-alpha From: Noah Goldstein Reply-To: Noah Goldstein Errors-To: libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org Sender: "Libc-alpha" ADAPTIVED_NP mutex is generally the most performant mutex option and is commonly used so it seems preferable for it to also benefit from the optimization. make check passes on linux-x86_64 --- nptl/pthread_mutex_cond_lock.c | 1 + nptl/pthread_mutex_lock.c | 40 ++++++++++++++++++++++++++-------- 2 files changed, 32 insertions(+), 9 deletions(-) diff --git a/nptl/pthread_mutex_cond_lock.c b/nptl/pthread_mutex_cond_lock.c index f3af514305..fcc90bb16c 100644 --- a/nptl/pthread_mutex_cond_lock.c +++ b/nptl/pthread_mutex_cond_lock.c @@ -3,6 +3,7 @@ #define LLL_MUTEX_LOCK(mutex) \ lll_cond_lock ((mutex)->__data.__lock, PTHREAD_MUTEX_PSHARED (mutex)) #define LLL_MUTEX_LOCK_OPTIMIZED(mutex) LLL_MUTEX_LOCK (mutex) +#define LLL_MUTEX_TRYLOCK_OPTIMIZED(mutex) LLL_MUTEX_TRYLOCK (mutex) /* Not actually elided so far. Needed? */ #define LLL_MUTEX_LOCK_ELISION(mutex) \ diff --git a/nptl/pthread_mutex_lock.c b/nptl/pthread_mutex_lock.c index d4f96c70ef..011bd7488d 100644 --- a/nptl/pthread_mutex_lock.c +++ b/nptl/pthread_mutex_lock.c @@ -30,9 +30,9 @@ /* Some of the following definitions differ when pthread_mutex_cond_lock.c includes this file. */ #ifndef LLL_MUTEX_LOCK -/* lll_lock with single-thread optimization. */ -static inline void -lll_mutex_lock_optimized (pthread_mutex_t *mutex) + +static inline int +lll_mutex_try_singlethreaded_opt (pthread_mutex_t *mutex, int private) { /* The single-threaded optimization is only valid for private mutexes. For process-shared mutexes, the mutex could be in a @@ -41,16 +41,38 @@ lll_mutex_lock_optimized (pthread_mutex_t *mutex) acquired, POSIX requires that pthread_mutex_lock deadlocks for normal mutexes, so skip the optimization in that case as well. */ - int private = PTHREAD_MUTEX_PSHARED (mutex); if (private == LLL_PRIVATE && SINGLE_THREAD_P && mutex->__data.__lock == 0) - mutex->__data.__lock = 1; - else + { + mutex->__data.__lock = 1; + return 0; + } + return 1; +} + +/* lll_trylock with single-thread optimization. */ +static inline int +lll_mutex_trylock_optimized (pthread_mutex_t *mutex) +{ + if (lll_mutex_try_singlethreaded_opt (mutex, PTHREAD_MUTEX_PSHARED (mutex)) + == 0) + return 0; + return lll_trylock (mutex->__data.__lock); +} + +/* lll_lock with single-thread optimization. */ +static inline void +lll_mutex_lock_optimized (pthread_mutex_t *mutex) +{ + int private = PTHREAD_MUTEX_PSHARED (mutex); + if (lll_mutex_try_singlethreaded_opt (mutex, private)) lll_lock (mutex->__data.__lock, private); } # define LLL_MUTEX_LOCK(mutex) \ lll_lock ((mutex)->__data.__lock, PTHREAD_MUTEX_PSHARED (mutex)) # define LLL_MUTEX_LOCK_OPTIMIZED(mutex) lll_mutex_lock_optimized (mutex) +# define LLL_MUTEX_TRYLOCK_OPTIMIZED(mutex) \ + lll_mutex_trylock_optimized (mutex) # define LLL_MUTEX_TRYLOCK(mutex) \ lll_trylock ((mutex)->__data.__lock) # define LLL_ROBUST_MUTEX_LOCK_MODIFIER 0 @@ -133,11 +155,11 @@ PTHREAD_MUTEX_LOCK (pthread_mutex_t *mutex) else if (__builtin_expect (PTHREAD_MUTEX_TYPE (mutex) == PTHREAD_MUTEX_ADAPTIVE_NP, 1)) { - if (LLL_MUTEX_TRYLOCK (mutex) != 0) + if (LLL_MUTEX_TRYLOCK_OPTIMIZED (mutex) != 0) { int cnt = 0; - int max_cnt = MIN (max_adaptive_count (), - mutex->__data.__spins * 2 + 10); + int max_cnt + = MIN (max_adaptive_count (), mutex->__data.__spins * 2 + 10); int spin_count, exp_backoff = 1; unsigned int jitter = get_jitter (); do