From patchwork Wed May 24 06:08:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1785539 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20221208 header.b=NFHsc9hw; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4QR13z2l4cz20Pb for ; Wed, 24 May 2023 16:09:35 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4QR13z1gdTz3fBX for ; Wed, 24 May 2023 16:09:35 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20221208 header.b=NFHsc9hw; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::42d; helo=mail-pf1-x42d.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20221208 header.b=NFHsc9hw; dkim-atps=neutral Received: from mail-pf1-x42d.google.com (mail-pf1-x42d.google.com [IPv6:2607:f8b0:4864:20::42d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4QR12n30hnz3bgn for ; Wed, 24 May 2023 16:08:33 +1000 (AEST) Received: by mail-pf1-x42d.google.com with SMTP id d2e1a72fcca58-64d2c865e4eso382034b3a.0 for ; Tue, 23 May 2023 23:08:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684908511; x=1687500511; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zHRrz+AGNmJliP5aOJUlFgtC9T/nLlGEtygMnlMlFwk=; b=NFHsc9hwv/Mc9mA6uYrnVOWvhi8x/MdNc66Dq6Uu5RQ3d/l0/7Oi0viuudvrrvq0Iv u6yPUhUZQUJpDI4xR5nxW61vnGcCedgzTwecaIFUGjdxj2XE4Z/5rVZQm3A/5zXtTj+1 8UE65Xj1+JFkPzRheZ8uGXgP1K+6M70x27n91UJ6nQwH0dQfi05JbRYEJcE5pRxrdnfk azXYPu5RGbw+FExt5Vq7vFM2+VxcD87+r1NIzGsE72Rggwpa5QEzcaifw9RfS9Ylj6+M LKVk909cbWXVBgHFwIbQTex/mfzAmd8imQnuoDxOoKuXhL/YAt0IsLGqYnEikWsIpqmy DoWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684908511; x=1687500511; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zHRrz+AGNmJliP5aOJUlFgtC9T/nLlGEtygMnlMlFwk=; b=hSNxuYQoy3nGlZsfw3TRUIvFGOCGil+m1m66I/Vvtccv5REF9coA0nfVDWPQXSWByb ZPml3nk3FA5eC+PeYtTmJwgu0cIHqlN8yuJE6yHIaV9wBHWIdy+6qixyAI3P2bS2GJIi l2uM9w27qCl1qCQHGFnbOQC+00kEglccXQAvbGAR6dIBlWtQLxI3XPu5paem0TtF7Wtk MZ9vc1W3+1YuDBFzzNSS3heqVqwZcxD9bimR3g6MUhNYFUwIpNdHdUwxWFXx9EpQhEaW CWN+FD6J7Zti3ce0Ca6kva/N5Whyxo9gGOmEsQHR4KcbxC1Ea9ss7Hk+N2yDzRYJHcEy 9keA== X-Gm-Message-State: AC+VfDyYLb/UYIKsMMik/ZpruZjz0qONaSJJfEeMIJDCT+2Igta0eY1+ /f+bs6WHHea39Hn0R/8Dvd4MuWQFxGk= X-Google-Smtp-Source: ACHHUZ7wfFfh9j5weGGZoSRb+OxRWyu8cyLicJpGhyfElakzQg0ANZ6XWrtM6mFAUIrHFJodaWYSyg== X-Received: by 2002:a17:902:e848:b0:1ad:d500:19d3 with SMTP id t8-20020a170902e84800b001add50019d3mr21348233plg.23.1684908510636; Tue, 23 May 2023 23:08:30 -0700 (PDT) Received: from wheely.local0.net ([220.240.241.243]) by smtp.gmail.com with ESMTPSA id 9-20020a170902c14900b001aaecb74f82sm7793019plj.72.2023.05.23.23.08.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 May 2023 23:08:30 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH 1/4] powerpc: Account mm_cpumask and active_cpus in init_mm Date: Wed, 24 May 2023 16:08:18 +1000 Message-Id: <20230524060821.148015-2-npiggin@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230524060821.148015-1-npiggin@gmail.com> References: <20230524060821.148015-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" init_mm mm_cpumask and context.active_cpus is not maintained at boot and hotplug. This seems to be harmless because init_mm does not have a userspace and so never gets user TLBs flushed, but it looks odd and it prevents some sanity checks being added. Signed-off-by: Nicholas Piggin --- arch/powerpc/kernel/setup-common.c | 6 +++++- arch/powerpc/kernel/smp.c | 12 ++++++++++++ arch/powerpc/mm/mmu_context.c | 1 + 3 files changed, 18 insertions(+), 1 deletion(-) diff --git a/arch/powerpc/kernel/setup-common.c b/arch/powerpc/kernel/setup-common.c index d2a446216444..16843294d978 100644 --- a/arch/powerpc/kernel/setup-common.c +++ b/arch/powerpc/kernel/setup-common.c @@ -969,8 +969,12 @@ void __init setup_arch(char **cmdline_p) klp_init_thread_info(&init_task); setup_initial_init_mm(_stext, _etext, _edata, _end); - + /* sched_init() does the mmgrab(&init_mm) for the primary CPU */ + VM_WARN_ON(cpumask_test_cpu(smp_processor_id(), mm_cpumask(&init_mm))); + cpumask_set_cpu(smp_processor_id(), mm_cpumask(&init_mm)); + inc_mm_active_cpus(&init_mm); mm_iommu_init(&init_mm); + irqstack_early_init(); exc_lvl_early_init(); emergency_stack_init(); diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c index 265801a3e94c..76a57dc753c8 100644 --- a/arch/powerpc/kernel/smp.c +++ b/arch/powerpc/kernel/smp.c @@ -47,6 +47,7 @@ #include #include #include +#include #include #include #include @@ -1615,6 +1616,9 @@ void start_secondary(void *unused) mmgrab_lazy_tlb(&init_mm); current->active_mm = &init_mm; + VM_WARN_ON(cpumask_test_cpu(smp_processor_id(), mm_cpumask(&init_mm))); + cpumask_set_cpu(cpu, mm_cpumask(&init_mm)); + inc_mm_active_cpus(&init_mm); smp_store_cpu_info(cpu); set_dec(tb_ticks_per_jiffy); @@ -1750,6 +1754,14 @@ int __cpu_disable(void) void __cpu_die(unsigned int cpu) { + /* + * This could perhaps be a generic call in idlea_task_dead(), but + * that requires testing from all archs, so first put it here to + */ + VM_WARN_ON_ONCE(!cpumask_test_cpu(cpu, mm_cpumask(&init_mm))); + dec_mm_active_cpus(&init_mm); + cpumask_clear_cpu(cpu, mm_cpumask(&init_mm)); + if (smp_ops->cpu_die) smp_ops->cpu_die(cpu); } diff --git a/arch/powerpc/mm/mmu_context.c b/arch/powerpc/mm/mmu_context.c index 1fb9c99f8679..894468975a44 100644 --- a/arch/powerpc/mm/mmu_context.c +++ b/arch/powerpc/mm/mmu_context.c @@ -47,6 +47,7 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, /* Mark this context has been used on the new CPU */ if (!cpumask_test_cpu(smp_processor_id(), mm_cpumask(next))) { + VM_WARN_ON_ONCE(next == &init_mm); cpumask_set_cpu(smp_processor_id(), mm_cpumask(next)); inc_mm_active_cpus(next); From patchwork Wed May 24 06:08:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1785540 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20221208 header.b=UF8PmRqz; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4QR14v0pNyz20PY for ; Wed, 24 May 2023 16:10:23 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4QR14t5hGsz3f6v for ; Wed, 24 May 2023 16:10:22 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20221208 header.b=UF8PmRqz; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::536; helo=mail-pg1-x536.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20221208 header.b=UF8PmRqz; dkim-atps=neutral Received: from mail-pg1-x536.google.com (mail-pg1-x536.google.com [IPv6:2607:f8b0:4864:20::536]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4QR12q5Tyrz3f6q for ; Wed, 24 May 2023 16:08:35 +1000 (AEST) Received: by mail-pg1-x536.google.com with SMTP id 41be03b00d2f7-5289cf35eeaso4072119a12.1 for ; Tue, 23 May 2023 23:08:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684908513; x=1687500513; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bF/yqgmO0OQmx/RXJ7rKmP80tBqdzMLehuYCltJdPfQ=; b=UF8PmRqzrtN4ASUn74FYkIANH2u8pIYq904IGeL44nmKfrWFqd2ccB2ryqxtAinad7 W+NRDj6UHTB8xspiNPNKBRIo9RCZEguooz4ArbkqzCavUw/zjVyDZP5n009JQqMDETmj joXwztiK08JsvWQJwdLTQqAgA+Ol7j2XFZvp8azgnBZaBl3k4e72ntL24f1rUiHe+9+S 0XTAw4W6soaWptvbGBfm2IjZzq6Pt7/JpGoK5u5a3qfe0Kp2j2TctgxC/nsSqSG6lktf /GqDRYxpNVQzwhz6DlSwZitoV9ioS4FgERCG2ZKsE+638c4D9YgBnN+zKpxNKKiu9D83 b0Fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684908513; x=1687500513; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bF/yqgmO0OQmx/RXJ7rKmP80tBqdzMLehuYCltJdPfQ=; b=V/snSdQ5/kpem7vTtKQLQ5WzGuMow5pVhsvsc/e5UiyTajZZcAwd+B7hSEe5bCXor6 DeZNCfJe8QRtdg9ZnrgSJetQDtU96H2nZfTOEXRiqtshg+xWwWJp4HA9e0WiOhDg7D5C aR3OqZQwrCX4Ns/Um7ey7jDJsNyszSH9Z62vhrhJgjWhAbni6z5KXaMa2Ru8yZ1AZkYe Ahaaqtwl7zARV6dIkO/UPMcehFTDJx9HEZGvr8fjD+wYNNY7ZcdnnfplMyHsCjuiC1dX UswpQqfpM2zSC1FrmwDdkbA89JXBrasuiEC2o/B5H76/O/tW4tk1QysL8/Wri3GFpIYE pbZg== X-Gm-Message-State: AC+VfDyzJH2fGIXHw8osp5bqymjXWgdjAa8L10U/M8zZVZcXfn1qaQ37 cIvzMW/974cmY4xijxSSgMeG42IL3u0= X-Google-Smtp-Source: ACHHUZ6DFXDPXPO0d2Iev7kqR06ozgfgiHVN51Fn0PtFjY9YaGODbrECoY8Y7seIH13Rox5j6QQWZA== X-Received: by 2002:a17:903:234a:b0:1ac:a88a:70b6 with SMTP id c10-20020a170903234a00b001aca88a70b6mr21730776plh.31.1684908513162; Tue, 23 May 2023 23:08:33 -0700 (PDT) Received: from wheely.local0.net ([220.240.241.243]) by smtp.gmail.com with ESMTPSA id 9-20020a170902c14900b001aaecb74f82sm7793019plj.72.2023.05.23.23.08.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 May 2023 23:08:32 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH 2/4] powerpc/64s: Use dec_mm_active_cpus helper Date: Wed, 24 May 2023 16:08:19 +1000 Message-Id: <20230524060821.148015-3-npiggin@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230524060821.148015-1-npiggin@gmail.com> References: <20230524060821.148015-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Avoid open-coded atomic_dec on mm->context.active_cpus and use the function made for it. Add CONFIG_DEBUG_VM underflow checking on the counter. Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/book3s/64/mmu.h | 2 +- arch/powerpc/include/asm/mmu_context.h | 1 + arch/powerpc/mm/book3s64/radix_tlb.c | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/mmu.h b/arch/powerpc/include/asm/book3s/64/mmu.h index 570a4960cf17..5cf0e9c953b3 100644 --- a/arch/powerpc/include/asm/book3s/64/mmu.h +++ b/arch/powerpc/include/asm/book3s/64/mmu.h @@ -261,7 +261,7 @@ static inline void radix_init_pseries(void) { } #define arch_clear_mm_cpumask_cpu(cpu, mm) \ do { \ if (cpumask_test_cpu(cpu, mm_cpumask(mm))) { \ - atomic_dec(&(mm)->context.active_cpus); \ + dec_mm_active_cpus(mm); \ cpumask_clear_cpu(cpu, mm_cpumask(mm)); \ } \ } while (0) diff --git a/arch/powerpc/include/asm/mmu_context.h b/arch/powerpc/include/asm/mmu_context.h index 57f5017111f4..37bffa0f7918 100644 --- a/arch/powerpc/include/asm/mmu_context.h +++ b/arch/powerpc/include/asm/mmu_context.h @@ -127,6 +127,7 @@ static inline void inc_mm_active_cpus(struct mm_struct *mm) static inline void dec_mm_active_cpus(struct mm_struct *mm) { + VM_WARN_ON_ONCE(atomic_read(&mm->context.active_cpus) <= 0); atomic_dec(&mm->context.active_cpus); } diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c b/arch/powerpc/mm/book3s64/radix_tlb.c index 90953cf9f648..8160c1630c3d 100644 --- a/arch/powerpc/mm/book3s64/radix_tlb.c +++ b/arch/powerpc/mm/book3s64/radix_tlb.c @@ -808,7 +808,7 @@ void exit_lazy_flush_tlb(struct mm_struct *mm, bool always_flush) * that's what the caller expects. */ if (cpumask_test_cpu(cpu, mm_cpumask(mm))) { - atomic_dec(&mm->context.active_cpus); + dec_mm_active_cpus(mm); cpumask_clear_cpu(cpu, mm_cpumask(mm)); always_flush = true; } From patchwork Wed May 24 06:08:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1785541 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20221208 header.b=jnqwi6DJ; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4QR15p5FMFz20PY for ; Wed, 24 May 2023 16:11:10 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4QR15p2GZYz3cjL for ; Wed, 24 May 2023 16:11:10 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20221208 header.b=jnqwi6DJ; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::62d; helo=mail-pl1-x62d.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20221208 header.b=jnqwi6DJ; dkim-atps=neutral Received: from mail-pl1-x62d.google.com (mail-pl1-x62d.google.com [IPv6:2607:f8b0:4864:20::62d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4QR12t3clwz3f5r for ; Wed, 24 May 2023 16:08:38 +1000 (AEST) Received: by mail-pl1-x62d.google.com with SMTP id d9443c01a7336-1ae40dcdc18so4212045ad.2 for ; Tue, 23 May 2023 23:08:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684908516; x=1687500516; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FNTKzgNHoYsYBJ1w6ox2ZWziujVaGJw7eJ2ap0bcafU=; b=jnqwi6DJC4WUuMuWbbNEgZvR/B264VWRneOlrpnRbq3cAzVd2M3AAb7bJigwLL3BAu fqSgWCIvvx98ehItGimgQt6DIQKYdcZ4f4iN3ihDidmTBWepEtES21u6s3iHGINuwC7o MuPP4pwqyUsBfo5HOVlztqY3fS5Xlf9i4Pb5L4B8otXtuwZDQnyK7KpUFUEeAS+lbUWi VHtLBlCcTjT4ju8++iGdv5SG+zslN/iiGVn02HNRWIkn1kqqLySPcNlQ27lqW8O7I3RH u8LuWzdKPEiKDFHIa4CQDHM1xHjULEhKSDEeubi1ui04Xt3mPKFC7lApNlcuvahyreTa msoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684908516; x=1687500516; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FNTKzgNHoYsYBJ1w6ox2ZWziujVaGJw7eJ2ap0bcafU=; b=YCyKRGjsczrP7d+DmaG6rCP/5pZL47nh24CH3UDCVuCBzTaeyIWjLI26IzZbGjzcA8 J6JIzXlBE01rvLBnsNrj9VHrQFqfjm9Jc6eDC4D51rPCpAXRkzMIEQBM55Oy1AZcu8gD 6vpPEBDrz9WYUd5uz292Pg0VSFZZn+hJH0Kx6yU1nvDsM+uL15hLxTuDoDKuqrCHQcuy 3Ao9SPb6IhoBG7A3YDR18XgANT8U+Uj8bcLEZ7CW5oqPPLoVLbvKE/BTu1XO1ouahD8a 6amZfbjOL8FARlnnSU3m58ovsdpgCEU8c0bcSmGcsO08BEGIhpIk55gN3IyjLOa5rLbF an/Q== X-Gm-Message-State: AC+VfDxCiEVw1V4EXpXlEhLeDnOqQwdB5ynkyv+bQhWM68+jM1gx+fFj bmwWQqp+m5SmkBqJIs2w66W3jUYH/VA= X-Google-Smtp-Source: ACHHUZ51XEqCLrHiWbknpM3feLfj7paXYm9dZfTfOigiY7HGI0Tx1SVUWuZ+wp8wJUWCKIFPrvFcIw== X-Received: by 2002:a17:902:e5ce:b0:1a6:4127:857 with SMTP id u14-20020a170902e5ce00b001a641270857mr20731309plf.5.1684908515832; Tue, 23 May 2023 23:08:35 -0700 (PDT) Received: from wheely.local0.net ([220.240.241.243]) by smtp.gmail.com with ESMTPSA id 9-20020a170902c14900b001aaecb74f82sm7793019plj.72.2023.05.23.23.08.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 May 2023 23:08:35 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH 3/4] powerpc: Add mm_cpumask warning when context switching Date: Wed, 24 May 2023 16:08:20 +1000 Message-Id: <20230524060821.148015-4-npiggin@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230524060821.148015-1-npiggin@gmail.com> References: <20230524060821.148015-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" When context switching away from an mm, add a CONFIG_DEBUG_VM warning check to ensure this CPU is still set in the mask. This could catch bugs where the mask is improperly trimmed while the CPU is still using the mm. Signed-off-by: Nicholas Piggin --- arch/powerpc/mm/mmu_context.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/arch/powerpc/mm/mmu_context.c b/arch/powerpc/mm/mmu_context.c index 894468975a44..b24c19078eb1 100644 --- a/arch/powerpc/mm/mmu_context.c +++ b/arch/powerpc/mm/mmu_context.c @@ -43,12 +43,13 @@ static inline void switch_mm_pgdir(struct task_struct *tsk, void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, struct task_struct *tsk) { + int cpu = smp_processor_id(); bool new_on_cpu = false; /* Mark this context has been used on the new CPU */ - if (!cpumask_test_cpu(smp_processor_id(), mm_cpumask(next))) { + if (!cpumask_test_cpu(cpu, mm_cpumask(next))) { VM_WARN_ON_ONCE(next == &init_mm); - cpumask_set_cpu(smp_processor_id(), mm_cpumask(next)); + cpumask_set_cpu(cpu, mm_cpumask(next)); inc_mm_active_cpus(next); /* @@ -101,6 +102,8 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, * sub architectures. Out of line for now */ switch_mmu_context(prev, next, tsk); + + VM_WARN_ON_ONCE(!cpumask_test_cpu(cpu, mm_cpumask(prev))); } #ifndef CONFIG_PPC_BOOK3S_64 From patchwork Wed May 24 06:08:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 1785542 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20221208 header.b=YVfR4h9r; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4QR16k2x92z20PY for ; Wed, 24 May 2023 16:11:58 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4QR16j5vzbz3fPJ for ; Wed, 24 May 2023 16:11:57 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20221208 header.b=YVfR4h9r; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::52e; helo=mail-pg1-x52e.google.com; envelope-from=npiggin@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20221208 header.b=YVfR4h9r; dkim-atps=neutral Received: from mail-pg1-x52e.google.com (mail-pg1-x52e.google.com [IPv6:2607:f8b0:4864:20::52e]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4QR12x1Jvdz3f7v for ; Wed, 24 May 2023 16:08:41 +1000 (AEST) Received: by mail-pg1-x52e.google.com with SMTP id 41be03b00d2f7-5343c3daff0so427017a12.0 for ; Tue, 23 May 2023 23:08:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684908518; x=1687500518; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yNILb4+cQ1eKYMAz1WBE8LcfWvbmXRGydxXui234anI=; b=YVfR4h9r61gNpNxdgsBp/tsaw5J1Yf+//AN5hJXt1uUxa/juzR5qD/miX2SLekqyBH NM1xrV7WiQRODbdSRn2K9rS45PRp2DZKBpzmN9okTD5y4gmUSLC4MiYJtc7Lxdey6hWQ D9vV339rLCSCTeLJmHSOxCIMEMX2thLLNdWPscRcDGMBa8DW/JSMJJHhDvb4jrAO1fhH YXk/5vgAA5zY6ilj8d/3cIzGRdqhdUThVZmydvgFFFD+o9Dh2SCPmLTCiT7lxpOYVigZ wyFfbOQLkdliFTFX0sDPoFZscenI5qRzY54sOAEUM9GAi1cAm7tUKUkW5ugzT+nvKeOB rfZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684908518; x=1687500518; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yNILb4+cQ1eKYMAz1WBE8LcfWvbmXRGydxXui234anI=; b=JGeg+PlTCTuUIuEV2lsqwqPzPxVmu4B56Zza1giRTB9qMrtlK+SPlRUij29FAUew+z 9NqrXjzOVdw2730dBZj8JDdpTxbtQakpYdiK5HD0IaNQrnFE8haNmmfwtKCuFUOcNN6I J7aY0u1r2h9PVUvXE1eXh3DW8UQxmQ/5G24BRMwosPAeQ6mThLvf+xNOTbM0TEWjaRdO N6rsIqu6y5Du2YhUEESmJxjkx7JmO+J9EiVqjcaqZU85g04iyyr8fEeex0T4cQ0hLpIK Inj/CkDPgHjU4xCk/iHAmgyddQ6LS9qGYfcRM2X/6+IqyggclyV6eJ6v3m5g3bVd95XA egjg== X-Gm-Message-State: AC+VfDyvpA+opBpD3bHnTerNSyTROTc1+VzmteeZF4FXwbbotAaIcF1y SFmquqM8AvPyPSGEzzQA4Ub3IP7w1iE= X-Google-Smtp-Source: ACHHUZ6FZolKd8yFWy74Jr7IyAijTLR9yo33/Pbk7T39hY2GS0YtQQ3pD5PcY/Ct8Vr5fkbwXtpsJA== X-Received: by 2002:a17:902:8d8a:b0:1ad:7bc5:b9ea with SMTP id v10-20020a1709028d8a00b001ad7bc5b9eamr13402174plo.60.1684908518594; Tue, 23 May 2023 23:08:38 -0700 (PDT) Received: from wheely.local0.net ([220.240.241.243]) by smtp.gmail.com with ESMTPSA id 9-20020a170902c14900b001aaecb74f82sm7793019plj.72.2023.05.23.23.08.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 May 2023 23:08:38 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH 4/4] powerpc/64s/radix: combine final TLB flush and lazy tlb mm shootdown IPIs Date: Wed, 24 May 2023 16:08:21 +1000 Message-Id: <20230524060821.148015-5-npiggin@gmail.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230524060821.148015-1-npiggin@gmail.com> References: <20230524060821.148015-1-npiggin@gmail.com> MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" This performs lazy tlb mm shootdown when doing the exit TLB flush when all mm users go away and user mappings are removed, which avoids having to do the lazy tlb mm shootdown IPIs on the final mmput when all kernel references disappear. powerpc/64s uses a broadcast TLBIE for the exit TLB flush if remote CPUs need to be invalidated (unless TLBIE is disabled), so this doesn't necessarily save IPIs but it does avoid a broadcast TLBIE which is quite expensive. Signed-off-by: Nicholas Piggin --- arch/powerpc/mm/book3s64/radix_tlb.c | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/arch/powerpc/mm/book3s64/radix_tlb.c b/arch/powerpc/mm/book3s64/radix_tlb.c index 8160c1630c3d..e2aaee6df1f6 100644 --- a/arch/powerpc/mm/book3s64/radix_tlb.c +++ b/arch/powerpc/mm/book3s64/radix_tlb.c @@ -1301,7 +1301,31 @@ void radix__tlb_flush(struct mmu_gather *tlb) * See the comment for radix in arch_exit_mmap(). */ if (tlb->fullmm) { - __flush_all_mm(mm, true); + if (IS_ENABLED(CONFIG_MMU_LAZY_TLB_SHOOTDOWN)) { + /* + * Shootdown based lazy tlb mm refcounting means we + * have to IPI everyone in the mm_cpumask anyway soon + * when the mm goes away, so might as well do it as + * part of the final flush now. + * + * If lazy shootdown was improved to reduce IPIs (e.g., + * by batching), then it may end up being better to use + * tlbies here instead. + */ + smp_mb(); /* see radix__flush_tlb_mm */ + exit_flush_lazy_tlbs(mm); + _tlbiel_pid(mm->context.id, RIC_FLUSH_ALL); + + /* + * It should not be possible to have coprocessors still + * attached here. + */ + if (WARN_ON_ONCE(atomic_read(&mm->context.copros) > 0)) + __flush_all_mm(mm, true); + } else { + __flush_all_mm(mm, true); + } + } else if ( (psize = radix_get_mmu_psize(page_size)) == -1) { if (!tlb->freed_tables) radix__flush_tlb_mm(mm);