From patchwork Thu Sep 28 20:21:27 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathieu Desnoyers X-Patchwork-Id: 819742 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3y35w35xGkz9sRg for ; Fri, 29 Sep 2017 06:32:19 +1000 (AEST) Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3y35w3553czDsQd for ; Fri, 29 Sep 2017 06:32:19 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=efficios.com (client-ip=167.114.142.141; helo=mail.efficios.com; envelope-from=mathieu.desnoyers@efficios.com; receiver=) X-Greylist: delayed 565 seconds by postgrey-1.36 at bilbo; Fri, 29 Sep 2017 06:31:18 AEST Received: from mail.efficios.com (mail.efficios.com [167.114.142.141]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3y35tt1TKNzDqNm for ; Fri, 29 Sep 2017 06:31:17 +1000 (AEST) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id 426BC3403CC; Thu, 28 Sep 2017 20:23:04 +0000 (UTC) Received: from mail.efficios.com ([127.0.0.1]) by localhost (evm-mail-1.efficios.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id TudG61WZhG6D; Thu, 28 Sep 2017 20:22:49 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.efficios.com (Postfix) with ESMTP id B43A93401E9; Thu, 28 Sep 2017 20:22:49 +0000 (UTC) X-Virus-Scanned: amavisd-new at efficios.com Received: from mail.efficios.com ([127.0.0.1]) by localhost (evm-mail-1.efficios.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id i6rITmHD7lT5; Thu, 28 Sep 2017 20:22:49 +0000 (UTC) Received: from thinkos.internal.efficios.com (cable-192.222.218.157.electronicbox.net [192.222.218.157]) by mail.efficios.com (Postfix) with ESMTPSA id 5994A3400F9; Thu, 28 Sep 2017 20:22:49 +0000 (UTC) From: Mathieu Desnoyers To: "Paul E . McKenney" , Peter Zijlstra , Nicholas Piggin Subject: [RFC PATCH for 4.14] membarrier powerpc: Move hook to switch_mm() Date: Thu, 28 Sep 2017 16:21:27 -0400 Message-Id: <20170928202127.18472-1-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.11.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.24 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Avi Kivity , Maged Michael , Boqun Feng , Dave Watson , Alan Stern , linux-kernel@vger.kernel.org, Will Deacon , Andrew Hunter , Ingo Molnar , Mathieu Desnoyers , Alexander Viro , Andy Lutomirski , Paul Mackerras , linuxppc-dev@lists.ozlabs.org, gromer@google.com Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Nick has a valid point that the sched_in() hook is a fast-path compared to switch_mm(). Adding an extra TIF test in a fast-path to save a barrier in a comparatively slow-path is therefore not such a good idea overall. Therefore, move the architecture hook to switch_mm() instead. [ This patch is aimed at Paul's tree. It applies on top of "membarrier: Provide register expedited private command (v4)" and "membarrier: Document scheduler barrier requirements (v4)". ] Signed-off-by: Mathieu Desnoyers CC: Peter Zijlstra CC: Paul E. McKenney CC: Nicholas Piggin CC: Boqun Feng CC: Andrew Hunter CC: Maged Michael CC: gromer@google.com CC: Avi Kivity CC: Benjamin Herrenschmidt CC: Paul Mackerras CC: Michael Ellerman CC: Dave Watson CC: Alan Stern CC: Will Deacon CC: Andy Lutomirski CC: Ingo Molnar CC: Alexander Viro CC: linuxppc-dev@lists.ozlabs.org CC: linux-arch@vger.kernel.org --- arch/powerpc/include/asm/membarrier.h | 9 ++++----- arch/powerpc/mm/mmu_context.c | 7 +++++++ include/linux/sched/mm.h | 13 ++++--------- kernel/sched/core.c | 6 ++---- 4 files changed, 17 insertions(+), 18 deletions(-) diff --git a/arch/powerpc/include/asm/membarrier.h b/arch/powerpc/include/asm/membarrier.h index 588154c1cf57..61152a7a3cf9 100644 --- a/arch/powerpc/include/asm/membarrier.h +++ b/arch/powerpc/include/asm/membarrier.h @@ -1,8 +1,8 @@ #ifndef _ASM_POWERPC_MEMBARRIER_H #define _ASM_POWERPC_MEMBARRIER_H -static inline void membarrier_arch_sched_in(struct task_struct *prev, - struct task_struct *next) +static inline void membarrier_arch_switch_mm(struct mm_struct *prev, + struct mm_struct *next, struct task_struct *tsk) { /* * Only need the full barrier when switching between processes. @@ -11,9 +11,8 @@ static inline void membarrier_arch_sched_in(struct task_struct *prev, * when switching from userspace to kernel is not needed after * store to rq->curr. */ - if (likely(!test_ti_thread_flag(task_thread_info(next), - TIF_MEMBARRIER_PRIVATE_EXPEDITED) - || !prev->mm || prev->mm == next->mm)) + if (likely(!test_ti_thread_flag(task_thread_info(tsk), + TIF_MEMBARRIER_PRIVATE_EXPEDITED) || !prev)) return; /* diff --git a/arch/powerpc/mm/mmu_context.c b/arch/powerpc/mm/mmu_context.c index 0f613bc63c50..22f5c91cdc38 100644 --- a/arch/powerpc/mm/mmu_context.c +++ b/arch/powerpc/mm/mmu_context.c @@ -12,6 +12,7 @@ #include #include +#include #include @@ -67,6 +68,10 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, * * On the read side the barrier is in pte_xchg(), which orders * the store to the PTE vs the load of mm_cpumask. + * + * This full barrier is needed by membarrier when switching + * between processes after store to rq->curr, before user-space + * memory accesses. */ smp_mb(); @@ -89,6 +94,8 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, if (new_on_cpu) radix_kvm_prefetch_workaround(next); + else + membarrier_arch_switch_mm(prev, next, tsk); /* * The actual HW switching method differs between the various diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index 1bd10c2c0893..d5a9ab8f3836 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -215,8 +215,8 @@ static inline void memalloc_noreclaim_restore(unsigned int flags) #ifdef CONFIG_ARCH_HAS_MEMBARRIER_HOOKS #include #else -static inline void membarrier_arch_sched_in(struct task_struct *prev, - struct task_struct *next) +static inline void membarrier_arch_switch_mm(struct mm_struct *prev, + struct mm_struct *next, struct task_struct *tsk) { } static inline void membarrier_arch_fork(struct task_struct *t, @@ -232,11 +232,6 @@ static inline void membarrier_arch_register_private_expedited( } #endif -static inline void membarrier_sched_in(struct task_struct *prev, - struct task_struct *next) -{ - membarrier_arch_sched_in(prev, next); -} static inline void membarrier_fork(struct task_struct *t, unsigned long clone_flags) { @@ -252,8 +247,8 @@ static inline void membarrier_execve(struct task_struct *t) membarrier_arch_execve(t); } #else -static inline void membarrier_sched_in(struct task_struct *prev, - struct task_struct *next) +static inline void membarrier_arch_switch_mm(struct mm_struct *prev, + struct mm_struct *next, struct task_struct *tsk) { } static inline void membarrier_fork(struct task_struct *t, diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 6254f87645de..8585abebe32b 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -2643,7 +2643,6 @@ static struct rq *finish_task_switch(struct task_struct *prev) */ prev_state = prev->state; vtime_task_switch(prev); - membarrier_sched_in(prev, current); perf_event_task_sched_in(prev, current); finish_lock_switch(rq, prev); finish_arch_post_lock_switch(); @@ -3357,13 +3356,12 @@ static void __sched notrace __schedule(bool preempt) * * Here are the schemes providing that barrier on the * various architectures: - * - mm ? switch_mm() : mmdrop() for x86, s390, sparc, + * - mm ? switch_mm() : mmdrop() for x86, s390, sparc, PowerPC. + * switch_mm() rely on membarrier_arch_switch_mm() on PowerPC. * - finish_lock_switch() for weakly-ordered * architectures where spin_unlock is a full barrier, * - switch_to() for arm64 (weakly-ordered, spin_unlock * is a RELEASE barrier), - * - membarrier_arch_sched_in() for PowerPC, - * (weakly-ordered, spin_unlock is a RELEASE barrier). */ ++*switch_count;