From patchwork Fri Apr 7 12:56:00 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 748250 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3w00CJ65Tzz9s80 for ; Fri, 7 Apr 2017 23:04:12 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="R+S+Oa2y"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3w00CJ4vJCzDqL8 for ; Fri, 7 Apr 2017 23:04:12 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="R+S+Oa2y"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Received: from mail-pg0-x243.google.com (mail-pg0-x243.google.com [IPv6:2607:f8b0:400e:c05::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3w002M1pbdzDqKv for ; Fri, 7 Apr 2017 22:56:27 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="R+S+Oa2y"; dkim-atps=neutral Received: by mail-pg0-x243.google.com with SMTP id g2so15399806pge.2 for ; Fri, 07 Apr 2017 05:56:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=5rr7vkTBwvWsxVy8t28QICQyo5Xpz5U5jUk4tJvHRp0=; b=R+S+Oa2y7Jayd9eoIuy1Ix2tpE3nvrEzeR2eRpDubBl/QkJKYDcmYa3Hzy+7SzpCwn Be5ZBjnYvconU5bIQMJYD4DnRR7C11BSt21UXTpy9Xm+IJThyFk1etjQE31H0bh639IU KnZ6cvC/tLFeGshLWB7BFHb3d+ER2E6X9WH1KVYTld8hA75ae3UJg0odkJDK1D/eLS8w 4Z+0/Lcx9vj5+Blhq6MY4256RAETbeEdrJKODB2x+APaStGEhxF6DUMWZ8FN76W6WhVn ZAaXskdYUctiXFxfamPz3sskayksczk6thnr+aSIQUDwknueU74/I79rKnHXBCrZXAJ3 rIiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=5rr7vkTBwvWsxVy8t28QICQyo5Xpz5U5jUk4tJvHRp0=; b=UUdjy7lEpWkHs2BBD+eg54uoyYFT5xQobEJKu/zoP71ebRmxNJky4AAz5cIOzReHWT u5+CbhZipVWpF3ewRNX9p8I/qgqejz58ucSM6eG0TCnXNoL9xre0TPQCAg2rbWPh04xn hIIAjcmePbw6cPJUE9pcctTLqdx895W37Q0vrq1EUudWAYP6QpXOpak7YzY8ycYeE7bW xJcJMwHSavlgmOJ7HwV5QUSmf9l0NHP/TM3GZSQW4h5HqIs/B8sjms2vUR7iGMbX7cnY DdTOeIao5TcycbLS2Ji/ZteESCWFd2tV6B6+aEsc+cDnI1pFDfxTF+z/1en4ZW9eepBW ubnA== X-Gm-Message-State: AFeK/H2RaU9ieYnOTvgRBWjugXV+XF/0QQyknZvPjgk6hs/mliAPqRVpXtB3zcenf0jE0w== X-Received: by 10.99.223.70 with SMTP id h6mr42015261pgj.50.1491569784633; Fri, 07 Apr 2017 05:56:24 -0700 (PDT) Received: from roar.au.ibm.com ([203.221.48.234]) by smtp.gmail.com with ESMTPSA id y6sm9715663pgc.40.2017.04.07.05.56.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 07 Apr 2017 05:56:23 -0700 (PDT) From: Nicholas Piggin To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH 3/5] powerpc: Introduce msgsnd/doorbell barrier primitives Date: Fri, 7 Apr 2017 22:56:00 +1000 Message-Id: <20170407125602.31146-4-npiggin@gmail.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20170407125602.31146-1-npiggin@gmail.com> References: <20170407125602.31146-1-npiggin@gmail.com> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Nicholas Piggin Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" POWER9 changes requirements and adds new instructions for synchronization. Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/dbell.h | 22 ++++++++++++++++++++++ arch/powerpc/include/asm/smp.h | 1 + arch/powerpc/kernel/dbell.c | 8 +++++--- arch/powerpc/kernel/smp.c | 10 ++++++++-- 4 files changed, 36 insertions(+), 5 deletions(-) diff --git a/arch/powerpc/include/asm/dbell.h b/arch/powerpc/include/asm/dbell.h index 5a7301f333a4..4db4cfdd829c 100644 --- a/arch/powerpc/include/asm/dbell.h +++ b/arch/powerpc/include/asm/dbell.h @@ -44,6 +44,17 @@ static inline void _ppc_msgsnd(u32 msg) __asm__ __volatile__ (PPC_MSGSNDP(%0) : : "r" (msg)); } +/* sync before sending message */ +static inline void ppc_msgsnd_sync(void) +{ + __asm__ __volatile__ ("sync" : : : "memory"); +} + +/* sync after taking message interrupt */ +static inline void ppc_msgsync(void) +{ +} + #else /* CONFIG_PPC_BOOK3S */ #define PPC_DBELL_MSGTYPE PPC_DBELL @@ -53,6 +64,17 @@ static inline void _ppc_msgsnd(u32 msg) __asm__ __volatile__ (PPC_MSGSND(%0) : : "r" (msg)); } +/* sync before sending message */ +static inline void ppc_msgsnd_sync(void) +{ + __asm__ __volatile__ ("sync" : : : "memory"); +} + +/* sync after taking message interrupt */ +static inline void ppc_msgsync(void) +{ +} + #endif /* CONFIG_PPC_BOOK3S */ extern void global_doorbell_cause_ipi(int cpu); diff --git a/arch/powerpc/include/asm/smp.h b/arch/powerpc/include/asm/smp.h index 0ada12e61fd7..0ee8a6cb1d87 100644 --- a/arch/powerpc/include/asm/smp.h +++ b/arch/powerpc/include/asm/smp.h @@ -128,6 +128,7 @@ extern const char *smp_ipi_name[]; extern void smp_muxed_ipi_message_pass(int cpu, int msg); extern void smp_muxed_ipi_set_message(int cpu, int msg); extern irqreturn_t smp_ipi_demux(void); +extern irqreturn_t smp_ipi_demux_relaxed(void); void smp_init_pSeries(void); void smp_init_cell(void); diff --git a/arch/powerpc/kernel/dbell.c b/arch/powerpc/kernel/dbell.c index 2b41f145de05..98de6d3d904e 100644 --- a/arch/powerpc/kernel/dbell.c +++ b/arch/powerpc/kernel/dbell.c @@ -38,7 +38,7 @@ void global_doorbell_cause_ipi(int cpu) kvmppc_set_host_ipi(cpu, 1); /* Order previous accesses vs. msgsnd, which is treated as a store */ - mb(); + ppc_msgsnd_sync(); ppc_msgsnd(PPC_DBELL_MSGTYPE, 0, tag); } @@ -53,7 +53,7 @@ void core_doorbell_cause_ipi(int cpu) kvmppc_set_host_ipi(cpu, 1); /* Order previous accesses vs. msgsnd, which is treated as a store */ - mb(); + ppc_msgsnd_sync(); ppc_msgsnd(PPC_DBELL_MSGTYPE, 0, tag); } @@ -82,12 +82,14 @@ void doorbell_exception(struct pt_regs *regs) irq_enter(); + ppc_msgsync(); + may_hard_irq_enable(); kvmppc_set_host_ipi(smp_processor_id(), 0); __this_cpu_inc(irq_stat.doorbell_irqs); - smp_ipi_demux(); + smp_ipi_demux_relaxed(); /* already performed the barrier */ irq_exit(); set_irq_regs(old_regs); diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c index f1f6e4e3906b..fd2441591b81 100644 --- a/arch/powerpc/kernel/smp.c +++ b/arch/powerpc/kernel/smp.c @@ -246,11 +246,17 @@ void smp_muxed_ipi_message_pass(int cpu, int msg) irqreturn_t smp_ipi_demux(void) { + mb(); /* order any irq clear */ + + return smp_ipi_demux_relaxed(); +} + +/* sync-free variant. Callers should ensure synchronization */ +irqreturn_t smp_ipi_demux_relaxed(void) +{ struct cpu_messages *info; unsigned long all; - mb(); /* order any irq clear */ - info = this_cpu_ptr(&ipi_message); do { all = xchg(&info->messages, 0);