From patchwork Mon Oct 10 00:29:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Gray X-Patchwork-Id: 1687879 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=jUshfk4W; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Mm0LH38d6z1yqk for ; Mon, 10 Oct 2022 11:35:03 +1100 (AEDT) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Mm0LH0z51z3f0y for ; Mon, 10 Oct 2022 11:35:03 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=jUshfk4W; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.158.5; helo=mx0b-001b2d01.pphosted.com; envelope-from=bgray@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=jUshfk4W; dkim-atps=neutral Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Mm0Fk4TN1z3bjc for ; Mon, 10 Oct 2022 11:31:05 +1100 (AEDT) Received: from pps.filterd (m0098417.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 29A0IEEv025523; Mon, 10 Oct 2022 00:30:39 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=Bcrzk33vNaa3RpMYHgLPkFNg9qHzXppyI/lvXZeUCDI=; b=jUshfk4WUSuiGTBVmjgrrZcUheO5805uvpb5LFjlDmZgV9ocNExYqLnOJ13btU47dQxX wb6cTfkIGvJhUb1Chsz2hQa8+W4Q4qpSpxzWO6a6NZwIL3RPzmTkYcRTCa+9s0bl2lHU bar/ghqFhpD++ltVlZ/E3wEPmsUsRFZ4uc4VG3aFvCbzUAKLYwUsGTTq0B0EuzMZ4DT6 eknEye7TxZpjI7Tmj0d+Tz1hXK0kU1q57IgpfwBwe+3jNk1Co4oXo+hYJ7ap8EJD9aA1 mQIRpOlcAAm6R6kdmjmoI700s4Mp2NITjusEI6uA7DYcWEkHYyhWXWpMRNvrPcrrmAmU Fg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3k3k7uv96y-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Oct 2022 00:30:38 +0000 Received: from m0098417.ppops.net (m0098417.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 29A0TH1B015688; Mon, 10 Oct 2022 00:30:38 GMT Received: from ppma04ams.nl.ibm.com (63.31.33a9.ip4.static.sl-reverse.com [169.51.49.99]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3k3k7uv967-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Oct 2022 00:30:38 +0000 Received: from pps.filterd (ppma04ams.nl.ibm.com [127.0.0.1]) by ppma04ams.nl.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 29A0KriH028930; Mon, 10 Oct 2022 00:30:36 GMT Received: from b06cxnps3074.portsmouth.uk.ibm.com (d06relay09.portsmouth.uk.ibm.com [9.149.109.194]) by ppma04ams.nl.ibm.com with ESMTP id 3k30u9a0v6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Oct 2022 00:30:36 +0000 Received: from d06av26.portsmouth.uk.ibm.com (d06av26.portsmouth.uk.ibm.com [9.149.105.62]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 29A0UYQZ7209574 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 10 Oct 2022 00:30:34 GMT Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0E40CAE053; Mon, 10 Oct 2022 00:30:34 +0000 (GMT) Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 62143AE04D; Mon, 10 Oct 2022 00:30:33 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av26.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 10 Oct 2022 00:30:33 +0000 (GMT) Received: from li-0d7fa1cc-2c9d-11b2-a85c-aed20764436d.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 85CFB6060A; Mon, 10 Oct 2022 11:30:30 +1100 (AEDT) From: Benjamin Gray To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v4 1/6] powerpc/code-patching: Implement generic text patching function Date: Mon, 10 Oct 2022 11:29:52 +1100 Message-Id: <20221010002957.128276-2-bgray@linux.ibm.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221010002957.128276-1-bgray@linux.ibm.com> References: <20221010002957.128276-1-bgray@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: OZLUW2jWMCycRAedqStLRZwMN0LCRvmR X-Proofpoint-GUID: S363xOpEY4sbWMn3U1znqkaJMtsWZz_U X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1 definitions=2022-10-07_04,2022-10-07_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 lowpriorityscore=0 phishscore=0 adultscore=0 bulkscore=0 mlxlogscore=770 mlxscore=0 priorityscore=1501 suspectscore=0 impostorscore=0 clxscore=1015 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2209130000 definitions=main-2210090155 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: ajd@linux.ibm.com, peterz@infradead.org, Benjamin Gray , npiggin@gmail.com, ardb@kernel.org, jbaron@akamai.com, rostedt@goodmis.org, jpoimboe@kernel.org Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Adds a generic memory patching mechanism for patches of size int or long bytes. The patch_instruction function is reimplemented in terms of this more generic function. This generic implementation allows patching of arbitrary long data, such as pointers on 64-bit. As a performance optimisation the patch will unconditionally flush the icache, as patch_instruction is used significantly more often and in more performance sensitive contexts (e.g., ftrace activation). On 32-bit patch_int is marked noinline to prevent a mis-optimisation. Without noinline, inside patch_branch the compiler may inline all the way to do_patch_memory, preventing the compiler from inlining do_patch_memory into patch_int. This would needlessly force patch_int to be a branch to do_patch_memory. The 'IS_ENABLED(CONFIG_PPC64) && ' part of the patch size condition helps GCC inline __patch_memory properly. Only 64-bit requires doubleword writes anyway, as ints and longs are the same size on 32-bit. Signed-off-by: Benjamin Gray Reviewed-by: Andrew Donnellan --- arch/powerpc/include/asm/code-patching.h | 29 ++++++++++ arch/powerpc/lib/code-patching.c | 73 ++++++++++++++++++------ 2 files changed, 85 insertions(+), 17 deletions(-) diff --git a/arch/powerpc/include/asm/code-patching.h b/arch/powerpc/include/asm/code-patching.h index 3f881548fb61..170bfa848c7c 100644 --- a/arch/powerpc/include/asm/code-patching.h +++ b/arch/powerpc/include/asm/code-patching.h @@ -72,7 +72,36 @@ static inline int create_branch(ppc_inst_t *instr, const u32 *addr, int create_cond_branch(ppc_inst_t *instr, const u32 *addr, unsigned long target, int flags); int patch_branch(u32 *addr, unsigned long target, int flags); + +/* patch_uint and patch_ulong must only be called on addresses where the patch + * does not cross a cacheline, otherwise it may not be flushed properly and + * mixes of new and stale data may be observed. + * + * patch_instruction and other instruction patchers automatically satisfy this + * requirement due to instruction alignment requirements. + */ + +int patch_uint(void *addr, unsigned int val); + +#ifdef CONFIG_PPC64 + +int patch_ulong(void *addr, unsigned long val); int patch_instruction(u32 *addr, ppc_inst_t instr); + +#else + +static inline int patch_ulong(void *addr, unsigned long val) +{ + return patch_uint(addr, val); +} + +static inline int patch_instruction(u32 *addr, ppc_inst_t instr) +{ + return patch_uint(addr, ppc_inst_val(instr)); +} + +#endif + int raw_patch_instruction(u32 *addr, ppc_inst_t instr); static inline unsigned long patch_site_addr(s32 *site) diff --git a/arch/powerpc/lib/code-patching.c b/arch/powerpc/lib/code-patching.c index 125c55e3e148..e41c0ccec79f 100644 --- a/arch/powerpc/lib/code-patching.c +++ b/arch/powerpc/lib/code-patching.c @@ -15,20 +15,24 @@ #include #include -static int __patch_instruction(u32 *exec_addr, ppc_inst_t instr, u32 *patch_addr) +static int __patch_memory(void *patch_addr, unsigned long val, void *exec_addr, + bool is_dword) { - if (!ppc_inst_prefixed(instr)) { - u32 val = ppc_inst_val(instr); - - __put_kernel_nofault(patch_addr, &val, u32, failed); - } else { - u64 val = ppc_inst_as_ulong(instr); + /* Prefixed instruction may cross cacheline if cacheline smaller than 64 bytes */ + BUILD_BUG_ON(IS_ENABLED(CONFIG_PPC64) && L1_CACHE_BYTES < 64); + if (IS_ENABLED(CONFIG_PPC64) && unlikely(is_dword)) __put_kernel_nofault(patch_addr, &val, u64, failed); - } + else + __put_kernel_nofault(patch_addr, &val, u32, failed); - asm ("dcbst 0, %0; sync; icbi 0,%1; sync; isync" :: "r" (patch_addr), - "r" (exec_addr)); + /* Assume data is inside a single cacheline */ + dcbst(patch_addr); + mb(); /* sync */ + /* Flush on the EA that may be executed in case of a non-coherent icache */ + icbi(exec_addr); + mb(); /* sync */ + isync(); return 0; @@ -38,7 +42,10 @@ static int __patch_instruction(u32 *exec_addr, ppc_inst_t instr, u32 *patch_addr int raw_patch_instruction(u32 *addr, ppc_inst_t instr) { - return __patch_instruction(addr, instr, addr); + if (ppc_inst_prefixed(instr)) + return __patch_memory(addr, ppc_inst_as_ulong(instr), addr, true); + else + return __patch_memory(addr, ppc_inst_val(instr), addr, false); } static DEFINE_PER_CPU(struct vm_struct *, text_poke_area); @@ -149,7 +156,7 @@ static void unmap_patch_area(unsigned long addr) flush_tlb_kernel_range(addr, addr + PAGE_SIZE); } -static int __do_patch_instruction(u32 *addr, ppc_inst_t instr) +static int __do_patch_memory(void *addr, unsigned long val, bool is_dword) { int err; u32 *patch_addr; @@ -166,7 +173,7 @@ static int __do_patch_instruction(u32 *addr, ppc_inst_t instr) if (radix_enabled()) asm volatile("ptesync": : :"memory"); - err = __patch_instruction(addr, instr, patch_addr); + err = __patch_memory(patch_addr, val, addr, is_dword); pte_clear(&init_mm, text_poke_addr, pte); flush_tlb_kernel_range(text_poke_addr, text_poke_addr + PAGE_SIZE); @@ -174,7 +181,7 @@ static int __do_patch_instruction(u32 *addr, ppc_inst_t instr) return err; } -int patch_instruction(u32 *addr, ppc_inst_t instr) +static int do_patch_memory(void *addr, unsigned long val, bool is_dword) { int err; unsigned long flags; @@ -186,15 +193,47 @@ int patch_instruction(u32 *addr, ppc_inst_t instr) */ if (!IS_ENABLED(CONFIG_STRICT_KERNEL_RWX) || !static_branch_likely(&poking_init_done)) - return raw_patch_instruction(addr, instr); + return __patch_memory(addr, val, addr, is_dword); local_irq_save(flags); - err = __do_patch_instruction(addr, instr); + err = __do_patch_memory(addr, val, is_dword); local_irq_restore(flags); return err; } -NOKPROBE_SYMBOL(patch_instruction); + +#ifdef CONFIG_PPC64 + +int patch_uint(void *addr, unsigned int val) +{ + return do_patch_memory(addr, val, false); +} +NOKPROBE_SYMBOL(patch_uint) + +int patch_ulong(void *addr, unsigned long val) +{ + return do_patch_memory(addr, val, true); +} +NOKPROBE_SYMBOL(patch_ulong) + +int patch_instruction(u32 *addr, ppc_inst_t instr) +{ + if (ppc_inst_prefixed(instr)) + return patch_ulong(addr, ppc_inst_as_ulong(instr)); + else + return patch_uint(addr, ppc_inst_val(instr)); +} +NOKPROBE_SYMBOL(patch_instruction) + +#else + +noinline int patch_uint(void *addr, unsigned int val) +{ + return do_patch_memory(addr, val, false); +} +NOKPROBE_SYMBOL(patch_uint) + +#endif int patch_branch(u32 *addr, unsigned long target, int flags) { From patchwork Mon Oct 10 00:29:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Gray X-Patchwork-Id: 1687882 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=VSczzPYz; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Mm0NM3yBmz20cX for ; Mon, 10 Oct 2022 11:36:51 +1100 (AEDT) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Mm0NM1MQsz3f1D for ; Mon, 10 Oct 2022 11:36:51 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=VSczzPYz; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=bgray@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=VSczzPYz; dkim-atps=neutral Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Mm0Fl0p1Zz3bjd for ; Mon, 10 Oct 2022 11:31:06 +1100 (AEDT) Received: from pps.filterd (m0098420.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 299NCBcZ003082; Mon, 10 Oct 2022 00:30:39 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=7AhaorhNNdVupXboT0plDfNanOhraIGnaqGo6BRjnUY=; b=VSczzPYzPdQDBX0ret01hSMpgtfbN+aLWJ/RzlPMOdsth0T37dQ2tDsM0u1yJ4UqmJkD VlRorJehdbDceicD77vlItN7eZpt13BiiDQ7XkoFG/v0MVEgJe1f5Yjq/TufHCExAxhs QUUZBL3T7QkNxf4dlbDTkiNmAdB39LunrqzilIHYrD4b0YohbMIoCRPmpdlJ15iLPXmR sZgJbPRj07f1JRrtOAomIhRj0VZQI+uQPMctErQnpofdBmyAd3IPlVTNi0jm2JyrfRZs gqmaGwPUMpCUzih1a0zD3F71lq/qF9lxI6orgIWzLMQzE3Zp65n/Crn7d/98kS+vP5d7 VQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com (PPS) with ESMTPS id 3k3k97v8ry-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Oct 2022 00:30:38 +0000 Received: from m0098420.ppops.net (m0098420.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 29A0UckV013314; Mon, 10 Oct 2022 00:30:38 GMT Received: from ppma03ams.nl.ibm.com (62.31.33a9.ip4.static.sl-reverse.com [169.51.49.98]) by mx0b-001b2d01.pphosted.com (PPS) with ESMTPS id 3k3k97v8rq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Oct 2022 00:30:38 +0000 Received: from pps.filterd (ppma03ams.nl.ibm.com [127.0.0.1]) by ppma03ams.nl.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 29A0MulQ010144; Mon, 10 Oct 2022 00:30:36 GMT Received: from b06avi18878370.portsmouth.uk.ibm.com (b06avi18878370.portsmouth.uk.ibm.com [9.149.26.194]) by ppma03ams.nl.ibm.com with ESMTP id 3k30u920xe-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Oct 2022 00:30:36 +0000 Received: from d06av24.portsmouth.uk.ibm.com (mk.ibm.com [9.149.105.60]) by b06avi18878370.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 29A0V4fN50528766 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 10 Oct 2022 00:31:04 GMT Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B990C4203F; Mon, 10 Oct 2022 00:30:33 +0000 (GMT) Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 61F9C42042; Mon, 10 Oct 2022 00:30:33 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av24.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 10 Oct 2022 00:30:33 +0000 (GMT) Received: from li-0d7fa1cc-2c9d-11b2-a85c-aed20764436d.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 89362602EB; Mon, 10 Oct 2022 11:30:30 +1100 (AEDT) From: Benjamin Gray To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v4 2/6] powerpc/module: Handle caller-saved TOC in module linker Date: Mon, 10 Oct 2022 11:29:53 +1100 Message-Id: <20221010002957.128276-3-bgray@linux.ibm.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221010002957.128276-1-bgray@linux.ibm.com> References: <20221010002957.128276-1-bgray@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: e-M2Y98EpjG7R0c3uQVwR3vxgA2mY8iN X-Proofpoint-ORIG-GUID: qZh5qYUjhi0SKznWzlPxiLBqCSZVVJth X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1 definitions=2022-10-07_04,2022-10-07_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 spamscore=0 mlxscore=0 suspectscore=0 clxscore=1015 lowpriorityscore=0 phishscore=0 priorityscore=1501 malwarescore=0 mlxlogscore=999 bulkscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2209130000 definitions=main-2210090155 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: ajd@linux.ibm.com, peterz@infradead.org, Benjamin Gray , npiggin@gmail.com, ardb@kernel.org, jbaron@akamai.com, rostedt@goodmis.org, jpoimboe@kernel.org Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" A function symbol may set a value in the st_other field to indicate the TOC should be treated as caller-saved. The linker should ensure the current TOC is saved before calling it and restore the TOC afterwards, much like external calls. This is necessary for supporting V2 ABI static calls that do not preserve the TOC. Signed-off-by: Benjamin Gray Reviewed-by: Andrew Donnellan --- arch/powerpc/kernel/module_64.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/arch/powerpc/kernel/module_64.c b/arch/powerpc/kernel/module_64.c index 7e45dc98df8a..83a6f6e22e3b 100644 --- a/arch/powerpc/kernel/module_64.c +++ b/arch/powerpc/kernel/module_64.c @@ -55,6 +55,12 @@ static unsigned int local_entry_offset(const Elf64_Sym *sym) * of function and try to derive r2 from it). */ return PPC64_LOCAL_ENTRY_OFFSET(sym->st_other); } + +static bool need_r2save_stub(unsigned char st_other) +{ + return (st_other & STO_PPC64_LOCAL_MASK) == (1 << STO_PPC64_LOCAL_BIT); +} + #else static func_desc_t func_desc(unsigned long addr) @@ -66,6 +72,11 @@ static unsigned int local_entry_offset(const Elf64_Sym *sym) return 0; } +static bool need_r2save_stub(unsigned char st_other) +{ + return false; +} + void *dereference_module_function_descriptor(struct module *mod, void *ptr) { if (ptr < (void *)mod->arch.start_opd || @@ -632,7 +643,8 @@ int apply_relocate_add(Elf64_Shdr *sechdrs, case R_PPC_REL24: /* FIXME: Handle weak symbols here --RR */ if (sym->st_shndx == SHN_UNDEF || - sym->st_shndx == SHN_LIVEPATCH) { + sym->st_shndx == SHN_LIVEPATCH || + need_r2save_stub(sym->st_other)) { /* External: go via stub */ value = stub_for_addr(sechdrs, value, me, strtab + sym->st_name); From patchwork Mon Oct 10 00:29:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Gray X-Patchwork-Id: 1687877 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=ZvJmGlEv; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Mm0JF0LZGz23jn for ; Mon, 10 Oct 2022 11:33:17 +1100 (AEDT) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Mm0JC1Rxvz3dqT for ; Mon, 10 Oct 2022 11:33:15 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=ZvJmGlEv; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=bgray@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=ZvJmGlEv; dkim-atps=neutral Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Mm0Fh73Gyz2yJQ for ; Mon, 10 Oct 2022 11:31:04 +1100 (AEDT) Received: from pps.filterd (m0098404.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 299Mtueg028542; Mon, 10 Oct 2022 00:30:39 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=D66I4qI/0wf5e+R3yzh7g8S98gPPFwfRQqT/YWWw1LQ=; b=ZvJmGlEvnNPYg9KoVK9pcjjW/YhL2f5lk9w/PyDLDqJfBY3u+Zmz5UVBMP5aY0z4ejZV XuVsClHhvyI84kVEy2GVgvICYVQfVMNfDAgB8MI7/MOPyddqhv+P/QmKIIxtIQUupBPt pP65vv28qNsrcbgjxDaq7t9Kxqf2ZL7TB97VUAdF1eGBxEftfiMJrgYg4vjfVj5UgH9X AfdSdWBMgwIljYQME1rV3P7VfHnk6suk0FBXKcchlEhHIorsYqCPO9EouJL1tDNmvexJ dtYtkdTsuETwyVC7NiMJ+hoR6maqoYE3bzaRkqLCIKlgsNZs2KRbV8BmADLAZAH5Kdbn kw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3k3jhaw2ud-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Oct 2022 00:30:39 +0000 Received: from m0098404.ppops.net (m0098404.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 29A0RoE1021299; Mon, 10 Oct 2022 00:30:39 GMT Received: from ppma06ams.nl.ibm.com (66.31.33a9.ip4.static.sl-reverse.com [169.51.49.102]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3k3jhaw2tx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Oct 2022 00:30:39 +0000 Received: from pps.filterd (ppma06ams.nl.ibm.com [127.0.0.1]) by ppma06ams.nl.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 29A0Kcw0028575; Mon, 10 Oct 2022 00:30:36 GMT Received: from b06avi18626390.portsmouth.uk.ibm.com (b06avi18626390.portsmouth.uk.ibm.com [9.149.26.192]) by ppma06ams.nl.ibm.com with ESMTP id 3k30fja1dy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Oct 2022 00:30:36 +0000 Received: from d06av26.portsmouth.uk.ibm.com (d06av26.portsmouth.uk.ibm.com [9.149.105.62]) by b06avi18626390.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 29A0PrbH38011224 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 10 Oct 2022 00:25:53 GMT Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B82EEAE051; Mon, 10 Oct 2022 00:30:33 +0000 (GMT) Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 61914AE045; Mon, 10 Oct 2022 00:30:33 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av26.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 10 Oct 2022 00:30:33 +0000 (GMT) Received: from li-0d7fa1cc-2c9d-11b2-a85c-aed20764436d.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 8C7436061A; Mon, 10 Oct 2022 11:30:30 +1100 (AEDT) From: Benjamin Gray To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v4 3/6] powerpc/module: Optimise nearby branches in ELF V2 ABI stub Date: Mon, 10 Oct 2022 11:29:54 +1100 Message-Id: <20221010002957.128276-4-bgray@linux.ibm.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221010002957.128276-1-bgray@linux.ibm.com> References: <20221010002957.128276-1-bgray@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: gTGNmI_hm7jDsE21SMvbmRWU97D-tqBW X-Proofpoint-ORIG-GUID: vmUH4B8oHlpKKpVe4caWny2v28q1mvl1 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1 definitions=2022-10-07_04,2022-10-07_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 spamscore=0 phishscore=0 adultscore=0 impostorscore=0 suspectscore=0 clxscore=1015 mlxscore=0 lowpriorityscore=0 malwarescore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2209130000 definitions=main-2210090155 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: ajd@linux.ibm.com, peterz@infradead.org, Benjamin Gray , npiggin@gmail.com, ardb@kernel.org, jbaron@akamai.com, rostedt@goodmis.org, jpoimboe@kernel.org Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Inserts a direct branch to the stub target when possible, replacing the mtctr/btctr sequence. The load into r12 could potentially be skipped too, but that change would need to refactor the arguments to indicate that the address does not have a separate local entry point. This helps the static call implementation, where modules calling their own trampolines are called through this stub and the trampoline is easily within range of a direct branch. Signed-off-by: Benjamin Gray Reviewed-by: Christophe Leroy Reviewed-by: Andrew Donnellan --- arch/powerpc/kernel/module_64.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/arch/powerpc/kernel/module_64.c b/arch/powerpc/kernel/module_64.c index 83a6f6e22e3b..3a90043b006c 100644 --- a/arch/powerpc/kernel/module_64.c +++ b/arch/powerpc/kernel/module_64.c @@ -141,6 +141,12 @@ static u32 ppc64_stub_insns[] = { PPC_RAW_BCTR(), }; +#ifdef CONFIG_PPC64_ELF_ABI_V1 +#define PPC64_STUB_MTCTR_OFFSET 5 +#else +#define PPC64_STUB_MTCTR_OFFSET 4 +#endif + /* Count how many different 24-bit relocations (different symbol, different addend) */ static unsigned int count_relocs(const Elf64_Rela *rela, unsigned int num) @@ -426,6 +432,7 @@ static inline int create_stub(const Elf64_Shdr *sechdrs, struct module *me, const char *name) { + int err; long reladdr; func_desc_t desc; int i; @@ -439,6 +446,11 @@ static inline int create_stub(const Elf64_Shdr *sechdrs, return 0; } + /* Replace indirect branch sequence with direct branch where possible */ + err = patch_branch(&entry->jump[PPC64_STUB_MTCTR_OFFSET], addr, 0); + if (err && err != -ERANGE) + return 0; + /* Stub uses address relative to r2. */ reladdr = (unsigned long)entry - my_r2(sechdrs, me); if (reladdr > 0x7FFFFFFF || reladdr < -(0x80000000L)) { From patchwork Mon Oct 10 00:29:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Gray X-Patchwork-Id: 1687881 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=qBRO1Tya; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Mm0MK2TyQz20cX for ; Mon, 10 Oct 2022 11:35:57 +1100 (AEDT) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Mm0MK1fSYz3drn for ; Mon, 10 Oct 2022 11:35:57 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=qBRO1Tya; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=bgray@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=qBRO1Tya; dkim-atps=neutral Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Mm0Fk41gNz2yJQ for ; Mon, 10 Oct 2022 11:31:05 +1100 (AEDT) Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 299NR9od024580; Mon, 10 Oct 2022 00:30:39 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=6jRzs0vGdfePZHWsxv0FN0NpYcLQI6j7e4PTsb4bLYo=; b=qBRO1Tya/VDRNfdHlH10sur+WfDTgmw79EAiUtKZaxHlaXOpApeco49o+X9OG13qMdq8 m9b1FJ8XRHoexUwkKhvh1p6H0F4mOa8xqY/a+X7Kkko8EyD/6fY6VIYWJqsEzQaFECxQ Af2S4eMIjAyf3ghOX06g/sBmp9kcYp8dIgIGaDfnepaorx6kup2BF99OBQSW7ziagUpk bjuYkn4M3x1QdvKYswCaqPq9wPv5N27/nrfB8u/KtrtEAFul8xP1DFJ4mjxMad7hhfCv 0iqphuUg67glVOEFTYYZYMMOtZ1FrQdZqQiBTza1cnghcOvdqgkDd6vgvxE+1bhScrSQ ag== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com (PPS) with ESMTPS id 3k3ju6mh1c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Oct 2022 00:30:39 +0000 Received: from m0098419.ppops.net (m0098419.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 29A0PQ7G030363; Mon, 10 Oct 2022 00:30:38 GMT Received: from ppma04ams.nl.ibm.com (63.31.33a9.ip4.static.sl-reverse.com [169.51.49.99]) by mx0b-001b2d01.pphosted.com (PPS) with ESMTPS id 3k3ju6mh0t-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Oct 2022 00:30:38 +0000 Received: from pps.filterd (ppma04ams.nl.ibm.com [127.0.0.1]) by ppma04ams.nl.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 29A0L3lu028961; Mon, 10 Oct 2022 00:30:36 GMT Received: from b06avi18878370.portsmouth.uk.ibm.com (b06avi18878370.portsmouth.uk.ibm.com [9.149.26.194]) by ppma04ams.nl.ibm.com with ESMTP id 3k30u9a0v5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Oct 2022 00:30:36 +0000 Received: from d06av25.portsmouth.uk.ibm.com (d06av25.portsmouth.uk.ibm.com [9.149.105.61]) by b06avi18878370.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 29A0V4G850528514 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 10 Oct 2022 00:31:04 GMT Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0E41011C04C; Mon, 10 Oct 2022 00:30:34 +0000 (GMT) Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6188D11C04A; Mon, 10 Oct 2022 00:30:33 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av25.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 10 Oct 2022 00:30:33 +0000 (GMT) Received: from li-0d7fa1cc-2c9d-11b2-a85c-aed20764436d.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 8FC3E60636; Mon, 10 Oct 2022 11:30:30 +1100 (AEDT) From: Benjamin Gray To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v4 4/6] static_call: Move static call selftest to static_call_selftest.c Date: Mon, 10 Oct 2022 11:29:55 +1100 Message-Id: <20221010002957.128276-5-bgray@linux.ibm.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221010002957.128276-1-bgray@linux.ibm.com> References: <20221010002957.128276-1-bgray@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: sdm0PKn4d8zhkL0Gw8dM4TwIPNesaBcC X-Proofpoint-ORIG-GUID: U98A-TiVgUbVlb9GnXACdad7oFwkPU5d X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1 definitions=2022-10-07_04,2022-10-07_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 adultscore=0 lowpriorityscore=0 spamscore=0 bulkscore=0 priorityscore=1501 clxscore=1015 mlxscore=0 suspectscore=0 malwarescore=0 impostorscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2209130000 definitions=main-2210090155 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: ajd@linux.ibm.com, peterz@infradead.org, Benjamin Gray , npiggin@gmail.com, ardb@kernel.org, jbaron@akamai.com, rostedt@goodmis.org, jpoimboe@kernel.org Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" These tests are out-of-line only, so moving them to the their own file allows them to be run when an arch does not implement inline static calls. Signed-off-by: Benjamin Gray Reviewed-by: Andrew Donnellan Reviewed-by: Christophe Leroy --- kernel/Makefile | 1 + kernel/static_call_inline.c | 43 ----------------------------------- kernel/static_call_selftest.c | 41 +++++++++++++++++++++++++++++++++ 3 files changed, 42 insertions(+), 43 deletions(-) create mode 100644 kernel/static_call_selftest.c diff --git a/kernel/Makefile b/kernel/Makefile index 318789c728d3..8ce8beaa3cc0 100644 --- a/kernel/Makefile +++ b/kernel/Makefile @@ -113,6 +113,7 @@ obj-$(CONFIG_KCSAN) += kcsan/ obj-$(CONFIG_SHADOW_CALL_STACK) += scs.o obj-$(CONFIG_HAVE_STATIC_CALL) += static_call.o obj-$(CONFIG_HAVE_STATIC_CALL_INLINE) += static_call_inline.o +obj-$(CONFIG_STATIC_CALL_SELFTEST) += static_call_selftest.o obj-$(CONFIG_CFI_CLANG) += cfi.o obj-$(CONFIG_PERF_EVENTS) += events/ diff --git a/kernel/static_call_inline.c b/kernel/static_call_inline.c index dc5665b62814..64d04d054698 100644 --- a/kernel/static_call_inline.c +++ b/kernel/static_call_inline.c @@ -498,46 +498,3 @@ int __init static_call_init(void) return 0; } early_initcall(static_call_init); - -#ifdef CONFIG_STATIC_CALL_SELFTEST - -static int func_a(int x) -{ - return x+1; -} - -static int func_b(int x) -{ - return x+2; -} - -DEFINE_STATIC_CALL(sc_selftest, func_a); - -static struct static_call_data { - int (*func)(int); - int val; - int expect; -} static_call_data [] __initdata = { - { NULL, 2, 3 }, - { func_b, 2, 4 }, - { func_a, 2, 3 } -}; - -static int __init test_static_call_init(void) -{ - int i; - - for (i = 0; i < ARRAY_SIZE(static_call_data); i++ ) { - struct static_call_data *scd = &static_call_data[i]; - - if (scd->func) - static_call_update(sc_selftest, scd->func); - - WARN_ON(static_call(sc_selftest)(scd->val) != scd->expect); - } - - return 0; -} -early_initcall(test_static_call_init); - -#endif /* CONFIG_STATIC_CALL_SELFTEST */ diff --git a/kernel/static_call_selftest.c b/kernel/static_call_selftest.c new file mode 100644 index 000000000000..246ad89f64eb --- /dev/null +++ b/kernel/static_call_selftest.c @@ -0,0 +1,41 @@ +// SPDX-License-Identifier: GPL-2.0 +#include + +static int func_a(int x) +{ + return x+1; +} + +static int func_b(int x) +{ + return x+2; +} + +DEFINE_STATIC_CALL(sc_selftest, func_a); + +static struct static_call_data { + int (*func)(int); + int val; + int expect; +} static_call_data [] __initdata = { + { NULL, 2, 3 }, + { func_b, 2, 4 }, + { func_a, 2, 3 } +}; + +static int __init test_static_call_init(void) +{ + int i; + + for (i = 0; i < ARRAY_SIZE(static_call_data); i++ ) { + struct static_call_data *scd = &static_call_data[i]; + + if (scd->func) + static_call_update(sc_selftest, scd->func); + + WARN_ON(static_call(sc_selftest)(scd->val) != scd->expect); + } + + return 0; +} +early_initcall(test_static_call_init); From patchwork Mon Oct 10 00:29:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Gray X-Patchwork-Id: 1687878 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=sF7s2bkT; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Mm0KF2cbBz23jn for ; Mon, 10 Oct 2022 11:34:09 +1100 (AEDT) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Mm0KF1nTpz3cfX for ; Mon, 10 Oct 2022 11:34:09 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=sF7s2bkT; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.158.5; helo=mx0b-001b2d01.pphosted.com; envelope-from=bgray@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=sF7s2bkT; dkim-atps=neutral Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Mm0Fj60BPz3bjd for ; Mon, 10 Oct 2022 11:31:05 +1100 (AEDT) Received: from pps.filterd (m0098421.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 29A0ExZ5004768; Mon, 10 Oct 2022 00:30:41 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=2P9CDSl9pVNcUvVvLJ1r2JJq4+Pd5MUTqDgY/Axgw+A=; b=sF7s2bkTo//u3bOMzZTXnGTB2yKIAu7CoE62tr0tPB/53vqhl5ZP7XRByZ0z0wrKJdSZ nSynvbEQkj+Oq+w2BvFvNVR71smpgxUFeQPl9prn3JUTEp3R1/qBIn4E5Zl5DcR2nWZt KacyXOsA+c9HYRLhb/oCKQQowxZYZSX7gT8Fs3RY5NyScLspWcsD/xUvGlEFH6EJHTTb FhkOzxpGlOQ3dE2PZKgA6Q0+AVaDEh1epq9O37uyieNj6f2mfth2Bqs42gil21G4fA2r 4M9Dg4pGmNZAqzsEdkzACwJkBJivCoNKIc43ipwreCxPtretaOFOtnMV+TbijGUnGy3Q Gw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3k3jvkcn8q-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Oct 2022 00:30:41 +0000 Received: from m0098421.ppops.net (m0098421.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 299NtHVF011418; Mon, 10 Oct 2022 00:30:40 GMT Received: from ppma06ams.nl.ibm.com (66.31.33a9.ip4.static.sl-reverse.com [169.51.49.102]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3k3jvkcn84-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Oct 2022 00:30:40 +0000 Received: from pps.filterd (ppma06ams.nl.ibm.com [127.0.0.1]) by ppma06ams.nl.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 29A0MM89029893; Mon, 10 Oct 2022 00:30:38 GMT Received: from b06cxnps3074.portsmouth.uk.ibm.com (d06relay09.portsmouth.uk.ibm.com [9.149.109.194]) by ppma06ams.nl.ibm.com with ESMTP id 3k30fja1e1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Oct 2022 00:30:37 +0000 Received: from d06av24.portsmouth.uk.ibm.com (mk.ibm.com [9.149.105.60]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 29A0UZDw8585898 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 10 Oct 2022 00:30:35 GMT Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9B2C342041; Mon, 10 Oct 2022 00:30:35 +0000 (GMT) Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id EDC6C4203F; Mon, 10 Oct 2022 00:30:34 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av24.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 10 Oct 2022 00:30:34 +0000 (GMT) Received: from li-0d7fa1cc-2c9d-11b2-a85c-aed20764436d.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 932F06060B; Mon, 10 Oct 2022 11:30:30 +1100 (AEDT) From: Benjamin Gray To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v4 5/6] powerpc/64: Add support for out-of-line static calls Date: Mon, 10 Oct 2022 11:29:56 +1100 Message-Id: <20221010002957.128276-6-bgray@linux.ibm.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221010002957.128276-1-bgray@linux.ibm.com> References: <20221010002957.128276-1-bgray@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: w1H1sVbuLKFlTMQ_tvBYrmJmoJG9tx-4 X-Proofpoint-ORIG-GUID: KyuD6gvUVcMcNFw0GDd_vrYiL5cQtqD0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1 definitions=2022-10-07_04,2022-10-07_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 priorityscore=1501 impostorscore=0 mlxscore=0 mlxlogscore=999 adultscore=0 suspectscore=0 bulkscore=0 clxscore=1015 spamscore=0 malwarescore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2209130000 definitions=main-2210090155 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: ajd@linux.ibm.com, peterz@infradead.org, Benjamin Gray , npiggin@gmail.com, ardb@kernel.org, jbaron@akamai.com, rostedt@goodmis.org, jpoimboe@kernel.org Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Implement static call support for 64 bit V2 ABI. This requires making sure the TOC is kept correct across kernel-module boundaries. As a secondary concern, it tries to use the local entry point of a target wherever possible. It does so by checking if both tramp & target are kernel code, and falls back to detecting the common global entry point patterns if modules are involved. Detecting the global entry point is also required for setting the local entry point as the trampoline target: if we cannot detect the local entry point, then we need to convservatively initialise r12 and use the global entry point. The trampolines are marked with `.localentry NAME, 1` to make the linker save and restore the TOC on each call to the trampoline. This allows the trampoline to safely target functions with different TOC values. However this directive also implies the TOC is not initialised on entry to the trampoline. The kernel TOC is easily found in the PACA, but not an arbitrary module TOC. Therefore the trampoline implementation depends on whether it's in the kernel or not. If in the kernel, we initialise the TOC using the PACA. If in a module, we have to initialise the TOC with zero context, so it's quite expensive. Signed-off-by: Benjamin Gray Reviewed-by: Christophe Leroy --- arch/powerpc/Kconfig | 14 ++- arch/powerpc/include/asm/code-patching.h | 1 + arch/powerpc/include/asm/static_call.h | 80 +++++++++++++- arch/powerpc/kernel/Makefile | 3 +- arch/powerpc/kernel/static_call.c | 130 +++++++++++++++++++++-- 5 files changed, 216 insertions(+), 12 deletions(-) diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 4c466acdc70d..962e36ec34ec 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -102,6 +102,18 @@ config GENERIC_HWEIGHT bool default y +config TOOLCHAIN_SUPPORTS_LOCALENTRY1 + bool + depends on PPC64_ELF_ABI_V2 + default y if LD_VERSION >= 23200 || LLD_VERSION >= 110000 + help + A section of the ELF symbol st_other field can be given the value 1 + using the directive '.localentry NAME, 1' to mean the local and global + entry points are the same, and r2 should be treated as caller-saved. + + Older versions of Clang and binutils do not recognise this form of the + directive and will error if it is used. + config PPC bool default y @@ -248,7 +260,7 @@ config PPC select HAVE_SOFTIRQ_ON_OWN_STACK select HAVE_STACKPROTECTOR if PPC32 && $(cc-option,-mstack-protector-guard=tls -mstack-protector-guard-reg=r2) select HAVE_STACKPROTECTOR if PPC64 && $(cc-option,-mstack-protector-guard=tls -mstack-protector-guard-reg=r13) - select HAVE_STATIC_CALL if PPC32 + select HAVE_STATIC_CALL if PPC32 || (PPC64_ELF_ABI_V2 && TOOLCHAIN_SUPPORTS_LOCALENTRY1) select HAVE_SYSCALL_TRACEPOINTS select HAVE_VIRT_CPU_ACCOUNTING select HUGETLB_PAGE_SIZE_VARIABLE if PPC_BOOK3S_64 && HUGETLB_PAGE diff --git a/arch/powerpc/include/asm/code-patching.h b/arch/powerpc/include/asm/code-patching.h index 170bfa848c7c..cb4629e55e57 100644 --- a/arch/powerpc/include/asm/code-patching.h +++ b/arch/powerpc/include/asm/code-patching.h @@ -152,6 +152,7 @@ int translate_branch(ppc_inst_t *instr, const u32 *dest, const u32 *src); bool is_conditional_branch(ppc_inst_t instr); #define OP_RT_RA_MASK 0xffff0000UL +#define OP_SI_MASK 0x0000ffffUL #define LIS_R2 (PPC_RAW_LIS(_R2, 0)) #define ADDIS_R2_R12 (PPC_RAW_ADDIS(_R2, _R12, 0)) #define ADDI_R2_R2 (PPC_RAW_ADDI(_R2, _R2, 0)) diff --git a/arch/powerpc/include/asm/static_call.h b/arch/powerpc/include/asm/static_call.h index de1018cc522b..3d6e82200cb7 100644 --- a/arch/powerpc/include/asm/static_call.h +++ b/arch/powerpc/include/asm/static_call.h @@ -2,12 +2,75 @@ #ifndef _ASM_POWERPC_STATIC_CALL_H #define _ASM_POWERPC_STATIC_CALL_H +#ifdef CONFIG_PPC64_ELF_ABI_V2 + +#ifdef MODULE + +#define __PPC_SCT(name, inst) \ + asm(".pushsection .text, \"ax\" \n" \ + ".align 6 \n" \ + ".globl " STATIC_CALL_TRAMP_STR(name) " \n" \ + ".localentry " STATIC_CALL_TRAMP_STR(name) ", 1 \n" \ + STATIC_CALL_TRAMP_STR(name) ": \n" \ + " mflr 11 \n" \ + " bcl 20, 31, $+4 \n" \ + "0: mflr 12 \n" \ + " mtlr 11 \n" \ + " addi 12, 12, (" STATIC_CALL_TRAMP_STR(name) " - 0b) \n" \ + " addis 2, 12, (.TOC.-" STATIC_CALL_TRAMP_STR(name) ")@ha \n" \ + " addi 2, 2, (.TOC.-" STATIC_CALL_TRAMP_STR(name) ")@l \n" \ + " " inst " \n" \ + " ld 12, (2f - " STATIC_CALL_TRAMP_STR(name) ")(12) \n" \ + " mtctr 12 \n" \ + " bctr \n" \ + "1: li 3, 0 \n" \ + " blr \n" \ + ".balign 8 \n" \ + "2: .8byte 0 \n" \ + ".type " STATIC_CALL_TRAMP_STR(name) ", @function \n" \ + ".size " STATIC_CALL_TRAMP_STR(name) ", . - " STATIC_CALL_TRAMP_STR(name) " \n" \ + ".popsection \n") + +#else /* KERNEL */ + +#define __PPC_SCT(name, inst) \ + asm(".pushsection .text, \"ax\" \n" \ + ".align 5 \n" \ + ".globl " STATIC_CALL_TRAMP_STR(name) " \n" \ + ".localentry " STATIC_CALL_TRAMP_STR(name) ", 1 \n" \ + STATIC_CALL_TRAMP_STR(name) ": \n" \ + " ld 2, 16(13) \n" \ + " " inst " \n" \ + " addis 12, 2, 2f@toc@ha \n" \ + " ld 12, 2f@toc@l(12) \n" \ + " mtctr 12 \n" \ + " bctr \n" \ + "1: li 3, 0 \n" \ + " blr \n" \ + ".balign 8 \n" \ + "2: .8byte 0 \n" \ + ".type " STATIC_CALL_TRAMP_STR(name) ", @function \n" \ + ".size " STATIC_CALL_TRAMP_STR(name) ", . - " STATIC_CALL_TRAMP_STR(name) " \n" \ + ".popsection \n") + +#endif /* MODULE */ + +#define PPC_SCT_INST_MODULE 28 /* Offset of instruction to update */ +#define PPC_SCT_RET0_MODULE 44 /* Offset of label 1 */ +#define PPC_SCT_DATA_MODULE 56 /* Offset of label 2 (aligned) */ + +#define PPC_SCT_INST_KERNEL 4 /* Offset of instruction to update */ +#define PPC_SCT_RET0_KERNEL 24 /* Offset of label 1 */ +#define PPC_SCT_DATA_KERNEL 32 /* Offset of label 2 (aligned) */ + +#elif defined(CONFIG_PPC32) + #define __PPC_SCT(name, inst) \ asm(".pushsection .text, \"ax\" \n" \ ".align 5 \n" \ ".globl " STATIC_CALL_TRAMP_STR(name) " \n" \ STATIC_CALL_TRAMP_STR(name) ": \n" \ - inst " \n" \ + " " inst " \n" \ " lis 12,2f@ha \n" \ " lwz 12,2f@l(12) \n" \ " mtctr 12 \n" \ @@ -19,11 +82,20 @@ ".size " STATIC_CALL_TRAMP_STR(name) ", . - " STATIC_CALL_TRAMP_STR(name) " \n" \ ".popsection \n") -#define PPC_SCT_RET0 20 /* Offset of label 1 */ -#define PPC_SCT_DATA 28 /* Offset of label 2 */ +#define PPC_SCT_INST_MODULE 0 /* Offset of instruction to update */ +#define PPC_SCT_RET0_MODULE 20 /* Offset of label 1 */ +#define PPC_SCT_DATA_MODULE 28 /* Offset of label 2 */ + +#define PPC_SCT_INST_KERNEL PPC_SCT_INST_MODULE +#define PPC_SCT_RET0_KERNEL PPC_SCT_RET0_MODULE +#define PPC_SCT_DATA_KERNEL PPC_SCT_DATA_MODULE + +#else /* !CONFIG_PPC64_ELF_ABI_V2 && !CONFIG_PPC32 */ +#error "Unsupported ABI" +#endif /* CONFIG_PPC64_ELF_ABI_V2 */ #define ARCH_DEFINE_STATIC_CALL_TRAMP(name, func) __PPC_SCT(name, "b " #func) #define ARCH_DEFINE_STATIC_CALL_NULL_TRAMP(name) __PPC_SCT(name, "blr") -#define ARCH_DEFINE_STATIC_CALL_RET0_TRAMP(name) __PPC_SCT(name, "b .+20") +#define ARCH_DEFINE_STATIC_CALL_RET0_TRAMP(name) __PPC_SCT(name, "b 1f") #endif /* _ASM_POWERPC_STATIC_CALL_H */ diff --git a/arch/powerpc/kernel/Makefile b/arch/powerpc/kernel/Makefile index 06d2d1f78f71..a30d0d0f5499 100644 --- a/arch/powerpc/kernel/Makefile +++ b/arch/powerpc/kernel/Makefile @@ -128,8 +128,9 @@ extra-y += vmlinux.lds obj-$(CONFIG_RELOCATABLE) += reloc_$(BITS).o -obj-$(CONFIG_PPC32) += entry_32.o setup_32.o early_32.o static_call.o +obj-$(CONFIG_PPC32) += entry_32.o setup_32.o early_32.o obj-$(CONFIG_PPC64) += dma-iommu.o iommu.o +obj-$(CONFIG_HAVE_STATIC_CALL) += static_call.o obj-$(CONFIG_KGDB) += kgdb.o obj-$(CONFIG_BOOTX_TEXT) += btext.o obj-$(CONFIG_SMP) += smp.o diff --git a/arch/powerpc/kernel/static_call.c b/arch/powerpc/kernel/static_call.c index 863a7aa24650..9211b2e189bb 100644 --- a/arch/powerpc/kernel/static_call.c +++ b/arch/powerpc/kernel/static_call.c @@ -1,33 +1,151 @@ // SPDX-License-Identifier: GPL-2.0 +#include #include #include #include +static long sign_extend_long(unsigned long value, int index) +{ + if (sizeof(long) == 8) + return sign_extend64(value, index); + else + return sign_extend32(value, index); +} + +static void *ppc_function_toc(u32 *func) +{ + if (IS_ENABLED(CONFIG_PPC64_ELF_ABI_V2)) { + /* There are two common global entry sequences we handle below + * + * 1. addis r2, r12, SI1 + * addi r2, SI2 + * + * 2. lis r2, SI1 + * addi r2, SI2 + * + * Where r12 contains the global entry point address (it is otherwise + * uninitialised, so doesn't matter what value we use if this is not + * a separate global entry point). + * + * Here we simulate running the given sequence and return the result it + * would calculate. If the sequence is not recognised we return NULL. + */ + u32 insn1 = *func; + u32 insn2 = *(func + 1); + unsigned long op_regs1 = insn1 & OP_RT_RA_MASK; + unsigned long op_regs2 = insn2 & OP_RT_RA_MASK; + unsigned long si1 = insn1 & OP_SI_MASK; + unsigned long si2 = insn2 & OP_SI_MASK; + unsigned long imm1 = sign_extend_long(si1 << 16, 31); + unsigned long imm2 = sign_extend_long(si2, 15); + unsigned long addr = 0; + + /* Simulate the first instruction */ + if (op_regs1 == ADDIS_R2_R12) + addr += (unsigned long)func + imm1; + else if (op_regs1 == LIS_R2) + addr += imm1; + else + return NULL; + + /* Simulate the second instruction */ + if (op_regs2 == ADDI_R2_R2) + addr += imm2; + else + return NULL; + + return (void *)addr; + } + + return NULL; +} + +static bool shares_toc(void *func1, void *func2) +{ + if (IS_ENABLED(CONFIG_PPC64_ELF_ABI_V2)) { + void *func1_toc; + void *func2_toc; + + if (func1 == NULL || func2 == NULL) + return false; + + /* Assume the kernel only uses a single TOC */ + if (core_kernel_text((unsigned long)func1) && + core_kernel_text((unsigned long)func2)) + return true; + + /* Fall back to calculating the TOC from common patterns + * if modules are involved + */ + func1_toc = ppc_function_toc(func1); + func2_toc = ppc_function_toc(func2); + return func1_toc != NULL && func2_toc != NULL && func1_toc == func2_toc; + } + + return true; +} + +static void *get_inst_addr(void *tramp) +{ + return tramp + (core_kernel_text((unsigned long)tramp) + ? PPC_SCT_INST_KERNEL + : PPC_SCT_INST_MODULE); +} + +static void *get_ret0_addr(void *tramp) +{ + return tramp + (core_kernel_text((unsigned long)tramp) + ? PPC_SCT_RET0_KERNEL + : PPC_SCT_RET0_MODULE); +} + +static void *get_data_addr(void *tramp) +{ + return tramp + (core_kernel_text((unsigned long) tramp) + ? PPC_SCT_DATA_KERNEL + : PPC_SCT_DATA_MODULE); +} + void arch_static_call_transform(void *site, void *tramp, void *func, bool tail) { int err; bool is_ret0 = (func == __static_call_return0); - unsigned long target = (unsigned long)(is_ret0 ? tramp + PPC_SCT_RET0 : func); - bool is_short = is_offset_in_branch_range((long)target - (long)tramp); + bool is_short; + void *target = is_ret0 ? get_ret0_addr(tramp) : func; + void *tramp_inst = get_inst_addr(tramp); if (!tramp) return; + if (is_ret0) + is_short = true; + else if (shares_toc(tramp, target)) + is_short = is_offset_in_branch_range( + (long)ppc_function_entry(target) - (long)tramp_inst); + else + /* Combine out-of-range with not sharing a TOC. Though it's possible + * an out-of-range target shares a TOC, handling this separately + * complicates the trampoline. It's simpler to always use the global + * entry point in this case. + */ + is_short = false; + mutex_lock(&text_mutex); if (func && !is_short) { - err = patch_instruction(tramp + PPC_SCT_DATA, ppc_inst(target)); + err = patch_ulong(get_data_addr(tramp), (unsigned long)target); if (err) goto out; } if (!func) - err = patch_instruction(tramp, ppc_inst(PPC_RAW_BLR())); + err = patch_instruction(tramp_inst, ppc_inst(PPC_RAW_BLR())); else if (is_short) - err = patch_branch(tramp, target, 0); + err = patch_branch(tramp_inst, ppc_function_entry(target), 0); else - err = patch_instruction(tramp, ppc_inst(PPC_RAW_NOP())); + err = patch_instruction(tramp_inst, ppc_inst(PPC_RAW_NOP())); + out: mutex_unlock(&text_mutex); From patchwork Mon Oct 10 00:29:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Gray X-Patchwork-Id: 1687875 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=Xz5t3f4a; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Mm0G83p8Dz23jn for ; Mon, 10 Oct 2022 11:31:27 +1100 (AEDT) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Mm0G64x8Yz3bjh for ; Mon, 10 Oct 2022 11:31:26 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=Xz5t3f4a; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=bgray@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=Xz5t3f4a; dkim-atps=neutral Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Mm0Fj0gvfz3bgh for ; Mon, 10 Oct 2022 11:31:04 +1100 (AEDT) Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 29A0I6FH026655; Mon, 10 Oct 2022 00:30:41 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=d2yAZ7F8TDw9fEWMm4S291jpj9nlvepl34zrMk8sw/s=; b=Xz5t3f4as2UyovdPY0rVH/p1JzhLifhiVC1xksL2yBgQvZKaAW4MmNHtAnx7cNtSQfxE h0hJw4XZFS/NWq3TsIUS8yccJs6+W3NWM8KhzN/M4EQkFqlOlRpAWZvyNOvQkzXHezZX ByeRUAF/GBljYUDOC+mZZsP+N/eMIVwGKjZwgTSTgGQSL5HfWdInn6D/oK8dMZeuNLa6 HTiyPjZe5V/7JAJ+t2vb8SrMG8KM5afORg9SMDYBsRwWAo6qCNvtmAu9+mrdXQ3DllL4 430AEWQACk7/fx+b7Qe9gT0xwfun240FeQbt5rma0OqAUXjPx8nSlkHoGFk8Yf0QEULI tQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3k3jgwd3ma-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Oct 2022 00:30:40 +0000 Received: from m0098409.ppops.net (m0098409.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 29A0MI1k004877; Mon, 10 Oct 2022 00:30:40 GMT Received: from ppma03ams.nl.ibm.com (62.31.33a9.ip4.static.sl-reverse.com [169.51.49.98]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3k3jgwd3ky-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Oct 2022 00:30:40 +0000 Received: from pps.filterd (ppma03ams.nl.ibm.com [127.0.0.1]) by ppma03ams.nl.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 29A0LVcd008659; Mon, 10 Oct 2022 00:30:38 GMT Received: from b06avi18626390.portsmouth.uk.ibm.com (b06avi18626390.portsmouth.uk.ibm.com [9.149.26.192]) by ppma03ams.nl.ibm.com with ESMTP id 3k30u920xf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 10 Oct 2022 00:30:37 +0000 Received: from d06av25.portsmouth.uk.ibm.com (d06av25.portsmouth.uk.ibm.com [9.149.105.61]) by b06avi18626390.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 29A0Ptrg37617996 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 10 Oct 2022 00:25:55 GMT Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9A7B111C04C; Mon, 10 Oct 2022 00:30:35 +0000 (GMT) Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id EDB3611C04A; Mon, 10 Oct 2022 00:30:34 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av25.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 10 Oct 2022 00:30:34 +0000 (GMT) Received: from li-0d7fa1cc-2c9d-11b2-a85c-aed20764436d.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 96E746063C; Mon, 10 Oct 2022 11:30:30 +1100 (AEDT) From: Benjamin Gray To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v4 6/6] powerpc: Add tests for out-of-line static calls Date: Mon, 10 Oct 2022 11:29:57 +1100 Message-Id: <20221010002957.128276-7-bgray@linux.ibm.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221010002957.128276-1-bgray@linux.ibm.com> References: <20221010002957.128276-1-bgray@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: y6mYGIoVD7rstE4On4qVYWp-hu4Sxc_Z X-Proofpoint-ORIG-GUID: zmjN3EJwHurVPZUEe1xQVsAdbx7eXkIX X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1 definitions=2022-10-07_04,2022-10-07_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 priorityscore=1501 adultscore=0 clxscore=1015 spamscore=0 impostorscore=0 mlxlogscore=999 lowpriorityscore=0 mlxscore=0 bulkscore=0 phishscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2209130000 definitions=main-2210090155 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: ajd@linux.ibm.com, peterz@infradead.org, Benjamin Gray , npiggin@gmail.com, ardb@kernel.org, jbaron@akamai.com, rostedt@goodmis.org, jpoimboe@kernel.org Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" KUnit tests for the various combinations of caller/trampoline/target and kernel/module. They must be run from a module loaded at runtime to guarantee they have a different TOC to the kernel (64-bit ELF V2) and increase the chance of testing the non-direct branch path of the trampoline. For 64-bit ELF V2 ABI the tests try to mitigate the chance of panicking by restoring the TOC after every static call. Not all possible errors can be caught by this (we can't stop a trampoline from using a bad TOC itself), but it makes certain errors easier to debug. Signed-off-by: Benjamin Gray --- arch/powerpc/Kconfig | 12 ++ arch/powerpc/kernel/Makefile | 1 + arch/powerpc/kernel/static_call.c | 53 +++++ arch/powerpc/kernel/static_call_test.c | 263 +++++++++++++++++++++++++ arch/powerpc/kernel/static_call_test.h | 56 ++++++ 5 files changed, 385 insertions(+) create mode 100644 arch/powerpc/kernel/static_call_test.c create mode 100644 arch/powerpc/kernel/static_call_test.h diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 962e36ec34ec..5b9d5fa96a9e 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -1035,6 +1035,18 @@ config PPC_RTAS_FILTER Say Y unless you know what you are doing and the filter is causing problems for you. +config PPC_STATIC_CALL_KUNIT_TEST + tristate "KUnit tests for static calls" + default KUNIT_ALL_TESTS + depends on HAVE_STATIC_CALL && KUNIT && m + help + Tests for static calls across all combinations of caller/trampoline/target + being kernel/module. On ELF ABI V2 the tests check the TOC is kept consistent. + + Must be built as a module and loaded at runtime to ensure the module has + a different TOC to the kernel and make it likely that non-direct branch + path of the trampoline is tested. + endmenu config ISA_DMA_API diff --git a/arch/powerpc/kernel/Makefile b/arch/powerpc/kernel/Makefile index a30d0d0f5499..22c07e3d34df 100644 --- a/arch/powerpc/kernel/Makefile +++ b/arch/powerpc/kernel/Makefile @@ -131,6 +131,7 @@ obj-$(CONFIG_RELOCATABLE) += reloc_$(BITS).o obj-$(CONFIG_PPC32) += entry_32.o setup_32.o early_32.o obj-$(CONFIG_PPC64) += dma-iommu.o iommu.o obj-$(CONFIG_HAVE_STATIC_CALL) += static_call.o +obj-$(CONFIG_PPC_STATIC_CALL_KUNIT_TEST) += static_call_test.o obj-$(CONFIG_KGDB) += kgdb.o obj-$(CONFIG_BOOTX_TEXT) += btext.o obj-$(CONFIG_SMP) += smp.o diff --git a/arch/powerpc/kernel/static_call.c b/arch/powerpc/kernel/static_call.c index 9211b2e189bb..44957ba91e3f 100644 --- a/arch/powerpc/kernel/static_call.c +++ b/arch/powerpc/kernel/static_call.c @@ -153,3 +153,56 @@ void arch_static_call_transform(void *site, void *tramp, void *func, bool tail) panic("%s: patching failed %pS at %pS\n", __func__, func, tramp); } EXPORT_SYMBOL_GPL(arch_static_call_transform); + + +#if IS_MODULE(CONFIG_PPC_STATIC_CALL_KUNIT_TEST) + +#include "static_call_test.h" + +int ppc_sc_kernel_target_1(struct kunit *test) +{ + toc_fixup(test); + return 1; +} + +int ppc_sc_kernel_target_2(struct kunit *test) +{ + toc_fixup(test); + return 2; +} + +DEFINE_STATIC_CALL(ppc_sc_kernel, ppc_sc_kernel_target_1); + +int ppc_sc_kernel_call(struct kunit *test) +{ + return PROTECTED_SC(test, int, static_call(ppc_sc_kernel)(test)); +} + +int ppc_sc_kernel_call_indirect(struct kunit *test, int (*fn)(struct kunit *test)) +{ + return PROTECTED_SC(test, int, fn(test)); +} + +long ppc_sc_kernel_target_big(struct kunit *test, long a, long b, long c, long d, + long e, long f, long g, long h, long i) +{ + toc_fixup(test); + KUNIT_EXPECT_EQ(test, a, b); + KUNIT_EXPECT_EQ(test, a, c); + KUNIT_EXPECT_EQ(test, a, d); + KUNIT_EXPECT_EQ(test, a, e); + KUNIT_EXPECT_EQ(test, a, f); + KUNIT_EXPECT_EQ(test, a, g); + KUNIT_EXPECT_EQ(test, a, h); + KUNIT_EXPECT_EQ(test, a, i); + return ~a; +} + +EXPORT_SYMBOL_GPL(ppc_sc_kernel_target_1); +EXPORT_SYMBOL_GPL(ppc_sc_kernel_target_2); +EXPORT_SYMBOL_GPL(ppc_sc_kernel_target_big); +EXPORT_STATIC_CALL_GPL(ppc_sc_kernel); +EXPORT_SYMBOL_GPL(ppc_sc_kernel_call); +EXPORT_SYMBOL_GPL(ppc_sc_kernel_call_indirect); + +#endif /* IS_MODULE(CONFIG_PPC_STATIC_CALL_KUNIT_TEST) */ diff --git a/arch/powerpc/kernel/static_call_test.c b/arch/powerpc/kernel/static_call_test.c new file mode 100644 index 000000000000..10a09ef455cf --- /dev/null +++ b/arch/powerpc/kernel/static_call_test.c @@ -0,0 +1,263 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include "static_call_test.h" + +#include +#include +#include + +/* Tests to ensure correctness in a variety of cases for static calls. + * + * The tests focus on ensuring the TOC is kept consistent across the + * module-kernel boundary, as compilers can't see that a trampoline + * defined locally to a caller might be jumping to a function with a + * different TOC. So it's important that these tests are compiled as + * a module to ensure the TOC will be different to the kernel's. + */ + +#ifdef CONFIG_PPC64_ELF_ABI_V2 + +/* Utils to hold a copy of the old register values while we test. + * + * We can't use the KUnit init/exit hooks because when the hooks and + * test cases return they will be in the KUnit context that doesn't know + * we've reserved and modified some non-volatile registers. + */ + +static void *saved_registers[2]; + +static void init_testcase(struct kunit *test) +{ + saved_registers[0] = actual_toc; + saved_registers[1] = module_toc; + module_toc = current_toc; + KUNIT_ASSERT_PTR_NE(test, module_toc, (void *)get_paca()->kernel_toc); +} + +static void exit_testcase(void) +{ + actual_toc = saved_registers[0]; + module_toc = saved_registers[1]; +} + +#else + +static void init_testcase(struct kunit *test) {} +static void exit_testcase(void) {} + +#endif /* CONFIG_PPC64_ELF_ABI_V2 */ + +static int module_target_11(struct kunit *test) +{ + toc_fixup(test); + return 11; +} + +static int module_target_12(struct kunit *test) +{ + toc_fixup(test); + return 12; +} + +DEFINE_STATIC_CALL(module_sc, module_target_11); + +DEFINE_STATIC_CALL_RET0(module_sc_ret0, int(void)); +DEFINE_STATIC_CALL_NULL(module_sc_null, int(int)); + +static int add_one(int *val) +{ + return (*val)++; +} + +static void null_function_test(struct kunit *test) +{ + int val = 0; + + init_testcase(test); + + /* Check argument unconditionally evaluated */ + static_call_cond(module_sc_null)(add_one(&val)); + KUNIT_ASSERT_EQ(test, 1, val); + + exit_testcase(); +} + +static void return_zero_test(struct kunit *test) +{ + int ret; + + init_testcase(test); + + ret = PROTECTED_SC(test, int, static_call(module_sc_ret0)()); + KUNIT_ASSERT_EQ(test, 0, ret); + + static_call_update(ppc_sc_kernel, (void *)__static_call_return0); + ret = PROTECTED_SC(test, int, static_call(ppc_sc_kernel)(test)); + KUNIT_ASSERT_EQ(test, 0, ret); + + static_call_update(module_sc, (void *)__static_call_return0); + ret = PROTECTED_SC(test, int, static_call(module_sc)(test)); + KUNIT_ASSERT_EQ(test, 0, ret); + + exit_testcase(); +} + +static void kernel_kernel_kernel_test(struct kunit *test) +{ + init_testcase(test); + + static_call_update(ppc_sc_kernel, ppc_sc_kernel_target_1); + KUNIT_ASSERT_EQ(test, 1, ppc_sc_kernel_call(test)); + + static_call_update(ppc_sc_kernel, ppc_sc_kernel_target_2); + KUNIT_ASSERT_EQ(test, 2, ppc_sc_kernel_call(test)); + + exit_testcase(); +} + +static void kernel_kernel_module_test(struct kunit *test) +{ + init_testcase(test); + + static_call_update(ppc_sc_kernel, module_target_11); + KUNIT_ASSERT_EQ(test, 11, ppc_sc_kernel_call(test)); + + static_call_update(ppc_sc_kernel, module_target_12); + KUNIT_ASSERT_EQ(test, 12, ppc_sc_kernel_call(test)); + + exit_testcase(); +} + +static void kernel_module_kernel_test(struct kunit *test) +{ + init_testcase(test); + + static_call_update(module_sc, ppc_sc_kernel_target_1); + KUNIT_ASSERT_EQ(test, 1, ppc_sc_kernel_call_indirect(test, static_call(module_sc))); + + static_call_update(module_sc, ppc_sc_kernel_target_2); + KUNIT_ASSERT_EQ(test, 2, ppc_sc_kernel_call_indirect(test, static_call(module_sc))); + + exit_testcase(); +} + +static void kernel_module_module_test(struct kunit *test) +{ + init_testcase(test); + + static_call_update(module_sc, module_target_11); + KUNIT_ASSERT_EQ(test, 11, ppc_sc_kernel_call_indirect(test, static_call(module_sc))); + + static_call_update(module_sc, module_target_12); + KUNIT_ASSERT_EQ(test, 12, ppc_sc_kernel_call_indirect(test, static_call(module_sc))); + + exit_testcase(); +} + +static void module_kernel_kernel_test(struct kunit *test) +{ + int ret; + + init_testcase(test); + + static_call_update(ppc_sc_kernel, ppc_sc_kernel_target_1); + ret = PROTECTED_SC(test, int, static_call(ppc_sc_kernel)(test)); + KUNIT_ASSERT_EQ(test, 1, ret); + + static_call_update(ppc_sc_kernel, ppc_sc_kernel_target_2); + ret = PROTECTED_SC(test, int, static_call(ppc_sc_kernel)(test)); + KUNIT_ASSERT_EQ(test, 2, ret); + + exit_testcase(); +} + +static void module_kernel_module_test(struct kunit *test) +{ + int ret; + + init_testcase(test); + + static_call_update(ppc_sc_kernel, module_target_11); + ret = PROTECTED_SC(test, int, static_call(ppc_sc_kernel)(test)); + KUNIT_ASSERT_EQ(test, 11, ret); + + static_call_update(ppc_sc_kernel, module_target_12); + ret = PROTECTED_SC(test, int, static_call(ppc_sc_kernel)(test)); + KUNIT_ASSERT_EQ(test, 12, ret); + + exit_testcase(); +} + +static void module_module_kernel_test(struct kunit *test) +{ + int ret; + + init_testcase(test); + + static_call_update(module_sc, ppc_sc_kernel_target_1); + ret = PROTECTED_SC(test, int, static_call(module_sc)(test)); + KUNIT_ASSERT_EQ(test, 1, ret); + + static_call_update(module_sc, ppc_sc_kernel_target_2); + ret = PROTECTED_SC(test, int, static_call(module_sc)(test)); + KUNIT_ASSERT_EQ(test, 2, ret); + + exit_testcase(); +} + +static void module_module_module_test(struct kunit *test) +{ + int ret; + + init_testcase(test); + + static_call_update(module_sc, module_target_11); + ret = PROTECTED_SC(test, int, static_call(module_sc)(test)); + KUNIT_ASSERT_EQ(test, 11, ret); + + static_call_update(module_sc, module_target_12); + ret = PROTECTED_SC(test, int, static_call(module_sc)(test)); + KUNIT_ASSERT_EQ(test, 12, ret); + + exit_testcase(); +} + +DEFINE_STATIC_CALL(module_sc_stack_params, ppc_sc_kernel_target_big); + +static void stack_parameters_test(struct kunit *test) +{ + long m = -0x87654321; + long ret; + + init_testcase(test); + + ret = PROTECTED_SC(test, long, + static_call(module_sc_stack_params)(test, m, m, m, m, m, m, m, m, m)); + KUNIT_ASSERT_EQ(test, ~m, ret); + + exit_testcase(); +} + +static struct kunit_case static_call_test_cases[] = { + KUNIT_CASE(null_function_test), + KUNIT_CASE(return_zero_test), + KUNIT_CASE(stack_parameters_test), + KUNIT_CASE(kernel_kernel_kernel_test), + KUNIT_CASE(kernel_kernel_module_test), + KUNIT_CASE(kernel_module_kernel_test), + KUNIT_CASE(kernel_module_module_test), + KUNIT_CASE(module_kernel_kernel_test), + KUNIT_CASE(module_kernel_module_test), + KUNIT_CASE(module_module_kernel_test), + KUNIT_CASE(module_module_module_test), + {} +}; + +static struct kunit_suite ppc_static_call_test_suite = { + .name = "ppc-static-call", + .test_cases = static_call_test_cases, +}; +kunit_test_suite(ppc_static_call_test_suite); + +MODULE_AUTHOR("Benjamin Gray "); +MODULE_LICENSE("GPL"); diff --git a/arch/powerpc/kernel/static_call_test.h b/arch/powerpc/kernel/static_call_test.h new file mode 100644 index 000000000000..71b5bc52c099 --- /dev/null +++ b/arch/powerpc/kernel/static_call_test.h @@ -0,0 +1,56 @@ +#ifndef _POWERPC_STATIC_CALL_TEST_ +#define _POWERPC_STATIC_CALL_TEST_ + +#include + +DECLARE_STATIC_CALL(ppc_sc_kernel, int(struct kunit *test)); +int ppc_sc_kernel_target_1(struct kunit *test); +int ppc_sc_kernel_target_2(struct kunit *test); +long ppc_sc_kernel_target_big(struct kunit *test, long a, long b, long c, long d, + long e, long f, long g, long h, long i); +int ppc_sc_kernel_call(struct kunit *test); +int ppc_sc_kernel_call_indirect(struct kunit *test, int(*fn)(struct kunit *test)); + +#ifdef CONFIG_PPC64_ELF_ABI_V2 + +/* Reserve these registers for testing so that a TOC error + * doesn't necessarily crash the whole kernel. + * + * The tests ensure the contents are restored before returning. + */ +register void *current_toc asm ("r2"); +register void *actual_toc asm ("r14"); /* Copy of TOC before fixup */ +register void *module_toc asm ("r15"); + +#ifdef MODULE + +#define toc_fixup(test) \ + actual_toc = current_toc; \ + current_toc = module_toc; \ + KUNIT_EXPECT_PTR_EQ(test, module_toc, actual_toc) + +#else /* KERNEL */ + +#define toc_fixup(test) \ + actual_toc = current_toc; \ + current_toc = (void *)get_paca()->kernel_toc; \ + KUNIT_EXPECT_PTR_EQ(test, (void *)get_paca()->kernel_toc, actual_toc) + +#endif /* MODULE */ + +#define PROTECTED_SC(test, ret_type, call) \ +({ \ + ret_type ret; \ + ret = call; \ + toc_fixup(test); \ + ret; \ +}) + +#else + +#define toc_fixup(test) {} +#define PROTECTED_SC(test, ret_type, call) call + +#endif /* CONFIG_PPC64_ELF_ABI_V2 */ + +#endif /* _POWERPC_STATIC_CALL_TEST_ */