From patchwork Mon Sep 26 06:43:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Gray X-Patchwork-Id: 1682351 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=jGu03Nql; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MbYD22LdNz1ypH for ; Mon, 26 Sep 2022 16:45:22 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4MbYD21r2fz3bYM for ; Mon, 26 Sep 2022 16:45:22 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=jGu03Nql; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=bgray@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=jGu03Nql; dkim-atps=neutral Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4MbYC46Ptfz2xHF for ; Mon, 26 Sep 2022 16:44:32 +1000 (AEST) Received: from pps.filterd (m0098420.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 28Q4ZPtI030992; Mon, 26 Sep 2022 06:44:00 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=K0Ez093Jir8pfoPaXkQTQDGA2NUEKxpG2TK0oAaWM9E=; b=jGu03NqlMhgRiT8dLVJPxxtcHXpXAIzj96qgfoYzgB42ar60EHSB2pshP9FrZulYAHma RhqA8cylHuihvjHmfb7TcAQGEtpvPBjCGO+A1AZtTSOI8xG/c6LxcgzXpmPVzK8zV1Gm fztcuJMpDufOa8zgRB/tsZVnhq/iJ44yJpf2x/H0VTUlftap23b1Y40VVONvS1pEstpn kJbSFMpqgMK2Q3nnXKR+FTHkE5fmfXEiYMYCejQ3UNLAl9+TIuIg8nY98hpvX0Y3OGJg x9VRlXY/5ntsB5y9qgKxxQ18wPMAZ2NX//h7XEzMqu5Wp3lZtxChg5XNrKlym8KlYDes YQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com (PPS) with ESMTPS id 3jtby74cf7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 26 Sep 2022 06:44:00 +0000 Received: from m0098420.ppops.net (m0098420.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 28Q5NBjg016938; Mon, 26 Sep 2022 06:43:59 GMT Received: from ppma04ams.nl.ibm.com (63.31.33a9.ip4.static.sl-reverse.com [169.51.49.99]) by mx0b-001b2d01.pphosted.com (PPS) with ESMTPS id 3jtby74cek-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 26 Sep 2022 06:43:59 +0000 Received: from pps.filterd (ppma04ams.nl.ibm.com [127.0.0.1]) by ppma04ams.nl.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 28Q6ZTCV003384; Mon, 26 Sep 2022 06:43:57 GMT Received: from b06cxnps4075.portsmouth.uk.ibm.com (d06relay12.portsmouth.uk.ibm.com [9.149.109.197]) by ppma04ams.nl.ibm.com with ESMTP id 3jssh8syqt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 26 Sep 2022 06:43:57 +0000 Received: from d06av26.portsmouth.uk.ibm.com (d06av26.portsmouth.uk.ibm.com [9.149.105.62]) by b06cxnps4075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 28Q6htmY1376890 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 26 Sep 2022 06:43:55 GMT Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 59EF6AE053; Mon, 26 Sep 2022 06:43:55 +0000 (GMT) Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id AD487AE045; Mon, 26 Sep 2022 06:43:54 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av26.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 26 Sep 2022 06:43:54 +0000 (GMT) Received: from beng-ozlabs-ibm-com.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 44EF66016C; Mon, 26 Sep 2022 16:43:45 +1000 (AEST) From: Benjamin Gray To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v2 1/6] powerpc/code-patching: Implement generic text patching function Date: Mon, 26 Sep 2022 16:43:11 +1000 Message-Id: <20220926064316.765967-2-bgray@linux.ibm.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220926064316.765967-1-bgray@linux.ibm.com> References: <20220926064316.765967-1-bgray@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: U6PTP-gfBgeBpkHqIJcVgUEt824QEy1C X-Proofpoint-GUID: 69dALQow2t9dl3FBUPaN0q1WWf4W4i93 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1 definitions=2022-09-26_04,2022-09-22_02,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 malwarescore=0 priorityscore=1501 adultscore=0 phishscore=0 spamscore=0 lowpriorityscore=0 suspectscore=0 impostorscore=0 mlxscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2209130000 definitions=main-2209260039 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: christophe.leroy@c-s.fr, ajd@linux.ibm.com, peterz@infradead.org, Benjamin Gray , npiggin@gmail.com, ardb@kernel.org, jbaron@akamai.com, rostedt@goodmis.org, jpoimboe@kernel.org Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Adds a generic text patching mechanism for patches of 1, 2, 4, or (64-bit) 8 bytes. The patcher conditionally syncs the icache depending on if the content will be executed (as opposed to, e.g., read-only data). The `patch_instruction` function is reimplemented in terms of this more generic function. This generic implementation allows patching of arbitrary 64-bit data, whereas the original `patch_instruction` decided the size based on the 'instruction' opcode, so was not suitable for arbitrary data. Signed-off-by: Benjamin Gray --- arch/powerpc/include/asm/code-patching.h | 7 ++ arch/powerpc/lib/code-patching.c | 90 +++++++++++++++++------- 2 files changed, 71 insertions(+), 26 deletions(-) -- 2.37.3 diff --git a/arch/powerpc/include/asm/code-patching.h b/arch/powerpc/include/asm/code-patching.h index 1c6316ec4b74..15efd8ab22da 100644 --- a/arch/powerpc/include/asm/code-patching.h +++ b/arch/powerpc/include/asm/code-patching.h @@ -76,6 +76,13 @@ int create_cond_branch(ppc_inst_t *instr, const u32 *addr, int patch_branch(u32 *addr, unsigned long target, int flags); int patch_instruction(u32 *addr, ppc_inst_t instr); int raw_patch_instruction(u32 *addr, ppc_inst_t instr); +int __patch_memory(void *dest, unsigned long src, size_t size); + +#define patch_memory(addr, val) \ +({ \ + BUILD_BUG_ON(!__native_word(val)); \ + __patch_memory(addr, (unsigned long) val, sizeof(val)); \ +}) static inline unsigned long patch_site_addr(s32 *site) { diff --git a/arch/powerpc/lib/code-patching.c b/arch/powerpc/lib/code-patching.c index ad0cf3108dd0..9979380d55ef 100644 --- a/arch/powerpc/lib/code-patching.c +++ b/arch/powerpc/lib/code-patching.c @@ -15,20 +15,47 @@ #include #include -static int __patch_instruction(u32 *exec_addr, ppc_inst_t instr, u32 *patch_addr) +static int __always_inline ___patch_memory(void *patch_addr, + unsigned long data, + void *prog_addr, + size_t size) { - if (!ppc_inst_prefixed(instr)) { - u32 val = ppc_inst_val(instr); + switch (size) { + case 1: + __put_kernel_nofault(patch_addr, &data, u8, failed); + break; + case 2: + __put_kernel_nofault(patch_addr, &data, u16, failed); + break; + case 4: + __put_kernel_nofault(patch_addr, &data, u32, failed); + break; +#ifdef CONFIG_PPC64 + case 8: + __put_kernel_nofault(patch_addr, &data, u64, failed); + break; +#endif + default: + unreachable(); + } - __put_kernel_nofault(patch_addr, &val, u32, failed); - } else { - u64 val = ppc_inst_as_ulong(instr); + dcbst(patch_addr); + dcbst(patch_addr + size - 1); /* Last byte of data may cross a cacheline */ - __put_kernel_nofault(patch_addr, &val, u64, failed); - } + mb(); /* sync */ + + /* Flush on the EA that may be executed in case of a non-coherent icache */ + icbi(prog_addr); + + /* Also flush the last byte of the instruction if it may be a + * prefixed instruction and we aren't assuming minimum 64-byte + * cacheline sizes + */ + if (IS_ENABLED(CONFIG_PPC64) && L1_CACHE_BYTES < 64) + icbi(prog_addr + size - 1); - asm ("dcbst 0, %0; sync; icbi 0,%1; sync; isync" :: "r" (patch_addr), - "r" (exec_addr)); + mb(); /* sync */ + isync(); return 0; @@ -38,7 +65,10 @@ static int __patch_instruction(u32 *exec_addr, ppc_inst_t instr, u32 *patch_addr int raw_patch_instruction(u32 *addr, ppc_inst_t instr) { - return __patch_instruction(addr, instr, addr); + if (ppc_inst_prefixed(instr)) + return ___patch_memory(addr, ppc_inst_as_ulong(instr), addr, sizeof(u64)); + else + return ___patch_memory(addr, ppc_inst_val(instr), addr, sizeof(u32)); } #ifdef CONFIG_STRICT_KERNEL_RWX @@ -147,24 +177,22 @@ static void unmap_patch_area(unsigned long addr) flush_tlb_kernel_range(addr, addr + PAGE_SIZE); } -static int __do_patch_instruction(u32 *addr, ppc_inst_t instr) +static int __always_inline __do_patch_memory(void *dest, unsigned long src, size_t size) { int err; u32 *patch_addr; - unsigned long text_poke_addr; pte_t *pte; - unsigned long pfn = get_patch_pfn(addr); - - text_poke_addr = (unsigned long)__this_cpu_read(text_poke_area)->addr & PAGE_MASK; - patch_addr = (u32 *)(text_poke_addr + offset_in_page(addr)); + unsigned long text_poke_addr = (unsigned long)__this_cpu_read(text_poke_area)->addr & PAGE_MASK; + unsigned long pfn = get_patch_pfn(dest); + patch_addr = (u32 *)(text_poke_addr + offset_in_page(dest)); pte = virt_to_kpte(text_poke_addr); __set_pte_at(&init_mm, text_poke_addr, pte, pfn_pte(pfn, PAGE_KERNEL), 0); /* See ptesync comment in radix__set_pte_at() */ if (radix_enabled()) asm volatile("ptesync": : :"memory"); - err = __patch_instruction(addr, instr, patch_addr); + err = ___patch_memory(patch_addr, src, dest, size); pte_clear(&init_mm, text_poke_addr, pte); flush_tlb_kernel_range(text_poke_addr, text_poke_addr + PAGE_SIZE); @@ -172,7 +200,7 @@ static int __do_patch_instruction(u32 *addr, ppc_inst_t instr) return err; } -static int do_patch_instruction(u32 *addr, ppc_inst_t instr) +static int __always_inline do_patch_memory(void *dest, unsigned long src, size_t size) { int err; unsigned long flags; @@ -183,32 +211,42 @@ static int do_patch_instruction(u32 *addr, ppc_inst_t instr) * to allow patching. We just do the plain old patching */ if (!static_branch_likely(&poking_init_done)) - return raw_patch_instruction(addr, instr); + return ___patch_memory(dest, src, dest, size); local_irq_save(flags); - err = __do_patch_instruction(addr, instr); + err = __do_patch_memory(dest, src, size); local_irq_restore(flags); return err; } + #else /* !CONFIG_STRICT_KERNEL_RWX */ -static int do_patch_instruction(u32 *addr, ppc_inst_t instr) +static int do_patch_memory(void *dest, unsigned long src, size_t size) { - return raw_patch_instruction(addr, instr); + return ___patch_memory(dest, src, dest, size); } #endif /* CONFIG_STRICT_KERNEL_RWX */ __ro_after_init DEFINE_STATIC_KEY_FALSE(init_mem_is_free); -int patch_instruction(u32 *addr, ppc_inst_t instr) +int __patch_memory(void *dest, unsigned long src, size_t size) { /* Make sure we aren't patching a freed init section */ - if (static_branch_likely(&init_mem_is_free) && init_section_contains(addr, 4)) + if (static_branch_likely(&init_mem_is_free) && init_section_contains(dest, 4)) return 0; - return do_patch_instruction(addr, instr); + return do_patch_memory(dest, src, size); +} +NOKPROBE_SYMBOL(__patch_memory); + +int patch_instruction(u32 *addr, ppc_inst_t instr) +{ + if (ppc_inst_prefixed(instr)) + return patch_memory(addr, ppc_inst_as_ulong(instr)); + else + return patch_memory(addr, ppc_inst_val(instr)); } NOKPROBE_SYMBOL(patch_instruction); From patchwork Mon Sep 26 06:43:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Gray X-Patchwork-Id: 1682355 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=jA6gM5le; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MbYHG3Z3cz1ypH for ; Mon, 26 Sep 2022 16:48:10 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4MbYHG2g87z3cjG for ; Mon, 26 Sep 2022 16:48:10 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=jA6gM5le; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=bgray@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=jA6gM5le; dkim-atps=neutral Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4MbYCN1GRtz3bcw for ; Mon, 26 Sep 2022 16:44:48 +1000 (AEST) Received: from pps.filterd (m0098410.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 28Q4ZQZZ005646; Mon, 26 Sep 2022 06:44:00 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=JjoakmGSN2j/oTNtddH3M+vnu2WJ36YPhlARa3ikhzw=; b=jA6gM5lelCDAbKS/HZdT0ko3+iEo43Lw0lKPVxXHgOQ1b/zhUVVPkqfvZFM+OWBQE/FH uj2MdB5P2YiJS7Q0S7gUBYcSbdPXC1xnCqg3lmJHU2LK1l0KuL2cZaG3B1HGYLV4s36F ZOXFCPC/bkCwGddM4oO4sPd2AmGTNGHsSreZ2oEQdhQNV2UbUT/ozmlRKnQElkT3UXWn nGFIeOxeukdi7mFM2gWUqoH4yOiQpElyw79KuktKtoPSHyZ3p0vfRZk2Y8/jbI/RQGC1 msAbwAiIsaf0h6YU5DnPJw0RRUvf63P06OleM0c337rkRAIlZBICj4DGDrIf02RrxALX 6A== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3jtbq8cjdg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 26 Sep 2022 06:44:00 +0000 Received: from m0098410.ppops.net (m0098410.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 28Q6e3Pa014431; Mon, 26 Sep 2022 06:44:00 GMT Received: from ppma04ams.nl.ibm.com (63.31.33a9.ip4.static.sl-reverse.com [169.51.49.99]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3jtbq8cjcu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 26 Sep 2022 06:43:59 +0000 Received: from pps.filterd (ppma04ams.nl.ibm.com [127.0.0.1]) by ppma04ams.nl.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 28Q6ZSo4003365; Mon, 26 Sep 2022 06:43:57 GMT Received: from b06avi18626390.portsmouth.uk.ibm.com (b06avi18626390.portsmouth.uk.ibm.com [9.149.26.192]) by ppma04ams.nl.ibm.com with ESMTP id 3jssh8syqs-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 26 Sep 2022 06:43:57 +0000 Received: from d06av25.portsmouth.uk.ibm.com (d06av25.portsmouth.uk.ibm.com [9.149.105.61]) by b06avi18626390.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 28Q6di9b37159364 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 26 Sep 2022 06:39:44 GMT Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0D3F511C054; Mon, 26 Sep 2022 06:43:55 +0000 (GMT) Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id AB32611C04C; Mon, 26 Sep 2022 06:43:54 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av25.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 26 Sep 2022 06:43:54 +0000 (GMT) Received: from beng-ozlabs-ibm-com.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id C1BD26058C; Mon, 26 Sep 2022 16:43:45 +1000 (AEST) From: Benjamin Gray To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v2 2/6] powerpc/module: Handle caller-saved TOC in module linker Date: Mon, 26 Sep 2022 16:43:12 +1000 Message-Id: <20220926064316.765967-3-bgray@linux.ibm.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220926064316.765967-1-bgray@linux.ibm.com> References: <20220926064316.765967-1-bgray@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: 1z5Vnnrv0KFoVkqbThVKXfnWBOZtOv02 X-Proofpoint-ORIG-GUID: GG3DUrlsywyd6RdpmkKnkVyL6vJdZYcY X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1 definitions=2022-09-26_04,2022-09-22_02,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 clxscore=1015 priorityscore=1501 phishscore=0 lowpriorityscore=0 mlxlogscore=999 bulkscore=0 adultscore=0 mlxscore=0 malwarescore=0 impostorscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2209130000 definitions=main-2209260039 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: christophe.leroy@c-s.fr, ajd@linux.ibm.com, peterz@infradead.org, Benjamin Gray , npiggin@gmail.com, ardb@kernel.org, jbaron@akamai.com, rostedt@goodmis.org, jpoimboe@kernel.org Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" The callee may set a field in `st_other` to 1 to indicate r2 should be treated as caller-saved. This means a trampoline must be used to save the current TOC before calling it and restore it afterwards, much like external calls. This is necessary for supporting V2 ABI static calls that do not preserve the TOC. Signed-off-by: Benjamin Gray --- arch/powerpc/kernel/module_64.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/arch/powerpc/kernel/module_64.c b/arch/powerpc/kernel/module_64.c index 7e45dc98df8a..4d816f7785b4 100644 --- a/arch/powerpc/kernel/module_64.c +++ b/arch/powerpc/kernel/module_64.c @@ -55,6 +55,12 @@ static unsigned int local_entry_offset(const Elf64_Sym *sym) * of function and try to derive r2 from it). */ return PPC64_LOCAL_ENTRY_OFFSET(sym->st_other); } + +static bool need_r2save_stub(unsigned char st_other) +{ + return ((st_other & STO_PPC64_LOCAL_MASK) >> STO_PPC64_LOCAL_BIT) == 1; +} + #else static func_desc_t func_desc(unsigned long addr) @@ -66,6 +72,11 @@ static unsigned int local_entry_offset(const Elf64_Sym *sym) return 0; } +static bool need_r2save_stub(unsigned char st_other) +{ + return false; +} + void *dereference_module_function_descriptor(struct module *mod, void *ptr) { if (ptr < (void *)mod->arch.start_opd || @@ -632,7 +643,8 @@ int apply_relocate_add(Elf64_Shdr *sechdrs, case R_PPC_REL24: /* FIXME: Handle weak symbols here --RR */ if (sym->st_shndx == SHN_UNDEF || - sym->st_shndx == SHN_LIVEPATCH) { + sym->st_shndx == SHN_LIVEPATCH || + need_r2save_stub(sym->st_other)) { /* External: go via stub */ value = stub_for_addr(sechdrs, value, me, strtab + sym->st_name); From patchwork Mon Sep 26 06:43:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Gray X-Patchwork-Id: 1682352 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=ZXBowBIU; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MbYDn4bc7z1ypH for ; Mon, 26 Sep 2022 16:46:01 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4MbYDn2kLnz3c0g for ; Mon, 26 Sep 2022 16:46:01 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=ZXBowBIU; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.158.5; helo=mx0a-001b2d01.pphosted.com; envelope-from=bgray@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=ZXBowBIU; dkim-atps=neutral Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4MbYC73h8Pz2xHF for ; Mon, 26 Sep 2022 16:44:35 +1000 (AEST) Received: from pps.filterd (m0098420.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 28Q4ZQo9031063; Mon, 26 Sep 2022 06:44:00 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=eqzHk4t2NLCPx8tq0XIFFTcpO5wYb8t2yFw/PQzZB24=; b=ZXBowBIUbvy8pVY6db3wWaeBJ83JMTmt284M/a5A71N3xEkF8zFuIn5ijneDhfShrKIP UHR4Mez+3pDGD7FTJKjEjaTZOyf4LnxmPtNirfJZ4hqiom/cLLVLsoHDnmNR/mFr70AC GgPKbEJ3LbTTqReAc5UZ/E7KY1rh++rdqAtqs6+FvKPmN5d9T4lkUb0LjY4MI+2h2fKW HKpasEwASRVAzAlgYLsu96igNGF6it1uTp/zmydZu+VanpEVBtACvlPpcDrTkF5eMXRl qdpRcwETZjm4prm42l7/+YAILo1nkMr28P2FPe2N5xZfNr/NdcIDOTUndZ5tIzHVS02Y jA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com (PPS) with ESMTPS id 3jtby74cf0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 26 Sep 2022 06:43:59 +0000 Received: from m0098420.ppops.net (m0098420.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 28Q5fCCM030566; Mon, 26 Sep 2022 06:43:59 GMT Received: from ppma01fra.de.ibm.com (46.49.7a9f.ip4.static.sl-reverse.com [159.122.73.70]) by mx0b-001b2d01.pphosted.com (PPS) with ESMTPS id 3jtby74cec-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 26 Sep 2022 06:43:59 +0000 Received: from pps.filterd (ppma01fra.de.ibm.com [127.0.0.1]) by ppma01fra.de.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 28Q6ZXHp026572; Mon, 26 Sep 2022 06:43:57 GMT Received: from b06avi18626390.portsmouth.uk.ibm.com (b06avi18626390.portsmouth.uk.ibm.com [9.149.26.192]) by ppma01fra.de.ibm.com with ESMTP id 3jssh8sfv6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 26 Sep 2022 06:43:57 +0000 Received: from d06av25.portsmouth.uk.ibm.com (d06av25.portsmouth.uk.ibm.com [9.149.105.61]) by b06avi18626390.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 28Q6diSl36372864 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 26 Sep 2022 06:39:44 GMT Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0D3EB11C050; Mon, 26 Sep 2022 06:43:55 +0000 (GMT) Received: from d06av25.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id ABC1311C052; Mon, 26 Sep 2022 06:43:54 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av25.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 26 Sep 2022 06:43:54 +0000 (GMT) Received: from beng-ozlabs-ibm-com.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 7F5346057F; Mon, 26 Sep 2022 16:43:53 +1000 (AEST) From: Benjamin Gray To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v2 3/6] powerpc/module: Optimise nearby branches in ELF V2 ABI stub Date: Mon, 26 Sep 2022 16:43:13 +1000 Message-Id: <20220926064316.765967-4-bgray@linux.ibm.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220926064316.765967-1-bgray@linux.ibm.com> References: <20220926064316.765967-1-bgray@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: JsWwzA0Pd1VqpQX6Is8-gYfyq8l_np1s X-Proofpoint-GUID: VbMF9-ryJEJKxa6VYRt3SZlbk9af_Pud X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1 definitions=2022-09-26_04,2022-09-22_02,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 malwarescore=0 priorityscore=1501 adultscore=0 phishscore=0 spamscore=0 lowpriorityscore=0 suspectscore=0 impostorscore=0 mlxscore=0 bulkscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2209130000 definitions=main-2209260039 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: christophe.leroy@c-s.fr, ajd@linux.ibm.com, peterz@infradead.org, Benjamin Gray , npiggin@gmail.com, ardb@kernel.org, jbaron@akamai.com, rostedt@goodmis.org, jpoimboe@kernel.org Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Inserts a direct branch to the stub target when possible, replacing the mtctr/btctr sequence. The load into r12 could potentially be skipped too, but that change would need to refactor the arguments to indicate that the address does not have a separate local entry point. This helps the static call implementation, where modules calling their own trampolines are called through this stub and the trampoline is easily within range of a direct branch. Signed-off-by: Benjamin Gray --- arch/powerpc/kernel/module_64.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/arch/powerpc/kernel/module_64.c b/arch/powerpc/kernel/module_64.c index 4d816f7785b4..745ce9097dcf 100644 --- a/arch/powerpc/kernel/module_64.c +++ b/arch/powerpc/kernel/module_64.c @@ -141,6 +141,12 @@ static u32 ppc64_stub_insns[] = { PPC_RAW_BCTR(), }; +#ifdef CONFIG_PPC64_ELF_ABI_V1 +#define PPC64_STUB_MTCTR_OFFSET 5 +#else +#define PPC64_STUB_MTCTR_OFFSET 4 +#endif + /* Count how many different 24-bit relocations (different symbol, different addend) */ static unsigned int count_relocs(const Elf64_Rela *rela, unsigned int num) @@ -429,6 +435,8 @@ static inline int create_stub(const Elf64_Shdr *sechdrs, long reladdr; func_desc_t desc; int i; + u32 *jump_seq_addr = &entry->jump[PPC64_STUB_MTCTR_OFFSET]; + ppc_inst_t direct; if (is_mprofile_ftrace_call(name)) return create_ftrace_stub(entry, addr, me); @@ -439,6 +447,11 @@ static inline int create_stub(const Elf64_Shdr *sechdrs, return 0; } + /* Replace indirect branch sequence with direct branch where possible */ + if (!create_branch(&direct, jump_seq_addr, addr, 0)) + if (patch_instruction(jump_seq_addr, direct)) + return 0; + /* Stub uses address relative to r2. */ reladdr = (unsigned long)entry - my_r2(sechdrs, me); if (reladdr > 0x7FFFFFFF || reladdr < -(0x80000000L)) { From patchwork Mon Sep 26 06:43:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Gray X-Patchwork-Id: 1682354 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=ZLJYLPl3; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MbYGc3M9cz1ypH for ; Mon, 26 Sep 2022 16:47:36 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4MbYGX4g7mz3f2s for ; Mon, 26 Sep 2022 16:47:32 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=ZLJYLPl3; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=bgray@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=ZLJYLPl3; dkim-atps=neutral Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4MbYCM1TXjz2xvJ for ; Mon, 26 Sep 2022 16:44:47 +1000 (AEST) Received: from pps.filterd (m0187473.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 28Q4ZRB6020407; Mon, 26 Sep 2022 06:44:01 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=9zEaH0k5pQzIL+oDYeijzfgTk3gU9Gj6CrTIUIx03i4=; b=ZLJYLPl3zJB23cd1erCUMH25CU6LvDRylhM0oGFW/Bl9ODtgQJwINCzDU+FDwQIxpJ0t Y4uyqUmXbYL0EWQknG+Y+3mPe37LbjzFAr3kx3AWXUwW7Teplsteb0Puxi4l1j7icknl WTHnUxAgMOgjTQjDaRqDgsws9ONptSB98h4A1Lm5VEg7iyuUwcaOwhxUorJR7uhfZiYr +4EOhpauQoWcwSM5ujWx0XKk7Enf16BLn/M+I0HWt3ocGjlxurd9c78FKI1b8WmzZTNR UEqUbF9NVFVrWw6IwYDgZE8e180NF23F2OXbAwRmVqdAdqEJqI5oPDo+DQWdTEPkLvx4 Hg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3jtbvf4h8b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 26 Sep 2022 06:44:01 +0000 Received: from m0187473.ppops.net (m0187473.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 28Q5RbNf026269; Mon, 26 Sep 2022 06:44:00 GMT Received: from ppma06ams.nl.ibm.com (66.31.33a9.ip4.static.sl-reverse.com [169.51.49.102]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3jtbvf4h7h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 26 Sep 2022 06:44:00 +0000 Received: from pps.filterd (ppma06ams.nl.ibm.com [127.0.0.1]) by ppma06ams.nl.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 28Q6ZM8O023324; Mon, 26 Sep 2022 06:43:58 GMT Received: from b06cxnps4075.portsmouth.uk.ibm.com (d06relay12.portsmouth.uk.ibm.com [9.149.109.197]) by ppma06ams.nl.ibm.com with ESMTP id 3jss5j20fv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 26 Sep 2022 06:43:58 +0000 Received: from d06av26.portsmouth.uk.ibm.com (d06av26.portsmouth.uk.ibm.com [9.149.105.62]) by b06cxnps4075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 28Q6huRw63373708 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 26 Sep 2022 06:43:56 GMT Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E49A8AE045; Mon, 26 Sep 2022 06:43:55 +0000 (GMT) Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 32426AE04D; Mon, 26 Sep 2022 06:43:55 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av26.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 26 Sep 2022 06:43:55 +0000 (GMT) Received: from beng-ozlabs-ibm-com.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 044DB60593; Mon, 26 Sep 2022 16:43:54 +1000 (AEST) From: Benjamin Gray To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v2 4/6] static_call: Move static call selftest to static_call_selftest.c Date: Mon, 26 Sep 2022 16:43:14 +1000 Message-Id: <20220926064316.765967-5-bgray@linux.ibm.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220926064316.765967-1-bgray@linux.ibm.com> References: <20220926064316.765967-1-bgray@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: Xqs2FmEPUG_Yi7NBEHAbXobE34nGPyGp X-Proofpoint-ORIG-GUID: -A5bfWcKz4E1ADS6one7XUcKc2rELFp2 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1 definitions=2022-09-26_04,2022-09-22_02,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 lowpriorityscore=0 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 mlxlogscore=999 impostorscore=0 mlxscore=0 priorityscore=1501 spamscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2209130000 definitions=main-2209260039 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: christophe.leroy@c-s.fr, ajd@linux.ibm.com, peterz@infradead.org, Benjamin Gray , npiggin@gmail.com, ardb@kernel.org, jbaron@akamai.com, rostedt@goodmis.org, jpoimboe@kernel.org Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" These tests are out-of-line only, so moving them to the their own file allows them to be run when an arch does not implement inline static calls. Signed-off-by: Benjamin Gray --- kernel/Makefile | 1 + kernel/static_call_inline.c | 43 ----------------------------------- kernel/static_call_selftest.c | 41 +++++++++++++++++++++++++++++++++ 3 files changed, 42 insertions(+), 43 deletions(-) create mode 100644 kernel/static_call_selftest.c diff --git a/kernel/Makefile b/kernel/Makefile index 318789c728d3..8ce8beaa3cc0 100644 --- a/kernel/Makefile +++ b/kernel/Makefile @@ -113,6 +113,7 @@ obj-$(CONFIG_KCSAN) += kcsan/ obj-$(CONFIG_SHADOW_CALL_STACK) += scs.o obj-$(CONFIG_HAVE_STATIC_CALL) += static_call.o obj-$(CONFIG_HAVE_STATIC_CALL_INLINE) += static_call_inline.o +obj-$(CONFIG_STATIC_CALL_SELFTEST) += static_call_selftest.o obj-$(CONFIG_CFI_CLANG) += cfi.o obj-$(CONFIG_PERF_EVENTS) += events/ diff --git a/kernel/static_call_inline.c b/kernel/static_call_inline.c index dc5665b62814..64d04d054698 100644 --- a/kernel/static_call_inline.c +++ b/kernel/static_call_inline.c @@ -498,46 +498,3 @@ int __init static_call_init(void) return 0; } early_initcall(static_call_init); - -#ifdef CONFIG_STATIC_CALL_SELFTEST - -static int func_a(int x) -{ - return x+1; -} - -static int func_b(int x) -{ - return x+2; -} - -DEFINE_STATIC_CALL(sc_selftest, func_a); - -static struct static_call_data { - int (*func)(int); - int val; - int expect; -} static_call_data [] __initdata = { - { NULL, 2, 3 }, - { func_b, 2, 4 }, - { func_a, 2, 3 } -}; - -static int __init test_static_call_init(void) -{ - int i; - - for (i = 0; i < ARRAY_SIZE(static_call_data); i++ ) { - struct static_call_data *scd = &static_call_data[i]; - - if (scd->func) - static_call_update(sc_selftest, scd->func); - - WARN_ON(static_call(sc_selftest)(scd->val) != scd->expect); - } - - return 0; -} -early_initcall(test_static_call_init); - -#endif /* CONFIG_STATIC_CALL_SELFTEST */ diff --git a/kernel/static_call_selftest.c b/kernel/static_call_selftest.c new file mode 100644 index 000000000000..246ad89f64eb --- /dev/null +++ b/kernel/static_call_selftest.c @@ -0,0 +1,41 @@ +// SPDX-License-Identifier: GPL-2.0 +#include + +static int func_a(int x) +{ + return x+1; +} + +static int func_b(int x) +{ + return x+2; +} + +DEFINE_STATIC_CALL(sc_selftest, func_a); + +static struct static_call_data { + int (*func)(int); + int val; + int expect; +} static_call_data [] __initdata = { + { NULL, 2, 3 }, + { func_b, 2, 4 }, + { func_a, 2, 3 } +}; + +static int __init test_static_call_init(void) +{ + int i; + + for (i = 0; i < ARRAY_SIZE(static_call_data); i++ ) { + struct static_call_data *scd = &static_call_data[i]; + + if (scd->func) + static_call_update(sc_selftest, scd->func); + + WARN_ON(static_call(sc_selftest)(scd->val) != scd->expect); + } + + return 0; +} +early_initcall(test_static_call_init); From patchwork Mon Sep 26 06:43:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Gray X-Patchwork-Id: 1682356 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=bAXIREln; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MbYJ10fp4z1ypH for ; Mon, 26 Sep 2022 16:48:48 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4MbYJ05spzz3c6S for ; Mon, 26 Sep 2022 16:48:48 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=bAXIREln; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.156.1; helo=mx0a-001b2d01.pphosted.com; envelope-from=bgray@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=bAXIREln; dkim-atps=neutral Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4MbYCQ0fPfz30MT for ; Mon, 26 Sep 2022 16:44:49 +1000 (AEST) Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 28Q4ZOMP024487; Mon, 26 Sep 2022 06:44:02 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=s+dyVnTUUL+TV0F1U7ucCK83xOS/uKhnZDux4PKWMVA=; b=bAXIRElnx2Mkk4ltPA75rEgZUk3h1XQROStCm+DQnPh+o1rfBro1qkZntWwh2b9j2M9v z8j5dC3U7Wiudxw240HAAdqsWmm6l7n7UTkIvefQnbQYCtWdS3DevaljeWEOQq1/f87m idtVaHaGeCKkZ56S7eBxvLe3YKHHmlUz0Ypc0Q/dDuPRcYN/w3jU4Wj7rzOp+kL5YOjf yEe8aiBT2p1pZT1YXXtbA87TMeXsgn61vLMVTIZhN+RysEALTorNbqwp+eikLveNIQoK 7MFXTAmQq3rLuIS6yZFE2LzvRnBRlknQnCYZwpxwpt4C89OSgMuS+04lYZxi91JVAIwe Zw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3jtb6u57k0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 26 Sep 2022 06:44:02 +0000 Received: from m0098409.ppops.net (m0098409.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 28Q6i1Er030474; Mon, 26 Sep 2022 06:44:01 GMT Received: from ppma06ams.nl.ibm.com (66.31.33a9.ip4.static.sl-reverse.com [169.51.49.102]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3jtb6u57j9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 26 Sep 2022 06:44:01 +0000 Received: from pps.filterd (ppma06ams.nl.ibm.com [127.0.0.1]) by ppma06ams.nl.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 28Q6ZOxa023332; Mon, 26 Sep 2022 06:43:59 GMT Received: from b06cxnps3074.portsmouth.uk.ibm.com (d06relay09.portsmouth.uk.ibm.com [9.149.109.194]) by ppma06ams.nl.ibm.com with ESMTP id 3jss5j20fx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 26 Sep 2022 06:43:59 +0000 Received: from d06av26.portsmouth.uk.ibm.com (d06av26.portsmouth.uk.ibm.com [9.149.105.62]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 28Q6hvCN47579452 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 26 Sep 2022 06:43:57 GMT Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E9E96AE051; Mon, 26 Sep 2022 06:43:56 +0000 (GMT) Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 46C1FAE045; Mon, 26 Sep 2022 06:43:56 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av26.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 26 Sep 2022 06:43:56 +0000 (GMT) Received: from beng-ozlabs-ibm-com.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 6C494605A5; Mon, 26 Sep 2022 16:43:54 +1000 (AEST) From: Benjamin Gray To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v2 5/6] powerpc/64: Add support for out-of-line static calls Date: Mon, 26 Sep 2022 16:43:15 +1000 Message-Id: <20220926064316.765967-6-bgray@linux.ibm.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220926064316.765967-1-bgray@linux.ibm.com> References: <20220926064316.765967-1-bgray@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: TyEHX6T5KA-HqW3QG7fzGRrkxWjDicVV X-Proofpoint-GUID: mYflydTudXsyrA5KR3kas7B91c6gF9RV X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1 definitions=2022-09-26_04,2022-09-22_02,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 clxscore=1015 priorityscore=1501 mlxlogscore=999 spamscore=0 lowpriorityscore=0 suspectscore=0 mlxscore=0 impostorscore=0 phishscore=0 adultscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2209130000 definitions=main-2209260039 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: christophe.leroy@c-s.fr, ajd@linux.ibm.com, peterz@infradead.org, Benjamin Gray , npiggin@gmail.com, ardb@kernel.org, jbaron@akamai.com, rostedt@goodmis.org, jpoimboe@kernel.org Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Implement static call support for 64 bit V2 ABI. This requires making sure the TOC is kept correct across kernel-module boundaries. As a secondary concern, it tries to use the local entry point of a target wherever possible. It does so by checking if both tramp & target are kernel code, and falls back to detecting the common global entry point patterns if modules are involved. Detecting the global entry point is also required for setting the local entry point as the trampoline target: if we cannot detect the local entry point, then we need to convservatively initialise r12 and use the global entry point. The trampolines are marked with `.localentry NAME, 1` to make the linker save and restore the TOC on each call to the trampoline. This allows the trampoline to safely target functions with different TOC values. However this directive also implies the TOC is not initialised on entry to the trampoline. The kernel TOC is easily found in the PACA, but not an arbitrary module TOC. Therefore the trampoline implementation depends on whether it's in the kernel or not. If in the kernel, we initialise the TOC using the PACA. If in a module, we have to initialise the TOC with zero context, so it's quite expensive. Signed-off-by: Benjamin Gray --- arch/powerpc/Kconfig | 2 +- arch/powerpc/include/asm/code-patching.h | 1 + arch/powerpc/include/asm/static_call.h | 80 +++++++++++++++++++-- arch/powerpc/kernel/Makefile | 3 +- arch/powerpc/kernel/static_call.c | 90 ++++++++++++++++++++++-- 5 files changed, 164 insertions(+), 12 deletions(-) diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 4c466acdc70d..e7a66635eade 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -248,7 +248,7 @@ config PPC select HAVE_SOFTIRQ_ON_OWN_STACK select HAVE_STACKPROTECTOR if PPC32 && $(cc-option,-mstack-protector-guard=tls -mstack-protector-guard-reg=r2) select HAVE_STACKPROTECTOR if PPC64 && $(cc-option,-mstack-protector-guard=tls -mstack-protector-guard-reg=r13) - select HAVE_STATIC_CALL if PPC32 + select HAVE_STATIC_CALL if PPC32 || PPC64_ELF_ABI_V2 select HAVE_SYSCALL_TRACEPOINTS select HAVE_VIRT_CPU_ACCOUNTING select HUGETLB_PAGE_SIZE_VARIABLE if PPC_BOOK3S_64 && HUGETLB_PAGE diff --git a/arch/powerpc/include/asm/code-patching.h b/arch/powerpc/include/asm/code-patching.h index 15efd8ab22da..8d1850080af8 100644 --- a/arch/powerpc/include/asm/code-patching.h +++ b/arch/powerpc/include/asm/code-patching.h @@ -132,6 +132,7 @@ int translate_branch(ppc_inst_t *instr, const u32 *dest, const u32 *src); bool is_conditional_branch(ppc_inst_t instr); #define OP_RT_RA_MASK 0xffff0000UL +#define OP_SI_MASK 0x0000ffffUL #define LIS_R2 (PPC_RAW_LIS(_R2, 0)) #define ADDIS_R2_R12 (PPC_RAW_ADDIS(_R2, _R12, 0)) #define ADDI_R2_R2 (PPC_RAW_ADDI(_R2, _R2, 0)) diff --git a/arch/powerpc/include/asm/static_call.h b/arch/powerpc/include/asm/static_call.h index de1018cc522b..3d6e82200cb7 100644 --- a/arch/powerpc/include/asm/static_call.h +++ b/arch/powerpc/include/asm/static_call.h @@ -2,12 +2,75 @@ #ifndef _ASM_POWERPC_STATIC_CALL_H #define _ASM_POWERPC_STATIC_CALL_H +#ifdef CONFIG_PPC64_ELF_ABI_V2 + +#ifdef MODULE + +#define __PPC_SCT(name, inst) \ + asm(".pushsection .text, \"ax\" \n" \ + ".align 6 \n" \ + ".globl " STATIC_CALL_TRAMP_STR(name) " \n" \ + ".localentry " STATIC_CALL_TRAMP_STR(name) ", 1 \n" \ + STATIC_CALL_TRAMP_STR(name) ": \n" \ + " mflr 11 \n" \ + " bcl 20, 31, $+4 \n" \ + "0: mflr 12 \n" \ + " mtlr 11 \n" \ + " addi 12, 12, (" STATIC_CALL_TRAMP_STR(name) " - 0b) \n" \ + " addis 2, 12, (.TOC.-" STATIC_CALL_TRAMP_STR(name) ")@ha \n" \ + " addi 2, 2, (.TOC.-" STATIC_CALL_TRAMP_STR(name) ")@l \n" \ + " " inst " \n" \ + " ld 12, (2f - " STATIC_CALL_TRAMP_STR(name) ")(12) \n" \ + " mtctr 12 \n" \ + " bctr \n" \ + "1: li 3, 0 \n" \ + " blr \n" \ + ".balign 8 \n" \ + "2: .8byte 0 \n" \ + ".type " STATIC_CALL_TRAMP_STR(name) ", @function \n" \ + ".size " STATIC_CALL_TRAMP_STR(name) ", . - " STATIC_CALL_TRAMP_STR(name) " \n" \ + ".popsection \n") + +#else /* KERNEL */ + +#define __PPC_SCT(name, inst) \ + asm(".pushsection .text, \"ax\" \n" \ + ".align 5 \n" \ + ".globl " STATIC_CALL_TRAMP_STR(name) " \n" \ + ".localentry " STATIC_CALL_TRAMP_STR(name) ", 1 \n" \ + STATIC_CALL_TRAMP_STR(name) ": \n" \ + " ld 2, 16(13) \n" \ + " " inst " \n" \ + " addis 12, 2, 2f@toc@ha \n" \ + " ld 12, 2f@toc@l(12) \n" \ + " mtctr 12 \n" \ + " bctr \n" \ + "1: li 3, 0 \n" \ + " blr \n" \ + ".balign 8 \n" \ + "2: .8byte 0 \n" \ + ".type " STATIC_CALL_TRAMP_STR(name) ", @function \n" \ + ".size " STATIC_CALL_TRAMP_STR(name) ", . - " STATIC_CALL_TRAMP_STR(name) " \n" \ + ".popsection \n") + +#endif /* MODULE */ + +#define PPC_SCT_INST_MODULE 28 /* Offset of instruction to update */ +#define PPC_SCT_RET0_MODULE 44 /* Offset of label 1 */ +#define PPC_SCT_DATA_MODULE 56 /* Offset of label 2 (aligned) */ + +#define PPC_SCT_INST_KERNEL 4 /* Offset of instruction to update */ +#define PPC_SCT_RET0_KERNEL 24 /* Offset of label 1 */ +#define PPC_SCT_DATA_KERNEL 32 /* Offset of label 2 (aligned) */ + +#elif defined(CONFIG_PPC32) + #define __PPC_SCT(name, inst) \ asm(".pushsection .text, \"ax\" \n" \ ".align 5 \n" \ ".globl " STATIC_CALL_TRAMP_STR(name) " \n" \ STATIC_CALL_TRAMP_STR(name) ": \n" \ - inst " \n" \ + " " inst " \n" \ " lis 12,2f@ha \n" \ " lwz 12,2f@l(12) \n" \ " mtctr 12 \n" \ @@ -19,11 +82,20 @@ ".size " STATIC_CALL_TRAMP_STR(name) ", . - " STATIC_CALL_TRAMP_STR(name) " \n" \ ".popsection \n") -#define PPC_SCT_RET0 20 /* Offset of label 1 */ -#define PPC_SCT_DATA 28 /* Offset of label 2 */ +#define PPC_SCT_INST_MODULE 0 /* Offset of instruction to update */ +#define PPC_SCT_RET0_MODULE 20 /* Offset of label 1 */ +#define PPC_SCT_DATA_MODULE 28 /* Offset of label 2 */ + +#define PPC_SCT_INST_KERNEL PPC_SCT_INST_MODULE +#define PPC_SCT_RET0_KERNEL PPC_SCT_RET0_MODULE +#define PPC_SCT_DATA_KERNEL PPC_SCT_DATA_MODULE + +#else /* !CONFIG_PPC64_ELF_ABI_V2 && !CONFIG_PPC32 */ +#error "Unsupported ABI" +#endif /* CONFIG_PPC64_ELF_ABI_V2 */ #define ARCH_DEFINE_STATIC_CALL_TRAMP(name, func) __PPC_SCT(name, "b " #func) #define ARCH_DEFINE_STATIC_CALL_NULL_TRAMP(name) __PPC_SCT(name, "blr") -#define ARCH_DEFINE_STATIC_CALL_RET0_TRAMP(name) __PPC_SCT(name, "b .+20") +#define ARCH_DEFINE_STATIC_CALL_RET0_TRAMP(name) __PPC_SCT(name, "b 1f") #endif /* _ASM_POWERPC_STATIC_CALL_H */ diff --git a/arch/powerpc/kernel/Makefile b/arch/powerpc/kernel/Makefile index 06d2d1f78f71..a30d0d0f5499 100644 --- a/arch/powerpc/kernel/Makefile +++ b/arch/powerpc/kernel/Makefile @@ -128,8 +128,9 @@ extra-y += vmlinux.lds obj-$(CONFIG_RELOCATABLE) += reloc_$(BITS).o -obj-$(CONFIG_PPC32) += entry_32.o setup_32.o early_32.o static_call.o +obj-$(CONFIG_PPC32) += entry_32.o setup_32.o early_32.o obj-$(CONFIG_PPC64) += dma-iommu.o iommu.o +obj-$(CONFIG_HAVE_STATIC_CALL) += static_call.o obj-$(CONFIG_KGDB) += kgdb.o obj-$(CONFIG_BOOTX_TEXT) += btext.o obj-$(CONFIG_SMP) += smp.o diff --git a/arch/powerpc/kernel/static_call.c b/arch/powerpc/kernel/static_call.c index 863a7aa24650..ecbb74e1b4d3 100644 --- a/arch/powerpc/kernel/static_call.c +++ b/arch/powerpc/kernel/static_call.c @@ -4,30 +4,108 @@ #include +static void* ppc_function_toc(u32 *func) +{ +#ifdef CONFIG_PPC64_ELF_ABI_V2 + u32 insn1 = *func; + u32 insn2 = *(func+1); + u64 si1 = sign_extend64((insn1 & OP_SI_MASK) << 16, 31); + u64 si2 = sign_extend64(insn2 & OP_SI_MASK, 15); + u64 addr = ((u64) func + si1) + si2; + + if ((((insn1 & OP_RT_RA_MASK) == ADDIS_R2_R12) || + ((insn1 & OP_RT_RA_MASK) == LIS_R2)) && + ((insn2 & OP_RT_RA_MASK) == ADDI_R2_R2)) + return (void*)addr; +#endif + return NULL; +} + +static bool shares_toc(void *func1, void *func2) +{ + if (IS_ENABLED(CONFIG_PPC64_ELF_ABI_V2)) { + void* func1_toc; + void* func2_toc; + + if (func1 == NULL || func2 == NULL) + return false; + + /* Assume the kernel only uses a single TOC */ + if (core_kernel_text((unsigned long)func1) && + core_kernel_text((unsigned long)func2)) + return true; + + /* Fall back to calculating the TOC from common patterns + * if modules are involved + */ + func1_toc = ppc_function_toc(func1); + func2_toc = ppc_function_toc(func2); + return func1_toc != NULL && func2_toc != NULL && (func1_toc == func2_toc); + } + + return true; +} + +static void* get_inst_addr(void *tramp) +{ + return tramp + (core_kernel_text((unsigned long)tramp) + ? PPC_SCT_INST_KERNEL + : PPC_SCT_INST_MODULE); +} + +static void* get_ret0_addr(void* tramp) +{ + return tramp + (core_kernel_text((unsigned long)tramp) + ? PPC_SCT_RET0_KERNEL + : PPC_SCT_RET0_MODULE); +} + +static void* get_data_addr(void *tramp) +{ + return tramp + (core_kernel_text((unsigned long) tramp) + ? PPC_SCT_DATA_KERNEL + : PPC_SCT_DATA_MODULE); +} + void arch_static_call_transform(void *site, void *tramp, void *func, bool tail) { int err; bool is_ret0 = (func == __static_call_return0); - unsigned long target = (unsigned long)(is_ret0 ? tramp + PPC_SCT_RET0 : func); - bool is_short = is_offset_in_branch_range((long)target - (long)tramp); + bool is_short; + void* target = is_ret0 ? get_ret0_addr(tramp) : func; + void* tramp_inst = get_inst_addr(tramp); if (!tramp) return; + if (is_ret0) + is_short = true; + else if (shares_toc(tramp, target)) + is_short = is_offset_in_branch_range( + (long)ppc_function_entry(target) - (long)tramp_inst); + else + /* Combine out-of-range with not sharing a TOC. Though it's possible an + * out-of-range target shares a TOC, handling this separately complicates + * the trampoline. It's simpler to always use the global entry point + * in this case. + */ + is_short = false; + mutex_lock(&text_mutex); if (func && !is_short) { - err = patch_instruction(tramp + PPC_SCT_DATA, ppc_inst(target)); + err = patch_memory(get_data_addr(tramp), target); if (err) goto out; } if (!func) - err = patch_instruction(tramp, ppc_inst(PPC_RAW_BLR())); + err = patch_instruction(tramp_inst, ppc_inst(PPC_RAW_BLR())); else if (is_short) - err = patch_branch(tramp, target, 0); + err = patch_branch(tramp_inst, ppc_function_entry(target), 0); else - err = patch_instruction(tramp, ppc_inst(PPC_RAW_NOP())); + err = patch_instruction(tramp_inst, ppc_inst(PPC_RAW_NOP())); + out: mutex_unlock(&text_mutex); From patchwork Mon Sep 26 06:43:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Benjamin Gray X-Patchwork-Id: 1682353 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=RAId01Zj; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4MbYFg0KGsz1ypH for ; Mon, 26 Sep 2022 16:46:47 +1000 (AEST) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4MbYFf6Xmbz3cjH for ; Mon, 26 Sep 2022 16:46:46 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=RAId01Zj; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=linux.ibm.com (client-ip=148.163.158.5; helo=mx0b-001b2d01.pphosted.com; envelope-from=bgray@linux.ibm.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=ibm.com header.i=@ibm.com header.a=rsa-sha256 header.s=pp1 header.b=RAId01Zj; dkim-atps=neutral Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4MbYCB43pzz30NS for ; Mon, 26 Sep 2022 16:44:38 +1000 (AEST) Received: from pps.filterd (m0127361.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 28Q4eHYI024999; Mon, 26 Sep 2022 06:44:02 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=9t0sGygSVuHlwZYJ1tqe7kFd9avIfZ0zik3DGzyiGh8=; b=RAId01ZjIy+iEsSvM2j5cs9oR/ZwuD/H2HYfYYXNaxlbyVo+tKuhrm24JsCqA+KCZ3E3 RFZ6lkmC0nv0MHTBa3FwV9FAnjDd0QF0UmxaNctsefSa9GW4NQWmQtU6DmUr7OJJL8HA un+JI/mq6dFGR/bVpd0mp6xSKHcTRBysD4vRtY+kUlYAHQhu0YYVWF4dYSYafFqXeeMR 1+N6xmfRIkWy1M8863mu6e74KxubMMUIk8vkqgYYvqWPX4ER2zAtKwGeOaIiXD1SHxj/ gkDSz6whG3nqgjuG90+K6+CF63jc3Fbu4hQiCT/I4rt9TCfFNF5HBBfFnrdQJ9kG2OCy Vg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3jtbxr41xc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 26 Sep 2022 06:44:02 +0000 Received: from m0127361.ppops.net (m0127361.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 28Q6i1hg030636; Mon, 26 Sep 2022 06:44:01 GMT Received: from ppma03fra.de.ibm.com (6b.4a.5195.ip4.static.sl-reverse.com [149.81.74.107]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3jtbxr41wv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 26 Sep 2022 06:44:01 +0000 Received: from pps.filterd (ppma03fra.de.ibm.com [127.0.0.1]) by ppma03fra.de.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 28Q6auJj004515; Mon, 26 Sep 2022 06:44:00 GMT Received: from b06cxnps3074.portsmouth.uk.ibm.com (d06relay09.portsmouth.uk.ibm.com [9.149.109.194]) by ppma03fra.de.ibm.com with ESMTP id 3jssh91fvb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 26 Sep 2022 06:43:59 +0000 Received: from d06av26.portsmouth.uk.ibm.com (d06av26.portsmouth.uk.ibm.com [9.149.105.62]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 28Q6hvtx47579458 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 26 Sep 2022 06:43:57 GMT Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 83EB5AE045; Mon, 26 Sep 2022 06:43:57 +0000 (GMT) Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D8AC7AE04D; Mon, 26 Sep 2022 06:43:56 +0000 (GMT) Received: from ozlabs.au.ibm.com (unknown [9.192.253.14]) by d06av26.portsmouth.uk.ibm.com (Postfix) with ESMTP; Mon, 26 Sep 2022 06:43:56 +0000 (GMT) Received: from beng-ozlabs-ibm-com.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id AE39760213; Mon, 26 Sep 2022 16:43:55 +1000 (AEST) From: Benjamin Gray To: linuxppc-dev@lists.ozlabs.org Subject: [PATCH v2 6/6] powerpc/64: Add tests for out-of-line static calls Date: Mon, 26 Sep 2022 16:43:16 +1000 Message-Id: <20220926064316.765967-7-bgray@linux.ibm.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220926064316.765967-1-bgray@linux.ibm.com> References: <20220926064316.765967-1-bgray@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: 16uBlOYwN0r2QzvAlsi1IXl3K6s94NCw X-Proofpoint-GUID: N-x4_dFEEHxNKWBG0U8fvj0aF4e77vwR X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1 definitions=2022-09-26_04,2022-09-22_02,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 mlxscore=0 spamscore=0 clxscore=1015 phishscore=0 lowpriorityscore=0 bulkscore=0 priorityscore=1501 mlxlogscore=999 suspectscore=0 impostorscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2209130000 definitions=main-2209260039 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: christophe.leroy@c-s.fr, ajd@linux.ibm.com, peterz@infradead.org, Benjamin Gray , npiggin@gmail.com, ardb@kernel.org, jbaron@akamai.com, rostedt@goodmis.org, jpoimboe@kernel.org Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" KUnit tests for the various combinations of caller/trampoline/target and kernel/module. They must be run from a module loaded at runtime to guarantee they have a different TOC to the kernel. The tests try to mitigate the chance of panicing by restoring the TOC after every static call. Not all possible errors can be caught by this (we can't stop a trampoline from using a bad TOC itself), but it makes certain errors easier to debug. Signed-off-by: Benjamin Gray --- arch/powerpc/Kconfig | 10 + arch/powerpc/kernel/Makefile | 1 + arch/powerpc/kernel/static_call.c | 61 ++++++ arch/powerpc/kernel/static_call_test.c | 251 +++++++++++++++++++++++++ arch/powerpc/kernel/static_call_test.h | 56 ++++++ 5 files changed, 379 insertions(+) create mode 100644 arch/powerpc/kernel/static_call_test.c create mode 100644 arch/powerpc/kernel/static_call_test.h diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index e7a66635eade..0ca60514c0e2 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -1023,6 +1023,16 @@ config PPC_RTAS_FILTER Say Y unless you know what you are doing and the filter is causing problems for you. +config PPC_STATIC_CALL_KUNIT_TEST + tristate "KUnit tests for PPC64 ELF ABI V2 static calls" + default KUNIT_ALL_TESTS + depends on HAVE_STATIC_CALL && PPC64_ELF_ABI_V2 && KUNIT && m + help + Tests that check the TOC is kept consistent across all combinations + of caller/trampoline/target being kernel/module. Must be built as a + module and loaded at runtime to ensure the module has a different + TOC to the kernel. + endmenu config ISA_DMA_API diff --git a/arch/powerpc/kernel/Makefile b/arch/powerpc/kernel/Makefile index a30d0d0f5499..22c07e3d34df 100644 --- a/arch/powerpc/kernel/Makefile +++ b/arch/powerpc/kernel/Makefile @@ -131,6 +131,7 @@ obj-$(CONFIG_RELOCATABLE) += reloc_$(BITS).o obj-$(CONFIG_PPC32) += entry_32.o setup_32.o early_32.o obj-$(CONFIG_PPC64) += dma-iommu.o iommu.o obj-$(CONFIG_HAVE_STATIC_CALL) += static_call.o +obj-$(CONFIG_PPC_STATIC_CALL_KUNIT_TEST) += static_call_test.o obj-$(CONFIG_KGDB) += kgdb.o obj-$(CONFIG_BOOTX_TEXT) += btext.o obj-$(CONFIG_SMP) += smp.o diff --git a/arch/powerpc/kernel/static_call.c b/arch/powerpc/kernel/static_call.c index ecbb74e1b4d3..8d338917b70e 100644 --- a/arch/powerpc/kernel/static_call.c +++ b/arch/powerpc/kernel/static_call.c @@ -113,3 +113,64 @@ void arch_static_call_transform(void *site, void *tramp, void *func, bool tail) panic("%s: patching failed %pS at %pS\n", __func__, func, tramp); } EXPORT_SYMBOL_GPL(arch_static_call_transform); + + +#if IS_MODULE(CONFIG_PPC_STATIC_CALL_KUNIT_TEST) + +#include "static_call_test.h" + +int ppc_sc_kernel_target_1(struct kunit* test) +{ + toc_fixup(test); + return 1; +} + +int ppc_sc_kernel_target_2(struct kunit* test) +{ + toc_fixup(test); + return 2; +} + +DEFINE_STATIC_CALL(ppc_sc_kernel, ppc_sc_kernel_target_1); + +int ppc_sc_kernel_call(struct kunit* test) +{ + return PROTECTED_SC(test, int, static_call(ppc_sc_kernel)(test)); +} + +int ppc_sc_kernel_call_indirect(struct kunit* test, int (*fn)(struct kunit*)) +{ + return PROTECTED_SC(test, int, fn(test)); +} + +long ppc_sc_kernel_target_big(struct kunit* test, + long a, + long b, + long c, + long d, + long e, + long f, + long g, + long h, + long i) +{ + toc_fixup(test); + KUNIT_EXPECT_EQ(test, a, b); + KUNIT_EXPECT_EQ(test, a, c); + KUNIT_EXPECT_EQ(test, a, d); + KUNIT_EXPECT_EQ(test, a, e); + KUNIT_EXPECT_EQ(test, a, f); + KUNIT_EXPECT_EQ(test, a, g); + KUNIT_EXPECT_EQ(test, a, h); + KUNIT_EXPECT_EQ(test, a, i); + return ~a; +} + +EXPORT_SYMBOL_GPL(ppc_sc_kernel_target_1); +EXPORT_SYMBOL_GPL(ppc_sc_kernel_target_2); +EXPORT_SYMBOL_GPL(ppc_sc_kernel_target_big); +EXPORT_STATIC_CALL_GPL(ppc_sc_kernel); +EXPORT_SYMBOL_GPL(ppc_sc_kernel_call); +EXPORT_SYMBOL_GPL(ppc_sc_kernel_call_indirect); + +#endif /* IS_MODULE(CONFIG_PPC_STATIC_CALL_KUNIT_TEST) */ diff --git a/arch/powerpc/kernel/static_call_test.c b/arch/powerpc/kernel/static_call_test.c new file mode 100644 index 000000000000..2d69524d935f --- /dev/null +++ b/arch/powerpc/kernel/static_call_test.c @@ -0,0 +1,251 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include "static_call_test.h" + +#include +#include +#include + +/* + * Tests to ensure correctness in a variety of cases for static calls. + * + * The tests focus on ensuring the TOC is kept consistent across the + * module-kernel boundary, as compilers can't see that a trampoline + * defined locally to a caller might be jumping to a function with a + * different TOC. So it's important that these tests are compiled as + * a module to ensure the TOC will be different to the kernel's. + */ + +/* Utils to hold a copy of the old register values while we test. + * + * We can't use the KUnit init/exit hooks because when the hooks and + * test cases return they will be in the KUnit context that doesn't know + * we've reserved and modified some non-volatile registers. + */ +static void* regsaves[3]; + +#define SAVE_REGS() \ + regsaves[0] = actual_toc; \ + regsaves[1] = module_toc; \ + regsaves[2] = kernel_toc; \ + module_toc = current_toc; \ + kernel_toc = (void*) get_paca()->kernel_toc; + +#define RESTORE_REGS() \ + actual_toc = regsaves[0]; \ + module_toc = regsaves[1]; \ + kernel_toc = regsaves[2]; + +static int module_target_11(struct kunit *test) +{ + toc_fixup(test); + return 11; +} + +static int module_target_12(struct kunit *test) +{ + toc_fixup(test); + return 12; +} + +DEFINE_STATIC_CALL(module_sc, module_target_11); + +DEFINE_STATIC_CALL_RET0(module_sc_ret0, int(void)); +DEFINE_STATIC_CALL_NULL(module_sc_null, int(int)); + +static int add_one(int *val) +{ + return (*val)++; +} + +static void null_function_test(struct kunit *test) +{ + int val = 0; + + SAVE_REGS(); + + /* Check argument unconditionally evaluated */ + static_call_cond(module_sc_null)(add_one(&val)); + KUNIT_ASSERT_EQ(test, 1, val); + + RESTORE_REGS(); +} + +static void return_zero_test(struct kunit *test) +{ + int ret; + + SAVE_REGS(); + + ret = PROTECTED_SC(test, int, static_call(module_sc_ret0)()); + KUNIT_ASSERT_EQ(test, 0, ret); + + static_call_update(ppc_sc_kernel, (void*)__static_call_return0); + ret = PROTECTED_SC(test, int, static_call(ppc_sc_kernel)(test)); + KUNIT_ASSERT_EQ(test, 0, ret); + + static_call_update(module_sc, (void*)__static_call_return0); + ret = PROTECTED_SC(test, int, static_call(module_sc)(test)); + KUNIT_ASSERT_EQ(test, 0, ret); + + RESTORE_REGS(); +} + +static void kernel_kernel_kernel_test(struct kunit *test) +{ + SAVE_REGS(); + + static_call_update(ppc_sc_kernel, ppc_sc_kernel_target_1); + KUNIT_ASSERT_EQ(test, 1, ppc_sc_kernel_call(test)); + + static_call_update(ppc_sc_kernel, ppc_sc_kernel_target_2); + KUNIT_ASSERT_EQ(test, 2, ppc_sc_kernel_call(test)); + + RESTORE_REGS(); +} + +static void kernel_kernel_module_test(struct kunit *test) +{ + SAVE_REGS(); + + static_call_update(ppc_sc_kernel, module_target_11); + KUNIT_ASSERT_EQ(test, 11, ppc_sc_kernel_call(test)); + + static_call_update(ppc_sc_kernel, module_target_12); + KUNIT_ASSERT_EQ(test, 12, ppc_sc_kernel_call(test)); + + RESTORE_REGS(); +} + +static void kernel_module_kernel_test(struct kunit *test) +{ + SAVE_REGS(); + + static_call_update(module_sc, ppc_sc_kernel_target_1); + KUNIT_ASSERT_EQ(test, 1, ppc_sc_kernel_call_indirect(test, static_call(module_sc))); + + static_call_update(module_sc, ppc_sc_kernel_target_2); + KUNIT_ASSERT_EQ(test, 2, ppc_sc_kernel_call_indirect(test, static_call(module_sc))); + + RESTORE_REGS(); +} + +static void kernel_module_module_test(struct kunit *test) +{ + SAVE_REGS(); + + static_call_update(module_sc, module_target_11); + KUNIT_ASSERT_EQ(test, 11, ppc_sc_kernel_call_indirect(test, static_call(module_sc))); + + static_call_update(module_sc, module_target_12); + KUNIT_ASSERT_EQ(test, 12, ppc_sc_kernel_call_indirect(test, static_call(module_sc))); + + RESTORE_REGS(); +} + +static void module_kernel_kernel_test(struct kunit *test) +{ + int ret; + + SAVE_REGS(); + + static_call_update(ppc_sc_kernel, ppc_sc_kernel_target_1); + ret = PROTECTED_SC(test, int, static_call(ppc_sc_kernel)(test)); + KUNIT_ASSERT_EQ(test, 1, ret); + + static_call_update(ppc_sc_kernel, ppc_sc_kernel_target_2); + ret = PROTECTED_SC(test, int, static_call(ppc_sc_kernel)(test)); + KUNIT_ASSERT_EQ(test, 2, ret); + + RESTORE_REGS(); +} + +static void module_kernel_module_test(struct kunit *test) +{ + int ret; + + SAVE_REGS(); + + static_call_update(ppc_sc_kernel, module_target_11); + ret = PROTECTED_SC(test, int, static_call(ppc_sc_kernel)(test)); + KUNIT_ASSERT_EQ(test, 11, ret); + + static_call_update(ppc_sc_kernel, module_target_12); + ret = PROTECTED_SC(test, int, static_call(ppc_sc_kernel)(test)); + KUNIT_ASSERT_EQ(test, 12, ret); + + RESTORE_REGS(); +} + +static void module_module_kernel_test(struct kunit *test) +{ + int ret; + + SAVE_REGS(); + + static_call_update(module_sc, ppc_sc_kernel_target_1); + ret = PROTECTED_SC(test, int, static_call(module_sc)(test)); + KUNIT_ASSERT_EQ(test, 1, ret); + + static_call_update(module_sc, ppc_sc_kernel_target_2); + ret = PROTECTED_SC(test, int, static_call(module_sc)(test)); + KUNIT_ASSERT_EQ(test, 2, ret); + + RESTORE_REGS(); +} + +static void module_module_module_test(struct kunit *test) +{ + int ret; + + SAVE_REGS(); + + static_call_update(module_sc, module_target_11); + ret = PROTECTED_SC(test, int, static_call(module_sc)(test)); + KUNIT_ASSERT_EQ(test, 11, ret); + + static_call_update(module_sc, module_target_12); + ret = PROTECTED_SC(test, int, static_call(module_sc)(test)); + KUNIT_ASSERT_EQ(test, 12, ret); + + RESTORE_REGS(); +} + +DEFINE_STATIC_CALL(module_sc_stack_params, ppc_sc_kernel_target_big); + +static void stack_parameters_test(struct kunit *test) +{ + long m = 0x1234567887654321; + long ret; + + SAVE_REGS(); + + ret = PROTECTED_SC(test, long, static_call(module_sc_stack_params)(test, m, m, m, m, m, m, m, m, m)); + KUNIT_ASSERT_EQ(test, ~m, ret); + + RESTORE_REGS(); +} + +static struct kunit_case static_call_test_cases[] = { + KUNIT_CASE(null_function_test), + KUNIT_CASE(return_zero_test), + KUNIT_CASE(stack_parameters_test), + KUNIT_CASE(kernel_kernel_kernel_test), + KUNIT_CASE(kernel_kernel_module_test), + KUNIT_CASE(kernel_module_kernel_test), + KUNIT_CASE(kernel_module_module_test), + KUNIT_CASE(module_kernel_kernel_test), + KUNIT_CASE(module_kernel_module_test), + KUNIT_CASE(module_module_kernel_test), + KUNIT_CASE(module_module_module_test), + {} +}; + +static struct kunit_suite ppc_static_call_test_suite = { + .name = "ppc-static-call", + .test_cases = static_call_test_cases, +}; +kunit_test_suite(ppc_static_call_test_suite); + +MODULE_AUTHOR("Benjamin Gray "); +MODULE_LICENSE("GPL"); diff --git a/arch/powerpc/kernel/static_call_test.h b/arch/powerpc/kernel/static_call_test.h new file mode 100644 index 000000000000..597da89297fa --- /dev/null +++ b/arch/powerpc/kernel/static_call_test.h @@ -0,0 +1,56 @@ +#ifndef _POWERPC_STATIC_CALL_TEST_ +#define _POWERPC_STATIC_CALL_TEST_ + +#include + +/* Reserve these registers for testing so that a TOC error + * doesn't necessarily crash the whole kernel. + * + * The tests ensure the contents are restored before returning. + */ +register void* current_toc asm ("r2"); +register void* actual_toc asm ("r14"); /* Copy of TOC before fixup */ +register void* module_toc asm ("r15"); +register void* kernel_toc asm ("r16"); + +DECLARE_STATIC_CALL(ppc_sc_kernel, int(struct kunit*)); +int ppc_sc_kernel_target_1(struct kunit* test); +int ppc_sc_kernel_target_2(struct kunit* test); +long ppc_sc_kernel_target_big(struct kunit* test, + long a, + long b, + long c, + long d, + long e, + long f, + long g, + long h, + long i); +int ppc_sc_kernel_call(struct kunit* test); +int ppc_sc_kernel_call_indirect(struct kunit* test, int(*fn)(struct kunit*)); + +#ifdef MODULE + +#define toc_fixup(test) \ + actual_toc = current_toc; \ + current_toc = module_toc; \ + KUNIT_EXPECT_PTR_EQ(test, module_toc, actual_toc) + +#else /* KERNEL */ + +#define toc_fixup(test) \ + actual_toc = current_toc; \ + current_toc = kernel_toc; \ + KUNIT_EXPECT_PTR_EQ(test, kernel_toc, actual_toc) + +#endif /* MODULE */ + +#define PROTECTED_SC(test, ret_type, call) \ +({ \ + ret_type ret; \ + ret = call; \ + toc_fixup(test); \ + ret; \ +}) + +#endif /* _POWERPC_STATIC_CALL_TEST_ */