From patchwork Mon Jun 19 09:47:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naveen N Rao X-Patchwork-Id: 1796452 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=HFxd+cnN; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Ql4r80BJlz214Z for ; Mon, 19 Jun 2023 19:55:03 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=HFxd+cnN; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Ql4r74gyTz30XX for ; Mon, 19 Jun 2023 19:55:03 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=HFxd+cnN; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=kernel.org (client-ip=2604:1380:4641:c500::1; helo=dfw.source.kernel.org; envelope-from=naveen@kernel.org; receiver=lists.ozlabs.org) Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Ql4my0ZJ8z30R4 for ; Mon, 19 Jun 2023 19:52:18 +1000 (AEST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6C192602E0; Mon, 19 Jun 2023 09:52:16 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4EA41C433C8; Mon, 19 Jun 2023 09:52:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1687168335; bh=aMr8hjlmZ/quV2LGV9FBITAOvkQNdJP40nE4J0o6PDs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HFxd+cnNy7eaX2+ucWyL9oCRTPEy1yheISSIZGK/A/yVs6V5H+mhNsF3WEWZ7G2rA mfvfSQg2W6IydSSF/Kr7hyn3GaIAYQB1NJs0BT6M3z+WG5fhkgpx8ipBS4J/gyodBS 8jOHkOfDFB8hAOi0fOP7C3Lz9K3k262s6g8NCDYmKcSq7nT95E/zelCg53lV742klu OSZYnJnHUWbxchBEdbt0xIWLMkky/ojFftd0VZc8txXdFoMu1vKgt7R7vCCWFZMurb C0ugf0rGNOEZ0697r0p0O+nDW/kbO0D2RPqaOKz2L8WcK5s3MCXljW7+tV78/6Di7n L2ihwtK/GMM5w== From: Naveen N Rao To: Subject: [PATCH 01/17] powerpc/ftrace: Fix dropping weak symbols with older toolchains Date: Mon, 19 Jun 2023 15:17:19 +0530 Message-Id: <7b265908a9461e38fc756ef9b569703860a80621.1687166935.git.naveen@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Steven Rostedt Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" The minimum level of gcc supported for building the kernel is v5.1. v5.x releases of gcc emitted a three instruction sequence for -mprofile-kernel: mflr r0 std r0, 16(r1) bl _mcount It is only with the v6.x releases that gcc started emitting the two instruction sequence for -mprofile-kernel, omitting the second store instruction. With the older three instruction sequence, the actual ftrace location can be the 5th instruction into a function. Update the allowed offset for ftrace location from 12 to 16 to accommodate the same. Cc: stable@vger.kernel.org Fixes: 7af82ff90a2b06 ("powerpc/ftrace: Ignore weak functions") Signed-off-by: Naveen N Rao Reviewed-by: Christophe Leroy --- arch/powerpc/include/asm/ftrace.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/powerpc/include/asm/ftrace.h b/arch/powerpc/include/asm/ftrace.h index 91c049d51d0e10..2edc6269b1a357 100644 --- a/arch/powerpc/include/asm/ftrace.h +++ b/arch/powerpc/include/asm/ftrace.h @@ -12,7 +12,7 @@ /* Ignore unused weak functions which will have larger offsets */ #ifdef CONFIG_MPROFILE_KERNEL -#define FTRACE_MCOUNT_MAX_OFFSET 12 +#define FTRACE_MCOUNT_MAX_OFFSET 16 #elif defined(CONFIG_PPC32) #define FTRACE_MCOUNT_MAX_OFFSET 8 #endif From patchwork Mon Jun 19 09:47:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naveen N Rao X-Patchwork-Id: 1796449 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=C8Epmbxm; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Ql4p06wvdz20XZ for ; Mon, 19 Jun 2023 19:53:12 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=C8Epmbxm; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Ql4p064mQz30fn for ; Mon, 19 Jun 2023 19:53:12 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=C8Epmbxm; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=kernel.org (client-ip=139.178.84.217; helo=dfw.source.kernel.org; envelope-from=naveen@kernel.org; receiver=lists.ozlabs.org) Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Ql4mm57CPz30Q0 for ; Mon, 19 Jun 2023 19:52:08 +1000 (AEST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id E832260909; Mon, 19 Jun 2023 09:52:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C151AC433C8; Mon, 19 Jun 2023 09:52:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1687168325; bh=Iw1Dv20up6Z064YBBB74k6x5kv8HOFxX1mSlj8e9zYU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=C8Epmbxm+rd5I555DhdkzK6hemXQyspSs8tgeBOmmm9VhYPiJYBVmCNd//Nlh2A14 aPRqYzsy5x6N7E6a2WfMKeGBtpyJ64LUpymU+v7wQEjb3xKNfmtLwYKPg20AW2tRmE jyBx2ZVu4DRvmYL7POuvvDCecCuXZ0wzlGyhJuVAESvPHwawZpUQYw1rx0eyveq7ZA bEQuM4wHMaUGYhge19MH0AfSTBcUeZ+vK63W3rbsjCtINTSqPo4v7v0o+/EXmQeWLn 7UUoVEeyol8diog1wIsmR9TLGaxxzEwskNDOqugF/xfGDsGONpt7DUZnaH6neL/Gb3 tyxqK7THFO0+A== From: Naveen N Rao To: Subject: [PATCH 02/17] powerpc/module: Remove unused .ftrace.tramp section Date: Mon, 19 Jun 2023 15:17:20 +0530 Message-Id: <9cf6d7f37ba82f7cb6dafecf660f44925c526d8d.1687166935.git.naveen@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Steven Rostedt Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" .ftrace.tramp section is not used for any purpose. This code was added all the way back in the original commit introducing support for dynamic ftrace on ppc64 modules. Remove it. Signed-off-by: Naveen N Rao Reviewed-by: Christophe Leroy --- arch/powerpc/include/asm/module.h | 4 ---- 1 file changed, 4 deletions(-) diff --git a/arch/powerpc/include/asm/module.h b/arch/powerpc/include/asm/module.h index ac53606c259430..a8e2e8339fb7f4 100644 --- a/arch/powerpc/include/asm/module.h +++ b/arch/powerpc/include/asm/module.h @@ -75,10 +75,6 @@ struct mod_arch_specific { #endif #ifdef CONFIG_DYNAMIC_FTRACE -# ifdef MODULE - asm(".section .ftrace.tramp,\"ax\",@nobits; .align 3; .previous"); -# endif /* MODULE */ - int module_trampoline_target(struct module *mod, unsigned long trampoline, unsigned long *target); int module_finalize_ftrace(struct module *mod, const Elf_Shdr *sechdrs); From patchwork Mon Jun 19 09:47:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naveen N Rao X-Patchwork-Id: 1796450 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=Hm2sv0OX; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Ql4q02mnDz20XZ for ; Mon, 19 Jun 2023 19:54:04 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=Hm2sv0OX; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Ql4pz6BV5z30hC for ; Mon, 19 Jun 2023 19:54:03 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=Hm2sv0OX; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=kernel.org (client-ip=139.178.84.217; helo=dfw.source.kernel.org; envelope-from=naveen@kernel.org; receiver=lists.ozlabs.org) Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Ql4ms2Xmxz30fW for ; Mon, 19 Jun 2023 19:52:13 +1000 (AEST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 8913C602E0; Mon, 19 Jun 2023 09:52:11 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 108ECC433C8; Mon, 19 Jun 2023 09:52:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1687168331; bh=PSfFBoSGu4k/xQfpRcfxKY0ifx/FH6efZ67001q3tqg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Hm2sv0OX+otFYQKbSyLlXkbzxkmZSnUMTPOHqKjsoJZCcROAfiDW+d2efGo2M2HS0 k2hPJ0Uez+wtAqJSf2EOFEoi/9fvwlkST9ii8A0ClRtdYl6TSn8kbj9FbHRdAiUiEP FtGJwDaXN7Kl3fPBMyFg8ewUxlxEPSbqxDZ/nBE+n07/T40MrWLIPq74nZbKtz9YEr lij4Ws1kfOPJtml7pqv2NvEcZY5EDzvx90xQyshX/7/RhP07eOlraLsrV6SM1rRZEJ WQO3f4g9NwcK+gc4Hq+94B7rFoO85ElouxhIMaFmKLFFCsHQfLG9n+7fjp6hKxPzf9 Dtke+2NxpVL3Q== From: Naveen N Rao To: Subject: [PATCH 03/17] powerpc64/ftrace: Move ELFv1 and -pg support code into a separate file Date: Mon, 19 Jun 2023 15:17:21 +0530 Message-Id: <1eb6cc6c3141ddb77a2a25f8a9e83d83ff312b02.1687166935.git.naveen@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Steven Rostedt Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" ELFv1 support is deprecated and on the way out. Pre -mprofile-kernel ftrace support (-pg only) is very limited and is retained primarily for clang builds. It won't be necessary once clang lands support for -fpatchable-function-entry. Copy the existing ftrace code supporting these into ftrace_pg.c. ftrace.c can then be refactored and enhanced with a focus on ppc32 and ppc64 ELFv2. Signed-off-by: Naveen N Rao Reviewed-by: Christophe Leroy --- arch/powerpc/kernel/trace/Makefile | 13 +- arch/powerpc/kernel/trace/ftrace.c | 10 - arch/powerpc/kernel/trace/ftrace_pg.c | 846 ++++++++++++++++++++++++++ 3 files changed, 855 insertions(+), 14 deletions(-) create mode 100644 arch/powerpc/kernel/trace/ftrace_pg.c diff --git a/arch/powerpc/kernel/trace/Makefile b/arch/powerpc/kernel/trace/Makefile index b16a9f9c0b35f2..342a2d1ae86cd0 100644 --- a/arch/powerpc/kernel/trace/Makefile +++ b/arch/powerpc/kernel/trace/Makefile @@ -6,15 +6,16 @@ ifdef CONFIG_FUNCTION_TRACER # do not trace tracer code CFLAGS_REMOVE_ftrace.o = $(CC_FLAGS_FTRACE) +CFLAGS_REMOVE_ftrace_pg.o = $(CC_FLAGS_FTRACE) endif -obj32-$(CONFIG_FUNCTION_TRACER) += ftrace_mprofile.o +obj32-$(CONFIG_FUNCTION_TRACER) += ftrace_mprofile.o ftrace.o ifdef CONFIG_MPROFILE_KERNEL -obj64-$(CONFIG_FUNCTION_TRACER) += ftrace_mprofile.o +obj64-$(CONFIG_FUNCTION_TRACER) += ftrace_mprofile.o ftrace.o else -obj64-$(CONFIG_FUNCTION_TRACER) += ftrace_64_pg.o +obj64-$(CONFIG_FUNCTION_TRACER) += ftrace_64_pg.o ftrace_pg.o endif -obj-$(CONFIG_FUNCTION_TRACER) += ftrace_low.o ftrace.o +obj-$(CONFIG_FUNCTION_TRACER) += ftrace_low.o obj-$(CONFIG_TRACING) += trace_clock.o obj-$(CONFIG_PPC64) += $(obj64-y) @@ -25,3 +26,7 @@ GCOV_PROFILE_ftrace.o := n KCOV_INSTRUMENT_ftrace.o := n KCSAN_SANITIZE_ftrace.o := n UBSAN_SANITIZE_ftrace.o := n +GCOV_PROFILE_ftrace_pg.o := n +KCOV_INSTRUMENT_ftrace_pg.o := n +KCSAN_SANITIZE_ftrace_pg.o := n +UBSAN_SANITIZE_ftrace_pg.o := n diff --git a/arch/powerpc/kernel/trace/ftrace.c b/arch/powerpc/kernel/trace/ftrace.c index a47f303734233b..81a121b56c4d7f 100644 --- a/arch/powerpc/kernel/trace/ftrace.c +++ b/arch/powerpc/kernel/trace/ftrace.c @@ -864,13 +864,3 @@ unsigned long prepare_ftrace_return(unsigned long parent, unsigned long ip, } #endif #endif /* CONFIG_FUNCTION_GRAPH_TRACER */ - -#ifdef CONFIG_PPC64_ELF_ABI_V1 -char *arch_ftrace_match_adjust(char *str, const char *search) -{ - if (str[0] == '.' && search[0] != '.') - return str + 1; - else - return str; -} -#endif /* CONFIG_PPC64_ELF_ABI_V1 */ diff --git a/arch/powerpc/kernel/trace/ftrace_pg.c b/arch/powerpc/kernel/trace/ftrace_pg.c new file mode 100644 index 00000000000000..7b85c3b460a3c0 --- /dev/null +++ b/arch/powerpc/kernel/trace/ftrace_pg.c @@ -0,0 +1,846 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Code for replacing ftrace calls with jumps. + * + * Copyright (C) 2007-2008 Steven Rostedt + * + * Thanks goes out to P.A. Semi, Inc for supplying me with a PPC64 box. + * + * Added function graph tracer code, taken from x86 that was written + * by Frederic Weisbecker, and ported to PPC by Steven Rostedt. + * + */ + +#define pr_fmt(fmt) "ftrace-powerpc: " fmt + +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +/* + * We generally only have a single long_branch tramp and at most 2 or 3 plt + * tramps generated. But, we don't use the plt tramps currently. We also allot + * 2 tramps after .text and .init.text. So, we only end up with around 3 usable + * tramps in total. Set aside 8 just to be sure. + */ +#define NUM_FTRACE_TRAMPS 8 +static unsigned long ftrace_tramps[NUM_FTRACE_TRAMPS]; + +static ppc_inst_t +ftrace_call_replace(unsigned long ip, unsigned long addr, int link) +{ + ppc_inst_t op; + + addr = ppc_function_entry((void *)addr); + + /* if (link) set op to 'bl' else 'b' */ + create_branch(&op, (u32 *)ip, addr, link ? BRANCH_SET_LINK : 0); + + return op; +} + +static inline int +ftrace_modify_code(unsigned long ip, ppc_inst_t old, ppc_inst_t new) +{ + ppc_inst_t replaced; + + /* + * Note: + * We are paranoid about modifying text, as if a bug was to happen, it + * could cause us to read or write to someplace that could cause harm. + * Carefully read and modify the code with probe_kernel_*(), and make + * sure what we read is what we expected it to be before modifying it. + */ + + /* read the text we want to modify */ + if (copy_inst_from_kernel_nofault(&replaced, (void *)ip)) + return -EFAULT; + + /* Make sure it is what we expect it to be */ + if (!ppc_inst_equal(replaced, old)) { + pr_err("%p: replaced (%08lx) != old (%08lx)", (void *)ip, + ppc_inst_as_ulong(replaced), ppc_inst_as_ulong(old)); + return -EINVAL; + } + + /* replace the text with the new text */ + return patch_instruction((u32 *)ip, new); +} + +/* + * Helper functions that are the same for both PPC64 and PPC32. + */ +static int test_24bit_addr(unsigned long ip, unsigned long addr) +{ + addr = ppc_function_entry((void *)addr); + + return is_offset_in_branch_range(addr - ip); +} + +static int is_bl_op(ppc_inst_t op) +{ + return (ppc_inst_val(op) & ~PPC_LI_MASK) == PPC_RAW_BL(0); +} + +static int is_b_op(ppc_inst_t op) +{ + return (ppc_inst_val(op) & ~PPC_LI_MASK) == PPC_RAW_BRANCH(0); +} + +static unsigned long find_bl_target(unsigned long ip, ppc_inst_t op) +{ + int offset; + + offset = PPC_LI(ppc_inst_val(op)); + /* make it signed */ + if (offset & 0x02000000) + offset |= 0xfe000000; + + return ip + (long)offset; +} + +#ifdef CONFIG_MODULES +static int +__ftrace_make_nop(struct module *mod, + struct dyn_ftrace *rec, unsigned long addr) +{ + unsigned long entry, ptr, tramp; + unsigned long ip = rec->ip; + ppc_inst_t op, pop; + + /* read where this goes */ + if (copy_inst_from_kernel_nofault(&op, (void *)ip)) { + pr_err("Fetching opcode failed.\n"); + return -EFAULT; + } + + /* Make sure that this is still a 24bit jump */ + if (!is_bl_op(op)) { + pr_err("Not expected bl: opcode is %08lx\n", ppc_inst_as_ulong(op)); + return -EINVAL; + } + + /* lets find where the pointer goes */ + tramp = find_bl_target(ip, op); + + pr_devel("ip:%lx jumps to %lx", ip, tramp); + + if (module_trampoline_target(mod, tramp, &ptr)) { + pr_err("Failed to get trampoline target\n"); + return -EFAULT; + } + + pr_devel("trampoline target %lx", ptr); + + entry = ppc_global_function_entry((void *)addr); + /* This should match what was called */ + if (ptr != entry) { + pr_err("addr %lx does not match expected %lx\n", ptr, entry); + return -EINVAL; + } + + if (IS_ENABLED(CONFIG_MPROFILE_KERNEL)) { + if (copy_inst_from_kernel_nofault(&op, (void *)(ip - 4))) { + pr_err("Fetching instruction at %lx failed.\n", ip - 4); + return -EFAULT; + } + + /* We expect either a mflr r0, or a std r0, LRSAVE(r1) */ + if (!ppc_inst_equal(op, ppc_inst(PPC_RAW_MFLR(_R0))) && + !ppc_inst_equal(op, ppc_inst(PPC_INST_STD_LR))) { + pr_err("Unexpected instruction %08lx around bl _mcount\n", + ppc_inst_as_ulong(op)); + return -EINVAL; + } + } else if (IS_ENABLED(CONFIG_PPC64)) { + /* + * Check what is in the next instruction. We can see ld r2,40(r1), but + * on first pass after boot we will see mflr r0. + */ + if (copy_inst_from_kernel_nofault(&op, (void *)(ip + 4))) { + pr_err("Fetching op failed.\n"); + return -EFAULT; + } + + if (!ppc_inst_equal(op, ppc_inst(PPC_INST_LD_TOC))) { + pr_err("Expected %08lx found %08lx\n", PPC_INST_LD_TOC, + ppc_inst_as_ulong(op)); + return -EINVAL; + } + } + + /* + * When using -mprofile-kernel or PPC32 there is no load to jump over. + * + * Otherwise our original call site looks like: + * + * bl + * ld r2,XX(r1) + * + * Milton Miller pointed out that we can not simply nop the branch. + * If a task was preempted when calling a trace function, the nops + * will remove the way to restore the TOC in r2 and the r2 TOC will + * get corrupted. + * + * Use a b +8 to jump over the load. + */ + if (IS_ENABLED(CONFIG_MPROFILE_KERNEL) || IS_ENABLED(CONFIG_PPC32)) + pop = ppc_inst(PPC_RAW_NOP()); + else + pop = ppc_inst(PPC_RAW_BRANCH(8)); /* b +8 */ + + if (patch_instruction((u32 *)ip, pop)) { + pr_err("Patching NOP failed.\n"); + return -EPERM; + } + + return 0; +} +#else +static int __ftrace_make_nop(struct module *mod, struct dyn_ftrace *rec, unsigned long addr) +{ + return 0; +} +#endif /* CONFIG_MODULES */ + +static unsigned long find_ftrace_tramp(unsigned long ip) +{ + int i; + + /* + * We have the compiler generated long_branch tramps at the end + * and we prefer those + */ + for (i = NUM_FTRACE_TRAMPS - 1; i >= 0; i--) + if (!ftrace_tramps[i]) + continue; + else if (is_offset_in_branch_range(ftrace_tramps[i] - ip)) + return ftrace_tramps[i]; + + return 0; +} + +static int add_ftrace_tramp(unsigned long tramp) +{ + int i; + + for (i = 0; i < NUM_FTRACE_TRAMPS; i++) + if (!ftrace_tramps[i]) { + ftrace_tramps[i] = tramp; + return 0; + } + + return -1; +} + +/* + * If this is a compiler generated long_branch trampoline (essentially, a + * trampoline that has a branch to _mcount()), we re-write the branch to + * instead go to ftrace_[regs_]caller() and note down the location of this + * trampoline. + */ +static int setup_mcount_compiler_tramp(unsigned long tramp) +{ + int i; + ppc_inst_t op; + unsigned long ptr; + + /* Is this a known long jump tramp? */ + for (i = 0; i < NUM_FTRACE_TRAMPS; i++) + if (ftrace_tramps[i] == tramp) + return 0; + + /* New trampoline -- read where this goes */ + if (copy_inst_from_kernel_nofault(&op, (void *)tramp)) { + pr_debug("Fetching opcode failed.\n"); + return -1; + } + + /* Is this a 24 bit branch? */ + if (!is_b_op(op)) { + pr_debug("Trampoline is not a long branch tramp.\n"); + return -1; + } + + /* lets find where the pointer goes */ + ptr = find_bl_target(tramp, op); + + if (ptr != ppc_global_function_entry((void *)_mcount)) { + pr_debug("Trampoline target %p is not _mcount\n", (void *)ptr); + return -1; + } + + /* Let's re-write the tramp to go to ftrace_[regs_]caller */ + if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_REGS)) + ptr = ppc_global_function_entry((void *)ftrace_regs_caller); + else + ptr = ppc_global_function_entry((void *)ftrace_caller); + + if (patch_branch((u32 *)tramp, ptr, 0)) { + pr_debug("REL24 out of range!\n"); + return -1; + } + + if (add_ftrace_tramp(tramp)) { + pr_debug("No tramp locations left\n"); + return -1; + } + + return 0; +} + +static int __ftrace_make_nop_kernel(struct dyn_ftrace *rec, unsigned long addr) +{ + unsigned long tramp, ip = rec->ip; + ppc_inst_t op; + + /* Read where this goes */ + if (copy_inst_from_kernel_nofault(&op, (void *)ip)) { + pr_err("Fetching opcode failed.\n"); + return -EFAULT; + } + + /* Make sure that this is still a 24bit jump */ + if (!is_bl_op(op)) { + pr_err("Not expected bl: opcode is %08lx\n", ppc_inst_as_ulong(op)); + return -EINVAL; + } + + /* Let's find where the pointer goes */ + tramp = find_bl_target(ip, op); + + pr_devel("ip:%lx jumps to %lx", ip, tramp); + + if (setup_mcount_compiler_tramp(tramp)) { + /* Are other trampolines reachable? */ + if (!find_ftrace_tramp(ip)) { + pr_err("No ftrace trampolines reachable from %ps\n", + (void *)ip); + return -EINVAL; + } + } + + if (patch_instruction((u32 *)ip, ppc_inst(PPC_RAW_NOP()))) { + pr_err("Patching NOP failed.\n"); + return -EPERM; + } + + return 0; +} + +int ftrace_make_nop(struct module *mod, + struct dyn_ftrace *rec, unsigned long addr) +{ + unsigned long ip = rec->ip; + ppc_inst_t old, new; + + /* + * If the calling address is more that 24 bits away, + * then we had to use a trampoline to make the call. + * Otherwise just update the call site. + */ + if (test_24bit_addr(ip, addr)) { + /* within range */ + old = ftrace_call_replace(ip, addr, 1); + new = ppc_inst(PPC_RAW_NOP()); + return ftrace_modify_code(ip, old, new); + } else if (core_kernel_text(ip)) { + return __ftrace_make_nop_kernel(rec, addr); + } else if (!IS_ENABLED(CONFIG_MODULES)) { + return -EINVAL; + } + + /* + * Out of range jumps are called from modules. + * We should either already have a pointer to the module + * or it has been passed in. + */ + if (!rec->arch.mod) { + if (!mod) { + pr_err("No module loaded addr=%lx\n", addr); + return -EFAULT; + } + rec->arch.mod = mod; + } else if (mod) { + if (mod != rec->arch.mod) { + pr_err("Record mod %p not equal to passed in mod %p\n", + rec->arch.mod, mod); + return -EINVAL; + } + /* nothing to do if mod == rec->arch.mod */ + } else + mod = rec->arch.mod; + + return __ftrace_make_nop(mod, rec, addr); +} + +#ifdef CONFIG_MODULES +/* + * Examine the existing instructions for __ftrace_make_call. + * They should effectively be a NOP, and follow formal constraints, + * depending on the ABI. Return false if they don't. + */ +static bool expected_nop_sequence(void *ip, ppc_inst_t op0, ppc_inst_t op1) +{ + if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_REGS)) + return ppc_inst_equal(op0, ppc_inst(PPC_RAW_NOP())); + else + return ppc_inst_equal(op0, ppc_inst(PPC_RAW_BRANCH(8))) && + ppc_inst_equal(op1, ppc_inst(PPC_INST_LD_TOC)); +} + +static int +__ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) +{ + ppc_inst_t op[2]; + void *ip = (void *)rec->ip; + unsigned long entry, ptr, tramp; + struct module *mod = rec->arch.mod; + + /* read where this goes */ + if (copy_inst_from_kernel_nofault(op, ip)) + return -EFAULT; + + if (!IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_REGS) && + copy_inst_from_kernel_nofault(op + 1, ip + 4)) + return -EFAULT; + + if (!expected_nop_sequence(ip, op[0], op[1])) { + pr_err("Unexpected call sequence at %p: %08lx %08lx\n", ip, + ppc_inst_as_ulong(op[0]), ppc_inst_as_ulong(op[1])); + return -EINVAL; + } + + /* If we never set up ftrace trampoline(s), then bail */ + if (!mod->arch.tramp || + (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_REGS) && !mod->arch.tramp_regs)) { + pr_err("No ftrace trampoline\n"); + return -EINVAL; + } + + if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_REGS) && rec->flags & FTRACE_FL_REGS) + tramp = mod->arch.tramp_regs; + else + tramp = mod->arch.tramp; + + if (module_trampoline_target(mod, tramp, &ptr)) { + pr_err("Failed to get trampoline target\n"); + return -EFAULT; + } + + pr_devel("trampoline target %lx", ptr); + + entry = ppc_global_function_entry((void *)addr); + /* This should match what was called */ + if (ptr != entry) { + pr_err("addr %lx does not match expected %lx\n", ptr, entry); + return -EINVAL; + } + + if (patch_branch(ip, tramp, BRANCH_SET_LINK)) { + pr_err("REL24 out of range!\n"); + return -EINVAL; + } + + return 0; +} +#else +static int __ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) +{ + return 0; +} +#endif /* CONFIG_MODULES */ + +static int __ftrace_make_call_kernel(struct dyn_ftrace *rec, unsigned long addr) +{ + ppc_inst_t op; + void *ip = (void *)rec->ip; + unsigned long tramp, entry, ptr; + + /* Make sure we're being asked to patch branch to a known ftrace addr */ + entry = ppc_global_function_entry((void *)ftrace_caller); + ptr = ppc_global_function_entry((void *)addr); + + if (ptr != entry && IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_REGS)) + entry = ppc_global_function_entry((void *)ftrace_regs_caller); + + if (ptr != entry) { + pr_err("Unknown ftrace addr to patch: %ps\n", (void *)ptr); + return -EINVAL; + } + + /* Make sure we have a nop */ + if (copy_inst_from_kernel_nofault(&op, ip)) { + pr_err("Unable to read ftrace location %p\n", ip); + return -EFAULT; + } + + if (!ppc_inst_equal(op, ppc_inst(PPC_RAW_NOP()))) { + pr_err("Unexpected call sequence at %p: %08lx\n", + ip, ppc_inst_as_ulong(op)); + return -EINVAL; + } + + tramp = find_ftrace_tramp((unsigned long)ip); + if (!tramp) { + pr_err("No ftrace trampolines reachable from %ps\n", ip); + return -EINVAL; + } + + if (patch_branch(ip, tramp, BRANCH_SET_LINK)) { + pr_err("Error patching branch to ftrace tramp!\n"); + return -EINVAL; + } + + return 0; +} + +int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) +{ + unsigned long ip = rec->ip; + ppc_inst_t old, new; + + /* + * If the calling address is more that 24 bits away, + * then we had to use a trampoline to make the call. + * Otherwise just update the call site. + */ + if (test_24bit_addr(ip, addr)) { + /* within range */ + old = ppc_inst(PPC_RAW_NOP()); + new = ftrace_call_replace(ip, addr, 1); + return ftrace_modify_code(ip, old, new); + } else if (core_kernel_text(ip)) { + return __ftrace_make_call_kernel(rec, addr); + } else if (!IS_ENABLED(CONFIG_MODULES)) { + /* We should not get here without modules */ + return -EINVAL; + } + + /* + * Out of range jumps are called from modules. + * Being that we are converting from nop, it had better + * already have a module defined. + */ + if (!rec->arch.mod) { + pr_err("No module loaded\n"); + return -EINVAL; + } + + return __ftrace_make_call(rec, addr); +} + +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_REGS +#ifdef CONFIG_MODULES +static int +__ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr, + unsigned long addr) +{ + ppc_inst_t op; + unsigned long ip = rec->ip; + unsigned long entry, ptr, tramp; + struct module *mod = rec->arch.mod; + + /* If we never set up ftrace trampolines, then bail */ + if (!mod->arch.tramp || !mod->arch.tramp_regs) { + pr_err("No ftrace trampoline\n"); + return -EINVAL; + } + + /* read where this goes */ + if (copy_inst_from_kernel_nofault(&op, (void *)ip)) { + pr_err("Fetching opcode failed.\n"); + return -EFAULT; + } + + /* Make sure that this is still a 24bit jump */ + if (!is_bl_op(op)) { + pr_err("Not expected bl: opcode is %08lx\n", ppc_inst_as_ulong(op)); + return -EINVAL; + } + + /* lets find where the pointer goes */ + tramp = find_bl_target(ip, op); + entry = ppc_global_function_entry((void *)old_addr); + + pr_devel("ip:%lx jumps to %lx", ip, tramp); + + if (tramp != entry) { + /* old_addr is not within range, so we must have used a trampoline */ + if (module_trampoline_target(mod, tramp, &ptr)) { + pr_err("Failed to get trampoline target\n"); + return -EFAULT; + } + + pr_devel("trampoline target %lx", ptr); + + /* This should match what was called */ + if (ptr != entry) { + pr_err("addr %lx does not match expected %lx\n", ptr, entry); + return -EINVAL; + } + } + + /* The new target may be within range */ + if (test_24bit_addr(ip, addr)) { + /* within range */ + if (patch_branch((u32 *)ip, addr, BRANCH_SET_LINK)) { + pr_err("REL24 out of range!\n"); + return -EINVAL; + } + + return 0; + } + + if (rec->flags & FTRACE_FL_REGS) + tramp = mod->arch.tramp_regs; + else + tramp = mod->arch.tramp; + + if (module_trampoline_target(mod, tramp, &ptr)) { + pr_err("Failed to get trampoline target\n"); + return -EFAULT; + } + + pr_devel("trampoline target %lx", ptr); + + entry = ppc_global_function_entry((void *)addr); + /* This should match what was called */ + if (ptr != entry) { + pr_err("addr %lx does not match expected %lx\n", ptr, entry); + return -EINVAL; + } + + if (patch_branch((u32 *)ip, tramp, BRANCH_SET_LINK)) { + pr_err("REL24 out of range!\n"); + return -EINVAL; + } + + return 0; +} +#else +static int __ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr, unsigned long addr) +{ + return 0; +} +#endif + +int ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr, + unsigned long addr) +{ + unsigned long ip = rec->ip; + ppc_inst_t old, new; + + /* + * If the calling address is more that 24 bits away, + * then we had to use a trampoline to make the call. + * Otherwise just update the call site. + */ + if (test_24bit_addr(ip, addr) && test_24bit_addr(ip, old_addr)) { + /* within range */ + old = ftrace_call_replace(ip, old_addr, 1); + new = ftrace_call_replace(ip, addr, 1); + return ftrace_modify_code(ip, old, new); + } else if (core_kernel_text(ip)) { + /* + * We always patch out of range locations to go to the regs + * variant, so there is nothing to do here + */ + return 0; + } else if (!IS_ENABLED(CONFIG_MODULES)) { + /* We should not get here without modules */ + return -EINVAL; + } + + /* + * Out of range jumps are called from modules. + */ + if (!rec->arch.mod) { + pr_err("No module loaded\n"); + return -EINVAL; + } + + return __ftrace_modify_call(rec, old_addr, addr); +} +#endif + +int ftrace_update_ftrace_func(ftrace_func_t func) +{ + unsigned long ip = (unsigned long)(&ftrace_call); + ppc_inst_t old, new; + int ret; + + old = ppc_inst_read((u32 *)&ftrace_call); + new = ftrace_call_replace(ip, (unsigned long)func, 1); + ret = ftrace_modify_code(ip, old, new); + + /* Also update the regs callback function */ + if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_REGS) && !ret) { + ip = (unsigned long)(&ftrace_regs_call); + old = ppc_inst_read((u32 *)&ftrace_regs_call); + new = ftrace_call_replace(ip, (unsigned long)func, 1); + ret = ftrace_modify_code(ip, old, new); + } + + return ret; +} + +/* + * Use the default ftrace_modify_all_code, but without + * stop_machine(). + */ +void arch_ftrace_update_code(int command) +{ + ftrace_modify_all_code(command); +} + +#ifdef CONFIG_PPC64 +#define PACATOC offsetof(struct paca_struct, kernel_toc) + +extern unsigned int ftrace_tramp_text[], ftrace_tramp_init[]; + +void ftrace_free_init_tramp(void) +{ + int i; + + for (i = 0; i < NUM_FTRACE_TRAMPS && ftrace_tramps[i]; i++) + if (ftrace_tramps[i] == (unsigned long)ftrace_tramp_init) { + ftrace_tramps[i] = 0; + return; + } +} + +int __init ftrace_dyn_arch_init(void) +{ + int i; + unsigned int *tramp[] = { ftrace_tramp_text, ftrace_tramp_init }; + u32 stub_insns[] = { + PPC_RAW_LD(_R12, _R13, PACATOC), + PPC_RAW_ADDIS(_R12, _R12, 0), + PPC_RAW_ADDI(_R12, _R12, 0), + PPC_RAW_MTCTR(_R12), + PPC_RAW_BCTR() + }; + unsigned long addr; + long reladdr; + + if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_REGS)) + addr = ppc_global_function_entry((void *)ftrace_regs_caller); + else + addr = ppc_global_function_entry((void *)ftrace_caller); + + reladdr = addr - kernel_toc_addr(); + + if (reladdr >= SZ_2G || reladdr < -(long)SZ_2G) { + pr_err("Address of %ps out of range of kernel_toc.\n", + (void *)addr); + return -1; + } + + for (i = 0; i < 2; i++) { + memcpy(tramp[i], stub_insns, sizeof(stub_insns)); + tramp[i][1] |= PPC_HA(reladdr); + tramp[i][2] |= PPC_LO(reladdr); + add_ftrace_tramp((unsigned long)tramp[i]); + } + + return 0; +} +#endif + +#ifdef CONFIG_FUNCTION_GRAPH_TRACER + +extern void ftrace_graph_call(void); +extern void ftrace_graph_stub(void); + +static int ftrace_modify_ftrace_graph_caller(bool enable) +{ + unsigned long ip = (unsigned long)(&ftrace_graph_call); + unsigned long addr = (unsigned long)(&ftrace_graph_caller); + unsigned long stub = (unsigned long)(&ftrace_graph_stub); + ppc_inst_t old, new; + + if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_ARGS)) + return 0; + + old = ftrace_call_replace(ip, enable ? stub : addr, 0); + new = ftrace_call_replace(ip, enable ? addr : stub, 0); + + return ftrace_modify_code(ip, old, new); +} + +int ftrace_enable_ftrace_graph_caller(void) +{ + return ftrace_modify_ftrace_graph_caller(true); +} + +int ftrace_disable_ftrace_graph_caller(void) +{ + return ftrace_modify_ftrace_graph_caller(false); +} + +/* + * Hook the return address and push it in the stack of return addrs + * in current thread info. Return the address we want to divert to. + */ +static unsigned long +__prepare_ftrace_return(unsigned long parent, unsigned long ip, unsigned long sp) +{ + unsigned long return_hooker; + int bit; + + if (unlikely(ftrace_graph_is_dead())) + goto out; + + if (unlikely(atomic_read(¤t->tracing_graph_pause))) + goto out; + + bit = ftrace_test_recursion_trylock(ip, parent); + if (bit < 0) + goto out; + + return_hooker = ppc_function_entry(return_to_handler); + + if (!function_graph_enter(parent, ip, 0, (unsigned long *)sp)) + parent = return_hooker; + + ftrace_test_recursion_unlock(bit); +out: + return parent; +} + +#ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS +void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, + struct ftrace_ops *op, struct ftrace_regs *fregs) +{ + fregs->regs.link = __prepare_ftrace_return(parent_ip, ip, fregs->regs.gpr[1]); +} +#else +unsigned long prepare_ftrace_return(unsigned long parent, unsigned long ip, + unsigned long sp) +{ + return __prepare_ftrace_return(parent, ip, sp); +} +#endif +#endif /* CONFIG_FUNCTION_GRAPH_TRACER */ + +#ifdef CONFIG_PPC64_ELF_ABI_V1 +char *arch_ftrace_match_adjust(char *str, const char *search) +{ + if (str[0] == '.' && search[0] != '.') + return str + 1; + else + return str; +} +#endif /* CONFIG_PPC64_ELF_ABI_V1 */ From patchwork Mon Jun 19 09:47:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naveen N Rao X-Patchwork-Id: 1796479 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=HCUqb8Gr; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Ql56P3PqPz20XZ for ; Mon, 19 Jun 2023 20:07:25 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=HCUqb8Gr; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Ql56N6JzGz3bmh for ; Mon, 19 Jun 2023 20:07:24 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=HCUqb8Gr; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=kernel.org (client-ip=2604:1380:4641:c500::1; helo=dfw.source.kernel.org; envelope-from=naveen@kernel.org; receiver=lists.ozlabs.org) Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Ql4pM0kjGz30f0 for ; Mon, 19 Jun 2023 19:53:31 +1000 (AEST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 7143F602E0; Mon, 19 Jun 2023 09:53:29 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 424ECC433C8; Mon, 19 Jun 2023 09:53:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1687168408; bh=fiYOSGx/yX+rBoVWfDjzvuQpt+N/h7M6dfetR+JOhWA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HCUqb8GrgIDA9dG2ZUhBvnsrKrltO+YCP5Nj7AFKGAZxolXAhJKoGjNjW16oohjob 00bVBx/H+hkc+mzK6+6L7VZrs67nOZMvB05s2mGdYT+Mz5hTkHTNL99aeLXe10MO9g 73FlAdsniSmO0mxs17MxYkkVlomgCImHlcdjNGMfaLAlHvIyLU2ODfH7p0kuRpyFG0 z+04JLXZuhiV9jWi3xnL50VxqwPEhBG3kSlbUmQ/Yyr+J02J/apcp5cAp2RTwtvM8I q07Uf4A8UOw1H38bd/oFk3uJ9jMwCoHK2I0GAMGnWFh51533TFSgpgf/+m3c2mCK25 0ugLcHd/avGBA== From: Naveen N Rao To: Subject: [PATCH 04/17] powerpc/ftrace: Simplify function_graph support in ftrace.c Date: Mon, 19 Jun 2023 15:17:22 +0530 Message-Id: <4dc92c4b1ed444dc62b748ae7327acdb9e096864.1687166935.git.naveen@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Steven Rostedt Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Since we now support DYNAMIC_FTRACE_WITH_ARGS across ppc32 and ppc64 ELFv2, we can simplify function_graph tracer support code in ftrace.c Signed-off-by: Naveen N Rao Reviewed-by: Christophe Leroy --- arch/powerpc/kernel/trace/ftrace.c | 64 ++++-------------------------- 1 file changed, 7 insertions(+), 57 deletions(-) diff --git a/arch/powerpc/kernel/trace/ftrace.c b/arch/powerpc/kernel/trace/ftrace.c index 81a121b56c4d7f..f117124c30325f 100644 --- a/arch/powerpc/kernel/trace/ftrace.c +++ b/arch/powerpc/kernel/trace/ftrace.c @@ -790,44 +790,10 @@ int __init ftrace_dyn_arch_init(void) #endif #ifdef CONFIG_FUNCTION_GRAPH_TRACER - -extern void ftrace_graph_call(void); -extern void ftrace_graph_stub(void); - -static int ftrace_modify_ftrace_graph_caller(bool enable) +void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, + struct ftrace_ops *op, struct ftrace_regs *fregs) { - unsigned long ip = (unsigned long)(&ftrace_graph_call); - unsigned long addr = (unsigned long)(&ftrace_graph_caller); - unsigned long stub = (unsigned long)(&ftrace_graph_stub); - ppc_inst_t old, new; - - if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_ARGS)) - return 0; - - old = ftrace_call_replace(ip, enable ? stub : addr, 0); - new = ftrace_call_replace(ip, enable ? addr : stub, 0); - - return ftrace_modify_code(ip, old, new); -} - -int ftrace_enable_ftrace_graph_caller(void) -{ - return ftrace_modify_ftrace_graph_caller(true); -} - -int ftrace_disable_ftrace_graph_caller(void) -{ - return ftrace_modify_ftrace_graph_caller(false); -} - -/* - * Hook the return address and push it in the stack of return addrs - * in current thread info. Return the address we want to divert to. - */ -static unsigned long -__prepare_ftrace_return(unsigned long parent, unsigned long ip, unsigned long sp) -{ - unsigned long return_hooker; + unsigned long sp = fregs->regs.gpr[1]; int bit; if (unlikely(ftrace_graph_is_dead())) @@ -836,31 +802,15 @@ __prepare_ftrace_return(unsigned long parent, unsigned long ip, unsigned long sp if (unlikely(atomic_read(¤t->tracing_graph_pause))) goto out; - bit = ftrace_test_recursion_trylock(ip, parent); + bit = ftrace_test_recursion_trylock(ip, parent_ip); if (bit < 0) goto out; - return_hooker = ppc_function_entry(return_to_handler); - - if (!function_graph_enter(parent, ip, 0, (unsigned long *)sp)) - parent = return_hooker; + if (!function_graph_enter(parent_ip, ip, 0, (unsigned long *)sp)) + parent_ip = ppc_function_entry(return_to_handler); ftrace_test_recursion_unlock(bit); out: - return parent; + fregs->regs.link = parent_ip; } - -#ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS -void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, - struct ftrace_ops *op, struct ftrace_regs *fregs) -{ - fregs->regs.link = __prepare_ftrace_return(parent_ip, ip, fregs->regs.gpr[1]); -} -#else -unsigned long prepare_ftrace_return(unsigned long parent, unsigned long ip, - unsigned long sp) -{ - return __prepare_ftrace_return(parent, ip, sp); -} -#endif #endif /* CONFIG_FUNCTION_GRAPH_TRACER */ From patchwork Mon Jun 19 09:47:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naveen N Rao X-Patchwork-Id: 1796470 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=Ecowpg2i; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Ql4xB6gYmz20Wk for ; Mon, 19 Jun 2023 19:59:26 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=Ecowpg2i; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Ql4xB5RFdz3c3b for ; Mon, 19 Jun 2023 19:59:26 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=Ecowpg2i; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=kernel.org (client-ip=139.178.84.217; helo=dfw.source.kernel.org; envelope-from=naveen@kernel.org; receiver=lists.ozlabs.org) Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Ql4nS2QQyz30f2 for ; Mon, 19 Jun 2023 19:52:44 +1000 (AEST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id ACF0460909; Mon, 19 Jun 2023 09:52:42 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 846CEC433C0; Mon, 19 Jun 2023 09:52:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1687168362; bh=ogEUXT2uop1wwWimAziuOSqdGoByA/s+m4cCyc0kb0o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Ecowpg2ianiRbd9CgHLvJ9v3DN6biRsXftFp72zI7gOyK0kkfHAdCxVIGSE/B6TIL LT7ICI00ZHdhumS1zR+b32FzQP5/TDjRTjhUZ8WcQ4REKEubAEJE4dH9VV/wAL6BR3 5+q6OyWRIiqPOJD4rZLm++7QtuSbn4uQuj9f0xOzMMrn8YWg2s0cAiPIqWX86ddjqd fhI8icSEtNN8L1tpM4/q3qid4fP/spxcZgDC9Wv+92ZmQ5zzE1Gsl2PIufvpiWAwZ2 BuFdFnbwHFWouF4dbpMjQ0k88u1O6a9pUSdRAZg19Mov1AsJDZRSXzlEO3OaxBQPpS 9lJORE8O6lh+Q== From: Naveen N Rao To: Subject: [PATCH 05/17] powerpc/ftrace: Use FTRACE_REGS_ADDR to identify the correct ftrace trampoline Date: Mon, 19 Jun 2023 15:17:23 +0530 Message-Id: <6045a280a57a7ea937a5bb13ccac747026dbfb07.1687166935.git.naveen@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Steven Rostedt Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Instead of keying off DYNAMIC_FTRACE_WITH_REGS, use FTRACE_REGS_ADDR to identify the proper ftrace trampoline address to use. Signed-off-by: Naveen N Rao Reviewed-by: Christophe Leroy --- arch/powerpc/kernel/trace/ftrace.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/arch/powerpc/kernel/trace/ftrace.c b/arch/powerpc/kernel/trace/ftrace.c index f117124c30325f..5aa36272617a03 100644 --- a/arch/powerpc/kernel/trace/ftrace.c +++ b/arch/powerpc/kernel/trace/ftrace.c @@ -745,14 +745,9 @@ int __init ftrace_dyn_arch_init(void) }; #endif - unsigned long addr; + unsigned long addr = FTRACE_REGS_ADDR; long reladdr; - if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_REGS)) - addr = ppc_global_function_entry((void *)ftrace_regs_caller); - else - addr = ppc_global_function_entry((void *)ftrace_caller); - if (IS_ENABLED(CONFIG_PPC_KERNEL_PCREL)) { for (i = 0; i < 2; i++) { reladdr = addr - (unsigned long)tramp[i]; From patchwork Mon Jun 19 09:47:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naveen N Rao X-Patchwork-Id: 1796471 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=DTv4iIMC; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Ql4yH43tDz20XZ for ; Mon, 19 Jun 2023 20:00:23 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=DTv4iIMC; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Ql4yH2vnlz3btc for ; Mon, 19 Jun 2023 20:00:23 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=DTv4iIMC; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=kernel.org (client-ip=139.178.84.217; helo=dfw.source.kernel.org; envelope-from=naveen@kernel.org; receiver=lists.ozlabs.org) Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Ql4nY4hTVz3bYF for ; Mon, 19 Jun 2023 19:52:49 +1000 (AEST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 02F7760B4B; Mon, 19 Jun 2023 09:52:48 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 86D08C433C0; Mon, 19 Jun 2023 09:52:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1687168367; bh=EldU9Xh7E1tCtYbsR3NaeaFeInCufKxrl11otRN2CWk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DTv4iIMCmcwJ1tRdZXc2B0apoZlbtUnEQyK8FWPl8Vrf4pnTtwtz0RHjoQ77DmQb3 od5GOf3zt8/TOqFbsUzDe9wtA1BKwRBmDHkcq12LXyWLGEeYATYVSrIj8dcghd+tmp KKCYU3bpfC3hWszIF8WbL1IbN0op09wiLM1qFuRZTBg4u37yRWNjhQaSnyKhpc6f6A Bz0yyplb53H8oG/AN2CSv2SevTUtj0PWXx96Xu9Vpoans+Hoj9ai7yWnFXmlfGeW0k BW260vRBDTjY6IFCiKZGvAbdp5Rqk0JhrFFm4MJ2U+TVaSuKaBLBdVOjzRI3lBIm9V KifZbJopMWMgQ== From: Naveen N Rao To: Subject: [PATCH 06/17] powerpc/ftrace: Extend ftrace support for large kernels to ppc32 Date: Mon, 19 Jun 2023 15:17:24 +0530 Message-Id: <9fa3258cbb9105cf8a0a8135214d44ffbc75fe84.1687166935.git.naveen@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Steven Rostedt Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Commit 67361cf8071286 ("powerpc/ftrace: Handle large kernel configs") added ftrace support for ppc64 kernel images with a text section larger than 32MB. The approach itself isn't specific to ppc64, so extend the same to also work on ppc32. While at it, reduce the space reserved for the stub from 64 bytes to 32 bytes since the different stub variants are all less than 8 instructions. To reduce use of #ifdef, a stub implementation is provided for kernel_toc_address() and -SZ_2G is cast to 'long long' to prevent errors on ppc32. Signed-off-by: Naveen N Rao Reviewed-by: Christophe Leroy --- arch/powerpc/include/asm/ftrace.h | 10 +++++-- arch/powerpc/include/asm/sections.h | 2 ++ arch/powerpc/kernel/trace/ftrace.c | 39 ++++++++++++++------------ arch/powerpc/kernel/trace/ftrace_low.S | 6 ++-- arch/powerpc/kernel/vmlinux.lds.S | 4 --- 5 files changed, 32 insertions(+), 29 deletions(-) diff --git a/arch/powerpc/include/asm/ftrace.h b/arch/powerpc/include/asm/ftrace.h index 2edc6269b1a357..702aaf2efa966c 100644 --- a/arch/powerpc/include/asm/ftrace.h +++ b/arch/powerpc/include/asm/ftrace.h @@ -124,15 +124,19 @@ static inline u8 this_cpu_get_ftrace_enabled(void) { return get_paca()->ftrace_enabled; } - -void ftrace_free_init_tramp(void); #else /* CONFIG_PPC64 */ static inline void this_cpu_disable_ftrace(void) { } static inline void this_cpu_enable_ftrace(void) { } static inline void this_cpu_set_ftrace_enabled(u8 ftrace_enabled) { } static inline u8 this_cpu_get_ftrace_enabled(void) { return 1; } -static inline void ftrace_free_init_tramp(void) { } #endif /* CONFIG_PPC64 */ + +#ifdef CONFIG_FUNCTION_TRACER +extern unsigned int ftrace_tramp_text[], ftrace_tramp_init[]; +void ftrace_free_init_tramp(void); +#else +static inline void ftrace_free_init_tramp(void) { } +#endif #endif /* !__ASSEMBLY__ */ #endif /* _ASM_POWERPC_FTRACE */ diff --git a/arch/powerpc/include/asm/sections.h b/arch/powerpc/include/asm/sections.h index 4e1f548c8d373d..ea26665f82cfc8 100644 --- a/arch/powerpc/include/asm/sections.h +++ b/arch/powerpc/include/asm/sections.h @@ -74,6 +74,8 @@ static inline int overlaps_kernel_text(unsigned long start, unsigned long end) (unsigned long)_stext < end; } +#else +static inline unsigned long kernel_toc_addr(void) { BUILD_BUG(); return -1UL; } #endif #endif /* __KERNEL__ */ diff --git a/arch/powerpc/kernel/trace/ftrace.c b/arch/powerpc/kernel/trace/ftrace.c index 5aa36272617a03..913c7aa63d3fa3 100644 --- a/arch/powerpc/kernel/trace/ftrace.c +++ b/arch/powerpc/kernel/trace/ftrace.c @@ -707,11 +707,6 @@ void arch_ftrace_update_code(int command) ftrace_modify_all_code(command); } -#ifdef CONFIG_PPC64 -#define PACATOC offsetof(struct paca_struct, kernel_toc) - -extern unsigned int ftrace_tramp_text[], ftrace_tramp_init[]; - void ftrace_free_init_tramp(void) { int i; @@ -725,28 +720,30 @@ void ftrace_free_init_tramp(void) int __init ftrace_dyn_arch_init(void) { - int i; unsigned int *tramp[] = { ftrace_tramp_text, ftrace_tramp_init }; -#ifdef CONFIG_PPC_KERNEL_PCREL + unsigned long addr = FTRACE_REGS_ADDR; + long reladdr; + int i; u32 stub_insns[] = { +#ifdef CONFIG_PPC_KERNEL_PCREL /* pla r12,addr */ PPC_PREFIX_MLS | __PPC_PRFX_R(1), PPC_INST_PADDI | ___PPC_RT(_R12), PPC_RAW_MTCTR(_R12), PPC_RAW_BCTR() - }; -#else - u32 stub_insns[] = { - PPC_RAW_LD(_R12, _R13, PACATOC), +#elif defined(CONFIG_PPC64) + PPC_RAW_LD(_R12, _R13, offsetof(struct paca_struct, kernel_toc)), PPC_RAW_ADDIS(_R12, _R12, 0), PPC_RAW_ADDI(_R12, _R12, 0), PPC_RAW_MTCTR(_R12), PPC_RAW_BCTR() - }; +#else + PPC_RAW_LIS(_R12, 0), + PPC_RAW_ADDI(_R12, _R12, 0), + PPC_RAW_MTCTR(_R12), + PPC_RAW_BCTR() #endif - - unsigned long addr = FTRACE_REGS_ADDR; - long reladdr; + }; if (IS_ENABLED(CONFIG_PPC_KERNEL_PCREL)) { for (i = 0; i < 2; i++) { @@ -763,10 +760,10 @@ int __init ftrace_dyn_arch_init(void) tramp[i][1] |= IMM_L(reladdr); add_ftrace_tramp((unsigned long)tramp[i]); } - } else { + } else if (IS_ENABLED(CONFIG_PPC64)) { reladdr = addr - kernel_toc_addr(); - if (reladdr >= (long)SZ_2G || reladdr < -(long)SZ_2G) { + if (reladdr >= (long)SZ_2G || reladdr < -(long long)SZ_2G) { pr_err("Address of %ps out of range of kernel_toc.\n", (void *)addr); return -1; @@ -778,11 +775,17 @@ int __init ftrace_dyn_arch_init(void) tramp[i][2] |= PPC_LO(reladdr); add_ftrace_tramp((unsigned long)tramp[i]); } + } else { + for (i = 0; i < 2; i++) { + memcpy(tramp[i], stub_insns, sizeof(stub_insns)); + tramp[i][0] |= PPC_HA(addr); + tramp[i][1] |= PPC_LO(addr); + add_ftrace_tramp((unsigned long)tramp[i]); + } } return 0; } -#endif #ifdef CONFIG_FUNCTION_GRAPH_TRACER void ftrace_graph_func(unsigned long ip, unsigned long parent_ip, diff --git a/arch/powerpc/kernel/trace/ftrace_low.S b/arch/powerpc/kernel/trace/ftrace_low.S index 294d1e05958aae..2fc7dd0a5ae968 100644 --- a/arch/powerpc/kernel/trace/ftrace_low.S +++ b/arch/powerpc/kernel/trace/ftrace_low.S @@ -10,19 +10,17 @@ #include #include -#ifdef CONFIG_PPC64 .pushsection ".tramp.ftrace.text","aw",@progbits; .globl ftrace_tramp_text ftrace_tramp_text: - .space 64 + .space 32 .popsection .pushsection ".tramp.ftrace.init","aw",@progbits; .globl ftrace_tramp_init ftrace_tramp_init: - .space 64 + .space 32 .popsection -#endif _GLOBAL(mcount) _GLOBAL(_mcount) diff --git a/arch/powerpc/kernel/vmlinux.lds.S b/arch/powerpc/kernel/vmlinux.lds.S index 13614f0b269cf4..1c5970df323366 100644 --- a/arch/powerpc/kernel/vmlinux.lds.S +++ b/arch/powerpc/kernel/vmlinux.lds.S @@ -107,9 +107,7 @@ SECTIONS #endif /* careful! __ftr_alt_* sections need to be close to .text */ *(.text.hot .text.hot.* TEXT_MAIN .text.fixup .text.unlikely .text.unlikely.* .fixup __ftr_alt_* .ref.text); -#ifdef CONFIG_PPC64 *(.tramp.ftrace.text); -#endif NOINSTR_TEXT SCHED_TEXT LOCK_TEXT @@ -276,9 +274,7 @@ SECTIONS */ . = ALIGN(PAGE_SIZE); _einittext = .; -#ifdef CONFIG_PPC64 *(.tramp.ftrace.init); -#endif } :text /* .exit.text is discarded at runtime, not link time, From patchwork Mon Jun 19 09:47:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naveen N Rao X-Patchwork-Id: 1796472 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=frbj6iLS; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Ql4zG4sqCz20XQ for ; Mon, 19 Jun 2023 20:01:14 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=frbj6iLS; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Ql4zG0WZHz3cNj for ; Mon, 19 Jun 2023 20:01:14 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=frbj6iLS; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=kernel.org (client-ip=2604:1380:4641:c500::1; helo=dfw.source.kernel.org; envelope-from=naveen@kernel.org; receiver=lists.ozlabs.org) Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Ql4nf4C7Hz30Q5 for ; Mon, 19 Jun 2023 19:52:54 +1000 (AEST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id E563D602E0; Mon, 19 Jun 2023 09:52:52 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BD908C433C8; Mon, 19 Jun 2023 09:52:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1687168372; bh=+h+3LyPm6CZYkFnOmOUKuPI8mQbyKGMHvHU2h6EHRe4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=frbj6iLSH71pXYpwb74At8c+/WFzsAuBqtb+UqKuzgXLIjmkOUaJIyEBHKFTHd66W Q+afLtC+tE0mYwCvZSf5tPXb4KlTjsCtAzhzGkpJKeFE2l/ciR5bVhr1DjWwXRN7IJ 0cM1ypyVsEsAG+1txcGrO7E/dYJc5lPI6k+3oqFo5VwRd6Bnlzi7aX1k2T9MSHdDdi UWm21oV7u+3dEoTRHjPUlcKxpZBRouI74uPo59pG+f/A7NxqUaMa5ja/RTtx8NWEdJ Onw3+Pj/3FW7PVulu7WxAfDWBX39phveShw/Bwze60JzKubIrjCSzsAoU8IUW7Fv4D TGuWdITNf/nFg== From: Naveen N Rao To: Subject: [PATCH 07/17] powerpc/ftrace: Consolidate ftrace support into fewer files Date: Mon, 19 Jun 2023 15:17:25 +0530 Message-Id: X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Steven Rostedt Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" ftrace_low.S has just the _mcount stub and return_to_handler(). Merge this back into ftrace_mprofile.S and ftrace_64_pg.S to keep all ftrace code together, and to allow those to evolve independently. ftrace_mprofile.S is also not an entirely accurate name since this also holds ppc32 code. This will be all the more incorrect once support for -fpatchable-function-entry is added. Rename files here to more accurately describe the code: - ftrace_mprofile.S is renamed to ftrace_entry.S - ftrace_pg.c is renamed to ftrace_64_pg.c - ftrace_64_pg.S is rename to ftrace_64_pg_entry.S Signed-off-by: Naveen N Rao Reviewed-by: Christophe Leroy --- arch/powerpc/kernel/trace/Makefile | 17 +++-- arch/powerpc/kernel/trace/ftrace_64_pg.S | 67 ------------------- .../trace/{ftrace_pg.c => ftrace_64_pg.c} | 0 .../{ftrace_low.S => ftrace_64_pg_entry.S} | 58 +++++++++++++++- .../{ftrace_mprofile.S => ftrace_entry.S} | 65 ++++++++++++++++++ 5 files changed, 130 insertions(+), 77 deletions(-) delete mode 100644 arch/powerpc/kernel/trace/ftrace_64_pg.S rename arch/powerpc/kernel/trace/{ftrace_pg.c => ftrace_64_pg.c} (100%) rename arch/powerpc/kernel/trace/{ftrace_low.S => ftrace_64_pg_entry.S} (55%) rename arch/powerpc/kernel/trace/{ftrace_mprofile.S => ftrace_entry.S} (83%) diff --git a/arch/powerpc/kernel/trace/Makefile b/arch/powerpc/kernel/trace/Makefile index 342a2d1ae86cd0..125f4ca588b98a 100644 --- a/arch/powerpc/kernel/trace/Makefile +++ b/arch/powerpc/kernel/trace/Makefile @@ -6,16 +6,15 @@ ifdef CONFIG_FUNCTION_TRACER # do not trace tracer code CFLAGS_REMOVE_ftrace.o = $(CC_FLAGS_FTRACE) -CFLAGS_REMOVE_ftrace_pg.o = $(CC_FLAGS_FTRACE) +CFLAGS_REMOVE_ftrace_64_pg.o = $(CC_FLAGS_FTRACE) endif -obj32-$(CONFIG_FUNCTION_TRACER) += ftrace_mprofile.o ftrace.o +obj32-$(CONFIG_FUNCTION_TRACER) += ftrace.o ftrace_entry.o ifdef CONFIG_MPROFILE_KERNEL -obj64-$(CONFIG_FUNCTION_TRACER) += ftrace_mprofile.o ftrace.o +obj64-$(CONFIG_FUNCTION_TRACER) += ftrace.o ftrace_entry.o else -obj64-$(CONFIG_FUNCTION_TRACER) += ftrace_64_pg.o ftrace_pg.o +obj64-$(CONFIG_FUNCTION_TRACER) += ftrace_64_pg.o ftrace_64_pg_entry.o endif -obj-$(CONFIG_FUNCTION_TRACER) += ftrace_low.o obj-$(CONFIG_TRACING) += trace_clock.o obj-$(CONFIG_PPC64) += $(obj64-y) @@ -26,7 +25,7 @@ GCOV_PROFILE_ftrace.o := n KCOV_INSTRUMENT_ftrace.o := n KCSAN_SANITIZE_ftrace.o := n UBSAN_SANITIZE_ftrace.o := n -GCOV_PROFILE_ftrace_pg.o := n -KCOV_INSTRUMENT_ftrace_pg.o := n -KCSAN_SANITIZE_ftrace_pg.o := n -UBSAN_SANITIZE_ftrace_pg.o := n +GCOV_PROFILE_ftrace_64_pg.o := n +KCOV_INSTRUMENT_ftrace_64_pg.o := n +KCSAN_SANITIZE_ftrace_64_pg.o := n +UBSAN_SANITIZE_ftrace_64_pg.o := n diff --git a/arch/powerpc/kernel/trace/ftrace_64_pg.S b/arch/powerpc/kernel/trace/ftrace_64_pg.S deleted file mode 100644 index 6708e24db0aba8..00000000000000 --- a/arch/powerpc/kernel/trace/ftrace_64_pg.S +++ /dev/null @@ -1,67 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * Split from ftrace_64.S - */ - -#include -#include -#include -#include -#include -#include - -_GLOBAL_TOC(ftrace_caller) - lbz r3, PACA_FTRACE_ENABLED(r13) - cmpdi r3, 0 - beqlr - - /* Taken from output of objdump from lib64/glibc */ - mflr r3 - ld r11, 0(r1) - stdu r1, -112(r1) - std r3, 128(r1) - ld r4, 16(r11) - subi r3, r3, MCOUNT_INSN_SIZE -.globl ftrace_call -ftrace_call: - bl ftrace_stub - nop -#ifdef CONFIG_FUNCTION_GRAPH_TRACER -.globl ftrace_graph_call -ftrace_graph_call: - b ftrace_graph_stub -_GLOBAL(ftrace_graph_stub) -#endif - ld r0, 128(r1) - mtlr r0 - addi r1, r1, 112 - -_GLOBAL(ftrace_stub) - blr - -#ifdef CONFIG_FUNCTION_GRAPH_TRACER -_GLOBAL(ftrace_graph_caller) - addi r5, r1, 112 - /* load r4 with local address */ - ld r4, 128(r1) - subi r4, r4, MCOUNT_INSN_SIZE - - /* Grab the LR out of the caller stack frame */ - ld r11, 112(r1) - ld r3, 16(r11) - - bl prepare_ftrace_return - nop - - /* - * prepare_ftrace_return gives us the address we divert to. - * Change the LR in the callers stack frame to this. - */ - ld r11, 112(r1) - std r3, 16(r11) - - ld r0, 128(r1) - mtlr r0 - addi r1, r1, 112 - blr -#endif /* CONFIG_FUNCTION_GRAPH_TRACER */ diff --git a/arch/powerpc/kernel/trace/ftrace_pg.c b/arch/powerpc/kernel/trace/ftrace_64_pg.c similarity index 100% rename from arch/powerpc/kernel/trace/ftrace_pg.c rename to arch/powerpc/kernel/trace/ftrace_64_pg.c diff --git a/arch/powerpc/kernel/trace/ftrace_low.S b/arch/powerpc/kernel/trace/ftrace_64_pg_entry.S similarity index 55% rename from arch/powerpc/kernel/trace/ftrace_low.S rename to arch/powerpc/kernel/trace/ftrace_64_pg_entry.S index 2fc7dd0a5ae968..81dbaf70b1513a 100644 --- a/arch/powerpc/kernel/trace/ftrace_low.S +++ b/arch/powerpc/kernel/trace/ftrace_64_pg_entry.S @@ -1,6 +1,6 @@ /* SPDX-License-Identifier: GPL-2.0-or-later */ /* - * Split from entry_64.S + * Split from ftrace_64.S */ #include @@ -10,6 +10,62 @@ #include #include +_GLOBAL_TOC(ftrace_caller) + lbz r3, PACA_FTRACE_ENABLED(r13) + cmpdi r3, 0 + beqlr + + /* Taken from output of objdump from lib64/glibc */ + mflr r3 + ld r11, 0(r1) + stdu r1, -112(r1) + std r3, 128(r1) + ld r4, 16(r11) + subi r3, r3, MCOUNT_INSN_SIZE +.globl ftrace_call +ftrace_call: + bl ftrace_stub + nop +#ifdef CONFIG_FUNCTION_GRAPH_TRACER +.globl ftrace_graph_call +ftrace_graph_call: + b ftrace_graph_stub +_GLOBAL(ftrace_graph_stub) +#endif + ld r0, 128(r1) + mtlr r0 + addi r1, r1, 112 + +_GLOBAL(ftrace_stub) + blr + +#ifdef CONFIG_FUNCTION_GRAPH_TRACER +_GLOBAL(ftrace_graph_caller) + addi r5, r1, 112 + /* load r4 with local address */ + ld r4, 128(r1) + subi r4, r4, MCOUNT_INSN_SIZE + + /* Grab the LR out of the caller stack frame */ + ld r11, 112(r1) + ld r3, 16(r11) + + bl prepare_ftrace_return + nop + + /* + * prepare_ftrace_return gives us the address we divert to. + * Change the LR in the callers stack frame to this. + */ + ld r11, 112(r1) + std r3, 16(r11) + + ld r0, 128(r1) + mtlr r0 + addi r1, r1, 112 + blr +#endif /* CONFIG_FUNCTION_GRAPH_TRACER */ + .pushsection ".tramp.ftrace.text","aw",@progbits; .globl ftrace_tramp_text ftrace_tramp_text: diff --git a/arch/powerpc/kernel/trace/ftrace_mprofile.S b/arch/powerpc/kernel/trace/ftrace_entry.S similarity index 83% rename from arch/powerpc/kernel/trace/ftrace_mprofile.S rename to arch/powerpc/kernel/trace/ftrace_entry.S index ffb1db38684998..e8339706e735b1 100644 --- a/arch/powerpc/kernel/trace/ftrace_mprofile.S +++ b/arch/powerpc/kernel/trace/ftrace_entry.S @@ -249,3 +249,68 @@ livepatch_handler: /* Return to original caller of live patched function */ blr #endif /* CONFIG_LIVEPATCH */ + +_GLOBAL(mcount) +_GLOBAL(_mcount) +EXPORT_SYMBOL(_mcount) + mflr r12 + mtctr r12 + mtlr r0 + bctr + +#ifdef CONFIG_FUNCTION_GRAPH_TRACER +_GLOBAL(return_to_handler) + /* need to save return values */ +#ifdef CONFIG_PPC64 + std r4, -32(r1) + std r3, -24(r1) + /* save TOC */ + std r2, -16(r1) + std r31, -8(r1) + mr r31, r1 + stdu r1, -112(r1) + + /* + * We might be called from a module. + * Switch to our TOC to run inside the core kernel. + */ + LOAD_PACA_TOC() +#else + stwu r1, -16(r1) + stw r3, 8(r1) + stw r4, 12(r1) +#endif + + bl ftrace_return_to_handler + nop + + /* return value has real return address */ + mtlr r3 + +#ifdef CONFIG_PPC64 + ld r1, 0(r1) + ld r4, -32(r1) + ld r3, -24(r1) + ld r2, -16(r1) + ld r31, -8(r1) +#else + lwz r3, 8(r1) + lwz r4, 12(r1) + addi r1, r1, 16 +#endif + + /* Jump back to real return address */ + blr +#endif /* CONFIG_FUNCTION_GRAPH_TRACER */ + +.pushsection ".tramp.ftrace.text","aw",@progbits; +.globl ftrace_tramp_text +ftrace_tramp_text: + .space 32 +.popsection + +.pushsection ".tramp.ftrace.init","aw",@progbits; +.globl ftrace_tramp_init +ftrace_tramp_init: + .space 32 +.popsection From patchwork Mon Jun 19 09:47:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naveen N Rao X-Patchwork-Id: 1796473 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=kQE3XR/0; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Ql50N3Nvmz20XQ for ; Mon, 19 Jun 2023 20:02:12 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=kQE3XR/0; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Ql50N2F1Kz3bWL for ; Mon, 19 Jun 2023 20:02:12 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=kQE3XR/0; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=kernel.org (client-ip=2604:1380:4641:c500::1; helo=dfw.source.kernel.org; envelope-from=naveen@kernel.org; receiver=lists.ozlabs.org) Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Ql4nl5mvCz3bTr for ; Mon, 19 Jun 2023 19:52:59 +1000 (AEST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 2CF6A601CD; Mon, 19 Jun 2023 09:52:58 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id F3378C433C8; Mon, 19 Jun 2023 09:52:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1687168377; bh=8JeQh63fPatsalzj0Vl6bvA1m4bFoaGQC3L7k3yskE0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kQE3XR/0y8+ni92uSmj1iv8i9vmd5MNVRf0IkIhh2zL40VaX1COKGwUb1ksdqTLk4 8XFSSbtJtvcQpREUep2KKm+t+uNFHcY/JZ0dkQksuQosibi76sAMZpaKVjqi+MO7Xb G3SuhGYUhfYa6Q4d2qlSDtiSS0lO0gJ15uzKRuzc8e5/nwuC+ELcRpk+4RHlOGg4SE sK4cwYzaYdvTfZrEZ+7IrORf6GCewsxU2QLtmzYGnmB/UMABvhe7NeRYJL+rnG4ZLi 8ShRWgva9IcGf7MvcVkqeGOQoL21c4CZ0cJBfz6hjn5x/M5AmJ1pqz5t4BxeDYNVG8 trZpLaWsIcW8Q== From: Naveen N Rao To: Subject: [PATCH 08/17] powerpc/ftrace: Refactor ftrace_modify_code() Date: Mon, 19 Jun 2023 15:17:26 +0530 Message-Id: X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Steven Rostedt Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Split up ftrace_modify_code() into a few helpers for future use. Also update error messages accordingly. Signed-off-by: Naveen N Rao Reviewed-by: Christophe Leroy --- arch/powerpc/kernel/trace/ftrace.c | 51 +++++++++++++++++------------- 1 file changed, 29 insertions(+), 22 deletions(-) diff --git a/arch/powerpc/kernel/trace/ftrace.c b/arch/powerpc/kernel/trace/ftrace.c index 913c7aa63d3fa3..ef4e49c2c37781 100644 --- a/arch/powerpc/kernel/trace/ftrace.c +++ b/arch/powerpc/kernel/trace/ftrace.c @@ -50,32 +50,39 @@ ftrace_call_replace(unsigned long ip, unsigned long addr, int link) return op; } -static inline int -ftrace_modify_code(unsigned long ip, ppc_inst_t old, ppc_inst_t new) +static inline int ftrace_read_inst(unsigned long ip, ppc_inst_t *op) { - ppc_inst_t replaced; - - /* - * Note: - * We are paranoid about modifying text, as if a bug was to happen, it - * could cause us to read or write to someplace that could cause harm. - * Carefully read and modify the code with probe_kernel_*(), and make - * sure what we read is what we expected it to be before modifying it. - */ - - /* read the text we want to modify */ - if (copy_inst_from_kernel_nofault(&replaced, (void *)ip)) + if (copy_inst_from_kernel_nofault(op, (void *)ip)) { + pr_err("0x%lx: fetching instruction failed\n", ip); return -EFAULT; - - /* Make sure it is what we expect it to be */ - if (!ppc_inst_equal(replaced, old)) { - pr_err("%p: replaced (%08lx) != old (%08lx)", (void *)ip, - ppc_inst_as_ulong(replaced), ppc_inst_as_ulong(old)); - return -EINVAL; } - /* replace the text with the new text */ - return patch_instruction((u32 *)ip, new); + return 0; +} + +static inline int ftrace_validate_inst(unsigned long ip, ppc_inst_t inst) +{ + ppc_inst_t op; + int ret; + + ret = ftrace_read_inst(ip, &op); + if (!ret && !ppc_inst_equal(op, inst)) { + pr_err("0x%lx: expected (%08lx) != found (%08lx)\n", + ip, ppc_inst_as_ulong(inst), ppc_inst_as_ulong(op)); + ret = -EINVAL; + } + + return ret; +} + +static inline int ftrace_modify_code(unsigned long ip, ppc_inst_t old, ppc_inst_t new) +{ + int ret = ftrace_validate_inst(ip, old); + + if (!ret) + ret = patch_instruction((u32 *)ip, new); + + return ret; } /* From patchwork Mon Jun 19 09:47:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naveen N Rao X-Patchwork-Id: 1796474 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=TkwcTdHn; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Ql51N1rWdz20XQ for ; Mon, 19 Jun 2023 20:03:04 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=TkwcTdHn; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Ql51L39lzz3cSN for ; Mon, 19 Jun 2023 20:03:02 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=TkwcTdHn; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=kernel.org (client-ip=2604:1380:4641:c500::1; helo=dfw.source.kernel.org; envelope-from=naveen@kernel.org; receiver=lists.ozlabs.org) Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Ql4nr5qDpz3bn2 for ; Mon, 19 Jun 2023 19:53:04 +1000 (AEST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 2D16F601CD; Mon, 19 Jun 2023 09:53:03 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 07DE5C433C0; Mon, 19 Jun 2023 09:53:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1687168382; bh=jk9z1qfTbnJ4M50k9sUONn0cX2iepmGzBY8gg5oa7Ew=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TkwcTdHnp3h7eqSZTi+5eHtkidPf4WRlOPbqrZN6u+hvpdl2BcgwG0Oq7OMIqvXqf YUvCMHJrOkf4nSzTlZ0KYKu9fH36vaBBLEkdbfOQKRGZFQqzkUz9OrHe4+iMWCl1xu g7EYne51tb0gj74Fu8KUtC/pt5L9WHigkuxAYO6QIRoxlX7+CAUwXDImR5ChZaEsWy E3tHozLqFbYXlxB0lrcPIswwn79FPD8+e6KLitJu+xhNXsMWLiTJR4X0e6NXxqAydR d/NtN4TbVyyyzQsNMrXF53RtEDoE6VMMrYOUru2eyx4Yal6Yyrh5WwXhAWrOkjfcqK dCGvmYKMd77NQ== From: Naveen N Rao To: Subject: [PATCH 09/17] powerpc/ftrace: Stop re-purposing linker generated long branches for ftrace Date: Mon, 19 Jun 2023 15:17:27 +0530 Message-Id: <33fa3be97f8e1f2171254ef2e1b0d5c8836c11fd.1687166935.git.naveen@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Steven Rostedt Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Commit 67361cf8071286 ("powerpc/ftrace: Handle large kernel configs") added ftrace support for ppc64 kernel images with a text section larger than 32MB. The patch did two things: 1. Add stubs at the end of .text to branch into ftrace_[regs_]caller for functions that were out of branch range. 2. Re-purpose linker-generated long branches to _mcount to instead branch to ftrace_[regs_]caller. Before that, we only supported kernel .text up to ~32MB. With the above, we now support up to ~96MB: - The first 32MB of kernel text can branch directly into ftrace_[regs_]caller since that symbol is usually at the beginning. - The modified long_branch from (2) above is used by the next 32MB of kernel text. - The next 32MB of kernel text can use the stub at the end of text to branch back to ftrace_[regs_]caller. While re-purposing the long branch works in practice, it still restricts ftrace to kernel text up to ~96MB. The stub at the end of kernel text from (1) already enables us to extend ftrace support for kernel text up to 64MB, which fulfils the original requirement. Further, once we switch to -fpatchable-function-entry, there will not be a long branch that we can use. Stop re-purposing the linker-generated long branches for ftrace to simplify the code. If there are good reasons to support ftrace on kernels beyond 64MB, we can consider adding support by using -fpatchable-function-entry. Signed-off-by: Naveen N Rao Reviewed-by: Christophe Leroy --- arch/powerpc/kernel/trace/ftrace.c | 110 +++++------------------------ 1 file changed, 17 insertions(+), 93 deletions(-) diff --git a/arch/powerpc/kernel/trace/ftrace.c b/arch/powerpc/kernel/trace/ftrace.c index ef4e49c2c37781..278bf8e52b6e89 100644 --- a/arch/powerpc/kernel/trace/ftrace.c +++ b/arch/powerpc/kernel/trace/ftrace.c @@ -28,13 +28,7 @@ #include #include -/* - * We generally only have a single long_branch tramp and at most 2 or 3 plt - * tramps generated. But, we don't use the plt tramps currently. We also allot - * 2 tramps after .text and .init.text. So, we only end up with around 3 usable - * tramps in total. Set aside 8 just to be sure. - */ -#define NUM_FTRACE_TRAMPS 8 +#define NUM_FTRACE_TRAMPS 2 static unsigned long ftrace_tramps[NUM_FTRACE_TRAMPS]; static ppc_inst_t @@ -100,11 +94,6 @@ static int is_bl_op(ppc_inst_t op) return (ppc_inst_val(op) & ~PPC_LI_MASK) == PPC_RAW_BL(0); } -static int is_b_op(ppc_inst_t op) -{ - return (ppc_inst_val(op) & ~PPC_LI_MASK) == PPC_RAW_BRANCH(0); -} - static unsigned long find_bl_target(unsigned long ip, ppc_inst_t op) { int offset; @@ -227,11 +216,7 @@ static unsigned long find_ftrace_tramp(unsigned long ip) { int i; - /* - * We have the compiler generated long_branch tramps at the end - * and we prefer those - */ - for (i = NUM_FTRACE_TRAMPS - 1; i >= 0; i--) + for (i = 0; i < NUM_FTRACE_TRAMPS; i++) if (!ftrace_tramps[i]) continue; else if (is_offset_in_branch_range(ftrace_tramps[i] - ip)) @@ -240,75 +225,6 @@ static unsigned long find_ftrace_tramp(unsigned long ip) return 0; } -static int add_ftrace_tramp(unsigned long tramp) -{ - int i; - - for (i = 0; i < NUM_FTRACE_TRAMPS; i++) - if (!ftrace_tramps[i]) { - ftrace_tramps[i] = tramp; - return 0; - } - - return -1; -} - -/* - * If this is a compiler generated long_branch trampoline (essentially, a - * trampoline that has a branch to _mcount()), we re-write the branch to - * instead go to ftrace_[regs_]caller() and note down the location of this - * trampoline. - */ -static int setup_mcount_compiler_tramp(unsigned long tramp) -{ - int i; - ppc_inst_t op; - unsigned long ptr; - - /* Is this a known long jump tramp? */ - for (i = 0; i < NUM_FTRACE_TRAMPS; i++) - if (ftrace_tramps[i] == tramp) - return 0; - - /* New trampoline -- read where this goes */ - if (copy_inst_from_kernel_nofault(&op, (void *)tramp)) { - pr_debug("Fetching opcode failed.\n"); - return -1; - } - - /* Is this a 24 bit branch? */ - if (!is_b_op(op)) { - pr_debug("Trampoline is not a long branch tramp.\n"); - return -1; - } - - /* lets find where the pointer goes */ - ptr = find_bl_target(tramp, op); - - if (ptr != ppc_global_function_entry((void *)_mcount)) { - pr_debug("Trampoline target %p is not _mcount\n", (void *)ptr); - return -1; - } - - /* Let's re-write the tramp to go to ftrace_[regs_]caller */ - if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_REGS)) - ptr = ppc_global_function_entry((void *)ftrace_regs_caller); - else - ptr = ppc_global_function_entry((void *)ftrace_caller); - - if (patch_branch((u32 *)tramp, ptr, 0)) { - pr_debug("REL24 out of range!\n"); - return -1; - } - - if (add_ftrace_tramp(tramp)) { - pr_debug("No tramp locations left\n"); - return -1; - } - - return 0; -} - static int __ftrace_make_nop_kernel(struct dyn_ftrace *rec, unsigned long addr) { unsigned long tramp, ip = rec->ip; @@ -331,13 +247,10 @@ static int __ftrace_make_nop_kernel(struct dyn_ftrace *rec, unsigned long addr) pr_devel("ip:%lx jumps to %lx", ip, tramp); - if (setup_mcount_compiler_tramp(tramp)) { - /* Are other trampolines reachable? */ - if (!find_ftrace_tramp(ip)) { - pr_err("No ftrace trampolines reachable from %ps\n", - (void *)ip); - return -EINVAL; - } + /* Are ftrace trampolines reachable? */ + if (!find_ftrace_tramp(ip)) { + pr_err("No ftrace trampolines reachable from %ps\n", (void *)ip); + return -EINVAL; } if (patch_instruction((u32 *)ip, ppc_inst(PPC_RAW_NOP()))) { @@ -725,6 +638,17 @@ void ftrace_free_init_tramp(void) } } +static void __init add_ftrace_tramp(unsigned long tramp) +{ + int i; + + for (i = 0; i < NUM_FTRACE_TRAMPS; i++) + if (!ftrace_tramps[i]) { + ftrace_tramps[i] = tramp; + return; + } +} + int __init ftrace_dyn_arch_init(void) { unsigned int *tramp[] = { ftrace_tramp_text, ftrace_tramp_init }; From patchwork Mon Jun 19 09:47:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naveen N Rao X-Patchwork-Id: 1796475 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=PvfJ86X/; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Ql52K1wgCz20XQ for ; Mon, 19 Jun 2023 20:03:53 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=PvfJ86X/; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Ql52K0lDZz3cLk for ; Mon, 19 Jun 2023 20:03:53 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=PvfJ86X/; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=kernel.org (client-ip=2604:1380:4641:c500::1; helo=dfw.source.kernel.org; envelope-from=naveen@kernel.org; receiver=lists.ozlabs.org) Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Ql4ny0xLdz3bng for ; Mon, 19 Jun 2023 19:53:10 +1000 (AEST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 7C261601CD; Mon, 19 Jun 2023 09:53:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4FFDBC433C8; Mon, 19 Jun 2023 09:53:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1687168387; bh=70K4tpgIiBf65QJ6rsPxfnt11i5FQV+b9UaDgRIJCh8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=PvfJ86X/7TuPKabDQMgj324RV1vJWxIG7QjQ4Q2kuvu4/+vluLB4wqwD0S04RAyZz Bco8/SsHElfnhoG5PH4wnwS1oPyJ6kMbmSKQQuq7fNSaB9QBc4X0xXxkIIIrglMMmI dxNiqEtMRkzhUr6dOXHp0EaWHkofLb46GLdMmewipZaPJ/34A+s5ZoYC8fflxrOULx qNUIZL3MPa7k21SO2ej7q+3Tjc+cSRbRyXyhsM1N5CKrdRIIfPQ/teTbC+iHQcZo0K fVe6pf5m1NHKqMAUmZSJuYH1Sfvm52AP0wqde3BeLO3H8EMpJhVKVUWj9zazpn5kej KN8WCc5Sg0CzQ== From: Naveen N Rao To: Subject: [PATCH 10/17] powerpc/ftrace: Add separate ftrace_init_nop() with additional validation Date: Mon, 19 Jun 2023 15:17:28 +0530 Message-Id: X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Steven Rostedt Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Currently, we validate instructions around the ftrace location every time we have to enable/disable ftrace. Introduce ftrace_init_nop() to instead perform all the validation during ftrace initialization. This allows us to simply patch the necessary instructions during enabling/disabling ftrace. Signed-off-by: Naveen N Rao Reviewed-by: Christophe Leroy --- arch/powerpc/include/asm/ftrace.h | 6 +++ arch/powerpc/kernel/trace/ftrace.c | 71 ++++++++++++++++++++++++++++++ 2 files changed, 77 insertions(+) diff --git a/arch/powerpc/include/asm/ftrace.h b/arch/powerpc/include/asm/ftrace.h index 702aaf2efa966c..ef9f0b97670d1c 100644 --- a/arch/powerpc/include/asm/ftrace.h +++ b/arch/powerpc/include/asm/ftrace.h @@ -29,11 +29,17 @@ static inline unsigned long ftrace_call_adjust(unsigned long addr) unsigned long prepare_ftrace_return(unsigned long parent, unsigned long ip, unsigned long sp); +struct module; +struct dyn_ftrace; struct dyn_arch_ftrace { struct module *mod; }; #ifdef CONFIG_DYNAMIC_FTRACE_WITH_ARGS +#define ftrace_need_init_nop() (true) +int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec); +#define ftrace_init_nop ftrace_init_nop + struct ftrace_regs { struct pt_regs regs; }; diff --git a/arch/powerpc/kernel/trace/ftrace.c b/arch/powerpc/kernel/trace/ftrace.c index 278bf8e52b6e89..98bd099c428ee0 100644 --- a/arch/powerpc/kernel/trace/ftrace.c +++ b/arch/powerpc/kernel/trace/ftrace.c @@ -31,6 +31,16 @@ #define NUM_FTRACE_TRAMPS 2 static unsigned long ftrace_tramps[NUM_FTRACE_TRAMPS]; +static ppc_inst_t ftrace_create_branch_inst(unsigned long ip, unsigned long addr, int link) +{ + ppc_inst_t op; + + WARN_ON(!is_offset_in_branch_range(addr - ip)); + create_branch(&op, (u32 *)ip, addr, link ? BRANCH_SET_LINK : 0); + + return op; +} + static ppc_inst_t ftrace_call_replace(unsigned long ip, unsigned long addr, int link) { @@ -597,6 +607,67 @@ int ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr, } #endif +int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec) +{ + unsigned long addr, ip = rec->ip; + ppc_inst_t old, new; + int ret = 0; + + /* Verify instructions surrounding the ftrace location */ + if (IS_ENABLED(CONFIG_PPC32)) { + /* Expected sequence: 'mflr r0', 'stw r0,4(r1)', 'bl _mcount' */ + ret = ftrace_validate_inst(ip - 8, ppc_inst(PPC_RAW_MFLR(_R0))); + if (!ret) + ret = ftrace_validate_inst(ip - 4, ppc_inst(PPC_RAW_STW(_R0, _R1, 4))); + } else if (IS_ENABLED(CONFIG_MPROFILE_KERNEL)) { + /* Expected sequence: 'mflr r0', ['std r0,16(r1)'], 'bl _mcount' */ + ret = ftrace_read_inst(ip - 4, &old); + if (!ret && !ppc_inst_equal(old, ppc_inst(PPC_RAW_MFLR(_R0)))) { + ret = ftrace_validate_inst(ip - 8, ppc_inst(PPC_RAW_MFLR(_R0))); + ret |= ftrace_validate_inst(ip - 4, ppc_inst(PPC_RAW_STD(_R0, _R1, 16))); + } + } else { + return -EINVAL; + } + + if (ret) + return ret; + + if (!core_kernel_text(ip)) { + if (!mod) { + pr_err("0x%lx: No module provided for non-kernel address\n", ip); + return -EFAULT; + } + rec->arch.mod = mod; + } + + /* Nop-out the ftrace location */ + new = ppc_inst(PPC_RAW_NOP()); + addr = MCOUNT_ADDR; + if (is_offset_in_branch_range(addr - ip)) { + /* Within range */ + old = ftrace_create_branch_inst(ip, addr, 1); + ret = ftrace_modify_code(ip, old, new); + } else if (core_kernel_text(ip) || (IS_ENABLED(CONFIG_MODULES) && mod)) { + /* + * We would be branching to a linker-generated stub, or to the module _mcount + * stub. Let's just confirm we have a 'bl' here. + */ + ret = ftrace_read_inst(ip, &old); + if (ret) + return ret; + if (!is_bl_op(old)) { + pr_err("0x%lx: expected (bl) != found (%08lx)\n", ip, ppc_inst_as_ulong(old)); + return -EINVAL; + } + ret = patch_instruction((u32 *)ip, new); + } else { + return -EINVAL; + } + + return ret; +} + int ftrace_update_ftrace_func(ftrace_func_t func) { unsigned long ip = (unsigned long)(&ftrace_call); From patchwork Mon Jun 19 09:47:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naveen N Rao X-Patchwork-Id: 1796476 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=k6cLAnt/; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Ql53H6D8Yz20XQ for ; Mon, 19 Jun 2023 20:04:43 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=k6cLAnt/; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Ql53H5169z30XV for ; Mon, 19 Jun 2023 20:04:43 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=k6cLAnt/; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=kernel.org (client-ip=139.178.84.217; helo=dfw.source.kernel.org; envelope-from=naveen@kernel.org; receiver=lists.ozlabs.org) Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Ql4p33PmDz3bNj for ; Mon, 19 Jun 2023 19:53:15 +1000 (AEST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id B817C601CD; Mon, 19 Jun 2023 09:53:13 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 90209C433C8; Mon, 19 Jun 2023 09:53:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1687168393; bh=kY59zIia92tnkVQ+n2QicgjFmTstOeOTi3k+EkFObTo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=k6cLAnt/4LIy65hrAOHg3mCPIr7YJwBkTrJu+KhbtI3y5a8K074SvMaj1MKheBkW/ XawIA1pUdR+LfkaLYP2BLf/ACFaKXXRlozaOM8YeJo1yfKz3e01vBQSpqY1h3oqsqK w3tdMxLRNL9m9a0Yv1o0LcIwZKN7Pu6uhlZrzy6xU9Ims6BtcPzwWnyDch2uROlApe PioMXoA+Pl9e5KvSH5GeCTZ1g1yNe0jXsGFtyZ0mF5IvHvM8/LiWzkZcIln8rSZOy4 J57EWj/v0hkg9aSL0cbe1qBhznIySROLaGAjxALYG0qLoaRoal6ZNe2QJcBBbqvxDa EFFcjn/7zuc/A== From: Naveen N Rao To: Subject: [PATCH 11/17] powerpc/ftrace: Simplify ftrace_make_nop() Date: Mon, 19 Jun 2023 15:17:29 +0530 Message-Id: X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Steven Rostedt Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Now that we validate the ftrace location during initialization in ftrace_init_nop(), we can simplify ftrace_make_nop() to patch-in the nop without worrying about the instructions surrounding the ftrace location. Note that we continue to ensure that we have a bl to ftrace_[regs_]caller at the ftrace location before nop-ing it out. Signed-off-by: Naveen N Rao Reviewed-by: Christophe Leroy --- arch/powerpc/kernel/trace/ftrace.c | 220 +++++------------------------ 1 file changed, 32 insertions(+), 188 deletions(-) diff --git a/arch/powerpc/kernel/trace/ftrace.c b/arch/powerpc/kernel/trace/ftrace.c index 98bd099c428ee0..05153a1038fdff 100644 --- a/arch/powerpc/kernel/trace/ftrace.c +++ b/arch/powerpc/kernel/trace/ftrace.c @@ -116,112 +116,6 @@ static unsigned long find_bl_target(unsigned long ip, ppc_inst_t op) return ip + (long)offset; } -#ifdef CONFIG_MODULES -static int -__ftrace_make_nop(struct module *mod, - struct dyn_ftrace *rec, unsigned long addr) -{ - unsigned long entry, ptr, tramp; - unsigned long ip = rec->ip; - ppc_inst_t op, pop; - - /* read where this goes */ - if (copy_inst_from_kernel_nofault(&op, (void *)ip)) { - pr_err("Fetching opcode failed.\n"); - return -EFAULT; - } - - /* Make sure that this is still a 24bit jump */ - if (!is_bl_op(op)) { - pr_err("Not expected bl: opcode is %08lx\n", ppc_inst_as_ulong(op)); - return -EINVAL; - } - - /* lets find where the pointer goes */ - tramp = find_bl_target(ip, op); - - pr_devel("ip:%lx jumps to %lx", ip, tramp); - - if (module_trampoline_target(mod, tramp, &ptr)) { - pr_err("Failed to get trampoline target\n"); - return -EFAULT; - } - - pr_devel("trampoline target %lx", ptr); - - entry = ppc_global_function_entry((void *)addr); - /* This should match what was called */ - if (ptr != entry) { - pr_err("addr %lx does not match expected %lx\n", ptr, entry); - return -EINVAL; - } - - if (IS_ENABLED(CONFIG_MPROFILE_KERNEL)) { - if (copy_inst_from_kernel_nofault(&op, (void *)(ip - 4))) { - pr_err("Fetching instruction at %lx failed.\n", ip - 4); - return -EFAULT; - } - - /* We expect either a mflr r0, or a std r0, LRSAVE(r1) */ - if (!ppc_inst_equal(op, ppc_inst(PPC_RAW_MFLR(_R0))) && - !ppc_inst_equal(op, ppc_inst(PPC_INST_STD_LR))) { - pr_err("Unexpected instruction %08lx around bl _mcount\n", - ppc_inst_as_ulong(op)); - return -EINVAL; - } - } else if (IS_ENABLED(CONFIG_PPC64)) { - /* - * Check what is in the next instruction. We can see ld r2,40(r1), but - * on first pass after boot we will see mflr r0. - */ - if (copy_inst_from_kernel_nofault(&op, (void *)(ip + 4))) { - pr_err("Fetching op failed.\n"); - return -EFAULT; - } - - if (!ppc_inst_equal(op, ppc_inst(PPC_INST_LD_TOC))) { - pr_err("Expected %08lx found %08lx\n", PPC_INST_LD_TOC, - ppc_inst_as_ulong(op)); - return -EINVAL; - } - } - - /* - * When using -mprofile-kernel or PPC32 there is no load to jump over. - * - * Otherwise our original call site looks like: - * - * bl - * ld r2,XX(r1) - * - * Milton Miller pointed out that we can not simply nop the branch. - * If a task was preempted when calling a trace function, the nops - * will remove the way to restore the TOC in r2 and the r2 TOC will - * get corrupted. - * - * Use a b +8 to jump over the load. - * XXX: could make PCREL depend on MPROFILE_KERNEL - * XXX: check PCREL && MPROFILE_KERNEL calling sequence - */ - if (IS_ENABLED(CONFIG_MPROFILE_KERNEL) || IS_ENABLED(CONFIG_PPC32)) - pop = ppc_inst(PPC_RAW_NOP()); - else - pop = ppc_inst(PPC_RAW_BRANCH(8)); /* b +8 */ - - if (patch_instruction((u32 *)ip, pop)) { - pr_err("Patching NOP failed.\n"); - return -EPERM; - } - - return 0; -} -#else -static int __ftrace_make_nop(struct module *mod, struct dyn_ftrace *rec, unsigned long addr) -{ - return 0; -} -#endif /* CONFIG_MODULES */ - static unsigned long find_ftrace_tramp(unsigned long ip) { int i; @@ -235,88 +129,6 @@ static unsigned long find_ftrace_tramp(unsigned long ip) return 0; } -static int __ftrace_make_nop_kernel(struct dyn_ftrace *rec, unsigned long addr) -{ - unsigned long tramp, ip = rec->ip; - ppc_inst_t op; - - /* Read where this goes */ - if (copy_inst_from_kernel_nofault(&op, (void *)ip)) { - pr_err("Fetching opcode failed.\n"); - return -EFAULT; - } - - /* Make sure that this is still a 24bit jump */ - if (!is_bl_op(op)) { - pr_err("Not expected bl: opcode is %08lx\n", ppc_inst_as_ulong(op)); - return -EINVAL; - } - - /* Let's find where the pointer goes */ - tramp = find_bl_target(ip, op); - - pr_devel("ip:%lx jumps to %lx", ip, tramp); - - /* Are ftrace trampolines reachable? */ - if (!find_ftrace_tramp(ip)) { - pr_err("No ftrace trampolines reachable from %ps\n", (void *)ip); - return -EINVAL; - } - - if (patch_instruction((u32 *)ip, ppc_inst(PPC_RAW_NOP()))) { - pr_err("Patching NOP failed.\n"); - return -EPERM; - } - - return 0; -} - -int ftrace_make_nop(struct module *mod, - struct dyn_ftrace *rec, unsigned long addr) -{ - unsigned long ip = rec->ip; - ppc_inst_t old, new; - - /* - * If the calling address is more that 24 bits away, - * then we had to use a trampoline to make the call. - * Otherwise just update the call site. - */ - if (test_24bit_addr(ip, addr)) { - /* within range */ - old = ftrace_call_replace(ip, addr, 1); - new = ppc_inst(PPC_RAW_NOP()); - return ftrace_modify_code(ip, old, new); - } else if (core_kernel_text(ip)) { - return __ftrace_make_nop_kernel(rec, addr); - } else if (!IS_ENABLED(CONFIG_MODULES)) { - return -EINVAL; - } - - /* - * Out of range jumps are called from modules. - * We should either already have a pointer to the module - * or it has been passed in. - */ - if (!rec->arch.mod) { - if (!mod) { - pr_err("No module loaded addr=%lx\n", addr); - return -EFAULT; - } - rec->arch.mod = mod; - } else if (mod) { - if (mod != rec->arch.mod) { - pr_err("Record mod %p not equal to passed in mod %p\n", - rec->arch.mod, mod); - return -EINVAL; - } - /* nothing to do if mod == rec->arch.mod */ - } else - mod = rec->arch.mod; - - return __ftrace_make_nop(mod, rec, addr); -} - #ifdef CONFIG_MODULES /* * Examine the existing instructions for __ftrace_make_call. @@ -607,6 +419,38 @@ int ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr, } #endif +int ftrace_make_nop(struct module *mod, struct dyn_ftrace *rec, unsigned long addr) +{ + unsigned long tramp, ip = rec->ip; + ppc_inst_t old, new; + + /* Nop-out the ftrace location */ + new = ppc_inst(PPC_RAW_NOP()); + if (is_offset_in_branch_range(addr - ip)) { + /* Within range */ + old = ftrace_create_branch_inst(ip, addr, 1); + return ftrace_modify_code(ip, old, new); + } else if (core_kernel_text(ip)) { + /* We would be branching to one of our ftrace tramps */ + tramp = find_ftrace_tramp(ip); + if (!tramp) { + pr_err("0x%lx: No ftrace trampolines reachable\n", ip); + return -EINVAL; + } + old = ftrace_create_branch_inst(ip, tramp, 1); + return ftrace_modify_code(ip, old, new); + } else if (IS_ENABLED(CONFIG_MODULES)) { + /* Module code would be going to one of the module stubs */ + if (!mod) + mod = rec->arch.mod; + tramp = (addr == (unsigned long)ftrace_caller ? mod->arch.tramp : mod->arch.tramp_regs); + old = ftrace_create_branch_inst(ip, tramp, 1); + return ftrace_modify_code(ip, old, new); + } + + return -EINVAL; +} + int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec) { unsigned long addr, ip = rec->ip; From patchwork Mon Jun 19 09:47:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naveen N Rao X-Patchwork-Id: 1796477 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=112.213.38.117; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=hye2PuV5; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Ql54R6nFbz20Z9 for ; Mon, 19 Jun 2023 20:05:43 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=hye2PuV5; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Ql54R5hMwz3dBy for ; Mon, 19 Jun 2023 20:05:43 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=hye2PuV5; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=kernel.org (client-ip=2604:1380:4641:c500::1; helo=dfw.source.kernel.org; envelope-from=naveen@kernel.org; receiver=lists.ozlabs.org) Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Ql4p82wDjz3bfK for ; Mon, 19 Jun 2023 19:53:20 +1000 (AEST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id BDD0E601CD; Mon, 19 Jun 2023 09:53:18 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9B1D3C433C8; Mon, 19 Jun 2023 09:53:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1687168398; bh=1WQ3lhGz5caRaaobUBPQvT/KIGPZ8w9+nESjHNgmp3c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hye2PuV57U7eBCAajtwe3MjBecp8HtNUJqe/O3c+AhMHQkJ9fPF0ofzIn40O4PcNo zrIrcQWhiSY17TTWKGs19afSBDDdw0cItIGJMQtYD2cEl8TMNXhhNyHnlQsUsVTm+d kbrdaUJ/WvABlQpfNkdMlHGOyy1PmYwMhl78Evf5zRGkPIkXy5gRpwSdXk/RPt/3rZ 9NFNQZs2+IDPSI1wnnkaNkcRvLFVB4iOVapkRp8FBNWcYGVrpQiyfbiUWv0rsQJjlA E+UrKtKhioiXLcf5GNB47L1Wrp3drxBJFCyovuIDO4s90kUE9WNCdGUFd1JLOTLfjW AXDWpuW91HGHw== From: Naveen N Rao To: Subject: [PATCH 12/17] powerpc/ftrace: Simplify ftrace_make_call() Date: Mon, 19 Jun 2023 15:17:30 +0530 Message-Id: <2d28866d2f556488a663981abe5621511efb207b.1687166935.git.naveen@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Steven Rostedt Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Now that we validate the ftrace location during initialization in ftrace_init_nop(), we can simplify ftrace_make_call() to replace the nop without worrying about the instructions surrounding the ftrace location. Note that we continue to ensure that we have a nop at the ftrace location before patching it. Signed-off-by: Naveen N Rao Reviewed-by: Christophe Leroy --- arch/powerpc/kernel/trace/ftrace.c | 187 +++++------------------------ 1 file changed, 31 insertions(+), 156 deletions(-) diff --git a/arch/powerpc/kernel/trace/ftrace.c b/arch/powerpc/kernel/trace/ftrace.c index 05153a1038fdff..6ea8b90246a540 100644 --- a/arch/powerpc/kernel/trace/ftrace.c +++ b/arch/powerpc/kernel/trace/ftrace.c @@ -129,162 +129,6 @@ static unsigned long find_ftrace_tramp(unsigned long ip) return 0; } -#ifdef CONFIG_MODULES -/* - * Examine the existing instructions for __ftrace_make_call. - * They should effectively be a NOP, and follow formal constraints, - * depending on the ABI. Return false if they don't. - */ -static bool expected_nop_sequence(void *ip, ppc_inst_t op0, ppc_inst_t op1) -{ - if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_REGS)) - return ppc_inst_equal(op0, ppc_inst(PPC_RAW_NOP())); - else - return ppc_inst_equal(op0, ppc_inst(PPC_RAW_BRANCH(8))) && - ppc_inst_equal(op1, ppc_inst(PPC_INST_LD_TOC)); -} - -static int -__ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) -{ - ppc_inst_t op[2]; - void *ip = (void *)rec->ip; - unsigned long entry, ptr, tramp; - struct module *mod = rec->arch.mod; - - /* read where this goes */ - if (copy_inst_from_kernel_nofault(op, ip)) - return -EFAULT; - - if (!IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_REGS) && - copy_inst_from_kernel_nofault(op + 1, ip + 4)) - return -EFAULT; - - if (!expected_nop_sequence(ip, op[0], op[1])) { - pr_err("Unexpected call sequence at %p: %08lx %08lx\n", ip, - ppc_inst_as_ulong(op[0]), ppc_inst_as_ulong(op[1])); - return -EINVAL; - } - - /* If we never set up ftrace trampoline(s), then bail */ - if (!mod->arch.tramp || - (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_REGS) && !mod->arch.tramp_regs)) { - pr_err("No ftrace trampoline\n"); - return -EINVAL; - } - - if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_REGS) && rec->flags & FTRACE_FL_REGS) - tramp = mod->arch.tramp_regs; - else - tramp = mod->arch.tramp; - - if (module_trampoline_target(mod, tramp, &ptr)) { - pr_err("Failed to get trampoline target\n"); - return -EFAULT; - } - - pr_devel("trampoline target %lx", ptr); - - entry = ppc_global_function_entry((void *)addr); - /* This should match what was called */ - if (ptr != entry) { - pr_err("addr %lx does not match expected %lx\n", ptr, entry); - return -EINVAL; - } - - if (patch_branch(ip, tramp, BRANCH_SET_LINK)) { - pr_err("REL24 out of range!\n"); - return -EINVAL; - } - - return 0; -} -#else -static int __ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) -{ - return 0; -} -#endif /* CONFIG_MODULES */ - -static int __ftrace_make_call_kernel(struct dyn_ftrace *rec, unsigned long addr) -{ - ppc_inst_t op; - void *ip = (void *)rec->ip; - unsigned long tramp, entry, ptr; - - /* Make sure we're being asked to patch branch to a known ftrace addr */ - entry = ppc_global_function_entry((void *)ftrace_caller); - ptr = ppc_global_function_entry((void *)addr); - - if (ptr != entry && IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_REGS)) - entry = ppc_global_function_entry((void *)ftrace_regs_caller); - - if (ptr != entry) { - pr_err("Unknown ftrace addr to patch: %ps\n", (void *)ptr); - return -EINVAL; - } - - /* Make sure we have a nop */ - if (copy_inst_from_kernel_nofault(&op, ip)) { - pr_err("Unable to read ftrace location %p\n", ip); - return -EFAULT; - } - - if (!ppc_inst_equal(op, ppc_inst(PPC_RAW_NOP()))) { - pr_err("Unexpected call sequence at %p: %08lx\n", - ip, ppc_inst_as_ulong(op)); - return -EINVAL; - } - - tramp = find_ftrace_tramp((unsigned long)ip); - if (!tramp) { - pr_err("No ftrace trampolines reachable from %ps\n", ip); - return -EINVAL; - } - - if (patch_branch(ip, tramp, BRANCH_SET_LINK)) { - pr_err("Error patching branch to ftrace tramp!\n"); - return -EINVAL; - } - - return 0; -} - -int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) -{ - unsigned long ip = rec->ip; - ppc_inst_t old, new; - - /* - * If the calling address is more that 24 bits away, - * then we had to use a trampoline to make the call. - * Otherwise just update the call site. - */ - if (test_24bit_addr(ip, addr)) { - /* within range */ - old = ppc_inst(PPC_RAW_NOP()); - new = ftrace_call_replace(ip, addr, 1); - return ftrace_modify_code(ip, old, new); - } else if (core_kernel_text(ip)) { - return __ftrace_make_call_kernel(rec, addr); - } else if (!IS_ENABLED(CONFIG_MODULES)) { - /* We should not get here without modules */ - return -EINVAL; - } - - /* - * Out of range jumps are called from modules. - * Being that we are converting from nop, it had better - * already have a module defined. - */ - if (!rec->arch.mod) { - pr_err("No module loaded\n"); - return -EINVAL; - } - - return __ftrace_make_call(rec, addr); -} - #ifdef CONFIG_DYNAMIC_FTRACE_WITH_REGS #ifdef CONFIG_MODULES static int @@ -419,6 +263,37 @@ int ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr, } #endif +int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) +{ + unsigned long tramp, ip = rec->ip; + ppc_inst_t old, new; + struct module *mod; + + old = ppc_inst(PPC_RAW_NOP()); + if (is_offset_in_branch_range(addr - ip)) { + /* Within range */ + new = ftrace_create_branch_inst(ip, addr, 1); + return ftrace_modify_code(ip, old, new); + } else if (core_kernel_text(ip)) { + /* We would be branching to one of our ftrace tramps */ + tramp = find_ftrace_tramp(ip); + if (!tramp) { + pr_err("0x%lx: No ftrace trampolines reachable\n", ip); + return -EINVAL; + } + new = ftrace_create_branch_inst(ip, tramp, 1); + return ftrace_modify_code(ip, old, new); + } else if (IS_ENABLED(CONFIG_MODULES)) { + /* Module code would be going to one of the module stubs */ + mod = rec->arch.mod; + tramp = (addr == (unsigned long)ftrace_caller ? mod->arch.tramp : mod->arch.tramp_regs); + new = ftrace_create_branch_inst(ip, tramp, 1); + return ftrace_modify_code(ip, old, new); + } + + return -EINVAL; +} + int ftrace_make_nop(struct module *mod, struct dyn_ftrace *rec, unsigned long addr) { unsigned long tramp, ip = rec->ip; From patchwork Mon Jun 19 09:47:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naveen N Rao X-Patchwork-Id: 1796478 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=Y50bS7Ip; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Ql55Q4y40z20XZ for ; Mon, 19 Jun 2023 20:06:34 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=Y50bS7Ip; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Ql55Q2S47z3c8L for ; Mon, 19 Jun 2023 20:06:34 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=Y50bS7Ip; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=kernel.org (client-ip=2604:1380:4641:c500::1; helo=dfw.source.kernel.org; envelope-from=naveen@kernel.org; receiver=lists.ozlabs.org) Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Ql4pF58bjz3bh5 for ; Mon, 19 Jun 2023 19:53:25 +1000 (AEST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 188E2601CD; Mon, 19 Jun 2023 09:53:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id DDC11C433C0; Mon, 19 Jun 2023 09:53:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1687168403; bh=4BIFSN0aQLOkDhEOEPJRJLNag5NGLIcNmv1ORO+002w=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Y50bS7IprMYs5zNAdqpJqlJ2YKG1bojwELtQPrLawaFjYN36E15EYIen55WT/amX+ x9HjX52cSNMXNmCBW3B7/xbQnX+uyPKGdjKbtyHZSWIM2Q3LZ9ERDrYsfm2JWdsPy4 Ma9KZTpZS/EbU/NOzq0+3C5vpEvFl65nuIuHZbJFxIK1LpOJTQybK1/1ZIZK0BCZ6+ TsjPjYteaS3m6yp+sCtq6HC+Q66kcgL6iepqSYMAHT8NF5Mc5WFZlOVr7haK0fQQL+ pr1kOnPsItz3fcsMtBi9eydc+Jj/ciwASRNXsztNfc3oq2e1uAZe47ecXeFzjDhxb+ NbL27HLAb4NOQ== From: Naveen N Rao To: Subject: [PATCH 13/17] powerpc/ftrace: Simplify ftrace_modify_call() Date: Mon, 19 Jun 2023 15:17:31 +0530 Message-Id: <06275720939f8ee4c2f61c9e9a3e89b1fa3c441d.1687166935.git.naveen@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Steven Rostedt Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Now that we validate the ftrace location during initialization in ftrace_init_nop(), we can simplify ftrace_modify_call() to patch-in the updated branch instruction without worrying about the instructions surrounding the ftrace location. Note that we continue to ensure we have the expected branch instruction at the ftrace location before patching it with the updated branch destination. Signed-off-by: Naveen N Rao Reviewed-by: Christophe Leroy --- arch/powerpc/kernel/trace/ftrace.c | 161 ++++------------------------- 1 file changed, 21 insertions(+), 140 deletions(-) diff --git a/arch/powerpc/kernel/trace/ftrace.c b/arch/powerpc/kernel/trace/ftrace.c index 6ea8b90246a540..c37e22c6c26521 100644 --- a/arch/powerpc/kernel/trace/ftrace.c +++ b/arch/powerpc/kernel/trace/ftrace.c @@ -89,33 +89,11 @@ static inline int ftrace_modify_code(unsigned long ip, ppc_inst_t old, ppc_inst_ return ret; } -/* - * Helper functions that are the same for both PPC64 and PPC32. - */ -static int test_24bit_addr(unsigned long ip, unsigned long addr) -{ - addr = ppc_function_entry((void *)addr); - - return is_offset_in_branch_range(addr - ip); -} - static int is_bl_op(ppc_inst_t op) { return (ppc_inst_val(op) & ~PPC_LI_MASK) == PPC_RAW_BL(0); } -static unsigned long find_bl_target(unsigned long ip, ppc_inst_t op) -{ - int offset; - - offset = PPC_LI(ppc_inst_val(op)); - /* make it signed */ - if (offset & 0x02000000) - offset |= 0xfe000000; - - return ip + (long)offset; -} - static unsigned long find_ftrace_tramp(unsigned long ip) { int i; @@ -130,115 +108,16 @@ static unsigned long find_ftrace_tramp(unsigned long ip) } #ifdef CONFIG_DYNAMIC_FTRACE_WITH_REGS -#ifdef CONFIG_MODULES -static int -__ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr, - unsigned long addr) +int ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr, unsigned long addr) { - ppc_inst_t op; - unsigned long ip = rec->ip; - unsigned long entry, ptr, tramp; - struct module *mod = rec->arch.mod; - - /* If we never set up ftrace trampolines, then bail */ - if (!mod->arch.tramp || !mod->arch.tramp_regs) { - pr_err("No ftrace trampoline\n"); - return -EINVAL; - } - - /* read where this goes */ - if (copy_inst_from_kernel_nofault(&op, (void *)ip)) { - pr_err("Fetching opcode failed.\n"); - return -EFAULT; - } - - /* Make sure that this is still a 24bit jump */ - if (!is_bl_op(op)) { - pr_err("Not expected bl: opcode is %08lx\n", ppc_inst_as_ulong(op)); - return -EINVAL; - } - - /* lets find where the pointer goes */ - tramp = find_bl_target(ip, op); - entry = ppc_global_function_entry((void *)old_addr); - - pr_devel("ip:%lx jumps to %lx", ip, tramp); - - if (tramp != entry) { - /* old_addr is not within range, so we must have used a trampoline */ - if (module_trampoline_target(mod, tramp, &ptr)) { - pr_err("Failed to get trampoline target\n"); - return -EFAULT; - } - - pr_devel("trampoline target %lx", ptr); - - /* This should match what was called */ - if (ptr != entry) { - pr_err("addr %lx does not match expected %lx\n", ptr, entry); - return -EINVAL; - } - } - - /* The new target may be within range */ - if (test_24bit_addr(ip, addr)) { - /* within range */ - if (patch_branch((u32 *)ip, addr, BRANCH_SET_LINK)) { - pr_err("REL24 out of range!\n"); - return -EINVAL; - } - - return 0; - } - - if (rec->flags & FTRACE_FL_REGS) - tramp = mod->arch.tramp_regs; - else - tramp = mod->arch.tramp; - - if (module_trampoline_target(mod, tramp, &ptr)) { - pr_err("Failed to get trampoline target\n"); - return -EFAULT; - } - - pr_devel("trampoline target %lx", ptr); - - entry = ppc_global_function_entry((void *)addr); - /* This should match what was called */ - if (ptr != entry) { - pr_err("addr %lx does not match expected %lx\n", ptr, entry); - return -EINVAL; - } - - if (patch_branch((u32 *)ip, tramp, BRANCH_SET_LINK)) { - pr_err("REL24 out of range!\n"); - return -EINVAL; - } - - return 0; -} -#else -static int __ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr, unsigned long addr) -{ - return 0; -} -#endif - -int ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr, - unsigned long addr) -{ - unsigned long ip = rec->ip; + unsigned long tramp, tramp_old, ip = rec->ip; ppc_inst_t old, new; + struct module *mod; - /* - * If the calling address is more that 24 bits away, - * then we had to use a trampoline to make the call. - * Otherwise just update the call site. - */ - if (test_24bit_addr(ip, addr) && test_24bit_addr(ip, old_addr)) { - /* within range */ - old = ftrace_call_replace(ip, old_addr, 1); - new = ftrace_call_replace(ip, addr, 1); + if (is_offset_in_branch_range(old_addr - ip) && is_offset_in_branch_range(addr - ip)) { + /* Within range */ + old = ftrace_create_branch_inst(ip, old_addr, 1); + new = ftrace_create_branch_inst(ip, addr, 1); return ftrace_modify_code(ip, old, new); } else if (core_kernel_text(ip)) { /* @@ -246,20 +125,22 @@ int ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr, * variant, so there is nothing to do here */ return 0; - } else if (!IS_ENABLED(CONFIG_MODULES)) { - /* We should not get here without modules */ - return -EINVAL; + } else if (IS_ENABLED(CONFIG_MODULES)) { + /* Module code would be going to one of the module stubs */ + mod = rec->arch.mod; + if (addr == (unsigned long)ftrace_caller) { + tramp_old = mod->arch.tramp_regs; + tramp = mod->arch.tramp; + } else { + tramp_old = mod->arch.tramp; + tramp = mod->arch.tramp_regs; + } + old = ftrace_create_branch_inst(ip, tramp_old, 1); + new = ftrace_create_branch_inst(ip, tramp, 1); + return ftrace_modify_code(ip, old, new); } - /* - * Out of range jumps are called from modules. - */ - if (!rec->arch.mod) { - pr_err("No module loaded\n"); - return -EINVAL; - } - - return __ftrace_modify_call(rec, old_addr, addr); + return -EINVAL; } #endif From patchwork Mon Jun 19 09:47:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naveen N Rao X-Patchwork-Id: 1796455 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=fpsi3CYW; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Ql4s74m8dz20Wk for ; Mon, 19 Jun 2023 19:55:55 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=fpsi3CYW; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Ql4s73cWtz3brV for ; Mon, 19 Jun 2023 19:55:55 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=fpsi3CYW; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=kernel.org (client-ip=2604:1380:4641:c500::1; helo=dfw.source.kernel.org; envelope-from=naveen@kernel.org; receiver=lists.ozlabs.org) Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Ql4n32spcz30f6 for ; Mon, 19 Jun 2023 19:52:23 +1000 (AEST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id BAF1E602E0; Mon, 19 Jun 2023 09:52:21 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 87343C433C8; Mon, 19 Jun 2023 09:52:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1687168341; bh=fvNzA4Q938jsAQnFgUt78HMwwQ810FzVX7BIJSSbuFU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fpsi3CYW+G70ajSQFtErV1h/yNfLRqjCbqErJA4Tk+z8P9FHKA7ugTKicQRN5x4E7 876nv4gi1Z8zZeghWcnjOG+nEc9zJRmaXmoqnG8TKO/5JfqpLwL4JMI8bMr1m7Np1o KsPVVC3lLQuPWRx5aeMuI0t6dgYGVr2glTKdMX+vw8SQDN9lVhkrYlYuvWfnsipBti RjOmM7uw/ivQuZ0SaEDv67lkYaHtCCJv++/jMOxHqEKUHiH+vUHvNRZFACjshiEtuY g2Btz4dIkLplGsAg8fWCq4FjkUSYB2iw3mChdkidoFWIa7Of9Fhgw4ujAlh3BUQaiH CUes9jsaRareQ== From: Naveen N Rao To: Subject: [PATCH 14/17] powerpc/ftrace: Replace use of ftrace_call_replace() with ftrace_create_branch_inst() Date: Mon, 19 Jun 2023 15:17:32 +0530 Message-Id: <953513b88fa922ba7a66d772dc1310710efe9177.1687166935.git.naveen@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Steven Rostedt Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" ftrace_create_branch_inst() is clearer about its intent than ftrace_call_replace(). Signed-off-by: Naveen N Rao Reviewed-by: Christophe Leroy --- arch/powerpc/kernel/trace/ftrace.c | 17 ++--------------- 1 file changed, 2 insertions(+), 15 deletions(-) diff --git a/arch/powerpc/kernel/trace/ftrace.c b/arch/powerpc/kernel/trace/ftrace.c index c37e22c6c26521..422dd760fbe013 100644 --- a/arch/powerpc/kernel/trace/ftrace.c +++ b/arch/powerpc/kernel/trace/ftrace.c @@ -41,19 +41,6 @@ static ppc_inst_t ftrace_create_branch_inst(unsigned long ip, unsigned long addr return op; } -static ppc_inst_t -ftrace_call_replace(unsigned long ip, unsigned long addr, int link) -{ - ppc_inst_t op; - - addr = ppc_function_entry((void *)addr); - - /* if (link) set op to 'bl' else 'b' */ - create_branch(&op, (u32 *)ip, addr, link ? BRANCH_SET_LINK : 0); - - return op; -} - static inline int ftrace_read_inst(unsigned long ip, ppc_inst_t *op) { if (copy_inst_from_kernel_nofault(op, (void *)ip)) { @@ -275,14 +262,14 @@ int ftrace_update_ftrace_func(ftrace_func_t func) int ret; old = ppc_inst_read((u32 *)&ftrace_call); - new = ftrace_call_replace(ip, (unsigned long)func, 1); + new = ftrace_create_branch_inst(ip, ppc_function_entry(func), 1); ret = ftrace_modify_code(ip, old, new); /* Also update the regs callback function */ if (IS_ENABLED(CONFIG_DYNAMIC_FTRACE_WITH_REGS) && !ret) { ip = (unsigned long)(&ftrace_regs_call); old = ppc_inst_read((u32 *)&ftrace_regs_call); - new = ftrace_call_replace(ip, (unsigned long)func, 1); + new = ftrace_create_branch_inst(ip, ppc_function_entry(func), 1); ret = ftrace_modify_code(ip, old, new); } From patchwork Mon Jun 19 09:47:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naveen N Rao X-Patchwork-Id: 1796458 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=rZ9SyYIP; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Ql4t62JDdz20Wk for ; Mon, 19 Jun 2023 19:56:46 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=rZ9SyYIP; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Ql4t619Hpz3bwj for ; Mon, 19 Jun 2023 19:56:46 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=rZ9SyYIP; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=kernel.org (client-ip=2604:1380:4641:c500::1; helo=dfw.source.kernel.org; envelope-from=naveen@kernel.org; receiver=lists.ozlabs.org) Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Ql4n86FW7z30XX for ; Mon, 19 Jun 2023 19:52:28 +1000 (AEST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 2215F60A50; Mon, 19 Jun 2023 09:52:27 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E6B2DC433C8; Mon, 19 Jun 2023 09:52:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1687168346; bh=JnzX964nCR1Bazu/wmQxgrJo1YVRNdq0Lxqc7/Cnfpk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rZ9SyYIPDJHrmXVV0mbKNb6xVNE6eFOe2LggYCQv7MVerj5MN8U7p8+E1qn+L5z5n OwIdxyhhxTkuu3zos6o1uyZajSs91a+zJoX0IrAsxsn45Umi3TfE45fCc8yuITV+FR mgaqYT7/Lz11yS4XMCKTJ0bMalfqvfmyaPHbsDR2OgEvG5/DmwiQIiNtxKiZDAN2tM DOjvO0B4Vl94rxLxpFgMwz+FUjgnJ2q5LRWSarytjRjP70MHFjfn2Y9sEGAKUxiur1 ZrWdD5SwLUTwE4vh+3bPmmScm7QpCRtLO8ZmTd2XJ8voFW+t6LxrVHgSCHIJjgdxFP j0SQvp5idemcg== From: Naveen N Rao To: Subject: [PATCH 15/17] powerpc/ftrace: Implement ftrace_replace_code() Date: Mon, 19 Jun 2023 15:17:33 +0530 Message-Id: X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Steven Rostedt Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Implement ftrace_replace_code() to consolidate logic from the different ftrace patching routines: ftrace_make_nop(), ftrace_make_call() and ftrace_modify_call(). Note that ftrace_make_call() is still required primarily to handle patching modules during their load time. The other two routines should no longer be called. This lays the groundwork to enable better control in patching ftrace locations, including the ability to nop-out preceding profiling instructions when ftrace is disabled. Signed-off-by: Naveen N Rao Reviewed-by: Christophe Leroy --- arch/powerpc/kernel/trace/ftrace.c | 173 ++++++++++++++++------------- 1 file changed, 96 insertions(+), 77 deletions(-) diff --git a/arch/powerpc/kernel/trace/ftrace.c b/arch/powerpc/kernel/trace/ftrace.c index 422dd760fbe013..cf9dce77527920 100644 --- a/arch/powerpc/kernel/trace/ftrace.c +++ b/arch/powerpc/kernel/trace/ftrace.c @@ -94,104 +94,123 @@ static unsigned long find_ftrace_tramp(unsigned long ip) return 0; } +static int ftrace_get_call_inst(struct dyn_ftrace *rec, unsigned long addr, ppc_inst_t *call_inst) +{ + unsigned long ip = rec->ip; + unsigned long stub; + + if (is_offset_in_branch_range(addr - ip)) { + /* Within range */ + stub = addr; +#ifdef CONFIG_MODULES + } else if (rec->arch.mod) { + /* Module code would be going to one of the module stubs */ + stub = (addr == (unsigned long)ftrace_caller ? rec->arch.mod->arch.tramp : + rec->arch.mod->arch.tramp_regs); +#endif + } else if (core_kernel_text(ip)) { + /* We would be branching to one of our ftrace stubs */ + stub = find_ftrace_tramp(ip); + if (!stub) { + pr_err("0x%lx: No ftrace stubs reachable\n", ip); + return -EINVAL; + } + } else { + return -EINVAL; + } + + *call_inst = ftrace_create_branch_inst(ip, stub, 1); + return 0; +} + #ifdef CONFIG_DYNAMIC_FTRACE_WITH_REGS int ftrace_modify_call(struct dyn_ftrace *rec, unsigned long old_addr, unsigned long addr) { - unsigned long tramp, tramp_old, ip = rec->ip; - ppc_inst_t old, new; - struct module *mod; - - if (is_offset_in_branch_range(old_addr - ip) && is_offset_in_branch_range(addr - ip)) { - /* Within range */ - old = ftrace_create_branch_inst(ip, old_addr, 1); - new = ftrace_create_branch_inst(ip, addr, 1); - return ftrace_modify_code(ip, old, new); - } else if (core_kernel_text(ip)) { - /* - * We always patch out of range locations to go to the regs - * variant, so there is nothing to do here - */ - return 0; - } else if (IS_ENABLED(CONFIG_MODULES)) { - /* Module code would be going to one of the module stubs */ - mod = rec->arch.mod; - if (addr == (unsigned long)ftrace_caller) { - tramp_old = mod->arch.tramp_regs; - tramp = mod->arch.tramp; - } else { - tramp_old = mod->arch.tramp; - tramp = mod->arch.tramp_regs; - } - old = ftrace_create_branch_inst(ip, tramp_old, 1); - new = ftrace_create_branch_inst(ip, tramp, 1); - return ftrace_modify_code(ip, old, new); - } - + /* This should never be called since we override ftrace_replace_code() */ + WARN_ON(1); return -EINVAL; } #endif int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) { - unsigned long tramp, ip = rec->ip; ppc_inst_t old, new; - struct module *mod; + int ret; + + /* This can only ever be called during module load */ + if (WARN_ON(!IS_ENABLED(CONFIG_MODULES) || core_kernel_text(rec->ip))) + return -EINVAL; old = ppc_inst(PPC_RAW_NOP()); - if (is_offset_in_branch_range(addr - ip)) { - /* Within range */ - new = ftrace_create_branch_inst(ip, addr, 1); - return ftrace_modify_code(ip, old, new); - } else if (core_kernel_text(ip)) { - /* We would be branching to one of our ftrace tramps */ - tramp = find_ftrace_tramp(ip); - if (!tramp) { - pr_err("0x%lx: No ftrace trampolines reachable\n", ip); - return -EINVAL; - } - new = ftrace_create_branch_inst(ip, tramp, 1); - return ftrace_modify_code(ip, old, new); - } else if (IS_ENABLED(CONFIG_MODULES)) { - /* Module code would be going to one of the module stubs */ - mod = rec->arch.mod; - tramp = (addr == (unsigned long)ftrace_caller ? mod->arch.tramp : mod->arch.tramp_regs); - new = ftrace_create_branch_inst(ip, tramp, 1); - return ftrace_modify_code(ip, old, new); - } + ret = ftrace_get_call_inst(rec, addr, &new); + if (ret) + return ret; - return -EINVAL; + return ftrace_modify_code(rec->ip, old, new); } int ftrace_make_nop(struct module *mod, struct dyn_ftrace *rec, unsigned long addr) { - unsigned long tramp, ip = rec->ip; - ppc_inst_t old, new; + /* + * This should never be called since we override ftrace_replace_code(), + * as well as ftrace_init_nop() + */ + WARN_ON(1); + return -EINVAL; +} - /* Nop-out the ftrace location */ - new = ppc_inst(PPC_RAW_NOP()); - if (is_offset_in_branch_range(addr - ip)) { - /* Within range */ - old = ftrace_create_branch_inst(ip, addr, 1); - return ftrace_modify_code(ip, old, new); - } else if (core_kernel_text(ip)) { - /* We would be branching to one of our ftrace tramps */ - tramp = find_ftrace_tramp(ip); - if (!tramp) { - pr_err("0x%lx: No ftrace trampolines reachable\n", ip); - return -EINVAL; +void ftrace_replace_code(int enable) +{ + ppc_inst_t old, new, call_inst, new_call_inst; + ppc_inst_t nop_inst = ppc_inst(PPC_RAW_NOP()); + unsigned long ip, new_addr, addr; + struct ftrace_rec_iter *iter; + struct dyn_ftrace *rec; + int ret = 0, update; + + for_ftrace_rec_iter(iter) { + rec = ftrace_rec_iter_record(iter); + ip = rec->ip; + + if (rec->flags & FTRACE_FL_DISABLED && !(rec->flags & FTRACE_FL_ENABLED)) + continue; + + addr = ftrace_get_addr_curr(rec); + new_addr = ftrace_get_addr_new(rec); + update = ftrace_update_record(rec, enable); + + switch (update) { + case FTRACE_UPDATE_IGNORE: + default: + continue; + case FTRACE_UPDATE_MODIFY_CALL: + ret = ftrace_get_call_inst(rec, new_addr, &new_call_inst); + ret |= ftrace_get_call_inst(rec, addr, &call_inst); + old = call_inst; + new = new_call_inst; + break; + case FTRACE_UPDATE_MAKE_NOP: + ret = ftrace_get_call_inst(rec, addr, &call_inst); + old = call_inst; + new = nop_inst; + break; + case FTRACE_UPDATE_MAKE_CALL: + ret = ftrace_get_call_inst(rec, new_addr, &call_inst); + old = nop_inst; + new = call_inst; + break; } - old = ftrace_create_branch_inst(ip, tramp, 1); - return ftrace_modify_code(ip, old, new); - } else if (IS_ENABLED(CONFIG_MODULES)) { - /* Module code would be going to one of the module stubs */ - if (!mod) - mod = rec->arch.mod; - tramp = (addr == (unsigned long)ftrace_caller ? mod->arch.tramp : mod->arch.tramp_regs); - old = ftrace_create_branch_inst(ip, tramp, 1); - return ftrace_modify_code(ip, old, new); + + if (!ret) + ret = ftrace_modify_code(ip, old, new); + if (ret) + goto out; } - return -EINVAL; +out: + if (ret) + ftrace_bug(ret, rec); + return; } int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec) From patchwork Mon Jun 19 09:47:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naveen N Rao X-Patchwork-Id: 1796468 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=RIWZlTaW; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Ql4v55nf4z20Wk for ; Mon, 19 Jun 2023 19:57:37 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=RIWZlTaW; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Ql4v50KWJz3c8h for ; Mon, 19 Jun 2023 19:57:37 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=RIWZlTaW; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=kernel.org (client-ip=2604:1380:4641:c500::1; helo=dfw.source.kernel.org; envelope-from=naveen@kernel.org; receiver=lists.ozlabs.org) Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Ql4nF620wz30h9 for ; Mon, 19 Jun 2023 19:52:33 +1000 (AEST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 2517D608CC; Mon, 19 Jun 2023 09:52:32 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 05920C433C8; Mon, 19 Jun 2023 09:52:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1687168351; bh=MkVRM0ul/nnBbHLoxxLQM/UvH1NA3bk6VPTLLF29Ozs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RIWZlTaWX2oc/15+x7SNglvBRGFKS+y/e1n5ttpZ7ZppHQY9oDx46laZ6rWkvDSb3 DFH0qOvJH1paXlO2JtXn2zpGG56F5+1Dr25RqG8gQ+5rSrI4B7+b/abXMC7X6rZQPx VCY3BSCkgUav77sCle4gFs68P/u7uROaTr8v9EQZzmCvkdZMWnoRuo36Sfa7jyg4h5 WPVHMJPUWcl9Lgv5I4giBjzrfO+jFStw274utXQPQ0DgkDONRpxtXEsK+llAXp41lT kBihk5H8kcyFivv8pXACJWn1zxRuAPg+uOiQGQ7muZ+yHvNDJg62Ck6jgJpBFe3VRl 8efmdx7XP27sw== From: Naveen N Rao To: Subject: [PATCH 16/17] powerpc/ftrace: Add support for -fpatchable-function-entry Date: Mon, 19 Jun 2023 15:17:34 +0530 Message-Id: <68586d22981a2c3bb45f27a2b621173d10a7d092.1687166935.git.naveen@kernel.org> X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Steven Rostedt Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" GCC v13.1 updated support for -fpatchable-function-entry on ppc64le to emit nops after the local entry point, rather than before it. This allows us to use this in the kernel for ftrace purposes. A new script is added under arch/powerpc/tools/ to help detect if nops are emitted after the function local entry point, or before the global entry point. With -fpatchable-function-entry, we no longer have the profiling instructions generated at function entry, so we only need to validate the presence of two nops at the ftrace location in ftrace_init_nop(). We patch the preceding instruction with 'mflr r0' to match the -mprofile-kernel ABI for subsequent ftrace use. This changes the profiling instructions used on ppc32. The default -pg option emits an additional 'stw' instruction after 'mflr r0' and before the branch to _mcount 'bl _mcount'. This is very similar to the original -mprofile-kernel implementation on ppc64le, where an additional 'std' instruction was used to save LR to its save location in the caller's stackframe. Subsequently, this additional store was removed in later compiler versions for performance reasons. The same reasons apply for ppc32 so we only patch in a 'mflr r0'. Signed-off-by: Naveen N Rao Reviewed-by: Christophe Leroy --- arch/powerpc/Kconfig | 14 +++++++--- arch/powerpc/Makefile | 5 ++++ arch/powerpc/include/asm/ftrace.h | 6 +++-- arch/powerpc/include/asm/vermagic.h | 4 ++- arch/powerpc/kernel/module_64.c | 2 +- arch/powerpc/kernel/trace/ftrace.c | 14 ++++++++-- arch/powerpc/kernel/trace/ftrace_entry.S | 2 ++ .../gcc-check-fpatchable-function-entry.sh | 26 +++++++++++++++++++ 8 files changed, 64 insertions(+), 9 deletions(-) create mode 100755 arch/powerpc/tools/gcc-check-fpatchable-function-entry.sh diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index bff5820b7cda14..9352d8e68152e1 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -187,6 +187,7 @@ config PPC select DYNAMIC_FTRACE if FUNCTION_TRACER select EDAC_ATOMIC_SCRUB select EDAC_SUPPORT + select FTRACE_MCOUNT_USE_PATCHABLE_FUNCTION_ENTRY if ARCH_USING_PATCHABLE_FUNCTION_ENTRY select GENERIC_ATOMIC64 if PPC32 select GENERIC_CLOCKEVENTS_BROADCAST if SMP select GENERIC_CMOS_UPDATE @@ -227,8 +228,8 @@ config PPC select HAVE_DEBUG_KMEMLEAK select HAVE_DEBUG_STACKOVERFLOW select HAVE_DYNAMIC_FTRACE - select HAVE_DYNAMIC_FTRACE_WITH_ARGS if MPROFILE_KERNEL || PPC32 - select HAVE_DYNAMIC_FTRACE_WITH_REGS if MPROFILE_KERNEL || PPC32 + select HAVE_DYNAMIC_FTRACE_WITH_ARGS if ARCH_USING_PATCHABLE_FUNCTION_ENTRY || MPROFILE_KERNEL || PPC32 + select HAVE_DYNAMIC_FTRACE_WITH_REGS if ARCH_USING_PATCHABLE_FUNCTION_ENTRY || MPROFILE_KERNEL || PPC32 select HAVE_EBPF_JIT select HAVE_EFFICIENT_UNALIGNED_ACCESS select HAVE_FAST_GUP @@ -256,7 +257,7 @@ config PPC select HAVE_MOD_ARCH_SPECIFIC select HAVE_NMI if PERF_EVENTS || (PPC64 && PPC_BOOK3S) select HAVE_OPTPROBES - select HAVE_OBJTOOL if PPC32 || MPROFILE_KERNEL + select HAVE_OBJTOOL if ARCH_USING_PATCHABLE_FUNCTION_ENTRY || MPROFILE_KERNEL || PPC32 select HAVE_OBJTOOL_MCOUNT if HAVE_OBJTOOL select HAVE_PERF_EVENTS select HAVE_PERF_EVENTS_NMI if PPC64 @@ -550,6 +551,13 @@ config MPROFILE_KERNEL depends on PPC64 && CPU_LITTLE_ENDIAN && FUNCTION_TRACER def_bool $(success,$(srctree)/arch/powerpc/tools/gcc-check-mprofile-kernel.sh $(CC) -I$(srctree)/include -D__KERNEL__) +config ARCH_USING_PATCHABLE_FUNCTION_ENTRY + depends on FUNCTION_TRACER && (PPC32 || PPC64_ELF_ABI_V2) + depends on $(cc-option,-fpatchable-function-entry=2) + def_bool y if PPC32 + def_bool $(success,$(srctree)/arch/powerpc/tools/gcc-check-fpatchable-function-entry.sh $(CC) -mlittle-endian) if PPC64 && CPU_LITTLE_ENDIAN + def_bool $(success,$(srctree)/arch/powerpc/tools/gcc-check-fpatchable-function-entry.sh $(CC) -mbig-endian) if PPC64 && CPU_BIG_ENDIAN + config HOTPLUG_CPU bool "Support for enabling/disabling CPUs" depends on SMP && (PPC_PSERIES || \ diff --git a/arch/powerpc/Makefile b/arch/powerpc/Makefile index dca73f673d7046..de39478b1c9e9f 100644 --- a/arch/powerpc/Makefile +++ b/arch/powerpc/Makefile @@ -148,11 +148,16 @@ CFLAGS-$(CONFIG_PPC32) += $(call cc-option, $(MULTIPLEWORD)) CFLAGS-$(CONFIG_PPC32) += $(call cc-option,-mno-readonly-in-sdata) ifdef CONFIG_FUNCTION_TRACER +ifdef CONFIG_ARCH_USING_PATCHABLE_FUNCTION_ENTRY +KBUILD_CPPFLAGS += -DCC_USING_PATCHABLE_FUNCTION_ENTRY +CC_FLAGS_FTRACE := -fpatchable-function-entry=2 +else CC_FLAGS_FTRACE := -pg ifdef CONFIG_MPROFILE_KERNEL CC_FLAGS_FTRACE += -mprofile-kernel endif endif +endif CFLAGS-$(CONFIG_TARGET_CPU_BOOL) += -mcpu=$(CONFIG_TARGET_CPU) AFLAGS-$(CONFIG_TARGET_CPU_BOOL) += -mcpu=$(CONFIG_TARGET_CPU) diff --git a/arch/powerpc/include/asm/ftrace.h b/arch/powerpc/include/asm/ftrace.h index ef9f0b97670d1c..9e5a39b6a3114b 100644 --- a/arch/powerpc/include/asm/ftrace.h +++ b/arch/powerpc/include/asm/ftrace.h @@ -11,7 +11,7 @@ #define HAVE_FUNCTION_GRAPH_RET_ADDR_PTR /* Ignore unused weak functions which will have larger offsets */ -#ifdef CONFIG_MPROFILE_KERNEL +#if defined(CONFIG_MPROFILE_KERNEL) || defined(CONFIG_ARCH_USING_PATCHABLE_FUNCTION_ENTRY) #define FTRACE_MCOUNT_MAX_OFFSET 16 #elif defined(CONFIG_PPC32) #define FTRACE_MCOUNT_MAX_OFFSET 8 @@ -22,7 +22,9 @@ extern void _mcount(void); static inline unsigned long ftrace_call_adjust(unsigned long addr) { - /* relocation of mcount call site is the same as the address */ + if (IS_ENABLED(CONFIG_ARCH_USING_PATCHABLE_FUNCTION_ENTRY)) + addr += MCOUNT_INSN_SIZE; + return addr; } diff --git a/arch/powerpc/include/asm/vermagic.h b/arch/powerpc/include/asm/vermagic.h index b054a8576e5deb..6f250fe506bd1c 100644 --- a/arch/powerpc/include/asm/vermagic.h +++ b/arch/powerpc/include/asm/vermagic.h @@ -2,7 +2,9 @@ #ifndef _ASM_VERMAGIC_H #define _ASM_VERMAGIC_H -#ifdef CONFIG_MPROFILE_KERNEL +#ifdef CONFIG_ARCH_USING_PATCHABLE_FUNCTION_ENTRY +#define MODULE_ARCH_VERMAGIC_FTRACE "patchable-function-entry " +#elif defined(CONFIG_MPROFILE_KERNEL) #define MODULE_ARCH_VERMAGIC_FTRACE "mprofile-kernel " #else #define MODULE_ARCH_VERMAGIC_FTRACE "" diff --git a/arch/powerpc/kernel/module_64.c b/arch/powerpc/kernel/module_64.c index 92570289ce08f5..7112adc597a80b 100644 --- a/arch/powerpc/kernel/module_64.c +++ b/arch/powerpc/kernel/module_64.c @@ -465,7 +465,7 @@ int module_frob_arch_sections(Elf64_Ehdr *hdr, return 0; } -#ifdef CONFIG_MPROFILE_KERNEL +#if defined(CONFIG_MPROFILE_KERNEL) || defined(CONFIG_ARCH_USING_PATCHABLE_FUNCTION_ENTRY) static u32 stub_insns[] = { #ifdef CONFIG_PPC_KERNEL_PCREL diff --git a/arch/powerpc/kernel/trace/ftrace.c b/arch/powerpc/kernel/trace/ftrace.c index cf9dce77527920..82010629cf887c 100644 --- a/arch/powerpc/kernel/trace/ftrace.c +++ b/arch/powerpc/kernel/trace/ftrace.c @@ -220,7 +220,12 @@ int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec) int ret = 0; /* Verify instructions surrounding the ftrace location */ - if (IS_ENABLED(CONFIG_PPC32)) { + if (IS_ENABLED(CONFIG_ARCH_USING_PATCHABLE_FUNCTION_ENTRY)) { + /* Expect nops */ + ret = ftrace_validate_inst(ip - 4, ppc_inst(PPC_RAW_NOP())); + if (!ret) + ret = ftrace_validate_inst(ip, ppc_inst(PPC_RAW_NOP())); + } else if (IS_ENABLED(CONFIG_PPC32)) { /* Expected sequence: 'mflr r0', 'stw r0,4(r1)', 'bl _mcount' */ ret = ftrace_validate_inst(ip - 8, ppc_inst(PPC_RAW_MFLR(_R0))); if (!ret) @@ -250,7 +255,12 @@ int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec) /* Nop-out the ftrace location */ new = ppc_inst(PPC_RAW_NOP()); addr = MCOUNT_ADDR; - if (is_offset_in_branch_range(addr - ip)) { + if (IS_ENABLED(CONFIG_ARCH_USING_PATCHABLE_FUNCTION_ENTRY)) { + /* we instead patch-in the 'mflr r0' */ + old = ppc_inst(PPC_RAW_NOP()); + new = ppc_inst(PPC_RAW_MFLR(_R0)); + ret = ftrace_modify_code(ip - 4, old, new); + } else if (is_offset_in_branch_range(addr - ip)) { /* Within range */ old = ftrace_create_branch_inst(ip, addr, 1); ret = ftrace_modify_code(ip, old, new); diff --git a/arch/powerpc/kernel/trace/ftrace_entry.S b/arch/powerpc/kernel/trace/ftrace_entry.S index e8339706e735b1..bab3ab1368a33f 100644 --- a/arch/powerpc/kernel/trace/ftrace_entry.S +++ b/arch/powerpc/kernel/trace/ftrace_entry.S @@ -250,6 +250,7 @@ livepatch_handler: blr #endif /* CONFIG_LIVEPATCH */ +#ifndef CONFIG_ARCH_USING_PATCHABLE_FUNCTION_ENTRY _GLOBAL(mcount) _GLOBAL(_mcount) EXPORT_SYMBOL(_mcount) @@ -257,6 +258,7 @@ EXPORT_SYMBOL(_mcount) mtctr r12 mtlr r0 bctr +#endif #ifdef CONFIG_FUNCTION_GRAPH_TRACER _GLOBAL(return_to_handler) diff --git a/arch/powerpc/tools/gcc-check-fpatchable-function-entry.sh b/arch/powerpc/tools/gcc-check-fpatchable-function-entry.sh new file mode 100755 index 00000000000000..06706903503b6c --- /dev/null +++ b/arch/powerpc/tools/gcc-check-fpatchable-function-entry.sh @@ -0,0 +1,26 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +set -e +set -o pipefail + +# To debug, uncomment the following line +# set -x + +# Output from -fpatchable-function-entry can only vary on ppc64 elfv2, so this +# should not be invoked for other targets. Therefore we can pass in -m64 and +# -mabi explicitly, to take care of toolchains defaulting to other targets. + +# Test whether the compile option -fpatchable-function-entry exists and +# generates appropriate code +echo "int func() { return 0; }" | \ + $* -m64 -mabi=elfv2 -S -x c -O2 -fpatchable-function-entry=2 - -o - 2> /dev/null | \ + grep -q "__patchable_function_entries" + +# Test whether nops are generated after the local entry point +echo "int x; int func() { return x; }" | \ + $* -m64 -mabi=elfv2 -S -x c -O2 -fpatchable-function-entry=2 - -o - 2> /dev/null | \ + awk 'BEGIN { RS = ";" } /\.localentry.*nop.*\n[[:space:]]*nop/ { print $0 }' | \ + grep -q "func:" + +exit 0 From patchwork Mon Jun 19 09:47:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Naveen N Rao X-Patchwork-Id: 1796469 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ozlabs.org (client-ip=2404:9400:2:0:216:3eff:fee1:b9f1; helo=lists.ozlabs.org; envelope-from=linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=AsFnztHC; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2404:9400:2:0:216:3eff:fee1:b9f1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Ql4wC6GfLz20Wk for ; Mon, 19 Jun 2023 19:58:35 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=AsFnztHC; dkim-atps=neutral Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4Ql4wC58dhz3cNY for ; Mon, 19 Jun 2023 19:58:35 +1000 (AEST) X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=AsFnztHC; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=kernel.org (client-ip=2604:1380:4641:c500::1; helo=dfw.source.kernel.org; envelope-from=naveen@kernel.org; receiver=lists.ozlabs.org) Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4Ql4nL7567z30gj for ; Mon, 19 Jun 2023 19:52:38 +1000 (AEST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 5B675602E0; Mon, 19 Jun 2023 09:52:37 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 30BAAC433C8; Mon, 19 Jun 2023 09:52:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1687168356; bh=P9e3L1q1deHAzNU7PwEGs+OQwBEU4pkJ+ZPVhwnpH+I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=AsFnztHCdXi4qBUuJG5Wny0B1W+LXMo52oXHhH40tVoLuoDVKwFlGBzJefEBUQWMB vbKtJUjb2xJ4gFVYIDMD1quGmUCVQjGDFyRTWleuqFHBmy84dIIQlZXHN+9fKZz1YL wx1zJ0LJQe1mTurCn6WAN3KibSOYwmMzk8KZbJczJfoQJmdvtaM8VCmUr9sZzhV2T5 FoGwzumvD+GOVnheENrtukAhdVwCVhc8kLeO2ZnU96tSngpVKzil3Uaxi+mAW5tsIs sOJky6zq63HUTIPNP8QiqdJVmafnaTEA59U0ST3LaKRCIJoTpuuB5/jYt78jFKwq3f ri14xbGconk1g== From: Naveen N Rao To: Subject: [PATCH 17/17] powerpc/ftrace: Create a dummy stackframe to fix stack unwind Date: Mon, 19 Jun 2023 15:17:35 +0530 Message-Id: X-Mailer: git-send-email 2.40.1 In-Reply-To: References: MIME-Version: 1.0 X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Steven Rostedt Errors-To: linuxppc-dev-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" With ppc64 -mprofile-kernel and ppc32 -pg, profiling instructions to call into ftrace are emitted right at function entry. The instruction sequence used is minimal to reduce overhead. Crucially, a stackframe is not created for the function being traced. This breaks stack unwinding since the function being traced does not have a stackframe for itself. As such, it never shows up in the backtrace: /sys/kernel/debug/tracing # echo 1 > /proc/sys/kernel/stack_tracer_enabled /sys/kernel/debug/tracing # cat stack_trace Depth Size Location (17 entries) ----- ---- -------- 0) 4144 32 ftrace_call+0x4/0x44 1) 4112 432 get_page_from_freelist+0x26c/0x1ad0 2) 3680 496 __alloc_pages+0x290/0x1280 3) 3184 336 __folio_alloc+0x34/0x90 4) 2848 176 vma_alloc_folio+0xd8/0x540 5) 2672 272 __handle_mm_fault+0x700/0x1cc0 6) 2400 208 handle_mm_fault+0xf0/0x3f0 7) 2192 80 ___do_page_fault+0x3e4/0xbe0 8) 2112 160 do_page_fault+0x30/0xc0 9) 1952 256 data_access_common_virt+0x210/0x220 10) 1696 400 0xc00000000f16b100 11) 1296 384 load_elf_binary+0x804/0x1b80 12) 912 208 bprm_execve+0x2d8/0x7e0 13) 704 64 do_execveat_common+0x1d0/0x2f0 14) 640 160 sys_execve+0x54/0x70 15) 480 64 system_call_exception+0x138/0x350 16) 416 416 system_call_common+0x160/0x2c4 Fix this by having ftrace create a dummy stackframe for the function being traced. Since this is only relevant when ftrace is active, we nop out the instruction to store LR in the LR save area in the profiling instruction sequence on ppc32 (and in ppc64 with older gcc versions). Instead, in ftrace, we store LR in the LR save area of the previous stackframe, and create a minimal stackframe to represent the function being traced. With this, backtraces now capture the function being traced: /sys/kernel/debug/tracing # cat stack_trace Depth Size Location (17 entries) ----- ---- -------- 0) 3888 32 _raw_spin_trylock+0x8/0x70 1) 3856 576 get_page_from_freelist+0x26c/0x1ad0 2) 3280 64 __alloc_pages+0x290/0x1280 3) 3216 336 __folio_alloc+0x34/0x90 4) 2880 176 vma_alloc_folio+0xd8/0x540 5) 2704 416 __handle_mm_fault+0x700/0x1cc0 6) 2288 96 handle_mm_fault+0xf0/0x3f0 7) 2192 48 ___do_page_fault+0x3e4/0xbe0 8) 2144 192 do_page_fault+0x30/0xc0 9) 1952 608 data_access_common_virt+0x210/0x220 10) 1344 16 0xc0000000334bbb50 11) 1328 416 load_elf_binary+0x804/0x1b80 12) 912 64 bprm_execve+0x2d8/0x7e0 13) 848 176 do_execveat_common+0x1d0/0x2f0 14) 672 192 sys_execve+0x54/0x70 15) 480 64 system_call_exception+0x138/0x350 16) 416 416 system_call_common+0x160/0x2c4 This results in two additional stores in the ftrace entry code, but produces reliable backtraces. Note that this change now aligns with other architectures (arm64, s390, x86). Signed-off-by: Naveen N Rao Reviewed-by: Christophe Leroy --- arch/powerpc/kernel/trace/ftrace.c | 6 ++++-- arch/powerpc/kernel/trace/ftrace_entry.S | 11 ++++++++--- 2 files changed, 12 insertions(+), 5 deletions(-) diff --git a/arch/powerpc/kernel/trace/ftrace.c b/arch/powerpc/kernel/trace/ftrace.c index 82010629cf887c..2956196c98ffdc 100644 --- a/arch/powerpc/kernel/trace/ftrace.c +++ b/arch/powerpc/kernel/trace/ftrace.c @@ -229,13 +229,15 @@ int ftrace_init_nop(struct module *mod, struct dyn_ftrace *rec) /* Expected sequence: 'mflr r0', 'stw r0,4(r1)', 'bl _mcount' */ ret = ftrace_validate_inst(ip - 8, ppc_inst(PPC_RAW_MFLR(_R0))); if (!ret) - ret = ftrace_validate_inst(ip - 4, ppc_inst(PPC_RAW_STW(_R0, _R1, 4))); + ret = ftrace_modify_code(ip - 4, ppc_inst(PPC_RAW_STW(_R0, _R1, 4)), + ppc_inst(PPC_RAW_NOP())); } else if (IS_ENABLED(CONFIG_MPROFILE_KERNEL)) { /* Expected sequence: 'mflr r0', ['std r0,16(r1)'], 'bl _mcount' */ ret = ftrace_read_inst(ip - 4, &old); if (!ret && !ppc_inst_equal(old, ppc_inst(PPC_RAW_MFLR(_R0)))) { ret = ftrace_validate_inst(ip - 8, ppc_inst(PPC_RAW_MFLR(_R0))); - ret |= ftrace_validate_inst(ip - 4, ppc_inst(PPC_RAW_STD(_R0, _R1, 16))); + ret |= ftrace_modify_code(ip - 4, ppc_inst(PPC_RAW_STD(_R0, _R1, 16)), + ppc_inst(PPC_RAW_NOP())); } } else { return -EINVAL; diff --git a/arch/powerpc/kernel/trace/ftrace_entry.S b/arch/powerpc/kernel/trace/ftrace_entry.S index bab3ab1368a33f..05e981fb526c2e 100644 --- a/arch/powerpc/kernel/trace/ftrace_entry.S +++ b/arch/powerpc/kernel/trace/ftrace_entry.S @@ -33,6 +33,11 @@ * and then arrange for the ftrace function to be called. */ .macro ftrace_regs_entry allregs + /* Create a minimal stack frame for representing B */ + PPC_STLU r1, -STACK_FRAME_MIN_SIZE(r1) + /* Save the original return address in A's stack frame */ + PPC_STL r0, LRSAVE+STACK_FRAME_MIN_SIZE(r1) + /* Create our stack frame + pt_regs */ PPC_STLU r1,-SWITCH_FRAME_SIZE(r1) @@ -41,8 +46,6 @@ SAVE_GPRS(3, 10, r1) #ifdef CONFIG_PPC64 - /* Save the original return address in A's stack frame */ - std r0, LRSAVE+SWITCH_FRAME_SIZE(r1) /* Ok to continue? */ lbz r3, PACA_FTRACE_ENABLED(r13) cmpdi r3, 0 @@ -77,6 +80,8 @@ mflr r7 /* Save it as pt_regs->nip */ PPC_STL r7, _NIP(r1) + /* Also save it in B's stackframe header for proper unwind */ + PPC_STL r7, LRSAVE+SWITCH_FRAME_SIZE(r1) /* Save the read LR in pt_regs->link */ PPC_STL r0, _LINK(r1) @@ -142,7 +147,7 @@ #endif /* Pop our stack frame */ - addi r1, r1, SWITCH_FRAME_SIZE + addi r1, r1, SWITCH_FRAME_SIZE+STACK_FRAME_MIN_SIZE #ifdef CONFIG_LIVEPATCH_64 /* Based on the cmpd above, if the NIP was altered handle livepatch */