From patchwork Thu Apr 6 15:12:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Cabaj X-Patchwork-Id: 1766195 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=OsXTiZ4Y; dkim-atps=neutral Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4PslP4500Rz1yZT for ; Fri, 7 Apr 2023 01:12:55 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1pkRI4-0008Bp-OR; Thu, 06 Apr 2023 15:12:44 +0000 Received: from smtp-relay-internal-0.internal ([10.131.114.225] helo=smtp-relay-internal-0.canonical.com) by huckleberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1pkRHz-00089v-Ac for kernel-team@lists.ubuntu.com; Thu, 06 Apr 2023 15:12:39 +0000 Received: from mail-yb1-f198.google.com (mail-yb1-f198.google.com [209.85.219.198]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-0.canonical.com (Postfix) with ESMTPS id 138B33F19A for ; Thu, 6 Apr 2023 15:12:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1680793959; bh=4gGk8er1P7vvDAs69hkD33F+mCgvm/K6H0P6BzQSZ98=; h=From:To:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=OsXTiZ4Y3kVB0CqyDwYI4Upby3vcZ7ZYU+HA2sLZMhq9bPcQTwYyvR5p7VfnCoK9K HQGvAig+VE/2wpWRudRUVhRHG3QubbDb07m/utFwEK8Y+lTlZFh3AfXiEoOxCJSPSW lBE0GjrmwXHNp7k4NKF/QuWdc4OCyJL2Ybo8ee9gxjwitKxF1VetcPzEc0w+2m/NQa rjaKA4+/yTpLaevl/ukL5Cd8gcZXJTvlbeqd7cZWAjEnmkdyyNHO+fxPOmWOAizLo0 OKba1Z1y9oQIcZna/Uqb7sneTRjxXRJ/ciAJD8HUjQtUCkvlKBOXffXHFlwGCCSuhp jhI3lhsdCrkSg== Received: by mail-yb1-f198.google.com with SMTP id c193-20020a25c0ca000000b00b868826cdfeso14330510ybf.0 for ; Thu, 06 Apr 2023 08:12:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680793958; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4gGk8er1P7vvDAs69hkD33F+mCgvm/K6H0P6BzQSZ98=; b=aiqV6k82N8NKCHRwaXhJkEInxpUM2/WAhuaoHDnDpLuPpTJw1AT/7Vex+KWl8TOwHE b6o+0REzN8FImEh7N18PamuQkoyVelkCDYtdxQjw3hWZLH7dm3OqzJxVRvx2eVdHFH69 0s2t6jp00VmIGXih3gWw7mTJSK4/cJ1M6mwH553LoDzzEeUR/UjWqiKXwkTgEqzufg/E EANX9GeLl3H94yyt02WQt4fuilD/CEGCX5DX/BLLetEa7r9a87ucNCYNbSH8g9GNtHFg jMwwz+qzpuopBCOYuFJykJP49MQcb0k3S8ErjyJCzW13FFWAWbVWXMTFitQBnzxVVuWz oPWQ== X-Gm-Message-State: AAQBX9esK6Yszulm4gYMdJoaovhZWHrr1IlWyunmSG/ZxeD8mD9bwVz+ S0p7ve/mQyFLC8ztHkIgyeWrR16w/39v43NOHj4HzLfA5hYWIkk6kHh5skKNGoiZNxQaQJ39ID3 +CLlxT7cLEzZeLLylkS1dlcR62YUt5sSsmAF1NaAvGYpsPpDJRg== X-Received: by 2002:a0d:d9d7:0:b0:541:bc8f:9c5f with SMTP id b206-20020a0dd9d7000000b00541bc8f9c5fmr9418845ywe.38.1680793957750; Thu, 06 Apr 2023 08:12:37 -0700 (PDT) X-Google-Smtp-Source: AKy350aFOCVvaxv3pEgt+8QSEZa8cXl37RntVN7GyoUjZoAk+f9dCa9a0KgXUCqjCWJQ5LCYnrPCwQ== X-Received: by 2002:a0d:d9d7:0:b0:541:bc8f:9c5f with SMTP id b206-20020a0dd9d7000000b00541bc8f9c5fmr9418809ywe.38.1680793957270; Thu, 06 Apr 2023 08:12:37 -0700 (PDT) Received: from smtp.gmail.com (h69-130-246-116.mdtnwi.broadband.dynamic.tds.net. [69.130.246.116]) by smtp.gmail.com with ESMTPSA id 80-20020a811453000000b00545a0818480sm373564ywu.16.2023.04.06.08.12.36 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Apr 2023 08:12:36 -0700 (PDT) From: John Cabaj To: kernel-team@lists.ubuntu.com Subject: [SRU][jammy][PATCH v3 3/5] kprobes: Add kretprobe_find_ret_addr() for searching return address Date: Thu, 6 Apr 2023 10:12:31 -0500 Message-Id: <20230406151233.405626-4-john.cabaj@canonical.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230406151233.405626-1-john.cabaj@canonical.com> References: <20230406151233.405626-1-john.cabaj@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: Masami Hiramatsu BugLink: https://bugs.launchpad.net/bugs/2013603 (Kernel livepatch ftrace graph fix) Introduce kretprobe_find_ret_addr() and is_kretprobe_trampoline(). These APIs will be used by the ORC stack unwinder and ftrace, so that they can check whether the given address points kretprobe trampoline code and query the correct return address in that case. Link: https://lkml.kernel.org/r/163163046461.489837.1044778356430293962.stgit@devnote2 Signed-off-by: Masami Hiramatsu Tested-by: Andrii Nakryiko Signed-off-by: Steven Rostedt (VMware) (cherry picked from commit 03bac0df2886882c43e6d0bfff9dee84a184fc7e) Signed-off-by: John Cabaj --- include/linux/kprobes.h | 22 ++++++++ kernel/kprobes.c | 109 ++++++++++++++++++++++++++++++---------- 2 files changed, 105 insertions(+), 26 deletions(-) diff --git a/include/linux/kprobes.h b/include/linux/kprobes.h index f6b0aef35f66..9e24d8b79100 100644 --- a/include/linux/kprobes.h +++ b/include/linux/kprobes.h @@ -511,6 +511,28 @@ static inline bool is_kprobe_optinsn_slot(unsigned long addr) } #endif +#ifdef CONFIG_KRETPROBES +static nokprobe_inline bool is_kretprobe_trampoline(unsigned long addr) +{ + return (void *)addr == kretprobe_trampoline_addr(); +} + +unsigned long kretprobe_find_ret_addr(struct task_struct *tsk, void *fp, + struct llist_node **cur); +#else +static nokprobe_inline bool is_kretprobe_trampoline(unsigned long addr) +{ + return false; +} + +static nokprobe_inline +unsigned long kretprobe_find_ret_addr(struct task_struct *tsk, void *fp, + struct llist_node **cur) +{ + return 0; +} +#endif + /* Returns true if kprobes handled the fault */ static nokprobe_inline bool kprobe_page_fault(struct pt_regs *regs, unsigned int trap) diff --git a/kernel/kprobes.c b/kernel/kprobes.c index d781b4b11ea8..bde06beb5111 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -1864,45 +1864,87 @@ unsigned long __weak arch_deref_entry_point(void *entry) #ifdef CONFIG_KRETPROBES -unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, - void *frame_pointer) +/* This assumes the 'tsk' is the current task or the is not running. */ +static kprobe_opcode_t *__kretprobe_find_ret_addr(struct task_struct *tsk, + struct llist_node **cur) { - kprobe_opcode_t *correct_ret_addr = NULL; struct kretprobe_instance *ri = NULL; - struct llist_node *first, *node; - struct kretprobe *rp; + struct llist_node *node = *cur; + + if (!node) + node = tsk->kretprobe_instances.first; + else + node = node->next; - /* Find all nodes for this frame. */ - first = node = current->kretprobe_instances.first; while (node) { ri = container_of(node, struct kretprobe_instance, llist); - - BUG_ON(ri->fp != frame_pointer); - if (ri->ret_addr != kretprobe_trampoline_addr()) { - correct_ret_addr = ri->ret_addr; - /* - * This is the real return address. Any other - * instances associated with this task are for - * other calls deeper on the call stack - */ - goto found; + *cur = node; + return ri->ret_addr; } - node = node->next; } - pr_err("kretprobe: Return address not found, not execute handler. Maybe there is a bug in the kernel.\n"); - BUG_ON(1); + return NULL; +} +NOKPROBE_SYMBOL(__kretprobe_find_ret_addr); -found: - /* Unlink all nodes for this frame. */ - current->kretprobe_instances.first = node->next; - node->next = NULL; +/** + * kretprobe_find_ret_addr -- Find correct return address modified by kretprobe + * @tsk: Target task + * @fp: A frame pointer + * @cur: a storage of the loop cursor llist_node pointer for next call + * + * Find the correct return address modified by a kretprobe on @tsk in unsigned + * long type. If it finds the return address, this returns that address value, + * or this returns 0. + * The @tsk must be 'current' or a task which is not running. @fp is a hint + * to get the currect return address - which is compared with the + * kretprobe_instance::fp field. The @cur is a loop cursor for searching the + * kretprobe return addresses on the @tsk. The '*@cur' should be NULL at the + * first call, but '@cur' itself must NOT NULL. + */ +unsigned long kretprobe_find_ret_addr(struct task_struct *tsk, void *fp, + struct llist_node **cur) +{ + struct kretprobe_instance *ri = NULL; + kprobe_opcode_t *ret; + + if (WARN_ON_ONCE(!cur)) + return 0; + + do { + ret = __kretprobe_find_ret_addr(tsk, cur); + if (!ret) + break; + ri = container_of(*cur, struct kretprobe_instance, llist); + } while (ri->fp != fp); - /* Run them.. */ + return (unsigned long)ret; +} +NOKPROBE_SYMBOL(kretprobe_find_ret_addr); + +unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, + void *frame_pointer) +{ + kprobe_opcode_t *correct_ret_addr = NULL; + struct kretprobe_instance *ri = NULL; + struct llist_node *first, *node = NULL; + struct kretprobe *rp; + + /* Find correct address and all nodes for this frame. */ + correct_ret_addr = __kretprobe_find_ret_addr(current, &node); + if (!correct_ret_addr) { + pr_err("kretprobe: Return address not found, not execute handler. Maybe there is a bug in the kernel.\n"); + BUG_ON(1); + } + + /* Run the user handler of the nodes. */ + first = current->kretprobe_instances.first; while (first) { ri = container_of(first, struct kretprobe_instance, llist); - first = first->next; + + if (WARN_ON_ONCE(ri->fp != frame_pointer)) + break; rp = get_kretprobe(ri); if (rp && rp->handler) { @@ -1913,6 +1955,21 @@ unsigned long __kretprobe_trampoline_handler(struct pt_regs *regs, rp->handler(ri, regs); __this_cpu_write(current_kprobe, prev); } + if (first == node) + break; + + first = first->next; + } + + /* Unlink all nodes for this frame. */ + first = current->kretprobe_instances.first; + current->kretprobe_instances.first = node->next; + node->next = NULL; + + /* Recycle free instances. */ + while (first) { + ri = container_of(first, struct kretprobe_instance, llist); + first = first->next; recycle_rp_inst(ri); }