From patchwork Tue Jan 2 20:05:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 854759 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zB5NR2ldFz9s75 for ; Wed, 3 Jan 2018 07:32:55 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="U4JVdCRA"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zB5NQ6wxdzDqJM for ; Wed, 3 Jan 2018 07:32:54 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="U4JVdCRA"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=linaro.org (client-ip=2a00:1450:400c:c09::241; helo=mail-wm0-x241.google.com; envelope-from=ard.biesheuvel@linaro.org; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="U4JVdCRA"; dkim-atps=neutral Received: from mail-wm0-x241.google.com (mail-wm0-x241.google.com [IPv6:2a00:1450:400c:c09::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zB4pt5h3DzF0Ph for ; Wed, 3 Jan 2018 07:07:18 +1100 (AEDT) Received: by mail-wm0-x241.google.com with SMTP id g75so63340670wme.0 for ; Tue, 02 Jan 2018 12:07:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=y3cZHthDjrZdZe+7+thZcDtJZzDXcVyA3ettxQyn2/I=; b=U4JVdCRAWxIu8asYC7VR7yKgASQ4tUAP3aRQgb65Z/v9h8Of/kX68XF7IQrKEQqRbL mILpXc3g0V1XmE0jvCO2bKvYotrOisM9fdYrmv1LQvUXHa3V9ocBPN5rZ/3uvp1owdcE JyUIUE3jrgb01ODtZJbvYgN0b++pSR4Jo52yo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=y3cZHthDjrZdZe+7+thZcDtJZzDXcVyA3ettxQyn2/I=; b=PlPjU12Kc+jemXbpPzEEo7tdfEkmWxpoSpdIO9aiO67IKd33jOOIr/5TYhcgl+4cK+ 1LAYqXQsiZJIGA/5qKCDhwXrW5dFwkrnSwyQteWuUa+I+6Q0nM4MfMkayLkaWAOptg3X dvCLmAQpO8IYbOw3PyiecHP7m8dWYztkV5PGNqI6HABhkU0U/zEi9KLEDwu9bkBQaKGn REpWEe2VOxe04eoAi18oAOGbbysr7Nj/kfJo0X2YHMFp25vh6AWJ2ly8PFb7YZCOJ3pU EnlcaJnPWNIBEia5QfO4/hkXd6bpnd6/JoeFcd84I05spBBcBdvNKRAGw5K1Dkuc5whk P5mA== X-Gm-Message-State: AKGB3mLxbCfwORPZbH61BDqwHe5yz4D0/2z2qlO1KNwec+/qwAa/STTs Yl1hy5Ia5kLRyRHjep+56Bx5KA== X-Google-Smtp-Source: ACJfBoviLzGvtRaGzz7XrqH0Ivxu6gFKE14Z5TUX+bR1ffelvedzDX5yPzkCFu05qY1xL5FDoDZ2MA== X-Received: by 10.28.74.147 with SMTP id n19mr40211357wmi.134.1514923636103; Tue, 02 Jan 2018 12:07:16 -0800 (PST) Received: from localhost.localdomain ([160.89.138.198]) by smtp.gmail.com with ESMTPSA id m70sm19128526wma.36.2018.01.02.12.07.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Jan 2018 12:07:15 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Subject: [PATCH v7 10/10] x86/kernel: jump_table: use relative references Date: Tue, 2 Jan 2018 20:05:49 +0000 Message-Id: <20180102200549.22984-11-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180102200549.22984-1-ard.biesheuvel@linaro.org> References: <20180102200549.22984-1-ard.biesheuvel@linaro.org> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.24 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-mips@linux-mips.org, Will Deacon , Paul Mackerras , "H. Peter Anvin" , sparclinux@vger.kernel.org, linux-s390@vger.kernel.org, Nicolas Pitre , x86@kernel.org, Russell King , Ingo Molnar , Catalin Marinas , "Serge E. Hallyn" , Petr Mladek , Kees Cook , Arnd Bergmann , Heiko Carstens , Steven Rostedt , James Morris , Josh Poimboeuf , Bjorn Helgaas , Thomas Gleixner , linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , linuxppc-dev@lists.ozlabs.org, Ralf Baechle , Thomas Garnier , Sergey Senozhatsky , Jessica Yu , Martin Schwidefsky , Andrew Morton , Linus Torvalds , "David S. Miller" Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Similar to the arm64 case, 64-bit x86 can benefit from using 32-bit relative references rather than 64-bit absolute ones when emitting struct jump_entry instances. Not only does this reduce the memory footprint of the entries themselves by 50%, it also removes the need for carrying relocation metadata on relocatable builds (i.e., for KASLR) which saves a fair chunk of .init space as well (although the savings are not as dramatic as on arm64) Signed-off-by: Ard Biesheuvel --- arch/x86/include/asm/jump_label.h | 35 ++++++++++++-------- arch/x86/kernel/jump_label.c | 16 +++++++++ tools/objtool/special.c | 4 +-- 3 files changed, 39 insertions(+), 16 deletions(-) diff --git a/arch/x86/include/asm/jump_label.h b/arch/x86/include/asm/jump_label.h index 009ff2699d07..35fc2c5ec846 100644 --- a/arch/x86/include/asm/jump_label.h +++ b/arch/x86/include/asm/jump_label.h @@ -36,8 +36,8 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool bran asm_volatile_goto("1:" ".byte " __stringify(STATIC_KEY_INIT_NOP) "\n\t" ".pushsection __jump_table, \"aw\" \n\t" - _ASM_ALIGN "\n\t" - _ASM_PTR "1b, %l[l_yes], %c0 + %c1 \n\t" + ".balign 4\n\t" + ".long 1b - ., %l[l_yes] - ., %c0 + %c1 - .\n\t" ".popsection \n\t" : : "i" (key), "i" (branch) : : l_yes); @@ -52,8 +52,8 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool ".byte 0xe9\n\t .long %l[l_yes] - 2f\n\t" "2:\n\t" ".pushsection __jump_table, \"aw\" \n\t" - _ASM_ALIGN "\n\t" - _ASM_PTR "1b, %l[l_yes], %c0 + %c1 \n\t" + ".balign 4\n\t" + ".long 1b - ., %l[l_yes] - ., %c0 + %c1 - .\n\t" ".popsection \n\t" : : "i" (key), "i" (branch) : : l_yes); @@ -69,19 +69,26 @@ typedef u32 jump_label_t; #endif struct jump_entry { - jump_label_t code; - jump_label_t target; - jump_label_t key; + s32 code; + s32 target; + s32 key; }; static inline jump_label_t jump_entry_code(const struct jump_entry *entry) { - return entry->code; + return (unsigned long)&entry->code + entry->code; +} + +static inline jump_label_t jump_entry_target(const struct jump_entry *entry) +{ + return (unsigned long)&entry->target + entry->target; } static inline struct static_key *jump_entry_key(const struct jump_entry *entry) { - return (struct static_key *)((unsigned long)entry->key & ~1UL); + unsigned long key = (unsigned long)&entry->key + entry->key; + + return (struct static_key *)(key & ~1UL); } static inline bool jump_entry_is_branch(const struct jump_entry *entry) @@ -99,7 +106,7 @@ static inline void jump_entry_set_module_init(struct jump_entry *entry) entry->code = 0; } -#define jump_label_swap NULL +void jump_label_swap(void *a, void *b, int size); #else /* __ASSEMBLY__ */ @@ -114,8 +121,8 @@ static inline void jump_entry_set_module_init(struct jump_entry *entry) .byte STATIC_KEY_INIT_NOP .endif .pushsection __jump_table, "aw" - _ASM_ALIGN - _ASM_PTR .Lstatic_jump_\@, \target, \key + .balign 4 + .long .Lstatic_jump_\@ - ., \target - ., \key - . .popsection .endm @@ -130,8 +137,8 @@ static inline void jump_entry_set_module_init(struct jump_entry *entry) .Lstatic_jump_after_\@: .endif .pushsection __jump_table, "aw" - _ASM_ALIGN - _ASM_PTR .Lstatic_jump_\@, \target, \key + 1 + .balign 4 + .long .Lstatic_jump_\@ - ., \target - ., \key + 1 - . .popsection .endm diff --git a/arch/x86/kernel/jump_label.c b/arch/x86/kernel/jump_label.c index d64296092ef5..cc5034b42335 100644 --- a/arch/x86/kernel/jump_label.c +++ b/arch/x86/kernel/jump_label.c @@ -149,4 +149,20 @@ __init_or_module void arch_jump_label_transform_static(struct jump_entry *entry, __jump_label_transform(entry, type, text_poke_early, 1); } +void jump_label_swap(void *a, void *b, int size) +{ + long delta = (unsigned long)a - (unsigned long)b; + struct jump_entry *jea = a; + struct jump_entry *jeb = b; + struct jump_entry tmp = *jea; + + jea->code = jeb->code - delta; + jea->target = jeb->target - delta; + jea->key = jeb->key - delta; + + jeb->code = tmp.code + delta; + jeb->target = tmp.target + delta; + jeb->key = tmp.key + delta; +} + #endif diff --git a/tools/objtool/special.c b/tools/objtool/special.c index 84f001d52322..98ae55b39037 100644 --- a/tools/objtool/special.c +++ b/tools/objtool/special.c @@ -30,9 +30,9 @@ #define EX_ORIG_OFFSET 0 #define EX_NEW_OFFSET 4 -#define JUMP_ENTRY_SIZE 24 +#define JUMP_ENTRY_SIZE 12 #define JUMP_ORIG_OFFSET 0 -#define JUMP_NEW_OFFSET 8 +#define JUMP_NEW_OFFSET 4 #define ALT_ENTRY_SIZE 13 #define ALT_ORIG_OFFSET 0