From patchwork Mon Dec 25 20:54:39 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 852865 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3z5BfL2MB9z9ryk for ; Tue, 26 Dec 2017 08:12:58 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="FoybLdU3"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3z5BfL16T5zDrXN for ; Tue, 26 Dec 2017 08:12:58 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="FoybLdU3"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=linaro.org (client-ip=2a00:1450:400c:c09::242; helo=mail-wm0-x242.google.com; envelope-from=ard.biesheuvel@linaro.org; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="FoybLdU3"; dkim-atps=neutral Received: from mail-wm0-x242.google.com (mail-wm0-x242.google.com [IPv6:2a00:1450:400c:c09::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3z5BGf6t7mzDrWL for ; Tue, 26 Dec 2017 07:55:54 +1100 (AEDT) Received: by mail-wm0-x242.google.com with SMTP id g75so33032774wme.0 for ; Mon, 25 Dec 2017 12:55:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=nmovwwxZQQa2ugEDLnXxoU69CsqWUnhsDto5qVfQljI=; b=FoybLdU34h7tD2YvNo1TOxrVEvcKxDgXeKMEmlTyHo14rK82VWAM7Gt4bG28T0C9j2 PlA5sqnwmPwYuhY3eMCsxXR2D9oaM0zNKDSw5VLk2RKm/LC0aaMgOXCdXgiHjieBY5D9 6dHlDOePmBPWtmBMMmFiCQS3T86vTRkDOcgI4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=nmovwwxZQQa2ugEDLnXxoU69CsqWUnhsDto5qVfQljI=; b=sWGiW9hEKDVf1W2oi8y6QnaATaeZXsUtoHmAj7lX5EkS+JME3+hYklz9Dp65RibAFZ JfD0oGbPQ2a5M9NHG1Eifyrt35bmLlCEBakTY1IBLH9EIT0KVuSNYLzau6pIouXipwWt JUItLhIjf+FvqG/HfuXo5+s5d/Xuy8OPUzZuj7yeag4j56EwThYZPyXdZmPslmBzeg1y VMEeOaDcYJHgj3Py/tGL1jn3GzSlPrdputjB38ND43m6044AM8MTE0kDAr4HXpw/ajkQ TZZna6DiOB78mCUUsDkFLZ/CdgSW1wmrJSdp5ugtFZGpwVCApgIEYOchA5e0LcjxWTMh Mbiw== X-Gm-Message-State: AKGB3mL0qzILuigF0aRqgjtsLGZyXZ81FFKvTI06pZ40MpIFfRb1OR/X 8zovj/RJgOVdsUmQA0bO8bJXVw== X-Google-Smtp-Source: ACJfBouSgIxtYrj51f7q3xEziJb3fJlQCFfKo17xCJ8hKXbNdYyAV4MmDu6vDezlowjQgsHETpwrUQ== X-Received: by 10.28.130.208 with SMTP id e199mr19345255wmd.75.1514235352290; Mon, 25 Dec 2017 12:55:52 -0800 (PST) Received: from localhost.localdomain ([160.171.216.245]) by smtp.gmail.com with ESMTPSA id y42sm39552441wrc.96.2017.12.25.12.55.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 25 Dec 2017 12:55:51 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Subject: [PATCH v5 7/8] arm64/kernel: jump_label: use relative references Date: Mon, 25 Dec 2017 20:54:39 +0000 Message-Id: <20171225205440.14575-8-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20171225205440.14575-1-ard.biesheuvel@linaro.org> References: <20171225205440.14575-1-ard.biesheuvel@linaro.org> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.24 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-mips@linux-mips.org, Will Deacon , Paul Mackerras , "H. Peter Anvin" , sparclinux@vger.kernel.org, linux-s390@vger.kernel.org, Nicolas Pitre , x86@kernel.org, Russell King , Ingo Molnar , Catalin Marinas , "Serge E. Hallyn" , Petr Mladek , Kees Cook , Arnd Bergmann , Heiko Carstens , Steven Rostedt , James Morris , Josh Poimboeuf , Bjorn Helgaas , Thomas Gleixner , linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , linuxppc-dev@lists.ozlabs.org, Ralf Baechle , Thomas Garnier , Sergey Senozhatsky , Jessica Yu , Martin Schwidefsky , Andrew Morton , Linus Torvalds , "David S. Miller" Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" On a randomly chosen distro kernel build for arm64, vmlinux.o shows the following sections, containing jump label entries, and the associated RELA relocation records, respectively: ... [38088] __jump_table PROGBITS 0000000000000000 00e19f30 000000000002ea10 0000000000000000 WA 0 0 8 [38089] .rela__jump_table RELA 0000000000000000 01fd8bb0 000000000008be30 0000000000000018 I 38178 38088 8 ... In other words, we have 190 KB worth of 'struct jump_entry' instances, and 573 KB worth of RELA entries to relocate each entry's code, target and key members. This means the RELA section occupies 10% of the .init segment, and the two sections combined represent 5% of vmlinux's entire memory footprint. So let's switch from 64-bit absolute references to 32-bit relative references: this reduces the size of the __jump_table by 50%, and gets rid of the RELA section entirely. Note that this requires some extra care in the sorting routine, given that the offsets change when entries are moved around in the jump_entry table. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/jump_label.h | 27 ++++++++++++-------- arch/arm64/kernel/jump_label.c | 22 +++++++++++++--- 2 files changed, 36 insertions(+), 13 deletions(-) diff --git a/arch/arm64/include/asm/jump_label.h b/arch/arm64/include/asm/jump_label.h index 9d6e46355c89..5cec68616125 100644 --- a/arch/arm64/include/asm/jump_label.h +++ b/arch/arm64/include/asm/jump_label.h @@ -30,8 +30,8 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool bran { asm goto("1: nop\n\t" ".pushsection __jump_table, \"aw\"\n\t" - ".align 3\n\t" - ".quad 1b, %l[l_yes], %c0\n\t" + ".align 2\n\t" + ".long 1b - ., %l[l_yes] - ., %c0 - .\n\t" ".popsection\n\t" : : "i"(&((char *)key)[branch]) : : l_yes); @@ -44,8 +44,8 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool { asm goto("1: b %l[l_yes]\n\t" ".pushsection __jump_table, \"aw\"\n\t" - ".align 3\n\t" - ".quad 1b, %l[l_yes], %c0\n\t" + ".align 2\n\t" + ".long 1b - ., %l[l_yes] - ., %c0 - .\n\t" ".popsection\n\t" : : "i"(&((char *)key)[branch]) : : l_yes); @@ -57,19 +57,26 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool typedef u64 jump_label_t; struct jump_entry { - jump_label_t code; - jump_label_t target; - jump_label_t key; + s32 code; + s32 target; + s32 key; }; static inline jump_label_t jump_entry_code(const struct jump_entry *entry) { - return entry->code; + return (jump_label_t)&entry->code + entry->code; +} + +static inline jump_label_t jump_entry_target(const struct jump_entry *entry) +{ + return (jump_label_t)&entry->target + entry->target; } static inline struct static_key *jump_entry_key(const struct jump_entry *entry) { - return (struct static_key *)((unsigned long)entry->key & ~1UL); + unsigned long key = (unsigned long)&entry->key + entry->key; + + return (struct static_key *)(key & ~1UL); } static inline bool jump_entry_is_branch(const struct jump_entry *entry) @@ -87,7 +94,7 @@ static inline void jump_entry_set_module_init(struct jump_entry *entry) entry->code = 0; } -#define jump_label_swap NULL +void jump_label_swap(void *a, void *b, int size); #endif /* __ASSEMBLY__ */ #endif /* __ASM_JUMP_LABEL_H */ diff --git a/arch/arm64/kernel/jump_label.c b/arch/arm64/kernel/jump_label.c index c2dd1ad3e648..2b8e459e91f7 100644 --- a/arch/arm64/kernel/jump_label.c +++ b/arch/arm64/kernel/jump_label.c @@ -25,12 +25,12 @@ void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type) { - void *addr = (void *)entry->code; + void *addr = (void *)jump_entry_code(entry); u32 insn; if (type == JUMP_LABEL_JMP) { - insn = aarch64_insn_gen_branch_imm(entry->code, - entry->target, + insn = aarch64_insn_gen_branch_imm(jump_entry_code(entry), + jump_entry_target(entry), AARCH64_INSN_BRANCH_NOLINK); } else { insn = aarch64_insn_gen_nop(); @@ -50,4 +50,20 @@ void arch_jump_label_transform_static(struct jump_entry *entry, */ } +void jump_label_swap(void *a, void *b, int size) +{ + long delta = (unsigned long)a - (unsigned long)b; + struct jump_entry *jea = a; + struct jump_entry *jeb = b; + struct jump_entry tmp = *jea; + + jea->code = jeb->code - delta; + jea->target = jeb->target - delta; + jea->key = jeb->key - delta; + + jeb->code = tmp.code + delta; + jeb->target = tmp.target + delta; + jeb->key = tmp.key + delta; +} + #endif /* HAVE_JUMP_LABEL */