From patchwork Tue Jan 2 20:05:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 854743 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zB4vZ6Nxvz9s7n for ; Wed, 3 Jan 2018 07:11:22 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="DYml98nl"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zB4vZ49j3zF0PR for ; Wed, 3 Jan 2018 07:11:22 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="DYml98nl"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=linaro.org (client-ip=2a00:1450:400c:c0c::244; helo=mail-wr0-x244.google.com; envelope-from=ard.biesheuvel@linaro.org; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="DYml98nl"; dkim-atps=neutral Received: from mail-wr0-x244.google.com (mail-wr0-x244.google.com [IPv6:2a00:1450:400c:c0c::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zB4nm4457zF0PK for ; Wed, 3 Jan 2018 07:06:20 +1100 (AEDT) Received: by mail-wr0-x244.google.com with SMTP id 36so16288767wrh.1 for ; Tue, 02 Jan 2018 12:06:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=8wbbwFOWuQMFP0SKy8UHj3NriHdOJ7kBnwcJ7OnyKjo=; b=DYml98nl5C9dxpX8PaphCwO1AwOOVT37wBSgEpvuWvz1+8lzAEk4lAznRm97Y/6LxX Y4BE3AmmwB7ErSHppbzjssrLWql23m3K278dd7UPXIhPBJLssy7YFhsS3HVTTffQ0BvS xMwTDlpUaHy6gjzAk7kS/8JHmrEGXAcyxkyKE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=8wbbwFOWuQMFP0SKy8UHj3NriHdOJ7kBnwcJ7OnyKjo=; b=oa8cD8VxuWs8aN9R0IXRv2jGNailsT8tgsTaJq1iUCYN+2KjovE/Y2UViJPfiDuoSN 0CKY7od8OSjGoy8SBk2q51p2C/a9OfV1k13waZqGz7DgORVGN7EBRj3K/MqcNL8+7wr8 JwKRNLSEZYFySMLQpgQQV/9WD4c8U5FkE6VMVfsR7p85ptuloh4RttyJR5lm0mPkbfEW /meSzFgVeG/jX8Ptw5dMT0yIPMWKfH7xlGukrijj6eqp1CIv954qN+KKbvsZShBHA6pQ Tv2fmuReNROtU5+VwnqTWbyj8TkpYmGKXRA3fHirJ6JK/vIBq82MryYdd0ADDJHpm6TJ Absg== X-Gm-Message-State: AKGB3mKIzXPrJN9aVuobLeHJFjdBtz6RkARz5/Vw+rwKvft6O6rJFtm4 w6n5E/hZ0z1tgixAeEQ0HydUtg== X-Google-Smtp-Source: ACJfBovJAzTpyTVLr4ekaaR7FsRgcOGRNFzsjbQL6nJa68x+hu03fqzi0XJ5w66c6hCO+U0seRwJLg== X-Received: by 10.223.151.146 with SMTP id s18mr20037055wrb.180.1514923577962; Tue, 02 Jan 2018 12:06:17 -0800 (PST) Received: from localhost.localdomain ([160.89.138.198]) by smtp.gmail.com with ESMTPSA id m70sm19128526wma.36.2018.01.02.12.06.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Jan 2018 12:06:17 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Subject: [PATCH v7 01/10] arch: enable relative relocations for arm64, power and x86 Date: Tue, 2 Jan 2018 20:05:40 +0000 Message-Id: <20180102200549.22984-2-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180102200549.22984-1-ard.biesheuvel@linaro.org> References: <20180102200549.22984-1-ard.biesheuvel@linaro.org> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.24 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-mips@linux-mips.org, Will Deacon , Paul Mackerras , "H. Peter Anvin" , sparclinux@vger.kernel.org, linux-s390@vger.kernel.org, Nicolas Pitre , x86@kernel.org, Russell King , Ingo Molnar , Catalin Marinas , "Serge E. Hallyn" , Petr Mladek , Kees Cook , Arnd Bergmann , Heiko Carstens , Steven Rostedt , James Morris , Josh Poimboeuf , Bjorn Helgaas , Thomas Gleixner , linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , linuxppc-dev@lists.ozlabs.org, Ralf Baechle , Thomas Garnier , Sergey Senozhatsky , Jessica Yu , Martin Schwidefsky , Andrew Morton , Linus Torvalds , "David S. Miller" Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Before updating certain subsystems to use place relative 32-bit relocations in special sections, to save space and reduce the number of absolute relocations that need to be processed at runtime by relocatable kernels, introduce the Kconfig symbol and define it for some architectures that should be able to support and benefit from it. Cc: Catalin Marinas Cc: Will Deacon Cc: Benjamin Herrenschmidt Cc: Paul Mackerras Cc: Michael Ellerman Cc: Thomas Gleixner Cc: Ingo Molnar Cc: "H. Peter Anvin" Cc: x86@kernel.org Signed-off-by: Ard Biesheuvel --- arch/Kconfig | 10 ++++++++++ arch/arm64/Kconfig | 1 + arch/powerpc/Kconfig | 1 + arch/x86/Kconfig | 1 + 4 files changed, 13 insertions(+) diff --git a/arch/Kconfig b/arch/Kconfig index 400b9e1b2f27..dbc036a7bd1b 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -959,4 +959,14 @@ config REFCOUNT_FULL against various use-after-free conditions that can be used in security flaw exploits. +config HAVE_ARCH_PREL32_RELOCATIONS + bool + help + May be selected by an architecture if it supports place-relative + 32-bit relocations, both in the toolchain and in the module loader, + in which case relative references can be used in special sections + for PCI fixup, initcalls etc which are only half the size on 64 bit + architectures, and don't require runtime relocation on relocatable + kernels. + source "kernel/gcov/Kconfig" diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index c9a7e9e1414f..66c7b9ab2a3d 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -89,6 +89,7 @@ config ARM64 select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS select HAVE_ARCH_MMAP_RND_COMPAT_BITS if COMPAT + select HAVE_ARCH_PREL32_RELOCATIONS select HAVE_ARCH_SECCOMP_FILTER select HAVE_ARCH_TRACEHOOK select HAVE_ARCH_TRANSPARENT_HUGEPAGE diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index c51e6ce42e7a..e172478e2ae7 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -177,6 +177,7 @@ config PPC select HAVE_ARCH_KGDB select HAVE_ARCH_MMAP_RND_BITS select HAVE_ARCH_MMAP_RND_COMPAT_BITS if COMPAT + select HAVE_ARCH_PREL32_RELOCATIONS select HAVE_ARCH_SECCOMP_FILTER select HAVE_ARCH_TRACEHOOK select ARCH_HAS_STRICT_KERNEL_RWX if ((PPC_BOOK3S_64 || PPC32) && !RELOCATABLE && !HIBERNATION) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index d4fc98c50378..9f2bb853aedb 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -115,6 +115,7 @@ config X86 select HAVE_ARCH_MMAP_RND_BITS if MMU select HAVE_ARCH_MMAP_RND_COMPAT_BITS if MMU && COMPAT select HAVE_ARCH_COMPAT_MMAP_BASES if MMU && COMPAT + select HAVE_ARCH_PREL32_RELOCATIONS select HAVE_ARCH_SECCOMP_FILTER select HAVE_ARCH_TRACEHOOK select HAVE_ARCH_TRANSPARENT_HUGEPAGE From patchwork Tue Jan 2 20:05:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 854744 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zB4yT2F0Pz9s7n for ; Wed, 3 Jan 2018 07:13:53 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="JvakjjXX"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zB4yS73B0zF0PR for ; Wed, 3 Jan 2018 07:13:52 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="JvakjjXX"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=linaro.org (client-ip=2a00:1450:400c:c09::241; helo=mail-wm0-x241.google.com; envelope-from=ard.biesheuvel@linaro.org; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="JvakjjXX"; dkim-atps=neutral Received: from mail-wm0-x241.google.com (mail-wm0-x241.google.com [IPv6:2a00:1450:400c:c09::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zB4nv41VnzF0Pg for ; Wed, 3 Jan 2018 07:06:27 +1100 (AEDT) Received: by mail-wm0-x241.google.com with SMTP id n138so62938594wmg.2 for ; Tue, 02 Jan 2018 12:06:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=9pf2nmv48uHqsDbsl+hfOY5WaUyvrm1nnNGc9/9m6Qg=; b=JvakjjXXh2ftac/uQdStpmRnyog8dVy1B1dV3/FBRYhclGcwN1Nf1ULl77WmER9ry4 qNvpqUIkOxaXMPtj6B3oJYPN87PgdaiNOi0rNpCB5XAS3BaMGwa31IA7QPoscn3u7sb0 uOQNLx6iterfuYqWKriCvDY/4G6QygREzc7s0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=9pf2nmv48uHqsDbsl+hfOY5WaUyvrm1nnNGc9/9m6Qg=; b=hmcyV464iCeOjb+MGydMGWJd+IVLWbbTOuH8UXGSaTIFkkTueHN8BSzCWB9hq7m62O eTWNK49WINYPiGX313sDUmbLoD8/QvvnepZ3NqlBVH4jkGT7Euyw+7IszFy18T6T8Tnt 1gXIoI/pJmomCHPDGKjaAWWBHU7swwJ+tlExEqq21XJr5oMLZUqb8az68kYAJNlua9Ax mkN/kjKFJZHU7z2WvMQAL1/wG0qudTiB+Zu65LodWBoJybldO1LoVTAXHLZWvTvU2lnS Gjor6RDgoYVRVueSiwMKCy/GGADCJmeTnWUdbOoPoGAVLmyNkmlJdqARnnm/HP5moxaL S3dQ== X-Gm-Message-State: AKGB3mJw0QGCNdr1pTK3ElK8MLPVYkXpxX7vjDhkD4xEX66nK9sLjHYh q5bthVjhoNJ5O4ryh7pp8uQT0g== X-Google-Smtp-Source: ACJfBosJ1YaW1H55WuPG4HZVNPR5sIYnqVBwvoCMpxmVYCKJYkCCK0WTpzNAV4SV0UZHkdO9ReayGQ== X-Received: by 10.28.97.194 with SMTP id v185mr11849434wmb.21.1514923584444; Tue, 02 Jan 2018 12:06:24 -0800 (PST) Received: from localhost.localdomain ([160.89.138.198]) by smtp.gmail.com with ESMTPSA id m70sm19128526wma.36.2018.01.02.12.06.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Jan 2018 12:06:23 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Subject: [PATCH v7 02/10] module: allow symbol exports to be disabled Date: Tue, 2 Jan 2018 20:05:41 +0000 Message-Id: <20180102200549.22984-3-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180102200549.22984-1-ard.biesheuvel@linaro.org> References: <20180102200549.22984-1-ard.biesheuvel@linaro.org> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.24 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-mips@linux-mips.org, matt@codeblueprint.co.uk, Will Deacon , Paul Mackerras , "H. Peter Anvin" , sparclinux@vger.kernel.org, linux-s390@vger.kernel.org, Nicolas Pitre , x86@kernel.org, Russell King , Ingo Molnar , Catalin Marinas , "Serge E. Hallyn" , Petr Mladek , Kees Cook , Arnd Bergmann , Heiko Carstens , Steven Rostedt , James Morris , Josh Poimboeuf , Bjorn Helgaas , Thomas Gleixner , linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , linuxppc-dev@lists.ozlabs.org, Ralf Baechle , Thomas Garnier , Sergey Senozhatsky , Jessica Yu , Martin Schwidefsky , Andrew Morton , Linus Torvalds , "David S. Miller" Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" To allow existing C code to be incorporated into the decompressor or the UEFI stub, introduce a CPP macro that turns all EXPORT_SYMBOL_xxx declarations into nops, and #define it in places where such exports are undesirable. Note that this gets rid of a rather dodgy redefine of linux/export.h's header guard. Cc: matt@codeblueprint.co.uk Cc: keescook@chromium.org Cc: jeyu@kernel.org Signed-off-by: Ard Biesheuvel --- arch/x86/boot/compressed/kaslr.c | 5 +---- drivers/firmware/efi/libstub/Makefile | 3 ++- include/linux/export.h | 9 +++++++++ 3 files changed, 12 insertions(+), 5 deletions(-) diff --git a/arch/x86/boot/compressed/kaslr.c b/arch/x86/boot/compressed/kaslr.c index 8199a6187251..3a2a6d7049e4 100644 --- a/arch/x86/boot/compressed/kaslr.c +++ b/arch/x86/boot/compressed/kaslr.c @@ -23,11 +23,8 @@ * _ctype[] in lib/ctype.c is needed by isspace() of linux/ctype.h. * While both lib/ctype.c and lib/cmdline.c will bring EXPORT_SYMBOL * which is meaningless and will cause compiling error in some cases. - * So do not include linux/export.h and define EXPORT_SYMBOL(sym) - * as empty. */ -#define _LINUX_EXPORT_H -#define EXPORT_SYMBOL(sym) +#define __DISABLE_EXPORTS #include "misc.h" #include "error.h" diff --git a/drivers/firmware/efi/libstub/Makefile b/drivers/firmware/efi/libstub/Makefile index adaa4a964f0c..312bd0b64a61 100644 --- a/drivers/firmware/efi/libstub/Makefile +++ b/drivers/firmware/efi/libstub/Makefile @@ -20,7 +20,8 @@ cflags-$(CONFIG_EFI_ARMSTUB) += -I$(srctree)/scripts/dtc/libfdt KBUILD_CFLAGS := $(cflags-y) -DDISABLE_BRANCH_PROFILING \ -D__NO_FORTIFY \ $(call cc-option,-ffreestanding) \ - $(call cc-option,-fno-stack-protector) + $(call cc-option,-fno-stack-protector) \ + -D__DISABLE_EXPORTS GCOV_PROFILE := n KASAN_SANITIZE := n diff --git a/include/linux/export.h b/include/linux/export.h index 1a1dfdb2a5c6..6dba2fb08f77 100644 --- a/include/linux/export.h +++ b/include/linux/export.h @@ -83,6 +83,15 @@ extern struct module __this_module; */ #define __EXPORT_SYMBOL(sym, sec) === __KSYM_##sym === +#elif defined(__DISABLE_EXPORTS) + +/* + * Allow symbol exports to be disabled completely so that C code may + * be reused in other execution contexts such as the UEFI stub or the + * decompressor. + */ +#define __EXPORT_SYMBOL(sym, sec) + #elif defined(CONFIG_TRIM_UNUSED_KSYMS) #include From patchwork Tue Jan 2 20:05:42 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 854745 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zB50w1BDXz9s7n for ; Wed, 3 Jan 2018 07:16:00 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="I7jWcnRZ"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zB50v6tfCzDqhL for ; Wed, 3 Jan 2018 07:15:59 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="I7jWcnRZ"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=linaro.org (client-ip=2a00:1450:400c:c09::242; helo=mail-wm0-x242.google.com; envelope-from=ard.biesheuvel@linaro.org; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="I7jWcnRZ"; dkim-atps=neutral Received: from mail-wm0-x242.google.com (mail-wm0-x242.google.com [IPv6:2a00:1450:400c:c09::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zB4p119f5zF0PZ for ; Wed, 3 Jan 2018 07:06:33 +1100 (AEDT) Received: by mail-wm0-x242.google.com with SMTP id 9so62960726wme.4 for ; Tue, 02 Jan 2018 12:06:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=YeY2Xm5F7YqCntxBJlOwEmhyquMbh7oq6A2LuKFDBk0=; b=I7jWcnRZxSQJ+r+NtiSwxZap+e4+EqIntXlM8zv+adVYkjRBnxpAYSFrC0dqGsT1QF 8rDSC94FmIB9N8KAC8sCCuRCErl1CHAnd3c/Uj+mcD10ddgBJm92tB35pktDiPzqQesh +AnUz+oxl5F+2nDDr5snXku1SfLcXf+ApxaMw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=YeY2Xm5F7YqCntxBJlOwEmhyquMbh7oq6A2LuKFDBk0=; b=fs+X7GOByONWNgXS8wEK226kaDwMaFAb6xkXUvJLPaO+9c2mgG/4TNnXsWfsUY+//H oUC7QSlGtyuTEf7vAzNnPIaeQac91qqA9slgiFPelDB2rPY267x7QuS7xjU8SZXdyL9D 6xfnEgLuu+NJGzG01xdlUWOAb4Lsr0I2jX5RWFptZxI2HsQvwrgK7VGtgtyNV31SIweW roBjwq71p4D0OaZdosl80CbD9cBYm0xy0kpyLGg8merhttEXfQ9fDyZdUCxHXcR6mPDO ZQftmDGwIfJZuf3H58h024EKAC1fa1hYv9jrTPPzaPHpvmSxqk8dREdWhqF5Nkg9eE71 H74w== X-Gm-Message-State: AKGB3mLK54OGc9uABOUrHGorNN3azi88t3LbK5zDgpKP7MsRUTgQYFR0 Ub6quY5RpvpTK5BwA44udXFJDw== X-Google-Smtp-Source: ACJfBot7xUb0qYmgPwxw/RXnKwp6js00bW9Gxhi5G8z2oVkjLuiiU4e/YG3F/p9nd9P/k33kbNIRBg== X-Received: by 10.28.129.69 with SMTP id c66mr38913551wmd.29.1514923590192; Tue, 02 Jan 2018 12:06:30 -0800 (PST) Received: from localhost.localdomain ([160.89.138.198]) by smtp.gmail.com with ESMTPSA id m70sm19128526wma.36.2018.01.02.12.06.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Jan 2018 12:06:29 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Subject: [PATCH v7 03/10] module: use relative references for __ksymtab entries Date: Tue, 2 Jan 2018 20:05:42 +0000 Message-Id: <20180102200549.22984-4-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180102200549.22984-1-ard.biesheuvel@linaro.org> References: <20180102200549.22984-1-ard.biesheuvel@linaro.org> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.24 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-mips@linux-mips.org, Will Deacon , Paul Mackerras , "H. Peter Anvin" , sparclinux@vger.kernel.org, Ingo Molnar , linux-s390@vger.kernel.org, Nicolas Pitre , x86@kernel.org, Russell King , Ingo Molnar , Catalin Marinas , "Serge E. Hallyn" , Petr Mladek , Kees Cook , Arnd Bergmann , Heiko Carstens , Steven Rostedt , James Morris , Josh Poimboeuf , Bjorn Helgaas , Thomas Gleixner , linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , linuxppc-dev@lists.ozlabs.org, Ralf Baechle , Thomas Garnier , Sergey Senozhatsky , Jessica Yu , Martin Schwidefsky , Andrew Morton , Linus Torvalds , "David S. Miller" Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" An ordinary arm64 defconfig build has ~64 KB worth of __ksymtab entries, each consisting of two 64-bit fields containing absolute references, to the symbol itself and to a char array containing its name, respectively. When we build the same configuration with KASLR enabled, we end up with an additional ~192 KB of relocations in the .init section, i.e., one 24 byte entry for each absolute reference, which all need to be processed at boot time. Given how the struct kernel_symbol that describes each entry is completely local to module.c (except for the references emitted by EXPORT_SYMBOL() itself), we can easily modify it to contain two 32-bit relative references instead. This reduces the size of the __ksymtab section by 50% for all 64-bit architectures, and gets rid of the runtime relocations entirely for architectures implementing KASLR, either via standard PIE linking (arm64) or using custom host tools (x86). Note that the binary search involving __ksymtab contents relies on each section being sorted by symbol name. This is implemented based on the input section names, not the names in the ksymtab entries, so this patch does not interfere with that. Given that the use of place-relative relocations requires support both in the toolchain and in the module loader, we cannot enable this feature for all architectures. So make it dependent on whether CONFIG_HAVE_ARCH_PREL32_RELOCATIONS is defined. Cc: Arnd Bergmann Cc: Andrew Morton Cc: Ingo Molnar Cc: Kees Cook Cc: Thomas Garnier Cc: Nicolas Pitre Acked-by: Jessica Yu Signed-off-by: Ard Biesheuvel --- arch/x86/include/asm/Kbuild | 1 + arch/x86/include/asm/export.h | 5 --- include/asm-generic/export.h | 12 ++++- include/linux/compiler.h | 10 +++++ include/linux/export.h | 46 +++++++++++++++----- kernel/module.c | 33 +++++++++++--- 6 files changed, 83 insertions(+), 24 deletions(-) diff --git a/arch/x86/include/asm/Kbuild b/arch/x86/include/asm/Kbuild index 5d6a53fd7521..3e8a88dcaa1d 100644 --- a/arch/x86/include/asm/Kbuild +++ b/arch/x86/include/asm/Kbuild @@ -9,5 +9,6 @@ generated-y += xen-hypercalls.h generic-y += clkdev.h generic-y += dma-contiguous.h generic-y += early_ioremap.h +generic-y += export.h generic-y += mcs_spinlock.h generic-y += mm-arch-hooks.h diff --git a/arch/x86/include/asm/export.h b/arch/x86/include/asm/export.h deleted file mode 100644 index 2a51d66689c5..000000000000 --- a/arch/x86/include/asm/export.h +++ /dev/null @@ -1,5 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifdef CONFIG_64BIT -#define KSYM_ALIGN 16 -#endif -#include diff --git a/include/asm-generic/export.h b/include/asm-generic/export.h index 719db1968d81..97ce606459ae 100644 --- a/include/asm-generic/export.h +++ b/include/asm-generic/export.h @@ -5,12 +5,10 @@ #define KSYM_FUNC(x) x #endif #ifdef CONFIG_64BIT -#define __put .quad #ifndef KSYM_ALIGN #define KSYM_ALIGN 8 #endif #else -#define __put .long #ifndef KSYM_ALIGN #define KSYM_ALIGN 4 #endif @@ -25,6 +23,16 @@ #define KSYM(name) name #endif +.macro __put, val, name +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS + .long \val - ., \name - . +#elif defined(CONFIG_64BIT) + .quad \val, \name +#else + .long \val, \name +#endif +.endm + /* * note on .section use: @progbits vs %progbits nastiness doesn't matter, * since we immediately emit into those sections anyway. diff --git a/include/linux/compiler.h b/include/linux/compiler.h index 52e611ab9a6c..79db4aa87d75 100644 --- a/include/linux/compiler.h +++ b/include/linux/compiler.h @@ -327,4 +327,14 @@ static __always_inline void __write_once_size(volatile void *p, void *res, int s compiletime_assert(__native_word(t), \ "Need native word sized stores/loads for atomicity.") +/* + * Force the compiler to emit 'sym' as a symbol, so that we can reference + * it from inline assembler. Necessary in case 'sym' could be inlined + * otherwise, or eliminated entirely due to lack of references that are + * visible to the compiler. + */ +#define __ADDRESSABLE(sym) \ + static void * const __attribute__((section(".discard"), used)) \ + __PASTE(__addressable_##sym, __LINE__) = (void *)&sym; + #endif /* __LINUX_COMPILER_H */ diff --git a/include/linux/export.h b/include/linux/export.h index 6dba2fb08f77..4744cf4736b0 100644 --- a/include/linux/export.h +++ b/include/linux/export.h @@ -24,12 +24,6 @@ #define VMLINUX_SYMBOL_STR(x) __VMLINUX_SYMBOL_STR(x) #ifndef __ASSEMBLY__ -struct kernel_symbol -{ - unsigned long value; - const char *name; -}; - #ifdef MODULE extern struct module __this_module; #define THIS_MODULE (&__this_module) @@ -60,17 +54,47 @@ extern struct module __this_module; #define __CRC_SYMBOL(sym, sec) #endif +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS +#include +/* + * Emit the ksymtab entry as a pair of relative references: this reduces + * the size by half on 64-bit architectures, and eliminates the need for + * absolute relocations that require runtime processing on relocatable + * kernels. + */ +#define __KSYMTAB_ENTRY(sym, sec) \ + __ADDRESSABLE(sym) \ + asm(" .section \"___ksymtab" sec "+" #sym "\", \"a\" \n" \ + " .balign 8 \n" \ + VMLINUX_SYMBOL_STR(__ksymtab_##sym) ": \n" \ + " .long " VMLINUX_SYMBOL_STR(sym) "- . \n" \ + " .long " VMLINUX_SYMBOL_STR(__kstrtab_##sym) "- .\n" \ + " .previous \n") + +struct kernel_symbol { + int value_offset; + int name_offset; +}; +#else +#define __KSYMTAB_ENTRY(sym, sec) \ + static const struct kernel_symbol __ksymtab_##sym \ + __attribute__((section("___ksymtab" sec "+" #sym), used)) \ + = { (unsigned long)&sym, __kstrtab_##sym } + +struct kernel_symbol { + unsigned long value; + const char *name; +}; +#endif + /* For every exported symbol, place a struct in the __ksymtab section */ #define ___EXPORT_SYMBOL(sym, sec) \ extern typeof(sym) sym; \ __CRC_SYMBOL(sym, sec) \ static const char __kstrtab_##sym[] \ - __attribute__((section("__ksymtab_strings"), aligned(1))) \ + __attribute__((section("__ksymtab_strings"), used, aligned(1))) \ = VMLINUX_SYMBOL_STR(sym); \ - static const struct kernel_symbol __ksymtab_##sym \ - __used \ - __attribute__((section("___ksymtab" sec "+" #sym), used)) \ - = { (unsigned long)&sym, __kstrtab_##sym } + __KSYMTAB_ENTRY(sym, sec) #if defined(__KSYM_DEPS__) diff --git a/kernel/module.c b/kernel/module.c index dea01ac9cb74..d3a908ffc42c 100644 --- a/kernel/module.c +++ b/kernel/module.c @@ -549,12 +549,31 @@ static bool check_symbol(const struct symsearch *syms, return true; } +static unsigned long kernel_symbol_value(const struct kernel_symbol *sym) +{ +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS + return (unsigned long)&sym->value_offset + sym->value_offset; +#else + return sym->value; +#endif +} + +static const char *kernel_symbol_name(const struct kernel_symbol *sym) +{ +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS + return (const char *)((unsigned long)&sym->name_offset + + sym->name_offset); +#else + return sym->name; +#endif +} + static int cmp_name(const void *va, const void *vb) { const char *a; const struct kernel_symbol *b; a = va; b = vb; - return strcmp(a, b->name); + return strcmp(a, kernel_symbol_name(b)); } static bool find_symbol_in_section(const struct symsearch *syms, @@ -2198,7 +2217,7 @@ void *__symbol_get(const char *symbol) sym = NULL; preempt_enable(); - return sym ? (void *)sym->value : NULL; + return sym ? (void *)kernel_symbol_value(sym) : NULL; } EXPORT_SYMBOL_GPL(__symbol_get); @@ -2228,10 +2247,12 @@ static int verify_export_symbols(struct module *mod) for (i = 0; i < ARRAY_SIZE(arr); i++) { for (s = arr[i].sym; s < arr[i].sym + arr[i].num; s++) { - if (find_symbol(s->name, &owner, NULL, true, false)) { + if (find_symbol(kernel_symbol_name(s), &owner, NULL, + true, false)) { pr_err("%s: exports duplicate symbol %s" " (owned by %s)\n", - mod->name, s->name, module_name(owner)); + mod->name, kernel_symbol_name(s), + module_name(owner)); return -ENOEXEC; } } @@ -2280,7 +2301,7 @@ static int simplify_symbols(struct module *mod, const struct load_info *info) ksym = resolve_symbol_wait(mod, info, name); /* Ok if resolved. */ if (ksym && !IS_ERR(ksym)) { - sym[i].st_value = ksym->value; + sym[i].st_value = kernel_symbol_value(ksym); break; } @@ -2540,7 +2561,7 @@ static int is_exported(const char *name, unsigned long value, ks = lookup_symbol(name, __start___ksymtab, __stop___ksymtab); else ks = lookup_symbol(name, mod->syms, mod->syms + mod->num_syms); - return ks != NULL && ks->value == value; + return ks != NULL && kernel_symbol_value(ks) == value; } /* As per nm */ From patchwork Tue Jan 2 20:05:43 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 854750 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zB53m6f8pz9s82 for ; Wed, 3 Jan 2018 07:18:28 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="MZO1fAVB"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zB53m5NFPzDqxT for ; Wed, 3 Jan 2018 07:18:28 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="MZO1fAVB"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=linaro.org (client-ip=2a00:1450:400c:c0c::243; helo=mail-wr0-x243.google.com; envelope-from=ard.biesheuvel@linaro.org; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="MZO1fAVB"; dkim-atps=neutral Received: from mail-wr0-x243.google.com (mail-wr0-x243.google.com [IPv6:2a00:1450:400c:c0c::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zB4p62QTlzF0Pc for ; Wed, 3 Jan 2018 07:06:38 +1100 (AEDT) Received: by mail-wr0-x243.google.com with SMTP id l19so37218706wrc.2 for ; Tue, 02 Jan 2018 12:06:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=/EOip1AIYHq0lLmjVJhDl5soNaKDqq6XOAGGb6F1q1o=; b=MZO1fAVB5d8JRhoU/dxiG9mAm1vs47AzcIZm005LKWPWqltWMORB0h8IrpNjc6h1yM 3w8IlJB11um83+6PPjZLrCjdybt+umM2Irsf7x405cLMxNqpTDhTjxDr8h/PKA1Z+vnf 9C6rc9lcQI9Zpfy3xhy/B3RqgGlocDfqmj+8I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=/EOip1AIYHq0lLmjVJhDl5soNaKDqq6XOAGGb6F1q1o=; b=XERqemhcH7hECk4KU3TIxuYgDjlcM04tLxZzaYIUgAv5JAI4H4kgeG0NmmfESgDi1l J7xGQLyuae3THuHmRoHxwOuC0dCFLog89RvCQ8GBsusmGMr4BLDc7GgSW8YKEUORmo5y wdBCLiGbUv9+uB61pBFv6R0B9O/c441Z8j3K/8pSmuv3dPusD1SfRQCFWWq4b0F8P4Aj jgvuWFloQ1F5lJMZlKUldsZs4vxA2vQeDsrExvbfRkTLdDlscX+KT04YfNxtw3Af/786 U3IqcipIU4p5LvC0JC+qq68kitc66z0L8gXp7C+GceHA6n2h8UwIjf3gVXrS1TWJlnyC 2mVA== X-Gm-Message-State: AKGB3mK0+QNlqMpYgsCNlbzUBxkusoMgj75ks63Do54+f0/M0u4MqM7+ KMzfAil+geSdnsaT+dArgAv5Ow== X-Google-Smtp-Source: ACJfBovUSe9pmuStmaR31Zld+E+Kyui49W96FWuvl4sK2MDMiOCPWHE6gOxFRrfnq/yGriItMlZ63w== X-Received: by 10.223.138.195 with SMTP id z3mr46074806wrz.152.1514923595658; Tue, 02 Jan 2018 12:06:35 -0800 (PST) Received: from localhost.localdomain ([160.89.138.198]) by smtp.gmail.com with ESMTPSA id m70sm19128526wma.36.2018.01.02.12.06.30 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Jan 2018 12:06:34 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Subject: [PATCH v7 04/10] init: allow initcall tables to be emitted using relative references Date: Tue, 2 Jan 2018 20:05:43 +0000 Message-Id: <20180102200549.22984-5-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180102200549.22984-1-ard.biesheuvel@linaro.org> References: <20180102200549.22984-1-ard.biesheuvel@linaro.org> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.24 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-mips@linux-mips.org, Will Deacon , Paul Mackerras , "H. Peter Anvin" , sparclinux@vger.kernel.org, linux-s390@vger.kernel.org, Nicolas Pitre , x86@kernel.org, Russell King , Ingo Molnar , Catalin Marinas , "Serge E. Hallyn" , Petr Mladek , Kees Cook , Arnd Bergmann , Heiko Carstens , Steven Rostedt , James Morris , Josh Poimboeuf , Bjorn Helgaas , Thomas Gleixner , linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , linuxppc-dev@lists.ozlabs.org, Ralf Baechle , Thomas Garnier , Sergey Senozhatsky , Jessica Yu , Martin Schwidefsky , Andrew Morton , Linus Torvalds , "David S. Miller" Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Allow the initcall tables to be emitted using relative references that are only half the size on 64-bit architectures and don't require fixups at runtime on relocatable kernels. Cc: Petr Mladek Cc: Sergey Senozhatsky Cc: Steven Rostedt Cc: James Morris Cc: "Serge E. Hallyn" Signed-off-by: Ard Biesheuvel --- include/linux/init.h | 44 +++++++++++++++----- init/main.c | 32 +++++++------- kernel/printk/printk.c | 4 +- security/security.c | 4 +- 4 files changed, 53 insertions(+), 31 deletions(-) diff --git a/include/linux/init.h b/include/linux/init.h index ea1b31101d9e..cef8e817e5a5 100644 --- a/include/linux/init.h +++ b/include/linux/init.h @@ -109,8 +109,24 @@ typedef int (*initcall_t)(void); typedef void (*exitcall_t)(void); -extern initcall_t __con_initcall_start[], __con_initcall_end[]; -extern initcall_t __security_initcall_start[], __security_initcall_end[]; +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS +typedef int initcall_entry_t; + +static inline initcall_t initcall_from_entry(initcall_entry_t *entry) +{ + return (initcall_t)((unsigned long)entry + *entry); +} +#else +typedef initcall_t initcall_entry_t; + +static inline initcall_t initcall_from_entry(initcall_entry_t *entry) +{ + return *entry; +} +#endif + +extern initcall_entry_t __con_initcall_start[], __con_initcall_end[]; +extern initcall_entry_t __security_initcall_start[], __security_initcall_end[]; /* Used for contructor calls. */ typedef void (*ctor_fn_t)(void); @@ -160,9 +176,20 @@ extern bool initcall_debug; * as KEEP() in the linker script. */ -#define __define_initcall(fn, id) \ +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS +#define ___define_initcall(fn, id, __sec) \ + __ADDRESSABLE(fn) \ + asm(".section \"" #__sec ".init\", \"a\" \n" \ + "__initcall_" #fn #id ": \n" \ + ".long " VMLINUX_SYMBOL_STR(fn) " - . \n" \ + ".previous \n"); +#else +#define ___define_initcall(fn, id, __sec) \ static initcall_t __initcall_##fn##id __used \ - __attribute__((__section__(".initcall" #id ".init"))) = fn; + __attribute__((__section__(#__sec ".init"))) = fn; +#endif + +#define __define_initcall(fn, id) ___define_initcall(fn, id, .initcall##id) /* * Early initcalls run before initializing SMP. @@ -201,13 +228,8 @@ extern bool initcall_debug; #define __exitcall(fn) \ static exitcall_t __exitcall_##fn __exit_call = fn -#define console_initcall(fn) \ - static initcall_t __initcall_##fn \ - __used __section(.con_initcall.init) = fn - -#define security_initcall(fn) \ - static initcall_t __initcall_##fn \ - __used __section(.security_initcall.init) = fn +#define console_initcall(fn) ___define_initcall(fn,, .con_initcall) +#define security_initcall(fn) ___define_initcall(fn,, .security_initcall) struct obs_kernel_param { const char *str; diff --git a/init/main.c b/init/main.c index a8100b954839..d81487cc126d 100644 --- a/init/main.c +++ b/init/main.c @@ -848,18 +848,18 @@ int __init_or_module do_one_initcall(initcall_t fn) } -extern initcall_t __initcall_start[]; -extern initcall_t __initcall0_start[]; -extern initcall_t __initcall1_start[]; -extern initcall_t __initcall2_start[]; -extern initcall_t __initcall3_start[]; -extern initcall_t __initcall4_start[]; -extern initcall_t __initcall5_start[]; -extern initcall_t __initcall6_start[]; -extern initcall_t __initcall7_start[]; -extern initcall_t __initcall_end[]; - -static initcall_t *initcall_levels[] __initdata = { +extern initcall_entry_t __initcall_start[]; +extern initcall_entry_t __initcall0_start[]; +extern initcall_entry_t __initcall1_start[]; +extern initcall_entry_t __initcall2_start[]; +extern initcall_entry_t __initcall3_start[]; +extern initcall_entry_t __initcall4_start[]; +extern initcall_entry_t __initcall5_start[]; +extern initcall_entry_t __initcall6_start[]; +extern initcall_entry_t __initcall7_start[]; +extern initcall_entry_t __initcall_end[]; + +static initcall_entry_t *initcall_levels[] __initdata = { __initcall0_start, __initcall1_start, __initcall2_start, @@ -885,7 +885,7 @@ static char *initcall_level_names[] __initdata = { static void __init do_initcall_level(int level) { - initcall_t *fn; + initcall_entry_t *fn; strcpy(initcall_command_line, saved_command_line); parse_args(initcall_level_names[level], @@ -895,7 +895,7 @@ static void __init do_initcall_level(int level) NULL, &repair_env_string); for (fn = initcall_levels[level]; fn < initcall_levels[level+1]; fn++) - do_one_initcall(*fn); + do_one_initcall(initcall_from_entry(fn)); } static void __init do_initcalls(void) @@ -926,10 +926,10 @@ static void __init do_basic_setup(void) static void __init do_pre_smp_initcalls(void) { - initcall_t *fn; + initcall_entry_t *fn; for (fn = __initcall_start; fn < __initcall0_start; fn++) - do_one_initcall(*fn); + do_one_initcall(initcall_from_entry(fn)); } /* diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c index b9006617710f..0516005261c7 100644 --- a/kernel/printk/printk.c +++ b/kernel/printk/printk.c @@ -2611,7 +2611,7 @@ EXPORT_SYMBOL(unregister_console); */ void __init console_init(void) { - initcall_t *call; + initcall_entry_t *call; /* Setup the default TTY line discipline. */ n_tty_init(); @@ -2622,7 +2622,7 @@ void __init console_init(void) */ call = __con_initcall_start; while (call < __con_initcall_end) { - (*call)(); + initcall_from_entry(call)(); call++; } } diff --git a/security/security.c b/security/security.c index 1cd8526cb0b7..f648eeff06de 100644 --- a/security/security.c +++ b/security/security.c @@ -45,10 +45,10 @@ static __initdata char chosen_lsm[SECURITY_NAME_MAX + 1] = static void __init do_security_initcalls(void) { - initcall_t *call; + initcall_entry_t *call; call = __security_initcall_start; while (call < __security_initcall_end) { - (*call) (); + initcall_from_entry(call)(); call++; } } From patchwork Tue Jan 2 20:05:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 854751 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zB5691XHgz9s82 for ; Wed, 3 Jan 2018 07:20:33 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="XNQ7JrGE"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zB5686wFdzDqpx for ; Wed, 3 Jan 2018 07:20:32 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="XNQ7JrGE"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=linaro.org (client-ip=2a00:1450:400c:c0c::244; helo=mail-wr0-x244.google.com; envelope-from=ard.biesheuvel@linaro.org; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="XNQ7JrGE"; dkim-atps=neutral Received: from mail-wr0-x244.google.com (mail-wr0-x244.google.com [IPv6:2a00:1450:400c:c0c::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zB4pD3W5zzF0PZ for ; Wed, 3 Jan 2018 07:06:44 +1100 (AEDT) Received: by mail-wr0-x244.google.com with SMTP id o101so20933070wrb.6 for ; Tue, 02 Jan 2018 12:06:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=mUgwtmE8XO4Hwp0PbFiB7DRxpjhYfUHYwRE5RAZQ5hI=; b=XNQ7JrGEjWPy3PKFnPYalcrQuqtv85mlGCtacZb9rZGxQtaUrqUYOpVSk04jltsOTo fgM5xlh15AjaahM/p+1MDFZVbDp/gPRs3DX0Lgg14Wv04BBRNOWjAx6nzkyW26YW1qt+ 925exYfDt17jeoetEnjl9dHLTCAyudN7VOq2M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=mUgwtmE8XO4Hwp0PbFiB7DRxpjhYfUHYwRE5RAZQ5hI=; b=pavvzzdlTFOTnCnP17JM/mzXWts3s1fPHyBAB2iUAfoYUqyoc8dS3tOPrdJ3WE62bL RCpU5jMPPbX8Qw1WE3AfUZl66/6WXYiYTTLjgtB0NYtK6uSw0chGQzh1+xx7/km1CHd+ LGJWbzPXRCu0jrHmzj/qvvzUOYB5ExD9shnu+R0L63LDxc+o0ICU7+INDngw50XOixiL 8aziWdqM9HHgV8FsQXvOsSzwKvkL4hSsRgP2IOz/Q+/rbloea5+ZrpE4eH7FgIzbjCxm 7/RcSfeQjCet31ccLuX0FLEcFCjuK/1m+Pn7Q9Nv593jJ0fJLwk7AQ7H4msCzFtNjwru MYOQ== X-Gm-Message-State: AKGB3mL+PmPgDUJCfO/sbKQ1+RHYrV6JpOAdelGxfy6kJXXeD9lKM4Jm O0cVEtwfjex698/vniDu9osiYw== X-Google-Smtp-Source: ACJfBot+fVisNnahT62GGiEfxafJjhU2CiVcZSDAlXFNCzxllw7nrvOdfiYlqYRAao5mIR8U+/29oQ== X-Received: by 10.223.179.215 with SMTP id x23mr3214340wrd.137.1514923601860; Tue, 02 Jan 2018 12:06:41 -0800 (PST) Received: from localhost.localdomain ([160.89.138.198]) by smtp.gmail.com with ESMTPSA id m70sm19128526wma.36.2018.01.02.12.06.35 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Jan 2018 12:06:41 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Subject: [PATCH v7 05/10] PCI: Add support for relative addressing in quirk tables Date: Tue, 2 Jan 2018 20:05:44 +0000 Message-Id: <20180102200549.22984-6-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180102200549.22984-1-ard.biesheuvel@linaro.org> References: <20180102200549.22984-1-ard.biesheuvel@linaro.org> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.24 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-mips@linux-mips.org, Will Deacon , Paul Mackerras , "H. Peter Anvin" , sparclinux@vger.kernel.org, linux-s390@vger.kernel.org, Nicolas Pitre , x86@kernel.org, Russell King , Ingo Molnar , Catalin Marinas , "Serge E. Hallyn" , Petr Mladek , Kees Cook , Arnd Bergmann , Heiko Carstens , Steven Rostedt , James Morris , Josh Poimboeuf , Bjorn Helgaas , Thomas Gleixner , linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , linuxppc-dev@lists.ozlabs.org, Ralf Baechle , Thomas Garnier , Sergey Senozhatsky , Jessica Yu , Martin Schwidefsky , Andrew Morton , Linus Torvalds , "David S. Miller" Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Allow the PCI quirk tables to be emitted in a way that avoids absolute references to the hook functions. This reduces the size of the entries, and, more importantly, makes them invariant under runtime relocation (e.g., for KASLR) Acked-by: Bjorn Helgaas Signed-off-by: Ard Biesheuvel --- drivers/pci/quirks.c | 13 ++++++++++--- include/linux/pci.h | 20 ++++++++++++++++++++ 2 files changed, 30 insertions(+), 3 deletions(-) diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index 10684b17d0bd..b6d51b4d5ce1 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -3556,9 +3556,16 @@ static void pci_do_fixups(struct pci_dev *dev, struct pci_fixup *f, f->vendor == (u16) PCI_ANY_ID) && (f->device == dev->device || f->device == (u16) PCI_ANY_ID)) { - calltime = fixup_debug_start(dev, f->hook); - f->hook(dev); - fixup_debug_report(dev, calltime, f->hook); + void (*hook)(struct pci_dev *dev); +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS + hook = (void *)((unsigned long)&f->hook_offset + + f->hook_offset); +#else + hook = f->hook; +#endif + calltime = fixup_debug_start(dev, hook); + hook(dev); + fixup_debug_report(dev, calltime, hook); } } diff --git a/include/linux/pci.h b/include/linux/pci.h index c170c9250c8b..086c3965710b 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -1792,7 +1792,11 @@ struct pci_fixup { u16 device; /* You can use PCI_ANY_ID here of course */ u32 class; /* You can use PCI_ANY_ID here too */ unsigned int class_shift; /* should be 0, 8, 16 */ +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS + int hook_offset; +#else void (*hook)(struct pci_dev *dev); +#endif }; enum pci_fixup_pass { @@ -1806,12 +1810,28 @@ enum pci_fixup_pass { pci_fixup_suspend_late, /* pci_device_suspend_late() */ }; +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS +#define __DECLARE_PCI_FIXUP_SECTION(sec, name, vendor, device, class, \ + class_shift, hook) \ + __ADDRESSABLE(hook) \ + asm(".section " #sec ", \"a\" \n" \ + ".balign 16 \n" \ + ".short " #vendor ", " #device " \n" \ + ".long " #class ", " #class_shift " \n" \ + ".long " VMLINUX_SYMBOL_STR(hook) " - . \n" \ + ".previous \n"); +#define DECLARE_PCI_FIXUP_SECTION(sec, name, vendor, device, class, \ + class_shift, hook) \ + __DECLARE_PCI_FIXUP_SECTION(sec, name, vendor, device, class, \ + class_shift, hook) +#else /* Anonymous variables would be nice... */ #define DECLARE_PCI_FIXUP_SECTION(section, name, vendor, device, class, \ class_shift, hook) \ static const struct pci_fixup __PASTE(__pci_fixup_##name,__LINE__) __used \ __attribute__((__section__(#section), aligned((sizeof(void *))))) \ = { vendor, device, class, class_shift, hook }; +#endif #define DECLARE_PCI_FIXUP_CLASS_EARLY(vendor, device, class, \ class_shift, hook) \ From patchwork Tue Jan 2 20:05:45 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 854755 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zB58h54wDz9s82 for ; Wed, 3 Jan 2018 07:22:44 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="OTP3P2Q0"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zB58h3lXGzDqlF for ; Wed, 3 Jan 2018 07:22:44 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="OTP3P2Q0"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=linaro.org (client-ip=2a00:1450:400c:c0c::244; helo=mail-wr0-x244.google.com; envelope-from=ard.biesheuvel@linaro.org; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="OTP3P2Q0"; dkim-atps=neutral Received: from mail-wr0-x244.google.com (mail-wr0-x244.google.com [IPv6:2a00:1450:400c:c0c::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zB4pL3XXrzF0Ph for ; Wed, 3 Jan 2018 07:06:50 +1100 (AEDT) Received: by mail-wr0-x244.google.com with SMTP id p17so25844407wre.7 for ; Tue, 02 Jan 2018 12:06:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=wKZwhkbHcAnKwdhVFtFBF5H65qthlsMyQbywDqRcj68=; b=OTP3P2Q0+30/oFxHDyVu3M4GplpoAAATn7kHNcGIe9IokkX7drMcUm7be/KmZtGjZD WsazG+awbUfxpOYDUSYbOhoZlElVtzJpO670EGPlB/IE9QlNm18N9TnV/I2WtyXjIXNO mkB3y33bzP1G9PF+9/zy9mGfxkST/iRm29Obo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=wKZwhkbHcAnKwdhVFtFBF5H65qthlsMyQbywDqRcj68=; b=HSw8J4JgnjKlpN2RLo/SeIMHY7W26xO/CTOuVjv6JHyg/FbkZ3bk4RDLUEkekEs8mE b8cCUHqS4nQzRYXRKsYpuLSzRzGAZkPrEme/ke2Gt7kxjyEYhQbSAjQtCko2By/s7SQF U80GY6i20McNkiFSZANZm1oXzUdSii3eK5napTarY3Jy1uQORqA7rtHoeAhsX/+oYHjx BhKaWgfOFXxpGjX96lPbJdd0sL3rrRk180dy7UgZ0lqL1upK+rkKbpJ+pgNLKK8tCcPE 2/f7Mn3OenA+jn5I1DP+p5ybij2ZICtoGXYniooE3oVEQfr7RbPkT4FUAY84lAGdwlzJ me6A== X-Gm-Message-State: AKGB3mJ0nUdRoDPzHNVnzGeSZuITlZQcES9V4P69wwOcAm7a80dbW329 5kIOrW67dKr/Z/yOUJ/zmfkoNQ== X-Google-Smtp-Source: ACJfBotnnYdFu2CknGomq9Vv0a+tURqGORyGb6VKbdJrMBb/kDfmRJGviPEpNIBY798zrTvWL9JwZw== X-Received: by 10.223.138.195 with SMTP id z3mr46075191wrz.152.1514923607843; Tue, 02 Jan 2018 12:06:47 -0800 (PST) Received: from localhost.localdomain ([160.89.138.198]) by smtp.gmail.com with ESMTPSA id m70sm19128526wma.36.2018.01.02.12.06.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Jan 2018 12:06:47 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Subject: [PATCH v7 06/10] kernel: tracepoints: add support for relative references Date: Tue, 2 Jan 2018 20:05:45 +0000 Message-Id: <20180102200549.22984-7-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180102200549.22984-1-ard.biesheuvel@linaro.org> References: <20180102200549.22984-1-ard.biesheuvel@linaro.org> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.24 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-mips@linux-mips.org, Will Deacon , Paul Mackerras , "H. Peter Anvin" , sparclinux@vger.kernel.org, linux-s390@vger.kernel.org, Nicolas Pitre , x86@kernel.org, Russell King , Ingo Molnar , Catalin Marinas , "Serge E. Hallyn" , Petr Mladek , Kees Cook , Arnd Bergmann , Heiko Carstens , Steven Rostedt , James Morris , Josh Poimboeuf , Bjorn Helgaas , Thomas Gleixner , linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , linuxppc-dev@lists.ozlabs.org, Ralf Baechle , Thomas Garnier , Sergey Senozhatsky , Jessica Yu , Martin Schwidefsky , Andrew Morton , Linus Torvalds , "David S. Miller" Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" To avoid the need for relocating absolute references to tracepoint structures at boot time when running relocatable kernels (which may take a disproportionate amount of space), add the option to emit these tables as relative references instead. Cc: Ingo Molnar Acked-by: Steven Rostedt (VMware) Signed-off-by: Ard Biesheuvel --- include/linux/tracepoint.h | 19 ++++++-- kernel/tracepoint.c | 50 +++++++++++--------- 2 files changed, 42 insertions(+), 27 deletions(-) diff --git a/include/linux/tracepoint.h b/include/linux/tracepoint.h index a26ffbe09e71..d02bf1a695e8 100644 --- a/include/linux/tracepoint.h +++ b/include/linux/tracepoint.h @@ -228,6 +228,19 @@ extern void syscall_unregfunc(void); return static_key_false(&__tracepoint_##name.key); \ } +#ifdef CONFIG_HAVE_ARCH_PREL32_RELOCATIONS +#define __TRACEPOINT_ENTRY(name) \ + asm(" .section \"__tracepoints_ptrs\", \"a\" \n" \ + " .balign 4 \n" \ + " .long " VMLINUX_SYMBOL_STR(__tracepoint_##name) " - .\n" \ + " .previous \n") +#else +#define __TRACEPOINT_ENTRY(name) \ + static struct tracepoint * const __tracepoint_ptr_##name __used \ + __attribute__((section("__tracepoints_ptrs"))) = \ + &__tracepoint_##name +#endif + /* * We have no guarantee that gcc and the linker won't up-align the tracepoint * structures, so we create an array of pointers that will be used for iteration @@ -237,11 +250,9 @@ extern void syscall_unregfunc(void); static const char __tpstrtab_##name[] \ __attribute__((section("__tracepoints_strings"))) = #name; \ struct tracepoint __tracepoint_##name \ - __attribute__((section("__tracepoints"))) = \ + __attribute__((section("__tracepoints"), used)) = \ { __tpstrtab_##name, STATIC_KEY_INIT_FALSE, reg, unreg, NULL };\ - static struct tracepoint * const __tracepoint_ptr_##name __used \ - __attribute__((section("__tracepoints_ptrs"))) = \ - &__tracepoint_##name; + __TRACEPOINT_ENTRY(name); #define DEFINE_TRACE(name) \ DEFINE_TRACE_FN(name, NULL, NULL); diff --git a/kernel/tracepoint.c b/kernel/tracepoint.c index 685c50ae6300..05649fef106c 100644 --- a/kernel/tracepoint.c +++ b/kernel/tracepoint.c @@ -327,6 +327,28 @@ int tracepoint_probe_unregister(struct tracepoint *tp, void *probe, void *data) } EXPORT_SYMBOL_GPL(tracepoint_probe_unregister); +static void for_each_tracepoint_range(struct tracepoint * const *begin, + struct tracepoint * const *end, + void (*fct)(struct tracepoint *tp, void *priv), + void *priv) +{ + if (!begin) + return; + + if (IS_ENABLED(CONFIG_HAVE_ARCH_PREL32_RELOCATIONS)) { + const int *iter; + + for (iter = (const int *)begin; iter < (const int *)end; iter++) + fct((struct tracepoint *)((unsigned long)iter + *iter), + priv); + } else { + struct tracepoint * const *iter; + + for (iter = begin; iter < end; iter++) + fct(*iter, priv); + } +} + #ifdef CONFIG_MODULES bool trace_module_has_bad_taint(struct module *mod) { @@ -391,15 +413,9 @@ EXPORT_SYMBOL_GPL(unregister_tracepoint_module_notifier); * Ensure the tracer unregistered the module's probes before the module * teardown is performed. Prevents leaks of probe and data pointers. */ -static void tp_module_going_check_quiescent(struct tracepoint * const *begin, - struct tracepoint * const *end) +static void tp_module_going_check_quiescent(struct tracepoint *tp, void *priv) { - struct tracepoint * const *iter; - - if (!begin) - return; - for (iter = begin; iter < end; iter++) - WARN_ON_ONCE((*iter)->funcs); + WARN_ON_ONCE(tp->funcs); } static int tracepoint_module_coming(struct module *mod) @@ -450,8 +466,9 @@ static void tracepoint_module_going(struct module *mod) * Called the going notifier before checking for * quiescence. */ - tp_module_going_check_quiescent(mod->tracepoints_ptrs, - mod->tracepoints_ptrs + mod->num_tracepoints); + for_each_tracepoint_range(mod->tracepoints_ptrs, + mod->tracepoints_ptrs + mod->num_tracepoints, + tp_module_going_check_quiescent, NULL); break; } } @@ -503,19 +520,6 @@ static __init int init_tracepoints(void) __initcall(init_tracepoints); #endif /* CONFIG_MODULES */ -static void for_each_tracepoint_range(struct tracepoint * const *begin, - struct tracepoint * const *end, - void (*fct)(struct tracepoint *tp, void *priv), - void *priv) -{ - struct tracepoint * const *iter; - - if (!begin) - return; - for (iter = begin; iter < end; iter++) - fct(*iter, priv); -} - /** * for_each_kernel_tracepoint - iteration on all kernel tracepoints * @fct: callback From patchwork Tue Jan 2 20:05:46 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 854756 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zB5CB50c2z9s8J for ; Wed, 3 Jan 2018 07:24:54 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="jeEb6ZB0"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zB5CB2shLzF0QN for ; Wed, 3 Jan 2018 07:24:54 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="jeEb6ZB0"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=linaro.org (client-ip=2a00:1450:400c:c0c::241; helo=mail-wr0-x241.google.com; envelope-from=ard.biesheuvel@linaro.org; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="jeEb6ZB0"; dkim-atps=neutral Received: from mail-wr0-x241.google.com (mail-wr0-x241.google.com [IPv6:2a00:1450:400c:c0c::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zB4pV1q09zF0Pg for ; Wed, 3 Jan 2018 07:06:58 +1100 (AEDT) Received: by mail-wr0-x241.google.com with SMTP id l41so37380910wre.11 for ; Tue, 02 Jan 2018 12:06:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=vJVJOehG9ztW1uyWcobdBB+SDC9ShNv9yhy7/gSRq5c=; b=jeEb6ZB0z2/zzzc3KaVGNrlfvf6s5dpCO+sLMLIshA7WAe6n5wRQLi93dLFKC0U/t1 RtaYo5PCH71xiMnB7L/+HADgXxGrsLlzKVHDKbSjZPJnUWWsvvpW9Z8YPBuR7MaKdEci dlLKq+Zd88nyl8pO4RVNbuEc9E8VLOxWlUvT0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=vJVJOehG9ztW1uyWcobdBB+SDC9ShNv9yhy7/gSRq5c=; b=OGBbiFoV2UsOjXbsAvvEYLfO92EWdkGM0OopsEq5FcCAQIKyQRGN5P0vuD3R6iQ50O 6Wa+RBqFPX/jpVydvkeejY47xQm7zWMy+1bGz0VSWWshN8rn9xmyUW73ziZQLyC0QONG SUcFAcza2GDtGszAb/8R5GWVS2QCYGHIFSt7YgRJJ4pvKRMV+ERnVL3gdjJ4/teT1cqt y83fDjkLb7tgB9kzbthMgg3lXut2JCsgheGp6P8CZ/L+EM3WIVGdwKprINV/dLYcV2ub 2IqPUrRySZIcTZsgvXa6YqgBUTqcg5+oqdKNccCou4BBk623kCz5IsNIscI12nKmhJkv VrDQ== X-Gm-Message-State: AKGB3mKwqGta7TPHmrWn+lOXKn7/idbg7qtJW+DUiyQ0Dirl76Lu8oRn ZPPgAGa72BMpuSnn4LZdcgs+5w== X-Google-Smtp-Source: ACJfBotcfbfLrbeVVNZVQ5j01/cki+Swlc3bVd36TNQ3p0kcrJSSaFQ9prQctoMtQNTrlJVL97JTRQ== X-Received: by 10.223.185.85 with SMTP id b21mr42063291wrg.264.1514923615474; Tue, 02 Jan 2018 12:06:55 -0800 (PST) Received: from localhost.localdomain ([160.89.138.198]) by smtp.gmail.com with ESMTPSA id m70sm19128526wma.36.2018.01.02.12.06.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Jan 2018 12:06:54 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Subject: [PATCH v7 07/10] kernel/jump_label: abstract jump_entry member accessors Date: Tue, 2 Jan 2018 20:05:46 +0000 Message-Id: <20180102200549.22984-8-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180102200549.22984-1-ard.biesheuvel@linaro.org> References: <20180102200549.22984-1-ard.biesheuvel@linaro.org> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.24 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-mips@linux-mips.org, Will Deacon , Paul Mackerras , "H. Peter Anvin" , sparclinux@vger.kernel.org, linux-s390@vger.kernel.org, Nicolas Pitre , x86@kernel.org, Russell King , Ingo Molnar , Catalin Marinas , "Serge E. Hallyn" , Petr Mladek , Kees Cook , Arnd Bergmann , Heiko Carstens , Steven Rostedt , James Morris , Josh Poimboeuf , Bjorn Helgaas , Thomas Gleixner , linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , linuxppc-dev@lists.ozlabs.org, Ralf Baechle , Thomas Garnier , Sergey Senozhatsky , Jessica Yu , Martin Schwidefsky , Andrew Morton , Linus Torvalds , "David S. Miller" Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" In preparation of allowing architectures to use relative references in jump_label entries [which can dramatically reduce the memory footprint], introduce abstractions for references to the 'code' and 'key' members of struct jump_entry. Signed-off-by: Ard Biesheuvel --- arch/arm/include/asm/jump_label.h | 27 ++++++++++++++ arch/arm64/include/asm/jump_label.h | 27 ++++++++++++++ arch/mips/include/asm/jump_label.h | 27 ++++++++++++++ arch/powerpc/include/asm/jump_label.h | 27 ++++++++++++++ arch/s390/include/asm/jump_label.h | 27 ++++++++++++++ arch/sparc/include/asm/jump_label.h | 27 ++++++++++++++ arch/tile/include/asm/jump_label.h | 27 ++++++++++++++ arch/x86/include/asm/jump_label.h | 27 ++++++++++++++ kernel/jump_label.c | 38 +++++++++----------- 9 files changed, 232 insertions(+), 22 deletions(-) diff --git a/arch/arm/include/asm/jump_label.h b/arch/arm/include/asm/jump_label.h index e12d7d096fc0..7b05b404063a 100644 --- a/arch/arm/include/asm/jump_label.h +++ b/arch/arm/include/asm/jump_label.h @@ -45,5 +45,32 @@ struct jump_entry { jump_label_t key; }; +static inline jump_label_t jump_entry_code(const struct jump_entry *entry) +{ + return entry->code; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + return (struct static_key *)((unsigned long)entry->key & ~1UL); +} + +static inline bool jump_entry_is_branch(const struct jump_entry *entry) +{ + return (unsigned long)entry->key & 1UL; +} + +static inline bool jump_entry_is_module_init(const struct jump_entry *entry) +{ + return entry->code == 0; +} + +static inline void jump_entry_set_module_init(struct jump_entry *entry) +{ + entry->code = 0; +} + +#define jump_label_swap NULL + #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/arm64/include/asm/jump_label.h b/arch/arm64/include/asm/jump_label.h index 1b5e0e843c3a..9d6e46355c89 100644 --- a/arch/arm64/include/asm/jump_label.h +++ b/arch/arm64/include/asm/jump_label.h @@ -62,5 +62,32 @@ struct jump_entry { jump_label_t key; }; +static inline jump_label_t jump_entry_code(const struct jump_entry *entry) +{ + return entry->code; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + return (struct static_key *)((unsigned long)entry->key & ~1UL); +} + +static inline bool jump_entry_is_branch(const struct jump_entry *entry) +{ + return (unsigned long)entry->key & 1UL; +} + +static inline bool jump_entry_is_module_init(const struct jump_entry *entry) +{ + return entry->code == 0; +} + +static inline void jump_entry_set_module_init(struct jump_entry *entry) +{ + entry->code = 0; +} + +#define jump_label_swap NULL + #endif /* __ASSEMBLY__ */ #endif /* __ASM_JUMP_LABEL_H */ diff --git a/arch/mips/include/asm/jump_label.h b/arch/mips/include/asm/jump_label.h index e77672539e8e..70df9293dc49 100644 --- a/arch/mips/include/asm/jump_label.h +++ b/arch/mips/include/asm/jump_label.h @@ -66,5 +66,32 @@ struct jump_entry { jump_label_t key; }; +static inline jump_label_t jump_entry_code(const struct jump_entry *entry) +{ + return entry->code; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + return (struct static_key *)((unsigned long)entry->key & ~1UL); +} + +static inline bool jump_entry_is_branch(const struct jump_entry *entry) +{ + return (unsigned long)entry->key & 1UL; +} + +static inline bool jump_entry_is_module_init(const struct jump_entry *entry) +{ + return entry->code == 0; +} + +static inline void jump_entry_set_module_init(struct jump_entry *entry) +{ + entry->code = 0; +} + +#define jump_label_swap NULL + #endif /* __ASSEMBLY__ */ #endif /* _ASM_MIPS_JUMP_LABEL_H */ diff --git a/arch/powerpc/include/asm/jump_label.h b/arch/powerpc/include/asm/jump_label.h index 9a287e0ac8b1..412b2699c9f6 100644 --- a/arch/powerpc/include/asm/jump_label.h +++ b/arch/powerpc/include/asm/jump_label.h @@ -59,6 +59,33 @@ struct jump_entry { jump_label_t key; }; +static inline jump_label_t jump_entry_code(const struct jump_entry *entry) +{ + return entry->code; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + return (struct static_key *)((unsigned long)entry->key & ~1UL); +} + +static inline bool jump_entry_is_branch(const struct jump_entry *entry) +{ + return (unsigned long)entry->key & 1UL; +} + +static inline bool jump_entry_is_module_init(const struct jump_entry *entry) +{ + return entry->code == 0; +} + +static inline void jump_entry_set_module_init(struct jump_entry *entry) +{ + entry->code = 0; +} + +#define jump_label_swap NULL + #else #define ARCH_STATIC_BRANCH(LABEL, KEY) \ 1098: nop; \ diff --git a/arch/s390/include/asm/jump_label.h b/arch/s390/include/asm/jump_label.h index 40f651292aa7..1ecfd46835d9 100644 --- a/arch/s390/include/asm/jump_label.h +++ b/arch/s390/include/asm/jump_label.h @@ -50,5 +50,32 @@ struct jump_entry { jump_label_t key; }; +static inline jump_label_t jump_entry_code(const struct jump_entry *entry) +{ + return entry->code; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + return (struct static_key *)((unsigned long)entry->key & ~1UL); +} + +static inline bool jump_entry_is_branch(const struct jump_entry *entry) +{ + return (unsigned long)entry->key & 1UL; +} + +static inline bool jump_entry_is_module_init(const struct jump_entry *entry) +{ + return entry->code == 0; +} + +static inline void jump_entry_set_module_init(struct jump_entry *entry) +{ + entry->code = 0; +} + +#define jump_label_swap NULL + #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/sparc/include/asm/jump_label.h b/arch/sparc/include/asm/jump_label.h index 94eb529dcb77..18e893687f7c 100644 --- a/arch/sparc/include/asm/jump_label.h +++ b/arch/sparc/include/asm/jump_label.h @@ -48,5 +48,32 @@ struct jump_entry { jump_label_t key; }; +static inline jump_label_t jump_entry_code(const struct jump_entry *entry) +{ + return entry->code; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + return (struct static_key *)((unsigned long)entry->key & ~1UL); +} + +static inline bool jump_entry_is_branch(const struct jump_entry *entry) +{ + return (unsigned long)entry->key & 1UL; +} + +static inline bool jump_entry_is_module_init(const struct jump_entry *entry) +{ + return entry->code == 0; +} + +static inline void jump_entry_set_module_init(struct jump_entry *entry) +{ + entry->code = 0; +} + +#define jump_label_swap NULL + #endif /* __ASSEMBLY__ */ #endif diff --git a/arch/tile/include/asm/jump_label.h b/arch/tile/include/asm/jump_label.h index cde7573f397b..86acaa6ff33d 100644 --- a/arch/tile/include/asm/jump_label.h +++ b/arch/tile/include/asm/jump_label.h @@ -55,4 +55,31 @@ struct jump_entry { jump_label_t key; }; +static inline jump_label_t jump_entry_code(const struct jump_entry *entry) +{ + return entry->code; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + return (struct static_key *)((unsigned long)entry->key & ~1UL); +} + +static inline bool jump_entry_is_branch(const struct jump_entry *entry) +{ + return (unsigned long)entry->key & 1UL; +} + +static inline bool jump_entry_is_module_init(const struct jump_entry *entry) +{ + return entry->code == 0; +} + +static inline void jump_entry_set_module_init(struct jump_entry *entry) +{ + entry->code = 0; +} + +#define jump_label_swap NULL + #endif /* _ASM_TILE_JUMP_LABEL_H */ diff --git a/arch/x86/include/asm/jump_label.h b/arch/x86/include/asm/jump_label.h index 8c0de4282659..009ff2699d07 100644 --- a/arch/x86/include/asm/jump_label.h +++ b/arch/x86/include/asm/jump_label.h @@ -74,6 +74,33 @@ struct jump_entry { jump_label_t key; }; +static inline jump_label_t jump_entry_code(const struct jump_entry *entry) +{ + return entry->code; +} + +static inline struct static_key *jump_entry_key(const struct jump_entry *entry) +{ + return (struct static_key *)((unsigned long)entry->key & ~1UL); +} + +static inline bool jump_entry_is_branch(const struct jump_entry *entry) +{ + return (unsigned long)entry->key & 1UL; +} + +static inline bool jump_entry_is_module_init(const struct jump_entry *entry) +{ + return entry->code == 0; +} + +static inline void jump_entry_set_module_init(struct jump_entry *entry) +{ + entry->code = 0; +} + +#define jump_label_swap NULL + #else /* __ASSEMBLY__ */ .macro STATIC_JUMP_IF_TRUE target, key, def diff --git a/kernel/jump_label.c b/kernel/jump_label.c index 8594d24e4adc..4f44db58d981 100644 --- a/kernel/jump_label.c +++ b/kernel/jump_label.c @@ -37,10 +37,12 @@ static int jump_label_cmp(const void *a, const void *b) const struct jump_entry *jea = a; const struct jump_entry *jeb = b; - if (jea->key < jeb->key) + if ((unsigned long)jump_entry_key(jea) < + (unsigned long)jump_entry_key(jeb)) return -1; - if (jea->key > jeb->key) + if ((unsigned long)jump_entry_key(jea) > + (unsigned long)jump_entry_key(jeb)) return 1; return 0; @@ -53,7 +55,8 @@ jump_label_sort_entries(struct jump_entry *start, struct jump_entry *stop) size = (((unsigned long)stop - (unsigned long)start) / sizeof(struct jump_entry)); - sort(start, size, sizeof(struct jump_entry), jump_label_cmp, NULL); + sort(start, size, sizeof(struct jump_entry), jump_label_cmp, + jump_label_swap); } static void jump_label_update(struct static_key *key); @@ -254,8 +257,8 @@ EXPORT_SYMBOL_GPL(jump_label_rate_limit); static int addr_conflict(struct jump_entry *entry, void *start, void *end) { - if (entry->code <= (unsigned long)end && - entry->code + JUMP_LABEL_NOP_SIZE > (unsigned long)start) + if (jump_entry_code(entry) <= (unsigned long)end && + jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE > (unsigned long)start) return 1; return 0; @@ -314,16 +317,6 @@ static inline void static_key_set_linked(struct static_key *key) key->type |= JUMP_TYPE_LINKED; } -static inline struct static_key *jump_entry_key(struct jump_entry *entry) -{ - return (struct static_key *)((unsigned long)entry->key & ~1UL); -} - -static bool jump_entry_branch(struct jump_entry *entry) -{ - return (unsigned long)entry->key & 1UL; -} - /*** * A 'struct static_key' uses a union such that it either points directly * to a table of 'struct jump_entry' or to a linked list of modules which in @@ -348,7 +341,7 @@ static enum jump_label_type jump_label_type(struct jump_entry *entry) { struct static_key *key = jump_entry_key(entry); bool enabled = static_key_enabled(key); - bool branch = jump_entry_branch(entry); + bool branch = jump_entry_is_branch(entry); /* See the comment in linux/jump_label.h */ return enabled ^ branch; @@ -364,7 +357,8 @@ static void __jump_label_update(struct static_key *key, * kernel_text_address() verifies we are not in core kernel * init code, see jump_label_invalidate_module_init(). */ - if (entry->code && kernel_text_address(entry->code)) + if (!jump_entry_is_module_init(entry) && + kernel_text_address(jump_entry_code(entry))) arch_jump_label_transform(entry, jump_label_type(entry)); } } @@ -417,7 +411,7 @@ static enum jump_label_type jump_label_init_type(struct jump_entry *entry) { struct static_key *key = jump_entry_key(entry); bool type = static_key_type(key); - bool branch = jump_entry_branch(entry); + bool branch = jump_entry_is_branch(entry); /* See the comment in linux/jump_label.h */ return type ^ branch; @@ -541,7 +535,7 @@ static int jump_label_add_module(struct module *mod) continue; key = iterk; - if (within_module(iter->key, mod)) { + if (within_module((unsigned long)key, mod)) { static_key_set_entries(key, iter); continue; } @@ -591,7 +585,7 @@ static void jump_label_del_module(struct module *mod) key = jump_entry_key(iter); - if (within_module(iter->key, mod)) + if (within_module((unsigned long)key, mod)) continue; /* No memory during module load */ @@ -634,8 +628,8 @@ static void jump_label_invalidate_module_init(struct module *mod) struct jump_entry *iter; for (iter = iter_start; iter < iter_stop; iter++) { - if (within_module_init(iter->code, mod)) - iter->code = 0; + if (within_module_init(jump_entry_code(iter), mod)) + jump_entry_set_module_init(iter); } } From patchwork Tue Jan 2 20:05:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 854757 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zB5GT21Dwz9rxj for ; Wed, 3 Jan 2018 07:27:45 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="ZXOsFAJi"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zB5GS6tPvzF0QT for ; Wed, 3 Jan 2018 07:27:44 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="ZXOsFAJi"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=linaro.org (client-ip=2a00:1450:400c:c0c::243; helo=mail-wr0-x243.google.com; envelope-from=ard.biesheuvel@linaro.org; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="ZXOsFAJi"; dkim-atps=neutral Received: from mail-wr0-x243.google.com (mail-wr0-x243.google.com [IPv6:2a00:1450:400c:c0c::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zB4pc609NzF0Pd for ; Wed, 3 Jan 2018 07:07:04 +1100 (AEDT) Received: by mail-wr0-x243.google.com with SMTP id w68so37152665wrc.10 for ; Tue, 02 Jan 2018 12:07:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=RtgdmqqE7QnNDS0Od8ctH/UMcFkuxkhGpZ1L+nRoK9Y=; b=ZXOsFAJioLobdvr9YpnYOCxgnNq32bEakvz+d8oHqbR+rWFbkbaVPtVyawqeeowoT1 Dzbj11m2Bi/29hzMeufYCDd4u72LhfE0tCsxWkG9HEh2S9NwOvXB6vLZxGTzpuOUsPI5 3M8pDezCKUo4tMXGIDfUWJ0EZBHr/1jyyzjGc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=RtgdmqqE7QnNDS0Od8ctH/UMcFkuxkhGpZ1L+nRoK9Y=; b=Xuknr0M3kkf/JJf5/oH6ZCp4NNNkEESaUMd6cZosNOIolY7SwtX1ZuopiAZhB229VS 9xnR13pjXysmBFdD9EaNzOl86wICMmP3As2FX8ugL4jP780bb/BsSUzLu1/8v4ch9nk0 jM8D+ht9j6nc+hurvYzcNVTYpsLtKYwYp0aPAhwdKxpy21RCw1MV+TXiQdWkgR1fH50n HyO+Q3qEvRenH78flzUrrdQ/igPk0HOVenu07ahjXMM6QyEOzEHP/0EogQm9P5E0zz3t Nr7vLzS5sHVHVYNRMlRYH9O5Ea1UrHsWYAdpRvSFjVsy9COAIwiSrVMlCFwYztUuZ6dn Xecw== X-Gm-Message-State: AKGB3mJIz+3eHAiHOoguEk8a4KPV8e6rfW/M6VaEq5ynsMncJ4Zk88hl c/meDmAFpyESrMcP2K8YW5xiUA== X-Google-Smtp-Source: ACJfBotnEYN73SoeBAeXFa2dgkyQaTjIm+dvcolnJ1DYqLR79c3pB4kn56LVZv46+GU71sfeYtgiCw== X-Received: by 10.223.186.13 with SMTP id o13mr15747922wrg.43.1514923622162; Tue, 02 Jan 2018 12:07:02 -0800 (PST) Received: from localhost.localdomain ([160.89.138.198]) by smtp.gmail.com with ESMTPSA id m70sm19128526wma.36.2018.01.02.12.06.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Jan 2018 12:07:01 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Subject: [PATCH v7 08/10] arm64/kernel: jump_label: use relative references Date: Tue, 2 Jan 2018 20:05:47 +0000 Message-Id: <20180102200549.22984-9-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180102200549.22984-1-ard.biesheuvel@linaro.org> References: <20180102200549.22984-1-ard.biesheuvel@linaro.org> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.24 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-mips@linux-mips.org, Will Deacon , Paul Mackerras , "H. Peter Anvin" , sparclinux@vger.kernel.org, linux-s390@vger.kernel.org, Nicolas Pitre , x86@kernel.org, Russell King , Ingo Molnar , Catalin Marinas , "Serge E. Hallyn" , Petr Mladek , Kees Cook , Arnd Bergmann , Heiko Carstens , Steven Rostedt , James Morris , Josh Poimboeuf , Bjorn Helgaas , Thomas Gleixner , linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , linuxppc-dev@lists.ozlabs.org, Ralf Baechle , Thomas Garnier , Sergey Senozhatsky , Jessica Yu , Martin Schwidefsky , Andrew Morton , Linus Torvalds , "David S. Miller" Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" On a randomly chosen distro kernel build for arm64, vmlinux.o shows the following sections, containing jump label entries, and the associated RELA relocation records, respectively: ... [38088] __jump_table PROGBITS 0000000000000000 00e19f30 000000000002ea10 0000000000000000 WA 0 0 8 [38089] .rela__jump_table RELA 0000000000000000 01fd8bb0 000000000008be30 0000000000000018 I 38178 38088 8 ... In other words, we have 190 KB worth of 'struct jump_entry' instances, and 573 KB worth of RELA entries to relocate each entry's code, target and key members. This means the RELA section occupies 10% of the .init segment, and the two sections combined represent 5% of vmlinux's entire memory footprint. So let's switch from 64-bit absolute references to 32-bit relative references: this reduces the size of the __jump_table by 50%, and gets rid of the RELA section entirely. Note that this requires some extra care in the sorting routine, given that the offsets change when entries are moved around in the jump_entry table. Signed-off-by: Ard Biesheuvel --- arch/arm64/include/asm/jump_label.h | 27 ++++++++++++-------- arch/arm64/kernel/jump_label.c | 22 +++++++++++++--- 2 files changed, 36 insertions(+), 13 deletions(-) diff --git a/arch/arm64/include/asm/jump_label.h b/arch/arm64/include/asm/jump_label.h index 9d6e46355c89..8f82adeb7b0b 100644 --- a/arch/arm64/include/asm/jump_label.h +++ b/arch/arm64/include/asm/jump_label.h @@ -30,8 +30,8 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool bran { asm goto("1: nop\n\t" ".pushsection __jump_table, \"aw\"\n\t" - ".align 3\n\t" - ".quad 1b, %l[l_yes], %c0\n\t" + ".align 2\n\t" + ".long 1b - ., %l[l_yes] - ., %c0 - .\n\t" ".popsection\n\t" : : "i"(&((char *)key)[branch]) : : l_yes); @@ -44,8 +44,8 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool { asm goto("1: b %l[l_yes]\n\t" ".pushsection __jump_table, \"aw\"\n\t" - ".align 3\n\t" - ".quad 1b, %l[l_yes], %c0\n\t" + ".align 2\n\t" + ".long 1b - ., %l[l_yes] - ., %c0 - .\n\t" ".popsection\n\t" : : "i"(&((char *)key)[branch]) : : l_yes); @@ -57,19 +57,26 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool typedef u64 jump_label_t; struct jump_entry { - jump_label_t code; - jump_label_t target; - jump_label_t key; + s32 code; + s32 target; + s32 key; }; static inline jump_label_t jump_entry_code(const struct jump_entry *entry) { - return entry->code; + return (unsigned long)&entry->code + entry->code; +} + +static inline jump_label_t jump_entry_target(const struct jump_entry *entry) +{ + return (unsigned long)&entry->target + entry->target; } static inline struct static_key *jump_entry_key(const struct jump_entry *entry) { - return (struct static_key *)((unsigned long)entry->key & ~1UL); + unsigned long key = (unsigned long)&entry->key + entry->key; + + return (struct static_key *)(key & ~1UL); } static inline bool jump_entry_is_branch(const struct jump_entry *entry) @@ -87,7 +94,7 @@ static inline void jump_entry_set_module_init(struct jump_entry *entry) entry->code = 0; } -#define jump_label_swap NULL +void jump_label_swap(void *a, void *b, int size); #endif /* __ASSEMBLY__ */ #endif /* __ASM_JUMP_LABEL_H */ diff --git a/arch/arm64/kernel/jump_label.c b/arch/arm64/kernel/jump_label.c index c2dd1ad3e648..2b8e459e91f7 100644 --- a/arch/arm64/kernel/jump_label.c +++ b/arch/arm64/kernel/jump_label.c @@ -25,12 +25,12 @@ void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type) { - void *addr = (void *)entry->code; + void *addr = (void *)jump_entry_code(entry); u32 insn; if (type == JUMP_LABEL_JMP) { - insn = aarch64_insn_gen_branch_imm(entry->code, - entry->target, + insn = aarch64_insn_gen_branch_imm(jump_entry_code(entry), + jump_entry_target(entry), AARCH64_INSN_BRANCH_NOLINK); } else { insn = aarch64_insn_gen_nop(); @@ -50,4 +50,20 @@ void arch_jump_label_transform_static(struct jump_entry *entry, */ } +void jump_label_swap(void *a, void *b, int size) +{ + long delta = (unsigned long)a - (unsigned long)b; + struct jump_entry *jea = a; + struct jump_entry *jeb = b; + struct jump_entry tmp = *jea; + + jea->code = jeb->code - delta; + jea->target = jeb->target - delta; + jea->key = jeb->key - delta; + + jeb->code = tmp.code + delta; + jeb->target = tmp.target + delta; + jeb->key = tmp.key + delta; +} + #endif /* HAVE_JUMP_LABEL */ From patchwork Tue Jan 2 20:05:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 854758 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zB5Kz2wNJz9rxj for ; Wed, 3 Jan 2018 07:30:47 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="IlsjC4yq"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zB5Kz1GpKzF0QH for ; Wed, 3 Jan 2018 07:30:47 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="IlsjC4yq"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=linaro.org (client-ip=2a00:1450:400c:c0c::244; helo=mail-wr0-x244.google.com; envelope-from=ard.biesheuvel@linaro.org; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="IlsjC4yq"; dkim-atps=neutral Received: from mail-wr0-x244.google.com (mail-wr0-x244.google.com [IPv6:2a00:1450:400c:c0c::244]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zB4pm239zzF0PX for ; Wed, 3 Jan 2018 07:07:12 +1100 (AEDT) Received: by mail-wr0-x244.google.com with SMTP id g17so40338530wrd.13 for ; Tue, 02 Jan 2018 12:07:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=3xW6hB18H7nCj1B8KM19meDl2NukzrBIHv9vfi9d94c=; b=IlsjC4yqku/Bgvxd3dSYA80TsqwMeZzyABWwlbxe827nworF3Z2Xw2NJJ9GC5u1EVj TlmJPcYtP0p5eP2s5u9UB06rVtkb1x+/J7q2wqmbSLsSc3bgNNB1+bkZWxdCYlujbUoP GxFZ5Jka238l0BCJThgTwy43cQ6Jj89D4IPmg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=3xW6hB18H7nCj1B8KM19meDl2NukzrBIHv9vfi9d94c=; b=N2DUBpKmFFJDrOxzQBW+6IEVLoZJQYedjuK9h6g6EwY+zdcuj0MC78H8HMUtWVbhkD GSXAaFdrse3ArHdgiQQAXM264lga2+Uy41GUYc+Obq9MgBGARg/l0wKBVfUvEvlxYluT M4s5QIhe3R+FmsZZwfCl4Ea/c4vmNimHgGn19CKk/ThXwx/LAsJXW7JW5rx3AfbY5eeL JMZQy/4MmbXMfjci/jcS96W5xFZDKocsY1pZ3BIpQihCgmQEgaYT/uojp5NaIQIzYvcG eRq2Z9GhcNlPBbG8tSCN+0c61CroAV6Qy86OXfBerQ8KC6++jS4vhowH9VamN3NJT1y9 U/sg== X-Gm-Message-State: AKGB3mIhXKSeDsQAeEk9WVxM3EBBZcIj/tVOguXG3NzKvC6shfwrWtFO MZgxK+tq3IqelX0O1ebtgz475Q== X-Google-Smtp-Source: ACJfBoud3bqGO7fUQcY4b1dNTrZOfP3q0fUJPUXWN6CkcwSAMQ/Py8E4OagotjFR3BvalO1ZlBxCBg== X-Received: by 10.223.177.143 with SMTP id q15mr14341520wra.42.1514923629540; Tue, 02 Jan 2018 12:07:09 -0800 (PST) Received: from localhost.localdomain ([160.89.138.198]) by smtp.gmail.com with ESMTPSA id m70sm19128526wma.36.2018.01.02.12.07.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Jan 2018 12:07:08 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Subject: [PATCH v7 09/10] x86: jump_label: switch to jump_entry accessors Date: Tue, 2 Jan 2018 20:05:48 +0000 Message-Id: <20180102200549.22984-10-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180102200549.22984-1-ard.biesheuvel@linaro.org> References: <20180102200549.22984-1-ard.biesheuvel@linaro.org> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.24 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-mips@linux-mips.org, Will Deacon , Paul Mackerras , "H. Peter Anvin" , sparclinux@vger.kernel.org, linux-s390@vger.kernel.org, Nicolas Pitre , x86@kernel.org, Russell King , Ingo Molnar , Catalin Marinas , "Serge E. Hallyn" , Petr Mladek , Kees Cook , Arnd Bergmann , Heiko Carstens , Steven Rostedt , James Morris , Josh Poimboeuf , Bjorn Helgaas , Thomas Gleixner , linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , linuxppc-dev@lists.ozlabs.org, Ralf Baechle , Thomas Garnier , Sergey Senozhatsky , Jessica Yu , Martin Schwidefsky , Andrew Morton , Linus Torvalds , "David S. Miller" Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" In preparation of switching x86 to use place-relative references for the code, target and key members of struct jump_entry, replace direct references to the struct member with invocations of the new accessors. This will allow us to make the switch by modifying the accessors only. Signed-off-by: Ard Biesheuvel --- arch/x86/kernel/jump_label.c | 43 ++++++++++++-------- 1 file changed, 26 insertions(+), 17 deletions(-) diff --git a/arch/x86/kernel/jump_label.c b/arch/x86/kernel/jump_label.c index e56c95be2808..d64296092ef5 100644 --- a/arch/x86/kernel/jump_label.c +++ b/arch/x86/kernel/jump_label.c @@ -52,22 +52,24 @@ static void __jump_label_transform(struct jump_entry *entry, * Jump label is enabled for the first time. * So we expect a default_nop... */ - if (unlikely(memcmp((void *)entry->code, default_nop, 5) - != 0)) - bug_at((void *)entry->code, __LINE__); + if (unlikely(memcmp((void *)jump_entry_code(entry), + default_nop, 5) != 0)) + bug_at((void *)jump_entry_code(entry), + __LINE__); } else { /* * ...otherwise expect an ideal_nop. Otherwise * something went horribly wrong. */ - if (unlikely(memcmp((void *)entry->code, ideal_nop, 5) - != 0)) - bug_at((void *)entry->code, __LINE__); + if (unlikely(memcmp((void *)jump_entry_code(entry), + ideal_nop, 5) != 0)) + bug_at((void *)jump_entry_code(entry), + __LINE__); } code.jump = 0xe9; - code.offset = entry->target - - (entry->code + JUMP_LABEL_NOP_SIZE); + code.offset = jump_entry_target(entry) - + (jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE); } else { /* * We are disabling this jump label. If it is not what @@ -76,14 +78,18 @@ static void __jump_label_transform(struct jump_entry *entry, * are converting the default nop to the ideal nop. */ if (init) { - if (unlikely(memcmp((void *)entry->code, default_nop, 5) != 0)) - bug_at((void *)entry->code, __LINE__); + if (unlikely(memcmp((void *)jump_entry_code(entry), + default_nop, 5) != 0)) + bug_at((void *)jump_entry_code(entry), + __LINE__); } else { code.jump = 0xe9; - code.offset = entry->target - - (entry->code + JUMP_LABEL_NOP_SIZE); - if (unlikely(memcmp((void *)entry->code, &code, 5) != 0)) - bug_at((void *)entry->code, __LINE__); + code.offset = jump_entry_target(entry) - + (jump_entry_code(entry) + JUMP_LABEL_NOP_SIZE); + if (unlikely(memcmp((void *)jump_entry_code(entry), + &code, 5) != 0)) + bug_at((void *)jump_entry_code(entry), + __LINE__); } memcpy(&code, ideal_nops[NOP_ATOMIC5], JUMP_LABEL_NOP_SIZE); } @@ -97,10 +103,13 @@ static void __jump_label_transform(struct jump_entry *entry, * */ if (poker) - (*poker)((void *)entry->code, &code, JUMP_LABEL_NOP_SIZE); + (*poker)((void *)jump_entry_code(entry), &code, + JUMP_LABEL_NOP_SIZE); else - text_poke_bp((void *)entry->code, &code, JUMP_LABEL_NOP_SIZE, - (void *)entry->code + JUMP_LABEL_NOP_SIZE); + text_poke_bp((void *)jump_entry_code(entry), &code, + JUMP_LABEL_NOP_SIZE, + (void *)jump_entry_code(entry) + + JUMP_LABEL_NOP_SIZE); } void arch_jump_label_transform(struct jump_entry *entry, From patchwork Tue Jan 2 20:05:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ard Biesheuvel X-Patchwork-Id: 854759 Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [103.22.144.68]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zB5NR2ldFz9s75 for ; Wed, 3 Jan 2018 07:32:55 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="U4JVdCRA"; dkim-atps=neutral Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 3zB5NQ6wxdzDqJM for ; Wed, 3 Jan 2018 07:32:54 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="U4JVdCRA"; dkim-atps=neutral X-Original-To: linuxppc-dev@lists.ozlabs.org Delivered-To: linuxppc-dev@lists.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=linaro.org (client-ip=2a00:1450:400c:c09::241; helo=mail-wm0-x241.google.com; envelope-from=ard.biesheuvel@linaro.org; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="U4JVdCRA"; dkim-atps=neutral Received: from mail-wm0-x241.google.com (mail-wm0-x241.google.com [IPv6:2a00:1450:400c:c09::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 3zB4pt5h3DzF0Ph for ; Wed, 3 Jan 2018 07:07:18 +1100 (AEDT) Received: by mail-wm0-x241.google.com with SMTP id g75so63340670wme.0 for ; Tue, 02 Jan 2018 12:07:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=y3cZHthDjrZdZe+7+thZcDtJZzDXcVyA3ettxQyn2/I=; b=U4JVdCRAWxIu8asYC7VR7yKgASQ4tUAP3aRQgb65Z/v9h8Of/kX68XF7IQrKEQqRbL mILpXc3g0V1XmE0jvCO2bKvYotrOisM9fdYrmv1LQvUXHa3V9ocBPN5rZ/3uvp1owdcE JyUIUE3jrgb01ODtZJbvYgN0b++pSR4Jo52yo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=y3cZHthDjrZdZe+7+thZcDtJZzDXcVyA3ettxQyn2/I=; b=PlPjU12Kc+jemXbpPzEEo7tdfEkmWxpoSpdIO9aiO67IKd33jOOIr/5TYhcgl+4cK+ 1LAYqXQsiZJIGA/5qKCDhwXrW5dFwkrnSwyQteWuUa+I+6Q0nM4MfMkayLkaWAOptg3X dvCLmAQpO8IYbOw3PyiecHP7m8dWYztkV5PGNqI6HABhkU0U/zEi9KLEDwu9bkBQaKGn REpWEe2VOxe04eoAi18oAOGbbysr7Nj/kfJo0X2YHMFp25vh6AWJ2ly8PFb7YZCOJ3pU EnlcaJnPWNIBEia5QfO4/hkXd6bpnd6/JoeFcd84I05spBBcBdvNKRAGw5K1Dkuc5whk P5mA== X-Gm-Message-State: AKGB3mLxbCfwORPZbH61BDqwHe5yz4D0/2z2qlO1KNwec+/qwAa/STTs Yl1hy5Ia5kLRyRHjep+56Bx5KA== X-Google-Smtp-Source: ACJfBoviLzGvtRaGzz7XrqH0Ivxu6gFKE14Z5TUX+bR1ffelvedzDX5yPzkCFu05qY1xL5FDoDZ2MA== X-Received: by 10.28.74.147 with SMTP id n19mr40211357wmi.134.1514923636103; Tue, 02 Jan 2018 12:07:16 -0800 (PST) Received: from localhost.localdomain ([160.89.138.198]) by smtp.gmail.com with ESMTPSA id m70sm19128526wma.36.2018.01.02.12.07.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 02 Jan 2018 12:07:15 -0800 (PST) From: Ard Biesheuvel To: linux-kernel@vger.kernel.org Subject: [PATCH v7 10/10] x86/kernel: jump_table: use relative references Date: Tue, 2 Jan 2018 20:05:49 +0000 Message-Id: <20180102200549.22984-11-ard.biesheuvel@linaro.org> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20180102200549.22984-1-ard.biesheuvel@linaro.org> References: <20180102200549.22984-1-ard.biesheuvel@linaro.org> X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.24 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-mips@linux-mips.org, Will Deacon , Paul Mackerras , "H. Peter Anvin" , sparclinux@vger.kernel.org, linux-s390@vger.kernel.org, Nicolas Pitre , x86@kernel.org, Russell King , Ingo Molnar , Catalin Marinas , "Serge E. Hallyn" , Petr Mladek , Kees Cook , Arnd Bergmann , Heiko Carstens , Steven Rostedt , James Morris , Josh Poimboeuf , Bjorn Helgaas , Thomas Gleixner , linux-arm-kernel@lists.infradead.org, Ard Biesheuvel , linuxppc-dev@lists.ozlabs.org, Ralf Baechle , Thomas Garnier , Sergey Senozhatsky , Jessica Yu , Martin Schwidefsky , Andrew Morton , Linus Torvalds , "David S. Miller" Errors-To: linuxppc-dev-bounces+patchwork-incoming=ozlabs.org@lists.ozlabs.org Sender: "Linuxppc-dev" Similar to the arm64 case, 64-bit x86 can benefit from using 32-bit relative references rather than 64-bit absolute ones when emitting struct jump_entry instances. Not only does this reduce the memory footprint of the entries themselves by 50%, it also removes the need for carrying relocation metadata on relocatable builds (i.e., for KASLR) which saves a fair chunk of .init space as well (although the savings are not as dramatic as on arm64) Signed-off-by: Ard Biesheuvel --- arch/x86/include/asm/jump_label.h | 35 ++++++++++++-------- arch/x86/kernel/jump_label.c | 16 +++++++++ tools/objtool/special.c | 4 +-- 3 files changed, 39 insertions(+), 16 deletions(-) diff --git a/arch/x86/include/asm/jump_label.h b/arch/x86/include/asm/jump_label.h index 009ff2699d07..35fc2c5ec846 100644 --- a/arch/x86/include/asm/jump_label.h +++ b/arch/x86/include/asm/jump_label.h @@ -36,8 +36,8 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool bran asm_volatile_goto("1:" ".byte " __stringify(STATIC_KEY_INIT_NOP) "\n\t" ".pushsection __jump_table, \"aw\" \n\t" - _ASM_ALIGN "\n\t" - _ASM_PTR "1b, %l[l_yes], %c0 + %c1 \n\t" + ".balign 4\n\t" + ".long 1b - ., %l[l_yes] - ., %c0 + %c1 - .\n\t" ".popsection \n\t" : : "i" (key), "i" (branch) : : l_yes); @@ -52,8 +52,8 @@ static __always_inline bool arch_static_branch_jump(struct static_key *key, bool ".byte 0xe9\n\t .long %l[l_yes] - 2f\n\t" "2:\n\t" ".pushsection __jump_table, \"aw\" \n\t" - _ASM_ALIGN "\n\t" - _ASM_PTR "1b, %l[l_yes], %c0 + %c1 \n\t" + ".balign 4\n\t" + ".long 1b - ., %l[l_yes] - ., %c0 + %c1 - .\n\t" ".popsection \n\t" : : "i" (key), "i" (branch) : : l_yes); @@ -69,19 +69,26 @@ typedef u32 jump_label_t; #endif struct jump_entry { - jump_label_t code; - jump_label_t target; - jump_label_t key; + s32 code; + s32 target; + s32 key; }; static inline jump_label_t jump_entry_code(const struct jump_entry *entry) { - return entry->code; + return (unsigned long)&entry->code + entry->code; +} + +static inline jump_label_t jump_entry_target(const struct jump_entry *entry) +{ + return (unsigned long)&entry->target + entry->target; } static inline struct static_key *jump_entry_key(const struct jump_entry *entry) { - return (struct static_key *)((unsigned long)entry->key & ~1UL); + unsigned long key = (unsigned long)&entry->key + entry->key; + + return (struct static_key *)(key & ~1UL); } static inline bool jump_entry_is_branch(const struct jump_entry *entry) @@ -99,7 +106,7 @@ static inline void jump_entry_set_module_init(struct jump_entry *entry) entry->code = 0; } -#define jump_label_swap NULL +void jump_label_swap(void *a, void *b, int size); #else /* __ASSEMBLY__ */ @@ -114,8 +121,8 @@ static inline void jump_entry_set_module_init(struct jump_entry *entry) .byte STATIC_KEY_INIT_NOP .endif .pushsection __jump_table, "aw" - _ASM_ALIGN - _ASM_PTR .Lstatic_jump_\@, \target, \key + .balign 4 + .long .Lstatic_jump_\@ - ., \target - ., \key - . .popsection .endm @@ -130,8 +137,8 @@ static inline void jump_entry_set_module_init(struct jump_entry *entry) .Lstatic_jump_after_\@: .endif .pushsection __jump_table, "aw" - _ASM_ALIGN - _ASM_PTR .Lstatic_jump_\@, \target, \key + 1 + .balign 4 + .long .Lstatic_jump_\@ - ., \target - ., \key + 1 - . .popsection .endm diff --git a/arch/x86/kernel/jump_label.c b/arch/x86/kernel/jump_label.c index d64296092ef5..cc5034b42335 100644 --- a/arch/x86/kernel/jump_label.c +++ b/arch/x86/kernel/jump_label.c @@ -149,4 +149,20 @@ __init_or_module void arch_jump_label_transform_static(struct jump_entry *entry, __jump_label_transform(entry, type, text_poke_early, 1); } +void jump_label_swap(void *a, void *b, int size) +{ + long delta = (unsigned long)a - (unsigned long)b; + struct jump_entry *jea = a; + struct jump_entry *jeb = b; + struct jump_entry tmp = *jea; + + jea->code = jeb->code - delta; + jea->target = jeb->target - delta; + jea->key = jeb->key - delta; + + jeb->code = tmp.code + delta; + jeb->target = tmp.target + delta; + jeb->key = tmp.key + delta; +} + #endif diff --git a/tools/objtool/special.c b/tools/objtool/special.c index 84f001d52322..98ae55b39037 100644 --- a/tools/objtool/special.c +++ b/tools/objtool/special.c @@ -30,9 +30,9 @@ #define EX_ORIG_OFFSET 0 #define EX_NEW_OFFSET 4 -#define JUMP_ENTRY_SIZE 24 +#define JUMP_ENTRY_SIZE 12 #define JUMP_ORIG_OFFSET 0 -#define JUMP_NEW_OFFSET 8 +#define JUMP_NEW_OFFSET 4 #define ALT_ENTRY_SIZE 13 #define ALT_ORIG_OFFSET 0