From patchwork Fri Apr 12 19:23:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yuxuan Luo X-Patchwork-Id: 1923234 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=185.125.189.65; helo=lists.ubuntu.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=patchwork.ozlabs.org) Received: from lists.ubuntu.com (lists.ubuntu.com [185.125.189.65]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VGRMK4fCcz1yYP for ; Sat, 13 Apr 2024 05:24:13 +1000 (AEST) Received: from localhost ([127.0.0.1] helo=lists.ubuntu.com) by lists.ubuntu.com with esmtp (Exim 4.86_2) (envelope-from ) id 1rvMVL-0004Om-6u; Fri, 12 Apr 2024 19:24:07 +0000 Received: from smtp-relay-internal-0.internal ([10.131.114.225] helo=smtp-relay-internal-0.canonical.com) by lists.ubuntu.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1rvMVH-0004Ii-Nt for kernel-team@lists.ubuntu.com; Fri, 12 Apr 2024 19:24:03 +0000 Received: from mail-vk1-f199.google.com (mail-vk1-f199.google.com [209.85.221.199]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-0.canonical.com (Postfix) with ESMTPS id C0CEF3F297 for ; Fri, 12 Apr 2024 19:24:02 +0000 (UTC) Received: by mail-vk1-f199.google.com with SMTP id 71dfb90a1353d-4dac0642ce5so401835e0c.3 for ; Fri, 12 Apr 2024 12:24:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712949840; x=1713554640; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7gSIvLhPzL35lqQ5aROrOAnbh3GW/6A13ZAHPpqdCCw=; b=DXNgtnLt65TNuBhqnjp7kKTTVHf31UaGs6s1K2jLYp7X5h0DtrZ3pDP8vRst4DYavQ 2oPVzWL1Y3yO6iiXwFiE6EC1cgtS7/vnzRhD4gepu9hJubIxeNbRsBtMSerK+01+O5Be rLoHHNkHAM29LDot4Vjxyglxx7sp/1BSA9SfwVZ/KdesOxEb0/1c/AUTMtchNSbQEs6e 0WDd4KH/I63ap8BdLJDw2E1K/cd9XL4nwdsPwFpAblUy87CBmpgZcdgPh7cBBYU7W3G+ AHWU0LaY5PU0c9Hgztp7vaX50BLE3+pGFfFB6q2CU+Vo1WpA1oD+akoiXr3/AwplIpvn Y7+A== X-Gm-Message-State: AOJu0Yzej1N5YqxuaIm8GW0C68Xl2erLijD5yRqxshi1fH38PS8XzdMi nJhlAHp0RARUeqz+dflpVh3G1oIIrEKAUQytasjkXqUpw+NcCF9qxp2HjyvzM8w4wNxc24OPOg+ LPjXKypg9S3826w99ECBqcuFZSBWFuuKYhB298XGX6hw+YzEbKqg4UmvE6qENpLc2hYxs3o91x/ zxkWTBQoools0L X-Received: by 2002:a05:6122:3c84:b0:4d4:2931:7d4d with SMTP id fy4-20020a0561223c8400b004d429317d4dmr4116735vkb.5.1712949840584; Fri, 12 Apr 2024 12:24:00 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGJApuNiUOVLK+UOAUY7y+Z5VZN28leHsaIM0PytkCgoMPW3dObYGePHwvcnm44LeJ2ToS8CQ== X-Received: by 2002:a05:6122:3c84:b0:4d4:2931:7d4d with SMTP id fy4-20020a0561223c8400b004d429317d4dmr4116710vkb.5.1712949840037; Fri, 12 Apr 2024 12:24:00 -0700 (PDT) Received: from cache-ubuntu.hsd1.nj.comcast.net ([2001:67c:1562:8007::aac:4795]) by smtp.gmail.com with ESMTPSA id z6-20020a0ce986000000b0069b47a53012sm2654071qvn.140.2024.04.12.12.23.58 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 12 Apr 2024 12:23:59 -0700 (PDT) From: Yuxuan Luo To: kernel-team@lists.ubuntu.com Subject: [PATCH 04/13] x86/bhi: Add support for clearing branch history at syscall entry Date: Fri, 12 Apr 2024 15:23:41 -0400 Message-Id: <20240412192351.89501-5-yuxuan.luo@canonical.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240412192351.89501-1-yuxuan.luo@canonical.com> References: <20240412192351.89501-1-yuxuan.luo@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: Pawan Gupta commit 7390db8aea0d64e9deb28b8e1ce716f5020c7ee5 upstream. Branch History Injection (BHI) attacks may allow a malicious application to influence indirect branch prediction in kernel by poisoning the branch history. eIBRS isolates indirect branch targets in ring0. The BHB can still influence the choice of indirect branch predictor entry, and although branch predictor entries are isolated between modes when eIBRS is enabled, the BHB itself is not isolated between modes. Alder Lake and new processors supports a hardware control BHI_DIS_S to mitigate BHI. For older processors Intel has released a software sequence to clear the branch history on parts that don't support BHI_DIS_S. Add support to execute the software sequence at syscall entry and VMexit to overwrite the branch history. For now, branch history is not cleared at interrupt entry, as malicious applications are not believed to have sufficient control over the registers, since previous register state is cleared at interrupt entry. Researchers continue to poke at this area and it may become necessary to clear at interrupt entry as well in the future. This mitigation is only defined here. It is enabled later. Signed-off-by: Pawan Gupta Co-developed-by: Daniel Sneddon Signed-off-by: Daniel Sneddon Signed-off-by: Thomas Gleixner Reviewed-by: Alexandre Chartre Reviewed-by: Josh Poimboeuf Signed-off-by: Daniel Sneddon Signed-off-by: Greg Kroah-Hartman (backported from commit bd53ec80f21839cfd4d852a6088279d602d67e5b stable/linux-5.15.y) [yuxuan.luo: manually backported the change for nospec-branch.h] CVE-2024-2201 Signed-off-by: Yuxuan Luo --- arch/x86/entry/entry_64.S | 61 ++++++++++++++++++++++++++++ arch/x86/entry/entry_64_compat.S | 3 ++ arch/x86/include/asm/cpufeatures.h | 8 ++++ arch/x86/include/asm/nospec-branch.h | 12 ++++++ arch/x86/kvm/vmx/vmenter.S | 2 + 5 files changed, 86 insertions(+) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 6082e281a8767..67c2f29b6f90d 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -114,6 +114,7 @@ SYM_INNER_LABEL(entry_SYSCALL_64_after_hwframe, SYM_L_GLOBAL) /* clobbers %rax, make sure it is after saving the syscall nr */ IBRS_ENTER UNTRAIN_RET + CLEAR_BRANCH_HISTORY call do_syscall_64 /* returns with IRQs disabled */ @@ -1498,3 +1499,63 @@ SYM_CODE_START(rewind_stack_and_make_dead) call make_task_dead SYM_CODE_END(rewind_stack_and_make_dead) .popsection + +/* + * This sequence executes branches in order to remove user branch information + * from the branch history tracker in the Branch Predictor, therefore removing + * user influence on subsequent BTB lookups. + * + * It should be used on parts prior to Alder Lake. Newer parts should use the + * BHI_DIS_S hardware control instead. If a pre-Alder Lake part is being + * virtualized on newer hardware the VMM should protect against BHI attacks by + * setting BHI_DIS_S for the guests. + * + * CALLs/RETs are necessary to prevent Loop Stream Detector(LSD) from engaging + * and not clearing the branch history. The call tree looks like: + * + * call 1 + * call 2 + * call 2 + * call 2 + * call 2 + * call 2 + * ret + * ret + * ret + * ret + * ret + * ret + * + * This means that the stack is non-constant and ORC can't unwind it with %rsp + * alone. Therefore we unconditionally set up the frame pointer, which allows + * ORC to unwind properly. + * + * The alignment is for performance and not for safety, and may be safely + * refactored in the future if needed. + */ +SYM_FUNC_START(clear_bhb_loop) + push %rbp + mov %rsp, %rbp + movl $5, %ecx + ANNOTATE_INTRA_FUNCTION_CALL + call 1f + jmp 5f + .align 64, 0xcc + ANNOTATE_INTRA_FUNCTION_CALL +1: call 2f + RET + .align 64, 0xcc +2: movl $5, %eax +3: jmp 4f + nop +4: sub $1, %eax + jnz 3b + sub $1, %ecx + jnz 1b + RET +5: lfence + pop %rbp + RET +SYM_FUNC_END(clear_bhb_loop) +EXPORT_SYMBOL_GPL(clear_bhb_loop) +STACK_FRAME_NON_STANDARD(clear_bhb_loop) diff --git a/arch/x86/entry/entry_64_compat.S b/arch/x86/entry/entry_64_compat.S index 4d637a965efbe..649e4b44f5ea9 100644 --- a/arch/x86/entry/entry_64_compat.S +++ b/arch/x86/entry/entry_64_compat.S @@ -116,6 +116,7 @@ SYM_INNER_LABEL(entry_SYSENTER_compat_after_hwframe, SYM_L_GLOBAL) IBRS_ENTER UNTRAIN_RET + CLEAR_BRANCH_HISTORY /* * SYSENTER doesn't filter flags, so we need to clear NT and AC @@ -259,6 +260,7 @@ SYM_INNER_LABEL(entry_SYSCALL_compat_after_hwframe, SYM_L_GLOBAL) IBRS_ENTER UNTRAIN_RET + CLEAR_BRANCH_HISTORY movq %rsp, %rdi call do_fast_syscall_32 @@ -421,6 +423,7 @@ SYM_CODE_START(entry_INT80_compat) IBRS_ENTER UNTRAIN_RET + CLEAR_BRANCH_HISTORY movq %rsp, %rdi call do_int80_syscall_32 diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 75d2dc704eb23..7c47e6aa9ea17 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -425,6 +425,14 @@ #define X86_FEATURE_IBPB_BRTYPE (20*32+28) /* "" MSR_PRED_CMD[IBPB] flushes all branch type predictions */ #define X86_FEATURE_SRSO_NO (20*32+29) /* "" CPU is not affected by SRSO */ +/* + * Extended auxiliary flags: Linux defined - for features scattered in various + * CPUID levels like 0x80000022, etc and Linux defined features. + * + * Reuse free bits when adding new feature flags! + */ +#define X86_FEATURE_CLEAR_BHB_LOOP (21*32+ 1) /* "" Clear branch history at syscall entry using SW loop */ + /* * BUG word(s) */ diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h index 940c15ee5650f..b1496b96ee529 100644 --- a/arch/x86/include/asm/nospec-branch.h +++ b/arch/x86/include/asm/nospec-branch.h @@ -182,6 +182,14 @@ #endif .endm +#ifdef CONFIG_X86_64 +.macro CLEAR_BRANCH_HISTORY + ALTERNATIVE "", "call clear_bhb_loop", X86_FEATURE_CLEAR_BHB_LOOP +.endm +#else +#define CLEAR_BRANCH_HISTORY +#endif + #else /* __ASSEMBLY__ */ #define ANNOTATE_RETPOLINE_SAFE \ @@ -207,6 +215,10 @@ extern void srso_alias_untrain_ret(void); extern void entry_untrain_ret(void); extern void entry_ibpb(void); +#ifdef CONFIG_X86_64 +extern void clear_bhb_loop(void); +#endif + #ifdef CONFIG_RETPOLINE typedef u8 retpoline_thunk_t[RETPOLINE_THUNK_SIZE]; diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S index 43cbf7e54dfff..5d8e1b01d6bf3 100644 --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -239,6 +239,8 @@ SYM_INNER_LABEL(vmx_vmexit, SYM_L_GLOBAL) call vmx_spec_ctrl_restore_host + CLEAR_BRANCH_HISTORY + /* Put return value in AX */ mov %_ASM_BX, %_ASM_AX