From patchwork Tue Jan 26 13:20:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Henrique Cerri X-Patchwork-Id: 1431685 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4DQ6nh4rdFz9sWH; Wed, 27 Jan 2021 00:21:28 +1100 (AEDT) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1l4OHc-000063-Mw; Tue, 26 Jan 2021 13:21:24 +0000 Received: from youngberry.canonical.com ([91.189.89.112]) by huckleberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1l4OHT-0008TU-Na for kernel-team@lists.ubuntu.com; Tue, 26 Jan 2021 13:21:15 +0000 Received: from mail-qt1-f199.google.com ([209.85.160.199]) by youngberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1l4OHT-0005Gw-BZ for kernel-team@lists.ubuntu.com; Tue, 26 Jan 2021 13:21:15 +0000 Received: by mail-qt1-f199.google.com with SMTP id r18so5505390qta.19 for ; Tue, 26 Jan 2021 05:21:15 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TqjcxlF+c+f6cS3PEbNBNHJz5Oyx0vB4B0xuek8cQhs=; b=F/z7otP/XPmXXIshsZVnsThZ+nQbjXMTdyM5dIBr9eiGOeoITU7QkMA0oTxP6tR68K mi60Jlgg0CvX2ne/Yo20ocr+wOHVpHnuoxtSWcXi2PN7JCFE3ZdGNcxVRFUoND/cyYwx tEA8Vt6yMB89oi3Ol+gXl3EnddvEH/KOFnAzT889DMKdCEDG0w8P6FF6113ockD2ZG6X dPDcjv3mRT3ABpRzuAVkKF6ie0bHIFcIyZFar57sB9//Ceer9KChLcz9ThPRULM/onWp DvuVBaweoKyoBAMVrobOqfBd4Vv8jIpJa6sJnam8JxM2Hogc3fPpdoAm71ymwrh2Bt5D I/MQ== X-Gm-Message-State: AOAM533+/i0Rsnb5/Ujq5HbKK0kyIpYFT5SC7VnxE+uLv/bOs8Bv19+q Wuf9uqJibs7pW5102DHwnEOlLHn3J1DxCWv+SYCpYKKg0yxRMi3F0rZgo3OQJBRe55YCGmo1Uw1 ZCv7uXinbKrqzE3FVfPAe7yAbbYvsVmuBL6tUY0a+ X-Received: by 2002:a0c:a366:: with SMTP id u93mr5388071qvu.53.1611667274142; Tue, 26 Jan 2021 05:21:14 -0800 (PST) X-Google-Smtp-Source: ABdhPJxeUBJsrpEJBtBh52rAAwpbTD/S1VKG3bAALP5i5q12EEavL/sspje+TPsUPQbJkV46bx/leA== X-Received: by 2002:a0c:a366:: with SMTP id u93mr5388050qvu.53.1611667273848; Tue, 26 Jan 2021 05:21:13 -0800 (PST) Received: from localhost.localdomain ([2804:431:cfed:edc:c86c:ab75:eda1:1e6c]) by smtp.gmail.com with ESMTPSA id i65sm14397153qkf.105.2021.01.26.05.21.12 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Jan 2021 05:21:13 -0800 (PST) From: Marcelo Henrique Cerri To: kernel-team@lists.ubuntu.com Subject: [focal:linux-azure][PATCH 1/2] x86/process/64: Make save_fsgs_for_kvm() ready for FSGSBASE Date: Tue, 26 Jan 2021 10:20:55 -0300 Message-Id: <20210126132056.1767826-2-marcelo.cerri@canonical.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210126132056.1767826-1-marcelo.cerri@canonical.com> References: <20210126131712.1744754-1-marcelo.cerri@canonical.com> <20210126132056.1767826-1-marcelo.cerri@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: Thomas Gleixner BugLink: https://bugs.launchpad.net/bugs/1913294 save_fsgs_for_kvm() is invoked via vcpu_enter_guest() kvm_x86_ops.prepare_guest_switch(vcpu) vmx_prepare_switch_to_guest() save_fsgs_for_kvm() with preemption disabled, but interrupts enabled. The upcoming FSGSBASE based GS safe needs interrupts to be disabled. This could be done in the helper function, but that function is also called from switch_to() which has interrupts disabled already. Disable interrupts inside save_fsgs_for_kvm() and rename the function to current_save_fsgs() so it can be invoked from other places. Signed-off-by: Thomas Gleixner Signed-off-by: Sasha Levin Signed-off-by: Thomas Gleixner Link: https://lkml.kernel.org/r/20200528201402.1708239-7-sashal@kernel.org (cherry picked from commit 6758034e4d6a7f0e26b748789ab1f83f3116d1b9) Signed-off-by: Marcelo Henrique Cerri --- arch/x86/include/asm/processor.h | 4 +--- arch/x86/kernel/process_64.c | 15 +++++++++------ arch/x86/kvm/vmx/vmx.c | 2 +- 3 files changed, 11 insertions(+), 10 deletions(-) diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index a07dfdf7759e..68601ae4d6cf 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -406,10 +406,8 @@ static inline unsigned long cpu_kernelmode_gs_base(int cpu) DECLARE_PER_CPU(unsigned int, irq_count); extern asmlinkage void ignore_sysret(void); -#if IS_ENABLED(CONFIG_KVM) /* Save actual FS/GS selectors and bases to current->thread */ -void save_fsgs_for_kvm(void); -#endif +void current_save_fsgs(void); #else /* X86_64 */ #ifdef CONFIG_STACKPROTECTOR /* diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 1112b9dc6ee8..98145d14f04a 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -214,18 +214,21 @@ static __always_inline void save_fsgs(struct task_struct *task) } } -#if IS_ENABLED(CONFIG_KVM) /* * While a process is running,current->thread.fsbase and current->thread.gsbase - * may not match the corresponding CPU registers (see save_base_legacy()). KVM - * wants an efficient way to save and restore FSBASE and GSBASE. - * When FSGSBASE extensions are enabled, this will have to use RD{FS,GS}BASE. + * may not match the corresponding CPU registers (see save_base_legacy()). */ -void save_fsgs_for_kvm(void) +void current_save_fsgs(void) { + unsigned long flags; + + /* Interrupts need to be off for FSGSBASE */ + local_irq_save(flags); save_fsgs(current); + local_irq_restore(flags); } -EXPORT_SYMBOL_GPL(save_fsgs_for_kvm); +#if IS_ENABLED(CONFIG_KVM) +EXPORT_SYMBOL_GPL(current_save_fsgs); #endif static __always_inline void loadseg(enum which_selector which, diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 2a1ed3aae100..486e67b4e1e6 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -1151,7 +1151,7 @@ void vmx_prepare_switch_to_guest(struct kvm_vcpu *vcpu) gs_base = cpu_kernelmode_gs_base(cpu); if (likely(is_64bit_mm(current->mm))) { - save_fsgs_for_kvm(); + current_save_fsgs(); fs_sel = current->thread.fsindex; gs_sel = current->thread.gsindex; fs_base = current->thread.fsbase; From patchwork Tue Jan 26 13:20:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Henrique Cerri X-Patchwork-Id: 1431684 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4DQ6nc12hMz9sWD; Wed, 27 Jan 2021 00:21:24 +1100 (AEDT) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1l4OHY-0008Us-DU; Tue, 26 Jan 2021 13:21:20 +0000 Received: from youngberry.canonical.com ([91.189.89.112]) by huckleberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1l4OHV-0008UA-F1 for kernel-team@lists.ubuntu.com; Tue, 26 Jan 2021 13:21:17 +0000 Received: from mail-qv1-f70.google.com ([209.85.219.70]) by youngberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1l4OHV-0005H1-3s for kernel-team@lists.ubuntu.com; Tue, 26 Jan 2021 13:21:17 +0000 Received: by mail-qv1-f70.google.com with SMTP id cu12so11386930qvb.10 for ; Tue, 26 Jan 2021 05:21:17 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4egTrz13VENVckQZxJguuP1oS22ABl7JO1tE33BebIE=; b=jmYZQ1CyuNsWtRtZ21BEQEJCiCtXZrN1vh3qSAVEJWI8tvCKdt7Qudp0cVNV7xanzF qXQdjlU3FA+V3NbkvwUX9T24R1TI82n++8n2hlTJC7qgTcWuIsFF+H38M5d2oyqthF9g KBfpuNYJZ0Puoho82XTZ8SFRPTG6uVuLsU2hsTjRe1CP7dVFiffXli/JCef4zUN91k98 Nt8t4AFCJsfunkMepuuHn3I+KLlX+J2Zd0rfj7RmQjOPjWFtyasQWhtH4QvvcEEGQMD4 Ovqb5ymgguejXvXf/KxT2jcNwmweMuX4ApbTcx/OXEwFer423Pe5Neuank1feeonSllQ F1QA== X-Gm-Message-State: AOAM530179sJbJ4NiZyl40rjc4oX3OgvvCfddk9dw3+utHO2YDsoTO3T fTwfC69QUdntbpLg+pDxl8jnAX3FudGcbwD/QvcdQkLkUQdCNv7rGWS0H/D2iSaAsoaNh57tZqQ DbnXmyUdX7qmFs5ch91GU1vb9Y/jBs8xtmpPOm73L X-Received: by 2002:a37:618f:: with SMTP id v137mr5334456qkb.461.1611667275899; Tue, 26 Jan 2021 05:21:15 -0800 (PST) X-Google-Smtp-Source: ABdhPJwh8WBsESd3m2ahSv43lX27tTFNeOm6RVruFjTwPv2JiM1bt/SOck+oGetB/3Cc5AqXnKZEcQ== X-Received: by 2002:a37:618f:: with SMTP id v137mr5334428qkb.461.1611667275609; Tue, 26 Jan 2021 05:21:15 -0800 (PST) Received: from localhost.localdomain ([2804:431:cfed:edc:c86c:ab75:eda1:1e6c]) by smtp.gmail.com with ESMTPSA id i65sm14397153qkf.105.2021.01.26.05.21.14 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 26 Jan 2021 05:21:15 -0800 (PST) From: Marcelo Henrique Cerri To: kernel-team@lists.ubuntu.com Subject: [focal:linux-azure][PATCH 2/2] x86/entry/64: Do not use RDPID in paranoid entry to accomodate KVM Date: Tue, 26 Jan 2021 10:20:56 -0300 Message-Id: <20210126132056.1767826-3-marcelo.cerri@canonical.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210126132056.1767826-1-marcelo.cerri@canonical.com> References: <20210126131712.1744754-1-marcelo.cerri@canonical.com> <20210126132056.1767826-1-marcelo.cerri@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" From: Sean Christopherson BugLink: https://bugs.launchpad.net/bugs/1913294 KVM has an optmization to avoid expensive MRS read/writes on VMENTER/EXIT. It caches the MSR values and restores them either when leaving the run loop, on preemption or when going out to user space. The affected MSRs are not required for kernel context operations. This changed with the recently introduced mechanism to handle FSGSBASE in the paranoid entry code which has to retrieve the kernel GSBASE value by accessing per CPU memory. The mechanism needs to retrieve the CPU number and uses either LSL or RDPID if the processor supports it. Unfortunately RDPID uses MSR_TSC_AUX which is in the list of cached and lazily restored MSRs, which means between the point where the guest value is written and the point of restore, MSR_TSC_AUX contains a random number. If an NMI or any other exception which uses the paranoid entry path happens in such a context, then RDPID returns the random guest MSR_TSC_AUX value. As a consequence this reads from the wrong memory location to retrieve the kernel GSBASE value. Kernel GS is used to for all regular this_cpu_*() operations. If the GSBASE in the exception handler points to the per CPU memory of a different CPU then this has the obvious consequences of data corruption and crashes. As the paranoid entry path is the only place which accesses MSR_TSX_AUX (via RDPID) and the fallback via LSL is not significantly slower, remove the RDPID alternative from the entry path and always use LSL. The alternative would be to write MSR_TSC_AUX on every VMENTER and VMEXIT which would be inflicting massive overhead on that code path. [ tglx: Rewrote changelog ] Fixes: eaad981291ee3 ("x86/entry/64: Introduce the FIND_PERCPU_BASE macro") Reported-by: Tom Lendacky Debugged-by: Tom Lendacky Suggested-by: Andy Lutomirski Suggested-by: Peter Zijlstra Signed-off-by: Sean Christopherson Signed-off-by: Paolo Bonzini Signed-off-by: Thomas Gleixner Link: https://lore.kernel.org/r/20200821105229.18938-1-pbonzini@redhat.com (cherry picked from commit 6a3ea3e68b8a8a26c4aaac03432ed92269c9a14e) Signed-off-by: Marcelo Henrique Cerri --- arch/x86/entry/calling.h | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/arch/x86/entry/calling.h b/arch/x86/entry/calling.h index a993f9f01901..a25d1e20b5d6 100644 --- a/arch/x86/entry/calling.h +++ b/arch/x86/entry/calling.h @@ -371,12 +371,14 @@ For 32-bit we have the following conventions - kernel is built with * Fetch the per-CPU GS base value for this processor and put it in @reg. * We normally use %gs for accessing per-CPU data, but we are setting up * %gs here and obviously can not use %gs itself to access per-CPU data. + * + * Do not use RDPID, because KVM loads guest's TSC_AUX on vm-entry and + * may not restore the host's value until the CPU returns to userspace. + * Thus the kernel would consume a guest's TSC_AUX if an NMI arrives + * while running KVM's run loop. */ .macro GET_PERCPU_BASE reg:req - ALTERNATIVE \ - "LOAD_CPU_AND_NODE_SEG_LIMIT \reg", \ - "RDPID \reg", \ - X86_FEATURE_RDPID + LOAD_CPU_AND_NODE_SEG_LIMIT \reg andq $VDSO_CPUNODE_MASK, \reg movq __per_cpu_offset(, \reg, 8), \reg .endm