From patchwork Sun Feb 18 08:19:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Ruffell X-Patchwork-Id: 1900595 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=185.125.189.65; helo=lists.ubuntu.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=patchwork.ozlabs.org) Received: from lists.ubuntu.com (lists.ubuntu.com [185.125.189.65]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Tcz9n5TzQz1ybm for ; Sun, 18 Feb 2024 19:19:56 +1100 (AEDT) Received: from localhost ([127.0.0.1] helo=lists.ubuntu.com) by lists.ubuntu.com with esmtp (Exim 4.86_2) (envelope-from ) id 1rbcOl-0006ZJ-Da; Sun, 18 Feb 2024 08:19:43 +0000 Received: from smtp-relay-internal-1.internal ([10.131.114.114] helo=smtp-relay-internal-1.canonical.com) by lists.ubuntu.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1rbcOg-0006Z0-9K for kernel-team@lists.ubuntu.com; Sun, 18 Feb 2024 08:19:38 +0000 Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-1.canonical.com (Postfix) with ESMTPS id 624833F670 for ; Sun, 18 Feb 2024 08:19:36 +0000 (UTC) Received: by mail-pf1-f198.google.com with SMTP id d2e1a72fcca58-6e45e3e814aso59219b3a.1 for ; Sun, 18 Feb 2024 00:19:36 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1708244375; x=1708849175; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mdbCecj4bQ0BECk0I76qevxadbNPW5ObjtzD57bZqEY=; b=NiWAH8uVk+vfVRuJsX/nGGm4CULxh3uakKe80ULyuzU6AM45xVRPBzydqTPoXjo5hD mDo0g9X5zyXKJPV4ONKL09ffagD1V27Li7LTTAjPVJzWqybLkVieLeVCFuL5RFAvNVZF vq4hhk5aM159Y/gXJDUKv6G/TzGkPj6aYN6Zk7ouLVid4exs2fo/ykzqY4Sr1FoWr/Fg TpqWW9nMDRrD/Q6yLdC6RJFcK5LFZKGe1POL76E9+J4uzyrwEsS47RM/7Eo2/ZTEi0Mr uetwd5JtY83GvTWsruvPh3IbIo48tpVrBnRKcBybipii/5waoTAmjvHF2Xfut2kP/0RY fDXA== X-Gm-Message-State: AOJu0Yy6vItCYWpiVjtDU6ogIVJPnZZynkpkTHN5Gs4AGEgZkyq+28AJ C78KjHYTlPJgfDRF9o8QptlzI/vOLneeTwYDfl2y6B+q9sEmZ3UKsMsg8GQeCX2OCUuxfi1p67L pa1hpucll3rK1uJbF82TjG24SrWbEnesPRI4zuPHYAhyW0k1Ha8mw1TRWUOvp5E7qvm59PFOQ+u lTO8YZ9FUVTw== X-Received: by 2002:a05:6a20:94c6:b0:19e:8866:9e56 with SMTP id ht6-20020a056a2094c600b0019e88669e56mr18292847pzb.11.1708244374929; Sun, 18 Feb 2024 00:19:34 -0800 (PST) X-Google-Smtp-Source: AGHT+IEFWEJxJAUTboZEQKJAN9mXWJd78XdAZapDDCMwTxXOTiayOWXb5mHF588j1gujKY0ekPs8SQ== X-Received: by 2002:a05:6a20:94c6:b0:19e:8866:9e56 with SMTP id ht6-20020a056a2094c600b0019e88669e56mr18292837pzb.11.1708244374558; Sun, 18 Feb 2024 00:19:34 -0800 (PST) Received: from ThinkPad-X1.. (222-154-76-179-fibre.sparkbb.co.nz. [222.154.76.179]) by smtp.gmail.com with ESMTPSA id k5-20020a635605000000b005dc85821c80sm2529207pgb.12.2024.02.18.00.19.33 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Feb 2024 00:19:34 -0800 (PST) From: Matthew Ruffell To: kernel-team@lists.ubuntu.com Subject: [SRU][Mantic][PATCH 0/1] kvm: Running perf against qemu processes results in page fault inside guest Date: Sun, 18 Feb 2024 21:19:26 +1300 Message-Id: <20240218081927.23118-1-matthew.ruffell@canonical.com> X-Mailer: git-send-email 2.40.1 MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" BugLink: https://bugs.launchpad.net/bugs/2054218 [Impact] Running perf against a QEMU/kvm process results in the guest suffering a page fault in trying to store Precise Event Based Sampling (PEBS) records for the host. This affects both using perf against a single process, in which it crashes the targeted guest, or using perf system wide, in which it crashes all running guests on the system. The issue was introduced in 6.0 by: commit c59a1f106f5cd4843c097069ff1bb2ad72103a67 Author: Like Xu Date: Mon Apr 11 18:19:36 2022 +0800 Subject: KVM: x86/pmu: Add IA32_PEBS_ENABLE MSR emulation for extended PEBS Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c59a1f106f5cd4843c097069ff1bb2ad72103a67 This affects all 6.2 and 6.5 kernels. There is no known workaround, apart from not using perf on affected systems. [Fix] The issue was fixed in 6.7 by: commit 971079464001c6856186ca137778e534d983174a Author: Paolo Bonzini Date: Thu Jan 4 16:15:17 2024 +0100 Subject: KVM: x86/pmu: fix masking logic for MSR_CORE_PERF_GLOBAL_CTRL Link: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=971079464001c6856186ca137778e534d983174a This reinstates the logic for setting MSR_CORE_PERF_GLOBAL_CTRL to what it was before "KVM: x86/pmu: Add IA32_PEBS_ENABLE MSR emulation for extended PEBS". - .guest = intel_ctrl & (~cpuc->intel_ctrl_host_mask | ~pebs_mask), + .guest = intel_ctrl & ~cpuc->intel_ctrl_host_mask & ~pebs_mask, The faulty logic includes any bit that isn't both marked as exclude_guest and using PEBS, while it should really be excluding PEBS from the host. [Testcase] Start a bare metal server. Enable KVM, start a few VMs. The VMs can be idle, they don't require any workload. $ sudo apt-get install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils uvtool $ sudo reboot $ ssh-keygen $ uvt-simplestreams-libvirt sync --source http://cloud-images.ubuntu.com/daily release=jammy arch=amd64 $ uvt-kvm create --cpu 4 --memory 4096 --disk 10 jammy-a release=jammy arch=amd64 $ uvt-kvm create --cpu 4 --memory 4096 --disk 10 jammy-b release=jammy arch=amd64 $ uvt-kvm create --cpu 4 --memory 4096 --disk 10 jammy-c release=jammy arch=amd64 $ virsh list Id Name State ------------------------- 2 jammy-a running 3 jammy-b running 4 jammy-c running $ uvt-kvm ssh jammy-a Check it works. $ ps aux | grep qemu Find the pid of jammy-a $ perf top -p $PID $ virsh console jammy-a Escape character is ^] (Ctrl + ]) [ 357.793039] BUG: unable to handle page fault for address: fffffe49178c6028 $ uvt-kvm ssh jammy-a (no response) Test packages are available in the following ppa: https://launchpad.net/~mruffell/+archive/ubuntu/sf379502-test If you install it, then running perf against the PID of qemu processes will no longer crash the guest, and they will be accessible by SSH afterward. [Where problems could occur] We are rearranging the logic of setting the PEBS MSRs, which affects processor sampling of events. This will affect any profiling tools running against KVM based virtual machines, namely perf against QEMU. If a regression were to occur, running perf against a VM could cause it to page fault and subsequently crash, resulting in downtime. The only workaround will be to disable all profiling tools until a fix is available. Paolo Bonzini (1): KVM: x86/pmu: fix masking logic for MSR_CORE_PERF_GLOBAL_CTRL arch/x86/events/intel/core.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) Acked-by: Manuel Diewald Acked-by: Tim Gardner