From patchwork Wed Feb 23 04:18:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 1596472 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=m5lClGvQ; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20210112 header.b=l+cbCeQK; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:e::133; helo=bombadil.infradead.org; envelope-from=kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=) Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:e::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4K3N8Y0GYTz9sG2 for ; Wed, 23 Feb 2022 15:19:09 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:Mime-Version: Message-Id:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Owner; bh=ltXKGeqJ+LLdHlUGiJDcO6qyyYAinex2AyQ0pj3g+hg=; b=m5l ClGvQTt21fD8dUQLRgdVgg8m9UBrVXzUsZrExsutyiNstgAkDsHKq3Mm28rdWakKrNkuaYV0RJ083 QM31svVgR8WWC4e3QJn+9vRMZaBR+q9W41rMf6fJ9isIYCqzrwrpqaPpdklWZXVZJ7oOy2majQteF NzH5w3rW/QTzfWQpF2a5HXVAzlRnHvBhcpcKw5jbac6zyzv2SRoOWg5jvp5d7RBhZuHOFd31UaJgd kOwdnIBr0VsYcOlZrJOSHVHPOjUiyS9GDiMaK4irv6JiI704iwAZc2r7kCEClx8X3iRI0j6owSEeo KGTZTKMGzInVe3PI5mbZiRcAE8uXZKg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nMj7K-00CmDu-Gp; Wed, 23 Feb 2022 04:19:06 +0000 Received: from mail-io1-xd4a.google.com ([2607:f8b0:4864:20::d4a]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nMj7H-00CmDO-N8 for kvm-riscv@lists.infradead.org; Wed, 23 Feb 2022 04:19:05 +0000 Received: by mail-io1-xd4a.google.com with SMTP id e23-20020a6b6917000000b006406b9433d6so11164004ioc.14 for ; Tue, 22 Feb 2022 20:19:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:message-id:mime-version:subject:from:to:cc; bh=rh/Pz7GShqLz5FQCLq9qwfWYqDB5jRhpe8UhyldNSc4=; b=l+cbCeQKyO8uknj64WCHlw/fpPURvdhvPLS4dVuTMTc8RIkeSAFNotw+gq/vQspKG6 fye1+QK2f/bGEG1dRabgbbnOVIs/yd+eKVtM+S4VoceT1CcEwLFKx2QLyEACBc/vUGS4 CzzVu2nFr1UOaIwl4XKOhZrshMWsuInlN/2Da24wc5oxkMZNdI5Jnx6kJJlmOW4W7o6Z bp0ZLHNuAWACEd2dC7wy4IWg2m7vE2GjrJCM4z9oJ4YTLNU3PVuxV32+Arf0E75eY8fj idTs/G/Mc6FbnyluIfqOc46IT8Dj4GUw5kthBZr3fPHJ3s+4LtdMhurUqDTxb34z9NGy qwMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=rh/Pz7GShqLz5FQCLq9qwfWYqDB5jRhpe8UhyldNSc4=; b=at7iPg9bRWm5j+WU22kcXq9EBYGKuk/15uBKcQ4C6aPFJaflLLSMrCf/vviYoAm8aB 3w+NxXOcy+AZFWdy/7HdZQYGGBgc9T3tlZgfhVBJm+03Eez+FhLU3JpKITe3Uun2seka +Sk2pITC7/lDvFotGDL6jhfIutMsV2kA5CaFMkLRdX73vxs5Mk/At6aD2v1XFESNzJAF PMoTu/AMmuMdtX7ODNPfadn/0o7sYrO67tcneRYQ77sXPR5HAtzUo0EwKtqlAiazEfbW trTaJP6mn72PdNI3TxDXyC5gr0BlkQxMu5Hs3xzlZm+KORhtxDQhPrJqC2RM5T0RpS2/ j8iQ== X-Gm-Message-State: AOAM532/wEChu0BG+peh1i+e6x3/ih9MkfPe8WY4HKmswiv13emNdpWV CUVWVxFnXQDskH615WgphWBkjCTr41k= X-Google-Smtp-Source: ABdhPJy1y64EgXE0D245F1bV1sdlGSK5971KnecjSD+vgnQFtsqE52fORA3Oa3hV4wWYMKsTBvpS1ErwXMk= X-Received: from oupton.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:404]) (user=oupton job=sendgmr) by 2002:a6b:205:0:b0:638:57c4:d51 with SMTP id 5-20020a6b0205000000b0063857c40d51mr21682341ioc.66.1645589941479; Tue, 22 Feb 2022 20:19:01 -0800 (PST) Date: Wed, 23 Feb 2022 04:18:25 +0000 Message-Id: <20220223041844.3984439-1-oupton@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.35.1.473.g83b2b277ed-goog Subject: [PATCH v3 00/19] KVM: arm64: Implement PSCI SYSTEM_SUSPEND From: Oliver Upton To: kvmarm@lists.cs.columbia.edu Cc: Paolo Bonzini , Marc Zyngier , James Morse , Alexandru Elisei , Suzuki K Poulose , Anup Patel , Atish Patra , Sean Christopherson , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, Peter Shier , Reiji Watanabe , Ricardo Koller , Raghavendra Rao Ananta , Jing Zhang , Oliver Upton X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220222_201903_795569_D10C7C61 X-CRM114-Status: GOOD ( 25.14 ) X-Spam-Score: -7.7 (-------) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: The PSCI v1.0 specification describes a call, SYSTEM_SUSPEND, which allows software to request that the system be placed into the lowest possible power state and await an IMPLEMENTATION DEFINED wakeup [...] Content analysis details: (-7.7 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:d4a listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.0 DKIMWL_WL_MED DKIMwl.org - Medium trust sender X-BeenThere: kvm-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "kvm-riscv" Errors-To: kvm-riscv-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org The PSCI v1.0 specification describes a call, SYSTEM_SUSPEND, which allows software to request that the system be placed into the lowest possible power state and await an IMPLEMENTATION DEFINED wakeup event. This call is optional in v1.0 and v1.1. KVM does not currently support it in the v1.0 implementation. This series adds support for the PSCI SYSTEM_SUSPEND call to KVM/arm64. By default, KVM will treat the call as equivalent to CPU_SUSPEND, wherein KVM handles the call as a guest WFI. However, this series also introduces an opt-in for the SYSTEM_SUSPEND call to exit to userspace. VMMs may use the event as a hint to save the VM to resume at a later time, freeing up system resources. Userspace can decide at the time of the exit whether or not to honor the SYSTEM_SUSPEND call. Patch 1 is a small cleanup already present in kvmarm/next, but the series depends on it so it has been included to guarantee the series builds. Patches 2-3 adds an additional check to the CPU_ON PSCI call. As Reiji noted, PSCI implementations can return INVALID_ADDRESS if it is determined that the provided entry address does not exist in the guest address space. Patch 4 is another small cleanup to generically filter SMC64 calls when running an AArch32 EL1, avoiding the need to add a special case for the new PSCI call introduced in this series. Patches 5-6 add support for tracking a vCPU's power state using KVM_MP_STATE_* values. This is significant as the series introduces an additional power state, which cannot be represented by the `vcpu->arch.power_off` boolean. Patch 7 is a nitpick regarding the naming of a KVM_REQ_ handler. Patches 8-9 provide the default implementation of PSCI SYSTEM_SUSPEND by synchronously resetting the calling vCPU and entering WFI. Patches 10-12 introduce a new MP state, KVM_MP_STATE_SUSPENDED, which implements 'sticky' suspension. If userspace puts a vCPU in this state, it will exit to userspace for every recognized wakeup event (pending interrupt). When userspace is satisfied that a VM should resume, it must explicitly unpark the vCPU by marking it runnable again. This is useful for userspace to implement PSCI SYSTEM_SUSPEND if it decides to trap the call. Patch 13 extends upon the implementation of PSCI SYSTEM_SUSPEND, granting userspace the opt-in capability of exiting to userspace on such a call. *NOTE* KVM_SYSTEM_EVENT_SUSPEND breaks away from the semantics of other system events. Userspace is required to manipulate the vCPU to either reset it or reject the call. Other PSCI calls that exit set an SMCCC return value before exiting, but doing so would clobber all of the pending reset state. I wanted to avoid adding additional API to convey the reset context to userspace so it may simply be expressed in the architected state. Patch 14 increments the reported PSCI version to 1.1, as KVM already meets the requirements. Patches 15-18 rework the PSCI selftest to make it amenable to additional test cases Lastly, patch 19 tests that the KVM_SYSTEM_EVENT_SUSPEND exits are working as intended, and that KVM rejects invalid calls to PSCI SYSTEM_SUSPEND. This series applies cleanly to v5.17-rc5. Testing was performed with the included selftest and suspending a QEMU guest (i.e. no system event exits) on an Ampere Altra machine. v2: https://patchwork.kernel.org/project/kvm/cover/20210923191610.3814698-1-oupton@google.com/ v2 -> v3: - rebase to 5.17-rc5 - Reject CPU_ON and SYSTEM_SUSPEND calls that provide an invalid IPA (Reiji) - do *not* defer WFI as a requested event (Marc) - Add support for userspace filtering of wakeup events if SUSPEND exits are enabled (Marc) - Bump the reported PSCI verision to v1.1 (Marc) Oliver Upton (19): KVM: arm64: Drop unused param from kvm_psci_version() KVM: arm64: Create a helper to check if IPA is valid KVM: arm64: Reject invalid addresses for CPU_ON PSCI call KVM: arm64: Clean up SMC64 PSCI filtering for AArch32 guests KVM: arm64: Dedupe vCPU power off helpers KVM: arm64: Track vCPU power state using MP state values KVM: arm64: Rename the KVM_REQ_SLEEP handler KVM: arm64: Add reset helper that accepts caller-provided reset state KVM: arm64: Implement PSCI SYSTEM_SUSPEND KVM: Create helper for setting a system event exit KVM: arm64: Return a value from check_vcpu_requests() KVM: arm64: Add support for userspace to suspend a vCPU KVM: arm64: Add support KVM_SYSTEM_EVENT_SUSPEND to PSCI SYSTEM_SUSPEND KVM: arm64: Raise default PSCI version to v1.1 selftests: KVM: Rename psci_cpu_on_test to psci_test selftests: KVM: Create helper for making SMCCC calls selftests: KVM: Use KVM_SET_MP_STATE to power off vCPU in psci_test selftests: KVM: Refactor psci_test to make it amenable to new tests selftests: KVM: Test SYSTEM_SUSPEND PSCI call Documentation/virt/kvm/api.rst | 62 ++++- arch/arm64/include/asm/kvm_host.h | 27 ++- arch/arm64/include/asm/kvm_mmu.h | 9 + arch/arm64/kvm/arm.c | 88 +++++-- arch/arm64/kvm/psci.c | 129 ++++++++--- arch/arm64/kvm/reset.c | 45 ++-- arch/arm64/kvm/vgic/vgic-kvm-device.c | 2 +- arch/riscv/kvm/vcpu_sbi_v01.c | 4 +- arch/x86/kvm/x86.c | 6 +- include/kvm/arm_psci.h | 9 +- include/linux/kvm_host.h | 7 + include/uapi/linux/kvm.h | 4 + tools/testing/selftests/kvm/.gitignore | 2 +- tools/testing/selftests/kvm/Makefile | 2 +- .../selftests/kvm/aarch64/psci_cpu_on_test.c | 121 ---------- .../testing/selftests/kvm/aarch64/psci_test.c | 218 ++++++++++++++++++ .../selftests/kvm/include/aarch64/processor.h | 22 ++ .../selftests/kvm/lib/aarch64/processor.c | 25 ++ tools/testing/selftests/kvm/steal_time.c | 13 +- 19 files changed, 571 insertions(+), 224 deletions(-) delete mode 100644 tools/testing/selftests/kvm/aarch64/psci_cpu_on_test.c create mode 100644 tools/testing/selftests/kvm/aarch64/psci_test.c