| Message ID | 20240209160039.677865-1-eric.auger@redhat.com |
|---|---|
| Headers | show |
| Series | ARM Nested Virt Support | expand |
On Fri, 9 Feb 2024 at 16:00, Eric Auger <eric.auger@redhat.com> wrote: > > This series adds ARM Nested Virtualization support in KVM mode. > This is a respin of previous contributions from Miguel [1] and Haibo [2]. > > This was tested with Marc's v11 [3] on Ampere HW with fedora L1 guest and > L2 guests booted without EDK2. However it does not work yet with > EDK2 but it looks unrelated to this qemu integration (host hard lockups). > > The host needs to be booted with "kvm-arm.mode=nested" option and > qemu needs to be invoked with : > > -machine virt,virtualization=on > > There is a known issue with hosts supporting SVE. Kernel does not support both > SVE and NV2 and the current qemu integration has an issue with the > scratch_host_vcpu startup because both are enabled if exposed by the kernel. > This is independent on whether sve is disabled on the command line. Unfortunately > I lost access to the HW that expose that issue so I couldn't fix it in this > version. You can probably repro that by running the whole setup under QEMU's FEAT_NV emulation, which will be able to give you a CPU with both FEAT_NV and SVE. Personally I think that this is a kernel missing-feature that should really be fixed as part of getting the kernel patches upstreamed. There's no cause to force every userspace VMM to develop extra complications for this. thanks -- PMM
Hi Peter, On 2/9/24 19:57, Peter Maydell wrote: > On Fri, 9 Feb 2024 at 16:00, Eric Auger <eric.auger@redhat.com> wrote: >> This series adds ARM Nested Virtualization support in KVM mode. >> This is a respin of previous contributions from Miguel [1] and Haibo [2]. >> >> This was tested with Marc's v11 [3] on Ampere HW with fedora L1 guest and >> L2 guests booted without EDK2. However it does not work yet with >> EDK2 but it looks unrelated to this qemu integration (host hard lockups). >> >> The host needs to be booted with "kvm-arm.mode=nested" option and >> qemu needs to be invoked with : >> >> -machine virt,virtualization=on >> >> There is a known issue with hosts supporting SVE. Kernel does not support both >> SVE and NV2 and the current qemu integration has an issue with the >> scratch_host_vcpu startup because both are enabled if exposed by the kernel. >> This is independent on whether sve is disabled on the command line. Unfortunately >> I lost access to the HW that expose that issue so I couldn't fix it in this >> version. > You can probably repro that by running the whole setup under > QEMU's FEAT_NV emulation, which will be able to give you a CPU > with both FEAT_NV and SVE. indeed, this should work now we have FEAT_NV. > > Personally I think that this is a kernel missing-feature that > should really be fixed as part of getting the kernel patches > upstreamed. There's no cause to force every userspace VMM to > develop extra complications for this. yes maybe this will be fixed later on. Thanks! Eric > > thanks > -- PMM >
On 2024-02-09 18:57, Peter Maydell wrote: > On Fri, 9 Feb 2024 at 16:00, Eric Auger <eric.auger@redhat.com> wrote: >> >> This series adds ARM Nested Virtualization support in KVM mode. >> This is a respin of previous contributions from Miguel [1] and Haibo >> [2]. >> >> This was tested with Marc's v11 [3] on Ampere HW with fedora L1 guest >> and >> L2 guests booted without EDK2. However it does not work yet with >> EDK2 but it looks unrelated to this qemu integration (host hard >> lockups). >> >> The host needs to be booted with "kvm-arm.mode=nested" option and >> qemu needs to be invoked with : >> >> -machine virt,virtualization=on >> >> There is a known issue with hosts supporting SVE. Kernel does not >> support both >> SVE and NV2 and the current qemu integration has an issue with the >> scratch_host_vcpu startup because both are enabled if exposed by the >> kernel. >> This is independent on whether sve is disabled on the command line. >> Unfortunately >> I lost access to the HW that expose that issue so I couldn't fix it in >> this >> version. > > You can probably repro that by running the whole setup under > QEMU's FEAT_NV emulation, which will be able to give you a CPU > with both FEAT_NV and SVE. > > Personally I think that this is a kernel missing-feature that > should really be fixed as part of getting the kernel patches > upstreamed. There's no cause to force every userspace VMM to > develop extra complications for this. I don't plan to make NV visible to userspace before this is fixed. Which may delay KVM NV by another year or five, but I don't think anyone is really waiting for it anyway. M.
On Fri, 9 Feb 2024 at 16:00, Eric Auger <eric.auger@redhat.com> wrote: > > This series adds ARM Nested Virtualization support in KVM mode. > This is a respin of previous contributions from Miguel [1] and Haibo [2]. > > This was tested with Marc's v11 [3] on Ampere HW with fedora L1 guest and > L2 guests booted without EDK2. However it does not work yet with > EDK2 but it looks unrelated to this qemu integration (host hard lockups). > > The host needs to be booted with "kvm-arm.mode=nested" option and > qemu needs to be invoked with : > > -machine virt,virtualization=on > > There is a known issue with hosts supporting SVE. Kernel does not support both > SVE and NV2 and the current qemu integration has an issue with the > scratch_host_vcpu startup because both are enabled if exposed by the kernel. > This is independent on whether sve is disabled on the command line. Unfortunately > I lost access to the HW that expose that issue so I couldn't fix it in this > version. > > This series can be found at: > https://github.com/eauger/qemu/tree/v8.2-nv-rfcv2 > > Previous version from Miguel: > [1] https://lore.kernel.org/all/20230227163718.62003-1-miguel.luis@oracle.com/ > Previous version from Haibo: > [2] https://lore.kernel.org/qemu-devel/cover.1617281290.git.haibo.xu@linaro.org/ > [3] Marc's kernel v11 series: > [PATCH v11 00/43] KVM: arm64: Nested Virtualization support (FEAT_NV2 only) > https://lore.kernel.org/linux-arm-kernel/20231120131027.854038-1-maz@kernel.org/T/ > available at: https://github.com/eauger/linux/tree/nv-6.8-nv2-v11 > > Haibo Xu (5): > [Placeholder] headers: Partial headers update for NV2 enablement > hw/arm: Allow setting KVM vGIC maintenance IRQ > target/arm/kvm: Add helper to detect EL2 when using KVM > target/arm: Enable feature ARM_FEATURE_EL2 if EL2 is supported > hw/arm/virt: Allow virt extensions with KVM > > hw/arm/virt.c | 6 +++++- > hw/intc/arm_gicv3_common.c | 1 + > hw/intc/arm_gicv3_kvm.c | 21 +++++++++++++++++++++ > include/hw/intc/arm_gicv3_common.h | 1 + > linux-headers/asm-arm64/kvm.h | 1 + > linux-headers/linux/kvm.h | 1 + > target/arm/kvm.c | 21 +++++++++++++++++++++ > target/arm/kvm_arm.h | 12 ++++++++++++ > 8 files changed, 63 insertions(+), 1 deletion(-) All the patches in this series seem reasonable, but the series as a whole is so short I wonder if we're missing something :-) Does migration Just Work? (I guess as long as the kernel exposes all the EL2 sysregs via the ONE_REG ioctl interface it ought to...) Anyway, I don't think there's anything that stood out as needing major changes, so for now I guess we just wait for whenever the KVM side patches eventually land. thanks -- PMM
Hi Peter, On 3/5/24 17:57, Peter Maydell wrote: > On Fri, 9 Feb 2024 at 16:00, Eric Auger <eric.auger@redhat.com> wrote: >> This series adds ARM Nested Virtualization support in KVM mode. >> This is a respin of previous contributions from Miguel [1] and Haibo [2]. >> >> This was tested with Marc's v11 [3] on Ampere HW with fedora L1 guest and >> L2 guests booted without EDK2. However it does not work yet with >> EDK2 but it looks unrelated to this qemu integration (host hard lockups). >> >> The host needs to be booted with "kvm-arm.mode=nested" option and >> qemu needs to be invoked with : >> >> -machine virt,virtualization=on >> >> There is a known issue with hosts supporting SVE. Kernel does not support both >> SVE and NV2 and the current qemu integration has an issue with the >> scratch_host_vcpu startup because both are enabled if exposed by the kernel. >> This is independent on whether sve is disabled on the command line. Unfortunately >> I lost access to the HW that expose that issue so I couldn't fix it in this >> version. >> >> This series can be found at: >> https://github.com/eauger/qemu/tree/v8.2-nv-rfcv2 >> >> Previous version from Miguel: >> [1] https://lore.kernel.org/all/20230227163718.62003-1-miguel.luis@oracle.com/ >> Previous version from Haibo: >> [2] https://lore.kernel.org/qemu-devel/cover.1617281290.git.haibo.xu@linaro.org/ >> [3] Marc's kernel v11 series: >> [PATCH v11 00/43] KVM: arm64: Nested Virtualization support (FEAT_NV2 only) >> https://lore.kernel.org/linux-arm-kernel/20231120131027.854038-1-maz@kernel.org/T/ >> available at: https://github.com/eauger/linux/tree/nv-6.8-nv2-v11 >> >> Haibo Xu (5): >> [Placeholder] headers: Partial headers update for NV2 enablement >> hw/arm: Allow setting KVM vGIC maintenance IRQ >> target/arm/kvm: Add helper to detect EL2 when using KVM >> target/arm: Enable feature ARM_FEATURE_EL2 if EL2 is supported >> hw/arm/virt: Allow virt extensions with KVM >> >> hw/arm/virt.c | 6 +++++- >> hw/intc/arm_gicv3_common.c | 1 + >> hw/intc/arm_gicv3_kvm.c | 21 +++++++++++++++++++++ >> include/hw/intc/arm_gicv3_common.h | 1 + >> linux-headers/asm-arm64/kvm.h | 1 + >> linux-headers/linux/kvm.h | 1 + >> target/arm/kvm.c | 21 +++++++++++++++++++++ >> target/arm/kvm_arm.h | 12 ++++++++++++ >> 8 files changed, 63 insertions(+), 1 deletion(-) > All the patches in this series seem reasonable, but the series > as a whole is so short I wonder if we're missing something :-) > Does migration Just Work? (I guess as long as the kernel exposes > all the EL2 sysregs via the ONE_REG ioctl interface it ought to...) To be honest I have not tested yet. I would also guess there is no mig blocker but that definitively deserves some testing. I will check before sending the next version. > > Anyway, I don't think there's anything that stood out as needing > major changes, so for now I guess we just wait for whenever the > KVM side patches eventually land. yup Thanks Eric > > thanks > -- PMM >
This series adds ARM Nested Virtualization support in KVM mode. This is a respin of previous contributions from Miguel [1] and Haibo [2]. This was tested with Marc's v11 [3] on Ampere HW with fedora L1 guest and L2 guests booted without EDK2. However it does not work yet with EDK2 but it looks unrelated to this qemu integration (host hard lockups). The host needs to be booted with "kvm-arm.mode=nested" option and qemu needs to be invoked with : -machine virt,virtualization=on There is a known issue with hosts supporting SVE. Kernel does not support both SVE and NV2 and the current qemu integration has an issue with the scratch_host_vcpu startup because both are enabled if exposed by the kernel. This is independent on whether sve is disabled on the command line. Unfortunately I lost access to the HW that expose that issue so I couldn't fix it in this version. This series can be found at: https://github.com/eauger/qemu/tree/v8.2-nv-rfcv2 Previous version from Miguel: [1] https://lore.kernel.org/all/20230227163718.62003-1-miguel.luis@oracle.com/ Previous version from Haibo: [2] https://lore.kernel.org/qemu-devel/cover.1617281290.git.haibo.xu@linaro.org/ [3] Marc's kernel v11 series: [PATCH v11 00/43] KVM: arm64: Nested Virtualization support (FEAT_NV2 only) https://lore.kernel.org/linux-arm-kernel/20231120131027.854038-1-maz@kernel.org/T/ available at: https://github.com/eauger/linux/tree/nv-6.8-nv2-v11 Haibo Xu (5): [Placeholder] headers: Partial headers update for NV2 enablement hw/arm: Allow setting KVM vGIC maintenance IRQ target/arm/kvm: Add helper to detect EL2 when using KVM target/arm: Enable feature ARM_FEATURE_EL2 if EL2 is supported hw/arm/virt: Allow virt extensions with KVM hw/arm/virt.c | 6 +++++- hw/intc/arm_gicv3_common.c | 1 + hw/intc/arm_gicv3_kvm.c | 21 +++++++++++++++++++++ include/hw/intc/arm_gicv3_common.h | 1 + linux-headers/asm-arm64/kvm.h | 1 + linux-headers/linux/kvm.h | 1 + target/arm/kvm.c | 21 +++++++++++++++++++++ target/arm/kvm_arm.h | 12 ++++++++++++ 8 files changed, 63 insertions(+), 1 deletion(-)