Message ID | 20200805175700.62775-1-maz@kernel.org |
---|---|
State | New |
Headers | show |
Series | [GIT,PULL] KVM/arm64 updates for 5.9 | expand |
On 05/08/20 19:56, Marc Zyngier wrote: > Hi Paolo, > > This is the bulk of the 5.9 patches for KVM/arm64. It is a pretty busy > merge window for us this time, thanks to the ongoing Protected KVM > work. We have changes all over the map, but the most important piece > probably is the way we now build the EL2 code on non-VHE systems. On > top of giving us better control over what gets pulled in there, it > allowed us to enable instrumentation on VHE systems. > > The rest is a mixed bag of new features (TTL TLB invalidation, Pointer > Auth on non-VHE), preliminary patches for NV, some early MMU rework > before the 5.10 onslaught, and tons of cleanups. > > A few things to notice: > > - We share a branch with the arm64 tree, which has gone in already. > > - There are a number of known conflicts with Sean's MMU cache rework, > as well as the late fixes that went in 5.8. The conflicts are pretty > simple to resolve, and -next has the right resolutions already. Ok, since I have already an x86 conflict I'll wait to pull this until next week. Paolo
On 05/08/20 19:56, Marc Zyngier wrote:
> git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git tags/kvmarm-5.9
Pulled, thanks.
Paolo
Hi Marc, -----Original Message----- From: kvmarm-bounces@lists.cs.columbia.edu <kvmarm-bounces@lists.cs.columbia.edu> On Behalf Of Marc Zyngier Sent: Thursday, August 6, 2020 1:57 AM To: Paolo Bonzini <pbonzini@redhat.com> Cc: Peng Hao <richard.peng@oppo.com>; kernel-team@android.com; kvm@vger.kernel.org; Will Deacon <will@kernel.org>; Catalin Marinas <Catalin.Marinas@arm.com>; Alexander Graf <graf@amazon.com>; kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org Subject: [PATCH 47/56] KVM: arm64: timers: Move timer registers to the sys_regs file Move the timer gsisters to the sysreg file. This will further help when they are directly changed by a nesting hypervisor in the VNCR page. This requires moving the initialisation of the timer struct so that some of the helpers (such as arch_timer_ctx_index) can work correctly at an early stage. Signed-off-by: Marc Zyngier <maz@kernel.org> --- arch/arm64/include/asm/kvm_host.h | 6 ++ arch/arm64/kvm/arch_timer.c | 155 +++++++++++++++++++++++------- arch/arm64/kvm/trace_arm.h | 8 +- include/kvm/arm_arch_timer.h | 11 +-- 4 files changed, 136 insertions(+), 44 deletions(-) +static u64 timer_get_offset(struct arch_timer_context *ctxt) { + struct kvm_vcpu *vcpu = ctxt->vcpu; + + switch(arch_timer_ctx_index(ctxt)) { + case TIMER_VTIMER: + return __vcpu_sys_reg(vcpu, CNTVOFF_EL2); + default: + return 0; + } +} + Can I export this helper? As in my ptp_kvm implementation I need get VCNT offset value separately not just give me a result of VCNT. Thanks Jianyong
On 2020-08-19 10:24, Jianyong Wu wrote: > Hi Marc, > > -----Original Message----- > From: kvmarm-bounces@lists.cs.columbia.edu > <kvmarm-bounces@lists.cs.columbia.edu> On Behalf Of Marc Zyngier > Sent: Thursday, August 6, 2020 1:57 AM > To: Paolo Bonzini <pbonzini@redhat.com> > Cc: Peng Hao <richard.peng@oppo.com>; kernel-team@android.com; > kvm@vger.kernel.org; Will Deacon <will@kernel.org>; Catalin Marinas > <Catalin.Marinas@arm.com>; Alexander Graf <graf@amazon.com>; > kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org > Subject: [PATCH 47/56] KVM: arm64: timers: Move timer registers to the > sys_regs file > > Move the timer gsisters to the sysreg file. This will further help > when they are directly changed by a nesting hypervisor in the VNCR > page. > > This requires moving the initialisation of the timer struct so that > some of the helpers (such as arch_timer_ctx_index) can work correctly > at an early stage. > > Signed-off-by: Marc Zyngier <maz@kernel.org> > --- > arch/arm64/include/asm/kvm_host.h | 6 ++ > arch/arm64/kvm/arch_timer.c | 155 +++++++++++++++++++++++------- > arch/arm64/kvm/trace_arm.h | 8 +- > include/kvm/arm_arch_timer.h | 11 +-- > 4 files changed, 136 insertions(+), 44 deletions(-) > > +static u64 timer_get_offset(struct arch_timer_context *ctxt) { > + struct kvm_vcpu *vcpu = ctxt->vcpu; > + > + switch(arch_timer_ctx_index(ctxt)) { > + case TIMER_VTIMER: > + return __vcpu_sys_reg(vcpu, CNTVOFF_EL2); > + default: > + return 0; > + } > +} > + > Can I export this helper? As in my ptp_kvm implementation I need get > VCNT offset value separately not just give me a result of VCNT. Sorry, you need to give me a bit more context. What do you need the offset for exactly? M.
> -----Original Message----- > From: Marc Zyngier <maz@kernel.org> > Sent: Wednesday, August 19, 2020 6:00 PM > To: Jianyong Wu <Jianyong.Wu@arm.com> > Cc: Paolo Bonzini <pbonzini@redhat.com>; Peng Hao > <richard.peng@oppo.com>; kernel-team@android.com; > kvm@vger.kernel.org; Will Deacon <will@kernel.org>; Catalin Marinas > <Catalin.Marinas@arm.com>; Alexander Graf <graf@amazon.com>; > kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org > Subject: Re: [PATCH 47/56] KVM: arm64: timers: Move timer registers to the > sys_regs file > > On 2020-08-19 10:24, Jianyong Wu wrote: > > Hi Marc, > > > > -----Original Message----- > > From: kvmarm-bounces@lists.cs.columbia.edu > > <kvmarm-bounces@lists.cs.columbia.edu> On Behalf Of Marc Zyngier > > Sent: Thursday, August 6, 2020 1:57 AM > > To: Paolo Bonzini <pbonzini@redhat.com> > > Cc: Peng Hao <richard.peng@oppo.com>; kernel-team@android.com; > > kvm@vger.kernel.org; Will Deacon <will@kernel.org>; Catalin Marinas > > <Catalin.Marinas@arm.com>; Alexander Graf <graf@amazon.com>; > > kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org > > Subject: [PATCH 47/56] KVM: arm64: timers: Move timer registers to the > > sys_regs file > > > > Move the timer gsisters to the sysreg file. This will further help > > when they are directly changed by a nesting hypervisor in the VNCR > > page. > > > > This requires moving the initialisation of the timer struct so that > > some of the helpers (such as arch_timer_ctx_index) can work correctly > > at an early stage. > > > > Signed-off-by: Marc Zyngier <maz@kernel.org> > > --- > > arch/arm64/include/asm/kvm_host.h | 6 ++ > > arch/arm64/kvm/arch_timer.c | 155 +++++++++++++++++++++++------- > > arch/arm64/kvm/trace_arm.h | 8 +- > > include/kvm/arm_arch_timer.h | 11 +-- > > 4 files changed, 136 insertions(+), 44 deletions(-) > > > > +static u64 timer_get_offset(struct arch_timer_context *ctxt) { > > + struct kvm_vcpu *vcpu = ctxt->vcpu; > > + > > + switch(arch_timer_ctx_index(ctxt)) { > > + case TIMER_VTIMER: > > + return __vcpu_sys_reg(vcpu, CNTVOFF_EL2); > > + default: > > + return 0; > > + } > > +} > > + > > Can I export this helper? As in my ptp_kvm implementation I need get > > VCNT offset value separately not just give me a result of VCNT. > > Sorry, you need to give me a bit more context. What do you need the offset > for exactly? Yeah, In my ptp_kvm implementation, I need acquire wall time and counter cycle in the same time in host. After get host counter cycle, I need subtract it by VCNT offset to obtain VCNT. See https://lkml.org/lkml/2020/6/19/441 https://lkml.org/lkml/2020/6/19/441 But now I can't get the VCNT offset easily like before using " vcpu_vtimer(vcpu)->cntvoff" and I can't use the helper like "kvm_arm_timer_read" as I need acquire the counter cycle in the same time with the host wall time. Thanks Jianyong > > M. > -- > Jazz is not dead. It just smells funny...
On 2020-08-19 11:18, Jianyong Wu wrote: >> -----Original Message----- >> From: Marc Zyngier <maz@kernel.org> >> Sent: Wednesday, August 19, 2020 6:00 PM >> To: Jianyong Wu <Jianyong.Wu@arm.com> >> Cc: Paolo Bonzini <pbonzini@redhat.com>; Peng Hao >> <richard.peng@oppo.com>; kernel-team@android.com; >> kvm@vger.kernel.org; Will Deacon <will@kernel.org>; Catalin Marinas >> <Catalin.Marinas@arm.com>; Alexander Graf <graf@amazon.com>; >> kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org >> Subject: Re: [PATCH 47/56] KVM: arm64: timers: Move timer registers to >> the >> sys_regs file >> >> On 2020-08-19 10:24, Jianyong Wu wrote: >> > Hi Marc, >> > >> > -----Original Message----- >> > From: kvmarm-bounces@lists.cs.columbia.edu >> > <kvmarm-bounces@lists.cs.columbia.edu> On Behalf Of Marc Zyngier >> > Sent: Thursday, August 6, 2020 1:57 AM >> > To: Paolo Bonzini <pbonzini@redhat.com> >> > Cc: Peng Hao <richard.peng@oppo.com>; kernel-team@android.com; >> > kvm@vger.kernel.org; Will Deacon <will@kernel.org>; Catalin Marinas >> > <Catalin.Marinas@arm.com>; Alexander Graf <graf@amazon.com>; >> > kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org >> > Subject: [PATCH 47/56] KVM: arm64: timers: Move timer registers to the >> > sys_regs file >> > >> > Move the timer gsisters to the sysreg file. This will further help >> > when they are directly changed by a nesting hypervisor in the VNCR >> > page. >> > >> > This requires moving the initialisation of the timer struct so that >> > some of the helpers (such as arch_timer_ctx_index) can work correctly >> > at an early stage. >> > >> > Signed-off-by: Marc Zyngier <maz@kernel.org> >> > --- >> > arch/arm64/include/asm/kvm_host.h | 6 ++ >> > arch/arm64/kvm/arch_timer.c | 155 +++++++++++++++++++++++------- >> > arch/arm64/kvm/trace_arm.h | 8 +- >> > include/kvm/arm_arch_timer.h | 11 +-- >> > 4 files changed, 136 insertions(+), 44 deletions(-) >> > >> > +static u64 timer_get_offset(struct arch_timer_context *ctxt) { >> > + struct kvm_vcpu *vcpu = ctxt->vcpu; >> > + >> > + switch(arch_timer_ctx_index(ctxt)) { >> > + case TIMER_VTIMER: >> > + return __vcpu_sys_reg(vcpu, CNTVOFF_EL2); >> > + default: >> > + return 0; >> > + } >> > +} >> > + >> > Can I export this helper? As in my ptp_kvm implementation I need get >> > VCNT offset value separately not just give me a result of VCNT. >> >> Sorry, you need to give me a bit more context. What do you need the >> offset >> for exactly? > > Yeah, > In my ptp_kvm implementation, I need acquire wall time and counter > cycle in the same time in host. After get host counter cycle, I need > subtract it by VCNT offset to obtain VCNT. See > https://lkml.org/lkml/2020/6/19/441 > https://lkml.org/lkml/2020/6/19/441 > But now I can't get the VCNT offset easily like before using " > vcpu_vtimer(vcpu)->cntvoff" and I can't use the helper like > "kvm_arm_timer_read" as I need acquire the counter cycle in the same > time with the host wall time. I must be missing something. CNTVOFF_EL2 is now implemented as a standard system register, and has the same visibility as any other vcpu sysreg. Why doesn't vcpu_read_sys_reg(vcpu, CNTVOFF_EL2) work for you? M.
> -----Original Message----- > From: Marc Zyngier <maz@kernel.org> > Sent: Wednesday, August 19, 2020 6:39 PM > To: Jianyong Wu <Jianyong.Wu@arm.com> > Cc: Paolo Bonzini <pbonzini@redhat.com>; Peng Hao > <richard.peng@oppo.com>; kernel-team@android.com; kvm@vger.kernel.org; > Will Deacon <will@kernel.org>; Catalin Marinas <Catalin.Marinas@arm.com>; > Alexander Graf <graf@amazon.com>; kvmarm@lists.cs.columbia.edu; linux- > arm-kernel@lists.infradead.org > Subject: Re: [PATCH 47/56] KVM: arm64: timers: Move timer registers to the > sys_regs file > > On 2020-08-19 11:18, Jianyong Wu wrote: > >> -----Original Message----- > >> From: Marc Zyngier <maz@kernel.org> > >> Sent: Wednesday, August 19, 2020 6:00 PM > >> To: Jianyong Wu <Jianyong.Wu@arm.com> > >> Cc: Paolo Bonzini <pbonzini@redhat.com>; Peng Hao > >> <richard.peng@oppo.com>; kernel-team@android.com; > >> kvm@vger.kernel.org; Will Deacon <will@kernel.org>; Catalin Marinas > >> <Catalin.Marinas@arm.com>; Alexander Graf <graf@amazon.com>; > >> kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org > >> Subject: Re: [PATCH 47/56] KVM: arm64: timers: Move timer registers > >> to the sys_regs file > >> > >> On 2020-08-19 10:24, Jianyong Wu wrote: > >> > Hi Marc, > >> > > >> > -----Original Message----- > >> > From: kvmarm-bounces@lists.cs.columbia.edu > >> > <kvmarm-bounces@lists.cs.columbia.edu> On Behalf Of Marc Zyngier > >> > Sent: Thursday, August 6, 2020 1:57 AM > >> > To: Paolo Bonzini <pbonzini@redhat.com> > >> > Cc: Peng Hao <richard.peng@oppo.com>; kernel-team@android.com; > >> > kvm@vger.kernel.org; Will Deacon <will@kernel.org>; Catalin Marinas > >> > <Catalin.Marinas@arm.com>; Alexander Graf <graf@amazon.com>; > >> > kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org > >> > Subject: [PATCH 47/56] KVM: arm64: timers: Move timer registers to > >> > the sys_regs file > >> > > >> > Move the timer gsisters to the sysreg file. This will further help > >> > when they are directly changed by a nesting hypervisor in the VNCR > >> > page. > >> > > >> > This requires moving the initialisation of the timer struct so that > >> > some of the helpers (such as arch_timer_ctx_index) can work > >> > correctly at an early stage. > >> > > >> > Signed-off-by: Marc Zyngier <maz@kernel.org> > >> > --- > >> > arch/arm64/include/asm/kvm_host.h | 6 ++ > >> > arch/arm64/kvm/arch_timer.c | 155 +++++++++++++++++++++++------- > >> > arch/arm64/kvm/trace_arm.h | 8 +- > >> > include/kvm/arm_arch_timer.h | 11 +-- > >> > 4 files changed, 136 insertions(+), 44 deletions(-) > >> > > >> > +static u64 timer_get_offset(struct arch_timer_context *ctxt) { > >> > + struct kvm_vcpu *vcpu = ctxt->vcpu; > >> > + > >> > + switch(arch_timer_ctx_index(ctxt)) { > >> > + case TIMER_VTIMER: > >> > + return __vcpu_sys_reg(vcpu, CNTVOFF_EL2); > >> > + default: > >> > + return 0; > >> > + } > >> > +} > >> > + > >> > Can I export this helper? As in my ptp_kvm implementation I need > >> > get VCNT offset value separately not just give me a result of VCNT. > >> > >> Sorry, you need to give me a bit more context. What do you need the > >> offset for exactly? > > > > Yeah, > > In my ptp_kvm implementation, I need acquire wall time and counter > > cycle in the same time in host. After get host counter cycle, I need > > subtract it by VCNT offset to obtain VCNT. See > > https://lkml.org/lkml/2020/6/19/441 > > https://lkml.org/lkml/2020/6/19/441 > > But now I can't get the VCNT offset easily like before using " > > vcpu_vtimer(vcpu)->cntvoff" and I can't use the helper like > > "kvm_arm_timer_read" as I need acquire the counter cycle in the same > > time with the host wall time. > > I must be missing something. CNTVOFF_EL2 is now implemented as a standard > system register, and has the same visibility as any other vcpu sysreg. > > Why doesn't vcpu_read_sys_reg(vcpu, CNTVOFF_EL2) work for you? Maybe it serves me, let me try it, thanks. Thanks Jianyong > > M. > -- > Jazz is not dead. It just smells funny...
Hi David, Marc, On 8/5/20 7:56 PM, Marc Zyngier wrote: > From: David Brazdil <dbrazdil@google.com> > > Add new folders arch/arm64/kvm/hyp/{vhe,nvhe} and Makefiles for building code > that runs in EL2 under VHE/nVHE KVM, repsectivelly. Add an include folder for > hyp-specific header files which will include code common to VHE/nVHE. > > Build nVHE code with -D__KVM_NVHE_HYPERVISOR__, VHE code with > -D__KVM_VHE_HYPERVISOR__. > > Under nVHE compile each source file into a `.hyp.tmp.o` object first, then > prefix all its symbols with "__kvm_nvhe_" using `objcopy` and produce > a `.hyp.o`. Suffixes were chosen so that it would be possible for VHE and nVHE > to share some source files, but compiled with different CFLAGS. > > The nVHE ELF symbol prefix is added to kallsyms.c as ignored. EL2-only symbols > will never appear in EL1 stack traces. > > Due to symbol prefixing, add a section in image-vars.h for aliases of symbols > that are defined in nVHE EL2 and accessed by kernel in EL1 or vice versa. > > Signed-off-by: David Brazdil <dbrazdil@google.com> > Signed-off-by: Marc Zyngier <maz@kernel.org> > Link: https://lore.kernel.org/r/20200625131420.71444-4-dbrazdil@google.com > --- > arch/arm64/kernel/image-vars.h | 14 +++++++++++++ > arch/arm64/kvm/hyp/Makefile | 10 +++++++--- > arch/arm64/kvm/hyp/nvhe/Makefile | 34 ++++++++++++++++++++++++++++++++ > arch/arm64/kvm/hyp/vhe/Makefile | 17 ++++++++++++++++ > scripts/kallsyms.c | 1 + > 5 files changed, 73 insertions(+), 3 deletions(-) > create mode 100644 arch/arm64/kvm/hyp/nvhe/Makefile > create mode 100644 arch/arm64/kvm/hyp/vhe/Makefile > > diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h > index be0a63ffed23..3dc27da47712 100644 > --- a/arch/arm64/kernel/image-vars.h > +++ b/arch/arm64/kernel/image-vars.h > @@ -51,4 +51,18 @@ __efistub__ctype = _ctype; > > #endif > > +#ifdef CONFIG_KVM > + > +/* > + * KVM nVHE code has its own symbol namespace prefixed with __kvm_nvhe_, to > + * separate it from the kernel proper. The following symbols are legally > + * accessed by it, therefore provide aliases to make them linkable. > + * Do not include symbols which may not be safely accessed under hypervisor > + * memory mappings. > + */ > + > +#define KVM_NVHE_ALIAS(sym) __kvm_nvhe_##sym = sym; > + > +#endif /* CONFIG_KVM */ > + > #endif /* __ARM64_KERNEL_IMAGE_VARS_H */ > diff --git a/arch/arm64/kvm/hyp/Makefile b/arch/arm64/kvm/hyp/Makefile > index 5d8357ddc234..9c5dfe6ff80b 100644 > --- a/arch/arm64/kvm/hyp/Makefile > +++ b/arch/arm64/kvm/hyp/Makefile > @@ -3,10 +3,14 @@ > # Makefile for Kernel-based Virtual Machine module, HYP part > # > > -ccflags-y += -fno-stack-protector -DDISABLE_BRANCH_PROFILING \ > - $(DISABLE_STACKLEAK_PLUGIN) > +incdir := $(srctree)/$(src)/include > +subdir-asflags-y := -I$(incdir) > +subdir-ccflags-y := -I$(incdir) \ > + -fno-stack-protector \ > + -DDISABLE_BRANCH_PROFILING \ > + $(DISABLE_STACKLEAK_PLUGIN) > > -obj-$(CONFIG_KVM) += hyp.o > +obj-$(CONFIG_KVM) += hyp.o nvhe/ > obj-$(CONFIG_KVM_INDIRECT_VECTORS) += smccc_wa.o > > hyp-y := vgic-v3-sr.o timer-sr.o aarch32.o vgic-v2-cpuif-proxy.o sysreg-sr.o \ > diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile > new file mode 100644 > index 000000000000..955f4188e00f > --- /dev/null > +++ b/arch/arm64/kvm/hyp/nvhe/Makefile > @@ -0,0 +1,34 @@ > +# SPDX-License-Identifier: GPL-2.0 > +# > +# Makefile for Kernel-based Virtual Machine module, HYP/nVHE part > +# > + > +asflags-y := -D__KVM_NVHE_HYPERVISOR__ > +ccflags-y := -D__KVM_NVHE_HYPERVISOR__ > + > +obj-y := > + > +obj-y := $(patsubst %.o,%.hyp.o,$(obj-y)) > +extra-y := $(patsubst %.hyp.o,%.hyp.tmp.o,$(obj-y)) > + > +$(obj)/%.hyp.tmp.o: $(src)/%.c FORCE > + $(call if_changed_rule,cc_o_c) > +$(obj)/%.hyp.tmp.o: $(src)/%.S FORCE > + $(call if_changed_rule,as_o_S) > +$(obj)/%.hyp.o: $(obj)/%.hyp.tmp.o FORCE > + $(call if_changed,hypcopy) > + > +quiet_cmd_hypcopy = HYPCOPY $@ > + cmd_hypcopy = $(OBJCOPY) --prefix-symbols=__kvm_nvhe_ $< $@ > + > +# KVM nVHE code is run at a different exception code with a different map, so > +# compiler instrumentation that inserts callbacks or checks into the code may > +# cause crashes. Just disable it. > +GCOV_PROFILE := n > +KASAN_SANITIZE := n > +UBSAN_SANITIZE := n > +KCOV_INSTRUMENT := n > + > +# Skip objtool checking for this directory because nVHE code is compiled with > +# non-standard build rules. > +OBJECT_FILES_NON_STANDARD := y > diff --git a/arch/arm64/kvm/hyp/vhe/Makefile b/arch/arm64/kvm/hyp/vhe/Makefile > new file mode 100644 > index 000000000000..e04375546081 > --- /dev/null > +++ b/arch/arm64/kvm/hyp/vhe/Makefile > @@ -0,0 +1,17 @@ > +# SPDX-License-Identifier: GPL-2.0 > +# > +# Makefile for Kernel-based Virtual Machine module, HYP/nVHE part > +# > + > +asflags-y := -D__KVM_VHE_HYPERVISOR__ > +ccflags-y := -D__KVM_VHE_HYPERVISOR__ > + > +obj-y := > + > +# KVM code is run at a different exception code with a different map, so > +# compiler instrumentation that inserts callbacks or checks into the code may > +# cause crashes. Just disable it. > +GCOV_PROFILE := n > +KASAN_SANITIZE := n > +UBSAN_SANITIZE := n > +KCOV_INSTRUMENT := n > diff --git a/scripts/kallsyms.c b/scripts/kallsyms.c > index 6dc3078649fa..0096cd965332 100644 > --- a/scripts/kallsyms.c > +++ b/scripts/kallsyms.c > @@ -109,6 +109,7 @@ static bool is_ignored_symbol(const char *name, char type) > ".LASANPC", /* s390 kasan local symbols */ > "__crc_", /* modversions */ > "__efistub_", /* arm64 EFI stub namespace */ > + "__kvm_nvhe_", /* arm64 non-VHE KVM namespace */ The addition of this line seems to have introduced errors on the 'vmlinux symtab matches kallsyms' perf test (perf test -v 1) which fails on aarch64 for all __kvm_nvhe_ prefixed symbols, like ERR : <addr> : __kvm_nvhe___invalid not on kallsyms ERR : <addr> : __kvm_nvhe___do_hyp_init not on kallsyms ERR : <addr> : __kvm_nvhe___kvm_handle_stub_hvc not on kallsyms ERR : <addr> : __kvm_nvhe_reset not on kallsyms ../.. I understand we willingly hided those symbols from /proc/kallsyms. Do you confirm the right fix is to upgrade the perf test suite accordingly? Thanks Eric > NULL > }; > >
Hi Marc, On 5/5/21 8:03 PM, Marc Zyngier wrote: > Hi Eric, > > On Tue, 04 May 2021 15:47:36 +0100, > Auger Eric <eric.auger@redhat.com> wrote: >> >> Hi David, Marc, >> >> On 8/5/20 7:56 PM, Marc Zyngier wrote: >>> From: David Brazdil <dbrazdil@google.com> >>> >>> Add new folders arch/arm64/kvm/hyp/{vhe,nvhe} and Makefiles for building code >>> that runs in EL2 under VHE/nVHE KVM, repsectivelly. Add an include folder for >>> hyp-specific header files which will include code common to VHE/nVHE. >>> >>> Build nVHE code with -D__KVM_NVHE_HYPERVISOR__, VHE code with >>> -D__KVM_VHE_HYPERVISOR__. >>> >>> Under nVHE compile each source file into a `.hyp.tmp.o` object first, then >>> prefix all its symbols with "__kvm_nvhe_" using `objcopy` and produce >>> a `.hyp.o`. Suffixes were chosen so that it would be possible for VHE and nVHE >>> to share some source files, but compiled with different CFLAGS. >>> >>> The nVHE ELF symbol prefix is added to kallsyms.c as ignored. EL2-only symbols >>> will never appear in EL1 stack traces. >>> >>> Due to symbol prefixing, add a section in image-vars.h for aliases of symbols >>> that are defined in nVHE EL2 and accessed by kernel in EL1 or vice versa. >>> >>> Signed-off-by: David Brazdil <dbrazdil@google.com> >>> Signed-off-by: Marc Zyngier <maz@kernel.org> >>> Link: https://lore.kernel.org/r/20200625131420.71444-4-dbrazdil@google.com >>> --- >>> arch/arm64/kernel/image-vars.h | 14 +++++++++++++ >>> arch/arm64/kvm/hyp/Makefile | 10 +++++++--- >>> arch/arm64/kvm/hyp/nvhe/Makefile | 34 ++++++++++++++++++++++++++++++++ >>> arch/arm64/kvm/hyp/vhe/Makefile | 17 ++++++++++++++++ >>> scripts/kallsyms.c | 1 + >>> 5 files changed, 73 insertions(+), 3 deletions(-) >>> create mode 100644 arch/arm64/kvm/hyp/nvhe/Makefile >>> create mode 100644 arch/arm64/kvm/hyp/vhe/Makefile > > [...] > >>> diff --git a/scripts/kallsyms.c b/scripts/kallsyms.c >>> index 6dc3078649fa..0096cd965332 100644 >>> --- a/scripts/kallsyms.c >>> +++ b/scripts/kallsyms.c >>> @@ -109,6 +109,7 @@ static bool is_ignored_symbol(const char *name, char type) >>> ".LASANPC", /* s390 kasan local symbols */ >>> "__crc_", /* modversions */ >>> "__efistub_", /* arm64 EFI stub namespace */ >>> + "__kvm_nvhe_", /* arm64 non-VHE KVM namespace */ >> The addition of this line seems to have introduced errors on the >> 'vmlinux symtab matches kallsyms' perf test (perf test -v 1) which fails >> on aarch64 for all __kvm_nvhe_ prefixed symbols, like >> >> ERR : <addr> : __kvm_nvhe___invalid not on kallsyms >> ERR : <addr> : __kvm_nvhe___do_hyp_init not on kallsyms >> ERR : <addr> : __kvm_nvhe___kvm_handle_stub_hvc not on kallsyms >> ERR : <addr> : __kvm_nvhe_reset not on kallsyms >> ../.. >> >> I understand we willingly hided those symbols from /proc/kallsyms. Do >> you confirm the right fix is to upgrade the perf test suite accordingly? > > Hmmm. This test always fail here, no matter whether I have this line > or not: sorry Marc I missed your reply, > > <quote> > maz@big-leg-emma:~$ sudo perf_5.10 test -v 1 > 1: vmlinux symtab matches kallsyms : > --- start --- > test child forked, pid 664 > /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module! > /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module! > /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module! > /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module! > /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module! > /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module! > /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module! > /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module! > /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module! > /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module! > Looking at the vmlinux_path (8 entries long) > symsrc__init: cannot get elf header. > symsrc__init: cannot get elf header. > Couldn't find a vmlinux that matches the kernel running on this machine, skipping test > test child finished with -2 > ---- end ---- > vmlinux symtab matches kallsyms: Skip > </quote> > > Rookie question: How do you provide a kernel to the test framework? On my end I compiled and installed a kernel and compiled perf from that same kernel. Should work. I don't know how to point to a specific vmlinux Thanks Eric > > Thanks, > > M. >
Hi David, Marc, On 5/4/21 4:47 PM, Auger Eric wrote: > Hi David, Marc, > > On 8/5/20 7:56 PM, Marc Zyngier wrote: >> From: David Brazdil <dbrazdil@google.com> >> >> Add new folders arch/arm64/kvm/hyp/{vhe,nvhe} and Makefiles for building code >> that runs in EL2 under VHE/nVHE KVM, repsectivelly. Add an include folder for >> hyp-specific header files which will include code common to VHE/nVHE. >> >> Build nVHE code with -D__KVM_NVHE_HYPERVISOR__, VHE code with >> -D__KVM_VHE_HYPERVISOR__. >> >> Under nVHE compile each source file into a `.hyp.tmp.o` object first, then >> prefix all its symbols with "__kvm_nvhe_" using `objcopy` and produce >> a `.hyp.o`. Suffixes were chosen so that it would be possible for VHE and nVHE >> to share some source files, but compiled with different CFLAGS. >> >> The nVHE ELF symbol prefix is added to kallsyms.c as ignored. EL2-only symbols >> will never appear in EL1 stack traces. >> >> Due to symbol prefixing, add a section in image-vars.h for aliases of symbols >> that are defined in nVHE EL2 and accessed by kernel in EL1 or vice versa. >> >> Signed-off-by: David Brazdil <dbrazdil@google.com> >> Signed-off-by: Marc Zyngier <maz@kernel.org> >> Link: https://lore.kernel.org/r/20200625131420.71444-4-dbrazdil@google.com >> --- >> arch/arm64/kernel/image-vars.h | 14 +++++++++++++ >> arch/arm64/kvm/hyp/Makefile | 10 +++++++--- >> arch/arm64/kvm/hyp/nvhe/Makefile | 34 ++++++++++++++++++++++++++++++++ >> arch/arm64/kvm/hyp/vhe/Makefile | 17 ++++++++++++++++ >> scripts/kallsyms.c | 1 + >> 5 files changed, 73 insertions(+), 3 deletions(-) >> create mode 100644 arch/arm64/kvm/hyp/nvhe/Makefile >> create mode 100644 arch/arm64/kvm/hyp/vhe/Makefile >> >> diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h >> index be0a63ffed23..3dc27da47712 100644 >> --- a/arch/arm64/kernel/image-vars.h >> +++ b/arch/arm64/kernel/image-vars.h >> @@ -51,4 +51,18 @@ __efistub__ctype = _ctype; >> >> #endif >> >> +#ifdef CONFIG_KVM >> + >> +/* >> + * KVM nVHE code has its own symbol namespace prefixed with __kvm_nvhe_, to >> + * separate it from the kernel proper. The following symbols are legally >> + * accessed by it, therefore provide aliases to make them linkable. >> + * Do not include symbols which may not be safely accessed under hypervisor >> + * memory mappings. >> + */ >> + >> +#define KVM_NVHE_ALIAS(sym) __kvm_nvhe_##sym = sym; >> + >> +#endif /* CONFIG_KVM */ >> + >> #endif /* __ARM64_KERNEL_IMAGE_VARS_H */ >> diff --git a/arch/arm64/kvm/hyp/Makefile b/arch/arm64/kvm/hyp/Makefile >> index 5d8357ddc234..9c5dfe6ff80b 100644 >> --- a/arch/arm64/kvm/hyp/Makefile >> +++ b/arch/arm64/kvm/hyp/Makefile >> @@ -3,10 +3,14 @@ >> # Makefile for Kernel-based Virtual Machine module, HYP part >> # >> >> -ccflags-y += -fno-stack-protector -DDISABLE_BRANCH_PROFILING \ >> - $(DISABLE_STACKLEAK_PLUGIN) >> +incdir := $(srctree)/$(src)/include >> +subdir-asflags-y := -I$(incdir) >> +subdir-ccflags-y := -I$(incdir) \ >> + -fno-stack-protector \ >> + -DDISABLE_BRANCH_PROFILING \ >> + $(DISABLE_STACKLEAK_PLUGIN) >> >> -obj-$(CONFIG_KVM) += hyp.o >> +obj-$(CONFIG_KVM) += hyp.o nvhe/ >> obj-$(CONFIG_KVM_INDIRECT_VECTORS) += smccc_wa.o >> >> hyp-y := vgic-v3-sr.o timer-sr.o aarch32.o vgic-v2-cpuif-proxy.o sysreg-sr.o \ >> diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile >> new file mode 100644 >> index 000000000000..955f4188e00f >> --- /dev/null >> +++ b/arch/arm64/kvm/hyp/nvhe/Makefile >> @@ -0,0 +1,34 @@ >> +# SPDX-License-Identifier: GPL-2.0 >> +# >> +# Makefile for Kernel-based Virtual Machine module, HYP/nVHE part >> +# >> + >> +asflags-y := -D__KVM_NVHE_HYPERVISOR__ >> +ccflags-y := -D__KVM_NVHE_HYPERVISOR__ >> + >> +obj-y := >> + >> +obj-y := $(patsubst %.o,%.hyp.o,$(obj-y)) >> +extra-y := $(patsubst %.hyp.o,%.hyp.tmp.o,$(obj-y)) >> + >> +$(obj)/%.hyp.tmp.o: $(src)/%.c FORCE >> + $(call if_changed_rule,cc_o_c) >> +$(obj)/%.hyp.tmp.o: $(src)/%.S FORCE >> + $(call if_changed_rule,as_o_S) >> +$(obj)/%.hyp.o: $(obj)/%.hyp.tmp.o FORCE >> + $(call if_changed,hypcopy) >> + >> +quiet_cmd_hypcopy = HYPCOPY $@ >> + cmd_hypcopy = $(OBJCOPY) --prefix-symbols=__kvm_nvhe_ $< $@ >> + >> +# KVM nVHE code is run at a different exception code with a different map, so >> +# compiler instrumentation that inserts callbacks or checks into the code may >> +# cause crashes. Just disable it. >> +GCOV_PROFILE := n >> +KASAN_SANITIZE := n >> +UBSAN_SANITIZE := n >> +KCOV_INSTRUMENT := n >> + >> +# Skip objtool checking for this directory because nVHE code is compiled with >> +# non-standard build rules. >> +OBJECT_FILES_NON_STANDARD := y >> diff --git a/arch/arm64/kvm/hyp/vhe/Makefile b/arch/arm64/kvm/hyp/vhe/Makefile >> new file mode 100644 >> index 000000000000..e04375546081 >> --- /dev/null >> +++ b/arch/arm64/kvm/hyp/vhe/Makefile >> @@ -0,0 +1,17 @@ >> +# SPDX-License-Identifier: GPL-2.0 >> +# >> +# Makefile for Kernel-based Virtual Machine module, HYP/nVHE part >> +# >> + >> +asflags-y := -D__KVM_VHE_HYPERVISOR__ >> +ccflags-y := -D__KVM_VHE_HYPERVISOR__ >> + >> +obj-y := >> + >> +# KVM code is run at a different exception code with a different map, so >> +# compiler instrumentation that inserts callbacks or checks into the code may >> +# cause crashes. Just disable it. >> +GCOV_PROFILE := n >> +KASAN_SANITIZE := n >> +UBSAN_SANITIZE := n >> +KCOV_INSTRUMENT := n >> diff --git a/scripts/kallsyms.c b/scripts/kallsyms.c >> index 6dc3078649fa..0096cd965332 100644 >> --- a/scripts/kallsyms.c >> +++ b/scripts/kallsyms.c >> @@ -109,6 +109,7 @@ static bool is_ignored_symbol(const char *name, char type) >> ".LASANPC", /* s390 kasan local symbols */ >> "__crc_", /* modversions */ >> "__efistub_", /* arm64 EFI stub namespace */ >> + "__kvm_nvhe_", /* arm64 non-VHE KVM namespace */ > The addition of this line seems to have introduced errors on the > 'vmlinux symtab matches kallsyms' perf test (perf test -v 1) which fails > on aarch64 for all __kvm_nvhe_ prefixed symbols, like > > ERR : <addr> : __kvm_nvhe___invalid not on kallsyms > ERR : <addr> : __kvm_nvhe___do_hyp_init not on kallsyms > ERR : <addr> : __kvm_nvhe___kvm_handle_stub_hvc not on kallsyms > ERR : <addr> : __kvm_nvhe_reset not on kallsyms > ../.. > > I understand we willingly hided those symbols from /proc/kallsyms. Do > you confirm the right fix is to upgrade the perf test suite accordingly? Were you eventually able to reproduce? Thanks Eric > > Thanks > > Eric > > >> NULL >> }; >> >>
Hi Eric, On Tue, 18 May 2021 12:48:41 +0100, Auger Eric <eric.auger@redhat.com> wrote: > > Hi David, Marc, > [...] > >> --- a/scripts/kallsyms.c > >> +++ b/scripts/kallsyms.c > >> @@ -109,6 +109,7 @@ static bool is_ignored_symbol(const char *name, char type) > >> ".LASANPC", /* s390 kasan local symbols */ > >> "__crc_", /* modversions */ > >> "__efistub_", /* arm64 EFI stub namespace */ > >> + "__kvm_nvhe_", /* arm64 non-VHE KVM namespace */ > > The addition of this line seems to have introduced errors on the > > 'vmlinux symtab matches kallsyms' perf test (perf test -v 1) which fails > > on aarch64 for all __kvm_nvhe_ prefixed symbols, like > > > > ERR : <addr> : __kvm_nvhe___invalid not on kallsyms > > ERR : <addr> : __kvm_nvhe___do_hyp_init not on kallsyms > > ERR : <addr> : __kvm_nvhe___kvm_handle_stub_hvc not on kallsyms > > ERR : <addr> : __kvm_nvhe_reset not on kallsyms > > ../.. > > > > I understand we willingly hided those symbols from /proc/kallsyms. Do > > you confirm the right fix is to upgrade the perf test suite accordingly? > > Were you eventually able to reproduce? Unfortunately not, I always end-up with the test failing even if I add the symbols back. I must be doing something wrong... M.