mbox series

[GIT,PULL] KVM/arm64 updates for 5.9

Message ID 20200805175700.62775-1-maz@kernel.org
State New
Headers show
Series [GIT,PULL] KVM/arm64 updates for 5.9 | expand

Pull-request

git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git tags/kvmarm-5.9

Message

Marc Zyngier Aug. 5, 2020, 5:56 p.m. UTC
Hi Paolo,

This is the bulk of the 5.9 patches for KVM/arm64. It is a pretty busy
merge window for us this time, thanks to the ongoing Protected KVM
work. We have changes all over the map, but the most important piece
probably is the way we now build the EL2 code on non-VHE systems. On
top of giving us better control over what gets pulled in there, it
allowed us to enable instrumentation on VHE systems.

The rest is a mixed bag of new features (TTL TLB invalidation, Pointer
Auth on non-VHE), preliminary patches for NV, some early MMU rework
before the 5.10 onslaught, and tons of cleanups.

A few things to notice:

- We share a branch with the arm64 tree, which has gone in already.

- There are a number of known conflicts with Sean's MMU cache rework,
  as well as the late fixes that went in 5.8. The conflicts are pretty
  simple to resolve, and -next has the right resolutions already.

Please pull,

	M.

The following changes since commit 9ebcfadb0610322ac537dd7aa5d9cbc2b2894c68:

  Linux 5.8-rc3 (2020-06-28 15:00:24 -0700)

are available in the Git repository at:

  git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git tags/kvmarm-5.9

for you to fetch changes up to 16314874b12b451bd5a1df86bcb69745eb487502:

  Merge branch 'kvm-arm64/misc-5.9' into kvmarm-master/next (2020-07-30 16:13:04 +0100)

----------------------------------------------------------------
KVM/arm64 updates for Linux 5.9:

- Split the VHE and nVHE hypervisor code bases, build the EL2 code
  separately, allowing for the VHE code to now be built with instrumentation

- Level-based TLB invalidation support

- Restructure of the vcpu register storage to accomodate the NV code

- Pointer Authentication available for guests on nVHE hosts

- Simplification of the system register table parsing

- MMU cleanups and fixes

- A number of post-32bit cleanups and other fixes

----------------------------------------------------------------
Alexander Graf (2):
      KVM: arm64: vgic-its: Change default outer cacheability for {PEND, PROP}BASER
      KVM: arm: Add trace name for ARM_NISV

Andrew Scull (3):
      arm64: kvm: Remove kern_hyp_va from get_vcpu_ptr
      KVM: arm64: Handle calls to prefixed hyp functions
      KVM: arm64: Move hyp-init.S to nVHE

Christoffer Dall (1):
      KVM: arm64: Factor out stage 2 page table data from struct kvm

David Brazdil (16):
      KVM: arm64: Fix symbol dependency in __hyp_call_panic_nvhe
      KVM: arm64: Move __smccc_workaround_1_smc to .rodata
      KVM: arm64: Add build rules for separate VHE/nVHE object files
      KVM: arm64: Use build-time defines in has_vhe()
      KVM: arm64: Build hyp-entry.S separately for VHE/nVHE
      KVM: arm64: Duplicate hyp/tlb.c for VHE/nVHE
      KVM: arm64: Split hyp/switch.c to VHE/nVHE
      KVM: arm64: Split hyp/debug-sr.c to VHE/nVHE
      KVM: arm64: Split hyp/sysreg-sr.c to VHE/nVHE
      KVM: arm64: Duplicate hyp/timer-sr.c for VHE/nVHE
      KVM: arm64: Compile remaining hyp/ files for both VHE/nVHE
      KVM: arm64: Remove __hyp_text macro, use build rules instead
      KVM: arm64: Lift instrumentation restrictions on VHE
      KVM: arm64: Make nVHE ASLR conditional on RANDOMIZE_BASE
      KVM: arm64: Substitute RANDOMIZE_BASE for HARDEN_EL2_VECTORS
      KVM: arm64: Ensure that all nVHE hyp code is in .hyp.text

Gavin Shan (1):
      KVM: arm64: Rename HSR to ESR

James Morse (5):
      KVM: arm64: Drop the target_table[] indirection
      KVM: arm64: Tolerate an empty target_table list
      KVM: arm64: Move ACTLR_EL1 emulation to the sys_reg_descs array
      KVM: arm64: Remove target_table from exit handlers
      KVM: arm64: Remove the target table

Marc Zyngier (31):
      KVM: arm64: Enable Address Authentication at EL2 if available
      KVM: arm64: Allow ARM64_PTR_AUTH when ARM64_VHE=n
      KVM: arm64: Allow PtrAuth to be enabled from userspace on non-VHE systems
      KVM: arm64: Check HCR_EL2 instead of shadow copy to swap PtrAuth registers
      KVM: arm64: Simplify PtrAuth alternative patching
      KVM: arm64: Allow in-atomic injection of SPIs
      arm64: Detect the ARMv8.4 TTL feature
      arm64: Document SW reserved PTE/PMD bits in Stage-2 descriptors
      arm64: Add level-hinted TLB invalidation helper
      Merge branch 'kvm-arm64/ttl-for-arm64' into HEAD
      KVM: arm64: Use TTL hint in when invalidating stage-2 translations
      KVM: arm64: Introduce accessor for ctxt->sys_reg
      KVM: arm64: hyp: Use ctxt_sys_reg/__vcpu_sys_reg instead of raw sys_regs access
      KVM: arm64: sve: Use __vcpu_sys_reg() instead of raw sys_regs access
      KVM: arm64: pauth: Use ctxt_sys_reg() instead of raw sys_regs access
      KVM: arm64: debug: Drop useless vpcu parameter
      KVM: arm64: Make struct kvm_regs userspace-only
      KVM: arm64: Move ELR_EL1 to the system register array
      KVM: arm64: Move SP_EL1 to the system register array
      KVM: arm64: Disintegrate SPSR array
      KVM: arm64: Move SPSR_EL1 to the system register array
      KVM: arm64: timers: Rename kvm_timer_sync_hwstate to kvm_timer_sync_user
      KVM: arm64: timers: Move timer registers to the sys_regs file
      KVM: arm64: Don't use has_vhe() for CHOOSE_HYP_SYM()
      Merge branch 'kvm-arm64/el2-obj-v4.1' into kvmarm-master/next-WIP
      Merge branch 'kvm-arm64/pre-nv-5.9' into kvmarm-master/next-WIP
      Merge branch 'kvm-arm64/ptrauth-nvhe' into kvmarm-master/next-WIP
      Merge branch 'kvm-arm64/target-table-no-more' into kvmarm-master/next-WIP
      Merge branch 'kvm-arm64/misc-5.9' into kvmarm-master/next-WIP
      Merge branch 'kvm-arm64/el2-obj-v4.1' into kvmarm-master/next
      Merge branch 'kvm-arm64/misc-5.9' into kvmarm-master/next

Peng Hao (1):
      KVM: arm64: Drop long gone function parameter documentation

Will Deacon (4):
      KVM: arm64: Rename kvm_vcpu_dabt_isextabt()
      KVM: arm64: Handle data and instruction external aborts the same way
      KVM: arm64: Don't skip cache maintenance for read-only memslots
      KVM: arm64: Move S1PTW S2 fault logic out of io_mem_abort()

 arch/arm64/Kconfig                                 |  20 +-
 arch/arm64/include/asm/cpucaps.h                   |   3 +-
 arch/arm64/include/asm/kvm_asm.h                   |  75 +-
 arch/arm64/include/asm/kvm_coproc.h                |   8 -
 arch/arm64/include/asm/kvm_emulate.h               |  75 +-
 arch/arm64/include/asm/kvm_host.h                  |  94 ++-
 arch/arm64/include/asm/kvm_hyp.h                   |  15 +-
 arch/arm64/include/asm/kvm_mmu.h                   |  16 +-
 arch/arm64/include/asm/kvm_ptrauth.h               |  34 +-
 arch/arm64/include/asm/mmu.h                       |   7 -
 arch/arm64/include/asm/pgtable-hwdef.h             |   2 +
 arch/arm64/include/asm/stage2_pgtable.h            |   9 +
 arch/arm64/include/asm/sysreg.h                    |   1 +
 arch/arm64/include/asm/tlbflush.h                  |  45 +
 arch/arm64/include/asm/virt.h                      |  13 +-
 arch/arm64/kernel/asm-offsets.c                    |   3 +-
 arch/arm64/kernel/cpu_errata.c                     |   4 +-
 arch/arm64/kernel/cpufeature.c                     |  11 +
 arch/arm64/kernel/image-vars.h                     |  54 ++
 arch/arm64/kvm/Kconfig                             |   2 +-
 arch/arm64/kvm/Makefile                            |   4 +-
 arch/arm64/kvm/arch_timer.c                        | 157 +++-
 arch/arm64/kvm/arm.c                               |  57 +-
 arch/arm64/kvm/fpsimd.c                            |   6 +-
 arch/arm64/kvm/guest.c                             |  79 +-
 arch/arm64/kvm/handle_exit.c                       |  32 +-
 arch/arm64/kvm/hyp/Makefile                        |  24 +-
 arch/arm64/kvm/hyp/aarch32.c                       |   8 +-
 arch/arm64/kvm/hyp/entry.S                         |   4 +-
 arch/arm64/kvm/hyp/fpsimd.S                        |   1 -
 arch/arm64/kvm/hyp/hyp-entry.S                     |  21 +-
 .../kvm/hyp/{debug-sr.c => include/hyp/debug-sr.h} |  88 +-
 arch/arm64/kvm/hyp/include/hyp/switch.h            | 511 +++++++++++
 arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h         | 193 +++++
 arch/arm64/kvm/hyp/nvhe/Makefile                   |  62 ++
 arch/arm64/kvm/hyp/nvhe/debug-sr.c                 |  77 ++
 arch/arm64/kvm/{ => hyp/nvhe}/hyp-init.S           |   5 +
 arch/arm64/kvm/hyp/nvhe/switch.c                   | 272 ++++++
 arch/arm64/kvm/hyp/nvhe/sysreg-sr.c                |  46 +
 arch/arm64/kvm/hyp/{ => nvhe}/timer-sr.c           |   6 +-
 arch/arm64/kvm/hyp/nvhe/tlb.c                      | 154 ++++
 arch/arm64/kvm/hyp/smccc_wa.S                      |  32 +
 arch/arm64/kvm/hyp/switch.c                        | 936 ---------------------
 arch/arm64/kvm/hyp/sysreg-sr.c                     | 333 --------
 arch/arm64/kvm/hyp/tlb.c                           | 242 ------
 arch/arm64/kvm/hyp/vgic-v2-cpuif-proxy.c           |   4 +-
 arch/arm64/kvm/hyp/vgic-v3-sr.c                    | 134 ++-
 arch/arm64/kvm/hyp/vhe/Makefile                    |  11 +
 arch/arm64/kvm/hyp/vhe/debug-sr.c                  |  26 +
 arch/arm64/kvm/hyp/vhe/switch.c                    | 219 +++++
 arch/arm64/kvm/hyp/vhe/sysreg-sr.c                 | 114 +++
 arch/arm64/kvm/hyp/vhe/timer-sr.c                  |  12 +
 arch/arm64/kvm/hyp/vhe/tlb.c                       | 162 ++++
 arch/arm64/kvm/inject_fault.c                      |   2 +-
 arch/arm64/kvm/mmio.c                              |   6 -
 arch/arm64/kvm/mmu.c                               | 311 ++++---
 arch/arm64/kvm/regmap.c                            |  37 +-
 arch/arm64/kvm/reset.c                             |  23 +-
 arch/arm64/kvm/sys_regs.c                          | 207 ++---
 arch/arm64/kvm/sys_regs_generic_v8.c               |  96 ---
 arch/arm64/kvm/trace_arm.h                         |   8 +-
 arch/arm64/kvm/va_layout.c                         |   2 +-
 arch/arm64/kvm/vgic/vgic-irqfd.c                   |  24 +-
 arch/arm64/kvm/vgic/vgic-its.c                     |   3 +-
 arch/arm64/kvm/vgic/vgic-mmio-v3.c                 |   2 +-
 include/kvm/arm_arch_timer.h                       |  13 +-
 include/trace/events/kvm.h                         |   2 +-
 scripts/kallsyms.c                                 |   1 +
 68 files changed, 2883 insertions(+), 2377 deletions(-)
 rename arch/arm64/kvm/hyp/{debug-sr.c => include/hyp/debug-sr.h} (66%)
 create mode 100644 arch/arm64/kvm/hyp/include/hyp/switch.h
 create mode 100644 arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h
 create mode 100644 arch/arm64/kvm/hyp/nvhe/Makefile
 create mode 100644 arch/arm64/kvm/hyp/nvhe/debug-sr.c
 rename arch/arm64/kvm/{ => hyp/nvhe}/hyp-init.S (95%)
 create mode 100644 arch/arm64/kvm/hyp/nvhe/switch.c
 create mode 100644 arch/arm64/kvm/hyp/nvhe/sysreg-sr.c
 rename arch/arm64/kvm/hyp/{ => nvhe}/timer-sr.c (84%)
 create mode 100644 arch/arm64/kvm/hyp/nvhe/tlb.c
 create mode 100644 arch/arm64/kvm/hyp/smccc_wa.S
 delete mode 100644 arch/arm64/kvm/hyp/switch.c
 delete mode 100644 arch/arm64/kvm/hyp/sysreg-sr.c
 delete mode 100644 arch/arm64/kvm/hyp/tlb.c
 create mode 100644 arch/arm64/kvm/hyp/vhe/Makefile
 create mode 100644 arch/arm64/kvm/hyp/vhe/debug-sr.c
 create mode 100644 arch/arm64/kvm/hyp/vhe/switch.c
 create mode 100644 arch/arm64/kvm/hyp/vhe/sysreg-sr.c
 create mode 100644 arch/arm64/kvm/hyp/vhe/timer-sr.c
 create mode 100644 arch/arm64/kvm/hyp/vhe/tlb.c
 delete mode 100644 arch/arm64/kvm/sys_regs_generic_v8.c

Comments

Paolo Bonzini Aug. 5, 2020, 6:27 p.m. UTC | #1
On 05/08/20 19:56, Marc Zyngier wrote:
> Hi Paolo,
> 
> This is the bulk of the 5.9 patches for KVM/arm64. It is a pretty busy
> merge window for us this time, thanks to the ongoing Protected KVM
> work. We have changes all over the map, but the most important piece
> probably is the way we now build the EL2 code on non-VHE systems. On
> top of giving us better control over what gets pulled in there, it
> allowed us to enable instrumentation on VHE systems.
> 
> The rest is a mixed bag of new features (TTL TLB invalidation, Pointer
> Auth on non-VHE), preliminary patches for NV, some early MMU rework
> before the 5.10 onslaught, and tons of cleanups.
> 
> A few things to notice:
> 
> - We share a branch with the arm64 tree, which has gone in already.
> 
> - There are a number of known conflicts with Sean's MMU cache rework,
>   as well as the late fixes that went in 5.8. The conflicts are pretty
>   simple to resolve, and -next has the right resolutions already.

Ok, since I have already an x86 conflict I'll wait to pull this until
next week.

Paolo
Paolo Bonzini Aug. 9, 2020, 4:13 p.m. UTC | #2
On 05/08/20 19:56, Marc Zyngier wrote:
>   git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm.git tags/kvmarm-5.9

Pulled, thanks.

Paolo
Jianyong Wu Aug. 19, 2020, 9:24 a.m. UTC | #3
Hi Marc,

-----Original Message-----
From: kvmarm-bounces@lists.cs.columbia.edu <kvmarm-bounces@lists.cs.columbia.edu> On Behalf Of Marc Zyngier
Sent: Thursday, August 6, 2020 1:57 AM
To: Paolo Bonzini <pbonzini@redhat.com>
Cc: Peng Hao <richard.peng@oppo.com>; kernel-team@android.com; kvm@vger.kernel.org; Will Deacon <will@kernel.org>; Catalin Marinas <Catalin.Marinas@arm.com>; Alexander Graf <graf@amazon.com>; kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org
Subject: [PATCH 47/56] KVM: arm64: timers: Move timer registers to the sys_regs file

Move the timer gsisters to the sysreg file. This will further help when they are directly changed by a nesting hypervisor in the VNCR page.

This requires moving the initialisation of the timer struct so that some of the helpers (such as arch_timer_ctx_index) can work correctly at an early stage.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/include/asm/kvm_host.h |   6 ++
 arch/arm64/kvm/arch_timer.c       | 155 +++++++++++++++++++++++-------
 arch/arm64/kvm/trace_arm.h        |   8 +-
 include/kvm/arm_arch_timer.h      |  11 +--
 4 files changed, 136 insertions(+), 44 deletions(-)

+static u64 timer_get_offset(struct arch_timer_context *ctxt) {
+	struct kvm_vcpu *vcpu = ctxt->vcpu;
+
+	switch(arch_timer_ctx_index(ctxt)) {
+	case TIMER_VTIMER:
+		return __vcpu_sys_reg(vcpu, CNTVOFF_EL2);
+	default:
+		return 0;
+	}
+}
+
Can I export this helper? As in my ptp_kvm implementation I need get VCNT offset value separately not just give me a result of VCNT.

Thanks
Jianyong
Marc Zyngier Aug. 19, 2020, 10 a.m. UTC | #4
On 2020-08-19 10:24, Jianyong Wu wrote:
> Hi Marc,
> 
> -----Original Message-----
> From: kvmarm-bounces@lists.cs.columbia.edu
> <kvmarm-bounces@lists.cs.columbia.edu> On Behalf Of Marc Zyngier
> Sent: Thursday, August 6, 2020 1:57 AM
> To: Paolo Bonzini <pbonzini@redhat.com>
> Cc: Peng Hao <richard.peng@oppo.com>; kernel-team@android.com;
> kvm@vger.kernel.org; Will Deacon <will@kernel.org>; Catalin Marinas
> <Catalin.Marinas@arm.com>; Alexander Graf <graf@amazon.com>;
> kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org
> Subject: [PATCH 47/56] KVM: arm64: timers: Move timer registers to the
> sys_regs file
> 
> Move the timer gsisters to the sysreg file. This will further help
> when they are directly changed by a nesting hypervisor in the VNCR
> page.
> 
> This requires moving the initialisation of the timer struct so that
> some of the helpers (such as arch_timer_ctx_index) can work correctly
> at an early stage.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>  arch/arm64/include/asm/kvm_host.h |   6 ++
>  arch/arm64/kvm/arch_timer.c       | 155 +++++++++++++++++++++++-------
>  arch/arm64/kvm/trace_arm.h        |   8 +-
>  include/kvm/arm_arch_timer.h      |  11 +--
>  4 files changed, 136 insertions(+), 44 deletions(-)
> 
> +static u64 timer_get_offset(struct arch_timer_context *ctxt) {
> +	struct kvm_vcpu *vcpu = ctxt->vcpu;
> +
> +	switch(arch_timer_ctx_index(ctxt)) {
> +	case TIMER_VTIMER:
> +		return __vcpu_sys_reg(vcpu, CNTVOFF_EL2);
> +	default:
> +		return 0;
> +	}
> +}
> +
> Can I export this helper? As in my ptp_kvm implementation I need get
> VCNT offset value separately not just give me a result of VCNT.

Sorry, you need to give me a bit more context. What do you need
the offset for exactly?

         M.
Jianyong Wu Aug. 19, 2020, 10:18 a.m. UTC | #5
> -----Original Message-----
> From: Marc Zyngier <maz@kernel.org>
> Sent: Wednesday, August 19, 2020 6:00 PM
> To: Jianyong Wu <Jianyong.Wu@arm.com>
> Cc: Paolo Bonzini <pbonzini@redhat.com>; Peng Hao
> <richard.peng@oppo.com>; kernel-team@android.com;
> kvm@vger.kernel.org; Will Deacon <will@kernel.org>; Catalin Marinas
> <Catalin.Marinas@arm.com>; Alexander Graf <graf@amazon.com>;
> kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org
> Subject: Re: [PATCH 47/56] KVM: arm64: timers: Move timer registers to the
> sys_regs file
> 
> On 2020-08-19 10:24, Jianyong Wu wrote:
> > Hi Marc,
> >
> > -----Original Message-----
> > From: kvmarm-bounces@lists.cs.columbia.edu
> > <kvmarm-bounces@lists.cs.columbia.edu> On Behalf Of Marc Zyngier
> > Sent: Thursday, August 6, 2020 1:57 AM
> > To: Paolo Bonzini <pbonzini@redhat.com>
> > Cc: Peng Hao <richard.peng@oppo.com>; kernel-team@android.com;
> > kvm@vger.kernel.org; Will Deacon <will@kernel.org>; Catalin Marinas
> > <Catalin.Marinas@arm.com>; Alexander Graf <graf@amazon.com>;
> > kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org
> > Subject: [PATCH 47/56] KVM: arm64: timers: Move timer registers to the
> > sys_regs file
> >
> > Move the timer gsisters to the sysreg file. This will further help
> > when they are directly changed by a nesting hypervisor in the VNCR
> > page.
> >
> > This requires moving the initialisation of the timer struct so that
> > some of the helpers (such as arch_timer_ctx_index) can work correctly
> > at an early stage.
> >
> > Signed-off-by: Marc Zyngier <maz@kernel.org>
> > ---
> >  arch/arm64/include/asm/kvm_host.h |   6 ++
> >  arch/arm64/kvm/arch_timer.c       | 155 +++++++++++++++++++++++-------
> >  arch/arm64/kvm/trace_arm.h        |   8 +-
> >  include/kvm/arm_arch_timer.h      |  11 +--
> >  4 files changed, 136 insertions(+), 44 deletions(-)
> >
> > +static u64 timer_get_offset(struct arch_timer_context *ctxt) {
> > +	struct kvm_vcpu *vcpu = ctxt->vcpu;
> > +
> > +	switch(arch_timer_ctx_index(ctxt)) {
> > +	case TIMER_VTIMER:
> > +		return __vcpu_sys_reg(vcpu, CNTVOFF_EL2);
> > +	default:
> > +		return 0;
> > +	}
> > +}
> > +
> > Can I export this helper? As in my ptp_kvm implementation I need get
> > VCNT offset value separately not just give me a result of VCNT.
> 
> Sorry, you need to give me a bit more context. What do you need the offset
> for exactly?

Yeah,
In my ptp_kvm implementation, I need acquire wall time and counter cycle in the same time in host. After get host counter cycle, I need subtract it by VCNT offset to obtain VCNT. See https://lkml.org/lkml/2020/6/19/441 https://lkml.org/lkml/2020/6/19/441
But now I can't get the VCNT offset easily like before using " vcpu_vtimer(vcpu)->cntvoff" and I can't use the helper like "kvm_arm_timer_read" as I need acquire the counter cycle in the same time with the host wall time.

Thanks
Jianyong

> 
>          M.
> --
> Jazz is not dead. It just smells funny...
Marc Zyngier Aug. 19, 2020, 10:39 a.m. UTC | #6
On 2020-08-19 11:18, Jianyong Wu wrote:
>> -----Original Message-----
>> From: Marc Zyngier <maz@kernel.org>
>> Sent: Wednesday, August 19, 2020 6:00 PM
>> To: Jianyong Wu <Jianyong.Wu@arm.com>
>> Cc: Paolo Bonzini <pbonzini@redhat.com>; Peng Hao
>> <richard.peng@oppo.com>; kernel-team@android.com;
>> kvm@vger.kernel.org; Will Deacon <will@kernel.org>; Catalin Marinas
>> <Catalin.Marinas@arm.com>; Alexander Graf <graf@amazon.com>;
>> kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org
>> Subject: Re: [PATCH 47/56] KVM: arm64: timers: Move timer registers to 
>> the
>> sys_regs file
>> 
>> On 2020-08-19 10:24, Jianyong Wu wrote:
>> > Hi Marc,
>> >
>> > -----Original Message-----
>> > From: kvmarm-bounces@lists.cs.columbia.edu
>> > <kvmarm-bounces@lists.cs.columbia.edu> On Behalf Of Marc Zyngier
>> > Sent: Thursday, August 6, 2020 1:57 AM
>> > To: Paolo Bonzini <pbonzini@redhat.com>
>> > Cc: Peng Hao <richard.peng@oppo.com>; kernel-team@android.com;
>> > kvm@vger.kernel.org; Will Deacon <will@kernel.org>; Catalin Marinas
>> > <Catalin.Marinas@arm.com>; Alexander Graf <graf@amazon.com>;
>> > kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org
>> > Subject: [PATCH 47/56] KVM: arm64: timers: Move timer registers to the
>> > sys_regs file
>> >
>> > Move the timer gsisters to the sysreg file. This will further help
>> > when they are directly changed by a nesting hypervisor in the VNCR
>> > page.
>> >
>> > This requires moving the initialisation of the timer struct so that
>> > some of the helpers (such as arch_timer_ctx_index) can work correctly
>> > at an early stage.
>> >
>> > Signed-off-by: Marc Zyngier <maz@kernel.org>
>> > ---
>> >  arch/arm64/include/asm/kvm_host.h |   6 ++
>> >  arch/arm64/kvm/arch_timer.c       | 155 +++++++++++++++++++++++-------
>> >  arch/arm64/kvm/trace_arm.h        |   8 +-
>> >  include/kvm/arm_arch_timer.h      |  11 +--
>> >  4 files changed, 136 insertions(+), 44 deletions(-)
>> >
>> > +static u64 timer_get_offset(struct arch_timer_context *ctxt) {
>> > +	struct kvm_vcpu *vcpu = ctxt->vcpu;
>> > +
>> > +	switch(arch_timer_ctx_index(ctxt)) {
>> > +	case TIMER_VTIMER:
>> > +		return __vcpu_sys_reg(vcpu, CNTVOFF_EL2);
>> > +	default:
>> > +		return 0;
>> > +	}
>> > +}
>> > +
>> > Can I export this helper? As in my ptp_kvm implementation I need get
>> > VCNT offset value separately not just give me a result of VCNT.
>> 
>> Sorry, you need to give me a bit more context. What do you need the 
>> offset
>> for exactly?
> 
> Yeah,
> In my ptp_kvm implementation, I need acquire wall time and counter
> cycle in the same time in host. After get host counter cycle, I need
> subtract it by VCNT offset to obtain VCNT. See
> https://lkml.org/lkml/2020/6/19/441
> https://lkml.org/lkml/2020/6/19/441
> But now I can't get the VCNT offset easily like before using "
> vcpu_vtimer(vcpu)->cntvoff" and I can't use the helper like
> "kvm_arm_timer_read" as I need acquire the counter cycle in the same
> time with the host wall time.

I must be missing something. CNTVOFF_EL2 is now implemented as
a standard system register, and has the same visibility as any
other vcpu sysreg.

Why doesn't vcpu_read_sys_reg(vcpu, CNTVOFF_EL2) work for you?

         M.
Jianyong Wu Aug. 19, 2020, 12:58 p.m. UTC | #7
> -----Original Message-----
> From: Marc Zyngier <maz@kernel.org>
> Sent: Wednesday, August 19, 2020 6:39 PM
> To: Jianyong Wu <Jianyong.Wu@arm.com>
> Cc: Paolo Bonzini <pbonzini@redhat.com>; Peng Hao
> <richard.peng@oppo.com>; kernel-team@android.com; kvm@vger.kernel.org;
> Will Deacon <will@kernel.org>; Catalin Marinas <Catalin.Marinas@arm.com>;
> Alexander Graf <graf@amazon.com>; kvmarm@lists.cs.columbia.edu; linux-
> arm-kernel@lists.infradead.org
> Subject: Re: [PATCH 47/56] KVM: arm64: timers: Move timer registers to the
> sys_regs file
> 
> On 2020-08-19 11:18, Jianyong Wu wrote:
> >> -----Original Message-----
> >> From: Marc Zyngier <maz@kernel.org>
> >> Sent: Wednesday, August 19, 2020 6:00 PM
> >> To: Jianyong Wu <Jianyong.Wu@arm.com>
> >> Cc: Paolo Bonzini <pbonzini@redhat.com>; Peng Hao
> >> <richard.peng@oppo.com>; kernel-team@android.com;
> >> kvm@vger.kernel.org; Will Deacon <will@kernel.org>; Catalin Marinas
> >> <Catalin.Marinas@arm.com>; Alexander Graf <graf@amazon.com>;
> >> kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org
> >> Subject: Re: [PATCH 47/56] KVM: arm64: timers: Move timer registers
> >> to the sys_regs file
> >>
> >> On 2020-08-19 10:24, Jianyong Wu wrote:
> >> > Hi Marc,
> >> >
> >> > -----Original Message-----
> >> > From: kvmarm-bounces@lists.cs.columbia.edu
> >> > <kvmarm-bounces@lists.cs.columbia.edu> On Behalf Of Marc Zyngier
> >> > Sent: Thursday, August 6, 2020 1:57 AM
> >> > To: Paolo Bonzini <pbonzini@redhat.com>
> >> > Cc: Peng Hao <richard.peng@oppo.com>; kernel-team@android.com;
> >> > kvm@vger.kernel.org; Will Deacon <will@kernel.org>; Catalin Marinas
> >> > <Catalin.Marinas@arm.com>; Alexander Graf <graf@amazon.com>;
> >> > kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org
> >> > Subject: [PATCH 47/56] KVM: arm64: timers: Move timer registers to
> >> > the sys_regs file
> >> >
> >> > Move the timer gsisters to the sysreg file. This will further help
> >> > when they are directly changed by a nesting hypervisor in the VNCR
> >> > page.
> >> >
> >> > This requires moving the initialisation of the timer struct so that
> >> > some of the helpers (such as arch_timer_ctx_index) can work
> >> > correctly at an early stage.
> >> >
> >> > Signed-off-by: Marc Zyngier <maz@kernel.org>
> >> > ---
> >> >  arch/arm64/include/asm/kvm_host.h |   6 ++
> >> >  arch/arm64/kvm/arch_timer.c       | 155 +++++++++++++++++++++++-------
> >> >  arch/arm64/kvm/trace_arm.h        |   8 +-
> >> >  include/kvm/arm_arch_timer.h      |  11 +--
> >> >  4 files changed, 136 insertions(+), 44 deletions(-)
> >> >
> >> > +static u64 timer_get_offset(struct arch_timer_context *ctxt) {
> >> > +	struct kvm_vcpu *vcpu = ctxt->vcpu;
> >> > +
> >> > +	switch(arch_timer_ctx_index(ctxt)) {
> >> > +	case TIMER_VTIMER:
> >> > +		return __vcpu_sys_reg(vcpu, CNTVOFF_EL2);
> >> > +	default:
> >> > +		return 0;
> >> > +	}
> >> > +}
> >> > +
> >> > Can I export this helper? As in my ptp_kvm implementation I need
> >> > get VCNT offset value separately not just give me a result of VCNT.
> >>
> >> Sorry, you need to give me a bit more context. What do you need the
> >> offset for exactly?
> >
> > Yeah,
> > In my ptp_kvm implementation, I need acquire wall time and counter
> > cycle in the same time in host. After get host counter cycle, I need
> > subtract it by VCNT offset to obtain VCNT. See
> > https://lkml.org/lkml/2020/6/19/441
> > https://lkml.org/lkml/2020/6/19/441
> > But now I can't get the VCNT offset easily like before using "
> > vcpu_vtimer(vcpu)->cntvoff" and I can't use the helper like
> > "kvm_arm_timer_read" as I need acquire the counter cycle in the same
> > time with the host wall time.
> 
> I must be missing something. CNTVOFF_EL2 is now implemented as a standard
> system register, and has the same visibility as any other vcpu sysreg.
> 
> Why doesn't vcpu_read_sys_reg(vcpu, CNTVOFF_EL2) work for you?

Maybe it serves me, let me try it, thanks.

Thanks
Jianyong
> 
>          M.
> --
> Jazz is not dead. It just smells funny...
Eric Auger May 4, 2021, 2:47 p.m. UTC | #8
Hi David, Marc,

On 8/5/20 7:56 PM, Marc Zyngier wrote:
> From: David Brazdil <dbrazdil@google.com>
> 
> Add new folders arch/arm64/kvm/hyp/{vhe,nvhe} and Makefiles for building code
> that runs in EL2 under VHE/nVHE KVM, repsectivelly. Add an include folder for
> hyp-specific header files which will include code common to VHE/nVHE.
> 
> Build nVHE code with -D__KVM_NVHE_HYPERVISOR__, VHE code with
> -D__KVM_VHE_HYPERVISOR__.
> 
> Under nVHE compile each source file into a `.hyp.tmp.o` object first, then
> prefix all its symbols with "__kvm_nvhe_" using `objcopy` and produce
> a `.hyp.o`. Suffixes were chosen so that it would be possible for VHE and nVHE
> to share some source files, but compiled with different CFLAGS.
> 
> The nVHE ELF symbol prefix is added to kallsyms.c as ignored. EL2-only symbols
> will never appear in EL1 stack traces.
> 
> Due to symbol prefixing, add a section in image-vars.h for aliases of symbols
> that are defined in nVHE EL2 and accessed by kernel in EL1 or vice versa.
> 
> Signed-off-by: David Brazdil <dbrazdil@google.com>
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> Link: https://lore.kernel.org/r/20200625131420.71444-4-dbrazdil@google.com
> ---
>  arch/arm64/kernel/image-vars.h   | 14 +++++++++++++
>  arch/arm64/kvm/hyp/Makefile      | 10 +++++++---
>  arch/arm64/kvm/hyp/nvhe/Makefile | 34 ++++++++++++++++++++++++++++++++
>  arch/arm64/kvm/hyp/vhe/Makefile  | 17 ++++++++++++++++
>  scripts/kallsyms.c               |  1 +
>  5 files changed, 73 insertions(+), 3 deletions(-)
>  create mode 100644 arch/arm64/kvm/hyp/nvhe/Makefile
>  create mode 100644 arch/arm64/kvm/hyp/vhe/Makefile
> 
> diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h
> index be0a63ffed23..3dc27da47712 100644
> --- a/arch/arm64/kernel/image-vars.h
> +++ b/arch/arm64/kernel/image-vars.h
> @@ -51,4 +51,18 @@ __efistub__ctype		= _ctype;
>  
>  #endif
>  
> +#ifdef CONFIG_KVM
> +
> +/*
> + * KVM nVHE code has its own symbol namespace prefixed with __kvm_nvhe_, to
> + * separate it from the kernel proper. The following symbols are legally
> + * accessed by it, therefore provide aliases to make them linkable.
> + * Do not include symbols which may not be safely accessed under hypervisor
> + * memory mappings.
> + */
> +
> +#define KVM_NVHE_ALIAS(sym) __kvm_nvhe_##sym = sym;
> +
> +#endif /* CONFIG_KVM */
> +
>  #endif /* __ARM64_KERNEL_IMAGE_VARS_H */
> diff --git a/arch/arm64/kvm/hyp/Makefile b/arch/arm64/kvm/hyp/Makefile
> index 5d8357ddc234..9c5dfe6ff80b 100644
> --- a/arch/arm64/kvm/hyp/Makefile
> +++ b/arch/arm64/kvm/hyp/Makefile
> @@ -3,10 +3,14 @@
>  # Makefile for Kernel-based Virtual Machine module, HYP part
>  #
>  
> -ccflags-y += -fno-stack-protector -DDISABLE_BRANCH_PROFILING \
> -		$(DISABLE_STACKLEAK_PLUGIN)
> +incdir := $(srctree)/$(src)/include
> +subdir-asflags-y := -I$(incdir)
> +subdir-ccflags-y := -I$(incdir)				\
> +		    -fno-stack-protector		\
> +		    -DDISABLE_BRANCH_PROFILING		\
> +		    $(DISABLE_STACKLEAK_PLUGIN)
>  
> -obj-$(CONFIG_KVM) += hyp.o
> +obj-$(CONFIG_KVM) += hyp.o nvhe/
>  obj-$(CONFIG_KVM_INDIRECT_VECTORS) += smccc_wa.o
>  
>  hyp-y := vgic-v3-sr.o timer-sr.o aarch32.o vgic-v2-cpuif-proxy.o sysreg-sr.o \
> diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile
> new file mode 100644
> index 000000000000..955f4188e00f
> --- /dev/null
> +++ b/arch/arm64/kvm/hyp/nvhe/Makefile
> @@ -0,0 +1,34 @@
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# Makefile for Kernel-based Virtual Machine module, HYP/nVHE part
> +#
> +
> +asflags-y := -D__KVM_NVHE_HYPERVISOR__
> +ccflags-y := -D__KVM_NVHE_HYPERVISOR__
> +
> +obj-y :=
> +
> +obj-y := $(patsubst %.o,%.hyp.o,$(obj-y))
> +extra-y := $(patsubst %.hyp.o,%.hyp.tmp.o,$(obj-y))
> +
> +$(obj)/%.hyp.tmp.o: $(src)/%.c FORCE
> +	$(call if_changed_rule,cc_o_c)
> +$(obj)/%.hyp.tmp.o: $(src)/%.S FORCE
> +	$(call if_changed_rule,as_o_S)
> +$(obj)/%.hyp.o: $(obj)/%.hyp.tmp.o FORCE
> +	$(call if_changed,hypcopy)
> +
> +quiet_cmd_hypcopy = HYPCOPY $@
> +      cmd_hypcopy = $(OBJCOPY) --prefix-symbols=__kvm_nvhe_ $< $@
> +
> +# KVM nVHE code is run at a different exception code with a different map, so
> +# compiler instrumentation that inserts callbacks or checks into the code may
> +# cause crashes. Just disable it.
> +GCOV_PROFILE	:= n
> +KASAN_SANITIZE	:= n
> +UBSAN_SANITIZE	:= n
> +KCOV_INSTRUMENT	:= n
> +
> +# Skip objtool checking for this directory because nVHE code is compiled with
> +# non-standard build rules.
> +OBJECT_FILES_NON_STANDARD := y
> diff --git a/arch/arm64/kvm/hyp/vhe/Makefile b/arch/arm64/kvm/hyp/vhe/Makefile
> new file mode 100644
> index 000000000000..e04375546081
> --- /dev/null
> +++ b/arch/arm64/kvm/hyp/vhe/Makefile
> @@ -0,0 +1,17 @@
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# Makefile for Kernel-based Virtual Machine module, HYP/nVHE part
> +#
> +
> +asflags-y := -D__KVM_VHE_HYPERVISOR__
> +ccflags-y := -D__KVM_VHE_HYPERVISOR__
> +
> +obj-y :=
> +
> +# KVM code is run at a different exception code with a different map, so
> +# compiler instrumentation that inserts callbacks or checks into the code may
> +# cause crashes. Just disable it.
> +GCOV_PROFILE	:= n
> +KASAN_SANITIZE	:= n
> +UBSAN_SANITIZE	:= n
> +KCOV_INSTRUMENT	:= n
> diff --git a/scripts/kallsyms.c b/scripts/kallsyms.c
> index 6dc3078649fa..0096cd965332 100644
> --- a/scripts/kallsyms.c
> +++ b/scripts/kallsyms.c
> @@ -109,6 +109,7 @@ static bool is_ignored_symbol(const char *name, char type)
>  		".LASANPC",		/* s390 kasan local symbols */
>  		"__crc_",		/* modversions */
>  		"__efistub_",		/* arm64 EFI stub namespace */
> +		"__kvm_nvhe_",		/* arm64 non-VHE KVM namespace */
The addition of this line seems to have introduced errors on the
'vmlinux symtab matches kallsyms' perf test (perf test -v 1) which fails
on aarch64 for all __kvm_nvhe_ prefixed symbols, like

ERR : <addr> : __kvm_nvhe___invalid not on kallsyms
ERR : <addr> : __kvm_nvhe___do_hyp_init not on kallsyms
ERR : <addr> : __kvm_nvhe___kvm_handle_stub_hvc not on kallsyms
ERR : <addr> : __kvm_nvhe_reset not on kallsyms
../..

I understand we willingly hided those symbols from /proc/kallsyms. Do
you confirm the right fix is to upgrade the perf test suite accordingly?

Thanks

Eric


>  		NULL
>  	};
>  
>
Eric Auger May 10, 2021, 8:43 a.m. UTC | #9
Hi Marc,

On 5/5/21 8:03 PM, Marc Zyngier wrote:
> Hi Eric,
> 
> On Tue, 04 May 2021 15:47:36 +0100,
> Auger Eric <eric.auger@redhat.com> wrote:
>>
>> Hi David, Marc,
>>
>> On 8/5/20 7:56 PM, Marc Zyngier wrote:
>>> From: David Brazdil <dbrazdil@google.com>
>>>
>>> Add new folders arch/arm64/kvm/hyp/{vhe,nvhe} and Makefiles for building code
>>> that runs in EL2 under VHE/nVHE KVM, repsectivelly. Add an include folder for
>>> hyp-specific header files which will include code common to VHE/nVHE.
>>>
>>> Build nVHE code with -D__KVM_NVHE_HYPERVISOR__, VHE code with
>>> -D__KVM_VHE_HYPERVISOR__.
>>>
>>> Under nVHE compile each source file into a `.hyp.tmp.o` object first, then
>>> prefix all its symbols with "__kvm_nvhe_" using `objcopy` and produce
>>> a `.hyp.o`. Suffixes were chosen so that it would be possible for VHE and nVHE
>>> to share some source files, but compiled with different CFLAGS.
>>>
>>> The nVHE ELF symbol prefix is added to kallsyms.c as ignored. EL2-only symbols
>>> will never appear in EL1 stack traces.
>>>
>>> Due to symbol prefixing, add a section in image-vars.h for aliases of symbols
>>> that are defined in nVHE EL2 and accessed by kernel in EL1 or vice versa.
>>>
>>> Signed-off-by: David Brazdil <dbrazdil@google.com>
>>> Signed-off-by: Marc Zyngier <maz@kernel.org>
>>> Link: https://lore.kernel.org/r/20200625131420.71444-4-dbrazdil@google.com
>>> ---
>>>  arch/arm64/kernel/image-vars.h   | 14 +++++++++++++
>>>  arch/arm64/kvm/hyp/Makefile      | 10 +++++++---
>>>  arch/arm64/kvm/hyp/nvhe/Makefile | 34 ++++++++++++++++++++++++++++++++
>>>  arch/arm64/kvm/hyp/vhe/Makefile  | 17 ++++++++++++++++
>>>  scripts/kallsyms.c               |  1 +
>>>  5 files changed, 73 insertions(+), 3 deletions(-)
>>>  create mode 100644 arch/arm64/kvm/hyp/nvhe/Makefile
>>>  create mode 100644 arch/arm64/kvm/hyp/vhe/Makefile
> 
> [...]
> 
>>> diff --git a/scripts/kallsyms.c b/scripts/kallsyms.c
>>> index 6dc3078649fa..0096cd965332 100644
>>> --- a/scripts/kallsyms.c
>>> +++ b/scripts/kallsyms.c
>>> @@ -109,6 +109,7 @@ static bool is_ignored_symbol(const char *name, char type)
>>>  		".LASANPC",		/* s390 kasan local symbols */
>>>  		"__crc_",		/* modversions */
>>>  		"__efistub_",		/* arm64 EFI stub namespace */
>>> +		"__kvm_nvhe_",		/* arm64 non-VHE KVM namespace */
>> The addition of this line seems to have introduced errors on the
>> 'vmlinux symtab matches kallsyms' perf test (perf test -v 1) which fails
>> on aarch64 for all __kvm_nvhe_ prefixed symbols, like
>>
>> ERR : <addr> : __kvm_nvhe___invalid not on kallsyms
>> ERR : <addr> : __kvm_nvhe___do_hyp_init not on kallsyms
>> ERR : <addr> : __kvm_nvhe___kvm_handle_stub_hvc not on kallsyms
>> ERR : <addr> : __kvm_nvhe_reset not on kallsyms
>> ../..
>>
>> I understand we willingly hided those symbols from /proc/kallsyms. Do
>> you confirm the right fix is to upgrade the perf test suite accordingly?
> 
> Hmmm. This test always fail here, no matter whether I have this line
> or not:

sorry Marc I missed your reply,
> 
> <quote>
> maz@big-leg-emma:~$ sudo perf_5.10 test -v 1
>  1: vmlinux symtab matches kallsyms                                 :
> --- start ---
> test child forked, pid 664
> /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module!
> /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module!
> /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module!
> /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module!
> /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module!
> /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module!
> /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module!
> /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module!
> /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module!
> /proc/{kallsyms,modules} inconsistency while looking for "[bpf]" module!
> Looking at the vmlinux_path (8 entries long)
> symsrc__init: cannot get elf header.
> symsrc__init: cannot get elf header.
> Couldn't find a vmlinux that matches the kernel running on this machine, skipping test
> test child finished with -2
> ---- end ----
> vmlinux symtab matches kallsyms: Skip
> </quote>
> 
> Rookie question: How do you provide a kernel to the test framework?
On my end I compiled and installed a kernel and compiled perf from that
same kernel. Should work. I don't know how to point to a specific vmlinux

Thanks

Eric
> 
> Thanks,
> 
> 	M.
>
Eric Auger May 18, 2021, 11:48 a.m. UTC | #10
Hi David, Marc,

On 5/4/21 4:47 PM, Auger Eric wrote:
> Hi David, Marc,
> 
> On 8/5/20 7:56 PM, Marc Zyngier wrote:
>> From: David Brazdil <dbrazdil@google.com>
>>
>> Add new folders arch/arm64/kvm/hyp/{vhe,nvhe} and Makefiles for building code
>> that runs in EL2 under VHE/nVHE KVM, repsectivelly. Add an include folder for
>> hyp-specific header files which will include code common to VHE/nVHE.
>>
>> Build nVHE code with -D__KVM_NVHE_HYPERVISOR__, VHE code with
>> -D__KVM_VHE_HYPERVISOR__.
>>
>> Under nVHE compile each source file into a `.hyp.tmp.o` object first, then
>> prefix all its symbols with "__kvm_nvhe_" using `objcopy` and produce
>> a `.hyp.o`. Suffixes were chosen so that it would be possible for VHE and nVHE
>> to share some source files, but compiled with different CFLAGS.
>>
>> The nVHE ELF symbol prefix is added to kallsyms.c as ignored. EL2-only symbols
>> will never appear in EL1 stack traces.
>>
>> Due to symbol prefixing, add a section in image-vars.h for aliases of symbols
>> that are defined in nVHE EL2 and accessed by kernel in EL1 or vice versa.
>>
>> Signed-off-by: David Brazdil <dbrazdil@google.com>
>> Signed-off-by: Marc Zyngier <maz@kernel.org>
>> Link: https://lore.kernel.org/r/20200625131420.71444-4-dbrazdil@google.com
>> ---
>>  arch/arm64/kernel/image-vars.h   | 14 +++++++++++++
>>  arch/arm64/kvm/hyp/Makefile      | 10 +++++++---
>>  arch/arm64/kvm/hyp/nvhe/Makefile | 34 ++++++++++++++++++++++++++++++++
>>  arch/arm64/kvm/hyp/vhe/Makefile  | 17 ++++++++++++++++
>>  scripts/kallsyms.c               |  1 +
>>  5 files changed, 73 insertions(+), 3 deletions(-)
>>  create mode 100644 arch/arm64/kvm/hyp/nvhe/Makefile
>>  create mode 100644 arch/arm64/kvm/hyp/vhe/Makefile
>>
>> diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h
>> index be0a63ffed23..3dc27da47712 100644
>> --- a/arch/arm64/kernel/image-vars.h
>> +++ b/arch/arm64/kernel/image-vars.h
>> @@ -51,4 +51,18 @@ __efistub__ctype		= _ctype;
>>  
>>  #endif
>>  
>> +#ifdef CONFIG_KVM
>> +
>> +/*
>> + * KVM nVHE code has its own symbol namespace prefixed with __kvm_nvhe_, to
>> + * separate it from the kernel proper. The following symbols are legally
>> + * accessed by it, therefore provide aliases to make them linkable.
>> + * Do not include symbols which may not be safely accessed under hypervisor
>> + * memory mappings.
>> + */
>> +
>> +#define KVM_NVHE_ALIAS(sym) __kvm_nvhe_##sym = sym;
>> +
>> +#endif /* CONFIG_KVM */
>> +
>>  #endif /* __ARM64_KERNEL_IMAGE_VARS_H */
>> diff --git a/arch/arm64/kvm/hyp/Makefile b/arch/arm64/kvm/hyp/Makefile
>> index 5d8357ddc234..9c5dfe6ff80b 100644
>> --- a/arch/arm64/kvm/hyp/Makefile
>> +++ b/arch/arm64/kvm/hyp/Makefile
>> @@ -3,10 +3,14 @@
>>  # Makefile for Kernel-based Virtual Machine module, HYP part
>>  #
>>  
>> -ccflags-y += -fno-stack-protector -DDISABLE_BRANCH_PROFILING \
>> -		$(DISABLE_STACKLEAK_PLUGIN)
>> +incdir := $(srctree)/$(src)/include
>> +subdir-asflags-y := -I$(incdir)
>> +subdir-ccflags-y := -I$(incdir)				\
>> +		    -fno-stack-protector		\
>> +		    -DDISABLE_BRANCH_PROFILING		\
>> +		    $(DISABLE_STACKLEAK_PLUGIN)
>>  
>> -obj-$(CONFIG_KVM) += hyp.o
>> +obj-$(CONFIG_KVM) += hyp.o nvhe/
>>  obj-$(CONFIG_KVM_INDIRECT_VECTORS) += smccc_wa.o
>>  
>>  hyp-y := vgic-v3-sr.o timer-sr.o aarch32.o vgic-v2-cpuif-proxy.o sysreg-sr.o \
>> diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile
>> new file mode 100644
>> index 000000000000..955f4188e00f
>> --- /dev/null
>> +++ b/arch/arm64/kvm/hyp/nvhe/Makefile
>> @@ -0,0 +1,34 @@
>> +# SPDX-License-Identifier: GPL-2.0
>> +#
>> +# Makefile for Kernel-based Virtual Machine module, HYP/nVHE part
>> +#
>> +
>> +asflags-y := -D__KVM_NVHE_HYPERVISOR__
>> +ccflags-y := -D__KVM_NVHE_HYPERVISOR__
>> +
>> +obj-y :=
>> +
>> +obj-y := $(patsubst %.o,%.hyp.o,$(obj-y))
>> +extra-y := $(patsubst %.hyp.o,%.hyp.tmp.o,$(obj-y))
>> +
>> +$(obj)/%.hyp.tmp.o: $(src)/%.c FORCE
>> +	$(call if_changed_rule,cc_o_c)
>> +$(obj)/%.hyp.tmp.o: $(src)/%.S FORCE
>> +	$(call if_changed_rule,as_o_S)
>> +$(obj)/%.hyp.o: $(obj)/%.hyp.tmp.o FORCE
>> +	$(call if_changed,hypcopy)
>> +
>> +quiet_cmd_hypcopy = HYPCOPY $@
>> +      cmd_hypcopy = $(OBJCOPY) --prefix-symbols=__kvm_nvhe_ $< $@
>> +
>> +# KVM nVHE code is run at a different exception code with a different map, so
>> +# compiler instrumentation that inserts callbacks or checks into the code may
>> +# cause crashes. Just disable it.
>> +GCOV_PROFILE	:= n
>> +KASAN_SANITIZE	:= n
>> +UBSAN_SANITIZE	:= n
>> +KCOV_INSTRUMENT	:= n
>> +
>> +# Skip objtool checking for this directory because nVHE code is compiled with
>> +# non-standard build rules.
>> +OBJECT_FILES_NON_STANDARD := y
>> diff --git a/arch/arm64/kvm/hyp/vhe/Makefile b/arch/arm64/kvm/hyp/vhe/Makefile
>> new file mode 100644
>> index 000000000000..e04375546081
>> --- /dev/null
>> +++ b/arch/arm64/kvm/hyp/vhe/Makefile
>> @@ -0,0 +1,17 @@
>> +# SPDX-License-Identifier: GPL-2.0
>> +#
>> +# Makefile for Kernel-based Virtual Machine module, HYP/nVHE part
>> +#
>> +
>> +asflags-y := -D__KVM_VHE_HYPERVISOR__
>> +ccflags-y := -D__KVM_VHE_HYPERVISOR__
>> +
>> +obj-y :=
>> +
>> +# KVM code is run at a different exception code with a different map, so
>> +# compiler instrumentation that inserts callbacks or checks into the code may
>> +# cause crashes. Just disable it.
>> +GCOV_PROFILE	:= n
>> +KASAN_SANITIZE	:= n
>> +UBSAN_SANITIZE	:= n
>> +KCOV_INSTRUMENT	:= n
>> diff --git a/scripts/kallsyms.c b/scripts/kallsyms.c
>> index 6dc3078649fa..0096cd965332 100644
>> --- a/scripts/kallsyms.c
>> +++ b/scripts/kallsyms.c
>> @@ -109,6 +109,7 @@ static bool is_ignored_symbol(const char *name, char type)
>>  		".LASANPC",		/* s390 kasan local symbols */
>>  		"__crc_",		/* modversions */
>>  		"__efistub_",		/* arm64 EFI stub namespace */
>> +		"__kvm_nvhe_",		/* arm64 non-VHE KVM namespace */
> The addition of this line seems to have introduced errors on the
> 'vmlinux symtab matches kallsyms' perf test (perf test -v 1) which fails
> on aarch64 for all __kvm_nvhe_ prefixed symbols, like
> 
> ERR : <addr> : __kvm_nvhe___invalid not on kallsyms
> ERR : <addr> : __kvm_nvhe___do_hyp_init not on kallsyms
> ERR : <addr> : __kvm_nvhe___kvm_handle_stub_hvc not on kallsyms
> ERR : <addr> : __kvm_nvhe_reset not on kallsyms
> ../..
> 
> I understand we willingly hided those symbols from /proc/kallsyms. Do
> you confirm the right fix is to upgrade the perf test suite accordingly?

Were you eventually able to reproduce?

Thanks

Eric
> 
> Thanks
> 
> Eric
> 
> 
>>  		NULL
>>  	};
>>  
>>
Marc Zyngier May 18, 2021, 11:57 a.m. UTC | #11
Hi Eric,

On Tue, 18 May 2021 12:48:41 +0100,
Auger Eric <eric.auger@redhat.com> wrote:
> 
> Hi David, Marc,
>

[...]

> >> --- a/scripts/kallsyms.c
> >> +++ b/scripts/kallsyms.c
> >> @@ -109,6 +109,7 @@ static bool is_ignored_symbol(const char *name, char type)
> >>  		".LASANPC",		/* s390 kasan local symbols */
> >>  		"__crc_",		/* modversions */
> >>  		"__efistub_",		/* arm64 EFI stub namespace */
> >> +		"__kvm_nvhe_",		/* arm64 non-VHE KVM namespace */
> > The addition of this line seems to have introduced errors on the
> > 'vmlinux symtab matches kallsyms' perf test (perf test -v 1) which fails
> > on aarch64 for all __kvm_nvhe_ prefixed symbols, like
> > 
> > ERR : <addr> : __kvm_nvhe___invalid not on kallsyms
> > ERR : <addr> : __kvm_nvhe___do_hyp_init not on kallsyms
> > ERR : <addr> : __kvm_nvhe___kvm_handle_stub_hvc not on kallsyms
> > ERR : <addr> : __kvm_nvhe_reset not on kallsyms
> > ../..
> > 
> > I understand we willingly hided those symbols from /proc/kallsyms. Do
> > you confirm the right fix is to upgrade the perf test suite accordingly?
> 
> Were you eventually able to reproduce?

Unfortunately not, I always end-up with the test failing even if I add
the symbols back. I must be doing something wrong...

	M.