[v6,0/8] KVM perf support

This series extends perf support for KVM. The KVM implementation relies
on the SBI PMU extension and trap n emulation of hpmcounter CSRs.
The KVM implementation exposes the virtual counters to the guest and internally
manage the counters using kernel perf counters. 

This series doesn't support the counter overflow as the Sscofpmf extension
doesn't allow trap & emulation mechanism of scountovf CSR yet. The required
changes to allow that are being under discussions. Supporting overflow interrupt
also requires AIA interrupt filtering support.

perf stat works in kvm guests with this series. 

Here is example of running perf stat in a guest running in KVM.

===========================================================================
/ # /host/apps/perf stat -e instructions -e cycles -e r8000000000000005 \
> -e r8000000000000006 -e r8000000000000007 -e r8000000000000008 \
> -e r800000000000000a perf bench sched messaging -g 10 -l 10

# Running 'sched/messaging' benchmark:
# 20 sender and receiver processes per group
# 10 groups == 400 processes run

     Total time: 7.769 [sec]
                 
 Performance counter stats for 'perf bench sched messaging -g 10 -l 10':

       73556259604      cycles
       73387266056      instructions              #    1.00  insn per cycle
                 0      dTLB-store-misses
                 0      iTLB-load-misses
                 0      r8000000000000005
              2595      r8000000000000006
              2272      r8000000000000007
                10      r8000000000000008
                 0      r800000000000000a

      12.173720400 seconds time elapsed

       1.002716000 seconds user
      21.931047000 seconds sys


Note: The SBI_PMU_FW_SET_TIMER (eventid : r8000000000000005) is zero
as kvm guest supports sstc now. 

This series can be found here as well.
https://github.com/atishp04/linux/tree/kvm_perf_v6

TODO:
1. Add sscofpmf support.
2. Add One reg interface for the following operations:
	1. Enable/Disable PMU (should it at VM level rather than vcpu ?)
	2. Number of hpmcounter and width of the counters
	3. Init PMU
	4. Allow guest user to access cycle & instret without trapping
3. Move counter mask to a bitmask instead of unsigned long so that it can work
   for RV32 systems where number of total counters are more than 32.
   This will also accomodate future systems which may define maximum counters
   to be more than 64. 

Changes from v5->v6:
1. Addressed remainig RB tags.
2. Rebased on kvm-queue which already has merged first 6 patches from v5.
3. Added static time compilation checks for max counters.
4. Addressed other nit comments.

Changes from v4->v5:
1. Few checkpatch --strict error fixes.
2. Some other minor nit comment addressed.
3. Fixed an issue around counter indexing.

Changes from v3->v4:
1. Addressed all the comments on v3.
2. Modified the vcpu_pmu_init to void return type.
3. Redirect illegal instruction trap to guest for invalid hpmcounter access
   instead of exiting to the userpsace.
4. Got rid of unecessary error messages.

Changes v2->v3:
1. Changed the exported functions to GPL only export.
2. Addressed all the nit comments on v2.
3. Split non-kvm related changes into separate patches.
4. Reorgainze the PATCH 11 and 10 based on Drew's suggestions.

Changes from v1->v2:
1. Addressed comments from Andrew.
2. Removed kvpmu sanity check.
3. Added a kvm pmu init flag and the sanity check to probe function.
4. Improved the linux vs sbi error code handling.
 

Atish Patra (8):
RISC-V: KVM: Add skeleton support for perf
RISC-V: KVM: Add SBI PMU extension support
RISC-V: KVM: Make PMU functionality depend on Sscofpmf
RISC-V: KVM: Disable all hpmcounter access for VS/VU mode
RISC-V: KVM: Implement trap & emulate for hpmcounters
RISC-V: KVM: Implement perf support without sampling
RISC-V: KVM: Support firmware events
RISC-V: KVM: Increment firmware pmu events

arch/riscv/include/asm/kvm_host.h     |   4 +
arch/riscv/include/asm/kvm_vcpu_pmu.h | 108 +++++
arch/riscv/kvm/Makefile               |   1 +
arch/riscv/kvm/main.c                 |   3 +-
arch/riscv/kvm/tlb.c                  |   4 +
arch/riscv/kvm/vcpu.c                 |   7 +
arch/riscv/kvm/vcpu_insn.c            |   4 +-
arch/riscv/kvm/vcpu_pmu.c             | 630 ++++++++++++++++++++++++++
arch/riscv/kvm/vcpu_sbi.c             |  11 +
arch/riscv/kvm/vcpu_sbi_pmu.c         |  86 ++++
arch/riscv/kvm/vcpu_sbi_replace.c     |   7 +
11 files changed, 863 insertions(+), 2 deletions(-)
create mode 100644 arch/riscv/include/asm/kvm_vcpu_pmu.h
create mode 100644 arch/riscv/kvm/vcpu_pmu.c
create mode 100644 arch/riscv/kvm/vcpu_sbi_pmu.c

--
2.25.1

On Tue, Feb 7, 2023 at 3:26 PM Atish Patra <atishp@rivosinc.com> wrote:
>
> This series extends perf support for KVM. The KVM implementation relies
> on the SBI PMU extension and trap n emulation of hpmcounter CSRs.
> The KVM implementation exposes the virtual counters to the guest and internally
> manage the counters using kernel perf counters.
>
> This series doesn't support the counter overflow as the Sscofpmf extension
> doesn't allow trap & emulation mechanism of scountovf CSR yet. The required
> changes to allow that are being under discussions. Supporting overflow interrupt
> also requires AIA interrupt filtering support.
>
> perf stat works in kvm guests with this series.
>
> Here is example of running perf stat in a guest running in KVM.
>
> ===========================================================================
> / # /host/apps/perf stat -e instructions -e cycles -e r8000000000000005 \
> > -e r8000000000000006 -e r8000000000000007 -e r8000000000000008 \
> > -e r800000000000000a perf bench sched messaging -g 10 -l 10
>
> # Running 'sched/messaging' benchmark:
> # 20 sender and receiver processes per group
> # 10 groups == 400 processes run
>
>      Total time: 7.769 [sec]
>
>  Performance counter stats for 'perf bench sched messaging -g 10 -l 10':
>
>        73556259604      cycles
>        73387266056      instructions              #    1.00  insn per cycle
>                  0      dTLB-store-misses
>                  0      iTLB-load-misses
>                  0      r8000000000000005
>               2595      r8000000000000006
>               2272      r8000000000000007
>                 10      r8000000000000008
>                  0      r800000000000000a
>
>       12.173720400 seconds time elapsed
>
>        1.002716000 seconds user
>       21.931047000 seconds sys
>
>
> Note: The SBI_PMU_FW_SET_TIMER (eventid : r8000000000000005) is zero
> as kvm guest supports sstc now.
>
> This series can be found here as well.
> https://github.com/atishp04/linux/tree/kvm_perf_v6
>
> TODO:
> 1. Add sscofpmf support.
> 2. Add One reg interface for the following operations:
>         1. Enable/Disable PMU (should it at VM level rather than vcpu ?)
>         2. Number of hpmcounter and width of the counters
>         3. Init PMU
>         4. Allow guest user to access cycle & instret without trapping
> 3. Move counter mask to a bitmask instead of unsigned long so that it can work
>    for RV32 systems where number of total counters are more than 32.
>    This will also accomodate future systems which may define maximum counters
>    to be more than 64.
>
> Changes from v5->v6:
> 1. Addressed remainig RB tags.
> 2. Rebased on kvm-queue which already has merged first 6 patches from v5.
> 3. Added static time compilation checks for max counters.
> 4. Addressed other nit comments.
>
> Changes from v4->v5:
> 1. Few checkpatch --strict error fixes.
> 2. Some other minor nit comment addressed.
> 3. Fixed an issue around counter indexing.
>
> Changes from v3->v4:
> 1. Addressed all the comments on v3.
> 2. Modified the vcpu_pmu_init to void return type.
> 3. Redirect illegal instruction trap to guest for invalid hpmcounter access
>    instead of exiting to the userpsace.
> 4. Got rid of unecessary error messages.
>
> Changes v2->v3:
> 1. Changed the exported functions to GPL only export.
> 2. Addressed all the nit comments on v2.
> 3. Split non-kvm related changes into separate patches.
> 4. Reorgainze the PATCH 11 and 10 based on Drew's suggestions.
>
> Changes from v1->v2:
> 1. Addressed comments from Andrew.
> 2. Removed kvpmu sanity check.
> 3. Added a kvm pmu init flag and the sanity check to probe function.
> 4. Improved the linux vs sbi error code handling.
>
>
> Atish Patra (8):
> RISC-V: KVM: Add skeleton support for perf
> RISC-V: KVM: Add SBI PMU extension support
> RISC-V: KVM: Make PMU functionality depend on Sscofpmf
> RISC-V: KVM: Disable all hpmcounter access for VS/VU mode
> RISC-V: KVM: Implement trap & emulate for hpmcounters
> RISC-V: KVM: Implement perf support without sampling
> RISC-V: KVM: Support firmware events
> RISC-V: KVM: Increment firmware pmu events

I have queued this series for Linux-6.3

Thanks,
Anup

>
> arch/riscv/include/asm/kvm_host.h     |   4 +
> arch/riscv/include/asm/kvm_vcpu_pmu.h | 108 +++++
> arch/riscv/kvm/Makefile               |   1 +
> arch/riscv/kvm/main.c                 |   3 +-
> arch/riscv/kvm/tlb.c                  |   4 +
> arch/riscv/kvm/vcpu.c                 |   7 +
> arch/riscv/kvm/vcpu_insn.c            |   4 +-
> arch/riscv/kvm/vcpu_pmu.c             | 630 ++++++++++++++++++++++++++
> arch/riscv/kvm/vcpu_sbi.c             |  11 +
> arch/riscv/kvm/vcpu_sbi_pmu.c         |  86 ++++
> arch/riscv/kvm/vcpu_sbi_replace.c     |   7 +
> 11 files changed, 863 insertions(+), 2 deletions(-)
> create mode 100644 arch/riscv/include/asm/kvm_vcpu_pmu.h
> create mode 100644 arch/riscv/kvm/vcpu_pmu.c
> create mode 100644 arch/riscv/kvm/vcpu_sbi_pmu.c
>
> --
> 2.25.1
>