From patchwork Wed Mar 30 23:51:49 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kamal Mostafa X-Patchwork-Id: 603930 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) by ozlabs.org (Postfix) with ESMTP id 3qb52929DRz9s5g; Thu, 31 Mar 2016 11:27:45 +1100 (AEDT) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.76) (envelope-from ) id 1alQSa-00089a-Jy; Thu, 31 Mar 2016 00:27:40 +0000 Received: from youngberry.canonical.com ([91.189.89.112]) by huckleberry.canonical.com with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.76) (envelope-from ) id 1alPtx-0004OD-OF for kernel-team@lists.ubuntu.com; Wed, 30 Mar 2016 23:51:53 +0000 Received: from 1.general.kamal.us.vpn ([10.172.68.52] helo=fourier) by youngberry.canonical.com with esmtpsa (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.76) (envelope-from ) id 1alPtx-0007Rk-2d; Wed, 30 Mar 2016 23:51:53 +0000 Received: from kamal by fourier with local (Exim 4.86_2) (envelope-from ) id 1alPtu-0000fv-CM; Wed, 30 Mar 2016 16:51:50 -0700 From: Kamal Mostafa To: Kan Liang Subject: [4.2.y-ckt stable] Patch "perf/x86/intel: Fix PEBS warning by only restoring active PMU in pmi" has been added to the 4.2.y-ckt tree Date: Wed, 30 Mar 2016 16:51:49 -0700 Message-Id: <1459381909-2553-1-git-send-email-kamal@canonical.com> X-Mailer: git-send-email 2.7.4 X-Extended-Stable: 4.2 X-Mailman-Approved-At: Thu, 31 Mar 2016 00:26:57 +0000 Cc: Vince Weaver , Peter Zijlstra , Linus Torvalds , Kamal Mostafa , Stephane Eranian , Arnaldo Carvalho de Melo , Alexander Shishkin , kernel-team@lists.ubuntu.com, Thomas Gleixner , Jiri Olsa , Ingo Molnar X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.14 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: kernel-team-bounces@lists.ubuntu.com This is a note to let you know that I have just added a patch titled perf/x86/intel: Fix PEBS warning by only restoring active PMU in pmi to the linux-4.2.y-queue branch of the 4.2.y-ckt extended stable tree which can be found at: http://kernel.ubuntu.com/git/ubuntu/linux.git/log/?h=linux-4.2.y-queue This patch is scheduled to be released in version 4.2.8-ckt7. If you, or anyone else, feels it should not be added to this tree, please reply to this email. For more information about the 4.2.y-ckt tree, see https://wiki.ubuntu.com/Kernel/Dev/ExtendedStable Thanks. -Kamal ---8<------------------------------------------------------------ From 2a659dd551e1f0ea4e287e10c848a40d9319cfd8 Mon Sep 17 00:00:00 2001 From: Kan Liang Date: Thu, 3 Mar 2016 18:07:28 -0500 Subject: perf/x86/intel: Fix PEBS warning by only restoring active PMU in pmi commit c3d266c8a9838cc141b69548bc3b1b18808ae8c4 upstream. This patch tries to fix a PEBS warning found in my stress test. The following perf command can easily trigger the pebs warning or spurious NMI error on Skylake/Broadwell/Haswell platforms: sudo perf record -e 'cpu/umask=0x04,event=0xc4/pp,cycles,branches,ref-cycles,cache-misses,cache-references' --call-graph fp -b -c1000 -a Also the NMI watchdog must be enabled. For this case, the events number is larger than counter number. So perf has to do multiplexing. In perf_mux_hrtimer_handler, it does perf_pmu_disable(), schedule out old events, rotate_ctx, schedule in new events and finally perf_pmu_enable(). If the old events include precise event, the MSR_IA32_PEBS_ENABLE should be cleared when perf_pmu_disable(). The MSR_IA32_PEBS_ENABLE should keep 0 until the perf_pmu_enable() is called and the new event is precise event. However, there is a corner case which could restore PEBS_ENABLE to stale value during the above period. In perf_pmu_disable(), GLOBAL_CTRL will be set to 0 to stop overflow and followed PMI. But there may be pending PMI from an earlier overflow, which cannot be stopped. So even GLOBAL_CTRL is cleared, the kernel still be possible to get PMI. At the end of the PMI handler, __intel_pmu_enable_all() will be called, which will restore the stale values if old events haven't scheduled out. Once the stale pebs value is set, it's impossible to be corrected if the new events are non-precise. Because the pebs_enabled will be set to 0. x86_pmu.enable_all() will ignore the MSR_IA32_PEBS_ENABLE setting. As a result, the following NMI with stale PEBS_ENABLE trigger pebs warning. The pending PMI after enabled=0 will become harmless if the NMI handler does not change the state. This patch checks cpuc->enabled in pmi and only restore the state when PMU is active. Here is the dump: Call Trace: [] dump_stack+0x63/0x85 [] warn_slowpath_common+0x82/0xc0 [] warn_slowpath_null+0x1a/0x20 [] intel_pmu_drain_pebs_nhm+0x2be/0x320 [] intel_pmu_handle_irq+0x279/0x460 [] ? native_write_msr_safe+0x6/0x40 [] ? vunmap_page_range+0x20d/0x330 [] ? unmap_kernel_range_noflush+0x11/0x20 [] ? ghes_copy_tofrom_phys+0x10f/0x2a0 [] ? ghes_read_estatus+0x98/0x170 [] perf_event_nmi_handler+0x2d/0x50 [] nmi_handle+0x69/0x120 [] default_do_nmi+0xe6/0x100 [] do_nmi+0xe2/0x130 [] end_repeat_nmi+0x1a/0x1e [] ? native_write_msr_safe+0x6/0x40 [] ? native_write_msr_safe+0x6/0x40 [] ? native_write_msr_safe+0x6/0x40 <> [] ? x86_perf_event_set_period+0xd8/0x180 [] x86_pmu_start+0x4c/0x100 [] x86_pmu_enable+0x28d/0x300 [] perf_pmu_enable.part.81+0x7/0x10 [] perf_mux_hrtimer_handler+0x200/0x280 [] ? __perf_install_in_context+0xc0/0xc0 [] __hrtimer_run_queues+0xfd/0x280 [] hrtimer_interrupt+0xa8/0x190 [] ? __perf_read_group_add.part.61+0x1a0/0x1a0 [] local_apic_timer_interrupt+0x38/0x60 [] smp_apic_timer_interrupt+0x3d/0x50 [] apic_timer_interrupt+0x8c/0xa0 [] ? __perf_read_group_add.part.61+0x1a0/0x1a0 [] ? smp_call_function_single+0xd5/0x130 [] ? smp_call_function_single+0xcb/0x130 [] ? __perf_read_group_add.part.61+0x1a0/0x1a0 [] event_function_call+0x10a/0x120 [] ? ctx_resched+0x90/0x90 [] ? cpu_clock_event_read+0x30/0x30 [] ? _perf_event_disable+0x60/0x60 [] _perf_event_enable+0x5b/0x70 [] perf_event_for_each_child+0x38/0xa0 [] ? _perf_event_disable+0x60/0x60 [] perf_ioctl+0x12d/0x3c0 [] ? selinux_file_ioctl+0x95/0x1e0 [] do_vfs_ioctl+0xa1/0x5a0 [] ? sched_clock+0x9/0x10 [] SyS_ioctl+0x79/0x90 [] entry_SYSCALL_64_fastpath+0x1a/0xa4 ---[ end trace aef202839fe9a71d ]--- Uhhuh. NMI received for unknown reason 2d on CPU 2. Do you have a strange power saving mode enabled? Signed-off-by: Kan Liang Signed-off-by: Peter Zijlstra (Intel) Cc: Alexander Shishkin Cc: Arnaldo Carvalho de Melo Cc: Jiri Olsa Cc: Linus Torvalds Cc: Peter Zijlstra Cc: Stephane Eranian Cc: Thomas Gleixner Cc: Vince Weaver Link: http://lkml.kernel.org/r/1457046448-6184-1-git-send-email-kan.liang@intel.com [ Fixed various typos and other small details. ] Signed-off-by: Ingo Molnar [ kamal: backport to 4.2-stable: files renamed ] Signed-off-by: Kamal Mostafa --- arch/x86/kernel/cpu/perf_event.c | 13 +++++++++++++ arch/x86/kernel/cpu/perf_event_intel.c | 15 +++++++++++++-- arch/x86/kernel/cpu/perf_event_knc.c | 4 +++- 3 files changed, 29 insertions(+), 3 deletions(-) -- 2.7.4 diff --git a/arch/x86/kernel/cpu/perf_event.c b/arch/x86/kernel/cpu/perf_event.c index a27d738..d93690c 100644 --- a/arch/x86/kernel/cpu/perf_event.c +++ b/arch/x86/kernel/cpu/perf_event.c @@ -593,6 +593,19 @@ void x86_pmu_disable_all(void) } } +/* + * There may be PMI landing after enabled=0. The PMI hitting could be before or + * after disable_all. + * + * If PMI hits before disable_all, the PMU will be disabled in the NMI handler. + * It will not be re-enabled in the NMI handler again, because enabled=0. After + * handling the NMI, disable_all will be called, which will not change the + * state either. If PMI hits after disable_all, the PMU is already disabled + * before entering NMI handler. The NMI handler will not change the state + * either. + * + * So either situation is harmless. + */ static void x86_pmu_disable(struct pmu *pmu) { struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events); diff --git a/arch/x86/kernel/cpu/perf_event_intel.c b/arch/x86/kernel/cpu/perf_event_intel.c index 1b09c42..9af4ba8 100644 --- a/arch/x86/kernel/cpu/perf_event_intel.c +++ b/arch/x86/kernel/cpu/perf_event_intel.c @@ -1244,7 +1244,15 @@ static __initconst const u64 slm_hw_cache_event_ids }; /* - * Use from PMIs where the LBRs are already disabled. + * Used from PMIs where the LBRs are already disabled. + * + * This function could be called consecutively. It is required to remain in + * disabled state if called consecutively. + * + * During consecutive calls, the same disable value will be written to related + * registers, so the PMU state remains unchanged. hw.state in + * intel_bts_disable_local will remain PERF_HES_STOPPED too in consecutive + * calls. */ static void __intel_pmu_disable_all(void) { @@ -1670,7 +1678,10 @@ again: goto again; done: - __intel_pmu_enable_all(0, true); + /* Only restore PMU state when it's active. See x86_pmu_disable(). */ + if (cpuc->enabled) + __intel_pmu_enable_all(0, true); + /* * Only unmask the NMI after the overflow counters * have been reset. This avoids spurious NMIs on diff --git a/arch/x86/kernel/cpu/perf_event_knc.c b/arch/x86/kernel/cpu/perf_event_knc.c index 5b0c232..b931095 100644 --- a/arch/x86/kernel/cpu/perf_event_knc.c +++ b/arch/x86/kernel/cpu/perf_event_knc.c @@ -263,7 +263,9 @@ again: goto again; done: - knc_pmu_enable_all(0); + /* Only restore PMU state when it's active. See x86_pmu_disable(). */ + if (cpuc->enabled) + knc_pmu_enable_all(0); return handled; }