From patchwork Thu Mar 3 04:21:57 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Stultz X-Patchwork-Id: 591227 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 29984140E14 for ; Thu, 3 Mar 2016 15:23:37 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b=E7k0N88D; dkim-atps=neutral Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756780AbcCCEW5 (ORCPT ); Wed, 2 Mar 2016 23:22:57 -0500 Received: from mail-pf0-f174.google.com ([209.85.192.174]:35108 "EHLO mail-pf0-f174.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756616AbcCCEWR (ORCPT ); Wed, 2 Mar 2016 23:22:17 -0500 Received: by mail-pf0-f174.google.com with SMTP id w128so7207153pfb.2 for ; Wed, 02 Mar 2016 20:22:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nxPmQZm8nk58sOg5YD8Zje/BMJ1oYRHt259UOLrvLlY=; b=E7k0N88DIQ/rg7V1vQ0k1yYDeFb12FHfXSFh/GFh9cIIMBK09pm4413U0lN+xZZ3K3 mO6/4PWXBfmD8HiQt5bmlNfiNyEdQHR3/1elZeezC8aAIm2VExg5P6gprzo0jI69s2G3 MzTrFm/gIZLXYLdBABT+fyi7YkqmYvO6LnMY0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nxPmQZm8nk58sOg5YD8Zje/BMJ1oYRHt259UOLrvLlY=; b=RPEuEPyv0rg4weR0JpsSJ6hIXxDDGMOYnzWTq5NurYmtd23C5imHfIThbwKMljJHTp oeCErbSi8WTnU22LO1gyi+vwwm6OTVtRdDSboTalpZ/ZnYkF/1AmCIUN8VNCR8Rap50C 5ofblBvqg6L1hRSAbBoMdpB9KRhNFPtoFxTcIehCwzVHVwj2eVbD0kvMcp1Aw3driU9d lj2YHBIcxtj87OfQuzxtvo2eUgpnEe97fXDM8g25STcyFUEqrOXppBAQQIIUIXHmJTMt 0uraexmoXTQb7QEbYEV/7tgqazcw5VeY0y6QiYsUY0NjkyCBrpesFV0t1npKOAruqoVU xFFA== X-Gm-Message-State: AD7BkJIgoqOEgT1SzWPBGtvFF95ofXJzaDaSrnS/Q1WMZM/2heAgpzYrjOAAE0jLpS+dEOS4 X-Received: by 10.98.74.74 with SMTP id x71mr716793pfa.135.1456978936920; Wed, 02 Mar 2016 20:22:16 -0800 (PST) Received: from localhost.localdomain (c-76-115-103-22.hsd1.or.comcast.net. [76.115.103.22]) by smtp.gmail.com with ESMTPSA id xu1sm56468698pab.31.2016.03.02.20.22.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 02 Mar 2016 20:22:16 -0800 (PST) From: John Stultz To: lkml Cc: "Christopher S. Hall" , Prarit Bhargava , Richard Cochran , Thomas Gleixner , Ingo Molnar , Andy Lutomirski , kevin.b.stanton@intel.com, kevin.j.clarke@intel.com, hpa@zytor.com, jeffrey.t.kirsher@intel.com, netdev@vger.kernel.org, John Stultz Subject: [PATCH 6/8] x86: tsc: Always Running Timer (ART) correlated clocksource Date: Wed, 2 Mar 2016 20:21:57 -0800 Message-Id: <1456978919-30076-7-git-send-email-john.stultz@linaro.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1456978919-30076-1-git-send-email-john.stultz@linaro.org> References: <1456978919-30076-1-git-send-email-john.stultz@linaro.org> MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: "Christopher S. Hall" On modern Intel systems TSC is derived from the new Always Running Timer (ART). ART can be captured simultaneous to the capture of audio and network device clocks, allowing a correlation between timebases to be constructed. Upon capture, the driver converts the captured ART value to the appropriate system clock using the correlated clocksource mechanism. On systems that support ART a new CPUID leaf (0x15) returns parameters “m” and “n” such that: TSC_value = (ART_value * m) / n + k [n >= 1] [k is an offset that can adjusted by a privileged agent. The IA32_TSC_ADJUST MSR is an example of an interface to adjust k. See 17.14.4 of the Intel SDM for more details] Cc: Prarit Bhargava Cc: Richard Cochran Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Andy Lutomirski Cc: kevin.b.stanton@intel.com Cc: kevin.j.clarke@intel.com Cc: hpa@zytor.com Cc: jeffrey.t.kirsher@intel.com Cc: netdev@vger.kernel.org Reviewed-by: Thomas Gleixner Signed-off-by: Christopher S. Hall [jstultz: Tweaked to fix build issue, also reworked math for 64bit division on 32bit systems, as well as !CONFIG_CPU_FREQ build fixes] Signed-off-by: John Stultz --- arch/x86/include/asm/cpufeature.h | 2 +- arch/x86/include/asm/tsc.h | 2 ++ arch/x86/kernel/tsc.c | 62 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 65 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/cpufeature.h b/arch/x86/include/asm/cpufeature.h index 7ad8c94..ff557b4 100644 --- a/arch/x86/include/asm/cpufeature.h +++ b/arch/x86/include/asm/cpufeature.h @@ -85,7 +85,7 @@ #define X86_FEATURE_P4 ( 3*32+ 7) /* "" P4 */ #define X86_FEATURE_CONSTANT_TSC ( 3*32+ 8) /* TSC ticks at a constant rate */ #define X86_FEATURE_UP ( 3*32+ 9) /* smp kernel running on up */ -/* free, was #define X86_FEATURE_FXSAVE_LEAK ( 3*32+10) * "" FXSAVE leaks FOP/FIP/FOP */ +#define X86_FEATURE_ART (3*32+10) /* Platform has always running timer (ART) */ #define X86_FEATURE_ARCH_PERFMON ( 3*32+11) /* Intel Architectural PerfMon */ #define X86_FEATURE_PEBS ( 3*32+12) /* Precise-Event Based Sampling */ #define X86_FEATURE_BTS ( 3*32+13) /* Branch Trace Store */ diff --git a/arch/x86/include/asm/tsc.h b/arch/x86/include/asm/tsc.h index 6d7c547..174c421 100644 --- a/arch/x86/include/asm/tsc.h +++ b/arch/x86/include/asm/tsc.h @@ -29,6 +29,8 @@ static inline cycles_t get_cycles(void) return rdtsc(); } +extern struct system_counterval_t convert_art_to_tsc(cycle_t art); + extern void tsc_init(void); extern void mark_tsc_unstable(char *reason); extern int unsynchronized_tsc(void); diff --git a/arch/x86/kernel/tsc.c b/arch/x86/kernel/tsc.c index 3d743da..683c8cb 100644 --- a/arch/x86/kernel/tsc.c +++ b/arch/x86/kernel/tsc.c @@ -43,6 +43,11 @@ static DEFINE_STATIC_KEY_FALSE(__use_tsc); int tsc_clocksource_reliable; +static u32 art_to_tsc_numerator; +static u32 art_to_tsc_denominator; +static u64 art_to_tsc_offset; +struct clocksource *art_related_clocksource; + /* * Use a ring-buffer like data structure, where a writer advances the head by * writing a new data entry and a reader advances the tail when it observes a @@ -949,6 +954,36 @@ static struct notifier_block time_cpufreq_notifier_block = { .notifier_call = time_cpufreq_notifier }; +#define ART_CPUID_LEAF (0x15) +#define ART_MIN_DENOMINATOR (1) + + +/* + * If ART is present detect the numerator:denominator to convert to TSC + */ +static void detect_art(void) +{ + unsigned int unused[2]; + + if (boot_cpu_data.cpuid_level < ART_CPUID_LEAF) + return; + + cpuid(ART_CPUID_LEAF, &art_to_tsc_denominator, + &art_to_tsc_numerator, unused, unused+1); + + /* Don't enable ART in a VM, non-stop TSC required */ + if (boot_cpu_has(X86_FEATURE_HYPERVISOR) || + !boot_cpu_has(X86_FEATURE_NONSTOP_TSC) || + art_to_tsc_denominator < ART_MIN_DENOMINATOR) + return; + + if (rdmsrl_safe(MSR_IA32_TSC_ADJUST, &art_to_tsc_offset)) + return; + + /* Make this sticky over multiple CPU init calls */ + setup_force_cpu_cap(X86_FEATURE_ART); +} + static int __init cpufreq_tsc(void) { if (!cpu_has_tsc) @@ -962,6 +997,10 @@ static int __init cpufreq_tsc(void) core_initcall(cpufreq_tsc); +#else + +#define detect_art() + #endif /* CONFIG_CPU_FREQ */ /* clocksource code */ @@ -1071,6 +1110,25 @@ int unsynchronized_tsc(void) return 0; } +/* + * Convert ART to TSC given numerator/denominator found in detect_art() + */ +struct system_counterval_t convert_art_to_tsc(cycle_t art) +{ + u64 tmp, res, rem; + + rem = do_div(art, art_to_tsc_denominator); + + res = art * art_to_tsc_numerator; + tmp = rem * art_to_tsc_numerator; + + do_div(tmp, art_to_tsc_denominator); + res += tmp + art_to_tsc_offset; + + return (struct system_counterval_t) {.cs = art_related_clocksource, + .cycles = res}; +} +EXPORT_SYMBOL(convert_art_to_tsc); static void tsc_refine_calibration_work(struct work_struct *work); static DECLARE_DELAYED_WORK(tsc_irqwork, tsc_refine_calibration_work); @@ -1142,6 +1200,8 @@ static void tsc_refine_calibration_work(struct work_struct *work) (unsigned long)tsc_khz % 1000); out: + if (boot_cpu_has(X86_FEATURE_ART)) + art_related_clocksource = &clocksource_tsc; clocksource_register_khz(&clocksource_tsc, tsc_khz); } @@ -1235,6 +1295,8 @@ void __init tsc_init(void) mark_tsc_unstable("TSCs unsynchronized"); check_system_tsc_reliable(); + + detect_art(); } #ifdef CONFIG_SMP