| Message ID | 20140423141407.6c38ee453d4c88c36fdfb062@linux-foundation.org |
|---|---|
| State | Not Applicable |
| Delegated to: | David Miller |
| Headers | show |
On Wed, Apr 23, 2014 at 02:14:07PM -0700, Andrew Morton wrote: > On Wed, 23 Apr 2014 16:40:05 -0400 Don Zickus <dzickus@redhat.com> wrote: > > > From: Aaron Tomlin <atomlin@redhat.com> > > > > A 'softlockup' is defined as a bug that causes the kernel to > > loop in kernel mode for more than a predefined period to > > time, without giving other tasks a chance to run. > > > > Currently, upon detection of this condition by the per-cpu > > watchdog task, debug information (including a stack trace) > > is sent to the system log. > > > > On some occasions, we have observed that the "victim" rather > > than the actual "culprit" (i.e. the owner/holder of the > > contended resource) is reported to the user. > > Often this information has proven to be insufficient to > > assist debugging efforts. > > > > To avoid loss of useful debug information, for architectures > > which support NMI, this patch makes it possible to improve > > soft lockup reporting. This is accomplished by issuing an > > NMI to each cpu to obtain a stack trace. > > > > If NMI is not supported we just revert back to the old method. > > A sysctl and boot-time parameter is available to toggle this > > feature. > > > > --- a/include/linux/nmi.h > > +++ b/include/linux/nmi.h > > @@ -57,6 +57,9 @@ int hw_nmi_is_cpu_stuck(struct pt_regs *); > > u64 hw_nmi_get_sample_period(int watchdog_thresh); > > extern int watchdog_user_enabled; > > extern int watchdog_thresh; > > +#ifdef CONFIG_SMP > > +extern int sysctl_softlockup_all_cpu_backtrace; > > +#endif > > The ifdefs aren't really needed here. If we omit them then error > reporting happens at link time rather than at compile time, but that's > a small price to pay for cleaning up the code. > > > + if (softlockup_all_cpu_backtrace) { > > + /* Prevent multiple soft-lockup reports if one cpu is already > > + * engaged in dumping cpu back traces > > + */ > > + if (test_and_set_bit(0, &soft_lockup_nmi_warn)) { > > + /* Someone else will report us. Let's give up */ > > + __this_cpu_write(soft_watchdog_warn, true); > > + return HRTIMER_RESTART; > > + } > > + } > > You missed my suggestion here. > > text data bss dec hex filename > 1519 524 24 2067 813 kernel/watchdog.o-before > 1471 520 16 2007 7d7 kernel/watchdog.o-after > > > --- a/include/linux/nmi.h~watchdog-printing-traces-for-all-cpus-on-lockup-detection-fix > +++ a/include/linux/nmi.h > @@ -57,9 +57,7 @@ int hw_nmi_is_cpu_stuck(struct pt_regs * > u64 hw_nmi_get_sample_period(int watchdog_thresh); > extern int watchdog_user_enabled; > extern int watchdog_thresh; > -#ifdef CONFIG_SMP > extern int sysctl_softlockup_all_cpu_backtrace; > -#endif > struct ctl_table; > extern int proc_dowatchdog(struct ctl_table *, int , > void __user *, size_t *, loff_t *); > --- a/kernel/watchdog.c~watchdog-printing-traces-for-all-cpus-on-lockup-detection-fix > +++ a/kernel/watchdog.c > @@ -31,7 +31,12 @@ > > int watchdog_user_enabled = 1; > int __read_mostly watchdog_thresh = 10; > +#ifdef CONFIG_SMP > int __read_mostly sysctl_softlockup_all_cpu_backtrace; > +#else > +#define sysctl_softlockup_all_cpu_backtrace 0 > +#endif > + > static int __read_mostly watchdog_running; > static u64 __read_mostly sample_period; > > _ Ah ok. I will respin the patch with that cleanup. Thanks! Cheers, Don > -- To unsubscribe from this list: send the line "unsubscribe sparclinux" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Apr 24, 2014 at 09:48:04AM -0400, Don Zickus wrote: > On Wed, Apr 23, 2014 at 02:14:07PM -0700, Andrew Morton wrote: > > On Wed, 23 Apr 2014 16:40:05 -0400 Don Zickus <dzickus@redhat.com> wrote: > > > > > From: Aaron Tomlin <atomlin@redhat.com> > > > > > > A 'softlockup' is defined as a bug that causes the kernel to > > > loop in kernel mode for more than a predefined period to > > > time, without giving other tasks a chance to run. > > > > > > Currently, upon detection of this condition by the per-cpu > > > watchdog task, debug information (including a stack trace) > > > is sent to the system log. > > > > > > On some occasions, we have observed that the "victim" rather > > > than the actual "culprit" (i.e. the owner/holder of the > > > contended resource) is reported to the user. > > > Often this information has proven to be insufficient to > > > assist debugging efforts. > > > > > > To avoid loss of useful debug information, for architectures > > > which support NMI, this patch makes it possible to improve > > > soft lockup reporting. This is accomplished by issuing an > > > NMI to each cpu to obtain a stack trace. > > > > > > If NMI is not supported we just revert back to the old method. > > > A sysctl and boot-time parameter is available to toggle this > > > feature. > > > > > > --- a/include/linux/nmi.h > > > +++ b/include/linux/nmi.h > > > @@ -57,6 +57,9 @@ int hw_nmi_is_cpu_stuck(struct pt_regs *); > > > u64 hw_nmi_get_sample_period(int watchdog_thresh); > > > extern int watchdog_user_enabled; > > > extern int watchdog_thresh; > > > +#ifdef CONFIG_SMP > > > +extern int sysctl_softlockup_all_cpu_backtrace; > > > +#endif > > > > The ifdefs aren't really needed here. If we omit them then error > > reporting happens at link time rather than at compile time, but that's > > a small price to pay for cleaning up the code. > > > > > + if (softlockup_all_cpu_backtrace) { > > > + /* Prevent multiple soft-lockup reports if one cpu is already > > > + * engaged in dumping cpu back traces > > > + */ > > > + if (test_and_set_bit(0, &soft_lockup_nmi_warn)) { > > > + /* Someone else will report us. Let's give up */ > > > + __this_cpu_write(soft_watchdog_warn, true); > > > + return HRTIMER_RESTART; > > > + } > > > + } > > > > You missed my suggestion here. > > > > text data bss dec hex filename > > 1519 524 24 2067 813 kernel/watchdog.o-before > > 1471 520 16 2007 7d7 kernel/watchdog.o-after > > > > > > --- a/include/linux/nmi.h~watchdog-printing-traces-for-all-cpus-on-lockup-detection-fix > > +++ a/include/linux/nmi.h > > @@ -57,9 +57,7 @@ int hw_nmi_is_cpu_stuck(struct pt_regs * > > u64 hw_nmi_get_sample_period(int watchdog_thresh); > > extern int watchdog_user_enabled; > > extern int watchdog_thresh; > > -#ifdef CONFIG_SMP > > extern int sysctl_softlockup_all_cpu_backtrace; > > -#endif > > struct ctl_table; > > extern int proc_dowatchdog(struct ctl_table *, int , > > void __user *, size_t *, loff_t *); > > --- a/kernel/watchdog.c~watchdog-printing-traces-for-all-cpus-on-lockup-detection-fix > > +++ a/kernel/watchdog.c > > @@ -31,7 +31,12 @@ > > > > int watchdog_user_enabled = 1; > > int __read_mostly watchdog_thresh = 10; > > +#ifdef CONFIG_SMP > > int __read_mostly sysctl_softlockup_all_cpu_backtrace; > > +#else > > +#define sysctl_softlockup_all_cpu_backtrace 0 > > +#endif > > + > > static int __read_mostly watchdog_running; > > static u64 __read_mostly sample_period; > > > > _ > > Ah ok. I will respin the patch with that cleanup. Thanks! Or I can just be happy you took care of that for me. :-) /me should read all his email first... Thanks, Don -- To unsubscribe from this list: send the line "unsubscribe sparclinux" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
--- a/include/linux/nmi.h~watchdog-printing-traces-for-all-cpus-on-lockup-detection-fix +++ a/include/linux/nmi.h @@ -57,9 +57,7 @@ int hw_nmi_is_cpu_stuck(struct pt_regs * u64 hw_nmi_get_sample_period(int watchdog_thresh); extern int watchdog_user_enabled; extern int watchdog_thresh; -#ifdef CONFIG_SMP extern int sysctl_softlockup_all_cpu_backtrace; -#endif struct ctl_table; extern int proc_dowatchdog(struct ctl_table *, int , void __user *, size_t *, loff_t *); --- a/kernel/watchdog.c~watchdog-printing-traces-for-all-cpus-on-lockup-detection-fix +++ a/kernel/watchdog.c @@ -31,7 +31,12 @@ int watchdog_user_enabled = 1; int __read_mostly watchdog_thresh = 10; +#ifdef CONFIG_SMP int __read_mostly sysctl_softlockup_all_cpu_backtrace; +#else +#define sysctl_softlockup_all_cpu_backtrace 0 +#endif + static int __read_mostly watchdog_running; static u64 __read_mostly sample_period;