| Message ID | 20190212194253.1951-3-mathieu.desnoyers@efficios.com |
|---|---|
| State | New |
| Headers | show |
| Series | Restartable Sequences support for glibc 2.30 | expand |
On 2/12/19 2:42 PM, Mathieu Desnoyers wrote: > When available, use the cpu_id field from __rseq_abi on Linux to > implement sched_getcpu(). Fall-back on the vgetcpu vDSO if unavailable. > > Benchmarks: > > x86-64: Intel E5-2630 v3@2.40GHz, 16-core, hyperthreading This patch looks good to me for master, but is blocked on patch 1/4 being reworked. Reviewed-by: Carlos O'Donell <carlos@redhat.com> > glibc sched_getcpu(): 13.7 ns (baseline) > glibc sched_getcpu() using rseq: 2.5 ns (speedup: 5.5x) > inline load cpuid from __rseq_abi TLS: 0.8 ns (speedup: 17.1x) > > Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> > CC: Carlos O'Donell <carlos@redhat.com> > CC: Florian Weimer <fweimer@redhat.com> > CC: Joseph Myers <joseph@codesourcery.com> > CC: Szabolcs Nagy <szabolcs.nagy@arm.com> > CC: Thomas Gleixner <tglx@linutronix.de> > CC: Ben Maurer <bmaurer@fb.com> > CC: Peter Zijlstra <peterz@infradead.org> > CC: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> > CC: Boqun Feng <boqun.feng@gmail.com> > CC: Will Deacon <will.deacon@arm.com> > CC: Dave Watson <davejwatson@fb.com> > CC: Paul Turner <pjt@google.com> > CC: libc-alpha@sourceware.org > CC: linux-kernel@vger.kernel.org > CC: linux-api@vger.kernel.org > --- > sysdeps/unix/sysv/linux/sched_getcpu.c | 25 +++++++++++++++++++++++-- > 1 file changed, 23 insertions(+), 2 deletions(-) > > diff --git a/sysdeps/unix/sysv/linux/sched_getcpu.c b/sysdeps/unix/sysv/linux/sched_getcpu.c > index fb0d317f83..8bfb03778b 100644 > --- a/sysdeps/unix/sysv/linux/sched_getcpu.c > +++ b/sysdeps/unix/sysv/linux/sched_getcpu.c > @@ -24,8 +24,8 @@ > #endif > #include <sysdep-vdso.h> > > -int > -sched_getcpu (void) > +static int > +vsyscall_sched_getcpu (void) OK. > { > #ifdef __NR_getcpu > unsigned int cpu; > @@ -37,3 +37,24 @@ sched_getcpu (void) > return -1; > #endif > } > + > +#ifdef __NR_rseq > +#include <linux/rseq.h> > + > +extern __attribute__ ((tls_model ("initial-exec"))) > +__thread volatile struct rseq __rseq_abi; OK. > + > +int > +sched_getcpu (void) > +{ > + int cpu_id = __rseq_abi.cpu_id; > + > + return cpu_id >= 0 ? cpu_id : vsyscall_sched_getcpu (); OK. Impressive :-) > +} > +#else > +int > +sched_getcpu (void) > +{ > + return vsyscall_sched_getcpu (); OK. > +} > +#endif >
diff --git a/sysdeps/unix/sysv/linux/sched_getcpu.c b/sysdeps/unix/sysv/linux/sched_getcpu.c index fb0d317f83..8bfb03778b 100644 --- a/sysdeps/unix/sysv/linux/sched_getcpu.c +++ b/sysdeps/unix/sysv/linux/sched_getcpu.c @@ -24,8 +24,8 @@ #endif #include <sysdep-vdso.h> -int -sched_getcpu (void) +static int +vsyscall_sched_getcpu (void) { #ifdef __NR_getcpu unsigned int cpu; @@ -37,3 +37,24 @@ sched_getcpu (void) return -1; #endif } + +#ifdef __NR_rseq +#include <linux/rseq.h> + +extern __attribute__ ((tls_model ("initial-exec"))) +__thread volatile struct rseq __rseq_abi; + +int +sched_getcpu (void) +{ + int cpu_id = __rseq_abi.cpu_id; + + return cpu_id >= 0 ? cpu_id : vsyscall_sched_getcpu (); +} +#else +int +sched_getcpu (void) +{ + return vsyscall_sched_getcpu (); +} +#endif
When available, use the cpu_id field from __rseq_abi on Linux to implement sched_getcpu(). Fall-back on the vgetcpu vDSO if unavailable. Benchmarks: x86-64: Intel E5-2630 v3@2.40GHz, 16-core, hyperthreading glibc sched_getcpu(): 13.7 ns (baseline) glibc sched_getcpu() using rseq: 2.5 ns (speedup: 5.5x) inline load cpuid from __rseq_abi TLS: 0.8 ns (speedup: 17.1x) Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> CC: Carlos O'Donell <carlos@redhat.com> CC: Florian Weimer <fweimer@redhat.com> CC: Joseph Myers <joseph@codesourcery.com> CC: Szabolcs Nagy <szabolcs.nagy@arm.com> CC: Thomas Gleixner <tglx@linutronix.de> CC: Ben Maurer <bmaurer@fb.com> CC: Peter Zijlstra <peterz@infradead.org> CC: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> CC: Boqun Feng <boqun.feng@gmail.com> CC: Will Deacon <will.deacon@arm.com> CC: Dave Watson <davejwatson@fb.com> CC: Paul Turner <pjt@google.com> CC: libc-alpha@sourceware.org CC: linux-kernel@vger.kernel.org CC: linux-api@vger.kernel.org --- sysdeps/unix/sysv/linux/sched_getcpu.c | 25 +++++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-)