[neleai/string-x64] Improve strcpy sse2 and avx2 implementation

Hi,

I wrote new strcpy on x64 and for some reason I thought that I had
commited it and forgot to ping it.

As there are other routines that I could improve I will use branch
neleai/string-x64 to collect these.

Here is revised version of what I sumbitted in 2013. Main change is that
I now target i7 instead core2 That simplifies things as unaligned loads
are cheap instead bit slower than aligned ones on core2. That mainly
concerns header as for core2 you could get better performance by
aligning loads or stores to 16 bytes after first bytes were read. I do
not know whats better I would need to test it.

That also makes less important support of ssse3 variant. I could send it
but it was one of my list on TODO list that now probably lost
importance. Problem is that on x64 for aligning by ssse3 or sse2 with
shifts you need to make 16 loops for each alignment as you don't have
variable shift. Also it needs to use jump table thats very expensive
For strcpy thats dubious as it increases instruction cache pressure 
and most copies are small. You would need to do switching from unaligned 
loads to aligning. I needed to do profiling to select correct treshold.

If somebody is interested in optimizing old pentiums4, athlon64 I will
provide a ssse3 variant that is also 50% faster than current one.
That is also reason why I omitted drawing current ssse3 implementation
performance.

In this version header first checks 128 bytes unaligned unless they
cross page boundary. That allows more effective loop as then at end of
loop we could simply write last 64 bytes instead specialcasing to avoid
writing before start.

I tried several variants of header, as we first read 16 bytes to xmm0
register question is if they could be reused. I used evolver to select
best variant, there was almost no difference in performance between
these.

Now I do checks for bytes 0-15, then 16-31, then 32-63, then 64-128.
There is possibility to get some cycles with different grouping, I will
post later improvement if I could find something. 

First problem was reading ahead. A rereading 8 bytes looked bit faster
than move from xmm.

Then I tried when to reuse/reread. In 4-7 byte case it was faster reread
than using bit shifts to get second half. For 1-3 bytes I use following
copy with s[0] and s[1] from rdx register with byte shifts.

  Test branch vs this branchless that works for i 0,1,2
   d[i] = 0;
   d[i/2] = s[1];
   d[0] = s[0];

I also added a avx2 loop. Problem why I shouldn't use them in headers
was high latency. I could test if using them for bytes 64-128 would give
speedup.

As technical issues go I needed to move old strcpy_sse_unaligned
implementation into strncpy_sse2_unaligned as strncpy is function that
should be optimized for size, not performance. For now I this will keep
these unchanged.

As performance these are 15%-30% faster than current one for gcc workload on
haswell and ivy bridge. 

As avx2 version its currently 6% on this workload mainly as its bash and
has lot of large loads so avx2 loop helps.

I used my profiler to show improvement, see here

http://kam.mff.cuni.cz/~ondra/benchmark_string/strcpy_profile.html

and source is here

http://kam.mff.cuni.cz/~ondra/benchmark_string/strcpy_profile170615.tar.bz2

Comments?

        * sysdeps/x86_64/multiarch/ifunc-impl-list.c (__libc_ifunc_impl_list):
	Add __strcpy_avx2 and __stpcpy_avx2
        * sysdeps/x86_64/multiarch/Makefile (routines): Add stpcpy_avx2.S and 
	strcpy_avx2.S
        * sysdeps/x86_64/multiarch/stpcpy-avx2.S: New file
        * sysdeps/x86_64/multiarch/strcpy-avx2.S: Likewise.
        * sysdeps/x86_64/multiarch/stpcpy-sse2-unaligned.S: Refactored
	implementation.
        * sysdeps/x86_64/multiarch/strcpy.S: Updated ifunc.
        * sysdeps/x86_64/multiarch/strncpy.S: Moved from strcpy.S.
        * sysdeps/x86_64/multiarch/strncpy-sse2-unaligned.S: Moved
	strcpy-sse2-unaligned.S here.
        * sysdeps/x86_64/multiarch/strcpy-sse2-unaligned.S: Likewise.
        * sysdeps/x86_64/multiarch/stpncpy-sse2-unaligned.S: Redirect
	from strcpy-sse2-unaligned.S to strncpy-sse2-unaligned.S 
        * sysdeps/x86_64/multiarch/stpncpy.S: Likewise.
        * sysdeps/x86_64/multiarch/strcat-sse2-unaligned.S: Likewise.

---
 sysdeps/x86_64/multiarch/Makefile                 |    2 +-
 sysdeps/x86_64/multiarch/ifunc-impl-list.c        |    2 +
 sysdeps/x86_64/multiarch/stpcpy-avx2.S            |    3 +
 sysdeps/x86_64/multiarch/stpcpy-sse2-unaligned.S  |  439 ++++-
 sysdeps/x86_64/multiarch/stpncpy-sse2-unaligned.S |    3 +-
 sysdeps/x86_64/multiarch/stpncpy.S                |    5 +-
 sysdeps/x86_64/multiarch/strcat-sse2-unaligned.S  |    2 +-
 sysdeps/x86_64/multiarch/strcpy-avx2.S            |    4 +
 sysdeps/x86_64/multiarch/strcpy-sse2-unaligned.S  | 1890 +-------------------
 sysdeps/x86_64/multiarch/strcpy.S                 |   22 +-
 sysdeps/x86_64/multiarch/strncpy-sse2-unaligned.S | 1891 ++++++++++++++++++++-
 sysdeps/x86_64/multiarch/strncpy.S                |   88 +-
 14 files changed, 2435 insertions(+), 1921 deletions(-)
 create mode 100644 sysdeps/x86_64/multiarch/stpcpy-avx2.S
 create mode 100644 sysdeps/x86_64/multiarch/strcpy-avx2.S

[neleai/string-x64] Improve strcpy sse2 and avx2 implementation

Commit Message

Comments

Patch