From patchwork Tue Aug 6 16:18:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Monakov X-Patchwork-Id: 1969638 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=ispras.ru header.i=@ispras.ru header.a=rsa-sha256 header.s=default header.b=AvoSLHEF; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces~incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Wddnl0w1yz1yYD for ; Wed, 7 Aug 2024 02:20:27 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 00D5E3858C56 for ; Tue, 6 Aug 2024 16:20:25 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail.ispras.ru (mail.ispras.ru [83.149.199.84]) by sourceware.org (Postfix) with ESMTPS id F07D1385842C for ; Tue, 6 Aug 2024 16:19:00 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org F07D1385842C Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=ispras.ru Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=ispras.ru ARC-Filter: OpenARC Filter v1.0.0 sourceware.org F07D1385842C Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=83.149.199.84 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1722961142; cv=none; b=o0tJxfYTPiPQKFIhPk8BybMxtNTGSHaHtAfHylzgMir6iij4X5ZxCmaVsY28emIAcxYsciGiq/xS7LnYo9/mFGljFYvIYqi1XCBUYUc4V95Z0P0yZrRppIewGIN2XdYD3n7YBUkacjlzrVoIBsFkHvIWAi/A3hVxkekTC+ufhko= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1722961142; c=relaxed/simple; bh=RxtTTCTxAbXMcpNXPBcC623cH+01LIio13f203y56aE=; h=DKIM-Signature:From:To:Subject:Date:Message-Id:MIME-Version; b=UQNL4H7L4MUm4dBr+Gie1loa1B6Lhtp/qB3T3a4CnsdsnwcKUM5yWqV4ew0kMMVqfPljWKWfmbhTGs4f48rTKuDsgJRmi2LtZRiskdML1LVs9Vz7B5bt+fi5pp98KTChGL7h/MbO0SHLOZYrm5CzaFuGfuyWg1zy9925eCDv4Jk= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from localhost.intra.ispras.ru (unknown [10.10.3.121]) by mail.ispras.ru (Postfix) with ESMTP id 9E82B4076738; Tue, 6 Aug 2024 16:18:59 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 mail.ispras.ru 9E82B4076738 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ispras.ru; s=default; t=1722961139; bh=NanUxrzhCYQxP/tm3A6lw9+S6NU38MBP0Yahd38VXPE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=AvoSLHEFDsQR501r2okLDjYebL8B08wODLnlLWUYdfu+h9adq1z7R+Jngnst2Idds IUZsePmdvThTUd2KMlAaab1a5krgyc+OmcN829mr4ggUfmlnjP3kuJe9WwCOcSaK/l Z6XO+V4Jon/jd9KNRun9DtUqCP4QZeKDkJRANRLQ= From: Alexander Monakov To: gcc-patches@gcc.gnu.org Cc: Andi Kleen , Alexander Monakov Subject: [PATCH 2/3] libcpp: replace SSE4.2 helper with an SSSE3 one Date: Tue, 6 Aug 2024 19:18:49 +0300 Message-Id: <20240806161850.18839-2-amonakov@ispras.ru> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20240806161850.18839-1-amonakov@ispras.ru> References: <20240806161850.18839-1-amonakov@ispras.ru> MIME-Version: 1.0 X-Spam-Status: No, score=-9.1 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~incoming=patchwork.ozlabs.org@gcc.gnu.org Since the characters we are searching for (CR, LF, '\', '?') all have distinct ASCII codes mod 16, PSHUFB can help match them all at once. libcpp/ChangeLog: * lex.cc (search_line_sse42): Replace with... (search_line_ssse3): ... this new function. Adjust the use... (init_vectorized_lexer): ... here. --- libcpp/lex.cc | 118 ++++++++++++++++++++------------------------------ 1 file changed, 46 insertions(+), 72 deletions(-) diff --git a/libcpp/lex.cc b/libcpp/lex.cc index fa9c03614c..815b8abd29 100644 --- a/libcpp/lex.cc +++ b/libcpp/lex.cc @@ -345,84 +345,58 @@ search_line_sse2 (const uchar *s, const uchar *end ATTRIBUTE_UNUSED) } #ifdef HAVE_AVX2 -/* A version of the fast scanner using SSE 4.2 vectorized string insns. */ +/* A version of the fast scanner using SSSE3 shuffle (PSHUFB) insns. */ static const uchar * -#ifndef __SSE4_2__ -__attribute__((__target__("sse4.2"))) +#ifndef __SSSE3__ +__attribute__((__target__("ssse3"))) #endif -search_line_sse42 (const uchar *s, const uchar *end) +search_line_ssse3 (const uchar *s, const uchar *end ATTRIBUTE_UNUSED) { typedef char v16qi __attribute__ ((__vector_size__ (16))); - static const v16qi search = { '\n', '\r', '?', '\\' }; - - uintptr_t si = (uintptr_t)s; - uintptr_t index; - - /* Check for unaligned input. */ - if (si & 15) - { - v16qi sv; - - if (__builtin_expect (end - s < 16, 0) - && __builtin_expect ((si & 0xfff) > 0xff0, 0)) - { - /* There are less than 16 bytes left in the buffer, and less - than 16 bytes left on the page. Reading 16 bytes at this - point might generate a spurious page fault. Defer to the - SSE2 implementation, which already handles alignment. */ - return search_line_sse2 (s, end); - } - - /* ??? The builtin doesn't understand that the PCMPESTRI read from - memory need not be aligned. */ - sv = __builtin_ia32_loaddqu ((const char *) s); - index = __builtin_ia32_pcmpestri128 (search, 4, sv, 16, 0); - - if (__builtin_expect (index < 16, 0)) - goto found; - - /* Advance the pointer to an aligned address. We will re-scan a - few bytes, but we no longer need care for reading past the - end of a page, since we're guaranteed a match. */ - s = (const uchar *)((si + 15) & -16); - } - - /* Main loop, processing 16 bytes at a time. */ -#ifdef __GCC_ASM_FLAG_OUTPUTS__ - while (1) + typedef v16qi v16qi_u __attribute__ ((__aligned__ (1))); + /* Helper vector for pshufb-based matching: + each character C we're searching for is at position (C % 16). */ + v16qi lut = { 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, '\n', 0, '\\', '\r', 0, '?' }; + static_assert ('\n' == 10 && '\r' == 13 && '\\' == 92 && '?' == 63); + + int found; + /* Process three 16-byte chunks per iteration. */ + for (; ; s += 48) { - char f; - - /* By using inline assembly instead of the builtin, - we can use the result, as well as the flags set. */ - __asm ("%vpcmpestri\t$0, %2, %3" - : "=c"(index), "=@ccc"(f) - : "m"(*s), "x"(search), "a"(4), "d"(16)); - if (f) - break; - - s += 16; + v16qi data, t; + /* Unaligned load. Reading beyond the final newline is safe, since + files.cc:read_file_guts pads the allocation. */ + data = *(const v16qi_u *)s; + /* Prevent propagation into pshufb and pcmp as memory operand. */ + __asm__ ("" : "+x" (data)); + t = __builtin_ia32_pshufb128 (lut, data); + if ((found = __builtin_ia32_pmovmskb128 (t == data))) + goto done; + /* Second chunk. */ + data = *(const v16qi_u *)(s + 16); + __asm__ ("" : "+x" (data)); + t = __builtin_ia32_pshufb128 (lut, data); + if ((found = __builtin_ia32_pmovmskb128 (t == data))) + goto add_16; + /* Third chunk. */ + data = *(const v16qi_u *)(s + 32); + __asm__ ("" : "+x" (data)); + t = __builtin_ia32_pshufb128 (lut, data); + if ((found = __builtin_ia32_pmovmskb128 (t == data))) + goto add_32; } -#else - s -= 16; - /* By doing the whole loop in inline assembly, - we can make proper use of the flags set. */ - __asm ( ".balign 16\n" - "0: add $16, %1\n" - " %vpcmpestri\t$0, (%1), %2\n" - " jnc 0b" - : "=&c"(index), "+r"(s) - : "x"(search), "a"(4), "d"(16)); -#endif - - found: - return s + index; +add_32: + s += 16; +add_16: + s += 16; +done: + return s + __builtin_ctz (found); } #else -/* Work around out-dated assemblers without sse4 support. */ -#define search_line_sse42 search_line_sse2 +/* Work around out-dated assemblers without SSSE3 support. */ +#define search_line_ssse3 search_line_sse2 #endif /* Check the CPU capabilities. */ @@ -440,18 +414,18 @@ init_vectorized_lexer (void) search_line_fast_type impl = search_line_acc_char; int minimum = 0; -#if defined(__SSE4_2__) +#if defined(__SSSE3__) minimum = 3; #elif defined(__SSE2__) minimum = 2; #endif if (minimum == 3) - impl = search_line_sse42; + impl = search_line_ssse3; else if (__get_cpuid (1, &dummy, &dummy, &ecx, &edx) || minimum == 2) { - if (minimum == 3 || (ecx & bit_SSE4_2)) - impl = search_line_sse42; + if (minimum == 3 || (ecx & bit_SSSE3)) + impl = search_line_ssse3; else if (minimum == 2 || (edx & bit_SSE2)) impl = search_line_sse2; }