From patchwork Wed Dec 7 08:52:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Noah Goldstein X-Patchwork-Id: 1713096 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=8.43.85.97; helo=sourceware.org; envelope-from=libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; secure) header.d=sourceware.org header.i=@sourceware.org header.a=rsa-sha256 header.s=default header.b=pZ51VETw; dkim-atps=neutral Received: from sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4NRrln0WRhz23yq for ; Wed, 7 Dec 2022 19:57:57 +1100 (AEDT) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 13A05396DC22 for ; Wed, 7 Dec 2022 08:57:55 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 13A05396DC22 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1670403475; bh=llydJqidZ8QGYG/xY08OJeng4+6T/Lp/64IqwnLOt8k=; h=To:Cc:Subject:Date:In-Reply-To:References:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=pZ51VETwsg6GOdqITLy9e5VWVlGNFh3xZdOaGmH37qAu8dTP5Ln+ykj/7NOUTsh6N pk19R126rOVYunglfBuEjeNbCf9umh9j4OykgNkEebRPxP0wM4N2D8mUJOaODNes4r 0sRRwrzfW61d7MMIvbbqUnrumQfsqQinU5/XdomE= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-ed1-x52d.google.com (mail-ed1-x52d.google.com [IPv6:2a00:1450:4864:20::52d]) by sourceware.org (Postfix) with ESMTPS id 273EE3945C3A for ; Wed, 7 Dec 2022 08:53:15 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org 273EE3945C3A Received: by mail-ed1-x52d.google.com with SMTP id d14so18978138edj.11 for ; Wed, 07 Dec 2022 00:53:15 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=llydJqidZ8QGYG/xY08OJeng4+6T/Lp/64IqwnLOt8k=; b=FWCMv1ILCwJ1uGVa8yB5Y8IHRLd6AS/Nq0XghAismYnJGc5jcoS6qzvnixsWp1PzIL WBboDgAj8WPJxGgDDFc3RJuOt4PeKguK/p5SDkTIb98KRnaaLx1GD4OvKYve91V7bzYa ChKNd6LfqWvhAxk4mlsFcDwGPt2DIJ9PSojf8JGdWe7NR0t8BE79Ks77Vh00ccIvAE5J ytUcGeMvTTHbo9Ie0o1rB5qwvgZblaxwgQ7cIuQYM08w8B++9wJdMZI6KKKbNz3Apart epWD4VREADScQ0Hc+bDB1nSi7VK8F1rJDH4pmnjf9ab7wzFtwcBF19ER46g+P6OpRwys jsTw== X-Gm-Message-State: ANoB5plIMn7iHRRN41/ZbFgCHOctGubq5CvGYh7v5Ock7Kvy27u1nG3m hvzdAO/Z4fHeLF6CLWRGVpciC6C/oQU= X-Google-Smtp-Source: AA0mqf4P2XkYj6f8Cv8gAjUvs8V//bAInt8VVltFbWz3+oe12pbMVOIY3oy95aYlbinjxlSoUkRwzQ== X-Received: by 2002:a05:6402:6c6:b0:46c:1326:e186 with SMTP id n6-20020a05640206c600b0046c1326e186mr20783246edy.228.1670403193447; Wed, 07 Dec 2022 00:53:13 -0800 (PST) Received: from noahgold-desk.lan (2603-8080-1301-76c6-feb7-1b9b-f2dd-08f7.res6.spectrum.com. [2603:8080:1301:76c6:feb7:1b9b:f2dd:8f7]) by smtp.gmail.com with ESMTPSA id k17-20020aa7c051000000b0046bd3b366f9sm1931767edo.32.2022.12.07.00.53.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Dec 2022 00:53:13 -0800 (PST) To: libc-alpha@sourceware.org Cc: goldstein.w.n@gmail.com, hjl.tools@gmail.com, andrey.kolesov@intel.com, carlos@systemhalted.org Subject: [PATCH v1 18/27] x86/fpu: Optimize svml_s_log10f16_core_avx512.S Date: Wed, 7 Dec 2022 00:52:27 -0800 Message-Id: <20221207085236.1424424-18-goldstein.w.n@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221207085236.1424424-1-goldstein.w.n@gmail.com> References: <20221207085236.1424424-1-goldstein.w.n@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-12.1 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Noah Goldstein via Libc-alpha From: Noah Goldstein Reply-To: Noah Goldstein Errors-To: libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org Sender: "Libc-alpha" 1. Improve special values case which ends up covering ~half of all float bit patterns. 2. Cleanup some missed optimizations in instruction selection / unnecissary repeated rodata references. 3. Remove unused rodata. 4. Use common data definitions where possible. Code Size Change: -52 Bytes (226 - 278) Input New Time / Old Time 0F (0x00000000) -> 0.9484 0F (0x0000ffff, Denorm) -> 0.9668 .1F (0x3dcccccd) -> 0.9934 5F (0x40a00000) -> 0.9859 2315255808F (0x4f0a0000) -> 0.9926 -NaN (0xffffffff) -> 0.9808 --- .../multiarch/svml_s_log10f16_core_avx512.S | 296 ++++++++---------- 1 file changed, 125 insertions(+), 171 deletions(-) diff --git a/sysdeps/x86_64/fpu/multiarch/svml_s_log10f16_core_avx512.S b/sysdeps/x86_64/fpu/multiarch/svml_s_log10f16_core_avx512.S index bea2124519..5b68fbea61 100644 --- a/sysdeps/x86_64/fpu/multiarch/svml_s_log10f16_core_avx512.S +++ b/sysdeps/x86_64/fpu/multiarch/svml_s_log10f16_core_avx512.S @@ -27,211 +27,165 @@ * */ -/* Offsets for data table __svml_slog10_data_internal_avx512 - */ -#define One 0 -#define coeff4 64 -#define coeff3 128 -#define coeff2 192 -#define coeff1 256 -#define L2 320 +#define LOCAL_DATA_NAME __svml_slog10_data_internal +#include "svml_s_common_evex512_rodata_offsets.h" + +/* Offsets for data table __svml_slog10_data_internal. */ +#define _Coeff_4 0 +#define _Coeff_3 64 +#define _Coeff_2 128 +#define _Coeff_1 192 +#define _L2 256 #include .section .text.evex512, "ax", @progbits ENTRY(_ZGVeN16v_log10f_skx) - pushq %rbp - cfi_def_cfa_offset(16) - movq %rsp, %rbp - cfi_def_cfa(6, 16) - cfi_offset(6, -16) - andq $-64, %rsp - subq $192, %rsp vgetmantps $11, {sae}, %zmm0, %zmm3 - vmovups __svml_slog10_data_internal_avx512(%rip), %zmm1 + vmovups COMMON_DATA(_OneF)(%rip), %zmm1 vgetexpps {sae}, %zmm0, %zmm5 - vmovups L2+__svml_slog10_data_internal_avx512(%rip), %zmm10 vpsrld $19, %zmm3, %zmm7 vgetexpps {sae}, %zmm3, %zmm6 vsubps {rn-sae}, %zmm1, %zmm3, %zmm11 - vpermps coeff4+__svml_slog10_data_internal_avx512(%rip), %zmm7, %zmm1 - vpermps coeff3+__svml_slog10_data_internal_avx512(%rip), %zmm7, %zmm2 + vpermps LOCAL_DATA(_Coeff_4)(%rip), %zmm7, %zmm1 + vpermps LOCAL_DATA(_Coeff_3)(%rip), %zmm7, %zmm2 vsubps {rn-sae}, %zmm6, %zmm5, %zmm9 - vpermps coeff2+__svml_slog10_data_internal_avx512(%rip), %zmm7, %zmm4 - vpermps coeff1+__svml_slog10_data_internal_avx512(%rip), %zmm7, %zmm8 + vpermps LOCAL_DATA(_Coeff_2)(%rip), %zmm7, %zmm4 + vpermps LOCAL_DATA(_Coeff_1)(%rip), %zmm7, %zmm8 - /* x<=0? */ - vfpclassps $94, %zmm0, %k0 + /* x<=0? */ + vfpclassps $0x5e, %zmm0, %k0 vfmadd213ps {rn-sae}, %zmm2, %zmm11, %zmm1 - vmulps {rn-sae}, %zmm10, %zmm9, %zmm12 + vmulps LOCAL_DATA(_L2)(%rip), %zmm9, %zmm12 vfmadd213ps {rn-sae}, %zmm4, %zmm11, %zmm1 - kmovw %k0, %edx vfmadd213ps {rn-sae}, %zmm8, %zmm11, %zmm1 vfmadd213ps {rn-sae}, %zmm12, %zmm11, %zmm1 + kmovd %k0, %edx testl %edx, %edx - /* Go to special inputs processing branch */ + /* Go to special inputs processing branch. */ jne L(SPECIAL_VALUES_BRANCH) - # LOE rbx r12 r13 r14 r15 edx zmm0 zmm1 - - /* Restore registers - * and exit the function - */ -L(EXIT): + /* Restore registers * and exit the function. */ vmovaps %zmm1, %zmm0 - movq %rbp, %rsp - popq %rbp - cfi_def_cfa(7, 8) - cfi_restore(6) ret - cfi_def_cfa(6, 16) - cfi_offset(6, -16) - /* Branch to process - * special inputs - */ + /* Cold case. edx has 1s where there was a special value that + needs to be handled by a tanf call. Optimize for code size + moreso than speed here. */ L(SPECIAL_VALUES_BRANCH): - vmovups %zmm0, 64(%rsp) - vmovups %zmm1, 128(%rsp) - # LOE rbx r12 r13 r14 r15 edx zmm1 - xorl %eax, %eax - # LOE rbx r12 r13 r14 r15 eax edx + /* Use r13 to save/restore the stack. This allows us to use rbp + as callee save register saving code size. */ + pushq %r13 + cfi_def_cfa (rsp, 16) + /* Need to callee save registers to preserve state across tanf + calls. */ + pushq %rbx + cfi_def_cfa (rsp, 24) + pushq %rbp + cfi_def_cfa (rsp, 32) + movq %rsp, %r13 + cfi_def_cfa (r13, 32) + + /* Align stack and make room for 2x zmm vectors. */ + andq $-64, %rsp + addq $-128, %rsp + + /* Save original input. */ + vmovaps %zmm0, 64(%rsp) + /* Save all already computed inputs. */ + vmovaps %zmm1, (%rsp) vzeroupper - movq %r12, 16(%rsp) - /* DW_CFA_expression: r12 (r12) (DW_OP_lit8; DW_OP_minus; DW_OP_const4s: -64; DW_OP_and; DW_OP_const4s: -176; DW_OP_plus) */ - .cfi_escape 0x10, 0x0c, 0x0e, 0x38, 0x1c, 0x0d, 0xc0, 0xff, 0xff, 0xff, 0x1a, 0x0d, 0x50, 0xff, 0xff, 0xff, 0x22 - movl %eax, %r12d - movq %r13, 8(%rsp) - /* DW_CFA_expression: r13 (r13) (DW_OP_lit8; DW_OP_minus; DW_OP_const4s: -64; DW_OP_and; DW_OP_const4s: -184; DW_OP_plus) */ - .cfi_escape 0x10, 0x0d, 0x0e, 0x38, 0x1c, 0x0d, 0xc0, 0xff, 0xff, 0xff, 0x1a, 0x0d, 0x48, 0xff, 0xff, 0xff, 0x22 - movl %edx, %r13d - movq %r14, (%rsp) - /* DW_CFA_expression: r14 (r14) (DW_OP_lit8; DW_OP_minus; DW_OP_const4s: -64; DW_OP_and; DW_OP_const4s: -192; DW_OP_plus) */ - .cfi_escape 0x10, 0x0e, 0x0e, 0x38, 0x1c, 0x0d, 0xc0, 0xff, 0xff, 0xff, 0x1a, 0x0d, 0x40, 0xff, 0xff, 0xff, 0x22 - # LOE rbx r15 r12d r13d - - /* Range mask - * bits check - */ - -L(RANGEMASK_CHECK): - btl %r12d, %r13d - - /* Call scalar math function */ - jc L(SCALAR_MATH_CALL) - # LOE rbx r15 r12d r13d - - /* Special inputs - * processing loop - */ + /* edx has 1s where there was a special value that needs to be + handled by a tanf call. */ + movl %edx, %ebx L(SPECIAL_VALUES_LOOP): - incl %r12d - cmpl $16, %r12d - - /* Check bits in range mask */ - jl L(RANGEMASK_CHECK) - # LOE rbx r15 r12d r13d - - movq 16(%rsp), %r12 - cfi_restore(12) - movq 8(%rsp), %r13 - cfi_restore(13) - movq (%rsp), %r14 - cfi_restore(14) - vmovups 128(%rsp), %zmm1 - - /* Go to exit */ - jmp L(EXIT) - /* DW_CFA_expression: r12 (r12) (DW_OP_lit8; DW_OP_minus; DW_OP_const4s: -64; DW_OP_and; DW_OP_const4s: -176; DW_OP_plus) */ - .cfi_escape 0x10, 0x0c, 0x0e, 0x38, 0x1c, 0x0d, 0xc0, 0xff, 0xff, 0xff, 0x1a, 0x0d, 0x50, 0xff, 0xff, 0xff, 0x22 - /* DW_CFA_expression: r13 (r13) (DW_OP_lit8; DW_OP_minus; DW_OP_const4s: -64; DW_OP_and; DW_OP_const4s: -184; DW_OP_plus) */ - .cfi_escape 0x10, 0x0d, 0x0e, 0x38, 0x1c, 0x0d, 0xc0, 0xff, 0xff, 0xff, 0x1a, 0x0d, 0x48, 0xff, 0xff, 0xff, 0x22 - /* DW_CFA_expression: r14 (r14) (DW_OP_lit8; DW_OP_minus; DW_OP_const4s: -64; DW_OP_and; DW_OP_const4s: -192; DW_OP_plus) */ - .cfi_escape 0x10, 0x0e, 0x0e, 0x38, 0x1c, 0x0d, 0xc0, 0xff, 0xff, 0xff, 0x1a, 0x0d, 0x40, 0xff, 0xff, 0xff, 0x22 - # LOE rbx r12 r13 r14 r15 zmm1 - - /* Scalar math fucntion call - * to process special input - */ - -L(SCALAR_MATH_CALL): - movl %r12d, %r14d - vmovss 64(%rsp, %r14, 4), %xmm0 + + /* use rbp as index for special value that is saved across calls + to tanf. We technically don't need a callee save register + here as offset to rsp is always [0, 56] so we can restore + rsp by realigning to 64. Essentially the tradeoff is 1 extra + save/restore vs 2 extra instructions in the loop. Realigning + also costs more code size. */ + xorl %ebp, %ebp + tzcntl %ebx, %ebp + + /* Scalar math fucntion call to process special input. */ + movss 64(%rsp, %rbp, 4), %xmm0 call log10f@PLT - # LOE rbx r14 r15 r12d r13d xmm0 - vmovss %xmm0, 128(%rsp, %r14, 4) + /* No good way to avoid the store-forwarding fault this will + cause on return. `lfence` avoids the SF fault but at greater + cost as it serialized stack/callee save restoration. */ + movss %xmm0, (%rsp, %rbp, 4) - /* Process special inputs in loop */ - jmp L(SPECIAL_VALUES_LOOP) - # LOE rbx r15 r12d r13d + blsrl %ebx, %ebx + jnz L(SPECIAL_VALUES_LOOP) + + + /* All results have been written to 64(%rsp). */ + vmovaps (%rsp), %zmm0 + /* Restore rsp. */ + movq %r13, %rsp + cfi_def_cfa (rsp, 32) + /* Restore callee save registers. */ + popq %rbp + cfi_def_cfa (rsp, 24) + popq %rbx + cfi_def_cfa (rsp, 16) + popq %r13 + ret END(_ZGVeN16v_log10f_skx) - .section .rodata, "a" + .section .rodata.evex512, "a" .align 64 -#ifdef __svml_slog10_data_internal_avx512_typedef -typedef unsigned int VUINT32; -typedef struct { - __declspec(align(64)) VUINT32 One[16][1]; - __declspec(align(64)) VUINT32 coeff4[16][1]; - __declspec(align(64)) VUINT32 coeff3[16][1]; - __declspec(align(64)) VUINT32 coeff2[16][1]; - __declspec(align(64)) VUINT32 coeff1[16][1]; - __declspec(align(64)) VUINT32 L2[16][1]; -} __svml_slog10_data_internal_avx512; -#endif -__svml_slog10_data_internal_avx512: - /* One */ - .long 0x3f800000, 0x3f800000, 0x3f800000, 0x3f800000, 0x3f800000, 0x3f800000, 0x3f800000, 0x3f800000, 0x3f800000, 0x3f800000, 0x3f800000, 0x3f800000, 0x3f800000, 0x3f800000, 0x3f800000, 0x3f800000 - // c4 - .align 64 - .long 0xbdc9ae9b, 0xbda6fcf4 - .long 0xbd8bac76, 0xbd6bca30 - .long 0xbd48a99b, 0xbd2c0a9f - .long 0xbd1480db, 0xbd00faf2 - .long 0xbe823aa9, 0xbe656348 - .long 0xbe4afbb9, 0xbe346895 - .long 0xbe20ffff, 0xbe103a0b - .long 0xbe01a91c, 0xbde9e84e - // c3 - .align 64 - .long 0x3e13d888, 0x3e10a87c - .long 0x3e0b95c3, 0x3e057f0b - .long 0x3dfde038, 0x3df080d9 - .long 0x3de34c1e, 0x3dd68333 - .long 0x3dac6e8e, 0x3dd54a51 - .long 0x3df30f40, 0x3e04235d - .long 0x3e0b7033, 0x3e102c90 - .long 0x3e12ebad, 0x3e141ff8 - // c2 - .align 64 - .long 0xbe5e5a9b, 0xbe5e2677 - .long 0xbe5d83f5, 0xbe5c6016 - .long 0xbe5abd0b, 0xbe58a6fd - .long 0xbe562e02, 0xbe5362f8 - .long 0xbe68e27c, 0xbe646747 - .long 0xbe619a73, 0xbe5ff05a - .long 0xbe5f0570, 0xbe5e92d0 - .long 0xbe5e662b, 0xbe5e5c08 - // c1 - .align 64 - .long 0x3ede5bd8, 0x3ede5b45 - .long 0x3ede57d8, 0x3ede4eb1 - .long 0x3ede3d37, 0x3ede2166 - .long 0x3eddf9d9, 0x3eddc5bb - .long 0x3ede08ed, 0x3ede32e7 - .long 0x3ede4967, 0x3ede5490 - .long 0x3ede597f, 0x3ede5b50 - .long 0x3ede5bca, 0x3ede5bd9 - /* L2 */ - .align 64 - .long 0x3e9a209b, 0x3e9a209b, 0x3e9a209b, 0x3e9a209b, 0x3e9a209b, 0x3e9a209b, 0x3e9a209b, 0x3e9a209b, 0x3e9a209b, 0x3e9a209b, 0x3e9a209b, 0x3e9a209b, 0x3e9a209b, 0x3e9a209b, 0x3e9a209b, 0x3e9a209b - .align 64 - .type __svml_slog10_data_internal_avx512, @object - .size __svml_slog10_data_internal_avx512, .-__svml_slog10_data_internal_avx512 +LOCAL_DATA_NAME: + float_block (LOCAL_DATA_NAME, _Coeff_4, + 0xbdc9ae9b, 0xbda6fcf4, + 0xbd8bac76, 0xbd6bca30, + 0xbd48a99b, 0xbd2c0a9f, + 0xbd1480db, 0xbd00faf2, + 0xbe823aa9, 0xbe656348, + 0xbe4afbb9, 0xbe346895, + 0xbe20ffff, 0xbe103a0b, + 0xbe01a91c, 0xbde9e84e) + + float_block (LOCAL_DATA_NAME, _Coeff_3, + 0x3e13d888, 0x3e10a87c, + 0x3e0b95c3, 0x3e057f0b, + 0x3dfde038, 0x3df080d9, + 0x3de34c1e, 0x3dd68333, + 0x3dac6e8e, 0x3dd54a51, + 0x3df30f40, 0x3e04235d, + 0x3e0b7033, 0x3e102c90, + 0x3e12ebad, 0x3e141ff8) + + float_block (LOCAL_DATA_NAME, _Coeff_2, + 0xbe5e5a9b, 0xbe5e2677, + 0xbe5d83f5, 0xbe5c6016, + 0xbe5abd0b, 0xbe58a6fd, + 0xbe562e02, 0xbe5362f8, + 0xbe68e27c, 0xbe646747, + 0xbe619a73, 0xbe5ff05a, + 0xbe5f0570, 0xbe5e92d0, + 0xbe5e662b, 0xbe5e5c08) + + float_block (LOCAL_DATA_NAME, _Coeff_1, + 0x3ede5bd8, 0x3ede5b45, + 0x3ede57d8, 0x3ede4eb1, + 0x3ede3d37, 0x3ede2166, + 0x3eddf9d9, 0x3eddc5bb, + 0x3ede08ed, 0x3ede32e7, + 0x3ede4967, 0x3ede5490, + 0x3ede597f, 0x3ede5b50, + 0x3ede5bca, 0x3ede5bd9) + + DATA_VEC (LOCAL_DATA_NAME, _L2, 0x3e9a209b) + + .type LOCAL_DATA_NAME, @object + .size LOCAL_DATA_NAME, .-LOCAL_DATA_NAME