From patchwork Fri Apr 1 07:51:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: liuhongt X-Patchwork-Id: 1612090 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=gcc.gnu.org header.i=@gcc.gnu.org header.a=rsa-sha256 header.s=default header.b=voV3B8iP; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=8.43.85.97; helo=sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=) Received: from sourceware.org (ip-8-43-85-97.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KVD243MfFz9sVB for ; Fri, 1 Apr 2022 19:32:42 +1100 (AEDT) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id E9BBC3945C14 for ; Fri, 1 Apr 2022 08:32:39 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org E9BBC3945C14 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1648801959; bh=u7MfySZN4OMh4WB4BNdlf9ONdZYm0Tr6cYxOJ/Tb71s=; h=To:Subject:Date:In-Reply-To:References:List-Id:List-Unsubscribe: List-Archive:List-Post:List-Help:List-Subscribe:From:Reply-To: From; b=voV3B8iPqS6dGM66bRE3P/CnDqGzGAY0YL03GM108wswBnqSI45le21f/c0BwBL1c Q+MB41Mvi4qZMEyfejsJ9Dlq1qmQtKXIEfSIc0X+9yYzT+Xa0JEseQrOspYq9BdElb xrD9Zv8DHucMGxLryM+uTQAXBod6yHUCX5oAhgEw= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by sourceware.org (Postfix) with ESMTPS id B14A43858D28 for ; Fri, 1 Apr 2022 08:32:18 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.1 sourceware.org B14A43858D28 X-IronPort-AV: E=McAfee;i="6200,9189,10303"; a="258910935" X-IronPort-AV: E=Sophos;i="5.90,226,1643702400"; d="scan'208";a="258910935" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 01 Apr 2022 01:32:17 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.90,226,1643702400"; d="scan'208";a="788732506" Received: from scymds01.sc.intel.com ([10.148.94.138]) by fmsmga006.fm.intel.com with ESMTP; 01 Apr 2022 01:32:17 -0700 Received: from shliclel051.sh.intel.com (shliclel051.sh.intel.com [10.239.236.51]) by scymds01.sc.intel.com with ESMTP id 2317pK2r013192; Fri, 1 Apr 2022 00:51:20 -0700 To: gcc-patches@gcc.gnu.org Subject: [PATCH V3] Split vector load from parm_del to elemental loads to avoid STLF stalls. Date: Fri, 1 Apr 2022 15:51:19 +0800 Message-Id: <20220401075119.99163-1-hongtao.liu@intel.com> X-Mailer: git-send-email 2.18.1 In-Reply-To: References: X-Spam-Status: No, score=-12.4 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, SPF_HELO_NONE, SPF_NONE, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: liuhongt via Gcc-patches From: liuhongt Reply-To: liuhongt Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org Sender: "Gcc-patches" Update in V3: 1. Add -param=x86-stlf-window-ninsns= (default 64). 2. Exclude call in the window. Since cfg is freed before machine_reorg, just do a rough calculation of the window according to the layout. Also according to an experiment on CLX, set window size to 64. Currently only handle V2DFmode load since it doesn't need any scratch registers, and it's sufficient to recover cray performance for -O2 compared to GCC11. gcc/ChangeLog: PR target/101908 * config/i386/i386.cc (ix86_split_stlf_stall_load): New function (ix86_reorg): Call ix86_split_stlf_stall_load. * config/i386/i386.opt (-param=x86-stlf-window-ninsns=): New param. gcc/testsuite/ChangeLog: * gcc.target/i386/pr101908-1.c: New test. * gcc.target/i386/pr101908-2.c: New test. * gcc.target/i386/pr101908-3.c: New test. --- gcc/config/i386/i386.cc | 61 ++++++++++++++++++++++ gcc/config/i386/i386.opt | 4 ++ gcc/testsuite/gcc.target/i386/pr101908-1.c | 12 +++++ gcc/testsuite/gcc.target/i386/pr101908-2.c | 12 +++++ gcc/testsuite/gcc.target/i386/pr101908-3.c | 14 +++++ 5 files changed, 103 insertions(+) create mode 100644 gcc/testsuite/gcc.target/i386/pr101908-1.c create mode 100644 gcc/testsuite/gcc.target/i386/pr101908-2.c create mode 100644 gcc/testsuite/gcc.target/i386/pr101908-3.c diff --git a/gcc/config/i386/i386.cc b/gcc/config/i386/i386.cc index 5a561966eb4..3f8a2c7932d 100644 --- a/gcc/config/i386/i386.cc +++ b/gcc/config/i386/i386.cc @@ -21933,6 +21933,65 @@ ix86_seh_fixup_eh_fallthru (void) emit_insn_after (gen_nops (const1_rtx), insn); } } +/* Split vector load from parm_decl to elemental loads to avoid STLF + stalls. */ +static void +ix86_split_stlf_stall_load () +{ + rtx_insn* insn, *start = get_insns (); + unsigned window = 0; + + for (insn = start; insn; insn = NEXT_INSN (insn)) + { + if (!NONDEBUG_INSN_P (insn)) + continue; + window++; + /* Insert 64 vaddps %xmm18, %xmm19, %xmm20(no dependence between each + other, just emulate for pipeline) before stalled load, stlf stall + case is as fast as no stall cases on CLX. + Since CFG is freed before machine_reorg, just do a rough + calculation of the window according to the layout. */ + if (window > (unsigned) x86_stlf_window_ninsns) + return; + + if (any_uncondjump_p (insn) + || ANY_RETURN_P (PATTERN (insn)) + || CALL_P (insn)) + return; + + rtx set = single_set (insn); + if (!set) + continue; + rtx src = SET_SRC (set); + if (!MEM_P (src) + /* Only handle V2DFmode load since it doesn't need any scratch + register. */ + || GET_MODE (src) != E_V2DFmode + || !MEM_EXPR (src) + || TREE_CODE (get_base_address (MEM_EXPR (src))) != PARM_DECL) + continue; + + rtx zero = CONST0_RTX (V2DFmode); + rtx dest = SET_DEST (set); + rtx m = adjust_address (src, DFmode, 0); + rtx loadlpd = gen_sse2_loadlpd (dest, zero, m); + emit_insn_before (loadlpd, insn); + m = adjust_address (src, DFmode, 8); + rtx loadhpd = gen_sse2_loadhpd (dest, dest, m); + if (dump_file && (dump_flags & TDF_DETAILS)) + { + fputs ("Due to potential STLF stall, split instruction:\n", + dump_file); + print_rtl_single (dump_file, insn); + fputs ("To:\n", dump_file); + print_rtl_single (dump_file, loadlpd); + print_rtl_single (dump_file, loadhpd); + } + PATTERN (insn) = loadhpd; + INSN_CODE (insn) = -1; + gcc_assert (recog_memoized (insn) != -1); + } +} /* Implement machine specific optimizations. We implement padding of returns for K8 CPUs and pass to avoid 4 jumps in the single 16 byte window. */ @@ -21948,6 +22007,8 @@ ix86_reorg (void) if (optimize && optimize_function_for_speed_p (cfun)) { + if (TARGET_SSE2) + ix86_split_stlf_stall_load (); if (TARGET_PAD_SHORT_FUNCTION) ix86_pad_short_function (); else if (TARGET_PAD_RETURNS) diff --git a/gcc/config/i386/i386.opt b/gcc/config/i386/i386.opt index d8e8656a8ab..a6b0e28f238 100644 --- a/gcc/config/i386/i386.opt +++ b/gcc/config/i386/i386.opt @@ -1210,3 +1210,7 @@ Support MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, AVX, AVX2, AVX512F and AVX5 mdirect-extern-access Target Var(ix86_direct_extern_access) Init(1) Do not use GOT to access external symbols. + +-param=x86-stlf-window-ninsns= +Target Joined UInteger Var(x86_stlf_window_ninsns) Init(64) Param +Instructions number above which STFL stall penalty can be compensated. diff --git a/gcc/testsuite/gcc.target/i386/pr101908-1.c b/gcc/testsuite/gcc.target/i386/pr101908-1.c new file mode 100644 index 00000000000..33d9684f0ad --- /dev/null +++ b/gcc/testsuite/gcc.target/i386/pr101908-1.c @@ -0,0 +1,12 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -msse2 -mno-avx" } */ +/* { dg-final { scan-assembler-not {(?n)movhpd[ \t]} } } */ + +struct X { double x[2]; }; +typedef double v2df __attribute__((vector_size(16))); + +v2df __attribute__((noipa)) +foo (struct X* x, struct X* y) +{ + return (v2df) {x->x[1], x->x[0] } + (v2df) { y->x[1], y->x[0] }; +} diff --git a/gcc/testsuite/gcc.target/i386/pr101908-2.c b/gcc/testsuite/gcc.target/i386/pr101908-2.c new file mode 100644 index 00000000000..45060b73c06 --- /dev/null +++ b/gcc/testsuite/gcc.target/i386/pr101908-2.c @@ -0,0 +1,12 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -msse2 -mno-avx" } */ +/* { dg-final { scan-assembler-times {(?n)movhpd[ \t]+} "2" } } */ + +struct X { double x[4]; }; +typedef double v2df __attribute__((vector_size(16))); + +v2df __attribute__((noipa)) +foo (struct X x, struct X y) +{ + return (v2df) {x.x[1], x.x[0] } + (v2df) { y.x[1], y.x[0] }; +} diff --git a/gcc/testsuite/gcc.target/i386/pr101908-3.c b/gcc/testsuite/gcc.target/i386/pr101908-3.c new file mode 100644 index 00000000000..ddd3e8eff33 --- /dev/null +++ b/gcc/testsuite/gcc.target/i386/pr101908-3.c @@ -0,0 +1,14 @@ +/* { dg-do compile } */ +/* { dg-options "-O2 -msse2 -mno-avx" } */ +/* { dg-final { scan-assembler-not {(?n)movhpd[ \t]+} } } */ + +struct X { double x[4]; }; +typedef double v2df __attribute__((vector_size(16))); + +extern void bar (void); +v2df __attribute__((noipa)) +foo (struct X x, struct X y) +{ + bar (); + return (v2df) {x.x[1], x.x[0] } + (v2df) { y.x[1], y.x[0] }; +}