From patchwork Thu Mar 2 14:28:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Sandiford X-Patchwork-Id: 1750757 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=8.43.85.97; helo=sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=gcc.gnu.org header.i=@gcc.gnu.org header.a=rsa-sha256 header.s=default header.b=Wcn+Citu; dkim-atps=neutral Received: from sourceware.org (ip-8-43-85-97.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4PSD4037Pkz1yWy for ; Fri, 3 Mar 2023 01:28:31 +1100 (AEDT) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 641C73858416 for ; Thu, 2 Mar 2023 14:28:27 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 641C73858416 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1677767307; bh=Nwf5RnxUai6TWaDt3LNEsTF0VC/p13fcP+56uEh+7Sw=; h=To:Subject:Date:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=Wcn+Citu1yUYnf+qiKzliadOX9gABD9cfKIdTBW/C+aRDXarshVW324XXFonAW0yY wmxauEe3Ee1usxXvV+5C7emsqCt2S+F8QMudYUJ4xYwNL6EXH0nRD40IOX5OFHlIWO ESPwm6iLJM8BPSyx3F9y5V5u5EjbkITjiaTUona8= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by sourceware.org (Postfix) with ESMTP id 9F6CF3858C83 for ; Thu, 2 Mar 2023 14:28:07 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 9F6CF3858C83 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9C76A1FB for ; Thu, 2 Mar 2023 06:28:50 -0800 (PST) Received: from localhost (e121540-lin.manchester.arm.com [10.32.99.50]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 0ADDF3F587 for ; Thu, 2 Mar 2023 06:28:06 -0800 (PST) To: gcc-patches@gcc.gnu.org Mail-Followup-To: gcc-patches@gcc.gnu.org, richard.sandiford@arm.com Subject: [PATCH] vect: Don't apply masks to operations on invariants [PR108979] Date: Thu, 02 Mar 2023 14:28:05 +0000 Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 X-Spam-Status: No, score=-34.1 required=5.0 tests=BAYES_00, GIT_PATCH_0, KAM_DMARC_NONE, KAM_DMARC_STATUS, KAM_LAZY_DOMAIN_SECURITY, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Richard Sandiford via Gcc-patches From: Richard Sandiford Reply-To: Richard Sandiford Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org Sender: "Gcc-patches" The loop body in the testcase contains an operation on invariants. SLP detects this and can hoist/schedule the operation outside of the loop. However, after the fix for PR96373, we would try to apply a loop mask to this operation, even though the mask is defined in the loop. The patch does what Richi suggested in the PR: suppress the masking for externs and constants. Tested on aarch64-linux-gnu. OK to install? Richard gcc/ PR tree-optimization/108979 * tree-vect-stmts.cc (vectorizable_operation): Don't mask operations on invariants. gcc/testsuite/ PR tree-optimization/108979 * gfortran.dg/vect/pr108979.f90: New test. --- gcc/testsuite/gfortran.dg/vect/pr108979.f90 | 21 +++++++++++++++++ gcc/tree-vect-stmts.cc | 25 ++++++++++++++++----- 2 files changed, 40 insertions(+), 6 deletions(-) create mode 100644 gcc/testsuite/gfortran.dg/vect/pr108979.f90 diff --git a/gcc/testsuite/gfortran.dg/vect/pr108979.f90 b/gcc/testsuite/gfortran.dg/vect/pr108979.f90 new file mode 100644 index 00000000000..623eb67826f --- /dev/null +++ b/gcc/testsuite/gfortran.dg/vect/pr108979.f90 @@ -0,0 +1,21 @@ +! { dg-do compile } +! { dg-additional-options "-fnon-call-exceptions" } +! { dg-additional-options "-march=armv8.2-a+sve" { target aarch64*-*-* } } + +MODULE hfx_contract_block + INTEGER, PARAMETER :: dp=8 +CONTAINS + SUBROUTINE block_2_1_2_1(kbd,kbc,kad,kac,pbd,pbc,pad,pac,prim,scale) + REAL(KIND=dp) :: kbd(1*1), kbc(1*2), kad(2*1), kac(2*2), pbd(1*1), & + pbc(1*2), pad(2*1), pac(2*2), prim(2*1*2*1), scale + DO md = 1,1 + DO mc = 1,2 + DO mb = 1,1 + DO ma = 1,2 + kac((mc-1)*2+ma) = kac((mc-1)*2+ma)-tmp*p_bd + END DO + END DO + END DO + END DO + END SUBROUTINE block_2_1_2_1 +END MODULE hfx_contract_block diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index 77ad8b78506..b56457617c0 100644 --- a/gcc/tree-vect-stmts.cc +++ b/gcc/tree-vect-stmts.cc @@ -6254,6 +6254,8 @@ vectorizable_operation (vec_info *vinfo, "use not simple.\n"); return false; } + bool is_invariant = (dt[0] == vect_external_def + || dt[0] == vect_constant_def); /* If op0 is an external or constant def, infer the vector type from the scalar type. */ if (!vectype) @@ -6307,6 +6309,8 @@ vectorizable_operation (vec_info *vinfo, "use not simple.\n"); return false; } + is_invariant &= (dt[1] == vect_external_def + || dt[1] == vect_constant_def); if (vectype2 && maybe_ne (nunits_out, TYPE_VECTOR_SUBPARTS (vectype2))) return false; @@ -6321,6 +6325,8 @@ vectorizable_operation (vec_info *vinfo, "use not simple.\n"); return false; } + is_invariant &= (dt[2] == vect_external_def + || dt[2] == vect_constant_def); if (vectype3 && maybe_ne (nunits_out, TYPE_VECTOR_SUBPARTS (vectype3))) return false; @@ -6426,16 +6432,23 @@ vectorizable_operation (vec_info *vinfo, int reduc_idx = STMT_VINFO_REDUC_IDX (stmt_info); vec_loop_masks *masks = (loop_vinfo ? &LOOP_VINFO_MASKS (loop_vinfo) : NULL); internal_fn cond_fn = get_conditional_internal_fn (code); - bool could_trap = gimple_could_trap_p (stmt); + + /* If operating on inactive elements could generate spurious traps, + we need to restrict the operation to active lanes. Note that this + specifically doesn't apply to unhoisted invariants, since they + operate on the same value for every lane. + + Similarly, if this operation is part of a reduction, a fully-masked + loop should only change the active lanes of the reduction chain, + keeping the inactive lanes as-is. */ + bool mask_out_inactive = ((!is_invariant && gimple_could_trap_p (stmt)) + || reduc_idx >= 0); if (!vec_stmt) /* transformation not required. */ { - /* If this operation is part of a reduction, a fully-masked loop - should only change the active lanes of the reduction chain, - keeping the inactive lanes as-is. */ if (loop_vinfo && LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) - && (could_trap || reduc_idx >= 0)) + && mask_out_inactive) { if (cond_fn == IFN_LAST || !direct_internal_fn_supported_p (cond_fn, vectype, @@ -6578,7 +6591,7 @@ vectorizable_operation (vec_info *vinfo, vop1 = ((op_type == binary_op || op_type == ternary_op) ? vec_oprnds1[i] : NULL_TREE); vop2 = ((op_type == ternary_op) ? vec_oprnds2[i] : NULL_TREE); - if (masked_loop_p && (reduc_idx >= 0 || could_trap)) + if (masked_loop_p && mask_out_inactive) { tree mask = vect_get_loop_mask (gsi, masks, vec_num * ncopies, vectype, i);