From patchwork Mon Nov 27 12:07:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Sandiford X-Patchwork-Id: 1868733 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Sf49Y4ScGz1ySP for ; Mon, 27 Nov 2023 23:08:10 +1100 (AEDT) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 072653857B9C for ; Mon, 27 Nov 2023 12:07:58 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by sourceware.org (Postfix) with ESMTP id AB3DE3858CD1 for ; Mon, 27 Nov 2023 12:07:46 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org AB3DE3858CD1 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=arm.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org AB3DE3858CD1 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=217.140.110.172 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1701086868; cv=none; b=yDVAN4JNxkP04swrYWOKPCc7LNr2dhyv7HnLIYBJ6Hx183iM1x0BQvVt6pJm/xs8r/SQ0TgSzy/NGV4fCyum4yPhCSZbRbuY/6lRwa1I7DKYTLAnCryghEC1a+H/O8AQuCcMzPfRZ2suT5RO2PmsSosVOVU6/WRMFTGXowla3SY= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1701086868; c=relaxed/simple; bh=NtNfrjK/i/2TTxLyVXqTc//pQCj9OqmobtDPyKpQ9Cw=; h=From:To:Subject:Date:Message-ID:MIME-Version; b=U5dQkV9Zoh2gkvUXlOZTe9VoIbggTK3/+RXHEXANSmmy0UCXlgm6htvlEX4QioaDgnm86RHT67TlrwA67Y5AP33h0fsKKW1mAO+Lb6UO+ZnNLO17eWJS/ne8qro5wUavlk8PUfZexfPog6dDQgf8RFo84F1bA0Yh30b0YwU8xXM= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id DA7512F4; Mon, 27 Nov 2023 04:08:33 -0800 (PST) Received: from localhost (e121540-lin.manchester.arm.com [10.32.110.72]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id CB8833F73F; Mon, 27 Nov 2023 04:07:45 -0800 (PST) From: Richard Sandiford To: gcc-patches@gcc.gnu.org Mail-Followup-To: gcc-patches@gcc.gnu.org, rguenther@suse.de, richard.sandiford@arm.com Cc: rguenther@suse.de Subject: [PATCH] vect: Avoid duplicate_and_interleave for uniform vectors [PR112661] Date: Mon, 27 Nov 2023 12:07:44 +0000 Message-ID: User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 X-Spam-Status: No, score=-22.6 required=5.0 tests=BAYES_00, GIT_PATCH_0, KAM_DMARC_NONE, KAM_DMARC_STATUS, KAM_LAZY_DOMAIN_SECURITY, SPF_HELO_NONE, SPF_NONE, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org can_duplicate_and_interleave_p checks whether we know a way of building a particular VLA SLP invariant. g:60034ecf25597bd515f skipped that test for booleans, to support MASK_LEN_GATHER_LOAD calls with a dummy all-ones mask. But there's nothing fundamentally different about VLA masks vs VLA data vectors. If we have a VLA mask that isn't all-ones, we need some way of loading it. This ultimately led to the ICE in the PR. This patch fixes it by applying can_duplicate_and_interleave_p to masks, while also adding a special path for uniform vectors (of all kinds) to support the MASK_LEN_GATHER_LOAD usage. This also fixes an XFAIL in pr36648.cc for SVE. The patch is mostly Richard's. My only changes were to skip redundant conversions and to use gimple_build_vector_from_val for all eligible vectors. Tested on aarch64-linux-gnu (with and without SVE) and x86_64-linux-gnu. OK to install? Richard 2023-11-27 Richard Biener Richard Sandiford gcc/ PR tree-optimization/112661 * tree-vect-slp.cc (vect_get_and_check_slp_defs): Defer duplicate-and- interleave test to... (vect_build_slp_tree_2): ...here, once we have all the operands. Skip the test for uniform vectors. (vect_create_constant_vectors): Detect uniform vectors. Avoid redundant conversions in that case. Use gimple_build_vector_from_val to build the vector. gcc/testsuite/ * g++.dg/vect/pr36648.cc: Remove XFAIL for VLA load-lanes. --- gcc/testsuite/g++.dg/vect/pr36648.cc | 2 +- gcc/tree-vect-slp.cc | 56 +++++++++++++++++++--------- 2 files changed, 40 insertions(+), 18 deletions(-) diff --git a/gcc/testsuite/g++.dg/vect/pr36648.cc b/gcc/testsuite/g++.dg/vect/pr36648.cc index 8d24d3d445d..7bda82899d0 100644 --- a/gcc/testsuite/g++.dg/vect/pr36648.cc +++ b/gcc/testsuite/g++.dg/vect/pr36648.cc @@ -25,6 +25,6 @@ int main() { } targets, ! vect_no_align is a sufficient test. */ /* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target { { { ! vect_no_align } && { ! powerpc*-*-* } } || { powerpc*-*-* && vect_hw_misalign } } } } } */ -/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 1 "vect" { target { { { ! vect_no_align } && { ! powerpc*-*-* } } || { powerpc*-*-* && vect_hw_misalign } } xfail { vect_variable_length && vect_load_lanes } } } } */ +/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 1 "vect" { target { { { ! vect_no_align } && { ! powerpc*-*-* } } || { powerpc*-*-* && vect_hw_misalign } } } } } */ diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc index 4a09b3c2aca..6799b9375ae 100644 --- a/gcc/tree-vect-slp.cc +++ b/gcc/tree-vect-slp.cc @@ -763,18 +763,6 @@ vect_get_and_check_slp_defs (vec_info *vinfo, unsigned char swap, { tree type = TREE_TYPE (oprnd); dt = dts[i]; - if ((dt == vect_constant_def - || dt == vect_external_def) - && !GET_MODE_SIZE (vinfo->vector_mode).is_constant () - && TREE_CODE (type) != BOOLEAN_TYPE - && !can_duplicate_and_interleave_p (vinfo, stmts.length (), type)) - { - if (dump_enabled_p ()) - dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, - "Build SLP failed: invalid type of def " - "for variable-length SLP %T\n", oprnd); - return -1; - } /* For the swapping logic below force vect_reduction_def for the reduction op in a SLP reduction group. */ @@ -2395,7 +2383,7 @@ out: /* Create SLP_TREE nodes for the definition node/s. */ FOR_EACH_VEC_ELT (oprnds_info, i, oprnd_info) { - slp_tree child; + slp_tree child = nullptr; unsigned int j; /* We're skipping certain operands from processing, for example @@ -2443,6 +2431,29 @@ out: if (oprnd_info->first_dt == vect_external_def || oprnd_info->first_dt == vect_constant_def) { + if (!GET_MODE_SIZE (vinfo->vector_mode).is_constant ()) + { + tree op0; + tree uniform_val = op0 = oprnd_info->ops[0]; + for (j = 1; j < oprnd_info->ops.length (); ++j) + if (!operand_equal_p (uniform_val, oprnd_info->ops[j])) + { + uniform_val = NULL_TREE; + break; + } + if (!uniform_val + && !can_duplicate_and_interleave_p (vinfo, + oprnd_info->ops.length (), + TREE_TYPE (op0))) + { + matches[j] = false; + if (dump_enabled_p ()) + dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location, + "Build SLP failed: invalid type of def " + "for variable-length SLP %T\n", op0); + goto fail; + } + } slp_tree invnode = vect_create_new_slp_node (oprnd_info->ops); SLP_TREE_DEF_TYPE (invnode) = oprnd_info->first_dt; oprnd_info->ops = vNULL; @@ -8157,6 +8168,7 @@ vect_create_constant_vectors (vec_info *vinfo, slp_tree op_node) number_of_places_left_in_vector = nunits; constant_p = true; + tree uniform_elt = NULL_TREE; tree_vector_builder elts (vector_type, nunits, 1); elts.quick_grow (nunits); stmt_vec_info insert_after = NULL; @@ -8166,8 +8178,14 @@ vect_create_constant_vectors (vec_info *vinfo, slp_tree op_node) for (i = group_size - 1; op_node->ops.iterate (i, &op); i--) { /* Create 'vect_ = {op0,op1,...,opn}'. */ - number_of_places_left_in_vector--; tree orig_op = op; + if (number_of_places_left_in_vector == nunits) + uniform_elt = op; + else if (uniform_elt && operand_equal_p (uniform_elt, op)) + op = elts[number_of_places_left_in_vector]; + else + uniform_elt = NULL_TREE; + number_of_places_left_in_vector--; if (!types_compatible_p (TREE_TYPE (vector_type), TREE_TYPE (op))) { if (CONSTANT_CLASS_P (op)) @@ -8236,9 +8254,13 @@ vect_create_constant_vectors (vec_info *vinfo, slp_tree op_node) if (number_of_places_left_in_vector == 0) { - if (constant_p - ? multiple_p (TYPE_VECTOR_SUBPARTS (vector_type), nunits) - : known_eq (TYPE_VECTOR_SUBPARTS (vector_type), nunits)) + auto type_nunits = TYPE_VECTOR_SUBPARTS (vector_type); + if (uniform_elt) + vec_cst = gimple_build_vector_from_val (&ctor_seq, vector_type, + elts[0]); + else if (constant_p + ? multiple_p (type_nunits, nunits) + : known_eq (type_nunits, nunits)) vec_cst = gimple_build_vector (&ctor_seq, &elts); else {