From patchwork Wed Jun 26 12:04:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Biener X-Patchwork-Id: 1952571 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=suse.de header.i=@suse.de header.a=rsa-sha256 header.s=susede2_rsa header.b=MRScML8r; dkim=pass header.d=suse.de header.i=@suse.de header.a=ed25519-sha256 header.s=susede2_ed25519 header.b=KTkjJ1Po; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.a=rsa-sha256 header.s=susede2_rsa header.b=MRScML8r; dkim=neutral header.d=suse.de header.i=@suse.de header.a=ed25519-sha256 header.s=susede2_ed25519 header.b=KTkjJ1Po; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=8.43.85.97; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4W8L4C3l23z20X1 for ; Wed, 26 Jun 2024 22:05:14 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 8BE833870C37 for ; Wed, 26 Jun 2024 12:05:11 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by sourceware.org (Postfix) with ESMTPS id C8829386D60C for ; Wed, 26 Jun 2024 12:04:53 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org C8829386D60C Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=suse.de ARC-Filter: OpenARC Filter v1.0.0 sourceware.org C8829386D60C Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=195.135.223.130 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1719403495; cv=none; b=hd7VEARrok4iFjakj3RSIHi8BkkOlATIluRIP+NWYwB/50qCLzWccfOvuuCa54Gxp8WJQaBcuElY7T2V3bVxhY0CoRPsIvGlIk6ST54tyrrGw3CQNlkS9DPQKVNDAGNZ6ewYStchOyInP5iJ1wJlKZOTkrONE+cSW8u2fpAMraY= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1719403495; c=relaxed/simple; bh=GQpDeqCdCdpct7pDeHz24nCiiH5q2eYHAtwcovW2k8w=; h=DKIM-Signature:DKIM-Signature:DKIM-Signature:DKIM-Signature:Date: From:To:Subject:MIME-Version; b=ELdb+315avgB7LWgg72v4z2tJbpzznosbZcMv07BPHViqpeaaG3F8XNapccaOBWDgGUMHBOxeWBCIQnmfvgquNw2rrn1q2lpdBpx61I6/aw3OeXy1N1zjkpDWqRBXvaVQu7al8oz+8fr8aAY8xTiCWLTTuSEpBe4ekgoe/9sJ7A= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from murzim.nue2.suse.org (unknown [10.168.4.243]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id B3D7E21A4F for ; Wed, 26 Jun 2024 12:04:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1719403492; h=from:from:reply-to:date:date:to:to:cc:mime-version:mime-version: content-type:content-type; bh=sdV1MBJANrXpJRt3hjn2lMxw6bO8XHRHRlRLQBNqwew=; b=MRScML8rKwrQH7iHq8AN59Gv0hU9LA9FRT6PuQZN1QrgRpJy3ITPhHD0mUC0sat8RWdHUV 7DpfqciiaX2/+PQNY3mj1RBW1W0ZrZIY7x6z9y+TEUqQuBalFCMiS6/arwixmmDdh9pqqz Tf78D1nvEZwasVjceFNM7Wg6miZ86u4= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1719403492; h=from:from:reply-to:date:date:to:to:cc:mime-version:mime-version: content-type:content-type; bh=sdV1MBJANrXpJRt3hjn2lMxw6bO8XHRHRlRLQBNqwew=; b=KTkjJ1PoDM6VfkFglXItLFZ080j8ss4B66UVfwKGW4VwiQ4562K3BcOl+QdSQvT4nmVEaU t19itdQ9NvWcQDBA== Authentication-Results: smtp-out1.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1719403492; h=from:from:reply-to:date:date:to:to:cc:mime-version:mime-version: content-type:content-type; bh=sdV1MBJANrXpJRt3hjn2lMxw6bO8XHRHRlRLQBNqwew=; b=MRScML8rKwrQH7iHq8AN59Gv0hU9LA9FRT6PuQZN1QrgRpJy3ITPhHD0mUC0sat8RWdHUV 7DpfqciiaX2/+PQNY3mj1RBW1W0ZrZIY7x6z9y+TEUqQuBalFCMiS6/arwixmmDdh9pqqz Tf78D1nvEZwasVjceFNM7Wg6miZ86u4= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1719403492; h=from:from:reply-to:date:date:to:to:cc:mime-version:mime-version: content-type:content-type; bh=sdV1MBJANrXpJRt3hjn2lMxw6bO8XHRHRlRLQBNqwew=; b=KTkjJ1PoDM6VfkFglXItLFZ080j8ss4B66UVfwKGW4VwiQ4562K3BcOl+QdSQvT4nmVEaU t19itdQ9NvWcQDBA== Date: Wed, 26 Jun 2024 14:04:52 +0200 (CEST) From: Richard Biener To: gcc-patches@gcc.gnu.org Subject: [PATCH] tree-optimization/115652 - adjust insertion gsi for SLP MIME-Version: 1.0 X-Spamd-Result: default: False [-1.78 / 50.00]; BAYES_HAM(-3.00)[100.00%]; MISSING_MID(2.50)[]; NEURAL_HAM_LONG(-1.00)[-0.999]; NEURAL_HAM_SHORT(-0.18)[-0.897]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_ZERO(0.00)[0]; ARC_NA(0.00)[]; MISSING_XM_UA(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; TO_DN_NONE(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[] X-Spam-Score: -1.78 X-Spam-Level: X-Spam-Status: No, score=-10.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, MISSING_MID, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org Message-Id: <20240626120511.8BE833870C37@sourceware.org> The following adjusts how SLP computes the insertion location. In particular it advanced the insert iterator of the found last_stmt. The vectorizer will later insert stmts _before_ it. But we also have the constraint that possibly masked ops may not be scheduled outside of the loop and as we do not model the loop mask in the SLP graph we have to adjust for that. The following moves this to after the advance since it isn't compatible with that as the current GIMPLE_COND exception shows. The PR is about in-order reduction vectorization which also isn't happy when that's the very first stmt. Bootstrapped and tested on x86_64-unknown-linux-gnu, pushed. PR tree-optimization/115652 * tree-vect-slp.cc (vect_schedule_slp_node): Advance the iterator based on last_stmt only for vector defs. --- gcc/tree-vect-slp.cc | 29 +++++++++++++---------------- 1 file changed, 13 insertions(+), 16 deletions(-) diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc index bb70a3fa5c2..0b12c821cbe 100644 --- a/gcc/tree-vect-slp.cc +++ b/gcc/tree-vect-slp.cc @@ -9932,16 +9932,6 @@ vect_schedule_slp_node (vec_info *vinfo, /* Emit other stmts after the children vectorized defs which is earliest possible. */ gimple *last_stmt = NULL; - if (auto loop_vinfo = dyn_cast (vinfo)) - if (LOOP_VINFO_FULLY_MASKED_P (loop_vinfo) - || LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo)) - { - /* But avoid scheduling internal defs outside of the loop when - we might have only implicitly tracked loop mask/len defs. */ - gimple_stmt_iterator si - = gsi_after_labels (LOOP_VINFO_LOOP (loop_vinfo)->header); - last_stmt = *si; - } bool seen_vector_def = false; FOR_EACH_VEC_ELT (SLP_TREE_CHILDREN (node), i, child) if (SLP_TREE_DEF_TYPE (child) == vect_internal_def) @@ -10050,12 +10040,19 @@ vect_schedule_slp_node (vec_info *vinfo, else { si = gsi_for_stmt (last_stmt); - /* When we're getting gsi_after_labels from the starting - condition of a fully masked/len loop avoid insertion - after a GIMPLE_COND that can appear as the only header - stmt with early break vectorization. */ - if (gimple_code (last_stmt) != GIMPLE_COND) - gsi_next (&si); + gsi_next (&si); + + /* Avoid scheduling internal defs outside of the loop when + we might have only implicitly tracked loop mask/len defs. */ + if (auto loop_vinfo = dyn_cast (vinfo)) + if (LOOP_VINFO_FULLY_MASKED_P (loop_vinfo) + || LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo)) + { + gimple_stmt_iterator si2 + = gsi_after_labels (LOOP_VINFO_LOOP (loop_vinfo)->header); + if (vect_stmt_dominates_stmt_p (last_stmt, *si2)) + si = si2; + } } }