From patchwork Mon Jan 22 09:02:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Biener X-Patchwork-Id: 1889045 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=suse.de header.i=@suse.de header.a=rsa-sha256 header.s=susede2_rsa header.b=Y88awFRm; dkim=pass header.d=suse.de header.i=@suse.de header.a=ed25519-sha256 header.s=susede2_ed25519 header.b=xXOOTyqU; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.a=rsa-sha256 header.s=susede2_rsa header.b=Y88awFRm; dkim=neutral header.d=suse.de header.i=@suse.de header.a=ed25519-sha256 header.s=susede2_ed25519 header.b=xXOOTyqU; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=8.43.85.97; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4TJPQm2JDsz20VD for ; Mon, 22 Jan 2024 20:03:44 +1100 (AEDT) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 54F2C38582AE for ; Mon, 22 Jan 2024 09:03:42 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtp-out2.suse.de (smtp-out2.suse.de [IPv6:2a07:de40:b251:101:10:150:64:2]) by sourceware.org (Postfix) with ESMTPS id CCB7A3858C2F for ; Mon, 22 Jan 2024 09:03:09 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org CCB7A3858C2F Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=suse.de ARC-Filter: OpenARC Filter v1.0.0 sourceware.org CCB7A3858C2F Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2a07:de40:b251:101:10:150:64:2 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1705914195; cv=none; b=s6cmpd1B+D3+eLl8S4nenrF9Un4r9IFK6BRFHVj9hPCwpgWKBStmFO/qB81vi7DbAtGHDyRKhm4S8p27xrhLLBeayiNYMT0DTu7ZWZDWnSDgRnQ3tenP+40Xk4TU3eBXGRSSL9/5GHxtccebaGqbiKS6FmmNBT3dmsyEqNE06ng= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1705914195; c=relaxed/simple; bh=RF4LqTMmRY9f/7eZML7QsF+TeuaYVj+Zb3MCAQnD2Ms=; h=DKIM-Signature:DKIM-Signature:DKIM-Signature:DKIM-Signature:Date: From:To:Subject:MIME-Version; b=xXOcSj9b2KcV3c1LMTzsxUi6fQe8xecAvMqck1q6nf8vdpSFLnFUFRYD9mPNYtifrAM24rTkXywQeSPGtF1cO4ico+PzLp73pRhuZNkrv4942sN86wOstPpwFmSxe33xBIE4uK18+FoOjDPhbiVimLXw7F27+cS6YqKx7ovhnDg= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from [10.168.4.150] (unknown [10.168.4.150]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id BFC061FB9E for ; Mon, 22 Jan 2024 09:03:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1705914188; h=from:from:reply-to:date:date:to:to:cc:mime-version:mime-version: content-type:content-type; bh=DqewgRezKtGRDZhbFwF/dW1i6YdZCSqeRW/DXZ7FDv4=; b=Y88awFRmq4fblBbErbkb9U/QYdPEpmc1mg5XGHihJRI2/B748do2vKtIJhn8JLVwIcFyT1 hkB7hBRZ4v6TkEdmaVPFP81jcmE7B1ML6wgKHs3QcaUUIJyRWIFtbxeL5fu8zg41ebQfLD Vni16s5SNvyl1B2ftFaFNbwjCO7xW+A= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1705914188; h=from:from:reply-to:date:date:to:to:cc:mime-version:mime-version: content-type:content-type; bh=DqewgRezKtGRDZhbFwF/dW1i6YdZCSqeRW/DXZ7FDv4=; b=xXOOTyqUYSDEYFAPJRTYsSzEQMvobtLD7vllyxftuZ0Bgr5MeP22lwjq+I+sLZL2xWl2ij PVhn8htXNn2VXmDQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1705914188; h=from:from:reply-to:date:date:to:to:cc:mime-version:mime-version: content-type:content-type; bh=DqewgRezKtGRDZhbFwF/dW1i6YdZCSqeRW/DXZ7FDv4=; b=Y88awFRmq4fblBbErbkb9U/QYdPEpmc1mg5XGHihJRI2/B748do2vKtIJhn8JLVwIcFyT1 hkB7hBRZ4v6TkEdmaVPFP81jcmE7B1ML6wgKHs3QcaUUIJyRWIFtbxeL5fu8zg41ebQfLD Vni16s5SNvyl1B2ftFaFNbwjCO7xW+A= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1705914188; h=from:from:reply-to:date:date:to:to:cc:mime-version:mime-version: content-type:content-type; bh=DqewgRezKtGRDZhbFwF/dW1i6YdZCSqeRW/DXZ7FDv4=; b=xXOOTyqUYSDEYFAPJRTYsSzEQMvobtLD7vllyxftuZ0Bgr5MeP22lwjq+I+sLZL2xWl2ij PVhn8htXNn2VXmDQ== Date: Mon, 22 Jan 2024 10:02:04 +0100 (CET) From: Richard Biener To: gcc-patches@gcc.gnu.org Subject: [PATCH] Refactor exit PHI handling in vectorizer epilogue peeling MIME-Version: 1.0 Authentication-Results: smtp-out2.suse.de; none X-Spam-Score: 5.22 X-Spamd-Result: default: False [5.22 / 50.00]; TO_DN_NONE(0.00)[]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; BAYES_HAM(-3.00)[100.00%]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_SPAM_SHORT(2.99)[0.997]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_ONE(0.00)[1]; MISSING_MID(2.50)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; NEURAL_SPAM_LONG(2.83)[0.807]; FUZZY_BLOCKED(0.00)[rspamd.com] X-Spam-Status: No, score=-10.8 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, MISSING_MID, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org Message-Id: <20240122090342.54F2C38582AE@sourceware.org> This refactors the handling of PHIs inbetween the main and the epilogue loop. Instead of trying to handle the multiple exit and original single exit case together the following separates these cases resulting in much easier to understand code. Bootstrap and regtest running on x86_64-unknown-linux-gnu. Richard. * tree-vect-loop-manip.cc (slpeel_tree_duplicate_loop_to_edge_cfg): Separate single and multi-exit case when creating PHIs between the main and epilogue. --- gcc/tree-vect-loop-manip.cc | 135 ++++++++++++++++++++---------------- 1 file changed, 74 insertions(+), 61 deletions(-) diff --git a/gcc/tree-vect-loop-manip.cc b/gcc/tree-vect-loop-manip.cc index eacbc022549..873a28d7c56 100644 --- a/gcc/tree-vect-loop-manip.cc +++ b/gcc/tree-vect-loop-manip.cc @@ -1682,77 +1682,60 @@ slpeel_tree_duplicate_loop_to_edge_cfg (class loop *loop, edge loop_exit, } /* Create the merge PHI nodes in new_preheader and populate the - arguments for the main exit. */ - for (auto gsi_from = gsi_start_phis (loop->header), - gsi_to = gsi_start_phis (new_loop->header); - !gsi_end_p (gsi_from) && !gsi_end_p (gsi_to); - gsi_next (&gsi_from), gsi_next (&gsi_to)) + arguments for the exits. */ + if (multiple_exits_p) { - gimple *from_phi = gsi_stmt (gsi_from); - gimple *to_phi = gsi_stmt (gsi_to); - tree new_arg = PHI_ARG_DEF_FROM_EDGE (from_phi, - loop_latch_edge (loop)); - - /* Check if we've already created a new phi node during edge - redirection. If we have, only propagate the value - downwards in case there is no merge block. */ - tree *res; - if ((res = new_phi_args.get (new_arg))) + for (auto gsi_from = gsi_start_phis (loop->header), + gsi_to = gsi_start_phis (new_loop->header); + !gsi_end_p (gsi_from) && !gsi_end_p (gsi_to); + gsi_next (&gsi_from), gsi_next (&gsi_to)) { - if (multiple_exits_p) - new_arg = *res; + gimple *from_phi = gsi_stmt (gsi_from); + gimple *to_phi = gsi_stmt (gsi_to); + + /* When the vector loop is peeled then we need to use the + value at start of the loop, otherwise the main loop exit + should use the final iter value. */ + tree new_arg; + if (peeled_iters) + new_arg = gimple_phi_result (from_phi); else - { - adjust_phi_and_debug_stmts (to_phi, loop_entry, *res); - continue; - } - } - /* If we have multiple exits and the vector loop is peeled then we - need to use the value at start of loop. If we're looking at - virtual operands we have to keep the original link. Virtual - operands don't all become the same because we'll corrupt the - vUSE chains among others. */ - if (peeled_iters) - { - tree tmp_arg = gimple_phi_result (from_phi); - /* Similar to the single exit case, If we have an existing - LCSSA variable thread through the original value otherwise - skip it and directly use the final value. */ - if ((res = new_phi_args.get (tmp_arg))) + new_arg = PHI_ARG_DEF_FROM_EDGE (from_phi, + loop_latch_edge (loop)); + + /* Check if we've already created a new phi node during edge + redirection and re-use it if so. Otherwise create a + LC PHI node to feed the merge PHI. */ + tree *res; + if (virtual_operand_p (new_arg)) + /* Use the existing virtual LC SSA from exit block. */ + new_arg = gimple_phi_result + (get_virtual_phi (main_loop_exit_block)); + else if ((res = new_phi_args.get (new_arg))) new_arg = *res; - else if (!virtual_operand_p (new_arg)) - new_arg = tmp_arg; - } - - tree new_res = copy_ssa_name (gimple_phi_result (from_phi)); - gphi *lcssa_phi = create_phi_node (new_res, new_preheader); - - /* Otherwise, main loop exit should use the final iter value. */ - if (multiple_exits_p) - { - /* Create a LC PHI if it doesn't already exist. */ - if (!virtual_operand_p (new_arg) && !res) + else { + /* Create the LC PHI node for the exit. */ tree new_def = copy_ssa_name (new_arg); gphi *lc_phi - = create_phi_node (new_def, main_loop_exit_block); + = create_phi_node (new_def, main_loop_exit_block); SET_PHI_ARG_DEF (lc_phi, 0, new_arg); new_arg = new_def; } - SET_PHI_ARG_DEF_ON_EDGE (lcssa_phi, - single_succ_edge (main_loop_exit_block), - new_arg); - } - else - SET_PHI_ARG_DEF_ON_EDGE (lcssa_phi, loop_exit, new_arg); - adjust_phi_and_debug_stmts (to_phi, loop_entry, new_res); - } + /* Create the PHI node in the merge block merging the + main and early exit values. */ + tree new_res = copy_ssa_name (gimple_phi_result (from_phi)); + gphi *lcssa_phi = create_phi_node (new_res, new_preheader); + edge main_e = single_succ_edge (main_loop_exit_block); + SET_PHI_ARG_DEF_ON_EDGE (lcssa_phi, main_e, new_arg); - /* Now fill in the values for the merge PHI in new_preheader - for the alternative exits. */ - if (multiple_exits_p) - { + /* And adjust the epilog entry value. */ + adjust_phi_and_debug_stmts (to_phi, loop_entry, new_res); + } + + /* After creating the merge PHIs handle the early exits those + should use the values at the start of the loop. */ for (auto gsi_from = gsi_start_phis (loop->header), gsi_to = gsi_start_phis (new_preheader); !gsi_end_p (gsi_from) && !gsi_end_p (gsi_to); @@ -1790,8 +1773,38 @@ slpeel_tree_duplicate_loop_to_edge_cfg (class loop *loop, edge loop_exit, SET_PHI_ARG_DEF (lc_phi, i, alt_arg); alt_arg = alt_def; } - edge main_e = single_succ_edge (alt_loop_exit_block); - SET_PHI_ARG_DEF_ON_EDGE (to_phi, main_e, alt_arg); + edge alt_e = single_succ_edge (alt_loop_exit_block); + SET_PHI_ARG_DEF_ON_EDGE (to_phi, alt_e, alt_arg); + } + } + /* For the single exit case only create the missing LC PHI nodes + for the continuation of the loop IVs that are not also already + reductions and thus had LC PHI nodes on the exit already. */ + else + { + for (auto gsi_from = gsi_start_phis (loop->header), + gsi_to = gsi_start_phis (new_loop->header); + !gsi_end_p (gsi_from) && !gsi_end_p (gsi_to); + gsi_next (&gsi_from), gsi_next (&gsi_to)) + { + gimple *from_phi = gsi_stmt (gsi_from); + gimple *to_phi = gsi_stmt (gsi_to); + tree new_arg = PHI_ARG_DEF_FROM_EDGE (from_phi, + loop_latch_edge (loop)); + + /* Check if we've already created a new phi node during edge + redirection. If we have, only propagate the value + downwards. */ + if (tree *res = new_phi_args.get (new_arg)) + { + adjust_phi_and_debug_stmts (to_phi, loop_entry, *res); + continue; + } + + tree new_res = copy_ssa_name (gimple_phi_result (from_phi)); + gphi *lcssa_phi = create_phi_node (new_res, new_preheader); + SET_PHI_ARG_DEF_ON_EDGE (lcssa_phi, loop_exit, new_arg); + adjust_phi_and_debug_stmts (to_phi, loop_entry, new_res); } } }