From patchwork Mon Sep 30 15:25:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Biener X-Patchwork-Id: 1991068 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=suse.de header.i=@suse.de header.a=rsa-sha256 header.s=susede2_rsa header.b=rJNxx0XI; dkim=pass header.d=suse.de header.i=@suse.de header.a=ed25519-sha256 header.s=susede2_ed25519 header.b=Kx4UiUs3; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.a=rsa-sha256 header.s=susede2_rsa header.b=Y0p6vQwm; dkim=neutral header.d=suse.de header.i=@suse.de header.a=ed25519-sha256 header.s=susede2_ed25519 header.b=4r98Kvw9; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=8.43.85.97; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces~incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4XHPzS4Qrcz1xtM for ; Tue, 1 Oct 2024 01:25:55 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 0DE3D384F4B8 for ; Mon, 30 Sep 2024 15:25:53 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by sourceware.org (Postfix) with ESMTPS id EEB113861035 for ; Mon, 30 Sep 2024 15:25:29 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org EEB113861035 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=suse.de ARC-Filter: OpenARC Filter v1.0.0 sourceware.org EEB113861035 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1727709932; cv=none; b=J+UBVvaytvbsEx1SFyvMiCz6sGsXUVThHEGXhXnBw5aZgWb2UvXzNthbNF1XzYA59IxRochk8VDR+qq9s+xUZTiv9TIhNSdJc2u7ijbB80/9LW0D0xqwwG5oPfOhZMpnoo95/+rZqPOLn3DMncxI2g7cuUH9JIGVo/B+WvcUlHA= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1727709932; c=relaxed/simple; bh=CgppcJ77tkq9zRZCdFp0zsvi1BpYtjx37lHow5Z3V0w=; h=DKIM-Signature:DKIM-Signature:DKIM-Signature:DKIM-Signature:Date: From:To:Subject:MIME-Version:Message-Id; b=KYrsba8HQrqyr/RFgdYgTtu+ANVevu6ff7/26tHmzO32pPbpggtMj2ytkkpZjYjB/6vlyvL7gLhqI1YcbynVD2iUGPntPNjwrJlCeBqJqmDdNJF3CW4bHPCI2xPJv9YFvYzvJtjRfumlKyMOYbbzFc26PH6/dXfBEN4Y0iJ1Sj8= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id EEE631FBA5; Mon, 30 Sep 2024 15:25:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1727709929; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type; bh=8ziYCGPyKz0Wq+Ln42wxFNl1odn4dwMrabRLCt3lEMI=; b=rJNxx0XIhgr3pwX7vVdR2aPuUZP7wTcTjj4rZl7H1wNgVYU7m3SV2wo0iTZQFSmT3yR/SK wqxV/tcJ9cFA3+ge78RkiH8v+aKje/vgXDe6uI/0UIVMIJBuGwtGc6Sr/SXiDlI25LOywM OvCME5F20MOP7I7ldxm1fWsZFAic4NI= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1727709929; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type; bh=8ziYCGPyKz0Wq+Ln42wxFNl1odn4dwMrabRLCt3lEMI=; b=Kx4UiUs32N3gPmaOT+be5H9VJMPLTUrj+KvbA+JdJOqTrkiy1z8IMQ24SDcsfN4rUbRcj+ U+BgEUzeHRjc6OBw== Authentication-Results: smtp-out2.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1727709928; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type; bh=8ziYCGPyKz0Wq+Ln42wxFNl1odn4dwMrabRLCt3lEMI=; b=Y0p6vQwmFKnJj5AoyapdZz3O/CmEkbpelAye4sjdv8/zX6PoxPnCOaJLNLmrQquL0F+eL8 1WCMXkrE9L9EDHu1SaTEZDjpjLSX01ycefPbJZ3HkyMTqI1fGGXnffCfI8BpV9XcW++xd/ 3yp0p17/gCjV9nf1iwZz36qiFD00qKU= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1727709928; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type; bh=8ziYCGPyKz0Wq+Ln42wxFNl1odn4dwMrabRLCt3lEMI=; b=4r98Kvw9btLsXZtgELyuuyP/bEzs5V2/ADkfdj/vwHPqCmcjCBI9KnZtBfGknodxoWchXd xruiySVvy7bYJOAg== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id CE381136CB; Mon, 30 Sep 2024 15:25:28 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id A5a7MOjC+mbzIwAAD6G6ig (envelope-from ); Mon, 30 Sep 2024 15:25:28 +0000 Date: Mon, 30 Sep 2024 17:25:24 +0200 (CEST) From: Richard Biener To: gcc-patches@gcc.gnu.org cc: RISC-V Subject: [PATCH] tree-optimization/116566 - single lane SLP for VLA inductions MIME-Version: 1.0 Message-Id: <20240930152528.CE381136CB@imap1.dmz-prg2.suse.org> X-Spam-Score: -4.30 X-Spamd-Result: default: False [-4.30 / 50.00]; BAYES_HAM(-3.00)[99.99%]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.20)[-1.000]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MISSING_XM_UA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FUZZY_BLOCKED(0.00)[rspamd.com]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; DBL_BLOCKED_OPENRESOLVER(0.00)[imap1.dmz-prg2.suse.org:mid, imap1.dmz-prg2.suse.org:helo] X-Spam-Level: X-Spam-Status: No, score=-11.7 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~incoming=patchwork.ozlabs.org@gcc.gnu.org The following adds SLP support for vectorizing single-lane inductions with variable length vectors. This is a WIP patch, local testing for SVE and riscv is fine but the CI might discover issues. PR tree-optimization/116566 * tree-vect-loop.cc (vectorizable_induction): Handle single-lane SLP for VLA vectors. --- gcc/tree-vect-loop.cc | 192 ++++++++++++++++++++++++++++++++++-------- 1 file changed, 156 insertions(+), 36 deletions(-) diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc index 0ce1bf8ebba..206c44226bd 100644 --- a/gcc/tree-vect-loop.cc +++ b/gcc/tree-vect-loop.cc @@ -10282,7 +10282,6 @@ vectorizable_induction (loop_vec_info loop_vinfo, gimple *new_stmt; gphi *induction_phi; tree induc_def, vec_dest; - tree init_expr, step_expr; poly_uint64 vf = LOOP_VINFO_VECT_FACTOR (loop_vinfo); unsigned i; tree expr; @@ -10368,7 +10367,7 @@ vectorizable_induction (loop_vec_info loop_vinfo, iv_loop = loop; gcc_assert (iv_loop == (gimple_bb (phi))->loop_father); - if (slp_node && !nunits.is_constant ()) + if (slp_node && (!nunits.is_constant () && SLP_TREE_LANES (slp_node) != 1)) { /* The current SLP code creates the step value element-by-element. */ if (dump_enabled_p ()) @@ -10386,7 +10385,7 @@ vectorizable_induction (loop_vec_info loop_vinfo, return false; } - step_expr = STMT_VINFO_LOOP_PHI_EVOLUTION_PART (stmt_info); + tree step_expr = STMT_VINFO_LOOP_PHI_EVOLUTION_PART (stmt_info); gcc_assert (step_expr != NULL_TREE); if (INTEGRAL_TYPE_P (TREE_TYPE (step_expr)) && !type_has_mode_precision_p (TREE_TYPE (step_expr))) @@ -10474,9 +10473,6 @@ vectorizable_induction (loop_vec_info loop_vinfo, [i2 + 2*S2, i0 + 3*S0, i1 + 3*S1, i2 + 3*S2]. */ if (slp_node) { - /* Enforced above. */ - unsigned int const_nunits = nunits.to_constant (); - /* The initial values are vectorized, but any lanes > group_size need adjustment. */ slp_tree init_node @@ -10498,11 +10494,12 @@ vectorizable_induction (loop_vec_info loop_vinfo, /* Now generate the IVs. */ unsigned nvects = SLP_TREE_NUMBER_OF_VEC_STMTS (slp_node); - gcc_assert ((const_nunits * nvects) % group_size == 0); + gcc_assert (multiple_p (nunits * nvects, group_size)); unsigned nivs; + unsigned HOST_WIDE_INT const_nunits; if (nested_in_vect_loop) nivs = nvects; - else + else if (nunits.is_constant (&const_nunits)) { /* Compute the number of distinct IVs we need. First reduce group_size if it is a multiple of const_nunits so we get @@ -10513,21 +10510,42 @@ vectorizable_induction (loop_vec_info loop_vinfo, nivs = least_common_multiple (group_sizep, const_nunits) / const_nunits; } + else + { + gcc_assert (SLP_TREE_LANES (slp_node) == 1); + nivs = 1; + } + gimple_seq init_stmts = NULL; tree stept = TREE_TYPE (step_vectype); tree lupdate_mul = NULL_TREE; if (!nested_in_vect_loop) { - /* The number of iterations covered in one vector iteration. */ - unsigned lup_mul = (nvects * const_nunits) / group_size; - lupdate_mul - = build_vector_from_val (step_vectype, - SCALAR_FLOAT_TYPE_P (stept) - ? build_real_from_wide (stept, lup_mul, - UNSIGNED) - : build_int_cstu (stept, lup_mul)); + if (nunits.is_constant ()) + { + /* The number of iterations covered in one vector iteration. */ + unsigned lup_mul = (nvects * const_nunits) / group_size; + lupdate_mul + = build_vector_from_val (step_vectype, + SCALAR_FLOAT_TYPE_P (stept) + ? build_real_from_wide (stept, lup_mul, + UNSIGNED) + : build_int_cstu (stept, lup_mul)); + } + else + { + if (SCALAR_FLOAT_TYPE_P (stept)) + { + tree tem = build_int_cst (integer_type_node, vf); + lupdate_mul = gimple_build (&init_stmts, FLOAT_EXPR, stept, tem); + } + else + lupdate_mul = build_int_cst (stept, vf); + lupdate_mul = gimple_build_vector_from_val (&init_stmts, + step_vectype, + lupdate_mul); + } } tree peel_mul = NULL_TREE; - gimple_seq init_stmts = NULL; if (LOOP_VINFO_MASK_SKIP_NITERS (loop_vinfo)) { if (SCALAR_FLOAT_TYPE_P (stept)) @@ -10539,10 +10557,15 @@ vectorizable_induction (loop_vec_info loop_vinfo, peel_mul = gimple_build_vector_from_val (&init_stmts, step_vectype, peel_mul); } + tree step_mul = NULL_TREE; unsigned ivn; auto_vec vec_steps; for (ivn = 0; ivn < nivs; ++ivn) { + gimple_seq stmts = NULL; + bool invariant = true; + if (nunits.is_constant ()) + { tree_vector_builder step_elts (step_vectype, const_nunits, 1); tree_vector_builder init_elts (vectype, const_nunits, 1); tree_vector_builder mul_elts (step_vectype, const_nunits, 1); @@ -10570,13 +10593,65 @@ vectorizable_induction (loop_vec_info loop_vinfo, : build_int_cstu (stept, mul_elt)); } vec_step = gimple_build_vector (&init_stmts, &step_elts); - vec_steps.safe_push (vec_step); - tree step_mul = gimple_build_vector (&init_stmts, &mul_elts); - if (peel_mul) - step_mul = gimple_build (&init_stmts, MINUS_EXPR, step_vectype, - step_mul, peel_mul); + step_mul = gimple_build_vector (&init_stmts, &mul_elts); if (!init_node) vec_init = gimple_build_vector (&init_stmts, &init_elts); + } + else + { + if (init_node) + ; + else if (INTEGRAL_TYPE_P (TREE_TYPE (steps[0]))) + { + new_name = gimple_convert (&init_stmts, stept, inits[0]); + /* Build the initial value directly from a VEC_SERIES_EXPR. */ + vec_init = gimple_build (&init_stmts, VEC_SERIES_EXPR, + step_vectype, new_name, steps[0]); + if (!useless_type_conversion_p (vectype, step_vectype)) + vec_init = gimple_build (&init_stmts, VIEW_CONVERT_EXPR, + vectype, vec_init); + } + else + { + /* Build: + [base, base, base, ...] + + (vectype) [0, 1, 2, ...] * [step, step, step, ...]. */ + gcc_assert (SCALAR_FLOAT_TYPE_P (TREE_TYPE (steps[0]))); + gcc_assert (flag_associative_math); + tree index = build_index_vector (step_vectype, 0, 1); + new_name = gimple_convert (&init_stmts, TREE_TYPE (steps[0]), + inits[0]); + tree base_vec = gimple_build_vector_from_val (&init_stmts, step_vectype, + new_name); + tree step_vec = gimple_build_vector_from_val (&init_stmts, step_vectype, + steps[0]); + vec_init = gimple_build (&init_stmts, FLOAT_EXPR, step_vectype, index); + vec_init = gimple_build (&init_stmts, MULT_EXPR, step_vectype, + vec_init, step_vec); + vec_init = gimple_build (&init_stmts, PLUS_EXPR, step_vectype, + vec_init, base_vec); + if (!useless_type_conversion_p (vectype, step_vectype)) + vec_init = gimple_build (&init_stmts, VIEW_CONVERT_EXPR, + vectype, vec_init); + } + /* iv_loop is nested in the loop to be vectorized. Generate: + vec_step = [S, S, S, S] */ + t = unshare_expr (steps[0]); + gcc_assert (CONSTANT_CLASS_P (t) + || TREE_CODE (t) == SSA_NAME); + vec_step = gimple_build_vector_from_val (&init_stmts, + step_vectype, t); + } + vec_steps.safe_push (vec_step); + if (peel_mul) + { + if (!step_mul) + step_mul = peel_mul; + else + step_mul = gimple_build (&init_stmts, + MINUS_EXPR, step_vectype, + step_mul, peel_mul); + } /* Create the induction-phi that defines the induction-operand. */ vec_dest = vect_get_new_vect_var (vectype, vect_simple_var, @@ -10587,9 +10662,35 @@ vectorizable_induction (loop_vec_info loop_vinfo, /* Create the iv update inside the loop */ tree up = vec_step; if (lupdate_mul) - up = gimple_build (&init_stmts, MULT_EXPR, step_vectype, - vec_step, lupdate_mul); - gimple_seq stmts = NULL; + { + if (LOOP_VINFO_USING_SELECT_VL_P (loop_vinfo)) + { + /* When we're using loop_len produced by SELEC_VL, the non-final + iterations are not always processing VF elements. So vectorize + induction variable instead of + + _21 = vect_vec_iv_.6_22 + { VF, ... }; + + We should generate: + + _35 = .SELECT_VL (ivtmp_33, VF); + vect_cst__22 = [vec_duplicate_expr] _35; + _21 = vect_vec_iv_.6_22 + vect_cst__22; */ + vec_loop_lens *lens = &LOOP_VINFO_LENS (loop_vinfo); + tree len = vect_get_loop_len (loop_vinfo, NULL, lens, 1, + vectype, 0, 0); + expr = gimple_convert (&stmts, stept, unshare_expr (len)); + lupdate_mul = gimple_build_vector_from_val (&stmts, + step_vectype, + expr); + up = gimple_build (&stmts, MULT_EXPR, + step_vectype, vec_step, lupdate_mul); + } + else + up = gimple_build (&init_stmts, + MULT_EXPR, step_vectype, + vec_step, lupdate_mul); + } vec_def = gimple_convert (&stmts, step_vectype, induc_def); vec_def = gimple_build (&stmts, PLUS_EXPR, step_vectype, vec_def, up); @@ -10601,8 +10702,10 @@ vectorizable_induction (loop_vec_info loop_vinfo, if (init_node) vec_init = vect_get_slp_vect_def (init_node, ivn); if (!nested_in_vect_loop + && step_mul && !integer_zerop (step_mul)) { + gcc_assert (invariant); vec_def = gimple_convert (&init_stmts, step_vectype, vec_init); up = gimple_build (&init_stmts, MULT_EXPR, step_vectype, vec_step, step_mul); @@ -10619,8 +10722,11 @@ vectorizable_induction (loop_vec_info loop_vinfo, if (!nested_in_vect_loop) { /* Fill up to the number of vectors we need for the whole group. */ - nivs = least_common_multiple (group_size, - const_nunits) / const_nunits; + if (nunits.is_constant ()) + nivs = least_common_multiple (group_size, + const_nunits) / const_nunits; + else + nivs = 1; vec_steps.reserve (nivs-ivn); for (; ivn < nivs; ++ivn) { @@ -10633,14 +10739,28 @@ vectorizable_induction (loop_vec_info loop_vinfo, stmts by adding VF' * stride to the IVs generated above. */ if (ivn < nvects) { - unsigned vfp - = least_common_multiple (group_size, const_nunits) / group_size; - tree lupdate_mul - = build_vector_from_val (step_vectype, - SCALAR_FLOAT_TYPE_P (stept) - ? build_real_from_wide (stept, - vfp, UNSIGNED) - : build_int_cstu (stept, vfp)); + if (nunits.is_constant ()) + { + unsigned vfp = least_common_multiple (group_size, const_nunits) / group_size; + lupdate_mul + = build_vector_from_val (step_vectype, + SCALAR_FLOAT_TYPE_P (stept) + ? build_real_from_wide (stept, + vfp, UNSIGNED) + : build_int_cstu (stept, vfp)); + } + else + { + if (SCALAR_FLOAT_TYPE_P (stept)) + { + tree tem = build_int_cst (integer_type_node, nunits); + lupdate_mul = gimple_build (&init_stmts, FLOAT_EXPR, stept, tem); + } + else + lupdate_mul = build_int_cst (stept, nunits); + lupdate_mul = gimple_build_vector_from_val (&init_stmts, + step_vectype, lupdate_mul); + } for (; ivn < nvects; ++ivn) { gimple *iv @@ -10672,7 +10792,7 @@ vectorizable_induction (loop_vec_info loop_vinfo, return true; } - init_expr = vect_phi_initial_value (phi); + tree init_expr = vect_phi_initial_value (phi); gimple_seq stmts = NULL; if (!nested_in_vect_loop)