From patchwork Thu May 16 04:05:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Li, Pan2" X-Patchwork-Id: 1935792 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=M6wu8U6P; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=8.43.85.97; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VfxNT3C6rz20KF for ; Thu, 16 May 2024 14:06:15 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id DD4C63849AE5 for ; Thu, 16 May 2024 04:06:12 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.7]) by sourceware.org (Postfix) with ESMTPS id 1582F384CB97 for ; Thu, 16 May 2024 04:05:49 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 1582F384CB97 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=intel.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 1582F384CB97 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=192.198.163.7 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1715832351; cv=none; b=JqAk9d3fiP4j8n8EQXkweVtF4/arAPofvpzMuVa/7HOOfN4oIa1m1JcuvIyzL84M7Z+1Smc/VqVk+K21U5FsDiLhcz1St5GgxdqNOgGHEfQYs+ByLJyQgs/aqTnCGIs09kxG5lI/SZP8RBgk98WlDXEMoZjAsNmjnzrxNeOl/n0= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1715832351; c=relaxed/simple; bh=/GVW5S01UQkVLewpSI50IYqMJGhZ5pRv72RYPUCzBIo=; h=DKIM-Signature:From:To:Subject:Date:Message-Id:MIME-Version; b=f2MYQFCvre3rudVc5rP6gRfpYUA7732S4cJiuJvSi9BACtUlu7vrh43X7nENxRlkobvzHUvuBzDeZughwr1RsHi2R53JfZ8lm4nFsgtucxFyRwtJh+veYb1RjPDcjWVpEco9KlzwRQzvbsQdwTDtuaT6o+bxFQ4awSS/YfC5puM= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1715832349; x=1747368349; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=/GVW5S01UQkVLewpSI50IYqMJGhZ5pRv72RYPUCzBIo=; b=M6wu8U6PfyjVq6L6FtbGBqE8QAPYMOC3MwMR3kxLKsgqaNukZY3j42b7 +kacQN/s6s3zCkIG6Wcd7ScPuuX4GRH7AR42iTxBPQUiZvIwRMPcCriTO z1ezS6fV9a2EUemSLChfFkbkY2Qubdv4jmwQFtlZGYXtPl89Vb9PQ3QWc uVH2WkMyJlbAivct7OB5LAleP72tCk//pDFdQBkWjjD27j1zoBdXFFFUE vjfjCI/4iNwtQmT/xPbxXYKF+p/+W4A43EAl8ARvT94yWfQPIK7+wWKS3 u8XsSzhu3jUX0GaY+vopNnXQogjvL5a7cxOrfJj/ZN2bFIj/YeIn/X/11 w==; X-CSE-ConnectionGUID: VF9hczsCRt2buUmMKhxl3Q== X-CSE-MsgGUID: 3zDzCBQcTqqN7s4bDZuh6Q== X-IronPort-AV: E=McAfee;i="6600,9927,11074"; a="37297984" X-IronPort-AV: E=Sophos;i="6.08,163,1712646000"; d="scan'208";a="37297984" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by fmvoesa101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 May 2024 21:05:47 -0700 X-CSE-ConnectionGUID: hTt0OgcVSESJhmDq13RFGw== X-CSE-MsgGUID: 63qNx/nlTBOOrIDGQKL3UQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.08,163,1712646000"; d="scan'208";a="68736863" Received: from shvmail02.sh.intel.com ([10.239.244.9]) by orviesa001.jf.intel.com with ESMTP; 15 May 2024 21:05:45 -0700 Received: from pli-ubuntu.sh.intel.com (pli-ubuntu.sh.intel.com [10.239.159.47]) by shvmail02.sh.intel.com (Postfix) with ESMTP id 14DD41005666; Thu, 16 May 2024 12:05:44 +0800 (CST) From: pan2.li@intel.com To: gcc-patches@gcc.gnu.org Cc: juzhe.zhong@rivai.ai, kito.cheng@gmail.com, tamar.christina@arm.com, richard.guenther@gmail.com, Richard.Sandiford@arm.com, Pan Li Subject: [PATCH v2 1/3] Vect: Support loop len in vectorizable early exit Date: Thu, 16 May 2024 12:05:40 +0800 Message-Id: <20240516040542.2734412-1-pan2.li@intel.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Spam-Status: No, score=-11.9 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org From: Pan Li This patch adds early break auto-vectorization support for target which use length on partial vectorization. Consider this following example: unsigned vect_a[802]; unsigned vect_b[802]; void test (unsigned x, int n) { for (int i = 0; i < n; i++) { vect_b[i] = x + i; if (vect_a[i] > x) break; vect_a[i] = x; } } We use VCOND_MASK_LEN to simulate the generate (mask && i < len + bias). And then the IR of RVV looks like below: ... _87 = .SELECT_VL (ivtmp_85, POLY_INT_CST [32, 32]); _55 = (int) _87; ... mask_patt_6.13_69 = vect_cst__62 < vect__3.12_67; vec_len_mask_72 = .VCOND_MASK_LEN (mask_patt_6.13_69, { -1, ... }, \ {0, ... }, _87, 0); if (vec_len_mask_72 != { 0, ... }) goto ; [5.50%] else goto ; [94.50%] The below tests are passed for this patch: 1. The riscv fully regression tests. 2. The x86 bootstrap tests. 3. The x86 fully regression tests. gcc/ChangeLog: * tree-vect-stmts.cc (vectorizable_early_exit): Add loop len handling for one or multiple stmt. gcc/ChangeLog: * tree-vect-loop.cc (vect_gen_loop_len_mask): New func to gen the loop len mask. * tree-vect-stmts.cc (vectorizable_early_exit): Invoke the vect_gen_loop_len_mask for 1 or more stmt(s). * tree-vectorizer.h (vect_gen_loop_len_mask): New func decl for vect_gen_loop_len_mask. Signed-off-by: Pan Li --- gcc/tree-vect-loop.cc | 27 +++++++++++++++++++++++++++ gcc/tree-vect-stmts.cc | 17 +++++++++++++++-- gcc/tree-vectorizer.h | 4 ++++ 3 files changed, 46 insertions(+), 2 deletions(-) diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc index 361aec06488..83c0544b6aa 100644 --- a/gcc/tree-vect-loop.cc +++ b/gcc/tree-vect-loop.cc @@ -11416,6 +11416,33 @@ vect_get_loop_len (loop_vec_info loop_vinfo, gimple_stmt_iterator *gsi, return loop_len; } +/* Generate the tree for the loop len mask and return it. Given the lens, + nvectors, vectype, index and factor to gen the len mask as below. + + tree len_mask = VCOND_MASK_LEN (compare_mask, ones, zero, len, bias) +*/ +tree +vect_gen_loop_len_mask (loop_vec_info loop_vinfo, gimple_stmt_iterator *gsi, + gimple_stmt_iterator *cond_gsi, vec_loop_lens *lens, + unsigned int nvectors, tree vectype, tree stmt, + unsigned int index, unsigned int factor) +{ + tree all_one_mask = build_all_ones_cst (vectype); + tree all_zero_mask = build_zero_cst (vectype); + tree len = vect_get_loop_len (loop_vinfo, gsi, lens, nvectors, vectype, index, + factor); + tree bias = build_int_cst (intQI_type_node, + LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo)); + tree len_mask = make_temp_ssa_name (TREE_TYPE (stmt), NULL, "vec_len_mask"); + gcall *call = gimple_build_call_internal (IFN_VCOND_MASK_LEN, 5, stmt, + all_one_mask, all_zero_mask, len, + bias); + gimple_call_set_lhs (call, len_mask); + gsi_insert_before (cond_gsi, call, GSI_SAME_STMT); + + return len_mask; +} + /* Scale profiling counters by estimation for LOOP which is vectorized by factor VF. If FLAT is true, the loop we started with had unrealistically flat diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index b8a71605f1b..672959501bb 100644 --- a/gcc/tree-vect-stmts.cc +++ b/gcc/tree-vect-stmts.cc @@ -12895,7 +12895,9 @@ vectorizable_early_exit (vec_info *vinfo, stmt_vec_info stmt_info, ncopies = vect_get_num_copies (loop_vinfo, vectype); vec_loop_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo); + vec_loop_lens *lens = &LOOP_VINFO_LENS (loop_vinfo); bool masked_loop_p = LOOP_VINFO_FULLY_MASKED_P (loop_vinfo); + bool len_loop_p = LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo); /* Now build the new conditional. Pattern gimple_conds get dropped during codegen so we must replace the original insn. */ @@ -12959,12 +12961,11 @@ vectorizable_early_exit (vec_info *vinfo, stmt_vec_info stmt_info, { if (direct_internal_fn_supported_p (IFN_VCOND_MASK_LEN, vectype, OPTIMIZE_FOR_SPEED)) - return false; + vect_record_loop_len (loop_vinfo, lens, ncopies, vectype, 1); else vect_record_loop_mask (loop_vinfo, masks, ncopies, vectype, NULL); } - return true; } @@ -13017,6 +13018,15 @@ vectorizable_early_exit (vec_info *vinfo, stmt_vec_info stmt_info, stmts[i], &cond_gsi); workset.quick_push (stmt_mask); } + else if (len_loop_p) + for (unsigned i = 0; i < stmts.length (); i++) + { + tree len_mask = vect_gen_loop_len_mask (loop_vinfo, gsi, &cond_gsi, + lens, ncopies, vectype, + stmts[i], i, 1); + + workset.quick_push (len_mask); + } else workset.splice (stmts); @@ -13041,6 +13051,9 @@ vectorizable_early_exit (vec_info *vinfo, stmt_vec_info stmt_info, new_temp = prepare_vec_mask (loop_vinfo, TREE_TYPE (mask), mask, new_temp, &cond_gsi); } + else if (len_loop_p) + new_temp = vect_gen_loop_len_mask (loop_vinfo, gsi, &cond_gsi, lens, + ncopies, vectype, new_temp, 0, 1); } gcc_assert (new_temp); diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h index db44d730b70..93bc30ef660 100644 --- a/gcc/tree-vectorizer.h +++ b/gcc/tree-vectorizer.h @@ -2408,6 +2408,10 @@ extern void vect_record_loop_len (loop_vec_info, vec_loop_lens *, unsigned int, extern tree vect_get_loop_len (loop_vec_info, gimple_stmt_iterator *, vec_loop_lens *, unsigned int, tree, unsigned int, unsigned int); +extern tree vect_gen_loop_len_mask (loop_vec_info, gimple_stmt_iterator *, + gimple_stmt_iterator *, vec_loop_lens *, + unsigned int, tree, tree, unsigned int, + unsigned int); extern gimple_seq vect_gen_len (tree, tree, tree, tree); extern stmt_vec_info info_for_reduction (vec_info *, stmt_vec_info); extern bool reduction_fn_for_scalar_code (code_helper, internal_fn *); From patchwork Thu May 16 04:05:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Li, Pan2" X-Patchwork-Id: 1935793 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=XrvjwREf; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=8.43.85.97; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VfxNd4xG2z20KF for ; Thu, 16 May 2024 14:06:25 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 5A40A3849AF3 for ; Thu, 16 May 2024 04:06:23 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.7]) by sourceware.org (Postfix) with ESMTPS id F19B33865479 for ; Thu, 16 May 2024 04:05:51 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org F19B33865479 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=intel.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org F19B33865479 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=192.198.163.7 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1715832358; cv=none; b=HnVbUoOzHK5eoGXQq1nqwvaGXcrUPXocYgIOVuxsYdZHaFJD3kyY8qbE+z7JWrXk6lYXMfUeRZL/OIDlN1gk5KotDLzxgwkuOL0q6EtxydPRmAUHRcSUq4G9zPyaDp+lfm7M2uYt77QeBynBY7zQSsif+nEXZ9UFLkStNJckmH8= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1715832358; c=relaxed/simple; bh=Qf6P85tP7tWxCqPae7mFiqoaeyZ2irlUUq8xVLSCi/s=; h=DKIM-Signature:From:To:Subject:Date:Message-Id:MIME-Version; b=oaWSHXOk9F7S9c2Ycd5FWlWY0seYmKWvFwAtKF5ukXlSjLpZjmWBurmoLT8Eye1aj6WeR8tdJ5bftKWFzMuhsn4BzdqhJ+/l9dw1Zx/Rxt1NMOaz/EIq0J/vuuCg0uGYLKuVt1aJBkFYny4ErVVhLNBEE3TXjoNbo319jtDyB54= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1715832352; x=1747368352; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Qf6P85tP7tWxCqPae7mFiqoaeyZ2irlUUq8xVLSCi/s=; b=XrvjwREfbaivWcBqRc4lTUCr4Rg50Tpdabp9PdbEc3FkieutF0C6Ou9l IvRFRhb8ZKFrQ4GDwKYJtkVToQeHyoJLZ16jwdXRXCFVMZC6QPu/UEMgJ EYNaBW55TmJdF79VmyS8/AtHP/rzM1aLCyE5ccDCnMlJI9VkpnrWSKHuM qOD+cIaG2d4EwT8GRE0rFRS6NyRofaHGfBKBQmA7pnoXRVgEELlPXpmFm PuMRD2OBeiyOhJYV2riu0HqKl6V1v8E2ANS+9c9BUr5LEfV69NlzVFWr2 5b2ahVfCgWw4NjDbQkOKtmOnd/VPuHgsgO4JqNjfjre1Q/F9n0VRlqvgH Q==; X-CSE-ConnectionGUID: iclsm7QdR7SGMGQgTNTQ+w== X-CSE-MsgGUID: 8eLswsRYT7a0l4xdmOYLQg== X-IronPort-AV: E=McAfee;i="6600,9927,11074"; a="37297988" X-IronPort-AV: E=Sophos;i="6.08,163,1712646000"; d="scan'208";a="37297988" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by fmvoesa101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 May 2024 21:05:48 -0700 X-CSE-ConnectionGUID: 0cq0zNqVTI2zlatznu8dbA== X-CSE-MsgGUID: zzucohdLTOCo+xpE4Qw83Q== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.08,163,1712646000"; d="scan'208";a="68736867" Received: from shvmail02.sh.intel.com ([10.239.244.9]) by orviesa001.jf.intel.com with ESMTP; 15 May 2024 21:05:46 -0700 Received: from pli-ubuntu.sh.intel.com (pli-ubuntu.sh.intel.com [10.239.159.47]) by shvmail02.sh.intel.com (Postfix) with ESMTP id E8C3F1005681; Thu, 16 May 2024 12:05:44 +0800 (CST) From: pan2.li@intel.com To: gcc-patches@gcc.gnu.org Cc: juzhe.zhong@rivai.ai, kito.cheng@gmail.com, tamar.christina@arm.com, richard.guenther@gmail.com, Richard.Sandiford@arm.com, Pan Li Subject: [PATCH v2 2/3] RISC-V: Implement vectorizable early exit with vcond_mask_len Date: Thu, 16 May 2024 12:05:41 +0800 Message-Id: <20240516040542.2734412-2-pan2.li@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240516040542.2734412-1-pan2.li@intel.com> References: <20240516040542.2734412-1-pan2.li@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-11.5 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_ASCII_DIVIDERS, KAM_SHORT, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org From: Pan Li After we support the loop lens for the vectorizable, we would like to implement the feature for the RISC-V target. Given below example: unsigned vect_a[1923]; unsigned vect_b[1923]; void test (unsigned limit, int n) { for (int i = 0; i < n; i++) { vect_b[i] = limit + i; if (vect_a[i] > limit) { ret = vect_b[i]; return ret; } vect_a[i] = limit; } } Before this patch: ... .L8: sw a3,0(a5) addiw a0,a0,1 addi a4,a4,4 addi a5,a5,4 beq a1,a0,.L2 .L4: sw a0,0(a4) lw a2,0(a5) bleu a2,a3,.L8 ret After this patch: ... .L5: vsetvli a5,a3,e8,mf4,ta,ma vmv1r.v v4,v2 vsetvli t4,zero,e32,m1,ta,ma vmv.v.x v1,a5 vadd.vv v2,v2,v1 vsetvli zero,a5,e32,m1,ta,ma vadd.vv v5,v4,v3 slli a6,a5,2 vle32.v v1,0(t1) vmsltu.vv v1,v3,v1 vcpop.m t4,v1 beq t4,zero,.L4 vmv.x.s a4,v4 .L3: ... The below tests are passed for this patch: 1. The riscv fully regression tests. gcc/ChangeLog: * config/riscv/autovec-opt.md (*vcond_mask_len_popcount_): New pattern of vcond_mask_len_popcount for vector bool mode. * config/riscv/autovec.md (vcond_mask_len_): New pattern of vcond_mask_len for vector bool mode. (cbranch4): New pattern for vector bool mode. * config/riscv/vector-iterators.md: Add new unspec UNSPEC_SELECT_MASK. * config/riscv/vector.md (@pred_popcount): Add VLS mode to popcount pattern. (@pred_popcount): Ditto. gcc/testsuite/ChangeLog: * gcc.target/riscv/rvv/autovec/early-break-1.c: New test. * gcc.target/riscv/rvv/autovec/early-break-2.c: New test. Signed-off-by: Pan Li Signed-off-by: Pan Li --- gcc/config/riscv/autovec-opt.md | 33 ++++++++++ gcc/config/riscv/autovec.md | 61 +++++++++++++++++++ gcc/config/riscv/vector-iterators.md | 1 + gcc/config/riscv/vector.md | 18 +++--- .../riscv/rvv/autovec/early-break-1.c | 34 +++++++++++ .../riscv/rvv/autovec/early-break-2.c | 37 +++++++++++ 6 files changed, 175 insertions(+), 9 deletions(-) create mode 100644 gcc/testsuite/gcc.target/riscv/rvv/autovec/early-break-1.c create mode 100644 gcc/testsuite/gcc.target/riscv/rvv/autovec/early-break-2.c diff --git a/gcc/config/riscv/autovec-opt.md b/gcc/config/riscv/autovec-opt.md index 645dc53d868..04f85d8e455 100644 --- a/gcc/config/riscv/autovec-opt.md +++ b/gcc/config/riscv/autovec-opt.md @@ -1436,3 +1436,36 @@ (define_insn_and_split "*n" DONE; } [(set_attr "type" "vmalu")]) + +;; Optimization pattern for early break auto-vectorization +;; vcond_mask_len (mask, ones, zeros, len, bias) + vlmax popcount +;; -> non vlmax popcount (mask, len) +(define_insn_and_split "*vcond_mask_len_popcount_" + [(set (match_operand:P 0 "register_operand") + (popcount:P + (unspec:VB_VLS [ + (unspec:VB_VLS [ + (match_operand:VB_VLS 1 "register_operand") + (match_operand:VB_VLS 2 "const_1_operand") + (match_operand:VB_VLS 3 "const_0_operand") + (match_operand 4 "autovec_length_operand") + (match_operand 5 "const_0_operand")] UNSPEC_SELECT_MASK) + (match_operand 6 "autovec_length_operand") + (const_int 1) + (reg:SI VL_REGNUM) + (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)))] + "TARGET_VECTOR + && can_create_pseudo_p () + && riscv_vector::get_vector_mode (Pmode, GET_MODE_NUNITS (mode)).exists ()" + "#" + "&& 1" + [(const_int 0)] + { + riscv_vector::emit_nonvlmax_insn ( + code_for_pred_popcount (mode, Pmode), + riscv_vector::CPOP_OP, + operands, operands[4]); + DONE; + } + [(set_attr "type" "vector")] +) diff --git a/gcc/config/riscv/autovec.md b/gcc/config/riscv/autovec.md index aa1ae0fe075..1ee3c8052fb 100644 --- a/gcc/config/riscv/autovec.md +++ b/gcc/config/riscv/autovec.md @@ -2612,3 +2612,64 @@ (define_expand "rawmemchr" DONE; } ) + +;; ========================================================================= +;; == Early break auto-vectorization patterns +;; ========================================================================= + +;; vcond_mask_len (mask, 1s, 0s, len, bias) +;; => mask[i] = mask[i] && i < len ? 1 : 0 +(define_insn_and_split "vcond_mask_len_" + [(set (match_operand:VB 0 "register_operand") + (unspec: VB [ + (match_operand:VB 1 "register_operand") + (match_operand:VB 2 "const_1_operand") + (match_operand:VB 3 "const_0_operand") + (match_operand 4 "autovec_length_operand") + (match_operand 5 "const_0_operand")] UNSPEC_SELECT_MASK))] + "TARGET_VECTOR + && can_create_pseudo_p () + && riscv_vector::get_vector_mode (Pmode, GET_MODE_NUNITS (mode)).exists ()" + "#" + "&& 1" + [(const_int 0)] + { + machine_mode mode = riscv_vector::get_vector_mode (Pmode, + GET_MODE_NUNITS (mode)).require (); + rtx reg = gen_reg_rtx (mode); + riscv_vector::expand_vec_series (reg, const0_rtx, const1_rtx); + rtx dup_rtx = gen_rtx_VEC_DUPLICATE (mode, operands[4]); + insn_code icode = code_for_pred_cmp_scalar (mode); + rtx cmp = gen_rtx_fmt_ee (LTU, mode, reg, dup_rtx); + rtx ops[] = {operands[0], operands[1], operands[1], cmp, reg, operands[4]}; + emit_vlmax_insn (icode, riscv_vector::COMPARE_OP_MU, ops); + DONE; + } + [(set_attr "type" "vector")]) + +;; cbranch +(define_expand "cbranch4" + [(set (pc) + (if_then_else + (match_operator 0 "equality_operator" + [(match_operand:VB_VLS 1 "register_operand") + (match_operand:VB_VLS 2 "reg_or_0_operand")]) + (label_ref (match_operand 3 "")) + (pc)))] + "TARGET_VECTOR" + { + rtx pred; + if (operands[2] == CONST0_RTX (mode)) + pred = operands[1]; + else + pred = expand_binop (mode, xor_optab, operands[1], + operands[2], NULL_RTX, 0, + OPTAB_DIRECT); + rtx reg = gen_reg_rtx (Pmode); + rtx cpop_ops[] = {reg, pred}; + emit_vlmax_insn (code_for_pred_popcount (mode, Pmode), + riscv_vector::CPOP_OP, cpop_ops); + operands[1] = reg; + operands[2] = const0_rtx; + } +) diff --git a/gcc/config/riscv/vector-iterators.md b/gcc/config/riscv/vector-iterators.md index a24e1bf078f..76c27035a73 100644 --- a/gcc/config/riscv/vector-iterators.md +++ b/gcc/config/riscv/vector-iterators.md @@ -102,6 +102,7 @@ (define_c_enum "unspec" [ UNSPEC_WREDUC_SUMU UNSPEC_WREDUC_SUM_ORDERED UNSPEC_WREDUC_SUM_UNORDERED + UNSPEC_SELECT_MASK ]) (define_c_enum "unspecv" [ diff --git a/gcc/config/riscv/vector.md b/gcc/config/riscv/vector.md index 228d0f9a766..95451dc762b 100644 --- a/gcc/config/riscv/vector.md +++ b/gcc/config/riscv/vector.md @@ -6121,21 +6121,21 @@ (define_insn "@pred_not" (set_attr "vl_op_idx" "4") (set (attr "avl_type_idx") (const_int 5))]) -(define_insn "@pred_popcount" - [(set (match_operand:P 0 "register_operand" "=r") +(define_insn "@pred_popcount" + [(set (match_operand:P 0 "register_operand" "=r") (popcount:P - (unspec:VB - [(and:VB - (match_operand:VB 1 "vector_mask_operand" "vmWc1") - (match_operand:VB 2 "register_operand" " vr")) - (match_operand 3 "vector_length_operand" " rK") - (match_operand 4 "const_int_operand" " i") + (unspec:VB_VLS + [(and:VB_VLS + (match_operand:VB_VLS 1 "vector_mask_operand" "vmWc1") + (match_operand:VB_VLS 2 "register_operand" " vr")) + (match_operand 3 "vector_length_operand" " rK") + (match_operand 4 "const_int_operand" " i") (reg:SI VL_REGNUM) (reg:SI VTYPE_REGNUM)] UNSPEC_VPREDICATE)))] "TARGET_VECTOR" "vcpop.m\t%0,%2%p1" [(set_attr "type" "vmpop") - (set_attr "mode" "")]) + (set_attr "mode" "")]) (define_insn "@pred_ffs" [(set (match_operand:P 0 "register_operand" "=r") diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/early-break-1.c b/gcc/testsuite/gcc.target/riscv/rvv/autovec/early-break-1.c new file mode 100644 index 00000000000..f70979e81f1 --- /dev/null +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/early-break-1.c @@ -0,0 +1,34 @@ +/* { dg-do compile } */ +/* { dg-options "-march=rv64gcv -mabi=lp64d -O3 -fno-schedule-insns -fno-schedule-insns2 -fdump-tree-vect-details" } */ +/* { dg-final { check-function-bodies "**" "" } } */ + +#define N 803 + +unsigned vect_a[N]; +unsigned vect_b[N]; + +/* +** test: +** ... +** vmsltu\.vv\s+v[0-9]+\s*,v[0-9]+,\s*v[0-9]+ +** vcpop\.m\s+[atx][0-9]+\s*,v[0-9]+ +** ... +*/ +unsigned test (unsigned x, int n) +{ + unsigned ret = 0; + + for (int i = 0; i < n; i++) + { + vect_b[i] = x + i; + + if (vect_a[i] > x) + break; + + vect_a[i] = x; + } + + return ret; +} + +/* { dg-final { scan-tree-dump-times "LOOP VECTORIZED" 1 "vect" } } */ diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/early-break-2.c b/gcc/testsuite/gcc.target/riscv/rvv/autovec/early-break-2.c new file mode 100644 index 00000000000..d405783d2c4 --- /dev/null +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/early-break-2.c @@ -0,0 +1,37 @@ +/* { dg-do compile } */ +/* { dg-options "-march=rv64gcv -mabi=lp64d -O3 -fno-schedule-insns -fno-schedule-insns2 -fdump-tree-vect-details" } */ +/* { dg-final { check-function-bodies "**" "" } } */ + +#define N 1728 + +unsigned vect_a[N]; +unsigned vect_b[N]; + +/* +** test: +** ... +** vmsltu\.vv\s+v[0-9]+\s*,v[0-9]+,\s*v[0-9]+ +** vcpop\.m\s+[atx][0-9]+\s*,v[0-9]+ +** ... +*/ +unsigned test (unsigned limit, int n) +{ + unsigned ret = 0; + + for (int i = 0; i < n; i++) + { + vect_b[i] = limit + i; + + if (vect_a[i] > limit) + { + ret = vect_b[i]; + return ret; + } + + vect_a[i] = limit; + } + + return ret; +} + +/* { dg-final { scan-tree-dump-times "LOOP VECTORIZED" 1 "vect" } } */ From patchwork Thu May 16 04:05:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Li, Pan2" X-Patchwork-Id: 1935794 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=intel.com header.i=@intel.com header.a=rsa-sha256 header.s=Intel header.b=HujuFkAX; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VfxNl41ztz20KF for ; Thu, 16 May 2024 14:06:31 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 68F7D3849AF5 for ; Thu, 16 May 2024 04:06:23 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.7]) by sourceware.org (Postfix) with ESMTPS id 8D2AE384AB45 for ; Thu, 16 May 2024 04:05:52 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 8D2AE384AB45 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=intel.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=intel.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 8D2AE384AB45 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=192.198.163.7 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1715832358; cv=none; b=sPTFkYu90k4/0dydLUbNT5coqiRrj9/sC+UsTYtBtTAwktZZTl62DlsmI+BLLN9q1eT3g8hBoxc/Y69YH5lASvngh2iaPKV5w47xXZkhVYM0OCOJoBtJLjbN2GQUZ1xVTQCMOe2o2PuK/xoTjlAlGJnYpfL6BxquWMNyq7vv01I= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1715832358; c=relaxed/simple; bh=p/iK8fYM1+EwDicUUtj/HcArWnO8G3qYkcfn2So+kzo=; h=DKIM-Signature:From:To:Subject:Date:Message-Id:MIME-Version; b=q6E1MwmycGD1cIWG/agCXwEplsR72/qNsEXLv4rVglHnWzrnVtbT3ot+xu8h/yohYCafv19njyh9FkwOPSpAxsGxJRiKThwUjpPOOjT4SLFjTeVyMcnSRuK+JLrn67tmV5BlLWtc7OTpSwZc/xEKavzChDYjN+L1fLCH68BQDmo= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1715832352; x=1747368352; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=p/iK8fYM1+EwDicUUtj/HcArWnO8G3qYkcfn2So+kzo=; b=HujuFkAX3fVVhisW73YEHYwQgCb3UYcsSXGq3Vnr9VjjzWKsLPBpPHtx gOqI8YiH+lcGgp/b2z92zg+veg1YSa+ZlOwlrEot2Bh2buXPSoY1vw1b9 y3hPU0r5/wPSq0OyZ47hvZ36d8+LkJdEMYc6teuBLKWrHme/MSl8cdLlT 9xLb+bMHthDNuKbhI9caRG5UHSJ7IGqrlB0CU4zgVLIRTlRveFUPSDP+w 42xA5BGHZdes+N6JtA0y2a1QuvfWKqXg+TUfB8SY6nGj5PVa3anOkpjJP /ZTUyv7/Q65rlPrDa8nP+lxKJvjBHhxAiDA/Q6Tr1iT4bRPpVP6zO/9y7 w==; X-CSE-ConnectionGUID: J47RNpyTRo22KdoteNpovA== X-CSE-MsgGUID: 3iQyCCkFRouURZTNUKMySg== X-IronPort-AV: E=McAfee;i="6600,9927,11074"; a="37297993" X-IronPort-AV: E=Sophos;i="6.08,163,1712646000"; d="scan'208";a="37297993" Received: from orviesa001.jf.intel.com ([10.64.159.141]) by fmvoesa101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 15 May 2024 21:05:48 -0700 X-CSE-ConnectionGUID: rqOLHlXUR/qmzhfZCrTLgQ== X-CSE-MsgGUID: /NWy9rt3RomXkPRLbgslhg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.08,163,1712646000"; d="scan'208";a="68736870" Received: from shvmail02.sh.intel.com ([10.239.244.9]) by orviesa001.jf.intel.com with ESMTP; 15 May 2024 21:05:47 -0700 Received: from pli-ubuntu.sh.intel.com (pli-ubuntu.sh.intel.com [10.239.159.47]) by shvmail02.sh.intel.com (Postfix) with ESMTP id AAEBA100727B; Thu, 16 May 2024 12:05:45 +0800 (CST) From: pan2.li@intel.com To: gcc-patches@gcc.gnu.org Cc: juzhe.zhong@rivai.ai, kito.cheng@gmail.com, tamar.christina@arm.com, richard.guenther@gmail.com, Richard.Sandiford@arm.com, Pan Li Subject: [PATCH v2 3/3] RISC-V: Enable vectorizable early exit testsuite Date: Thu, 16 May 2024 12:05:42 +0800 Message-Id: <20240516040542.2734412-3-pan2.li@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240516040542.2734412-1-pan2.li@intel.com> References: <20240516040542.2734412-1-pan2.li@intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-11.9 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org From: Pan Li After we supported vectorizable early exit in RISC-V, we would like to enable the gcc vect test for vectorizable early test. The vect-early-break_124-pr114403.c failed to vectorize for now. Because that the __builtin_memcpy with 8 bytes failed to folded into int64 assignment during ccp1. We will improve that first and mark this as xfail for RISC-V. The below tests are passed for this patch: 1. The riscv fully regression tests. gcc/testsuite/ChangeLog: * gcc.dg/vect/slp-mask-store-1.c: Add pragma novector as it will have 2 times LOOP VECTORIZED in RISC-V. * gcc.dg/vect/vect-early-break_124-pr114403.c: Xfail for the riscv backend. * lib/target-supports.exp: Add RISC-V backend. Signed-off-by: Pan Li Signed-off-by: Pan Li --- gcc/testsuite/gcc.dg/vect/slp-mask-store-1.c | 2 ++ gcc/testsuite/gcc.dg/vect/vect-early-break_124-pr114403.c | 2 +- gcc/testsuite/lib/target-supports.exp | 2 ++ 3 files changed, 5 insertions(+), 1 deletion(-) diff --git a/gcc/testsuite/gcc.dg/vect/slp-mask-store-1.c b/gcc/testsuite/gcc.dg/vect/slp-mask-store-1.c index fdd9032da98..2f80bf89e5e 100644 --- a/gcc/testsuite/gcc.dg/vect/slp-mask-store-1.c +++ b/gcc/testsuite/gcc.dg/vect/slp-mask-store-1.c @@ -28,6 +28,8 @@ main () if (__builtin_memcmp (x, res, sizeof (x)) != 0) abort (); + +#pragma GCC novector for (int i = 0; i < 32; ++i) if (flag[i] != 0 && flag[i] != 1) abort (); diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break_124-pr114403.c b/gcc/testsuite/gcc.dg/vect/vect-early-break_124-pr114403.c index 51abf245ccb..101ae1e0eaa 100644 --- a/gcc/testsuite/gcc.dg/vect/vect-early-break_124-pr114403.c +++ b/gcc/testsuite/gcc.dg/vect/vect-early-break_124-pr114403.c @@ -2,7 +2,7 @@ /* { dg-require-effective-target vect_early_break_hw } */ /* { dg-require-effective-target vect_long_long } */ -/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */ +/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" { xfail riscv*-*-* } } } */ #include "tree-vect.h" diff --git a/gcc/testsuite/lib/target-supports.exp b/gcc/testsuite/lib/target-supports.exp index 6f5d477b128..ec9baa4f32a 100644 --- a/gcc/testsuite/lib/target-supports.exp +++ b/gcc/testsuite/lib/target-supports.exp @@ -4099,6 +4099,7 @@ proc check_effective_target_vect_early_break { } { || [check_effective_target_arm_v8_neon_ok] || [check_effective_target_sse4] || [istarget amdgcn-*-*] + || [check_effective_target_riscv_v] }}] } @@ -4114,6 +4115,7 @@ proc check_effective_target_vect_early_break_hw { } { || [check_effective_target_arm_v8_neon_hw] || [check_sse4_hw_available] || [istarget amdgcn-*-*] + || [check_effective_target_riscv_v_ok] }}] }