From patchwork Thu Sep 14 03:11:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kewen.Lin" X-Patchwork-Id: 1833954 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=gcc.gnu.org header.i=@gcc.gnu.org header.a=rsa-sha256 header.s=default header.b=HhI5ARmV; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4RmMnc0cQhz1yhd for ; Thu, 14 Sep 2023 13:12:34 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 703803858032 for ; Thu, 14 Sep 2023 03:12:32 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 703803858032 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1694661152; bh=hBnjte7FFsy6RRoK9snmkszsO0pbNv4vGDg463ngIMw=; h=To:Cc:Subject:Date:In-Reply-To:References:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=HhI5ARmVbmAp03l93Js8ryuBdNcYwV8YmCXFwnrJw/iGQ07Mv3YEoEtZFL8YsHzpY wtW5jYNnrVpyoUBrT4APOi5Yoeqv99Ivd+9WbL2bRFTu9Q42n9iaNxOB2xpXZsOJhJ rd9Sr5EBkR04j8JLfjj1apN3NL3XNzlEPjBJytF8= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by sourceware.org (Postfix) with ESMTPS id EC26C3858C2B for ; Thu, 14 Sep 2023 03:12:08 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org EC26C3858C2B Received: from pps.filterd (m0356516.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 38E38tU5028540; Thu, 14 Sep 2023 03:12:07 GMT Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3t3sq3rar4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 14 Sep 2023 03:12:06 +0000 Received: from m0356516.ppops.net (m0356516.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 38E3ALtU002343; Thu, 14 Sep 2023 03:12:06 GMT Received: from ppma12.dal12v.mail.ibm.com (dc.9e.1632.ip4.static.sl-reverse.com [50.22.158.220]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3t3sq3raqy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 14 Sep 2023 03:12:06 +0000 Received: from pps.filterd (ppma12.dal12v.mail.ibm.com [127.0.0.1]) by ppma12.dal12v.mail.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 38E18D4F024088; Thu, 14 Sep 2023 03:12:05 GMT Received: from smtprelay05.fra02v.mail.ibm.com ([9.218.2.225]) by ppma12.dal12v.mail.ibm.com (PPS) with ESMTPS id 3t131tg7j6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 14 Sep 2023 03:12:05 +0000 Received: from smtpav06.fra02v.mail.ibm.com (smtpav06.fra02v.mail.ibm.com [10.20.54.105]) by smtprelay05.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 38E3C3hq16056960 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 14 Sep 2023 03:12:03 GMT Received: from smtpav06.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B0D5E20040; Thu, 14 Sep 2023 03:12:03 +0000 (GMT) Received: from smtpav06.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E89932004B; Thu, 14 Sep 2023 03:12:02 +0000 (GMT) Received: from trout.aus.stglabs.ibm.com (unknown [9.40.194.100]) by smtpav06.fra02v.mail.ibm.com (Postfix) with ESMTP; Thu, 14 Sep 2023 03:12:02 +0000 (GMT) To: gcc-patches@gcc.gnu.org Cc: richard.guenther@gmail.com, richard.sandiford@arm.com Subject: [PATCH 02/10] vect: Move vect_model_store_cost next to the transform in vectorizable_store Date: Wed, 13 Sep 2023 22:11:51 -0500 Message-Id: <1539ec7d34af4e38467420b3aed342d708a64a48.1694657494.git.linkw@linux.ibm.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: References: MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: DF7KICfyTP2M0VtLkqkznXEtcQJoW0n4 X-Proofpoint-ORIG-GUID: cpsy1P73M4FF_amiWGjShHfXRIcpDQe7 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.601,FMLib:17.11.176.26 definitions=2023-09-13_19,2023-09-13_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 suspectscore=0 adultscore=0 spamscore=0 clxscore=1015 malwarescore=0 phishscore=0 bulkscore=0 lowpriorityscore=0 priorityscore=1501 impostorscore=0 mlxlogscore=973 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2308100000 definitions=main-2309140025 X-Spam-Status: No, score=-12.3 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_EF, GIT_PATCH_0, RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Kewen Lin via Gcc-patches From: "Kewen.Lin" Reply-To: Kewen Lin Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org Sender: "Gcc-patches" This patch is an initial patch to move costing next to the transform, it still adopts vect_model_store_cost for costing but moves and duplicates it down according to the handlings of different vect_memory_access_types or some special handling need, hope it can make the subsequent patches easy to review. This patch should not have any functional changes. gcc/ChangeLog: * tree-vect-stmts.cc (vectorizable_store): Move and duplicate the call to vect_model_store_cost down to some different transform paths according to the handlings of different vect_memory_access_types or some special handling need. --- gcc/tree-vect-stmts.cc | 79 ++++++++++++++++++++++++++++++++---------- 1 file changed, 60 insertions(+), 19 deletions(-) diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc index a5caaf0bca2..36f7c5b9f4b 100644 --- a/gcc/tree-vect-stmts.cc +++ b/gcc/tree-vect-stmts.cc @@ -8372,7 +8372,8 @@ vectorizable_store (vec_info *vinfo, return false; } - if (!vec_stmt) /* transformation not required. */ + bool costing_p = !vec_stmt; + if (costing_p) /* transformation not required. */ { STMT_VINFO_MEMORY_ACCESS_TYPE (stmt_info) = memory_access_type; @@ -8401,11 +8402,6 @@ vectorizable_store (vec_info *vinfo, "Vectorizing an unaligned access.\n"); STMT_VINFO_TYPE (stmt_info) = store_vec_info_type; - vect_model_store_cost (vinfo, stmt_info, ncopies, - memory_access_type, &gs_info, - alignment_support_scheme, - misalignment, vls_type, slp_node, cost_vec); - return true; } gcc_assert (memory_access_type == STMT_VINFO_MEMORY_ACCESS_TYPE (stmt_info)); @@ -8415,12 +8411,27 @@ vectorizable_store (vec_info *vinfo, if (memory_access_type == VMAT_GATHER_SCATTER && gs_info.decl) { - vect_build_scatter_store_calls (vinfo, stmt_info, gsi, vec_stmt, - &gs_info, mask); + if (costing_p) + vect_model_store_cost (vinfo, stmt_info, ncopies, memory_access_type, + &gs_info, alignment_support_scheme, misalignment, + vls_type, slp_node, cost_vec); + else + vect_build_scatter_store_calls (vinfo, stmt_info, gsi, vec_stmt, + &gs_info, mask); return true; } else if (STMT_VINFO_SIMD_LANE_ACCESS_P (stmt_info) >= 3) - return vectorizable_scan_store (vinfo, stmt_info, gsi, vec_stmt, ncopies); + { + gcc_assert (memory_access_type == VMAT_CONTIGUOUS); + if (costing_p) + { + vect_model_store_cost (vinfo, stmt_info, ncopies, memory_access_type, + &gs_info, alignment_support_scheme, + misalignment, vls_type, slp_node, cost_vec); + return true; + } + return vectorizable_scan_store (vinfo, stmt_info, gsi, vec_stmt, ncopies); + } if (grouped_store) { @@ -8449,13 +8460,21 @@ vectorizable_store (vec_info *vinfo, else ref_type = reference_alias_ptr_type (DR_REF (first_dr_info->dr)); - if (dump_enabled_p ()) - dump_printf_loc (MSG_NOTE, vect_location, - "transform store. ncopies = %d\n", ncopies); + if (!costing_p && dump_enabled_p ()) + dump_printf_loc (MSG_NOTE, vect_location, "transform store. ncopies = %d\n", + ncopies); if (memory_access_type == VMAT_ELEMENTWISE || memory_access_type == VMAT_STRIDED_SLP) { + if (costing_p) + { + vect_model_store_cost (vinfo, stmt_info, ncopies, memory_access_type, + &gs_info, alignment_support_scheme, + misalignment, vls_type, slp_node, cost_vec); + return true; + } + gimple_stmt_iterator incr_gsi; bool insert_after; gimple *incr; @@ -8718,8 +8737,9 @@ vectorizable_store (vec_info *vinfo, else if (memory_access_type == VMAT_GATHER_SCATTER) { aggr_type = elem_type; - vect_get_strided_load_store_ops (stmt_info, loop_vinfo, gsi, &gs_info, - &bump, &vec_offset, loop_lens); + if (!costing_p) + vect_get_strided_load_store_ops (stmt_info, loop_vinfo, gsi, &gs_info, + &bump, &vec_offset, loop_lens); } else { @@ -8731,7 +8751,7 @@ vectorizable_store (vec_info *vinfo, memory_access_type, loop_lens); } - if (mask) + if (mask && !costing_p) LOOP_VINFO_HAS_MASK_STORE (loop_vinfo) = true; /* In case the vectorization factor (VF) is bigger than the number @@ -8782,6 +8802,13 @@ vectorizable_store (vec_info *vinfo, if (memory_access_type == VMAT_LOAD_STORE_LANES) { gcc_assert (!slp && grouped_store); + if (costing_p) + { + vect_model_store_cost (vinfo, stmt_info, ncopies, memory_access_type, + &gs_info, alignment_support_scheme, + misalignment, vls_type, slp_node, cost_vec); + return true; + } for (j = 0; j < ncopies; j++) { gimple *new_stmt; @@ -8927,6 +8954,13 @@ vectorizable_store (vec_info *vinfo, if (memory_access_type == VMAT_GATHER_SCATTER) { gcc_assert (!slp && !grouped_store); + if (costing_p) + { + vect_model_store_cost (vinfo, stmt_info, ncopies, memory_access_type, + &gs_info, alignment_support_scheme, + misalignment, vls_type, slp_node, cost_vec); + return true; + } auto_vec vec_offsets; for (j = 0; j < ncopies; j++) { @@ -9091,7 +9125,7 @@ vectorizable_store (vec_info *vinfo, for (j = 0; j < ncopies; j++) { gimple *new_stmt; - if (j == 0) + if (j == 0 && !costing_p) { if (slp) { @@ -9158,7 +9192,7 @@ vectorizable_store (vec_info *vinfo, offset, &dummy, gsi, &ptr_incr, simd_lane_access_p, bump); } - else + else if (!costing_p) { gcc_assert (!LOOP_VINFO_USING_SELECT_VL_P (loop_vinfo)); /* DR_CHAIN is then used as an input to vect_permute_store_chain(). @@ -9179,7 +9213,7 @@ vectorizable_store (vec_info *vinfo, } new_stmt = NULL; - if (grouped_store) + if (!costing_p && grouped_store) /* Permute. */ vect_permute_store_chain (vinfo, dr_chain, group_size, stmt_info, gsi, &result_chain); @@ -9187,6 +9221,8 @@ vectorizable_store (vec_info *vinfo, stmt_vec_info next_stmt_info = first_stmt_info; for (i = 0; i < vec_num; i++) { + if (costing_p) + continue; unsigned misalign; unsigned HOST_WIDE_INT align; @@ -9361,7 +9397,7 @@ vectorizable_store (vec_info *vinfo, if (!next_stmt_info) break; } - if (!slp) + if (!slp && !costing_p) { if (j == 0) *vec_stmt = new_stmt; @@ -9369,6 +9405,11 @@ vectorizable_store (vec_info *vinfo, } } + if (costing_p) + vect_model_store_cost (vinfo, stmt_info, ncopies, memory_access_type, + &gs_info, alignment_support_scheme, misalignment, + vls_type, slp_node, cost_vec); + return true; }