diff mbox series

[4/5] Disable gather/scatter for non-first vectorized epilogue

Message ID 20241106143026.598B03858D28@sourceware.org
State New
Headers show
Series [1/5] Check LOOP_VINFO_PEELING_FOR_GAPS on epilog is supported | expand

Commit Message

Richard Biener Nov. 6, 2024, 2:29 p.m. UTC 
We currently make vect_check_gather_scatter happy by replacing SSA
name references in DR_REF for gather/scatter DRs but the replacement
process only works once since for the second epilogue we have SSA
names from the first epilogue in DR_REF but as we copied from the
original loop the SSA mapping doesn't work.

The following simply punts for non-first epilogues, those gather/scatter
recognized by patterns to IFNs are already analyzed and should work
fine.

Bootstrapped and tested on x86_64-unknown-linux-gnu.

	* tree-vect-data-refs.cc (vect_check_gather_scatter): Refuse
	to analyze DR_REF if from an epilogue that's not first.
	* tree-vect-loop.cc (update_epilogue_loop_vinfo): Add comment
	how the substitution in DR_REF is broken.
---
 gcc/tree-vect-data-refs.cc | 7 +++++++
 gcc/tree-vect-loop.cc      | 3 +++
 2 files changed, 10 insertions(+)
diff mbox series

Patch

diff --git a/gcc/tree-vect-data-refs.cc b/gcc/tree-vect-data-refs.cc
index 54ad5c8f3dc..6276139b71f 100644
--- a/gcc/tree-vect-data-refs.cc
+++ b/gcc/tree-vect-data-refs.cc
@@ -4319,6 +4319,13 @@  vect_check_gather_scatter (stmt_vec_info stmt_info, loop_vec_info loop_vinfo,
       masked_p = (ifn == IFN_MASK_LOAD || ifn == IFN_MASK_STORE);
     }
 
+  /* ???  For epilogues we adjust DR_REF to make the following stmt-based
+     analysis work, but this adjustment doesn't work for epilogues of
+     epilogues during transform, so disable gather/scatter in that case.  */
+  if (LOOP_VINFO_EPILOGUE_P (loop_vinfo)
+      && LOOP_VINFO_EPILOGUE_P (LOOP_VINFO_ORIG_LOOP_INFO (loop_vinfo)))
+    return false;
+
   /* True if we should aim to use internal functions rather than
      built-in functions.  */
   bool use_ifn_p = (DR_IS_READ (dr)
diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
index 5de90b6573a..41875683595 100644
--- a/gcc/tree-vect-loop.cc
+++ b/gcc/tree-vect-loop.cc
@@ -12294,6 +12294,9 @@  update_epilogue_loop_vinfo (class loop *epilogue, tree advance)
       if (STMT_VINFO_MEMORY_ACCESS_TYPE (vstmt_vinfo) == VMAT_GATHER_SCATTER
 	  || STMT_VINFO_GATHER_SCATTER_P (vstmt_vinfo))
 	{
+	  /* ???  As we copy epilogues from the main loop incremental
+	     replacement from an already replaced DR_REF from vectorizing
+	     the first epilogue will fail.  */
 	  DR_REF (dr)
 	    = simplify_replace_tree (DR_REF (dr), NULL_TREE, NULL_TREE,
 				     &find_in_mapping, &mapping);