From patchwork Mon Aug 17 16:34:07 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Enkovich X-Patchwork-Id: 508024 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from sourceware.org (server1.sourceware.org [209.132.180.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 76DBA1402A0 for ; Tue, 18 Aug 2015 02:34:35 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=gcc.gnu.org header.i=@gcc.gnu.org header.b=ptlYh3yE; dkim-atps=neutral DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:date :from:to:subject:message-id:mime-version:content-type; q=dns; s= default; b=Tr2I8fSPv+HCvOFGDxyRY3+v4QrAjDEvtRhkhZI4ai2hbD4sCKeT0 NHHElrqKAeARWT/ilh8xtRwrdghCI+l20jAkzUzcZ5pAeq+tFy9q39GO0AByQbKK nLkFZSOQI9dtuk/+mu8bmXHpWu2IDjKVZaDxG102HGywWcKPokoioY= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender:date :from:to:subject:message-id:mime-version:content-type; s= default; bh=DnU/7tP8dGEjEDsGmmb5UMysUP4=; b=ptlYh3yEDHA50Qg0yHrr iGlBP7dFM2EUzB65CwNRbf4Z0ZK7mocZnj2c8n2QQ6tHpKsSjEXWx2KLl1ATFn/G KlujV1f8JdHVnf+MHyBX65NrwyS1fHNtaWICpvPFvUkiiCObxLKe9ESX5enDZo0G FQYQeMcCJX+vdj0Chy61omw= Received: (qmail 40953 invoked by alias); 17 Aug 2015 16:34:27 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 40937 invoked by uid 89); 17 Aug 2015 16:34:25 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-1.5 required=5.0 tests=AWL, BAYES_00, FREEMAIL_FROM, RCVD_IN_DNSWL_LOW, SPF_PASS autolearn=ham version=3.3.2 X-HELO: mail-qg0-f45.google.com Received: from mail-qg0-f45.google.com (HELO mail-qg0-f45.google.com) (209.85.192.45) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (AES128-GCM-SHA256 encrypted) ESMTPS; Mon, 17 Aug 2015 16:34:23 +0000 Received: by qgeg42 with SMTP id g42so98137399qge.1 for ; Mon, 17 Aug 2015 09:34:21 -0700 (PDT) X-Received: by 10.140.130.196 with SMTP id 187mr4292208qhc.58.1439829260938; Mon, 17 Aug 2015 09:34:20 -0700 (PDT) Received: from msticlxl57.ims.intel.com ([192.55.54.40]) by smtp.gmail.com with ESMTPSA id q28sm1441936qkl.37.2015.08.17.09.34.18 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 17 Aug 2015 09:34:20 -0700 (PDT) Date: Mon, 17 Aug 2015 19:34:07 +0300 From: Ilya Enkovich To: gcc-patches@gcc.gnu.org Subject: [Scalar masks 3/x] Support scalar masks in MASK_LOAD, MASK_STORE and VEC_COND_EXPR Message-ID: <20150817163407.GC12565@msticlxl57.ims.intel.com> MIME-Version: 1.0 Content-Disposition: inline User-Agent: Mutt/1.5.23 (2014-03-12) X-IsSubscribed: yes Hi, This patch adds scalar masks support for MASK_LOAD, MASK_STORE and VEC_COND_EXPR. Each one gets new optab for scalar mask case and optab is chosen depending on operands type. For VEC_COND_EXPR it's actually unclear which mask to check in case comparison is used as the first operand. Thus I consider VEC_COND_EXPR with scalar mask just always use SSA_NAME or a constant as the first operand. Otherwise old optab is used. I think it's OK because existing vcond_mask_optab is already extended to handle 512-bit vectors. Alternatively both optabs may be checked in such cases. Any comments? Thanks, Ilya --- gcc/ChangeLog: 2015-08-17 Ilya Enkovich * internal-fn.c (expand_MASK_LOAD): Support scalar mask. (expand_MASK_STORE): Likewise. * optabs.c (get_vcond_icode): Likewise. (expand_vec_cond_expr_p): Likewise. (expand_vec_cond_expr): Likewise. (can_vec_mask_load_store_p): Add SCALAR_MASK arg. * optabs.h (can_vec_mask_load_store_p): Likewise. * optabs.def (smaskload_optab): New. (smaskstore_optab): New. (vcond_mask_optab): New. * tree-cfg.c (verify_gimple_assign_ternary): Support scalar mask. * tree-if-conv.c (ifcvt_can_use_mask_load_store): Adjust can_vec_mask_load_store_p call. * tree-vect-stmts.c (vectorizable_mask_load_store): Support scalar mask. (vectorizable_assignment): Skip scalars. (vectorizable_operation): Likewise. (vectorizable_store): Likewise. (vectorizable_load): Likewise. (vect_is_simple_cond): Support scalar mask. (vectorizable_condition): Likewise. diff --git a/gcc/internal-fn.c b/gcc/internal-fn.c index e785946..5f8c21a 100644 --- a/gcc/internal-fn.c +++ b/gcc/internal-fn.c @@ -1869,6 +1869,7 @@ expand_MASK_LOAD (gcall *stmt) struct expand_operand ops[3]; tree type, lhs, rhs, maskt; rtx mem, target, mask; + optab tab; maskt = gimple_call_arg (stmt, 2); lhs = gimple_call_lhs (stmt); @@ -1885,7 +1886,10 @@ expand_MASK_LOAD (gcall *stmt) create_output_operand (&ops[0], target, TYPE_MODE (type)); create_fixed_operand (&ops[1], mem); create_input_operand (&ops[2], mask, TYPE_MODE (TREE_TYPE (maskt))); - expand_insn (optab_handler (maskload_optab, TYPE_MODE (type)), 3, ops); + + tab = (VECTOR_TYPE_P (TREE_TYPE (maskt))) + ? maskload_optab : smaskload_optab; + expand_insn (optab_handler (tab, TYPE_MODE (type)), 3, ops); } static void @@ -1894,6 +1898,7 @@ expand_MASK_STORE (gcall *stmt) struct expand_operand ops[3]; tree type, lhs, rhs, maskt; rtx mem, reg, mask; + optab tab; maskt = gimple_call_arg (stmt, 2); rhs = gimple_call_arg (stmt, 3); @@ -1908,7 +1913,10 @@ expand_MASK_STORE (gcall *stmt) create_fixed_operand (&ops[0], mem); create_input_operand (&ops[1], reg, TYPE_MODE (type)); create_input_operand (&ops[2], mask, TYPE_MODE (TREE_TYPE (maskt))); - expand_insn (optab_handler (maskstore_optab, TYPE_MODE (type)), 3, ops); + + tab = (VECTOR_TYPE_P (TREE_TYPE (maskt))) + ? maskstore_optab : smaskstore_optab; + expand_insn (optab_handler (tab, TYPE_MODE (type)), 3, ops); } static void diff --git a/gcc/optabs.c b/gcc/optabs.c index bf466ca..4c5e683 100644 --- a/gcc/optabs.c +++ b/gcc/optabs.c @@ -6870,7 +6870,9 @@ static inline enum insn_code get_vcond_icode (machine_mode vmode, machine_mode cmode, bool uns) { enum insn_code icode = CODE_FOR_nothing; - if (uns) + if (!VECTOR_MODE_P (cmode)) + icode = optab_handler (vcond_mask_optab, vmode); + else if (uns) icode = convert_optab_handler (vcondu_optab, vmode, cmode); else icode = convert_optab_handler (vcond_optab, vmode, cmode); @@ -6886,8 +6888,9 @@ expand_vec_cond_expr_p (tree value_type, tree cmp_op_type) { machine_mode value_mode = TYPE_MODE (value_type); machine_mode cmp_op_mode = TYPE_MODE (cmp_op_type); - if (GET_MODE_SIZE (value_mode) != GET_MODE_SIZE (cmp_op_mode) - || GET_MODE_NUNITS (value_mode) != GET_MODE_NUNITS (cmp_op_mode) + if ((VECTOR_MODE_P (cmp_op_mode) + && (GET_MODE_SIZE (value_mode) != GET_MODE_SIZE (cmp_op_mode) + || GET_MODE_NUNITS (value_mode) != GET_MODE_NUNITS (cmp_op_mode))) || get_vcond_icode (TYPE_MODE (value_type), TYPE_MODE (cmp_op_type), TYPE_UNSIGNED (cmp_op_type)) == CODE_FOR_nothing) return false; @@ -6909,6 +6912,7 @@ expand_vec_cond_expr (tree vec_cond_type, tree op0, tree op1, tree op2, bool unsignedp; tree op0a, op0b; enum tree_code tcode; + bool masked = false; if (COMPARISON_CLASS_P (op0)) { @@ -6916,6 +6920,8 @@ expand_vec_cond_expr (tree vec_cond_type, tree op0, tree op1, tree op2, op0b = TREE_OPERAND (op0, 1); tcode = TREE_CODE (op0); } + else if (!VECTOR_TYPE_P (TREE_TYPE (op0))) + masked = true; else { /* Fake op0 < 0. */ @@ -6924,28 +6930,51 @@ expand_vec_cond_expr (tree vec_cond_type, tree op0, tree op1, tree op2, op0b = build_zero_cst (TREE_TYPE (op0)); tcode = LT_EXPR; } - unsignedp = TYPE_UNSIGNED (TREE_TYPE (op0a)); - cmp_op_mode = TYPE_MODE (TREE_TYPE (op0a)); + if (masked) + { + cmp_op_mode = TYPE_MODE (TREE_TYPE (op0)); + icode = get_vcond_icode (mode, cmp_op_mode, 0); + if (icode == CODE_FOR_nothing) + return 0; - gcc_assert (GET_MODE_SIZE (mode) == GET_MODE_SIZE (cmp_op_mode) - && GET_MODE_NUNITS (mode) == GET_MODE_NUNITS (cmp_op_mode)); + comparison = expand_normal (op0); + rtx_op1 = expand_normal (op1); + rtx_op2 = expand_normal (op2); - icode = get_vcond_icode (mode, cmp_op_mode, unsignedp); - if (icode == CODE_FOR_nothing) - return 0; + comparison = force_reg (GET_MODE (comparison), comparison); + rtx_op1 = force_reg (GET_MODE (rtx_op1), rtx_op1); - comparison = vector_compare_rtx (tcode, op0a, op0b, unsignedp, icode, 4); - rtx_op1 = expand_normal (op1); - rtx_op2 = expand_normal (op2); + create_output_operand (&ops[0], target, mode); + create_input_operand (&ops[1], rtx_op1, mode); + create_input_operand (&ops[2], rtx_op2, mode); + create_input_operand (&ops[3], comparison, cmp_op_mode); + expand_insn (icode, 4, ops); + } + else + { + unsignedp = TYPE_UNSIGNED (TREE_TYPE (op0a)); + cmp_op_mode = TYPE_MODE (TREE_TYPE (op0a)); - create_output_operand (&ops[0], target, mode); - create_input_operand (&ops[1], rtx_op1, mode); - create_input_operand (&ops[2], rtx_op2, mode); - create_fixed_operand (&ops[3], comparison); - create_fixed_operand (&ops[4], XEXP (comparison, 0)); - create_fixed_operand (&ops[5], XEXP (comparison, 1)); - expand_insn (icode, 6, ops); + gcc_assert (GET_MODE_SIZE (mode) == GET_MODE_SIZE (cmp_op_mode) + && GET_MODE_NUNITS (mode) == GET_MODE_NUNITS (cmp_op_mode)); + + icode = get_vcond_icode (mode, cmp_op_mode, unsignedp); + if (icode == CODE_FOR_nothing) + return 0; + + comparison = vector_compare_rtx (tcode, op0a, op0b, unsignedp, icode, 4); + rtx_op1 = expand_normal (op1); + rtx_op2 = expand_normal (op2); + + create_output_operand (&ops[0], target, mode); + create_input_operand (&ops[1], rtx_op1, mode); + create_input_operand (&ops[2], rtx_op2, mode); + create_fixed_operand (&ops[3], comparison); + create_fixed_operand (&ops[4], XEXP (comparison, 0)); + create_fixed_operand (&ops[5], XEXP (comparison, 1)); + expand_insn (icode, 6, ops); + } return ops[0].value; } @@ -7074,9 +7103,11 @@ expand_mult_highpart (machine_mode mode, rtx op0, rtx op1, /* Return true if target supports vector masked load/store for mode. */ bool -can_vec_mask_load_store_p (machine_mode mode, bool is_load) +can_vec_mask_load_store_p (machine_mode mode, bool is_load, bool scalar_mask) { - optab op = is_load ? maskload_optab : maskstore_optab; + optab op = scalar_mask + ? is_load ? smaskload_optab : smaskstore_optab + : is_load ? maskload_optab : maskstore_optab; machine_mode vmode; unsigned int vector_sizes; diff --git a/gcc/optabs.def b/gcc/optabs.def index dc664d1..52316e4 100644 --- a/gcc/optabs.def +++ b/gcc/optabs.def @@ -266,8 +266,11 @@ OPTAB_D (usad_optab, "usad$I$a") OPTAB_D (ssad_optab, "ssad$I$a") OPTAB_D (maskload_optab, "maskload$a") OPTAB_D (maskstore_optab, "maskstore$a") +OPTAB_D (smaskload_optab, "load$a_mask") +OPTAB_D (smaskstore_optab, "store$a_mask") OPTAB_D (gen_mask_optab, "gen_mask$a") OPTAB_D (gen_masku_optab, "gen_masku$a") +OPTAB_D (vcond_mask_optab, "vcond$a_mask") OPTAB_D (vec_extract_optab, "vec_extract$a") OPTAB_D (vec_init_optab, "vec_init$a") OPTAB_D (vec_pack_sfix_trunc_optab, "vec_pack_sfix_trunc_$a") diff --git a/gcc/optabs.h b/gcc/optabs.h index d7b7fb0..770ed24 100644 --- a/gcc/optabs.h +++ b/gcc/optabs.h @@ -514,7 +514,7 @@ extern int can_mult_highpart_p (machine_mode, bool); extern rtx expand_mult_highpart (machine_mode, rtx, rtx, rtx, bool); /* Return true if target supports vector masked load/store for mode. */ -extern bool can_vec_mask_load_store_p (machine_mode, bool); +extern bool can_vec_mask_load_store_p (machine_mode, bool, bool); /* Return true if there is an inline compare and swap pattern. */ extern bool can_compare_and_swap_p (machine_mode, bool); diff --git a/gcc/tree-cfg.c b/gcc/tree-cfg.c index 5bae411..2f7060b 100644 --- a/gcc/tree-cfg.c +++ b/gcc/tree-cfg.c @@ -3980,11 +3980,15 @@ verify_gimple_assign_ternary (gassign *stmt) break; case VEC_COND_EXPR: - if (!VECTOR_INTEGER_TYPE_P (rhs1_type) - || TYPE_SIGN (rhs1_type) != SIGNED - || TYPE_SIZE (rhs1_type) != TYPE_SIZE (lhs_type) - || TYPE_VECTOR_SUBPARTS (rhs1_type) - != TYPE_VECTOR_SUBPARTS (lhs_type)) + if ((!VECTOR_INTEGER_TYPE_P (rhs1_type) + || TYPE_SIGN (rhs1_type) != SIGNED + || TYPE_SIZE (rhs1_type) != TYPE_SIZE (lhs_type) + || TYPE_VECTOR_SUBPARTS (rhs1_type) + != TYPE_VECTOR_SUBPARTS (lhs_type)) + && (!INTEGRAL_TYPE_P (rhs1_type) + || TYPE_SIGN (rhs1_type) != UNSIGNED + || TYPE_PRECISION (rhs1_type) + != TYPE_VECTOR_SUBPARTS (lhs_type))) { error ("the first argument of a VEC_COND_EXPR must be of a signed " "integral vector type of the same size and number of " diff --git a/gcc/tree-if-conv.c b/gcc/tree-if-conv.c index 73dcecd..27c575e 100644 --- a/gcc/tree-if-conv.c +++ b/gcc/tree-if-conv.c @@ -812,7 +812,8 @@ ifcvt_can_use_mask_load_store (gimple stmt) || VECTOR_MODE_P (mode)) return false; - if (can_vec_mask_load_store_p (mode, is_load)) + if (can_vec_mask_load_store_p (mode, is_load, + targetm.vectorize.use_scalar_mask_p ())) return true; return false; diff --git a/gcc/tree-vect-stmts.c b/gcc/tree-vect-stmts.c index 2ddd434..d0e0833 100644 --- a/gcc/tree-vect-stmts.c +++ b/gcc/tree-vect-stmts.c @@ -1776,17 +1776,25 @@ vectorizable_mask_load_store (gimple stmt, gimple_stmt_iterator *gsi, gimple def_stmt; tree def; enum vect_def_type dt; + bool scalar_mask; if (slp_node != NULL) return false; + if (!VECTOR_TYPE_P (vectype)) + return false; + + nunits = TYPE_VECTOR_SUBPARTS (vectype); + ncopies = LOOP_VINFO_VECT_FACTOR (loop_vinfo) / nunits; gcc_assert (ncopies >= 1); is_store = gimple_call_internal_fn (stmt) == IFN_MASK_STORE; mask = gimple_call_arg (stmt, 2); + scalar_mask = (TREE_CODE (TREE_TYPE (mask)) == BOOLEAN_TYPE); if (TYPE_PRECISION (TREE_TYPE (mask)) - != GET_MODE_BITSIZE (TYPE_MODE (TREE_TYPE (vectype)))) + != GET_MODE_BITSIZE (TYPE_MODE (TREE_TYPE (vectype))) + && !scalar_mask) return false; /* FORNOW. This restriction should be relaxed. */ @@ -1848,7 +1856,8 @@ vectorizable_mask_load_store (gimple stmt, gimple_stmt_iterator *gsi, : DR_STEP (dr), size_zero_node) <= 0) return false; else if (!VECTOR_MODE_P (TYPE_MODE (vectype)) - || !can_vec_mask_load_store_p (TYPE_MODE (vectype), !is_store)) + || !can_vec_mask_load_store_p (TYPE_MODE (vectype), !is_store, + scalar_mask)) return false; if (TREE_CODE (mask) != SSA_NAME) @@ -1864,6 +1873,12 @@ vectorizable_mask_load_store (gimple stmt, gimple_stmt_iterator *gsi, if (!vect_is_simple_use (rhs, stmt, loop_vinfo, NULL, &def_stmt, &def, &dt)) return false; + /* Can't store a mask to a vector. Patterns should catch + and handle it similar to regular bool stores. */ + if ((TYPE_PRECISION (TREE_TYPE (rhs)) == 1 + && TYPE_UNSIGNED (TREE_TYPE (rhs))) + || TREE_CODE (TREE_TYPE (rhs)) == BOOLEAN_TYPE) + return false; } if (!vec_stmt) /* transformation not required. */ @@ -4189,6 +4204,10 @@ vectorizable_assignment (gimple stmt, gimple_stmt_iterator *gsi, op = TREE_OPERAND (op, 0); tree vectype = STMT_VINFO_VECTYPE (stmt_info); + + if (!VECTOR_TYPE_P (vectype)) + return false; + unsigned int nunits = TYPE_VECTOR_SUBPARTS (vectype); /* Multiple types in SLP are handled by creating the appropriate number of @@ -4772,10 +4791,14 @@ vectorizable_operation (gimple stmt, gimple_stmt_iterator *gsi, scalar_dest = gimple_assign_lhs (stmt); vectype_out = STMT_VINFO_VECTYPE (stmt_info); + if (!VECTOR_TYPE_P (vectype_out)) + return false; + /* Most operations cannot handle bit-precision types without extra truncations. */ if ((TYPE_PRECISION (TREE_TYPE (scalar_dest)) != GET_MODE_PRECISION (TYPE_MODE (TREE_TYPE (scalar_dest)))) + && TREE_CODE (TREE_TYPE (scalar_dest)) != BOOLEAN_TYPE /* Exception are bitwise binary operations. */ && code != BIT_IOR_EXPR && code != BIT_XOR_EXPR @@ -4816,6 +4839,9 @@ vectorizable_operation (gimple stmt, gimple_stmt_iterator *gsi, return false; } + if (!VECTOR_TYPE_P (vectype)) + return false; + nunits_out = TYPE_VECTOR_SUBPARTS (vectype_out); nunits_in = TYPE_VECTOR_SUBPARTS (vectype); if (nunits_out != nunits_in) @@ -5172,6 +5198,10 @@ vectorizable_store (gimple stmt, gimple_stmt_iterator *gsi, gimple *vec_stmt, gcc_assert (gimple_assign_single_p (stmt)); tree vectype = STMT_VINFO_VECTYPE (stmt_info); + + if (!VECTOR_TYPE_P (vectype)) + return false; + unsigned int nunits = TYPE_VECTOR_SUBPARTS (vectype); if (loop_vinfo) @@ -6013,6 +6043,10 @@ vectorizable_load (gimple stmt, gimple_stmt_iterator *gsi, gimple *vec_stmt, return false; tree vectype = STMT_VINFO_VECTYPE (stmt_info); + + if (!VECTOR_TYPE_P (vectype)) + return false; + int nunits = TYPE_VECTOR_SUBPARTS (vectype); if (loop_vinfo) @@ -7086,6 +7120,16 @@ vect_is_simple_cond (tree cond, gimple stmt, loop_vec_info loop_vinfo, enum vect_def_type dt; tree vectype1 = NULL_TREE, vectype2 = NULL_TREE; + if (TREE_CODE (cond) == SSA_NAME + && TREE_TYPE (cond) == boolean_type_node) + { + gimple lhs_def_stmt = SSA_NAME_DEF_STMT (cond); + if (!vect_is_simple_use_1 (cond, stmt, loop_vinfo, bb_vinfo, + &lhs_def_stmt, &def, &dt, comp_vectype)) + return false; + return true; + } + if (!COMPARISON_CLASS_P (cond)) return false; @@ -7158,6 +7202,7 @@ vectorizable_condition (gimple stmt, gimple_stmt_iterator *gsi, vec vec_oprnds2 = vNULL; vec vec_oprnds3 = vNULL; tree vec_cmp_type; + bool masked; if (reduc_index && STMT_SLP_TYPE (stmt_info)) return false; @@ -7189,6 +7234,10 @@ vectorizable_condition (gimple stmt, gimple_stmt_iterator *gsi, return false; tree vectype = STMT_VINFO_VECTYPE (stmt_info); + + if (!VECTOR_TYPE_P (vectype)) + return false; + int nunits = TYPE_VECTOR_SUBPARTS (vectype); if (slp_node || PURE_SLP_STMT (stmt_info)) @@ -7233,6 +7282,9 @@ vectorizable_condition (gimple stmt, gimple_stmt_iterator *gsi, && TREE_CODE (else_clause) != FIXED_CST) return false; + masked = (TREE_CODE (cond_expr) == SSA_NAME); + gcc_assert (masked ^ VECTOR_TYPE_P (comp_vectype)); + unsigned int prec = GET_MODE_BITSIZE (TYPE_MODE (TREE_TYPE (vectype))); /* The result of a vector comparison should be signed type. */ tree cmp_type = build_nonstandard_integer_type (prec, 0); @@ -7271,14 +7323,20 @@ vectorizable_condition (gimple stmt, gimple_stmt_iterator *gsi, auto_vec ops; auto_vec, 4> vec_defs; - ops.safe_push (TREE_OPERAND (cond_expr, 0)); - ops.safe_push (TREE_OPERAND (cond_expr, 1)); + if (masked) + ops.safe_push (cond_expr); + else + { + ops.safe_push (TREE_OPERAND (cond_expr, 0)); + ops.safe_push (TREE_OPERAND (cond_expr, 1)); + } ops.safe_push (then_clause); ops.safe_push (else_clause); vect_get_slp_defs (ops, slp_node, &vec_defs, -1); vec_oprnds3 = vec_defs.pop (); vec_oprnds2 = vec_defs.pop (); - vec_oprnds1 = vec_defs.pop (); + if (!masked) + vec_oprnds1 = vec_defs.pop (); vec_oprnds0 = vec_defs.pop (); ops.release (); @@ -7287,17 +7345,29 @@ vectorizable_condition (gimple stmt, gimple_stmt_iterator *gsi, else { gimple gtemp; - vec_cond_lhs = - vect_get_vec_def_for_operand (TREE_OPERAND (cond_expr, 0), - stmt, NULL); - vect_is_simple_use (TREE_OPERAND (cond_expr, 0), stmt, - loop_vinfo, NULL, >emp, &def, &dts[0]); - - vec_cond_rhs = - vect_get_vec_def_for_operand (TREE_OPERAND (cond_expr, 1), - stmt, NULL); - vect_is_simple_use (TREE_OPERAND (cond_expr, 1), stmt, - loop_vinfo, NULL, >emp, &def, &dts[1]); + + if (masked) + { + vec_cond_lhs = + vect_get_vec_def_for_operand (cond_expr, stmt, NULL); + vect_is_simple_use (cond_expr, stmt, loop_vinfo, NULL, + >emp, &def, &dts[0]); + } + else + { + vec_cond_lhs = + vect_get_vec_def_for_operand (TREE_OPERAND (cond_expr, 0), + stmt, NULL); + vect_is_simple_use (TREE_OPERAND (cond_expr, 0), stmt, + loop_vinfo, NULL, >emp, &def, &dts[0]); + + vec_cond_rhs = + vect_get_vec_def_for_operand (TREE_OPERAND (cond_expr, 1), + stmt, NULL); + vect_is_simple_use (TREE_OPERAND (cond_expr, 1), stmt, + loop_vinfo, NULL, >emp, &def, &dts[1]); + } + if (reduc_index == 1) vec_then_clause = reduc_def; else @@ -7322,8 +7392,9 @@ vectorizable_condition (gimple stmt, gimple_stmt_iterator *gsi, { vec_cond_lhs = vect_get_vec_def_for_stmt_copy (dts[0], vec_oprnds0.pop ()); - vec_cond_rhs = vect_get_vec_def_for_stmt_copy (dts[1], - vec_oprnds1.pop ()); + if (!masked) + vec_cond_rhs = vect_get_vec_def_for_stmt_copy (dts[1], + vec_oprnds1.pop ()); vec_then_clause = vect_get_vec_def_for_stmt_copy (dts[2], vec_oprnds2.pop ()); vec_else_clause = vect_get_vec_def_for_stmt_copy (dts[3], @@ -7333,7 +7404,8 @@ vectorizable_condition (gimple stmt, gimple_stmt_iterator *gsi, if (!slp_node) { vec_oprnds0.quick_push (vec_cond_lhs); - vec_oprnds1.quick_push (vec_cond_rhs); + if (!masked) + vec_oprnds1.quick_push (vec_cond_rhs); vec_oprnds2.quick_push (vec_then_clause); vec_oprnds3.quick_push (vec_else_clause); } @@ -7341,12 +7413,17 @@ vectorizable_condition (gimple stmt, gimple_stmt_iterator *gsi, /* Arguments are ready. Create the new vector stmt. */ FOR_EACH_VEC_ELT (vec_oprnds0, i, vec_cond_lhs) { - vec_cond_rhs = vec_oprnds1[i]; vec_then_clause = vec_oprnds2[i]; vec_else_clause = vec_oprnds3[i]; - vec_compare = build2 (TREE_CODE (cond_expr), vec_cmp_type, - vec_cond_lhs, vec_cond_rhs); + if (masked) + vec_compare = vec_cond_lhs; + else + { + vec_cond_rhs = vec_oprnds1[i]; + vec_compare = build2 (TREE_CODE (cond_expr), vec_cmp_type, + vec_cond_lhs, vec_cond_rhs); + } vec_cond_expr = build3 (VEC_COND_EXPR, vectype, vec_compare, vec_then_clause, vec_else_clause); @@ -7359,12 +7436,12 @@ vectorizable_condition (gimple stmt, gimple_stmt_iterator *gsi, } if (slp_node) - continue; + continue; if (j == 0) - STMT_VINFO_VEC_STMT (stmt_info) = *vec_stmt = new_stmt; + STMT_VINFO_VEC_STMT (stmt_info) = *vec_stmt = new_stmt; else - STMT_VINFO_RELATED_STMT (prev_stmt_info) = new_stmt; + STMT_VINFO_RELATED_STMT (prev_stmt_info) = new_stmt; prev_stmt_info = vinfo_for_stmt (new_stmt); }