From patchwork Sun Aug 11 21:00:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robin Dapp X-Patchwork-Id: 1971384 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=ivhIDmdK; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces~incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Whqpc33zWz1yYl for ; Mon, 12 Aug 2024 07:02:16 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 4560F385DDCF for ; Sun, 11 Aug 2024 21:02:14 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail-lf1-x12b.google.com (mail-lf1-x12b.google.com [IPv6:2a00:1450:4864:20::12b]) by sourceware.org (Postfix) with ESMTPS id BE5AF3858D39 for ; Sun, 11 Aug 2024 21:00:51 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org BE5AF3858D39 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=gmail.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org BE5AF3858D39 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2a00:1450:4864:20::12b ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1723410054; cv=none; b=PjEQrhT/AyyH8FzE3s3Nrk++IIuYrtvsatjq8XKjTTKofcR5ShqwUCpgXs9Xr6CLi0tNyShfPvUvD2VWW0/bA2LemVDmye6mSAGwMTshwWZQFlBbThjgd0prc5BfuInjZSnU8l0XN8VEbTvMX4/E7B/eaZxY579VeqmdHAAbUZI= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1723410054; c=relaxed/simple; bh=2OQX0oeM8HBVbUKX7p5hdFsivlG0iV2tUIV+PM4w3js=; h=DKIM-Signature:Mime-Version:Date:Message-Id:From:Subject:To; b=QyJ1K868DsnbZi7OIMjzXKtt+P0HnmweZLlhWptj1GUEDW4+RVYda47teNAjtRBR64KFBBGpNWpBdNlbDoT0ZpabFWBjyUGo54+OnbN2TzffIS3PnbP4wgNkU46a/wKrRJ2bpzjO1LRjZa0zRTmcVC7qrPdlXdeLirnLZB11eiY= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-lf1-x12b.google.com with SMTP id 2adb3069b0e04-530e062217eso4498010e87.1 for ; Sun, 11 Aug 2024 14:00:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1723410050; x=1724014850; darn=gcc.gnu.org; h=to:cc:subject:from:message-id:date:content-transfer-encoding :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=NJFt71B4HgU/z+9vIACDv6CKrcm9xRFQGZKF5gXAQgg=; b=ivhIDmdKhwYUvbuEXdbL+F4D3pslWwY6XqrKPtxLXwQsrkMFDj8Y8m0aQMIiEeardd qzaRFAkeSUQup1Xdz6FnAvLdAMxKUp/wMJwQ27TTBbXME5UOUXYR7Oo7FP/N5lyoWasa Ec1aP1CORbNGuFvRQ/hTCuHTu4xFOEdkuRP0ox9r9c7CwRanFGFsuOF5vtS0Y9OV3cUX jCJ79xIOE7CPncGTUCbsV/ngoNKPUqn3c5vv+wy+J3A3RUpRqllUuMKfLGp3PJ+AvgCt YYHlBRZsKuI6+Q3dOfWzmanNkDFHdRLzIVjzPwW/i46mczrYJx8dWSeG+MencpQ+UlNk odqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723410050; x=1724014850; h=to:cc:subject:from:message-id:date:content-transfer-encoding :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=NJFt71B4HgU/z+9vIACDv6CKrcm9xRFQGZKF5gXAQgg=; b=ErC3ArZBAd+zBUTQLl820BlPVLM7ZSpR6hPlW1QBL5HNm2LDWfYj9DNcRgOBMKNYX9 B1arNaIH/Xk86Jfrm1X+LHi69ItTi9UTijIPgLmul7fuArSa0AZIsYlOZ/DR4UhSk23J 1/uTyh+LlpD+pIBlTXXixPNtoOn0rcFLb+019mQ+JpFpD2rjbHTHfvmiDivkfu8+LEjf F38DYzxbWud6DcYXV7vSEDGP/i3a+u+4Hoo+MKyTDH8NIN8MFgse81feb4wPtxC4qL4u Z/sjm7TRC4KEXqR1GXdV1jFx1tgdsvFIyC1YAggqnA7FOQX7C5XbNh0PVCpheEgC7FpV rokg== X-Gm-Message-State: AOJu0YzPHdB+15yIpUZ4J9A1KbMV/Puky1i5ULXl75OVK2ebQe2VwIOn fJqxo85zsdqngT2n6f1y0b/aFHp5TxZOuKi4VSE3YDbjjkCdd7g5pWeL1w== X-Google-Smtp-Source: AGHT+IHx2KLDcHbtV6uSjdM/5vYGw4FWsFpfHOnRIvBpfonzxo4HPYXz3G3CAg0BzcG9Vtv2H/5LUQ== X-Received: by 2002:a05:6512:b08:b0:530:dab8:7dd9 with SMTP id 2adb3069b0e04-530ee96f49amr6197701e87.12.1723410049041; Sun, 11 Aug 2024 14:00:49 -0700 (PDT) Received: from localhost (ip-149-172-150-237.um42.pools.vodafone-ip.de. [149.172.150.237]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5bd187f5d3asm1601249a12.12.2024.08.11.14.00.48 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 11 Aug 2024 14:00:48 -0700 (PDT) Mime-Version: 1.0 Date: Sun, 11 Aug 2024 23:00:48 +0200 Message-Id: From: "Robin Dapp" Subject: [PATCH 3/8] tree-ifcvt: Enforce zero else value after maskload. Cc: , "" , "Richard Biener" To: "gcc-patches" X-Spam-Status: No, score=-9.0 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, FREEMAIL_FROM, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~incoming=patchwork.ozlabs.org@gcc.gnu.org When predicating a load we implicitly assume that the else value is zero. In order to formalize this this patch queries the target for its supported else operand and uses that for the maskload call. Subsequently, if the else operand is nonzero, a cond_expr enforcing a zero else value is emitted. gcc/ChangeLog: * tree-if-conv.cc (predicate_load_or_store): Enforce zero else value. (predicate_statements): Use sequence instead of statement. --- gcc/tree-if-conv.cc | 78 +++++++++++++++++++++++++++++++++++---------- 1 file changed, 62 insertions(+), 16 deletions(-) diff --git a/gcc/tree-if-conv.cc b/gcc/tree-if-conv.cc index 57992b6deca..54cb9ef0ef1 100644 --- a/gcc/tree-if-conv.cc +++ b/gcc/tree-if-conv.cc @@ -2452,10 +2452,12 @@ mask_exists (int size, const vec &vec) write and it needs to be predicated by MASK. Return a statement that does so. */ -static gimple * -predicate_load_or_store (gimple_stmt_iterator *gsi, gassign *stmt, tree mask) +static gimple_seq +predicate_load_or_store (gimple_stmt_iterator *gsi, gassign *stmt, tree mask, + hash_set *ssa_names) { - gcall *new_stmt; + gimple_seq stmts = NULL; + gcall *call_stmt; tree lhs = gimple_assign_lhs (stmt); tree rhs = gimple_assign_rhs1 (stmt); @@ -2471,21 +2473,59 @@ predicate_load_or_store (gimple_stmt_iterator *gsi, gassign *stmt, tree mask) ref); if (TREE_CODE (lhs) == SSA_NAME) { - new_stmt - = gimple_build_call_internal (IFN_MASK_LOAD, 3, addr, - ptr, mask); - gimple_call_set_lhs (new_stmt, lhs); - gimple_set_vuse (new_stmt, gimple_vuse (stmt)); + /* Get the preferred vector mode and its corresponding mask for the + masked load. We need this to query the target's supported else + operands. */ + machine_mode mode = TYPE_MODE (TREE_TYPE (addr)); + scalar_mode smode = as_a (mode); + + machine_mode vmode = targetm.vectorize.preferred_simd_mode (smode); + machine_mode mask_mode + = targetm.vectorize.get_mask_mode (vmode).require (); + + int elsval; + internal_fn ifn; + target_supports_mask_load_store_p (vmode, mask_mode, true, &ifn, &elsval); + tree els = vect_get_mask_load_else (elsval, TREE_TYPE (lhs)); + + call_stmt + = gimple_build_call_internal (IFN_MASK_LOAD, 4, addr, + ptr, mask, els); + + /* Build the load call and, if the else value is nonzero, + a COND_EXPR that enforces it. */ + tree loadlhs; + if (elsval == MASK_LOAD_ELSE_ZERO) + gimple_call_set_lhs (call_stmt, gimple_get_lhs (stmt)); + else + { + loadlhs = make_temp_ssa_name (TREE_TYPE (lhs), NULL, "_ifc_"); + ssa_names->add (loadlhs); + gimple_call_set_lhs (call_stmt, loadlhs); + } + gimple_set_vuse (call_stmt, gimple_vuse (stmt)); + gimple_seq_add_stmt (&stmts, call_stmt); + + if (elsval != MASK_LOAD_ELSE_ZERO) + { + tree cond_rhs + = fold_build_cond_expr (TREE_TYPE (loadlhs), mask, loadlhs, + build_zero_cst (TREE_TYPE (loadlhs))); + gassign *cond_stmt + = gimple_build_assign (gimple_get_lhs (stmt), cond_rhs); + gimple_seq_add_stmt (&stmts, cond_stmt); + } } else { - new_stmt + call_stmt = gimple_build_call_internal (IFN_MASK_STORE, 4, addr, ptr, mask, rhs); - gimple_move_vops (new_stmt, stmt); + gimple_move_vops (call_stmt, stmt); + gimple_seq_add_stmt (&stmts, call_stmt); } - gimple_call_set_nothrow (new_stmt, true); - return new_stmt; + gimple_call_set_nothrow (call_stmt, true); + return stmts; } /* STMT uses OP_LHS. Check whether it is equivalent to: @@ -2789,11 +2829,17 @@ predicate_statements (loop_p loop) vect_masks.safe_push (mask); } if (gimple_assign_single_p (stmt)) - new_stmt = predicate_load_or_store (&gsi, stmt, mask); - else - new_stmt = predicate_rhs_code (stmt, mask, cond, &ssa_names); + { + gimple_seq call_seq + = predicate_load_or_store (&gsi, stmt, mask, &ssa_names); - gsi_replace (&gsi, new_stmt, true); + gsi_replace_with_seq (&gsi, call_seq, true); + } + else + { + new_stmt = predicate_rhs_code (stmt, mask, cond, &ssa_names); + gsi_replace (&gsi, new_stmt, true); + } } else if (((lhs = gimple_assign_lhs (stmt)), true) && (INTEGRAL_TYPE_P (TREE_TYPE (lhs))