From patchwork Fri Oct 4 09:58:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jakub Jelinek X-Patchwork-Id: 1992671 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=gx3Vr4ZK; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=8.43.85.97; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces~incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4XKkXd3w1xz1xt7 for ; Fri, 4 Oct 2024 19:59:13 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 52B76385E45A for ; Fri, 4 Oct 2024 09:59:11 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by sourceware.org (Postfix) with ESMTP id 15277385E45A for ; Fri, 4 Oct 2024 09:58:51 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 15277385E45A Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 15277385E45A Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1728035934; cv=none; b=p04LCgjybwnoV5GChFuA95NThEA/zrlVNCGbhuUTdwP+9t4EaKajBc/ItecmwjNOJ6AmtWqlZGSgAcHF2EZn+rFyoDFNjFn23SellDK5C7RsOOiNPoAPWm8RHVbIIaxcDQE+gH0S5cEzxWgMbrQ52rWFhbQwAGfsHNUmFRY5UHE= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1728035934; c=relaxed/simple; bh=rhR+YjhzfV++yu8rJOJg1ZIg4m0Di3SNNxdLqpJT4nU=; h=DKIM-Signature:Date:From:To:Subject:Message-ID:MIME-Version; b=X+cGqiZ0b7wbc/rxlYqneaqKySwmBIpoYZlHeOadaq/8zmCtxbLaKGDyoSxJoy52gZ6zq3lV24Rh5RH4Ys7VPw8PgkTZDp9MPBjJqPhth3bUQ0+v8yTIFY+8tvJfgxTKIRhQ95qdM6yY2UIOkPVAlHkSdlhFroj1zN/oLEz0rVE= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1728035930; h=from:from:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type; bh=Ei4BVOHvH0ykkTuRizf4ODy4QCdB2FY+croijRPkYlQ=; b=gx3Vr4ZKIKAIhXdY4dbKH+C7n2fjZVCx1b+80BMvFzaB/IOp/AMXBc+GU/ZugaR6QVMVQ1 wp0rYOO6B6Q47U1zsrSb3Bxsf6tXEThni3F7YitbMX4MUn9SeYp8RSbfK46psxKJAh3CvK h5aUZShPiZLVnpY516YaD1fRutmI6uE= Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-193-l7wvrWEUNtyywdieZ3fAIw-1; Fri, 04 Oct 2024 05:58:49 -0400 X-MC-Unique: l7wvrWEUNtyywdieZ3fAIw-1 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 2775D19560AE; Fri, 4 Oct 2024 09:58:48 +0000 (UTC) Received: from tucnak.zalov.cz (unknown [10.45.224.61]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 327B73000198; Fri, 4 Oct 2024 09:58:46 +0000 (UTC) Received: from tucnak.zalov.cz (localhost [127.0.0.1]) by tucnak.zalov.cz (8.17.1/8.17.1) with ESMTPS id 4949wivG2241987 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NOT); Fri, 4 Oct 2024 11:58:44 +0200 Received: (from jakub@localhost) by tucnak.zalov.cz (8.17.1/8.17.1/Submit) id 4949wium2241986; Fri, 4 Oct 2024 11:58:44 +0200 Date: Fri, 4 Oct 2024 11:58:43 +0200 From: Jakub Jelinek To: Richard Biener , Uros Bizjak Cc: gcc-patches@gcc.gnu.org Subject: [PATCH] ssa-math-opts, i386: Improve spaceship expansion [PR116896] Message-ID: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Disposition: inline X-Spam-Status: No, score=-13.9 required=5.0 tests=BAYES_00, DKIM_INVALID, DKIM_SIGNED, KAM_DMARC_NONE, KAM_DMARC_STATUS, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SCC_5_SHORT_WORD_LINES, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: Jakub Jelinek Errors-To: gcc-patches-bounces~incoming=patchwork.ozlabs.org@gcc.gnu.org Hi! The PR notes that we don't emit optimal code for C++ spaceship operator if the result is returned as an integer rather than the result just being compared against different values and different code executed based on that. So e.g. for template auto foo (T x, T y) { return x <=> y; } for both floating point types, signed integer types and unsigned integer types. auto in that case is std::strong_ordering or std::partial_ordering, which are fancy C++ abstractions around struct with signed char member which is -1, 0, 1 for the strong ordering and -1, 0, 1, 2 for the partial ordering (but for -ffast-math 2 is never the case). I'm afraid functions like that are fairly common and unless they are inlined, we really need to map the comparison to those -1, 0, 1 or -1, 0, 1, 2 values. Now, for floating point spaceship I've in the past already added an optimization (with tree-ssa-math-opts.cc discovery and named optab, the optab only defined on x86 though right now), which ensures there is just a single comparison instruction and then just tests based on flags. Now, if we have code like: auto a = x <=> y; if (a == std::partial_ordering::less) bar (); else if (a == std::partial_ordering::greater) baz (); else if (a == std::partial_ordering::equivalent) qux (); else if (a == std::partial_ordering::unordered) corge (); etc., that results in decent code generation, the spaceship named pattern on x86 optimizes for the jumps, so emits comparisons on the flags, followed by setting the result to -1, 0, 1, 2 and subsequent jump pass optimizes that well. But if the result needs to be stored into an integer and just returned that way or there are no immediate jumps based on it (or turned into some non-standard integer values like -42, 0, 36, 75 etc.), then CE doesn't do a good job for that, we end up with say comiss %xmm1, %xmm0 jp .L4 seta %al movl $0, %edx leal -1(%rax,%rax), %eax cmove %edx, %eax ret .L4: movl $2, %eax ret The jp is good, that is the unlikely case and can't be easily handled in straight line code due to the layout of the flags, but the rest uses cmov which often isn't a win and a weird math. With the patch below we can get instead xorl %eax, %eax comiss %xmm1, %xmm0 jp .L2 seta %al sbbl $0, %eax ret .L2: movl $2, %eax ret The patch changes the discovery in the generic code, by detecting if the future .SPACESHIP result is just used in a PHI with -1, 0, 1 or -1, 0, 1, 2 values (the latter for HONOR_NANS) and passes that as a flag in a new argument to .SPACESHIP ifn, so that the named pattern is told whether it should optimize for branches or for loading the result into a -1, 0, 1 (, 2) integer. Additionally, it doesn't detect just floating point <=> anymore, but also integer and unsigned integer, but in those cases only if an integer -1, 0, 1 is wanted (otherwise == and > or similar comparisons result in good code). The backend then can for those integer or unsigned integer <=>s return effectively (x > y) - (x < y) in a way that is efficient on the target (so for x86 with ensuring zero initialization first when needed before setcc; one for floating point and unsigned, where there is just one setcc and the second one optimized into sbb instruction, two for the signed int case). So e.g. for signed int we now emit xorl %edx, %edx xorl %eax, %eax cmpl %esi, %edi setl %dl setg %al subl %edx, %eax ret and for unsigned xorl %eax, %eax cmpl %esi, %edi seta %al sbbb $0, %al ret Bootstrapped/regtested on x86_64-linux and i686-linux, ok for trunk? Note, I wonder if other targets wouldn't benefit from defining the named optab too... 2024-10-04 Jakub Jelinek PR middle-end/116896 * optabs.def (spaceship_optab): Use spaceship$a4 rather than spaceship$a3. * internal-fn.cc (expand_SPACESHIP): Expect 3 call arguments rather than 2, expand the last one, expect 4 operands of spaceship_optab. * tree-ssa-math-opts.cc: Include cfghooks.h. (optimize_spaceship): Check if a single PHI is initialized to -1, 0, 1, 2 or -1, 0, 1 values, in that case pass 1 as last (new) argument to .SPACESHIP and optimize away the comparisons, otherwise pass 0. Also check for integer comparisons rather than floating point, in that case do it only if there is a single PHI with -1, 0, 1 values and pass 1 to last argument of .SPACESHIP if the <=> is signed, 2 if unsigned. * config/i386/i386-protos.h (ix86_expand_fp_spaceship): Add another rtx argument. (ix86_expand_int_spaceship): Declare. * config/i386/i386-expand.cc (ix86_expand_fp_spaceship): Add arg3 argument, if it is const0_rtx, expand like before, otherwise emit optimized sequence for setting the result into a GPR. (ix86_expand_int_spaceship): New function. * config/i386/i386.md (UNSPEC_SETCC_SI_SLP): New UNSPEC code. (setcc_si_slp): New define_expand. (*setcc_si_slp): New define_insn_and_split. (setcc + setcc + movzbl): New define_peephole2. (spaceship3): Renamed to ... (spaceship4): ... this. Add an extra operand, pass it to ix86_expand_fp_spaceship. (spaceshipxf3): Renamed to ... (spaceshipxf4): ... this. Add an extra operand, pass it to ix86_expand_fp_spaceship. (spaceship4): New define_expand for SWI modes. * doc/md.texi (spaceship@var{m}3): Renamed to ... (spaceship@var{m}4): ... this. Document the meaning of last operand. * g++.target/i386/pr116896-1.C: New test. * g++.target/i386/pr116896-2.C: New test. Jakub --- gcc/optabs.def.jj 2024-10-01 09:38:58.143960049 +0200 +++ gcc/optabs.def 2024-10-02 13:52:53.352550358 +0200 @@ -308,7 +308,7 @@ OPTAB_D (negv3_optab, "negv$I$a3") OPTAB_D (uaddc5_optab, "uaddc$I$a5") OPTAB_D (usubc5_optab, "usubc$I$a5") OPTAB_D (addptr3_optab, "addptr$a3") -OPTAB_D (spaceship_optab, "spaceship$a3") +OPTAB_D (spaceship_optab, "spaceship$a4") OPTAB_D (smul_highpart_optab, "smul$a3_highpart") OPTAB_D (umul_highpart_optab, "umul$a3_highpart") --- gcc/internal-fn.cc.jj 2024-10-01 09:38:41.482192804 +0200 +++ gcc/internal-fn.cc 2024-10-02 13:52:53.354550330 +0200 @@ -5107,6 +5107,7 @@ expand_SPACESHIP (internal_fn, gcall *st tree lhs = gimple_call_lhs (stmt); tree rhs1 = gimple_call_arg (stmt, 0); tree rhs2 = gimple_call_arg (stmt, 1); + tree rhs3 = gimple_call_arg (stmt, 2); tree type = TREE_TYPE (rhs1); do_pending_stack_adjust (); @@ -5114,13 +5115,15 @@ expand_SPACESHIP (internal_fn, gcall *st rtx target = expand_expr (lhs, NULL_RTX, VOIDmode, EXPAND_WRITE); rtx op1 = expand_normal (rhs1); rtx op2 = expand_normal (rhs2); + rtx op3 = expand_normal (rhs3); - class expand_operand ops[3]; + class expand_operand ops[4]; create_call_lhs_operand (&ops[0], target, TYPE_MODE (TREE_TYPE (lhs))); create_input_operand (&ops[1], op1, TYPE_MODE (type)); create_input_operand (&ops[2], op2, TYPE_MODE (type)); + create_input_operand (&ops[3], op3, TYPE_MODE (TREE_TYPE (rhs3))); insn_code icode = optab_handler (spaceship_optab, TYPE_MODE (type)); - expand_insn (icode, 3, ops); + expand_insn (icode, 4, ops); assign_call_lhs (lhs, target, &ops[0]); } --- gcc/tree-ssa-math-opts.cc.jj 2024-10-01 09:38:58.331957422 +0200 +++ gcc/tree-ssa-math-opts.cc 2024-10-03 15:00:15.893473504 +0200 @@ -117,6 +117,7 @@ along with GCC; see the file COPYING3. #include "domwalk.h" #include "tree-ssa-math-opts.h" #include "dbgcnt.h" +#include "cfghooks.h" /* This structure represents one basic block that either computes a division, or is a common dominator for basic block that compute a @@ -5869,7 +5870,7 @@ convert_mult_to_highpart (gassign *stmt, [local count: 1073741824]: and turn it into: [local count: 1073741824]: - _1 = .SPACESHIP (a_2(D), b_3(D)); + _1 = .SPACESHIP (a_2(D), b_3(D), 0); if (_1 == 0) goto ; [34.00%] else @@ -5891,7 +5892,13 @@ convert_mult_to_highpart (gassign *stmt, [local count: 1073741824]: so that the backend can emit optimal comparison and - conditional jump sequence. */ + conditional jump sequence. If the + [local count: 1073741824]: + above has a single PHI like: + # _27 = PHI<0(2), -1(3), 2(4), 1(5)> + then replace it with effectively + _1 = .SPACESHIP (a_2(D), b_3(D), 1); + _27 = _1; */ static void optimize_spaceship (gcond *stmt) @@ -5901,7 +5908,8 @@ optimize_spaceship (gcond *stmt) return; tree arg1 = gimple_cond_lhs (stmt); tree arg2 = gimple_cond_rhs (stmt); - if (!SCALAR_FLOAT_TYPE_P (TREE_TYPE (arg1)) + if ((!SCALAR_FLOAT_TYPE_P (TREE_TYPE (arg1)) + && !INTEGRAL_TYPE_P (TREE_TYPE (arg1))) || optab_handler (spaceship_optab, TYPE_MODE (TREE_TYPE (arg1))) == CODE_FOR_nothing || operand_equal_p (arg1, arg2, 0)) @@ -6013,12 +6021,105 @@ optimize_spaceship (gcond *stmt) } } - gcall *gc = gimple_build_call_internal (IFN_SPACESHIP, 2, arg1, arg2); + /* Check if there is a single bb into which all failed conditions + jump to (perhaps through an empty block) and if it results in + a single integral PHI which just sets it to -1, 0, 1, 2 + (or -1, 0, 1 when NaNs can't happen). In that case use 1 rather + than 0 as last .SPACESHIP argument to tell backends it might + consider different code generation and just cast the result + of .SPACESHIP to the PHI result. */ + tree arg3 = integer_zero_node; + edge e = EDGE_SUCC (bb0, 0); + if (e->dest == bb1) + e = EDGE_SUCC (bb0, 1); + basic_block bbp = e->dest; + gphi *phi = NULL; + for (gphi_iterator psi = gsi_start_phis (bbp); + !gsi_end_p (psi); gsi_next (&psi)) + { + gphi *gp = psi.phi (); + tree res = gimple_phi_result (gp); + + if (phi != NULL + || virtual_operand_p (res) + || !INTEGRAL_TYPE_P (TREE_TYPE (res)) + || TYPE_PRECISION (TREE_TYPE (res)) < 2) + { + phi = NULL; + break; + } + phi = gp; + } + if (phi + && integer_zerop (gimple_phi_arg_def_from_edge (phi, e)) + && EDGE_COUNT (bbp->preds) == (HONOR_NANS (TREE_TYPE (arg1)) ? 4 : 3)) + { + for (unsigned i = 0; phi && i < EDGE_COUNT (bbp->preds) - 1; ++i) + { + edge e3 = i == 0 ? e1 : i == 1 ? em1 : e2; + if (e3->dest != bbp) + { + if (!empty_block_p (e3->dest) + || !single_succ_p (e3->dest) + || single_succ (e3->dest) != bbp) + { + phi = NULL; + break; + } + e3 = single_succ_edge (e3->dest); + } + tree a = gimple_phi_arg_def_from_edge (phi, e3); + if (TREE_CODE (a) != INTEGER_CST + || (i == 0 && !integer_onep (a)) + || (i == 1 && !integer_all_onesp (a)) + || (i == 2 && wi::to_widest (a) != 2)) + { + phi = NULL; + break; + } + } + if (phi) + arg3 = build_int_cst (integer_type_node, + TYPE_UNSIGNED (TREE_TYPE (arg1)) ? 2 : 1); + } + + /* For integral <=> comparisons only use .SPACESHIP if it is turned + into an integer (-1, 0, 1). */ + if (!SCALAR_FLOAT_TYPE_P (TREE_TYPE (arg1)) && arg3 == integer_zero_node) + return; + + gcall *gc = gimple_build_call_internal (IFN_SPACESHIP, 3, arg1, arg2, arg3); tree lhs = make_ssa_name (integer_type_node); gimple_call_set_lhs (gc, lhs); gimple_stmt_iterator gsi = gsi_for_stmt (stmt); gsi_insert_before (&gsi, gc, GSI_SAME_STMT); + wide_int wm1 = wi::minus_one (TYPE_PRECISION (integer_type_node)); + wide_int w2 = (HONOR_NANS (TREE_TYPE (arg1)) + ? wi::two (TYPE_PRECISION (integer_type_node)) + : wi::one (TYPE_PRECISION (integer_type_node))); + int_range<1> vr (TREE_TYPE (lhs), wm1, w2); + set_range_info (lhs, vr); + + if (arg3 != integer_zero_node) + { + tree type = TREE_TYPE (gimple_phi_result (phi)); + if (!useless_type_conversion_p (type, integer_type_node)) + { + tree tem = make_ssa_name (type); + gimple *gcv = gimple_build_assign (tem, NOP_EXPR, lhs); + gsi_insert_before (&gsi, gcv, GSI_SAME_STMT); + lhs = tem; + } + SET_PHI_ARG_DEF_ON_EDGE (phi, e, lhs); + gimple_cond_set_lhs (stmt, boolean_false_node); + gimple_cond_set_rhs (stmt, boolean_false_node); + gimple_cond_set_code (stmt, (e->flags & EDGE_TRUE_VALUE) + ? EQ_EXPR : NE_EXPR); + update_stmt (stmt); + return; + } + gimple_cond_set_lhs (stmt, lhs); gimple_cond_set_rhs (stmt, integer_zero_node); update_stmt (stmt); @@ -6055,11 +6156,6 @@ optimize_spaceship (gcond *stmt) (e2->flags & EDGE_TRUE_VALUE) ? NE_EXPR : EQ_EXPR); update_stmt (cond); } - - wide_int wm1 = wi::minus_one (TYPE_PRECISION (integer_type_node)); - wide_int w2 = wi::two (TYPE_PRECISION (integer_type_node)); - int_range<1> vr (TREE_TYPE (lhs), wm1, w2); - set_range_info (lhs, vr); } --- gcc/config/i386/i386-protos.h.jj 2024-10-01 09:38:41.237196225 +0200 +++ gcc/config/i386/i386-protos.h 2024-10-02 18:45:12.807284696 +0200 @@ -164,7 +164,8 @@ extern bool ix86_expand_fp_vec_cmp (rtx[ extern void ix86_expand_sse_movcc (rtx, rtx, rtx, rtx); extern void ix86_expand_sse_extend (rtx, rtx, bool); extern void ix86_expand_sse_unpack (rtx, rtx, bool, bool); -extern void ix86_expand_fp_spaceship (rtx, rtx, rtx); +extern void ix86_expand_fp_spaceship (rtx, rtx, rtx, rtx); +extern void ix86_expand_int_spaceship (rtx, rtx, rtx, rtx); extern bool ix86_expand_int_addcc (rtx[]); extern void ix86_expand_carry (rtx arg); extern rtx_insn *ix86_expand_call (rtx, rtx, rtx, rtx, rtx, bool); --- gcc/config/i386/i386-expand.cc.jj 2024-10-01 09:38:41.181197008 +0200 +++ gcc/config/i386/i386-expand.cc 2024-10-04 11:05:09.647022644 +0200 @@ -3144,12 +3144,15 @@ ix86_expand_setcc (rtx dest, enum rtx_co dest = op0 == op1 ? 0 : op0 < op1 ? -1 : op0 > op1 ? 1 : 2. */ void -ix86_expand_fp_spaceship (rtx dest, rtx op0, rtx op1) +ix86_expand_fp_spaceship (rtx dest, rtx op0, rtx op1, rtx op2) { gcc_checking_assert (ix86_fp_comparison_strategy (GT) != IX86_FPCMP_ARITH); + rtx zero = NULL_RTX; + if (op2 != const0_rtx && TARGET_IEEE_FP && GET_MODE (dest) == SImode) + zero = force_reg (SImode, const0_rtx); rtx gt = ix86_expand_fp_compare (GT, op0, op1); - rtx l0 = gen_label_rtx (); - rtx l1 = gen_label_rtx (); + rtx l0 = op2 == const0_rtx ? gen_label_rtx () : NULL_RTX; + rtx l1 = op2 == const0_rtx ? gen_label_rtx () : NULL_RTX; rtx l2 = TARGET_IEEE_FP ? gen_label_rtx () : NULL_RTX; rtx lend = gen_label_rtx (); rtx tmp; @@ -3163,23 +3166,68 @@ ix86_expand_fp_spaceship (rtx dest, rtx jmp = emit_jump_insn (gen_rtx_SET (pc_rtx, tmp)); add_reg_br_prob_note (jmp, profile_probability:: very_unlikely ()); } - rtx eq = gen_rtx_fmt_ee (UNEQ, VOIDmode, - gen_rtx_REG (CCFPmode, FLAGS_REG), const0_rtx); - tmp = gen_rtx_IF_THEN_ELSE (VOIDmode, eq, - gen_rtx_LABEL_REF (VOIDmode, l0), pc_rtx); - jmp = emit_jump_insn (gen_rtx_SET (pc_rtx, tmp)); - add_reg_br_prob_note (jmp, profile_probability::unlikely ()); - tmp = gen_rtx_IF_THEN_ELSE (VOIDmode, gt, - gen_rtx_LABEL_REF (VOIDmode, l1), pc_rtx); - jmp = emit_jump_insn (gen_rtx_SET (pc_rtx, tmp)); - add_reg_br_prob_note (jmp, profile_probability::even ()); - emit_move_insn (dest, constm1_rtx); - emit_jump (lend); - emit_label (l0); - emit_move_insn (dest, const0_rtx); - emit_jump (lend); - emit_label (l1); - emit_move_insn (dest, const1_rtx); + if (op2 == const0_rtx) + { + rtx eq = gen_rtx_fmt_ee (UNEQ, VOIDmode, + gen_rtx_REG (CCFPmode, FLAGS_REG), const0_rtx); + tmp = gen_rtx_IF_THEN_ELSE (VOIDmode, eq, + gen_rtx_LABEL_REF (VOIDmode, l0), pc_rtx); + jmp = emit_jump_insn (gen_rtx_SET (pc_rtx, tmp)); + add_reg_br_prob_note (jmp, profile_probability::unlikely ()); + tmp = gen_rtx_IF_THEN_ELSE (VOIDmode, gt, + gen_rtx_LABEL_REF (VOIDmode, l1), pc_rtx); + jmp = emit_jump_insn (gen_rtx_SET (pc_rtx, tmp)); + add_reg_br_prob_note (jmp, profile_probability::even ()); + emit_move_insn (dest, constm1_rtx); + emit_jump (lend); + emit_label (l0); + emit_move_insn (dest, const0_rtx); + emit_jump (lend); + emit_label (l1); + emit_move_insn (dest, const1_rtx); + } + else + { + rtx lt_tmp = gen_reg_rtx (QImode); + ix86_expand_setcc (lt_tmp, UNLT, gen_rtx_REG (CCFPmode, FLAGS_REG), + const0_rtx); + if (GET_MODE (dest) != QImode) + { + tmp = gen_reg_rtx (GET_MODE (dest)); + emit_insn (gen_rtx_SET (tmp, gen_rtx_ZERO_EXTEND (GET_MODE (dest), + lt_tmp))); + lt_tmp = tmp; + } + rtx gt_tmp; + if (zero) + { + /* If TARGET_IEEE_FP and dest has SImode, emit SImode clear + before the floating point comparison and use setcc_si_slp + pattern to hide it from the combiner, so that it doesn't + undo it. */ + tmp = ix86_expand_compare (GT, XEXP (gt, 0), const0_rtx); + PUT_MODE (tmp, QImode); + emit_insn (gen_setcc_si_slp (zero, tmp, zero)); + gt_tmp = zero; + } + else + { + gt_tmp = gen_reg_rtx (QImode); + ix86_expand_setcc (gt_tmp, GT, XEXP (gt, 0), const0_rtx); + if (GET_MODE (dest) != QImode) + { + tmp = gen_reg_rtx (GET_MODE (dest)); + emit_insn (gen_rtx_SET (tmp, + gen_rtx_ZERO_EXTEND (GET_MODE (dest), + gt_tmp))); + gt_tmp = tmp; + } + } + tmp = expand_simple_binop (GET_MODE (dest), MINUS, gt_tmp, lt_tmp, dest, + 0, OPTAB_DIRECT); + if (!rtx_equal_p (tmp, dest)) + emit_move_insn (dest, tmp); + } emit_jump (lend); if (l2) { @@ -3189,6 +3237,46 @@ ix86_expand_fp_spaceship (rtx dest, rtx emit_label (lend); } +/* Expand integral op0 <=> op1, i.e. + dest = op0 == op1 ? 0 : op0 < op1 ? -1 : 1. */ + +void +ix86_expand_int_spaceship (rtx dest, rtx op0, rtx op1, rtx op2) +{ + gcc_assert (INTVAL (op2)); + /* Not using ix86_expand_int_compare here, so that it doesn't swap + operands nor optimize CC mode - we need a mode usable for both + LT and GT resp. LTU and GTU comparisons with the same unswapped + operands. */ + rtx flags = gen_rtx_REG (INTVAL (op2) == 1 ? CCGCmode : CCmode, FLAGS_REG); + rtx tmp = gen_rtx_COMPARE (GET_MODE (flags), op0, op1); + emit_insn (gen_rtx_SET (flags, tmp)); + rtx lt_tmp = gen_reg_rtx (QImode); + ix86_expand_setcc (lt_tmp, INTVAL (op2) == 1 ? LT : LTU, flags, + const0_rtx); + if (GET_MODE (dest) != QImode) + { + tmp = gen_reg_rtx (GET_MODE (dest)); + emit_insn (gen_rtx_SET (tmp, gen_rtx_ZERO_EXTEND (GET_MODE (dest), + lt_tmp))); + lt_tmp = tmp; + } + rtx gt_tmp = gen_reg_rtx (QImode); + ix86_expand_setcc (gt_tmp, INTVAL (op2) == 1 ? GT : GTU, flags, + const0_rtx); + if (GET_MODE (dest) != QImode) + { + tmp = gen_reg_rtx (GET_MODE (dest)); + emit_insn (gen_rtx_SET (tmp, gen_rtx_ZERO_EXTEND (GET_MODE (dest), + gt_tmp))); + gt_tmp = tmp; + } + tmp = expand_simple_binop (GET_MODE (dest), MINUS, gt_tmp, lt_tmp, dest, + 0, OPTAB_DIRECT); + if (!rtx_equal_p (tmp, dest)) + emit_move_insn (dest, tmp); +} + /* Expand comparison setting or clearing carry flag. Return true when successful and set pop for the operation. */ static bool --- gcc/config/i386/i386.md.jj 2024-10-01 09:38:41.296195402 +0200 +++ gcc/config/i386/i386.md 2024-10-03 14:32:24.661446577 +0200 @@ -118,6 +118,7 @@ (define_c_enum "unspec" [ UNSPEC_PUSHFL UNSPEC_POPFL UNSPEC_OPTCOMX + UNSPEC_SETCC_SI_SLP ;; For SSE/MMX support: UNSPEC_FIX_NOTRUNC @@ -19281,6 +19282,27 @@ (define_insn "*setcc_qi_slp" [(set_attr "type" "setcc") (set_attr "mode" "QI")]) +(define_expand "setcc_si_slp" + [(set (match_operand:SI 0 "register_operand") + (unspec:SI + [(match_operand:QI 1) + (match_operand:SI 2 "register_operand")] UNSPEC_SETCC_SI_SLP))]) + +(define_insn_and_split "*setcc_si_slp" + [(set (match_operand:SI 0 "register_operand" "=q") + (unspec:SI + [(match_operator:QI 1 "ix86_comparison_operator" + [(reg FLAGS_REG) (const_int 0)]) + (match_operand:SI 2 "register_operand" "0")] UNSPEC_SETCC_SI_SLP))] + "ix86_pre_reload_split ()" + "#" + "&& 1" + [(set (match_dup 0) (match_dup 2)) + (set (strict_low_part (match_dup 3)) (match_dup 1))] +{ + operands[3] = gen_lowpart (QImode, operands[0]); +}) + ;; In general it is not safe to assume too much about CCmode registers, ;; so simplify-rtx stops when it sees a second one. Under certain ;; conditions this is safe on x86, so help combine not create @@ -19776,6 +19798,32 @@ (define_peephole2 operands[8] = gen_lowpart (QImode, operands[4]); ix86_expand_clear (operands[4]); }) + +(define_peephole2 + [(set (match_operand 4 "flags_reg_operand") (match_operand 0)) + (set (strict_low_part (match_operand:QI 5 "register_operand")) + (match_operator:QI 6 "ix86_comparison_operator" + [(reg FLAGS_REG) (const_int 0)])) + (set (match_operand:QI 1 "register_operand") + (match_operator:QI 2 "ix86_comparison_operator" + [(reg FLAGS_REG) (const_int 0)])) + (set (match_operand 3 "any_QIreg_operand") + (zero_extend (match_dup 1)))] + "(peep2_reg_dead_p (4, operands[1]) + || operands_match_p (operands[1], operands[3])) + && ! reg_overlap_mentioned_p (operands[3], operands[0]) + && ! reg_overlap_mentioned_p (operands[3], operands[5]) + && ! reg_overlap_mentioned_p (operands[1], operands[5]) + && peep2_regno_dead_p (0, FLAGS_REG)" + [(set (match_dup 4) (match_dup 0)) + (set (strict_low_part (match_dup 5)) + (match_dup 6)) + (set (strict_low_part (match_dup 7)) + (match_dup 2))] +{ + operands[7] = gen_lowpart (QImode, operands[3]); + ix86_expand_clear (operands[3]); +}) ;; Call instructions. @@ -29494,24 +29542,40 @@ (define_insn "hreset" (set_attr "length" "4")]) ;; Spaceship optimization -(define_expand "spaceship3" +(define_expand "spaceship4" [(match_operand:SI 0 "register_operand") (match_operand:MODEF 1 "cmp_fp_expander_operand") - (match_operand:MODEF 2 "cmp_fp_expander_operand")] + (match_operand:MODEF 2 "cmp_fp_expander_operand") + (match_operand:SI 3 "const_int_operand")] "(TARGET_80387 || (SSE_FLOAT_MODE_P (mode) && TARGET_SSE_MATH)) && (TARGET_CMOVE || (TARGET_SAHF && TARGET_USE_SAHF))" { - ix86_expand_fp_spaceship (operands[0], operands[1], operands[2]); + ix86_expand_fp_spaceship (operands[0], operands[1], operands[2], + operands[3]); DONE; }) -(define_expand "spaceshipxf3" +(define_expand "spaceshipxf4" [(match_operand:SI 0 "register_operand") (match_operand:XF 1 "nonmemory_operand") - (match_operand:XF 2 "nonmemory_operand")] + (match_operand:XF 2 "nonmemory_operand") + (match_operand:SI 3 "const_int_operand")] "TARGET_80387 && (TARGET_CMOVE || (TARGET_SAHF && TARGET_USE_SAHF))" { - ix86_expand_fp_spaceship (operands[0], operands[1], operands[2]); + ix86_expand_fp_spaceship (operands[0], operands[1], operands[2], + operands[3]); + DONE; +}) + +(define_expand "spaceship4" + [(match_operand:SI 0 "register_operand") + (match_operand:SWI 1 "nonimmediate_operand") + (match_operand:SWI 2 "") + (match_operand:SI 3 "const_int_operand")] + "" +{ + ix86_expand_int_spaceship (operands[0], operands[1], operands[2], + operands[3]); DONE; }) --- gcc/doc/md.texi.jj 2024-10-01 09:38:58.035961557 +0200 +++ gcc/doc/md.texi 2024-10-02 20:16:22.502329039 +0200 @@ -8568,11 +8568,15 @@ inclusive and operand 1 exclusive. If this pattern is not defined, a call to the library function @code{__clear_cache} is used. -@cindex @code{spaceship@var{m}3} instruction pattern -@item @samp{spaceship@var{m}3} +@cindex @code{spaceship@var{m}4} instruction pattern +@item @samp{spaceship@var{m}4} Initialize output operand 0 with mode of integer type to -1, 0, 1 or 2 if operand 1 with mode @var{m} compares less than operand 2, equal to operand 2, greater than operand 2 or is unordered with operand 2. +Operand 3 should be @code{const0_rtx} if the result is used in comparisons, +@code{const1_rtx} if the result is used as integer value and the comparison +is signed, @code{const2_rtx} if the result is used as integer value and +the comparison is unsigned. @var{m} should be a scalar floating point mode. This pattern is not allowed to @code{FAIL}. --- gcc/testsuite/g++.target/i386/pr116896-1.C.jj 2024-10-03 14:40:27.071813336 +0200 +++ gcc/testsuite/g++.target/i386/pr116896-1.C 2024-10-03 15:52:06.243819660 +0200 @@ -0,0 +1,35 @@ +// PR middle-end/116896 +// { dg-do compile { target c++20 } } +// { dg-options "-O2 -masm=att -fno-stack-protector" } +// { dg-final { scan-assembler-times "\tjp\t" 1 } } +// { dg-final { scan-assembler-not "\tj\[^mp\]\[a-z\]*\t" } } +// { dg-final { scan-assembler-times "\tsbb\[bl\]\t\\\$0, " 3 } } +// { dg-final { scan-assembler-times "\tseta\t" 3 } } +// { dg-final { scan-assembler-times "\tsetg\t" 1 } } +// { dg-final { scan-assembler-times "\tsetl\t" 1 } } + +#include + +[[gnu::noipa]] auto +foo (float x, float y) +{ + return x <=> y; +} + +[[gnu::noipa, gnu::optimize ("fast-math")]] auto +bar (float x, float y) +{ + return x <=> y; +} + +[[gnu::noipa]] auto +baz (int x, int y) +{ + return x <=> y; +} + +[[gnu::noipa]] auto +qux (unsigned x, unsigned y) +{ + return x <=> y; +} --- gcc/testsuite/g++.target/i386/pr116896-2.C.jj 2024-10-03 14:40:37.203674018 +0200 +++ gcc/testsuite/g++.target/i386/pr116896-2.C 2024-10-04 10:55:07.468396073 +0200 @@ -0,0 +1,41 @@ +// PR middle-end/116896 +// { dg-do run { target c++20 } } +// { dg-options "-O2" } + +#include "pr116896-1.C" + +[[gnu::noipa]] auto +corge (int x) +{ + return x <=> 0; +} + +[[gnu::noipa]] auto +garply (unsigned x) +{ + return x <=> 0; +} + +int +main () +{ + if (foo (-1.0f, 1.0f) != std::partial_ordering::less + || foo (1.0f, -1.0f) != std::partial_ordering::greater + || foo (1.0f, 1.0f) != std::partial_ordering::equivalent + || foo (__builtin_nanf (""), 1.0f) != std::partial_ordering::unordered + || bar (-2.0f, 2.0f) != std::partial_ordering::less + || bar (2.0f, -2.0f) != std::partial_ordering::greater + || bar (-5.0f, -5.0f) != std::partial_ordering::equivalent + || baz (-42, 42) != std::strong_ordering::less + || baz (42, -42) != std::strong_ordering::greater + || baz (42, 42) != std::strong_ordering::equal + || qux (40, 42) != std::strong_ordering::less + || qux (42, 40) != std::strong_ordering::greater + || qux (40, 40) != std::strong_ordering::equal + || corge (-15) != std::strong_ordering::less + || corge (15) != std::strong_ordering::greater + || corge (0) != std::strong_ordering::equal + || garply (15) != std::strong_ordering::greater + || garply (0) != std::strong_ordering::equal) + __builtin_abort (); +}