From patchwork Thu Dec 10 16:59:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tamar Christina X-Patchwork-Id: 1414367 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=sourceware.org; envelope-from=gcc-patches-bounces@gcc.gnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gcc.gnu.org Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=gcc.gnu.org header.i=@gcc.gnu.org header.a=rsa-sha256 header.s=default header.b=yg6RknnQ; dkim-atps=neutral Received: from sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4CsKt80hV3z9sRK for ; Fri, 11 Dec 2020 04:00:27 +1100 (AEDT) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 06D743972422; Thu, 10 Dec 2020 17:00:25 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 06D743972422 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1607619625; bh=8BPUtg6uYp7y4psF8m9X78zn3yoK6oYqtkb5r+e2m7A=; h=Date:To:Subject:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:Cc:From; b=yg6RknnQAM6xGFol3GO5PBfD6TrQ1gQbTqa0cSjhytRpDH/JnrjuvO/HjwTzj6jYB 2lMLKOzxbcRbe/sbUQtJiZAaDj99RViBCFO8VbZOOo1RVR5qsbefSG5790aU3PoF8S NKdu2Gtm8z0ndmNBgxR5YOyRfvD+a36HXyAqdd/U= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from EUR03-VE1-obe.outbound.protection.outlook.com (mail-eopbgr50080.outbound.protection.outlook.com [40.107.5.80]) by sourceware.org (Postfix) with ESMTPS id ED19B386F44F for ; Thu, 10 Dec 2020 17:00:19 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.3.2 sourceware.org ED19B386F44F Received: from AM6P192CA0012.EURP192.PROD.OUTLOOK.COM (2603:10a6:209:83::25) by VI1PR0802MB2351.eurprd08.prod.outlook.com (2603:10a6:800:a0::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3632.19; Thu, 10 Dec 2020 17:00:17 +0000 Received: from VE1EUR03FT059.eop-EUR03.prod.protection.outlook.com (2603:10a6:209:83:cafe::c5) by AM6P192CA0012.outlook.office365.com (2603:10a6:209:83::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3654.12 via Frontend Transport; Thu, 10 Dec 2020 17:00:17 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; gcc.gnu.org; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;gcc.gnu.org; dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by VE1EUR03FT059.mail.protection.outlook.com (10.152.19.60) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3654.12 via Frontend Transport; Thu, 10 Dec 2020 17:00:16 +0000 Received: ("Tessian outbound 76bd5a04122f:v71"); Thu, 10 Dec 2020 17:00:16 +0000 X-CheckRecipientChecked: true X-CR-MTA-CID: c6441e6c2e7a2e97 X-CR-MTA-TID: 64aa7808 Received: from ce5ae83fb9ea.2 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 6857171B-4B50-4BAC-BDEF-2A0C7E819C18.1; Thu, 10 Dec 2020 16:59:59 +0000 Received: from EUR04-HE1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id ce5ae83fb9ea.2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Thu, 10 Dec 2020 16:59:59 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DG/KrpZvCSufSxb/3O+wRerZf8wCEFExOgXf8hAjK/D3G7LoMNlrvYjdRXV7PyRvYOJKtzBc1f+lt1NG1F+LX3F18tIDHuCAK2jJ5X8LODTF36Qyz3bf4NfFwPUXB/trcLphtXakhGgr0rShX0+XmczKE2MrxlVQK9KvJsmxpZW9UoBlpRn1uopvEbM6xbRIsEvTH0uKLqqDbfizrBHkjPFAyHV93uJqn9PoctBoKSv+kkluhn6bBbeQZwspQXod+4RPSO2M8qhLS0S6c+gxnnKekJ0IHNa3DzX1WCMuBym3H08vBxgw32GCxsz0GIGX2ruh9vvVrYyqjYKVnz0KTA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8BPUtg6uYp7y4psF8m9X78zn3yoK6oYqtkb5r+e2m7A=; b=mPQ1FaVIQTWhiSCrwpIUZ+a8j7r/b3nnHOGgvceOBnW1xrOieUTF+APjdxyXh05sS10hlB11xHBHRG1e/UfwFxEMBrofG6MJmGA9LGnVwAJHyE/lbu2nQ3Izr80ZBPPkHaKvcc8mqkT9Lgx/JNabBQggcHzFCJrLRKgvOp4btWAVV0oaE5GO9jpfiJpjRdvqeba/aqf2pU4jI/NP90sMsCoKJT39qflJKdOfuPnxdBTFrVFuR4WBGorC+/hyAn6RNzCy6bBgRwl7JY4hsDIw+26LECTTOW25oSeiouOd8bnI/O+Su44fJ7JkWm6qUjtAK9rRk8NpIVG6wKU7reg4NQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none Authentication-Results-Original: gcc.gnu.org; dkim=none (message not signed) header.d=none;gcc.gnu.org; dmarc=none action=none header.from=arm.com; Received: from VI1PR08MB5325.eurprd08.prod.outlook.com (2603:10a6:803:13e::17) by VI1PR08MB5326.eurprd08.prod.outlook.com (2603:10a6:803:12d::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3632.17; Thu, 10 Dec 2020 16:59:56 +0000 Received: from VI1PR08MB5325.eurprd08.prod.outlook.com ([fe80::f937:5b3:12e1:8297]) by VI1PR08MB5325.eurprd08.prod.outlook.com ([fe80::f937:5b3:12e1:8297%6]) with mapi id 15.20.3632.023; Thu, 10 Dec 2020 16:59:56 +0000 Date: Thu, 10 Dec 2020 16:59:53 +0000 To: gcc-patches@gcc.gnu.org Subject: [PATCH]AArch64: Add NEON, SVE and SVE2 RTL patterns for Complex Addition, Multiply and FMA. Message-ID: Content-Disposition: inline User-Agent: Mutt/1.9.4 (2018-02-28) X-Originating-IP: [217.140.106.53] X-ClientProxiedBy: LO2P265CA0226.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:b::22) To VI1PR08MB5325.eurprd08.prod.outlook.com (2603:10a6:803:13e::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from arm.com (217.140.106.53) by LO2P265CA0226.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:b::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3654.12 via Frontend Transport; Thu, 10 Dec 2020 16:59:55 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 98e88f14-fbb5-4bc2-3478-08d89d2d11cf X-MS-TrafficTypeDiagnostic: VI1PR08MB5326:|VI1PR0802MB2351: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: x-checkrecipientrouted: true NoDisclaimer: true X-MS-Oob-TLC-OOBClassifiers: OLM:1284;OLM:1284; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: 13gs+YVp0f8CqIqtR/ZDs99nsy8qnVjCtCmWc0qHYdA/etFCudUqBgp5TbRovj/ssnyLTZ33/4h5CjiPffTvkTuNe7h/szTDM9XHOuc33CSaaSYHiOImp18bnb0FMo+WAno4WLUw/V65brWueF07wc4Z38mtgeZXdKEE6I+nJboNxE479kbbRp3enfeUEmHF5dBxk73FnkIyC0U0X8Z1kHXV7mtWYGtYAJTBlr7zYDC6NJhvmoJgzSxatJp7h3xCR6Mt4kHBurjtf/Xd4tPbqVAlmoRXR72AEFT1/Az/NuwIcmDL4IeVEPafbM5yFnAq691Vik46kAQpYSd0b/tbO57Qsswf1wcHr72BmpAGsSRzbAZf2A3TAuyYJUf1xwRbJ4h2Jn9F8hzeXjwlIdLiog== X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:VI1PR08MB5325.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(136003)(346002)(39860400002)(396003)(366004)(376002)(52116002)(55016002)(8676002)(86362001)(33964004)(66616009)(316002)(44832011)(44144004)(83380400001)(26005)(956004)(6916009)(2616005)(16526019)(66556008)(186003)(8936002)(8886007)(66476007)(4326008)(7696005)(2906002)(478600001)(36756003)(235185007)(5660300002)(66946007)(4743002)(4216001)(2700100001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?utf-8?q?0g9xTrYrfAFyGyS8DgXtSubP3yEblk?= =?utf-8?q?rGIr2Ja/BtPUuqP4curc+04q2MwgsKkWzUnPRPjKyqaQ1Ii50PCmnRmPYDIgEKeTw?= =?utf-8?q?EEvUR//ZUo6RlYQBHTGe8fmOdQy4YfcqDjOPCm5izGJy2P9n6XKS6PZHdpW3Whqt+?= =?utf-8?q?4qUsByBtv1jMattUol278svznRf4qIkXiFK5gYOInUHxr+Zj0Gr9sSG5Dty9o0fTZ?= =?utf-8?q?8Cz5KkitHmz/pgH/enntsIa5XzkUjwz9Oo4f8Umt04NXLQDZzsrJScc9e18KnZbW0?= =?utf-8?q?z4aIxzH8lZzD+FFY74gBTuQenokxuuBK0z/qCPBmVTHhaeoM0Ut86Pa+QECTTMfmo?= =?utf-8?q?W8kppT4kJvruAS9RN4eAO5ctNaZLts5wFw6G6jnpwt4HTeOicHl3TCs88HOkY0pvt?= =?utf-8?q?lvV9FnecOAJVydguEbzfOY9GCtgAf/Ipcg8L5fb9mavHShLLon6AOsx/4wxHJ0WqF?= =?utf-8?q?JFBu0uS3OZRH+Oav6qkzOT5wCqofu19f8y0WAe/o7xBpetQ13+S/ecs9GdnLBgjjI?= =?utf-8?q?QDaZ3fZtti1LBWVaUc5Hn4rThRV1EH9OoH7ZzgjCZZwXfbS5CQ18a3By75GbCVJcq?= =?utf-8?q?z5RF9bbWAKmokIkqba+aMws3HgIlOWtcVEAf/ssWFk8eCp92JhS1JWlrihBC3wDKS?= =?utf-8?q?As+/u89wi5SU1jANoVvk+gQu1qLTwabZhwVhAP44LLN5pq6gNcTGoTYnkTBNsbEfU?= =?utf-8?q?tZFHkzi9f0Cx9LXspKaF7JFvbEfdHn8gB1hBseZuaroZJZ7aHwUeIXkvYjbz0FmpO?= =?utf-8?q?XmWT2zOFac6pNb/pM0pQe8e8p5Kdur27+BvM6egIW9O43SZ+t+3mxeQS8QxsgBIUU?= =?utf-8?q?bJnioDj7mXqJKugStwHnFNgWHd0jeEcf1SB6+eFLxRKwlTVB4fHoKcL2UC4F1I8RA?= =?utf-8?q?eKVqLuAGn4OpWiDTn1o6BSl02Y9kl1+ds15rFSTnK6MB+uoP9YfhOlRbKVfvzVpEf?= =?utf-8?q?NlvWHfYo4CVlf9mVEDp?= X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR08MB5326 Original-Authentication-Results: gcc.gnu.org; dkim=none (message not signed) header.d=none;gcc.gnu.org; dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: VE1EUR03FT059.eop-EUR03.prod.protection.outlook.com X-MS-Office365-Filtering-Correlation-Id-Prvs: 69565b8d-9ff0-4c7a-4ec8-08d89d2d057e X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: zxdNAj8P9V7kO8iEJABjh6g2zoQ7CxmHGtIRjqaVQU+KMdOzSIxADp55kwoMahsvrfP49ehQWqcaRSayrpYbX6ZdXsSAOaKLl8V3iYc9A7ivLjB3pE1Cnee89Co/oTnSlAa5E2K0VjFIhfHYid/XoozhiSR0yaOnOkKVVkrOtGZAoxu2ipeWRSsmTgypPJo4L/SElCZ/y64Vj3EKD4v6PFFMc8O83kt5rWIHxt3t8o66zlTVA4TMti45xzMit2hXph244oNynM4siiy1qmnAcVklVhlmqSYcETsimZcnkOMkGL+NdzUv5/4VZTsf9nSeZIo6TJGDoHvaJOneHepUFsgqat9aNL/wBRtyF66/um5sNJAQu9iB2aeIcgKvSGOMTLaN2QoFDcTDMN4Obnx18atozEvay2z5v3N5i8i35mq9L5frjejdsPk5tkon6jQX X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(4636009)(346002)(396003)(39860400002)(136003)(376002)(46966005)(356005)(66616009)(55016002)(336012)(82310400003)(86362001)(44832011)(186003)(82740400003)(8936002)(2616005)(70586007)(47076004)(4326008)(956004)(44144004)(70206006)(33964004)(83380400001)(81166007)(36756003)(4743002)(6916009)(235185007)(2906002)(26005)(7696005)(8886007)(478600001)(8676002)(316002)(16526019)(5660300002)(4216001)(2700100001); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 Dec 2020 17:00:16.9486 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 98e88f14-fbb5-4bc2-3478-08d89d2d11cf X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: VE1EUR03FT059.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0802MB2351 X-Spam-Status: No, score=-13.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, KAM_ASCII_DIVIDERS, KAM_LOTSOFHASH, MSGID_FROM_MTA_HEADER, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_PASS, TXREP, UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Tamar Christina via Gcc-patches From: Tamar Christina Reply-To: Tamar Christina Cc: Richard.Earnshaw@arm.com, nd@arm.com, richard.sandiford@arm.com, Marcus.Shawcroft@arm.com Errors-To: gcc-patches-bounces@gcc.gnu.org Sender: "Gcc-patches" Hi All, This adds implementation for the optabs for complex operations. With this the following C code: void f90 (float complex a[restrict N], float complex b[restrict N], float complex c[restrict N]) { for (int i=0; i < N; i++) c[i] = a[i] + (b[i] * I); } generates f90: mov x3, 0 .p2align 3,,7 .L2: ldr q0, [x0, x3] ldr q1, [x1, x3] fcadd v0.4s, v0.4s, v1.4s, #90 str q0, [x2, x3] add x3, x3, 16 cmp x3, 1600 bne .L2 ret instead of f90: add x3, x1, 1600 .p2align 3,,7 .L2: ld2 {v4.4s - v5.4s}, [x0], 32 ld2 {v2.4s - v3.4s}, [x1], 32 fsub v0.4s, v4.4s, v3.4s fadd v1.4s, v5.4s, v2.4s st2 {v0.4s - v1.4s}, [x2], 32 cmp x3, x1 bne .L2 ret It defined a new iterator VALL_ARITH which contains types for which we can do general arithmetic (excludes bfloat16). Bootstrapped Regtested on aarch64-none-linux-gnu and no issues. Checked with armv8-a+sve2+fp16 and no issues. Note that sue to a mid-end limitation SLP for SVE currently fails for some permutes. The tests have these marked as XFAIL. I do intend to fix this soon. Matching tests for these are in the mid-end patches. Note that The mid-end patches are still being respun and I may need to change the order of some parameters but no other change is expected and would like to decrease the size of future patches. As such.. Ok for master? Thanks, Tamar gcc/ChangeLog: * config/aarch64/aarch64-simd.md (cadd3, cml4, cmul3): New. * config/aarch64/iterators.md (VALL_ARITH, UNSPEC_FCMUL, UNSPEC_FCMUL180, UNSPEC_FCMLS, UNSPEC_FCMLS180, UNSPEC_CMLS, UNSPEC_CMLS180, UNSPEC_CMUL, UNSPEC_CMUL180, FCMLA_OP, FCMUL_OP, rot_op, rotsplit1, rotsplit2, fcmac1, sve_rot1, sve_rot2, SVE2_INT_CMLA_OP, SVE2_INT_CMUL_OP, SVE2_INT_CADD_OP): New.): New.): New. (rot): Add UNSPEC_FCMLS, UNSPEC_FCMUL, UNSPEC_FCMUL180. * config/aarch64/aarch64-sve.md (cadd3, cml4, cmul3): New. * config/aarch64/aarch64-sve2.md (cadd3, cml4, cmul3): New. diff --git a/gcc/config/aarch64/aarch64-simd.md b/gcc/config/aarch64/aarch64-simd.md index 68baf416045178b0ebcfeb8de2d201f625f1c317..1aa74beeee154e054f2a01f8843dfed218fe850b 100644 --- a/gcc/config/aarch64/aarch64-simd.md +++ b/gcc/config/aarch64/aarch64-simd.md @@ -449,6 +449,14 @@ (define_insn "aarch64_fcadd" [(set_attr "type" "neon_fcadd")] ) +(define_expand "cadd3" + [(set (match_operand:VHSDF 0 "register_operand") + (unspec:VHSDF [(match_operand:VHSDF 1 "register_operand") + (match_operand:VHSDF 2 "register_operand")] + FCADD))] + "TARGET_COMPLEX && !BYTES_BIG_ENDIAN" +) + (define_insn "aarch64_fcmla" [(set (match_operand:VHSDF 0 "register_operand" "=w") (plus:VHSDF (match_operand:VHSDF 1 "register_operand" "0") @@ -508,6 +516,47 @@ (define_insn "aarch64_fcmlaq_lane" [(set_attr "type" "neon_fcmla")] ) +;; The complex mla/mls operations always need to expand to two instructions. +;; The first operation does half the computation and the second does the +;; remainder. Because of this, expand early. +(define_expand "cml4" + [(set (match_operand:VHSDF 0 "register_operand") + (plus:VHSDF (match_operand:VHSDF 1 "register_operand") + (unspec:VHSDF [(match_operand:VHSDF 2 "register_operand") + (match_operand:VHSDF 3 "register_operand")] + FCMLA_OP)))] + "TARGET_COMPLEX && !BYTES_BIG_ENDIAN" +{ + rtx tmp = gen_reg_rtx (mode); + emit_insn (gen_aarch64_fcmla (tmp, operands[1], + operands[2], operands[3])); + emit_insn (gen_aarch64_fcmla (operands[0], tmp, + operands[2], operands[3])); + DONE; +}) + +;; The complex mul operations always need to expand to two instructions. +;; The first operation does half the computation and the second does the +;; remainder. Because of this, expand early. +(define_expand "cmul3" + [(set (match_operand:VHSDF 0 "register_operand") + (unspec:VHSDF [(match_operand:VHSDF 1 "register_operand") + (match_operand:VHSDF 2 "register_operand")] + FCMUL_OP))] + "TARGET_COMPLEX && !BYTES_BIG_ENDIAN" +{ + rtx tmp = gen_reg_rtx (mode); + rtx res1 = gen_reg_rtx (mode); + emit_move_insn (tmp, CONST0_RTX (mode)); + emit_insn (gen_aarch64_fcmla (res1, tmp, + operands[1], operands[2])); + emit_insn (gen_aarch64_fcmla (operands[0], res1, + operands[1], operands[2])); + DONE; +}) + + + ;; These instructions map to the __builtins for the Dot Product operations. (define_insn "aarch64_dot" [(set (match_operand:VS 0 "register_operand" "=w") diff --git a/gcc/config/aarch64/aarch64-sve.md b/gcc/config/aarch64/aarch64-sve.md index 6359c40bdecda6c126bd70bef66561dd1da44dc9..7d27a84016d687cb6c019f98b99a7aacf8b3a031 100644 --- a/gcc/config/aarch64/aarch64-sve.md +++ b/gcc/config/aarch64/aarch64-sve.md @@ -5480,6 +5480,20 @@ (define_expand "@cond_" "TARGET_SVE" ) +;; Predicated FCADD using ptrue for unpredicated optab for auto-vectorizer +(define_expand "@cadd3" + [(set (match_operand:SVE_FULL_F 0 "register_operand") + (unspec:SVE_FULL_F + [(match_dup 3) + (const_int SVE_RELAXED_GP) + (match_operand:SVE_FULL_F 1 "register_operand") + (match_operand:SVE_FULL_F 2 "register_operand")] + SVE_COND_FCADD))] + "TARGET_SVE" +{ + operands[3] = aarch64_ptrue_reg (mode); +}) + ;; Predicated FCADD, merging with the first input. (define_insn_and_rewrite "*cond__2_relaxed" [(set (match_operand:SVE_FULL_F 0 "register_operand" "=w, ?&w") @@ -7152,6 +7166,64 @@ (define_insn "@aarch64_pred_" [(set_attr "movprfx" "*,yes")] ) +;; unpredicated optab pattern for auto-vectorizer +;; The complex mla/mls operations always need to expand to two instructions. +;; The first operation does half the computation and the second does the +;; remainder. Because of this, expand early. +(define_expand "cml4" + [(set (match_operand:SVE_FULL_F 0 "register_operand") + (unspec:SVE_FULL_F + [(match_dup 4) + (match_dup 5) + (match_operand:SVE_FULL_F 1 "register_operand") + (match_operand:SVE_FULL_F 2 "register_operand") + (match_operand:SVE_FULL_F 3 "register_operand")] + FCMLA_OP))] + "TARGET_SVE && !BYTES_BIG_ENDIAN" +{ + operands[4] = aarch64_ptrue_reg (mode); + operands[5] = gen_int_mode (SVE_RELAXED_GP, SImode); + rtx tmp = gen_reg_rtx (mode); + emit_insn ( + gen_aarch64_pred_fcmla (tmp, operands[4], + operands[1], operands[2], + operands[3], operands[5])); + emit_insn ( + gen_aarch64_pred_fcmla (operands[0], operands[4], + tmp, operands[2], + operands[3], operands[5])); + DONE; +}) + +;; unpredicated optab pattern for auto-vectorizer +;; The complex mul operations always need to expand to two instructions. +;; The first operation does half the computation and the second does the +;; remainder. Because of this, expand early. +(define_expand "cmul3" + [(set (match_operand:SVE_FULL_F 0 "register_operand") + (unspec:SVE_FULL_F + [(match_dup 3) + (match_dup 4) + (match_operand:SVE_FULL_F 1 "register_operand") + (match_operand:SVE_FULL_F 2 "register_operand") + (match_dup 5)] + FCMUL_OP))] + "TARGET_SVE && !BYTES_BIG_ENDIAN" +{ + operands[3] = aarch64_ptrue_reg (mode); + operands[4] = gen_int_mode (SVE_RELAXED_GP, SImode); + operands[5] = force_reg (mode, CONST0_RTX (mode)); + rtx tmp = gen_reg_rtx (mode); + emit_insn ( + gen_aarch64_pred_fcmla (tmp, operands[3], operands[1], + operands[2], operands[5], operands[4])); + emit_insn ( + gen_aarch64_pred_fcmla (operands[0], operands[3], operands[1], + operands[2], tmp, + operands[4])); + DONE; +}) + ;; Predicated FCMLA with merging. (define_expand "@cond_" [(set (match_operand:SVE_FULL_F 0 "register_operand") diff --git a/gcc/config/aarch64/aarch64-sve2.md b/gcc/config/aarch64/aarch64-sve2.md index 772c35079c9441448534471fba4dba622322b8fc..58594f985e5a98a188f32d96c6f71c9f4e0a6f05 100644 --- a/gcc/config/aarch64/aarch64-sve2.md +++ b/gcc/config/aarch64/aarch64-sve2.md @@ -1799,6 +1799,16 @@ (define_insn "@aarch64_sve_" [(set_attr "movprfx" "*,yes")] ) +;; unpredicated optab pattern for auto-vectorizer +(define_expand "cadd3" + [(set (match_operand:SVE_FULL_I 0 "register_operand") + (unspec:SVE_FULL_I + [(match_operand:SVE_FULL_I 1 "register_operand") + (match_operand:SVE_FULL_I 2 "register_operand")] + SVE2_INT_CADD_OP))] + "TARGET_SVE2 && !BYTES_BIG_ENDIAN" +) + ;; ------------------------------------------------------------------------- ;; ---- [INT] Complex ternary operations ;; ------------------------------------------------------------------------- @@ -1838,6 +1848,49 @@ (define_insn "@aarch64__lane_" [(set_attr "movprfx" "*,yes")] ) +;; unpredicated optab pattern for auto-vectorizer +;; The complex mla/mls operations always need to expand to two instructions. +;; The first operation does half the computation and the second does the +;; remainder. Because of this, expand early. +(define_expand "cml4" + [(set (match_operand:SVE_FULL_I 0 "register_operand") + (plus:SVE_FULL_I (match_operand:SVE_FULL_I 1 "register_operand") + (unspec:SVE_FULL_I + [(match_operand:SVE_FULL_I 2 "register_operand") + (match_operand:SVE_FULL_I 3 "register_operand")] + SVE2_INT_CMLA_OP)))] + "TARGET_SVE2 && !BYTES_BIG_ENDIAN" +{ + rtx tmp = gen_reg_rtx (mode); + emit_insn (gen_aarch64_sve_cmla (tmp, operands[1], + operands[2], operands[3])); + emit_insn (gen_aarch64_sve_cmla (operands[0], tmp, + operands[2], operands[3])); + DONE; +}) + +;; unpredicated optab pattern for auto-vectorizer +;; The complex mul operations always need to expand to two instructions. +;; The first operation does half the computation and the second does the +;; remainder. Because of this, expand early. +(define_expand "cmul3" + [(set (match_operand:SVE_FULL_I 0 "register_operand") + (unspec:SVE_FULL_I + [(match_operand:SVE_FULL_I 1 "register_operand") + (match_operand:SVE_FULL_I 2 "register_operand") + (match_dup 3)] + SVE2_INT_CMUL_OP))] + "TARGET_SVE2 && !BYTES_BIG_ENDIAN" +{ + operands[3] = force_reg (mode, CONST0_RTX (mode)); + rtx tmp = gen_reg_rtx (mode); + emit_insn (gen_aarch64_sve_cmla (tmp, operands[3], + operands[1], operands[2])); + emit_insn (gen_aarch64_sve_cmla (operands[0], tmp, + operands[1], operands[2])); + DONE; +}) + ;; ------------------------------------------------------------------------- ;; ---- [INT] Complex dot product ;; ------------------------------------------------------------------------- diff --git a/gcc/config/aarch64/iterators.md b/gcc/config/aarch64/iterators.md index fb1426b7752890848cb49722ef7442d96cb1408b..dd88e63f4e3a60ffe0d0276f13c6068161511cb9 100644 --- a/gcc/config/aarch64/iterators.md +++ b/gcc/config/aarch64/iterators.md @@ -182,6 +182,11 @@ (define_mode_iterator V2F [V2SF V2DF]) ;; All Advanced SIMD modes on which we support any arithmetic operations. (define_mode_iterator VALL [V8QI V16QI V4HI V8HI V2SI V4SI V2DI V2SF V4SF V2DF]) +;; All Advanced SIMD modes suitable for performing arithmetics. +(define_mode_iterator VALL_ARITH [V8QI V16QI V4HI V8HI V2SI V4SI V2DI + (V4HF "TARGET_SIMD_F16INST") (V8HF "TARGET_SIMD_F16INST") + V2SF V4SF V2DF]) + ;; All Advanced SIMD modes suitable for moving, loading, and storing. (define_mode_iterator VALL_F16 [V8QI V16QI V4HI V8HI V2SI V4SI V2DI V4HF V8HF V4BF V8BF V2SF V4SF V2DF]) @@ -708,6 +713,10 @@ (define_c_enum "unspec" UNSPEC_FCMLA90 ; Used in aarch64-simd.md. UNSPEC_FCMLA180 ; Used in aarch64-simd.md. UNSPEC_FCMLA270 ; Used in aarch64-simd.md. + UNSPEC_FCMUL ; Used in aarch64-simd.md. + UNSPEC_FCMUL180 ; Used in aarch64-simd.md. + UNSPEC_FCMLS ; Used in aarch64-simd.md. + UNSPEC_FCMLS180 ; Used in aarch64-simd.md. UNSPEC_ASRD ; Used in aarch64-sve.md. UNSPEC_ADCLB ; Used in aarch64-sve2.md. UNSPEC_ADCLT ; Used in aarch64-sve2.md. @@ -726,6 +735,10 @@ (define_c_enum "unspec" UNSPEC_CMLA180 ; Used in aarch64-sve2.md. UNSPEC_CMLA270 ; Used in aarch64-sve2.md. UNSPEC_CMLA90 ; Used in aarch64-sve2.md. + UNSPEC_CMLS ; Used in aarch64-sve2.md. + UNSPEC_CMLS180 ; Used in aarch64-sve2.md. + UNSPEC_CMUL ; Used in aarch64-sve2.md. + UNSPEC_CMUL180 ; Used in aarch64-sve2.md. UNSPEC_COND_FCVTLT ; Used in aarch64-sve2.md. UNSPEC_COND_FCVTNT ; Used in aarch64-sve2.md. UNSPEC_COND_FCVTX ; Used in aarch64-sve2.md. @@ -2598,6 +2611,23 @@ (define_int_iterator SVE2_INT_CMLA [UNSPEC_CMLA UNSPEC_SQRDCMLAH180 UNSPEC_SQRDCMLAH270]) +;; Unlike the normal CMLA instructions these represent the actual operation you +;; to be performed. They will always need to be expanded into multiple +;; sequences consisting of CMLA. +(define_int_iterator SVE2_INT_CMLA_OP [UNSPEC_CMLA + UNSPEC_CMLA180 + UNSPEC_CMLS]) + +;; Unlike the normal CMLA instructions these represent the actual operation you +;; to be performed. They will always need to be expanded into multiple +;; sequences consisting of CMLA. +(define_int_iterator SVE2_INT_CMUL_OP [UNSPEC_CMUL + UNSPEC_CMUL180]) + +;; Same as SVE2_INT_CADD but exclude the saturating instructions +(define_int_iterator SVE2_INT_CADD_OP [UNSPEC_CADD90 + UNSPEC_CADD270]) + (define_int_iterator SVE2_INT_CDOT [UNSPEC_CDOT UNSPEC_CDOT90 UNSPEC_CDOT180 @@ -2708,6 +2738,14 @@ (define_int_iterator FMMLA [UNSPEC_FMMLA]) (define_int_iterator BF_MLA [UNSPEC_BFMLALB UNSPEC_BFMLALT]) +(define_int_iterator FCMLA_OP [UNSPEC_FCMLA + UNSPEC_FCMLA180 + UNSPEC_FCMLS + UNSPEC_FCMLS180]) + +(define_int_iterator FCMUL_OP [UNSPEC_FCMUL + UNSPEC_FCMUL180]) + ;; Iterators for atomic operations. (define_int_iterator ATOMIC_LDOP @@ -3403,6 +3441,7 @@ (define_int_attr rot [(UNSPEC_CADD90 "90") (UNSPEC_CMLA270 "270") (UNSPEC_FCADD90 "90") (UNSPEC_FCADD270 "270") + (UNSPEC_FCMLS "0") (UNSPEC_FCMLA "0") (UNSPEC_FCMLA90 "90") (UNSPEC_FCMLA180 "180") @@ -3418,7 +3457,85 @@ (define_int_attr rot [(UNSPEC_CADD90 "90") (UNSPEC_COND_FCMLA "0") (UNSPEC_COND_FCMLA90 "90") (UNSPEC_COND_FCMLA180 "180") - (UNSPEC_COND_FCMLA270 "270")]) + (UNSPEC_COND_FCMLA270 "270") + (UNSPEC_FCMUL "0") + (UNSPEC_FCMUL180 "180")]) + +;; A conjucate is a negation of the imaginary component +;; The number in the inspecs are the rotation component of the instruction, e.g +;; FCMLS180 means use the instruction with #180. +;; The iterator is used to produce the right name mangling for the function. +;; +;; The rotation value does not directly correlate to a rotation along the argant +;; plane as the instructions only perform half the computation. +;; +;; For the implementation we threat any rotation by 0 as normal and 180 as +;; conjucate. This is only for implementing the vectorizer patterns. +(define_int_attr rot_op [(UNSPEC_FCMLS "") + (UNSPEC_FCMLS180 "_conj") + (UNSPEC_FCMLA "") + (UNSPEC_FCMLA180 "_conj") + (UNSPEC_FCMUL "") + (UNSPEC_FCMUL180 "_conj") + (UNSPEC_CMLS "") + (UNSPEC_CMLA "") + (UNSPEC_CMLA180 "_conj") + (UNSPEC_CMUL "") + (UNSPEC_CMUL180 "_conj")]) + +;; The complex operations when performed on a real complex number require two +;; instructions to perform the operation. e.g. complex multiplication requires +;; two FCMUL with a particular rotation value. +;; +;; These values can be looked up in rotsplit1 and rotsplit2. as an example +;; FCMUL needs the first instruction to use #0 and the second #90. +(define_int_attr rotsplit1 [(UNSPEC_FCMLA "0") + (UNSPEC_FCMLA180 "0") + (UNSPEC_FCMUL "0") + (UNSPEC_FCMUL180 "0") + (UNSPEC_FCMLS "270") + (UNSPEC_FCMLS180 "90")]) + +(define_int_attr rotsplit2 [(UNSPEC_FCMLA "90") + (UNSPEC_FCMLA180 "270") + (UNSPEC_FCMUL "90") + (UNSPEC_FCMUL180 "270") + (UNSPEC_FCMLS "180") + (UNSPEC_FCMLS180 "180")]) + +;; SVE has slightly different namings from NEON so we have to split these +;; iterators. +(define_int_attr sve_rot1 [(UNSPEC_FCMLA "") + (UNSPEC_FCMLA180 "") + (UNSPEC_FCMUL "") + (UNSPEC_FCMUL180 "") + (UNSPEC_FCMLS "270") + (UNSPEC_FCMLS180 "90") + (UNSPEC_CMLA "") + (UNSPEC_CMLA180 "") + (UNSPEC_CMUL "") + (UNSPEC_CMUL180 "") + (UNSPEC_CMLS "270") + (UNSPEC_CMLS180 "90")]) + +(define_int_attr sve_rot2 [(UNSPEC_FCMLA "90") + (UNSPEC_FCMLA180 "270") + (UNSPEC_FCMUL "90") + (UNSPEC_FCMUL180 "270") + (UNSPEC_FCMLS "180") + (UNSPEC_FCMLS180 "180") + (UNSPEC_CMLA "90") + (UNSPEC_CMLA180 "270") + (UNSPEC_CMUL "90") + (UNSPEC_CMUL180 "270") + (UNSPEC_CMLS "180") + (UNSPEC_CMLS180 "180")]) + + +(define_int_attr fcmac1 [(UNSPEC_FCMLA "a") (UNSPEC_FCMLA180 "a") + (UNSPEC_FCMLS "s") (UNSPEC_FCMLS180 "s") + (UNSPEC_CMLA "a") (UNSPEC_CMLA180 "a") + (UNSPEC_CMLS "s") (UNSPEC_CMLS180 "s")]) (define_int_attr sve_fmla_op [(UNSPEC_COND_FMLA "fmla") (UNSPEC_COND_FMLS "fmls") diff --git a/gcc/config/arm/iterators.md b/gcc/config/arm/iterators.md index 592af35f038f48b5f4ac622a0ed944ffc2a140f2..43e1ebd87cf69e716474bb6ee9bcdd405523d8da 100644 --- a/gcc/config/arm/iterators.md +++ b/gcc/config/arm/iterators.md @@ -712,7 +712,7 @@ (define_mode_attr v_cmp_result [(V8QI "v8qi") (V16QI "v16qi") (DI "di") (V2DI "v2di") (V2SF "v2si") (V4SF "v4si")]) -;; Get element type from double-width mode, for operations where we +;; Get element type from double-width mode, for operations where we ;; don't care about signedness. (define_mode_attr V_if_elem [(V8QI "i8") (V16QI "i8") (V4HI "i16") (V8HI "i16") @@ -1180,7 +1180,49 @@ (define_int_attr rot [(UNSPEC_VCADD90 "90") (UNSPEC_VCMLA "0") (UNSPEC_VCMLA90 "90") (UNSPEC_VCMLA180 "180") - (UNSPEC_VCMLA270 "270")]) + (UNSPEC_VCMLA270 "270") + (UNSPEC_VCMUL "0") + (UNSPEC_VCMUL180 "180")]) + +;; A conjucate is a negation of the imaginary component +;; The number in the inspecs are the rotation component of the instruction, e.g +;; FCMLS180 means use the instruction with #180. +;; The iterator is used to produce the right name mangling for the function. +;; +;; The rotation value does not directly correlate to a rotation along the argant +;; plane as the instructions only perform half the computation. +;; +;; For the implementation we threat any rotation by 0 as normal and 180 as +;; conjucate. This is only for implementing the vectorizer patterns. +(define_int_attr rot_op [(UNSPEC_VCMLS "") + (UNSPEC_VCMLS180 "_conj") + (UNSPEC_VCMLA "") + (UNSPEC_VCMLA180 "_conj") + (UNSPEC_VCMUL "") + (UNSPEC_VCMUL180 "_conj")]) + +;; The complex operations when performed on a real complex number require two +;; instructions to perform the operation. e.g. complex multiplication requires +;; two FCMUL with a particular rotation value. +;; +;; These values can be looked up in rotsplit1 and rotsplit2. as an example +;; FCMUL needs the first instruction to use #0 and the second #90. +(define_int_attr rotsplit1 [(UNSPEC_VCMLA "0") + (UNSPEC_VCMLA180 "0") + (UNSPEC_VCMUL "0") + (UNSPEC_VCMUL180 "0") + (UNSPEC_VCMLS "270") + (UNSPEC_VCMLS180 "90")]) + +(define_int_attr rotsplit2 [(UNSPEC_VCMLA "90") + (UNSPEC_VCMLA180 "270") + (UNSPEC_VCMUL "90") + (UNSPEC_VCMUL180 "270") + (UNSPEC_VCMLS "180") + (UNSPEC_VCMLS180 "180")]) + +(define_int_attr fcmac1 [(UNSPEC_VCMLA "a") (UNSPEC_VCMLA180 "a") + (UNSPEC_VCMLS "s") (UNSPEC_VCMLS180 "s")]) (define_int_attr simd32_op [(UNSPEC_QADD8 "qadd8") (UNSPEC_QSUB8 "qsub8") (UNSPEC_SHADD8 "shadd8") (UNSPEC_SHSUB8 "shsub8")