From patchwork Wed Sep 13 14:54:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Wilco Dijkstra X-Patchwork-Id: 1833625 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=gcc.gnu.org header.i=@gcc.gnu.org header.a=rsa-sha256 header.s=default header.b=GyPM5LO2; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Rm3Qj038Sz1yh6 for ; Thu, 14 Sep 2023 00:55:09 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 088143858D3C for ; Wed, 13 Sep 2023 14:55:07 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 088143858D3C DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gcc.gnu.org; s=default; t=1694616907; bh=lc1pISpndslPvEp4wc+lBNAOTqo8o3ZNpYeaXSsc5dw=; h=To:CC:Subject:Date:List-Id:List-Unsubscribe:List-Archive: List-Post:List-Help:List-Subscribe:From:Reply-To:From; b=GyPM5LO2Olp1gfT/Z+4Zz+LedBB/IMewLsxkSDX3HxjNOerwk7GmQuObwBKRqllzv dR0cUUh3/pG5y5cyWUhfoZfBEoH9DUdseEkfqL0h6T9zFH0BTaRN2M3zEm4VhyJAKj MQ5HvND7PEY3kgZBanlyryZwa+nNJxMpiCoftT9E= X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from EUR05-DB8-obe.outbound.protection.outlook.com (mail-db8eur05on2075.outbound.protection.outlook.com [40.107.20.75]) by sourceware.org (Postfix) with ESMTPS id BCABE3858D39 for ; Wed, 13 Sep 2023 14:54:45 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org BCABE3858D39 Received: from AM6P191CA0047.EURP191.PROD.OUTLOOK.COM (2603:10a6:209:7f::24) by AS2PR08MB9149.eurprd08.prod.outlook.com (2603:10a6:20b:57b::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6745.34; Wed, 13 Sep 2023 14:54:42 +0000 Received: from AM7EUR03FT005.eop-EUR03.prod.protection.outlook.com (2603:10a6:209:7f:cafe::5b) by AM6P191CA0047.outlook.office365.com (2603:10a6:209:7f::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.19 via Frontend Transport; Wed, 13 Sep 2023 14:54:42 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; pr=C Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by AM7EUR03FT005.mail.protection.outlook.com (100.127.140.218) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.16 via Frontend Transport; Wed, 13 Sep 2023 14:54:42 +0000 Received: ("Tessian outbound 30c9f5e988c5:v175"); Wed, 13 Sep 2023 14:54:42 +0000 X-CheckRecipientChecked: true X-CR-MTA-CID: 1e2be0ba8ff30cb2 X-CR-MTA-TID: 64aa7808 Received: from 5bf6b3768a99.2 by 64aa7808-outbound-1.mta.getcheckrecipient.com id D58948B4-910A-41ED-A9D5-A280850FF7CD.1; Wed, 13 Sep 2023 14:54:30 +0000 Received: from EUR04-HE1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 5bf6b3768a99.2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Wed, 13 Sep 2023 14:54:30 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HPxBXCMOi9zCva5wPN5CoUE15iPiR6dDzE3NhoTEElYAZ+YDiECp4KQGC2Wd1i1f+cLg/x6L3L3MqP8BbbE5oFz5+8lH6BYgLk1Kpi67LZwB0i17z8lwWa3hQGc6JPK8+bHOwa46pD+RJVpHRjrEWicJaJqygmS5P1TcmlvFMOxAph5DfET4k5eafy8iGZGEy2MjH00E+gMvw0Jhg7qf4765zWjWT9nXYa2g7qMUDHzbqSTXErqMCnZyea8Grt3anFap/bIRCDUDR8Jnh1f4agX0Fk6ha2UYEsLp403+bT4MLnYUgDSzM14cIZ9pZkqTzNRhHuuQIVB0+55E8xoARA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=lc1pISpndslPvEp4wc+lBNAOTqo8o3ZNpYeaXSsc5dw=; b=GCKbYQR4+IgMz/QeH0SGSYe7J5BwPct26rjq9RBlbUkcE2Xp+iioMJUSaw35oaSf0WInUw0WV60yMQmV52dzD3t1Yk6H/pACQNNuOA+uw03olmgR555dIGP2DD/QECeWn8olxVKDOkZF9/lsyB5cfBVo61I9OuVb7ibkeWifhga/oY3Uzby0KvD+A/FhGn1jnLlqNCdkF2SA1G06NFpDcIWAYDvlJG0UxwkkL37cMf+jUlEF2jZqwu4UKpQATY1l3C6wHFnnU+8T8deBNr7lHDiOmafBfD3uiAcsjiarLE6IZAGEAaWv/vBaRrjugLPMwjYpqncRqwHFQg6iBj217Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=arm.com; dmarc=pass action=none header.from=arm.com; dkim=pass header.d=arm.com; arc=none Received: from PAWPR08MB8982.eurprd08.prod.outlook.com (2603:10a6:102:33f::20) by DU2PR08MB7325.eurprd08.prod.outlook.com (2603:10a6:10:2e4::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6792.19; Wed, 13 Sep 2023 14:54:28 +0000 Received: from PAWPR08MB8982.eurprd08.prod.outlook.com ([fe80::ff3d:6e95:9971:a7e]) by PAWPR08MB8982.eurprd08.prod.outlook.com ([fe80::ff3d:6e95:9971:a7e%5]) with mapi id 15.20.6745.034; Wed, 13 Sep 2023 14:54:28 +0000 To: GCC Patches CC: Richard Sandiford , Kyrylo Tkachov Subject: [PATCH] AArch64: Fix __sync_val_compare_and_swap [PR111404] Thread-Topic: [PATCH] AArch64: Fix __sync_val_compare_and_swap [PR111404] Thread-Index: AQHZ5lFymK1NCxAT/0uZ1wjY9RCv4w== Date: Wed, 13 Sep 2023 14:54:28 +0000 Message-ID: Accept-Language: en-GB, en-US Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: Authentication-Results-Original: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; x-ms-traffictypediagnostic: PAWPR08MB8982:EE_|DU2PR08MB7325:EE_|AM7EUR03FT005:EE_|AS2PR08MB9149:EE_ X-MS-Office365-Filtering-Correlation-Id: 48873f7b-5a97-418b-2c96-08dbb4695cff x-checkrecipientrouted: true nodisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: QLLj1xXJ6rF2glQQ89p9XXvyHxD0Z8mwM05aoF99FZYNbxRphwJFpPx6pZspIATcQfdB37AEFmCfErfgDgS3LVMcGSJ5/QeOjfs3hygOIHWymMMR1AigOTnUlwWEiBoq+b1nyHT8mA+os0on1+WvZrzrqD1koSF7GUztozqurapBocOn2zH78hIayZMfZX6Ya/CJKwKUBLRcvcfAQdRMT4+1QjYbVTi12h1P7oDhySxCxWm8t7gpMv+AKvvj4zk/tv7jau5xkg7zWIbcyKyenBn2e48b0/1WDkl5eHNYiskKbBkNnoGDoYDGZ+xIx/ewsLgkelIJVyYIadislWSSe+M7QJHe2/IA/9D2tcy2Z6fVxNSdrc0Glr7puYHoZbwafIwBYGReHG1o5VKXviIEOOGU6x3YC756YMeXq/tO3b68n5wleZQUwDuk2XvimPrGXzBD8OADY/DHGLRL2+b/DDmje9n8LLdHHZRHWZpbrCRjO2HZHLGNYA5w3tq6ZH7MCBzG/+GZFC4PIkJJ+lZpQk5L3uwLGze7HyCD+scfinu7tYhkW27GKDtnUEEszSvQZMbE859ypni/lDDiFHiqWSNKcSE99sWdyUP15pL9Lllk0rP7JnlUU0KyPeDiKm5r X-Forefront-Antispam-Report-Untrusted: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PAWPR08MB8982.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230031)(366004)(396003)(376002)(39860400002)(136003)(346002)(1800799009)(186009)(451199024)(2906002)(52536014)(86362001)(5660300002)(122000001)(33656002)(38070700005)(38100700002)(4326008)(8936002)(8676002)(55016003)(316002)(54906003)(41300700001)(6916009)(26005)(66446008)(66476007)(76116006)(64756008)(66556008)(66946007)(91956017)(9686003)(71200400001)(7696005)(6506007)(478600001); DIR:OUT; SFP:1101; MIME-Version: 1.0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU2PR08MB7325 Original-Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=arm.com; X-EOPAttributedMessage: 0 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM7EUR03FT005.eop-EUR03.prod.protection.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: 924c72c7-5b64-4b98-911c-08dbb469543b X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: YX3ua7PF5X2TmSqE5LJMJQbvIHwNB5yYL9dw+653n4LONoZ/chC+GZQYMX9UEHcqQxM8OsPtBGae1fkF5Ux0Ga4NosJgYL3iQn4ZyVwI7kOoFxe4l73GPxapUCce6tVtPqW4goEQu1iHzrFDYCQqUtNpHMgK6rlhTqe1FDWom7Dp45JL0AgG2Vef0bwfH0mAxOtlH+NX23y5dT6we9PJv9Hp9C4Um4pZ0vbR1yfqLRWUQaEdv53FyTe467C69Eb740Hjsw3QOP01HKhhgjdA4j9UgXDXrjjJbCDkniEAwfUB49FlOQcxyKM5V9pQA42RmwGqrv45Yu9e0tq1+fCOWRxsJujFVFKh3IpPR6NNuD+V0+WBT1lPgqtvnG7HuCta3719CPfsLMfmCT46yLASuT4Crc7AgAjmwbrTZ6nLyIDyH+8+vHA/snV5qu5GvuBIstRaRoWHuAxk/nmz/uUdTaZy0LBLOulO4U/JX+Egbf+h8eXuOe8ChhM/QhD6YwPOUGKESWZ8DvmNCVn/VQYTmAzsjbmcMl/jJ6ijYAeXZhoOgVyTpHFo/sAt3w1w2yTgfDxHPSftTXE3opDe6OlVla56gg0tvPHh/pxdvdGz8kp9FXd8mUTJFN9mjh9/7ZGmpSIuDxk2TWIUTtfLmWvGculUHtcpVyK4ZgQKUib5bdUOcPV/uDNKYewXQZufkqCBNt9VPjolLXLZKo+kqz3JwTl5XLPCnU/f0eoiXTJIJTqFJt0C6GSlpCofnqnTq9AB X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(13230031)(4636009)(39860400002)(346002)(376002)(396003)(136003)(82310400011)(1800799009)(186009)(451199024)(36840700001)(40470700004)(46966006)(6506007)(7696005)(356005)(40480700001)(40460700003)(36860700001)(82740400003)(86362001)(55016003)(47076005)(33656002)(81166007)(26005)(5660300002)(2906002)(336012)(41300700001)(478600001)(9686003)(4326008)(52536014)(316002)(8676002)(8936002)(70586007)(6916009)(54906003)(70206006); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Sep 2023 14:54:42.7224 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 48873f7b-5a97-418b-2c96-08dbb4695cff X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: AM7EUR03FT005.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS2PR08MB9149 X-Spam-Status: No, score=-10.7 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, FORGED_SPF_HELO, GIT_PATCH_0, KAM_DMARC_NONE, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_NONE, TXREP, UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Wilco Dijkstra via Gcc-patches From: Wilco Dijkstra Reply-To: Wilco Dijkstra Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org Sender: "Gcc-patches" __sync_val_compare_and_swap may be used on 128-bit types and either calls the outline atomic code or uses an inline loop. On AArch64 LDXP is only atomic if the value is stored successfully using STXP, but the current implementations do not perform the store if the comparison fails. In this case the value returned is not read atomically. Passes regress/bootstrap, OK for commit? gcc/ChangeLog/ PR target/111404 * config/aarch64/aarch64.cc (aarch64_split_compare_and_swap): For 128-bit store the loaded value and loop if needed. libgcc/ChangeLog/ PR target/111404 * config/aarch64/lse.S (__aarch64_cas16_acq_rel): Execute STLXP using either new value or loaded value. diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc index 5e8d0a0c91bc7719de2a8c5627b354cf905a4db0..c44c0b979d0cc3755c61dcf566cfddedccebf1ea 100644 --- a/gcc/config/aarch64/aarch64.cc +++ b/gcc/config/aarch64/aarch64.cc @@ -23413,11 +23413,11 @@ aarch64_split_compare_and_swap (rtx operands[]) mem = operands[1]; oldval = operands[2]; newval = operands[3]; - is_weak = (operands[4] != const0_rtx); model_rtx = operands[5]; scratch = operands[7]; mode = GET_MODE (mem); model = memmodel_from_int (INTVAL (model_rtx)); + is_weak = operands[4] != const0_rtx && mode != TImode; /* When OLDVAL is zero and we want the strong version we can emit a tighter loop: @@ -23478,6 +23478,33 @@ aarch64_split_compare_and_swap (rtx operands[]) else aarch64_gen_compare_reg (NE, scratch, const0_rtx); + /* 128-bit LDAXP is not atomic unless STLXP succeeds. So for a mismatch, + store the returned value and loop if the STLXP fails. */ + if (mode == TImode) + { + rtx_code_label *label3 = gen_label_rtx (); + emit_jump_insn (gen_rtx_SET (pc_rtx, gen_rtx_LABEL_REF (Pmode, label3))); + emit_barrier (); + + emit_label (label2); + aarch64_emit_store_exclusive (mode, scratch, mem, rval, model_rtx); + + if (aarch64_track_speculation) + { + /* Emit an explicit compare instruction, so that we can correctly + track the condition codes. */ + rtx cc_reg = aarch64_gen_compare_reg (NE, scratch, const0_rtx); + x = gen_rtx_NE (GET_MODE (cc_reg), cc_reg, const0_rtx); + } + else + x = gen_rtx_NE (VOIDmode, scratch, const0_rtx); + x = gen_rtx_IF_THEN_ELSE (VOIDmode, x, + gen_rtx_LABEL_REF (Pmode, label1), pc_rtx); + aarch64_emit_unlikely_jump (gen_rtx_SET (pc_rtx, x)); + + label2 = label3; + } + emit_label (label2); /* If we used a CBNZ in the exchange loop emit an explicit compare with RVAL diff --git a/libgcc/config/aarch64/lse.S b/libgcc/config/aarch64/lse.S index dde3a28e07b13669533dfc5e8fac0a9a6ac33dbd..ba05047ff02b6fc5752235bffa924fc4a2f48c04 100644 --- a/libgcc/config/aarch64/lse.S +++ b/libgcc/config/aarch64/lse.S @@ -160,6 +160,8 @@ see the files COPYING3 and COPYING.RUNTIME respectively. If not, see #define tmp0 16 #define tmp1 17 #define tmp2 15 +#define tmp3 14 +#define tmp4 13 #define BTI_C hint 34 @@ -233,10 +235,11 @@ STARTFN NAME(cas) 0: LDXP x0, x1, [x4] cmp x0, x(tmp0) ccmp x1, x(tmp1), #0, eq - bne 1f - STXP w(tmp2), x2, x3, [x4] - cbnz w(tmp2), 0b -1: BARRIER + csel x(tmp2), x2, x0, eq + csel x(tmp3), x3, x1, eq + STXP w(tmp4), x(tmp2), x(tmp3), [x4] + cbnz w(tmp4), 0b + BARRIER ret #endif