From patchwork Wed Aug 30 14:34:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Szabolcs Nagy X-Patchwork-Id: 1827814 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; secure) header.d=sourceware.org header.i=@sourceware.org header.a=rsa-sha256 header.s=default header.b=y0H2F/Zw; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4RbRdr4XT1z1yfX for ; Thu, 31 Aug 2023 00:34:55 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 54338385840D for ; Wed, 30 Aug 2023 14:34:53 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 54338385840D DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1693406093; bh=V7aWR6b+4WCf14n5w+wtvWcQ+mJFbJNP9WumbFasx/o=; h=To:Subject:Date:List-Id:List-Unsubscribe:List-Archive:List-Post: List-Help:List-Subscribe:From:Reply-To:From; b=y0H2F/ZwvVK+ronuCw1CugWUEeTtFrIAPHkRE02XnDs2XhbNpyCjUaGPY7mEC3Hx3 HZvw/xfNGBeshFZEaiPYGbOF0m9BrwqFBGEWlucgR9sTCZui2TiOfJpXkr2PFx4aF6 len+YGDyY1LfdBsNKIBnPIGt2fhuC+MjRxP3p4G4= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from EUR05-VI1-obe.outbound.protection.outlook.com (mail-vi1eur05on2089.outbound.protection.outlook.com [40.107.21.89]) by sourceware.org (Postfix) with ESMTPS id 6E1433858D28 for ; Wed, 30 Aug 2023 14:34:32 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 6E1433858D28 Received: from AM6PR10CA0107.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:209:8c::48) by AS8PR08MB6501.eurprd08.prod.outlook.com (2603:10a6:20b:338::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.35; Wed, 30 Aug 2023 14:34:28 +0000 Received: from AM7EUR03FT050.eop-EUR03.prod.protection.outlook.com (2603:10a6:209:8c:cafe::31) by AM6PR10CA0107.outlook.office365.com (2603:10a6:209:8c::48) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6745.20 via Frontend Transport; Wed, 30 Aug 2023 14:34:28 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 63.35.35.123) smtp.mailfrom=arm.com; dkim=pass (signature was verified) header.d=armh.onmicrosoft.com;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 63.35.35.123 as permitted sender) receiver=protection.outlook.com; client-ip=63.35.35.123; helo=64aa7808-outbound-1.mta.getcheckrecipient.com; pr=C Received: from 64aa7808-outbound-1.mta.getcheckrecipient.com (63.35.35.123) by AM7EUR03FT050.mail.protection.outlook.com (100.127.141.27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6745.21 via Frontend Transport; Wed, 30 Aug 2023 14:34:28 +0000 Received: ("Tessian outbound 169aaa6bf2b7:v175"); Wed, 30 Aug 2023 14:34:28 +0000 X-CheckRecipientChecked: true X-CR-MTA-CID: bc284d9f28f71b63 X-CR-MTA-TID: 64aa7808 Received: from 53a98b16413b.1 by 64aa7808-outbound-1.mta.getcheckrecipient.com id 4868AE59-003D-434E-A1A0-7CFE5EAA1884.1; Wed, 30 Aug 2023 14:34:21 +0000 Received: from EUR04-HE1-obe.outbound.protection.outlook.com by 64aa7808-outbound-1.mta.getcheckrecipient.com with ESMTPS id 53a98b16413b.1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384); Wed, 30 Aug 2023 14:34:21 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lLtlNbGlkW6Nbcn2hqWAKIrMo8Lfn4aKmQR3+SWato+XF54QmG5Wy73FGUwgArLmeBO7GtMocTDn1W9ry1cuKQysbXQ08ZHawRAkScKYytEdh5fPwl/bx6QIsjB/QWjfmxVgo5gkkiG50ODPZXWjCgZ6JyepANbhLlVvx4JM1jNQK9Zerh1kgx0X4vJ7f8Y10wcZ9cxKi6xRJalw7debBfH7KIdYGpivWN1eHdC1HV1/5yno6QZkvzXCTp4PggwSkkWgzvjR2B0gGKcM8/Shpd03pJRmPtkm5PTgSf13d+GOP6jnK6BIedhJSOdoTb7KLbLRtk0m3XoWPXXhm+P8ZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=V7aWR6b+4WCf14n5w+wtvWcQ+mJFbJNP9WumbFasx/o=; b=LKv1stpZF/gF2RMpjsThSnSi3eLC4iYZEh3xQt9KUtJ6WpWX/yuSQRCU8fl3RL4vvSrVvbPss4pp+OqcQMrp81+I/9Ef1s+b2PEHgeAFz83YJ2c+f/hbsIrabVNs/Bnsd41OiFYHDw3ru1S6PHvr3G6tQQjRH6ua39Vmls/qyZKInCiXMrOiJF2p8UGZjqSfTNQytuWLE28sX3dhLSZLHsdeSHrp553q3rM31O10CRNa/xokhEhxOTfJL/vYowoR0Csmd04Q+ugDzJEB312YT1xlc/zYgeYU+LtTAWir/rILYQL81kGM+LweSAfqFfru+nVlO1vLqC2XPQ2Tuq+7Hg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 40.67.248.234) smtp.rcpttodomain=sourceware.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=none (message not signed); arc=none Received: from DB3PR08CA0021.eurprd08.prod.outlook.com (2603:10a6:8::34) by AS2PR08MB9570.eurprd08.prod.outlook.com (2603:10a6:20b:60a::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.34; Wed, 30 Aug 2023 14:34:18 +0000 Received: from DBAEUR03FT034.eop-EUR03.prod.protection.outlook.com (2603:10a6:8:0:cafe::3c) by DB3PR08CA0021.outlook.office365.com (2603:10a6:8::34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6699.36 via Frontend Transport; Wed, 30 Aug 2023 14:34:18 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 40.67.248.234) smtp.mailfrom=arm.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=arm.com; Received-SPF: Pass (protection.outlook.com: domain of arm.com designates 40.67.248.234 as permitted sender) receiver=protection.outlook.com; client-ip=40.67.248.234; helo=nebula.arm.com; pr=C Received: from nebula.arm.com (40.67.248.234) by DBAEUR03FT034.mail.protection.outlook.com (100.127.142.97) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.6745.20 via Frontend Transport; Wed, 30 Aug 2023 14:34:18 +0000 Received: from AZ-NEU-EX03.Arm.com (10.251.24.31) by AZ-NEU-EX03.Arm.com (10.251.24.31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Wed, 30 Aug 2023 14:34:17 +0000 Received: from armchair.cambridge.arm.com (10.2.80.71) by mail.arm.com (10.251.24.31) with Microsoft SMTP Server id 15.1.2507.27 via Frontend Transport; Wed, 30 Aug 2023 14:34:17 +0000 To: , Paul Zimmermann , Adhemerval Zanella Netto Subject: [PATCH v4] elf: Fix slow tls access after dlopen [BZ #19924] Date: Wed, 30 Aug 2023 15:34:16 +0100 Message-ID: <20230830143416.3457251-1-szabolcs.nagy@arm.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-EOPAttributedMessage: 1 X-MS-TrafficTypeDiagnostic: DBAEUR03FT034:EE_|AS2PR08MB9570:EE_|AM7EUR03FT050:EE_|AS8PR08MB6501:EE_ X-MS-Office365-Filtering-Correlation-Id: 17570610-3cb3-4106-bb28-08dba9663789 x-checkrecipientrouted: true NoDisclaimer: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam-Untrusted: BCL:0; X-Microsoft-Antispam-Message-Info-Original: 6peQ/XnpiFbghKVbvHAvRN38GCKnZcxBhNVpdcv6zirRgsqhdNQ7wGv2O4QaXy5eJyKMVWGqurWM+mj3E7jQyfrdodogjCBmhpduC9uGjPlm7i4zapqM931PmOAmn0u5a5sKFG/A6SIhW75IrLuxHMjFOvCaXpDBmJ36tKdkHF+frYAZXxzpJvB5i5Xrajbr9PHaEg+U4NICFiOCXdKMzWgpUcX8Vo828sk/PC2yC/RkK8mkd4JqEfUbb4KBuh3OfL1OeL5oNZ/Okr/q4S+JaNwUk9MAILCyQBHL86bIEvtt4iiSDGnfKY5qkDwEtim68rx4l4vh4asUa1qX2HrF51Xo2Z8p7oUbgwmlAxg20cl8upaVAh4F7A78mXYLYHm50GgPqGv1uPz1pXfFHnc5Z9sIgH4CqVVK+uphvN60KL5IPRSYzB0kkzope8uXOafFlq3bzA107l0F/4KcLY5+Ar25vI3pwUIvbnl5T/v1keigAMn4lKiAkMa+iXQ7hgf+yyAnjOld0cJVmwQ+VprkKvoHjU9NaIONdhwRjQXtxS6MNwc939ayPxkCCV+J6mWaaJSYuxlMiVA4TIGQ7X3p1Yh+yMWVl+xOwcZKPu9Wa+p0shxRccvdRkRPwvkRmtn8baVBFzarvwxrbGa2xhRrtMqAFXO4e62IeXTACUGm5ISjKd4B10rinUJeQhx5nm85GK/qEAxQyho+b+Y4ZRQ2nvB3tyuCyjfdRp58NqAKuBjAA+Gdjsd5gxSPC+UCdCpyAK6XC1HVcem1TfnYcxrTtU+0MAzPOC7TY8p3uRhrXaw= X-Forefront-Antispam-Report-Untrusted: CIP:40.67.248.234; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:nebula.arm.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230031)(4636009)(396003)(136003)(376002)(346002)(39860400002)(186009)(82310400011)(1800799009)(451199024)(36840700001)(40470700004)(46966006)(41300700001)(66899024)(336012)(40460700003)(7696005)(2616005)(81166007)(82740400003)(86362001)(47076005)(478600001)(356005)(1076003)(36860700001)(426003)(40480700001)(30864003)(26005)(70586007)(70206006)(110136005)(36756003)(316002)(2906002)(5660300002)(83380400001)(8676002)(8936002)(44832011)(21314003)(36900700001); DIR:OUT; SFP:1101; X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS2PR08MB9570 X-MS-Exchange-Transport-CrossTenantHeadersStripped: AM7EUR03FT050.eop-EUR03.prod.protection.outlook.com X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id-Prvs: e9c2e074-cdb5-466d-9f42-08dba9663185 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: L40sWc8jy5sz2PmrJuFLKOF/8wQDQ+IcuUddarHoRq6XVeTzeOBndyD9zsBxlOif8QxU6jJMZOpC8w78mUeBO+tP1YAeSHRnw828L1zjrmZTnIIDvCjs/3L6DHi8zEvftj1pKPTXSN+FuH390SIbhzIV8QcBnoSnb31tjUlVyg4S47OBO1I/ehDTB22RcjGzBGRpQLM+4L9/YD2TWrdwYoH/Ql5bQhUM5aKUs1n6nwMauHBb43DUhpOG+IO8sTkv2GiyHz3Gch6YIfJelBJrtFP4as16gigVODMxpiZOMvWa6sWy3KuALFTZbhwGQ/1YvJmkPaYsRc8oKdEWkK2UbkJaLFwDG/Tytl7H1zSJSQEPp5GfR//dHJ05GXT3Hggvly1lz8AuuuBml+5jhe2BE8mRdoj8dBHMKjFuPRyIj20gPiF9642kt2XUFxwLLypszPmXbsPXtLwcexrsTCh9Ikcy5e9Yfq4BVihxY9vFJTA6bu8ZqmjXDqSkBy+7GAMQNK4tqW4JXCC4lUYOJFdk9LnjXbFulFVPsPGuUe9mrNaKPwhfB4BYtKOfQq/Xa+b5wlb/ay7HLSx2aig0OH+K5AoT8otlmgHVjOjPNy3C2UVqaXdy6Rmw3Oxom+aHZ7EuuTwcIfYhejSLZixhf384fIlOtwIWBnnGQZL2qv0aBk6CXB0XWDlF+pzSZneHvecdHpfrE2nq8J/oyraHHIxSUGsVBb712iPZefjxtJlgo21/yL3Cmke/upPc/9+5OOEVrNzXgExtyGdasjTqHZ1Riw== X-Forefront-Antispam-Report: CIP:63.35.35.123; CTRY:IE; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:64aa7808-outbound-1.mta.getcheckrecipient.com; PTR:ec2-63-35-35-123.eu-west-1.compute.amazonaws.com; CAT:NONE; SFS:(13230031)(4636009)(376002)(396003)(346002)(136003)(39860400002)(1800799009)(186009)(82310400011)(451199024)(40470700004)(46966006)(36840700001)(81166007)(82740400003)(8936002)(70206006)(70586007)(478600001)(110136005)(7696005)(316002)(41300700001)(66899024)(426003)(40460700003)(5660300002)(8676002)(44832011)(26005)(36756003)(1076003)(2906002)(83380400001)(2616005)(30864003)(86362001)(47076005)(40480700001)(36860700001)(336012)(21314003); DIR:OUT; SFP:1101; X-OriginatorOrg: arm.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Aug 2023 14:34:28.5807 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 17570610-3cb3-4106-bb28-08dba9663789 X-MS-Exchange-CrossTenant-Id: f34e5979-57d9-4aaa-ad4d-b122a662184d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=f34e5979-57d9-4aaa-ad4d-b122a662184d; Ip=[63.35.35.123]; Helo=[64aa7808-outbound-1.mta.getcheckrecipient.com] X-MS-Exchange-CrossTenant-AuthSource: AM7EUR03FT050.eop-EUR03.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR08MB6501 X-Spam-Status: No, score=-11.6 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, FORGED_SPF_HELO, GIT_PATCH_0, KAM_DMARC_NONE, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_PASS, SPF_NONE, TXREP, UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Szabolcs Nagy via Libc-alpha From: Szabolcs Nagy Reply-To: Szabolcs Nagy Errors-To: libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org Sender: "Libc-alpha" In short: __tls_get_addr checks the global generation counter and if the current dtv is older then _dl_update_slotinfo updates dtv up to the generation of the accessed module. So if the global generation is newer than generation of the module then __tls_get_addr keeps hitting the slow dtv update path. The dtv update path includes a number of checks to see if any update is needed and this already causes measurable tls access slow down after dlopen. It may be possible to detect up-to-date dtv faster. But if there are many modules loaded (> TLS_SLOTINFO_SURPLUS) then this requires at least walking the slotinfo list. This patch tries to update the dtv to the global generation instead, so after a dlopen the tls access slow path is only hit once. The modules with larger generation than the accessed one were not necessarily synchronized before, so additional synchronization is needed. This patch uses acquire/release synchronization when accessing the generation counter. Note: in the x86_64 version of dl-tls.c the generation is only loaded once, since relaxed mo is not faster than acquire mo load. I have not benchmarked this. Tested by Adhemerval Zanella on aarch64, powerpc, sparc, x86 who reported that it fixes the performance issue of bug 19924. Reviewed-by: Adhemerval Zanella --- v4: - fix typos and wording in the commit log and concurrency note. - add tested by note to commit log. - add comment in _dl_try_allocate_static_tls. v3: updated concurrency notes. v2: rebased and updated the commit message a bit. still RFC quality. --- elf/dl-close.c | 2 +- elf/dl-open.c | 8 +-- elf/dl-reloc.c | 6 +- elf/dl-tls.c | 117 ++++++++++++++++++++----------------- sysdeps/generic/ldsodefs.h | 3 +- sysdeps/x86_64/dl-tls.c | 4 +- 6 files changed, 74 insertions(+), 66 deletions(-) diff --git a/elf/dl-close.c b/elf/dl-close.c index b887a44888..1c7a861db1 100644 --- a/elf/dl-close.c +++ b/elf/dl-close.c @@ -703,7 +703,7 @@ _dl_close_worker (struct link_map *map, bool force) if (__glibc_unlikely (newgen == 0)) _dl_fatal_printf ("TLS generation counter wrapped! Please report as described in "REPORT_BUGS_TO".\n"); /* Can be read concurrently. */ - atomic_store_relaxed (&GL(dl_tls_generation), newgen); + atomic_store_release (&GL(dl_tls_generation), newgen); if (tls_free_end == GL(dl_tls_static_used)) GL(dl_tls_static_used) = tls_free_start; diff --git a/elf/dl-open.c b/elf/dl-open.c index 2d985e21d8..351931af04 100644 --- a/elf/dl-open.c +++ b/elf/dl-open.c @@ -405,7 +405,7 @@ update_tls_slotinfo (struct link_map *new) _dl_fatal_printf (N_("\ TLS generation counter wrapped! Please report this.")); /* Can be read concurrently. */ - atomic_store_relaxed (&GL(dl_tls_generation), newgen); + atomic_store_release (&GL(dl_tls_generation), newgen); /* We need a second pass for static tls data, because _dl_update_slotinfo must not be run while calls to @@ -422,8 +422,8 @@ TLS generation counter wrapped! Please report this.")); now, but we can delay updating the DTV. */ imap->l_need_tls_init = 0; #ifdef SHARED - /* Update the slot information data for at least the - generation of the DSO we are allocating data for. */ + /* Update the slot information data for the current + generation. */ /* FIXME: This can terminate the process on memory allocation failure. It is not possible to raise @@ -431,7 +431,7 @@ TLS generation counter wrapped! Please report this.")); _dl_update_slotinfo would have to be split into two operations, similar to resize_scopes and update_scopes above. This is related to bug 16134. */ - _dl_update_slotinfo (imap->l_tls_modid); + _dl_update_slotinfo (imap->l_tls_modid, newgen); #endif dl_init_static_tls (imap); diff --git a/elf/dl-reloc.c b/elf/dl-reloc.c index 1d558c1e0c..e5c555d82c 100644 --- a/elf/dl-reloc.c +++ b/elf/dl-reloc.c @@ -112,11 +112,11 @@ _dl_try_allocate_static_tls (struct link_map *map, bool optional) if (map->l_real->l_relocated) { #ifdef SHARED + /* Update the DTV of the current thread. Note: GL(dl_load_tls_lock) + is held here so normal load of the generation counter is valid. */ if (__builtin_expect (THREAD_DTV()[0].counter != GL(dl_tls_generation), 0)) - /* Update the slot information data for at least the generation of - the DSO we are allocating data for. */ - (void) _dl_update_slotinfo (map->l_tls_modid); + (void) _dl_update_slotinfo (map->l_tls_modid, GL(dl_tls_generation)); #endif dl_init_static_tls (map); diff --git a/elf/dl-tls.c b/elf/dl-tls.c index 99b83ca696..c192b5a13a 100644 --- a/elf/dl-tls.c +++ b/elf/dl-tls.c @@ -715,57 +715,57 @@ allocate_and_init (struct link_map *map) struct link_map * -_dl_update_slotinfo (unsigned long int req_modid) +_dl_update_slotinfo (unsigned long int req_modid, size_t new_gen) { struct link_map *the_map = NULL; dtv_t *dtv = THREAD_DTV (); - /* The global dl_tls_dtv_slotinfo array contains for each module - index the generation counter current when the entry was created. + /* CONCURRENCY NOTES: + + The global dl_tls_dtv_slotinfo_list array contains for each module + index the generation counter current when that entry was updated. This array never shrinks so that all module indices which were - valid at some time can be used to access it. Before the first - use of a new module index in this function the array was extended - appropriately. Access also does not have to be guarded against - modifications of the array. It is assumed that pointer-size - values can be read atomically even in SMP environments. It is - possible that other threads at the same time dynamically load - code and therefore add to the slotinfo list. This is a problem - since we must not pick up any information about incomplete work. - The solution to this is to ignore all dtv slots which were - created after the one we are currently interested. We know that - dynamic loading for this module is completed and this is the last - load operation we know finished. */ - unsigned long int idx = req_modid; + valid at some time can be used to access it. Concurrent loading + and unloading of modules can update slotinfo entries or extend + the array. The updates happen under the GL(dl_load_tls_lock) and + finish with the release store of the generation counter to + GL(dl_tls_generation) which is synchronized with the load of + new_gen in the caller. So updates up to new_gen are synchronized + but updates for later generations may not be. + + Here we update the thread dtv from old_gen (== dtv[0].counter) to + new_gen generation. For this, each dtv[i] entry is either set to + an unallocated state (set), or left unmodified (nop). Where (set) + may resize the dtv first if modid i >= dtv[-1].counter. The rules + for the decision between (set) and (nop) are + + (1) If slotinfo entry i is concurrently updated then either (set) + or (nop) is valid: TLS access cannot use dtv[i] unless it is + synchronized with a generation > new_gen. + + Otherwise, if the generation of slotinfo entry i is gen and the + loaded module for this entry is map then + + (2) If gen <= old_gen then do (nop). + + (3) If old_gen < gen <= new_gen then + (3.1) if map != 0 then (set) + (3.2) if map == 0 then either (set) or (nop). + + Note that (1) cannot be reliably detected, but since both actions + are valid it does not have to be. Only (2) and (3.1) cases need + to be distinguished for which relaxed mo access of gen and map is + enough: their value is synchronized when it matters. + + Note that a relaxed mo load may give an out-of-thin-air value since + it is used in decisions that can affect concurrent stores. But this + should only happen if the OOTA value causes UB that justifies the + concurrent store of the value. This is not expected to be an issue + in practice. */ struct dtv_slotinfo_list *listp = GL(dl_tls_dtv_slotinfo_list); - while (idx >= listp->len) + if (dtv[0].counter < new_gen) { - idx -= listp->len; - listp = listp->next; - } - - if (dtv[0].counter < listp->slotinfo[idx].gen) - { - /* CONCURRENCY NOTES: - - Here the dtv needs to be updated to new_gen generation count. - - This code may be called during TLS access when GL(dl_load_tls_lock) - is not held. In that case the user code has to synchronize with - dlopen and dlclose calls of relevant modules. A module m is - relevant if the generation of m <= new_gen and dlclose of m is - synchronized: a memory access here happens after the dlopen and - before the dlclose of relevant modules. The dtv entries for - relevant modules need to be updated, other entries can be - arbitrary. - - This e.g. means that the first part of the slotinfo list can be - accessed race free, but the tail may be concurrently extended. - Similarly relevant slotinfo entries can be read race free, but - other entries are racy. However updating a non-relevant dtv - entry does not affect correctness. For a relevant module m, - max_modid >= modid of m. */ - size_t new_gen = listp->slotinfo[idx].gen; size_t total = 0; size_t max_modid = atomic_load_relaxed (&GL(dl_tls_max_dtv_idx)); assert (max_modid >= req_modid); @@ -778,31 +778,33 @@ _dl_update_slotinfo (unsigned long int req_modid) { size_t modid = total + cnt; - /* Later entries are not relevant. */ + /* Case (1) for all later modids. */ if (modid > max_modid) break; size_t gen = atomic_load_relaxed (&listp->slotinfo[cnt].gen); + /* Case (1). */ if (gen > new_gen) - /* Not relevant. */ continue; - /* If the entry is older than the current dtv layout we - know we don't have to handle it. */ + /* Case (2) or (1). */ if (gen <= dtv[0].counter) continue; + /* Case (3) or (1). */ + /* If there is no map this means the entry is empty. */ struct link_map *map = atomic_load_relaxed (&listp->slotinfo[cnt].map); /* Check whether the current dtv array is large enough. */ if (dtv[-1].counter < modid) { + /* Case (3.2) or (1). */ if (map == NULL) continue; - /* Resize the dtv. */ + /* Resizing the dtv aborts on failure: bug 16134. */ dtv = _dl_resize_dtv (dtv, max_modid); assert (modid <= dtv[-1].counter); @@ -813,7 +815,7 @@ _dl_update_slotinfo (unsigned long int req_modid) } /* If there is currently memory allocate for this - dtv entry free it. */ + dtv entry free it. Note: this is not AS-safe. */ /* XXX Ideally we will at some point create a memory pool. */ free (dtv[modid].pointer.to_free); @@ -908,9 +910,9 @@ tls_get_addr_tail (GET_ADDR_ARGS, dtv_t *dtv, struct link_map *the_map) static struct link_map * __attribute_noinline__ -update_get_addr (GET_ADDR_ARGS) +update_get_addr (GET_ADDR_ARGS, size_t gen) { - struct link_map *the_map = _dl_update_slotinfo (GET_ADDR_MODULE); + struct link_map *the_map = _dl_update_slotinfo (GET_ADDR_MODULE, gen); dtv_t *dtv = THREAD_DTV (); void *p = dtv[GET_ADDR_MODULE].pointer.val; @@ -940,12 +942,17 @@ __tls_get_addr (GET_ADDR_ARGS) dtv_t *dtv = THREAD_DTV (); /* Update is needed if dtv[0].counter < the generation of the accessed - module. The global generation counter is used here as it is easier - to check. Synchronization for the relaxed MO access is guaranteed - by user code, see CONCURRENCY NOTES in _dl_update_slotinfo. */ + module, but the global generation counter is easier to check (which + must be synchronized up to the generation of the accessed module by + user code doing the TLS access so relaxed mo read is enough). */ size_t gen = atomic_load_relaxed (&GL(dl_tls_generation)); if (__glibc_unlikely (dtv[0].counter != gen)) - return update_get_addr (GET_ADDR_PARAM); + { + /* Update DTV up to the global generation, see CONCURRENCY NOTES + in _dl_update_slotinfo. */ + gen = atomic_load_acquire (&GL(dl_tls_generation)); + return update_get_addr (GET_ADDR_PARAM, gen); + } void *p = dtv[GET_ADDR_MODULE].pointer.val; diff --git a/sysdeps/generic/ldsodefs.h b/sysdeps/generic/ldsodefs.h index e8b7359b04..ed69c6babd 100644 --- a/sysdeps/generic/ldsodefs.h +++ b/sysdeps/generic/ldsodefs.h @@ -1251,7 +1251,8 @@ extern void _dl_add_to_slotinfo (struct link_map *l, bool do_add) /* Update slot information data for at least the generation of the module with the given index. */ -extern struct link_map *_dl_update_slotinfo (unsigned long int req_modid) +extern struct link_map *_dl_update_slotinfo (unsigned long int req_modid, + size_t gen) attribute_hidden; /* Look up the module's TLS block as for __tls_get_addr, diff --git a/sysdeps/x86_64/dl-tls.c b/sysdeps/x86_64/dl-tls.c index 7a7fe38625..e9b6ab9970 100644 --- a/sysdeps/x86_64/dl-tls.c +++ b/sysdeps/x86_64/dl-tls.c @@ -40,9 +40,9 @@ __tls_get_addr_slow (GET_ADDR_ARGS) { dtv_t *dtv = THREAD_DTV (); - size_t gen = atomic_load_relaxed (&GL(dl_tls_generation)); + size_t gen = atomic_load_acquire (&GL(dl_tls_generation)); if (__glibc_unlikely (dtv[0].counter != gen)) - return update_get_addr (GET_ADDR_PARAM); + return update_get_addr (GET_ADDR_PARAM, gen); return tls_get_addr_tail (GET_ADDR_PARAM, dtv, NULL); }