From patchwork Wed Jun 12 11:42:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Stubbs X-Patchwork-Id: 1946813 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=baylibre-com.20230601.gappssmtp.com header.i=@baylibre-com.20230601.gappssmtp.com header.a=rsa-sha256 header.s=20230601 header.b=yz1jGgJr; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VzkF70Z4jz20Wd for ; Wed, 12 Jun 2024 21:43:07 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 42162388207D for ; Wed, 12 Jun 2024 11:43:05 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail-lj1-x22c.google.com (mail-lj1-x22c.google.com [IPv6:2a00:1450:4864:20::22c]) by sourceware.org (Postfix) with ESMTPS id 6A4413877015 for ; Wed, 12 Jun 2024 11:42:34 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 6A4413877015 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=baylibre.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=baylibre.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 6A4413877015 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2a00:1450:4864:20::22c ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1718192557; cv=none; b=cVvMCU7hai3iUXFkSPxBrn/h74DZ0yxuy4jjK0MPtai79KFVILncc56S9/E+D8DMdgKHywjJT780KNLIPbzGATmhcYt1AKxFCaTe7U/88r3vETjQgJfdOfUFphxFsFRtBy7rdtrv3ZsIwvFL9OqYxz/n3lzExrbKsXAdnF0rXWU= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1718192557; c=relaxed/simple; bh=qDiEON0RvuMgfQqNsBnOwJdxSYEgIjXaQpbs54jzC4U=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=G+ijhlRKzNYrLHPxvnlRFfTF3t3lF5rr11EbGTvf5zgk3XjVIbDUUx4PHhgUaEAG0uxwsiMALJ0usmlajNKEh6MDmh9REFeP+bW+A8dlbB0jjvB8viHXwqEbbmGtsAcq6OoQmsGVq2f7gq/apA2V182g48tlReC/5q6JWEsHZew= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-lj1-x22c.google.com with SMTP id 38308e7fff4ca-2eabd22d3f4so80225701fa.1 for ; Wed, 12 Jun 2024 04:42:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1718192552; x=1718797352; darn=gcc.gnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4zClnyshK7E6HIt6ufmul2gNpuLyX8SjCBAFt48GZPA=; b=yz1jGgJr4IuuTKflorWuw4ialmqPwVtSkyQmhz4y0oiM9Cz//ZSBgvvtgcPT4ognhi Bvk57sPfgtcouzZVBHFdesohvuSpo9AXPzc8RR7hzihCun3LlN5TFQQTMQNIxX9JsahQ RqufQf+U+foX3vH3z5pa+zOPkz+lOgA8bGN4PWbQftcI/ibba9NKJ27VFZd2r8rslhd+ mjFDslLk+TBkdb6gbZqmd9zhpDOLXVj7OnUlDz03VnEsy1xhb7H4PCvbQtVLyxRvxn9/ wV0t7NM9BrXuOcaUc6as5Lb5P7ZBxqSio3yjb1FqdjvCofsk0CfxvgFFVdhrHpe6J9xw LWjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718192552; x=1718797352; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4zClnyshK7E6HIt6ufmul2gNpuLyX8SjCBAFt48GZPA=; b=DTWrIu47TBW14ZWcSdTDCZ0cUQzTvNC4a0wIdTFyOil/apc5wSlPM7KB4pqKlC7aKA ZreX99TYsrBdHxR4KOjy607OJSbmV9YLXWx+ByPs91gUypV6OuhN/4eopmIPKgIXeWwS p0VOsS1xw7SKkZSWDDPy3shOkCyABUvPCoWPFsBmsNcMhZCNtDitTQOi5W2dmaTwJz0A ARCSw2BQ/qSq3fdUUcR0qzStSQ7C46csoilj0wKn9D9QN6lWZ/f2DL5AXxe1SGzvGW56 8pEDPySn1GVpPCIZEXdOnI2m2hnqeZpYzFwE+ZtNWTpxmbvOkrmerE04dH99obOFAXPX zTUA== X-Gm-Message-State: AOJu0YwVC8Yjb6w0in9VsHrGXyEQDJGnSb7Z7ADbzNbIU64g71bc+xuW fRhYOclJ2gfS66+q2ROIBbM4+o2iBL+C0W1YgrFvCywPAN8eYMGfDpKjUK/sEV5iMaKMARsnSGm i7aQ= X-Google-Smtp-Source: AGHT+IHUXelC+QYyl4HKFKuUi6wrM+GLGJZQollqr2dN7+rFby40bg9NbR5aZ6z80NoyYFSqQ9OylQ== X-Received: by 2002:a2e:a310:0:b0:2eb:d825:e720 with SMTP id 38308e7fff4ca-2ebfc9295bbmr10370811fa.22.1718192551888; Wed, 12 Jun 2024 04:42:31 -0700 (PDT) Received: from arnold.baylibre (laubervilliers-658-1-213-31.w90-63.abo.wanadoo.fr. [90.63.244.31]) by smtp.googlemail.com with ESMTPSA id 5b1f17b1804b1-42286fe75ebsm23366235e9.9.2024.06.12.04.42.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Jun 2024 04:42:31 -0700 (PDT) From: Andrew Stubbs To: gcc-patches@gcc.gnu.org Cc: tburnus@baylibre.com, jakub@redhat.com Subject: [PATCH v5 1/6] libgomp: change alloc-pinned tests failure mode Date: Wed, 12 Jun 2024 11:42:18 +0000 Message-ID: <20240612114223.2206735-2-ams@baylibre.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20240612114223.2206735-1-ams@baylibre.com> References: <20240612114223.2206735-1-ams@baylibre.com> MIME-Version: 1.0 X-Spam-Status: No, score=-10.2 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org The feature doesn't work on non-Linux hosts, at present, so skip the tests entirely. On Linux systems that have insufficient lockable memory configured we still need to fail or else the feature won't be getting tested when we think it is, but now there's a message to explain why. libgomp/ChangeLog: * testsuite/libgomp.c/alloc-pinned-1.c: Change dg-xfail-run-if to dg-skip-if. Correct spelling mistake. Abort on insufficient lockable memory. Use #error on non-linux hosts. * testsuite/libgomp.c/alloc-pinned-2.c: Likewise. --- libgomp/testsuite/libgomp.c/alloc-pinned-1.c | 20 ++++++-------------- libgomp/testsuite/libgomp.c/alloc-pinned-2.c | 20 ++++++-------------- 2 files changed, 12 insertions(+), 28 deletions(-) diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-1.c b/libgomp/testsuite/libgomp.c/alloc-pinned-1.c index 4185accf2e6..672f2453a78 100644 --- a/libgomp/testsuite/libgomp.c/alloc-pinned-1.c +++ b/libgomp/testsuite/libgomp.c/alloc-pinned-1.c @@ -1,6 +1,6 @@ /* { dg-do run } */ -/* { dg-xfail-run-if "Pinning not implemented on this host" { ! *-*-linux-gnu } } */ +/* { dg-skip-if "Pinning not implemented on this host" { ! *-*-linux-gnu* } } */ /* Test that pinned memory works. */ @@ -19,7 +19,10 @@ struct rlimit limit; \ if (getrlimit (RLIMIT_MEMLOCK, &limit) \ || limit.rlim_cur <= SIZE) \ - fprintf (stderr, "unsufficient lockable memory; please increase ulimit\n"); \ + { \ + fprintf (stderr, "insufficient lockable memory; please increase ulimit\n"); \ + abort (); \ + } \ } int @@ -44,18 +47,7 @@ get_pinned_mem () abort (); } #else -#define PAGE_SIZE 1024 /* unknown */ -#define CHECK_SIZE(SIZE) { \ - fprintf (stderr, "OS unsupported\n"); \ - abort (); \ - } -#define EXPECT_OMP_NULL_ALLOCATOR - -int -get_pinned_mem () -{ - return 0; -} +#error "OS unsupported" #endif static void diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-2.c b/libgomp/testsuite/libgomp.c/alloc-pinned-2.c index 0b9c11d0315..b6d1d83fb6f 100644 --- a/libgomp/testsuite/libgomp.c/alloc-pinned-2.c +++ b/libgomp/testsuite/libgomp.c/alloc-pinned-2.c @@ -1,6 +1,6 @@ /* { dg-do run } */ -/* { dg-xfail-run-if "Pinning not implemented on this host" { ! *-*-linux-gnu } } */ +/* { dg-skip-if "Pinning not implemented on this host" { ! *-*-linux-gnu* } } */ /* Test that pinned memory works (pool_size code path). */ @@ -19,7 +19,10 @@ struct rlimit limit; \ if (getrlimit (RLIMIT_MEMLOCK, &limit) \ || limit.rlim_cur <= SIZE) \ - fprintf (stderr, "unsufficient lockable memory; please increase ulimit\n"); \ + { \ + fprintf (stderr, "insufficient lockable memory; please increase ulimit\n"); \ + abort (); \ + } \ } int @@ -44,18 +47,7 @@ get_pinned_mem () abort (); } #else -#define PAGE_SIZE 1024 /* unknown */ -#define CHECK_SIZE(SIZE) { \ - fprintf (stderr, "OS unsupported\n"); \ - abort (); \ - } -#define EXPECT_OMP_NULL_ALLOCATOR - -int -get_pinned_mem () -{ - return 0; -} +#error "OS unsupported" #endif static void From patchwork Wed Jun 12 11:42:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Stubbs X-Patchwork-Id: 1946816 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=baylibre-com.20230601.gappssmtp.com header.i=@baylibre-com.20230601.gappssmtp.com header.a=rsa-sha256 header.s=20230601 header.b=mutQOY+e; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VzkFH3Mdjz20Wd for ; Wed, 12 Jun 2024 21:43:15 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id B3DBE3882121 for ; Wed, 12 Jun 2024 11:43:13 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail-lj1-x235.google.com (mail-lj1-x235.google.com [IPv6:2a00:1450:4864:20::235]) by sourceware.org (Postfix) with ESMTPS id DA2233877020 for ; Wed, 12 Jun 2024 11:42:34 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org DA2233877020 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=baylibre.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=baylibre.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org DA2233877020 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2a00:1450:4864:20::235 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1718192560; cv=none; b=EbKNen1MXyIOVDW4dIH5COSuPjTaseUGuY5J/AaUtKOtOG4jTQ/s7F0I029OYyMwZ8xO+G6arF7djIa1s7kykVt7EamMaaIotf7d9te4jaEiZSbx1zSDv97CboHRmx0eeWMmAgklb2laIl5D+A/HXhay4fXzkSsQpjT3M5HD73Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1718192560; c=relaxed/simple; bh=uKNkOHEZGbeDxrZl9LWD43i9vyxBU/XBC/N1dLLdrT0=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=eUtCZKgKSPg23yJzMmaolvrpUVHgYs6q7VNlXefWUhk6eBk9fzmqBm3gEPWMeHTAGNFaGmSej2qgn4boa7BvLgFSC9p6z/ylC7w/v+JXCz1/CbtTX3qCyQjngCPAFSlmwwx2Yx8TA156K6vh0AA+/AhXaD+KfzkSChzK0izW2x8= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-lj1-x235.google.com with SMTP id 38308e7fff4ca-2eaae2a6dc1so131611751fa.0 for ; Wed, 12 Jun 2024 04:42:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1718192553; x=1718797353; darn=gcc.gnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=mCU0gywpZAF7zSTURtjrBz0orj7JXmdSacyc0teyb7o=; b=mutQOY+eyojXPoWhG9QRCqogiUZ7mRkh3MFvsa7ZEx41IF1Bu4AeBLLZ3RlKQIQ3ON 46RFqfijKkrDTrdXxPcJGGi+BD0USOVh5cAFIvCM8k7uU4fr6Dchc4VCp+WGioU/gt3C 0DR/6gF+cmuskakidH0kZqTGK12T80tZ9hYB7FeHIf0O2+isD5Mir5vSxm+OewI2Z+6s 83iUaHHCDxWBO2Va+A828k03iwMgwR/NZIz017wVXvbp2wdU5TAWsP92UgzlaHDi2b/5 gikHWPn5rdHb6LmSVv0xKlcGUUs6xKxK4J59v7BwagRSxgEcdmACQwMHSiOBqjCaZ31O WSpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718192553; x=1718797353; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mCU0gywpZAF7zSTURtjrBz0orj7JXmdSacyc0teyb7o=; b=nweZn54KVGWWwzKuTAK9erd0v7ZgN+O5d9/5VCkVnYL9nORfaPFp9+FkRsGZQEGxcR 7YwcycMyF25wEGzEqp5Wo+U7v+12Yw3D0iQZQeazUzJTPEmpFtpIOuPh4CNO/tsz+a6S vbU5K3E9Lw0d4B3BnUYXCXiMcG3c0rzxaV9fvrKbJKXq49cCoBvxf4IjoMtZyfX5hInb a/kHm3f0/3sxKJ3esKZTbCu18SZ4WTBUf91JKdTbUNVKflpRkhzs5zrPSaYiWgIKOVJl D9mlv+iJVPo4xAJid9E9MB3Fa1aba+rgqPM5LoPuon3j4mv/+xa4Peevui35aPxItv3v 7ZFg== X-Gm-Message-State: AOJu0Yw9ZkwRYv4MiRet7BnnxI+u4oPZdKUrjgCp7AghArXq/pLeGkkF V1/yYe5V1Mag8lLygkMcf5AphyQ4BOI/acqa4hdqW0kR5uQ4AP6QaJV/7aF/ERpwYKtdhQcarE+ ilXw= X-Google-Smtp-Source: AGHT+IE3e72gkh1wLfBseMpr2qUzs15y8PrYWkjswWkdOSttVeiwIACSLHo1Y5dJGp3/jJcr08BhKg== X-Received: by 2002:a2e:9ac4:0:b0:2ec:fb:b464 with SMTP id 38308e7fff4ca-2ec00fbb629mr1001091fa.1.1718192552569; Wed, 12 Jun 2024 04:42:32 -0700 (PDT) Received: from arnold.baylibre (laubervilliers-658-1-213-31.w90-63.abo.wanadoo.fr. [90.63.244.31]) by smtp.googlemail.com with ESMTPSA id 5b1f17b1804b1-42286fe75ebsm23366235e9.9.2024.06.12.04.42.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Jun 2024 04:42:32 -0700 (PDT) From: Andrew Stubbs To: gcc-patches@gcc.gnu.org Cc: tburnus@baylibre.com, jakub@redhat.com Subject: [PATCH v5 2/6] libgomp, openmp: Add ompx_gnu_pinned_mem_alloc Date: Wed, 12 Jun 2024 11:42:19 +0000 Message-ID: <20240612114223.2206735-3-ams@baylibre.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20240612114223.2206735-1-ams@baylibre.com> References: <20240612114223.2206735-1-ams@baylibre.com> MIME-Version: 1.0 X-Spam-Status: No, score=-10.4 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org Compared to the previous v4 (1/5) posting of this patch: - The enumeration of the ompx allocators have been moved (again) to 200 (as 100 is already in use by another toolchain vendor and this seems like a possible source of confusion). - The "ompx" has also been changed to "ompx_gnu" to highlight that these are specifically GNU extensions. - The failure mode of the testcases had been modified, including adding an abort in CHECK_SIZE and skipping the test on unsupported platforms. - The OMP_ALLOCATE environment variable now supports the new allocator. - The Fortran frontend allows use of the new allocator in "allocator" clauses. --- This creates a new predefined allocator as a shortcut for using pinned memory with OpenMP. This is not in the OpenMP standard so it uses the "ompx" namespace and an independent enum baseline of 200 (selected to not clash with other known implementations). The allocator is equivalent to using a custom allocator with the pinned trait and the null fallback trait. One motivation for having this feature is for use by the (planned) -foffload-memory=pinned feature. gcc/fortran/ChangeLog: * openmp.cc (is_predefined_allocator): Update valid ranges to incorporate ompx_gnu_pinned_mem_alloc. libgomp/ChangeLog: * allocator.c (ompx_gnu_min_predefined_alloc): New. (ompx_gnu_max_predefined_alloc): New. (predefined_alloc_mapping): Rename to ... (predefined_omp_alloc_mapping): ... this. (predefined_ompx_gnu_alloc_mapping): New. (_Static_assert): Adjust for the new name, and add a new assert for the new table. (predefined_allocator_p): New. (predefined_alloc_mapping): New. (omp_aligned_alloc): Support ompx_gnu_pinned_mem_alloc. Use predefined_allocator_p and predefined_alloc_mapping. (omp_free): Likewise. (omp_alligned_calloc): Likewise. (omp_realloc): Likewise. * env.c (parse_allocator): Add ompx_gnu_pinned_mem_alloc. * libgomp.texi: Document ompx_gnu_pinned_mem_alloc. * omp.h.in (omp_allocator_handle_t): Add ompx_gnu_pinned_mem_alloc. * omp_lib.f90.in: Add ompx_gnu_pinned_mem_alloc. * omp_lib.h.in: Add ompx_gnu_pinned_mem_alloc. * testsuite/libgomp.c/alloc-pinned-5.c: New test. * testsuite/libgomp.c/alloc-pinned-6.c: New test. * testsuite/libgomp.fortran/alloc-pinned-1.f90: New test. gcc/testsuite/ChangeLog: * gfortran.dg/gomp/allocate-pinned-1.f90: New test. Co-Authored-By: Thomas Schwinge --- gcc/fortran/openmp.cc | 11 +- .../gfortran.dg/gomp/allocate-pinned-1.f90 | 16 +++ libgomp/allocator.c | 115 +++++++++++++----- libgomp/env.c | 1 + libgomp/libgomp.texi | 7 +- libgomp/omp.h.in | 1 + libgomp/omp_lib.f90.in | 2 + libgomp/omp_lib.h.in | 2 + libgomp/testsuite/libgomp.c/alloc-pinned-5.c | 100 +++++++++++++++ libgomp/testsuite/libgomp.c/alloc-pinned-6.c | 102 ++++++++++++++++ .../libgomp.fortran/alloc-pinned-1.f90 | 16 +++ 11 files changed, 336 insertions(+), 37 deletions(-) create mode 100644 gcc/testsuite/gfortran.dg/gomp/allocate-pinned-1.f90 create mode 100644 libgomp/testsuite/libgomp.c/alloc-pinned-5.c create mode 100644 libgomp/testsuite/libgomp.c/alloc-pinned-6.c create mode 100644 libgomp/testsuite/libgomp.fortran/alloc-pinned-1.f90 diff --git a/gcc/fortran/openmp.cc b/gcc/fortran/openmp.cc index 5246647e6f8..a177afb4974 100644 --- a/gcc/fortran/openmp.cc +++ b/gcc/fortran/openmp.cc @@ -7352,8 +7352,9 @@ resolve_omp_udr_clause (gfc_omp_namelist *n, gfc_namespace *ns, } /* Assume that a constant expression in the range 1 (omp_default_mem_alloc) - to 8 (omp_thread_mem_alloc) range is fine. The original symbol name is - already lost during matching via gfc_match_expr. */ + to 8 (omp_thread_mem_alloc) range, or 200 (ompx_gnu_pinned_mem_alloc) is + fine. The original symbol name is already lost during matching via + gfc_match_expr. */ static bool is_predefined_allocator (gfc_expr *expr) { @@ -7362,8 +7363,10 @@ is_predefined_allocator (gfc_expr *expr) && expr->ts.type == BT_INTEGER && expr->ts.kind == gfc_c_intptr_kind && expr->expr_type == EXPR_CONSTANT - && mpz_sgn (expr->value.integer) > 0 - && mpz_cmp_si (expr->value.integer, 8) <= 0); + && ((mpz_sgn (expr->value.integer) > 0 + && mpz_cmp_si (expr->value.integer, 8) <= 0) + || (mpz_cmp_si (expr->value.integer, 200) >= 0 + && mpz_cmp_si (expr->value.integer, 200) <= 0))); } /* Resolve declarative ALLOCATE statement. Note: Common block vars only appear diff --git a/gcc/testsuite/gfortran.dg/gomp/allocate-pinned-1.f90 b/gcc/testsuite/gfortran.dg/gomp/allocate-pinned-1.f90 new file mode 100644 index 00000000000..0e6619b7853 --- /dev/null +++ b/gcc/testsuite/gfortran.dg/gomp/allocate-pinned-1.f90 @@ -0,0 +1,16 @@ +! Test that the ompx_gnu_pinned_mem_alloc is accepted by the parser + +module m +use iso_c_binding +integer, parameter :: omp_allocator_handle_kind = c_intptr_t + integer (kind=omp_allocator_handle_kind), & + parameter :: ompx_gnu_pinned_mem_alloc = 200 +end + +subroutine f () + use m + implicit none + ! The "Sorry" is here temporarily only to avoid excess error failures. + integer, save :: i ! { dg-error "Sorry, !.OMP allocate for variable 'i' at .1. with SAVE attribute not yet implemented" } + !$omp allocate(i) allocator(ompx_gnu_pinned_mem_alloc) +end diff --git a/libgomp/allocator.c b/libgomp/allocator.c index cdedc7d80e9..91aa58e162b 100644 --- a/libgomp/allocator.c +++ b/libgomp/allocator.c @@ -99,6 +99,8 @@ GOMP_is_alloc (void *ptr) #define omp_max_predefined_alloc omp_thread_mem_alloc +#define ompx_gnu_min_predefined_alloc ompx_gnu_pinned_mem_alloc +#define ompx_gnu_max_predefined_alloc ompx_gnu_pinned_mem_alloc /* These macros may be overridden in config//allocator.c. The defaults (no override) are to return NULL for pinned memory requests @@ -131,7 +133,7 @@ GOMP_is_alloc (void *ptr) The index to this table is the omp_allocator_handle_t enum value. When the user calls omp_alloc with a predefined allocator this table determines what memory they get. */ -static const omp_memspace_handle_t predefined_alloc_mapping[] = { +static const omp_memspace_handle_t predefined_omp_alloc_mapping[] = { omp_default_mem_space, /* omp_null_allocator doesn't actually use this. */ omp_default_mem_space, /* omp_default_mem_alloc. */ omp_large_cap_mem_space, /* omp_large_cap_mem_alloc. */ @@ -142,11 +144,41 @@ static const omp_memspace_handle_t predefined_alloc_mapping[] = { omp_low_lat_mem_space, /* omp_pteam_mem_alloc (implementation defined). */ omp_low_lat_mem_space, /* omp_thread_mem_alloc (implementation defined). */ }; +static const omp_memspace_handle_t predefined_ompx_gnu_alloc_mapping[] = { + omp_default_mem_space, /* ompx_gnu_pinned_mem_alloc. */ +}; #define ARRAY_SIZE(A) (sizeof (A) / sizeof ((A)[0])) -_Static_assert (ARRAY_SIZE (predefined_alloc_mapping) +_Static_assert (ARRAY_SIZE (predefined_omp_alloc_mapping) == omp_max_predefined_alloc + 1, - "predefined_alloc_mapping must match omp_memspace_handle_t"); + "predefined_omp_alloc_mapping must match omp_memspace_handle_t"); +_Static_assert (ARRAY_SIZE (predefined_ompx_gnu_alloc_mapping) + == (ompx_gnu_max_predefined_alloc + - ompx_gnu_min_predefined_alloc) + 1, + "predefined_ompx_gnu_alloc_mapping must match" + " omp_memspace_handle_t"); + +static inline bool +predefined_allocator_p (omp_allocator_handle_t allocator) +{ + return allocator <= ompx_gnu_max_predefined_alloc; +} + +static inline omp_memspace_handle_t +predefined_alloc_mapping (omp_allocator_handle_t allocator) +{ + if (allocator <= omp_max_predefined_alloc) + return predefined_omp_alloc_mapping[allocator]; + else if (allocator >= ompx_gnu_min_predefined_alloc + && allocator <= ompx_gnu_max_predefined_alloc) + { + int index = allocator - ompx_gnu_min_predefined_alloc; + return predefined_ompx_gnu_alloc_mapping[index]; + } + else + /* This should never happen. */ + return omp_default_mem_space; +} enum gomp_numa_memkind_kind { @@ -556,7 +588,7 @@ retry: allocator = (omp_allocator_handle_t) thr->ts.def_allocator; } - if (allocator > omp_max_predefined_alloc) + if (!predefined_allocator_p (allocator)) { allocator_data = (struct omp_allocator_data *) allocator; if (new_alignment < allocator_data->alignment) @@ -685,9 +717,11 @@ retry: omp_memspace_handle_t memspace; memspace = (allocator_data ? allocator_data->memspace - : predefined_alloc_mapping[allocator]); - ptr = MEMSPACE_ALLOC (memspace, new_size, - allocator_data && allocator_data->pinned); + : predefined_alloc_mapping (allocator)); + int pinned = (allocator_data + ? allocator_data->pinned + : allocator == ompx_gnu_pinned_mem_alloc); + ptr = MEMSPACE_ALLOC (memspace, new_size, pinned); } if (ptr == NULL) goto fail; @@ -708,7 +742,8 @@ retry: fail:; int fallback = (allocator_data ? allocator_data->fallback - : allocator == omp_default_mem_alloc + : (allocator == omp_default_mem_alloc + || allocator == ompx_gnu_pinned_mem_alloc) ? omp_atv_null_fb : omp_atv_default_mem_fb); switch (fallback) @@ -764,7 +799,7 @@ omp_free (void *ptr, omp_allocator_handle_t allocator) return; (void) allocator; data = &((struct omp_mem_header *) ptr)[-1]; - if (data->allocator > omp_max_predefined_alloc) + if (!predefined_allocator_p (data->allocator)) { struct omp_allocator_data *allocator_data = (struct omp_allocator_data *) (data->allocator); @@ -822,7 +857,8 @@ omp_free (void *ptr, omp_allocator_handle_t allocator) } #endif - memspace = predefined_alloc_mapping[data->allocator]; + memspace = predefined_alloc_mapping (data->allocator); + pinned = (data->allocator == ompx_gnu_pinned_mem_alloc); } MEMSPACE_FREE (memspace, data->ptr, data->size, pinned); @@ -860,7 +896,7 @@ retry: allocator = (omp_allocator_handle_t) thr->ts.def_allocator; } - if (allocator > omp_max_predefined_alloc) + if (!predefined_allocator_p (allocator)) { allocator_data = (struct omp_allocator_data *) allocator; if (new_alignment < allocator_data->alignment) @@ -995,9 +1031,11 @@ retry: omp_memspace_handle_t memspace; memspace = (allocator_data ? allocator_data->memspace - : predefined_alloc_mapping[allocator]); - ptr = MEMSPACE_CALLOC (memspace, new_size, - allocator_data && allocator_data->pinned); + : predefined_alloc_mapping (allocator)); + int pinned = (allocator_data + ? allocator_data->pinned + : allocator == ompx_gnu_pinned_mem_alloc); + ptr = MEMSPACE_CALLOC (memspace, new_size, pinned); } if (ptr == NULL) goto fail; @@ -1018,7 +1056,8 @@ retry: fail:; int fallback = (allocator_data ? allocator_data->fallback - : allocator == omp_default_mem_alloc + : (allocator == omp_default_mem_alloc + || allocator == ompx_gnu_pinned_mem_alloc) ? omp_atv_null_fb : omp_atv_default_mem_fb); switch (fallback) @@ -1076,7 +1115,7 @@ retry: if (allocator == omp_null_allocator) allocator = free_allocator; - if (allocator > omp_max_predefined_alloc) + if (!predefined_allocator_p (allocator)) { allocator_data = (struct omp_allocator_data *) allocator; if (new_alignment < allocator_data->alignment) @@ -1104,7 +1143,7 @@ retry: } #endif } - if (free_allocator > omp_max_predefined_alloc) + if (!predefined_allocator_p (free_allocator)) { free_allocator_data = (struct omp_allocator_data *) free_allocator; #if defined(LIBGOMP_USE_MEMKIND) || defined(LIBGOMP_USE_LIBNUMA) @@ -1228,11 +1267,14 @@ retry: else #endif if (prev_size) - new_ptr = MEMSPACE_REALLOC (allocator_data->memspace, data->ptr, - data->size, new_size, - (free_allocator_data - && free_allocator_data->pinned), - allocator_data->pinned); + { + int was_pinned = (free_allocator_data + ? free_allocator_data->pinned + : free_allocator == ompx_gnu_pinned_mem_alloc); + new_ptr = MEMSPACE_REALLOC (allocator_data->memspace, data->ptr, + data->size, new_size, was_pinned, + allocator_data->pinned); + } else new_ptr = MEMSPACE_ALLOC (allocator_data->memspace, new_size, allocator_data->pinned); @@ -1287,11 +1329,15 @@ retry: omp_memspace_handle_t memspace; memspace = (allocator_data ? allocator_data->memspace - : predefined_alloc_mapping[allocator]); + : predefined_alloc_mapping (allocator)); + int was_pinned = (free_allocator_data + ? free_allocator_data->pinned + : free_allocator == ompx_gnu_pinned_mem_alloc); + int pinned = (allocator_data + ? allocator_data->pinned + : allocator == ompx_gnu_pinned_mem_alloc); new_ptr = MEMSPACE_REALLOC (memspace, data->ptr, data->size, new_size, - (free_allocator_data - && free_allocator_data->pinned), - allocator_data && allocator_data->pinned); + was_pinned, pinned); } if (new_ptr == NULL) goto fail; @@ -1324,9 +1370,11 @@ retry: omp_memspace_handle_t memspace; memspace = (allocator_data ? allocator_data->memspace - : predefined_alloc_mapping[allocator]); - new_ptr = MEMSPACE_ALLOC (memspace, new_size, - allocator_data && allocator_data->pinned); + : predefined_alloc_mapping (allocator)); + int pinned = (allocator_data + ? allocator_data->pinned + : allocator == ompx_gnu_pinned_mem_alloc); + new_ptr = MEMSPACE_ALLOC (memspace, new_size, pinned); } if (new_ptr == NULL) goto fail; @@ -1380,8 +1428,10 @@ retry: omp_memspace_handle_t was_memspace; was_memspace = (free_allocator_data ? free_allocator_data->memspace - : predefined_alloc_mapping[free_allocator]); - int was_pinned = (free_allocator_data && free_allocator_data->pinned); + : predefined_alloc_mapping (free_allocator)); + int was_pinned = (free_allocator_data + ? free_allocator_data->pinned + : free_allocator == ompx_gnu_pinned_mem_alloc); MEMSPACE_FREE (was_memspace, data->ptr, data->size, was_pinned); } return ret; @@ -1389,7 +1439,8 @@ retry: fail:; int fallback = (allocator_data ? allocator_data->fallback - : allocator == omp_default_mem_alloc + : (allocator == omp_default_mem_alloc + || allocator == ompx_gnu_pinned_mem_alloc) ? omp_atv_null_fb : omp_atv_default_mem_fb); switch (fallback) diff --git a/libgomp/env.c b/libgomp/env.c index 47d8c637b66..7f1f74f387d 100644 --- a/libgomp/env.c +++ b/libgomp/env.c @@ -1264,6 +1264,7 @@ parse_allocator (const char *env, const char *val, void *const params[]) C (omp_cgroup_mem_alloc, false) C (omp_pteam_mem_alloc, false) C (omp_thread_mem_alloc, false) + C (ompx_gnu_pinned_mem_alloc, false) C (omp_default_mem_space, true) C (omp_large_cap_mem_space, true) C (omp_const_mem_space, true) diff --git a/libgomp/libgomp.texi b/libgomp/libgomp.texi index d612488ad10..b3abb3dbdd9 100644 --- a/libgomp/libgomp.texi +++ b/libgomp/libgomp.texi @@ -3440,6 +3440,7 @@ value. @item omp_cgroup_mem_alloc @tab omp_low_lat_mem_space (implementation defined) @item omp_pteam_mem_alloc @tab omp_low_lat_mem_space (implementation defined) @item omp_thread_mem_alloc @tab omp_low_lat_mem_space (implementation defined) +@item ompx_gnu_pinned_mem_alloc @tab omp_default_mem_space (GNU extension) @end multitable The predefined allocators use the default values for the traits, @@ -3465,7 +3466,7 @@ as listed below. Except that the last three allocators have the @item @code{fb_data} @tab @emph{unsupported as it needs an allocator handle} @tab (none) @item @code{pinned} @tab @code{true}, @code{false} - @tab @code{false} + @tab See below @item @code{partition} @tab @code{environment}, @code{nearest}, @code{blocked}, @code{interleaved} @tab @code{environment} @@ -3476,6 +3477,10 @@ For the @code{fallback} trait, the default value is @code{null_fb} for the with device memory; for all other allocators, it is @code{default_mem_fb} by default. +For the @code{pinned} trait, the default value is @code{true} for +predefined allocator @code{ompx_gnu_pinned_mem_alloc} (a GNU extension), and +@code{false} for all others. + Examples: @smallexample OMP_ALLOCATOR=omp_high_bw_mem_alloc diff --git a/libgomp/omp.h.in b/libgomp/omp.h.in index 9b00647339e..4438d341160 100644 --- a/libgomp/omp.h.in +++ b/libgomp/omp.h.in @@ -134,6 +134,7 @@ typedef enum omp_allocator_handle_t __GOMP_UINTPTR_T_ENUM omp_cgroup_mem_alloc = 6, omp_pteam_mem_alloc = 7, omp_thread_mem_alloc = 8, + ompx_gnu_pinned_mem_alloc = 200, __omp_allocator_handle_t_max__ = __UINTPTR_MAX__ } omp_allocator_handle_t; diff --git a/libgomp/omp_lib.f90.in b/libgomp/omp_lib.f90.in index 65365e4497b..545a79fcec9 100644 --- a/libgomp/omp_lib.f90.in +++ b/libgomp/omp_lib.f90.in @@ -158,6 +158,8 @@ parameter :: omp_pteam_mem_alloc = 7 integer (kind=omp_allocator_handle_kind), & parameter :: omp_thread_mem_alloc = 8 + integer (kind=omp_allocator_handle_kind), & + parameter :: ompx_gnu_pinned_mem_alloc = 200 integer (omp_memspace_handle_kind), & parameter :: omp_default_mem_space = 0 integer (omp_memspace_handle_kind), & diff --git a/libgomp/omp_lib.h.in b/libgomp/omp_lib.h.in index 9f4019bf492..b5144bb4144 100644 --- a/libgomp/omp_lib.h.in +++ b/libgomp/omp_lib.h.in @@ -155,6 +155,7 @@ integer (omp_allocator_handle_kind) omp_cgroup_mem_alloc integer (omp_allocator_handle_kind) omp_pteam_mem_alloc integer (omp_allocator_handle_kind) omp_thread_mem_alloc + integer (omp_allocator_handle_kind) ompx_gnu_pinned_mem_alloc parameter (omp_null_allocator = 0) parameter (omp_default_mem_alloc = 1) parameter (omp_large_cap_mem_alloc = 2) @@ -164,6 +165,7 @@ parameter (omp_cgroup_mem_alloc = 6) parameter (omp_pteam_mem_alloc = 7) parameter (omp_thread_mem_alloc = 8) + parameter (ompx_gnu_pinned_mem_alloc = 200) integer (omp_memspace_handle_kind) omp_default_mem_space integer (omp_memspace_handle_kind) omp_large_cap_mem_space integer (omp_memspace_handle_kind) omp_const_mem_space diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-5.c b/libgomp/testsuite/libgomp.c/alloc-pinned-5.c new file mode 100644 index 00000000000..0ba2feb8d96 --- /dev/null +++ b/libgomp/testsuite/libgomp.c/alloc-pinned-5.c @@ -0,0 +1,100 @@ +/* { dg-do run } */ + +/* { dg-skip-if "Pinning not implemented on this host" { ! *-*-linux-gnu* } } */ + +/* Test that ompx_gnu_pinned_mem_alloc works. */ + +#include +#include + +#ifdef __linux__ +#include +#include + +#include +#include + +#define PAGE_SIZE sysconf(_SC_PAGESIZE) +#define CHECK_SIZE(SIZE) { \ + struct rlimit limit; \ + if (getrlimit (RLIMIT_MEMLOCK, &limit) \ + || limit.rlim_cur <= SIZE) \ + { \ + fprintf (stderr, "insufficient lockable memory; please increase ulimit\n"); \ + abort (); \ + } \ + } + +int +get_pinned_mem () +{ + int pid = getpid (); + char buf[100]; + sprintf (buf, "/proc/%d/status", pid); + + FILE *proc = fopen (buf, "r"); + if (!proc) + abort (); + while (fgets (buf, 100, proc)) + { + int val; + if (sscanf (buf, "VmLck: %d", &val)) + { + fclose (proc); + return val; + } + } + abort (); +} +#else +#error "OS unsupported" +#endif + +static void +verify0 (char *p, size_t s) +{ + for (size_t i = 0; i < s; ++i) + if (p[i] != 0) + abort (); +} + +#include + +int +main () +{ + /* Allocate at least a page each time, allowing space for overhead, + but stay within the ulimit. */ + const int SIZE = PAGE_SIZE - 128; + CHECK_SIZE (SIZE * 5); + + // Sanity check + if (get_pinned_mem () != 0) + abort (); + + void *p = omp_alloc (SIZE, ompx_gnu_pinned_mem_alloc); + if (!p) + abort (); + + int amount = get_pinned_mem (); + if (amount == 0) + abort (); + + p = omp_realloc (p, SIZE * 2, ompx_gnu_pinned_mem_alloc, + ompx_gnu_pinned_mem_alloc); + + int amount2 = get_pinned_mem (); + if (amount2 <= amount) + abort (); + + /* SIZE*2 ensures that it doesn't slot into the space possibly + vacated by realloc. */ + p = omp_calloc (1, SIZE * 2, ompx_gnu_pinned_mem_alloc); + + if (get_pinned_mem () <= amount2) + abort (); + + verify0 (p, SIZE * 2); + + return 0; +} diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-6.c b/libgomp/testsuite/libgomp.c/alloc-pinned-6.c new file mode 100644 index 00000000000..7ed68740ae6 --- /dev/null +++ b/libgomp/testsuite/libgomp.c/alloc-pinned-6.c @@ -0,0 +1,102 @@ +/* { dg-do run } */ + +/* Test that ompx_gnu_pinned_mem_alloc fails correctly. */ + +#include +#include + +#ifdef __linux__ +#include +#include + +#include +#include + +#define PAGE_SIZE sysconf(_SC_PAGESIZE) + +int +get_pinned_mem () +{ + int pid = getpid (); + char buf[100]; + sprintf (buf, "/proc/%d/status", pid); + + FILE *proc = fopen (buf, "r"); + if (!proc) + abort (); + while (fgets (buf, 100, proc)) + { + int val; + if (sscanf (buf, "VmLck: %d", &val)) + { + fclose (proc); + return val; + } + } + abort (); +} + +void +set_pin_limit (int size) +{ + struct rlimit limit; + if (getrlimit (RLIMIT_MEMLOCK, &limit)) + abort (); + limit.rlim_cur = (limit.rlim_max < size ? limit.rlim_max : size); + if (setrlimit (RLIMIT_MEMLOCK, &limit)) + abort (); +} +#else +#define PAGE_SIZE 10000 * 1024 /* unknown */ + +int +get_pinned_mem () +{ + return 0; +} + +void +set_pin_limit () +{ +} +#endif + +#include + +int +main () +{ + /* Allocate at least a page each time, but stay within the ulimit. */ + const int SIZE = PAGE_SIZE * 4; + + /* Ensure that the limit is smaller than the allocation. */ + set_pin_limit (SIZE / 2); + + // Sanity check + if (get_pinned_mem () != 0) + abort (); + + // Should fail + void *p = omp_alloc (SIZE, ompx_gnu_pinned_mem_alloc); + if (p) + abort (); + + // Should fail + p = omp_calloc (1, SIZE, ompx_gnu_pinned_mem_alloc); + if (p) + abort (); + + // Should fail to realloc + void *notpinned = omp_alloc (SIZE, omp_default_mem_alloc); + p = omp_realloc (notpinned, SIZE, ompx_gnu_pinned_mem_alloc, + omp_default_mem_alloc); + if (!notpinned || p) + abort (); + + // No memory should have been pinned + int amount = get_pinned_mem (); + if (amount != 0) + abort (); + + return 0; +} diff --git a/libgomp/testsuite/libgomp.fortran/alloc-pinned-1.f90 b/libgomp/testsuite/libgomp.fortran/alloc-pinned-1.f90 new file mode 100644 index 00000000000..379fffad932 --- /dev/null +++ b/libgomp/testsuite/libgomp.fortran/alloc-pinned-1.f90 @@ -0,0 +1,16 @@ +! Ensure that the ompx_gnu_pinned_mem_alloc predefined allocator is present and +! accepted. The majority of the functionality testing lives in the C tests. +! +! { dg-xfail-run-if "Pinning not implemented on this host" { ! *-*-linux-gnu } } + +program main + use omp_lib + use ISO_C_Binding + implicit none (external, type) + + type(c_ptr) :: p + + p = omp_alloc (10_c_size_t, ompx_gnu_pinned_mem_alloc); + if (.not. c_associated (p)) stop 1 + call omp_free (p, ompx_gnu_pinned_mem_alloc); +end program main From patchwork Wed Jun 12 11:42:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Stubbs X-Patchwork-Id: 1946815 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=baylibre-com.20230601.gappssmtp.com header.i=@baylibre-com.20230601.gappssmtp.com header.a=rsa-sha256 header.s=20230601 header.b=JS11RszX; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VzkFF0GJnz20Wd for ; Wed, 12 Jun 2024 21:43:13 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 333FB3882053 for ; Wed, 12 Jun 2024 11:43:11 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by sourceware.org (Postfix) with ESMTPS id E46D13877021 for ; Wed, 12 Jun 2024 11:42:34 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org E46D13877021 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=baylibre.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=baylibre.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org E46D13877021 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2a00:1450:4864:20::334 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1718192557; cv=none; b=tzBrh5lJOYT5emQYQq2T/kQRhfrWJjdXsEFL180Lm4fzQIksYCoQxlkt6wOIANqlhh7DpKa3Kzmodgczu39Ou7PW57mK82OnMvCI4azi0uEqNrkBXNMYU1YhdelNAlMxDEE3mW0ALaPO+1IF6GCOkSbeve8nVREeywYWC8yCo0k= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1718192557; c=relaxed/simple; bh=rpp0wAJEZMVyW1+CD1UE8pgqceR8rFXhGtrNUh9pESw=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=HVRm4IOVsabyIP0WT04MB+CDBTTepT/I+tEVoGZFXmKhtEAiKOu5jVCnDJV/4K1ZrLVU9e5zySc0hgKiMrOH68F53sL2wgcVsRKQEHAObA2+mtBlNST12kdr+REkCNdUwszN5006y1ziz4jrPuclxFe/wPwyITOgp9pSUTNCnIY= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-wm1-x334.google.com with SMTP id 5b1f17b1804b1-421757d217aso47444435e9.3 for ; Wed, 12 Jun 2024 04:42:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1718192553; x=1718797353; darn=gcc.gnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=hD1PRTGrbjS5BO4hwunh8RPV8Q6+mTKV5OJby39ZosM=; b=JS11RszXu9wnFvljNLfyPpT8Hb5n8q15gyvcwI3YFnfsaHBhvgZXhEZAnWnaL2utJY bbs9IaLt0WjL036b/L8cylB7VZTpNJ6RaeoAak7mi96Lb+utdOaqqccjbZARQRmjBxOG fNJBKafK6L5EuH/M2PmBQN/6TGxFLxae0oyddIms+E08lSHTQnYsog1+KXurbqggqkXP Aqz33bBWB3aVBHcLFaPJUd14Nh8Y/EDtSO43J6ODwu+yoNt9ZNIbOKZYmEVjKfWkqeDY v6HOln4ksofQy3FHOjfY7wJ5ZD0uxe7dYs3QwT+o6mg7Tm5dSVrBka6qf/cr9aOqauAn zeJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718192553; x=1718797353; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hD1PRTGrbjS5BO4hwunh8RPV8Q6+mTKV5OJby39ZosM=; b=LkcJYPYLKQnglVFP3/fGXWdgVvHgIbThvglGU6ln1LoDq0/rX0lIYhMB1WkkMO+58A yiLqMXOloLBk4Hcwc59eRpEuvArjNqR4IW4ZbHcsIvvnZCDsMb2LKwQ8Pn3UpFENCPru fw6IE81uLz1pR9nCYkhAWboI3YcC9DEMULhaQjR2cKg0ggOWMeT5VouK9XUj57qHG/cV +RJdUwHytYs+oakroewrs1UIInfp63PZ7IzHNcaOiWs6cCSis/KB78fQaOVnKsln9IOm zA0nxTeXBal+nC1dgkx5hck8CzACgrgONk77CI5c2f48WfNsSeBfSv8V93NdQJKNhMEI L/yA== X-Gm-Message-State: AOJu0YwZl7lTX1vK8j4Ugjjq2u7333ILOMBFszoBsI25EUbvZpT/6mLn z8PD7zn5hQIXcLHmNcN47mfMKb9DdbsyFJjm26hvoch+LaCGOh0X8LX/mLyv7M1j/9FV32EUeoK LiZk= X-Google-Smtp-Source: AGHT+IFJL26jePNnnWW1XwR9pI0l0S7PxS+W56YSEqoae3dmAf9iMszWB1gyIN4JW7K1K67aHjBdRQ== X-Received: by 2002:a05:600c:42c4:b0:422:6755:17de with SMTP id 5b1f17b1804b1-422867c90f9mr15528375e9.41.1718192553346; Wed, 12 Jun 2024 04:42:33 -0700 (PDT) Received: from arnold.baylibre (laubervilliers-658-1-213-31.w90-63.abo.wanadoo.fr. [90.63.244.31]) by smtp.googlemail.com with ESMTPSA id 5b1f17b1804b1-42286fe75ebsm23366235e9.9.2024.06.12.04.42.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Jun 2024 04:42:32 -0700 (PDT) From: Andrew Stubbs To: gcc-patches@gcc.gnu.org Cc: tburnus@baylibre.com, jakub@redhat.com Subject: [PATCH v5 3/6] openmp: Add -foffload-memory Date: Wed, 12 Jun 2024 11:42:20 +0000 Message-ID: <20240612114223.2206735-4-ams@baylibre.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20240612114223.2206735-1-ams@baylibre.com> References: <20240612114223.2206735-1-ams@baylibre.com> MIME-Version: 1.0 X-Spam-Status: No, score=-10.3 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org Add a new option. It's inactive until I add some follow-up patches. gcc/ChangeLog: * common.opt: Add -foffload-memory and its enum values. * coretypes.h (enum offload_memory): New. * doc/invoke.texi: Document -foffload-memory. --- gcc/common.opt | 16 ++++++++++++++++ gcc/coretypes.h | 7 +++++++ gcc/doc/invoke.texi | 16 +++++++++++++++- 3 files changed, 38 insertions(+), 1 deletion(-) diff --git a/gcc/common.opt b/gcc/common.opt index 2c078fdd1f8..e874e88d3e1 100644 --- a/gcc/common.opt +++ b/gcc/common.opt @@ -2349,6 +2349,22 @@ Enum(offload_abi) String(ilp32) Value(OFFLOAD_ABI_ILP32) EnumValue Enum(offload_abi) String(lp64) Value(OFFLOAD_ABI_LP64) +foffload-memory= +Common Joined RejectNegative Enum(offload_memory) Var(flag_offload_memory) Init(OFFLOAD_MEMORY_NONE) +-foffload-memory=[none|unified|pinned] Use an offload memory optimization. + +Enum +Name(offload_memory) Type(enum offload_memory) UnknownError(Unknown offload memory option %qs) + +EnumValue +Enum(offload_memory) String(none) Value(OFFLOAD_MEMORY_NONE) + +EnumValue +Enum(offload_memory) String(unified) Value(OFFLOAD_MEMORY_UNIFIED) + +EnumValue +Enum(offload_memory) String(pinned) Value(OFFLOAD_MEMORY_PINNED) + fomit-frame-pointer Common Var(flag_omit_frame_pointer) Optimization When possible do not generate stack frames. diff --git a/gcc/coretypes.h b/gcc/coretypes.h index 1ac6f0abea3..938cfa93753 100644 --- a/gcc/coretypes.h +++ b/gcc/coretypes.h @@ -218,6 +218,13 @@ enum offload_abi { OFFLOAD_ABI_ILP32 }; +/* Types of memory optimization for an offload device. */ +enum offload_memory { + OFFLOAD_MEMORY_NONE, + OFFLOAD_MEMORY_UNIFIED, + OFFLOAD_MEMORY_PINNED +}; + /* Types of profile update methods. */ enum profile_update { PROFILE_UPDATE_SINGLE, diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi index 45115b5fbed..eb0f8b4a58d 100644 --- a/gcc/doc/invoke.texi +++ b/gcc/doc/invoke.texi @@ -202,7 +202,7 @@ in the following sections. -fno-builtin -fno-builtin-@var{function} -fcond-mismatch -ffreestanding -fgimple -fgnu-tm -fgnu89-inline -fhosted -flax-vector-conversions -fms-extensions --foffload=@var{arg} -foffload-options=@var{arg} +-foffload=@var{arg} -foffload-options=@var{arg} -foffload-memory=@var{arg} -fopenacc -fopenacc-dim=@var{geom} -fopenmp -fopenmp-simd -fopenmp-target-simd-clone@r{[}=@var{device-type}@r{]} -fpermitted-flt-eval-methods=@var{standard} @@ -2786,6 +2786,20 @@ Typical command lines are -foffload-options=amdgcn-amdhsa=-march=gfx906 @end smallexample +@opindex foffload-memory +@cindex OpenMP offloading memory modes +@item -foffload-memory=none +@itemx -foffload-memory=unified +@itemx -foffload-memory=pinned +Enable a memory optimization mode to use with OpenMP. The default behavior, +@option{-foffload-memory=none}, is to do nothing special (unless enabled via +a requires directive in the code). @option{-foffload-memory=unified} is +equivalent to @code{#pragma omp requires unified_shared_memory}. +@option{-foffload-memory=pinned} forces all host memory to be pinned (this +mode may require the user to increase the ulimit setting for locked memory). +All translation units must select the same setting to avoid undefined +behavior. + @opindex fopenacc @cindex OpenACC accelerator programming @item -fopenacc From patchwork Wed Jun 12 11:42:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Stubbs X-Patchwork-Id: 1946818 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=baylibre-com.20230601.gappssmtp.com header.i=@baylibre-com.20230601.gappssmtp.com header.a=rsa-sha256 header.s=20230601 header.b=jlZnYAUp; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=8.43.85.97; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VzkGR5TKMz20Wd for ; Wed, 12 Jun 2024 21:44:15 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 4F71C3882AD0 for ; Wed, 12 Jun 2024 11:44:08 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail-lj1-x236.google.com (mail-lj1-x236.google.com [IPv6:2a00:1450:4864:20::236]) by sourceware.org (Postfix) with ESMTPS id 1DC763877030 for ; Wed, 12 Jun 2024 11:42:36 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 1DC763877030 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=baylibre.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=baylibre.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 1DC763877030 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2a00:1450:4864:20::236 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1718192560; cv=none; b=TKMKrQ2X76AKV/U4r5DWuG+Bye1IGjitMfpnASNmCudCweXVLs4x+k6lQjFVNrVUGfOd6cYKyuxrx+OrY4feKJnN2CLLYid/zeGCUYNauKFKrHyqdyb/Z6HrTyQNVMUnfRajmNG0bYWW7/2w7PL3b0N8j8KNbgxvXalgzbaClEs= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1718192560; c=relaxed/simple; bh=YOkjWy+px+zYu4ewB7s/kvLzgqvJD7XlYhXnX1oWZeE=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=fNFxfiws4hmfMqzknJJvfyhDgT+V498EmyLyAyMW8ybqRrq4X6Yi/f4tO7eN0Di7JpvMJf3i3MUpuiGqUwpxAi4igt2Gw9FXJUIFOLZLOtqh+N3g96rj7U0MrJMcDRZcE9OQyU2d0V02DD0pj4ZNl22njbBWUTWWNNkqV1hzhOs= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-lj1-x236.google.com with SMTP id 38308e7fff4ca-2ebed33cb65so31751521fa.2 for ; Wed, 12 Jun 2024 04:42:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1718192554; x=1718797354; darn=gcc.gnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QGENu5iHm7jO8pzPZHUxQih9UFe3gSgx6ruCPk27lxs=; b=jlZnYAUp7tpPiK8z5o5oWzK9KQuep8Qo802LaQw0FHEjGPNVvJTMadd1Xzd4a6Yixa GNGpSh1EED9sbLP7hLNxvB4FXeL/vzROCKcPNwHPWLGJYvxtNKlMCjIsUTwc27nm8cle 8PkMQyMsxGIdkpAZ9UuxXnp/IIrvMcOYAFT3xXFJFQhGg2M8emp4a6+jZxtlMntGIK3C a3J6q+xyFknraQj0SxzB4i0z7ygLOG08RJ2TvWhmhAkb7Q2Z6z5mET7tPc8UDtiucJtk brkQz0w4OXFNuOK08PDLUE1aotrKAPE3Iu6FT+CrDzQeIcRJmpaluEwHJJ0y1sCv3kSF L5Zg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718192554; x=1718797354; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QGENu5iHm7jO8pzPZHUxQih9UFe3gSgx6ruCPk27lxs=; b=v5sDzro7CktTeSlKFZ4O+YNK0ivxYYlgRytVEVmQ7rBcWIoA7eOzcwRdGFDsF4XfCV G3k6sLq/woeiDyp8457zkWAzWEfjpUA1cmffg6+o8giqA/I1tHAfIH7ITAIbl0E9rPaU d4dGVmQui97Gf6uK5FXUwEFamfvZ2Umlt3nEUuJXkLqMGOW0yuVS7NqZDwL6jTUPsbCb hQTnfKqYvN8o44jmIx29wBxgnq3HuTvxBT3UMWpBnVeaTw87ax62aMY+ELPPK9u/hhvR tK6Y/WAIsLq0qIcawuHwnkSfq/fJzbFS2AYE9z6teNFg+e+SY4qJbOX26St4WBigjzs5 UTkg== X-Gm-Message-State: AOJu0Yzw3t1VbN1BMUIuTIRM1IvKBPIEKAAckOZ7Gy1w0C+0q4IXdSZ6 WZkJ8TaDYa0qQM/Pb3tqIk2JgRmJ38Is0f3YN5Fzx1SU2dO6UMzeQo9QTRCnW36h9Bx2Ge/HX4A CaHA= X-Google-Smtp-Source: AGHT+IGsqqzrCBLSGb1a+EbL5ayRT4sZgXpg/t3/JLNPfYlIslUgYV0CdqwhhwHJocQouBSLc9JJkA== X-Received: by 2002:a2e:6a14:0:b0:2eb:fc08:5d83 with SMTP id 38308e7fff4ca-2ebfca5dac4mr9879481fa.44.1718192554241; Wed, 12 Jun 2024 04:42:34 -0700 (PDT) Received: from arnold.baylibre (laubervilliers-658-1-213-31.w90-63.abo.wanadoo.fr. [90.63.244.31]) by smtp.googlemail.com with ESMTPSA id 5b1f17b1804b1-42286fe75ebsm23366235e9.9.2024.06.12.04.42.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Jun 2024 04:42:33 -0700 (PDT) From: Andrew Stubbs To: gcc-patches@gcc.gnu.org Cc: tburnus@baylibre.com, jakub@redhat.com Subject: [PATCH v5 4/6] openmp: -foffload-memory=pinned Date: Wed, 12 Jun 2024 11:42:21 +0000 Message-ID: <20240612114223.2206735-5-ams@baylibre.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20240612114223.2206735-1-ams@baylibre.com> References: <20240612114223.2206735-1-ams@baylibre.com> MIME-Version: 1.0 X-Spam-Status: No, score=-10.4 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org Implement the -foffload-memory=pinned option such that libgomp is instructed to enable fully-pinned memory at start-up. The option is intended to provide a performance boost to certain offload programs without modifying the code. This feature only works on Linux, at present, and simply calls mlockall to enable always-on memory pinning. It requires that the ulimit feature is set high enough to accommodate all the program's memory usage. In this mode the ompx_gnu_pinned_memory_alloc feature is disabled as it is not needed and may conflict. gcc/ChangeLog: * omp-builtins.def (BUILT_IN_GOMP_ENABLE_PINNED_MODE): New. * omp-low.cc (omp_enable_pinned_mode): New function. (execute_lower_omp): Call omp_enable_pinned_mode. libgomp/ChangeLog: * config/linux/allocator.c (always_pinned_mode): New variable. (GOMP_enable_pinned_mode): New function. (linux_memspace_alloc): Disable pinning when always_pinned_mode set. (linux_memspace_calloc): Likewise. (linux_memspace_free): Likewise. (linux_memspace_realloc): Likewise. * libgomp.map: Add GOMP_enable_pinned_mode. * testsuite/libgomp.c/alloc-pinned-7.c: New test. * testsuite/libgomp.c-c++-common/alloc-pinned-1.c: New test. --- gcc/omp-builtins.def | 3 + gcc/omp-low.cc | 66 +++++++++++++++++++ libgomp/config/linux/allocator.c | 26 ++++++++ libgomp/libgomp.map | 1 + .../libgomp.c-c++-common/alloc-pinned-1.c | 28 ++++++++ libgomp/testsuite/libgomp.c/alloc-pinned-7.c | 63 ++++++++++++++++++ 6 files changed, 187 insertions(+) create mode 100644 libgomp/testsuite/libgomp.c-c++-common/alloc-pinned-1.c create mode 100644 libgomp/testsuite/libgomp.c/alloc-pinned-7.c diff --git a/gcc/omp-builtins.def b/gcc/omp-builtins.def index 044d5d087b6..aefc52e5f9f 100644 --- a/gcc/omp-builtins.def +++ b/gcc/omp-builtins.def @@ -476,3 +476,6 @@ DEF_GOMP_BUILTIN (BUILT_IN_GOMP_WARNING, "GOMP_warning", BT_FN_VOID_CONST_PTR_SIZE, ATTR_NOTHROW_LEAF_LIST) DEF_GOMP_BUILTIN (BUILT_IN_GOMP_ERROR, "GOMP_error", BT_FN_VOID_CONST_PTR_SIZE, ATTR_COLD_NORETURN_NOTHROW_LEAF_LIST) +DEF_GOMP_BUILTIN (BUILT_IN_GOMP_ENABLE_PINNED_MODE, + "GOMP_enable_pinned_mode", + BT_FN_VOID, ATTR_NOTHROW_LIST) diff --git a/gcc/omp-low.cc b/gcc/omp-low.cc index 4d003f42098..cf3f57748d8 100644 --- a/gcc/omp-low.cc +++ b/gcc/omp-low.cc @@ -14596,6 +14596,68 @@ lower_omp (gimple_seq *body, omp_context *ctx) input_location = saved_location; } +/* Emit a constructor function to enable -foffload-memory=pinned + at runtime. Libgomp handles the OS mode setting, but we need to trigger + it by calling GOMP_enable_pinned mode before the program proper runs. */ + +static void +omp_enable_pinned_mode () +{ + static bool visited = false; + if (visited) + return; + visited = true; + + /* Create a new function like this: + + static void __attribute__((constructor)) + __set_pinned_mode () + { + GOMP_enable_pinned_mode (); + } + */ + + tree name = get_identifier ("__set_pinned_mode"); + tree voidfntype = build_function_type_list (void_type_node, NULL_TREE); + tree decl = build_decl (UNKNOWN_LOCATION, FUNCTION_DECL, name, voidfntype); + + TREE_STATIC (decl) = 1; + TREE_USED (decl) = 1; + DECL_ARTIFICIAL (decl) = 1; + DECL_IGNORED_P (decl) = 0; + TREE_PUBLIC (decl) = 0; + DECL_UNINLINABLE (decl) = 1; + DECL_EXTERNAL (decl) = 0; + DECL_CONTEXT (decl) = NULL_TREE; + DECL_INITIAL (decl) = make_node (BLOCK); + BLOCK_SUPERCONTEXT (DECL_INITIAL (decl)) = decl; + DECL_STATIC_CONSTRUCTOR (decl) = 1; + DECL_ATTRIBUTES (decl) = tree_cons (get_identifier ("constructor"), + NULL_TREE, NULL_TREE); + + tree t = build_decl (UNKNOWN_LOCATION, RESULT_DECL, NULL_TREE, + void_type_node); + DECL_ARTIFICIAL (t) = 1; + DECL_IGNORED_P (t) = 1; + DECL_CONTEXT (t) = decl; + DECL_RESULT (decl) = t; + + push_struct_function (decl); + init_tree_ssa (cfun); + + tree calldecl = builtin_decl_explicit (BUILT_IN_GOMP_ENABLE_PINNED_MODE); + gcall *call = gimple_build_call (calldecl, 0); + + gimple_seq seq = NULL; + gimple_seq_add_stmt (&seq, call); + gimple_set_body (decl, gimple_build_bind (NULL_TREE, seq, NULL)); + + cfun->function_end_locus = UNKNOWN_LOCATION; + cfun->curr_properties |= PROP_gimple_any; + pop_cfun (); + cgraph_node::add_new_function (decl, true); +} + /* Main entry point. */ static unsigned int @@ -14652,6 +14714,10 @@ execute_lower_omp (void) for (auto task_stmt : task_cpyfns) finalize_task_copyfn (task_stmt); task_cpyfns.release (); + + if (flag_offload_memory == OFFLOAD_MEMORY_PINNED) + omp_enable_pinned_mode (); + return 0; } diff --git a/libgomp/config/linux/allocator.c b/libgomp/config/linux/allocator.c index de98c04289f..7e09ba44b2f 100644 --- a/libgomp/config/linux/allocator.c +++ b/libgomp/config/linux/allocator.c @@ -54,11 +54,28 @@ # include /* For PRIu64. */ #endif +static bool always_pinned_mode = false; + +/* This function is called by the compiler when -foffload-memory=pinned + is used. */ + +void +GOMP_enable_pinned_mode () +{ + if (mlockall (MCL_CURRENT | MCL_FUTURE) != 0) + gomp_error ("failed to pin all memory (ulimit too low?)"); + else + always_pinned_mode = true; +} + static void * linux_memspace_alloc (omp_memspace_handle_t memspace, size_t size, int pin) { (void)memspace; + /* Explicit pinning may not be required. */ + pin = pin && !always_pinned_mode; + if (pin) { /* Note that mmap always returns zeroed memory and is therefore also a @@ -90,6 +107,9 @@ linux_memspace_alloc (omp_memspace_handle_t memspace, size_t size, int pin) static void * linux_memspace_calloc (omp_memspace_handle_t memspace, size_t size, int pin) { + /* Explicit pinning may not be required. */ + pin = pin && !always_pinned_mode; + if (pin) return linux_memspace_alloc (memspace, size, pin); else @@ -102,6 +122,9 @@ linux_memspace_free (omp_memspace_handle_t memspace, void *addr, size_t size, { (void)memspace; + /* Explicit pinning may not be required. */ + pin = pin && !always_pinned_mode; + if (pin) munmap (addr, size); else @@ -112,6 +135,9 @@ static void * linux_memspace_realloc (omp_memspace_handle_t memspace, void *addr, size_t oldsize, size_t size, int oldpin, int pin) { + /* Explicit pinning may not be required. */ + pin = pin && !always_pinned_mode; + if (oldpin && pin) { void *newaddr = mremap (addr, oldsize, size, MREMAP_MAYMOVE); diff --git a/libgomp/libgomp.map b/libgomp/libgomp.map index 65901dff235..0fee53046e4 100644 --- a/libgomp/libgomp.map +++ b/libgomp/libgomp.map @@ -406,6 +406,7 @@ GOMP_5.0.1 { global: GOMP_alloc; GOMP_free; + GOMP_enable_pinned_mode; } GOMP_5.0; GOMP_5.1 { diff --git a/libgomp/testsuite/libgomp.c-c++-common/alloc-pinned-1.c b/libgomp/testsuite/libgomp.c-c++-common/alloc-pinned-1.c new file mode 100644 index 00000000000..77333951ca7 --- /dev/null +++ b/libgomp/testsuite/libgomp.c-c++-common/alloc-pinned-1.c @@ -0,0 +1,28 @@ +/* { dg-do run } */ +/* { dg-additional-options "-foffload-memory=pinned" } */ +/* { dg-skip-if "Pinning not implemented on this host" { ! *-*-linux-gnu* } } */ + +#if __cplusplus +#define EXTERNC extern "C" +#else +#define EXTERNC +#endif + +/* Intercept the libgomp initialization call to check it happens. */ + +int good = 0; + +EXTERNC void +GOMP_enable_pinned_mode () +{ + good = 1; +} + +int +main () +{ + if (!good) + __builtin_exit (1); + + return 0; +} diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-7.c b/libgomp/testsuite/libgomp.c/alloc-pinned-7.c new file mode 100644 index 00000000000..44652aac874 --- /dev/null +++ b/libgomp/testsuite/libgomp.c/alloc-pinned-7.c @@ -0,0 +1,63 @@ +/* { dg-do run } */ +/* { dg-additional-options "-foffload-memory=pinned" } */ + +/* { dg-skip-if "Pinning not implemented on this host" { ! *-*-linux-gnu* } } */ + +/* Test that -foffload-memory=pinned works. */ + +#include +#include + +#ifdef __linux__ +#include +#include + +#include + +int +get_pinned_mem () +{ + int pid = getpid (); + char buf[100]; + sprintf (buf, "/proc/%d/status", pid); + + FILE *proc = fopen (buf, "r"); + if (!proc) + abort (); + while (fgets (buf, 100, proc)) + { + int val; + if (sscanf (buf, "VmLck: %d", &val)) + { + fclose (proc); + return val; + } + } + abort (); +} +#else +int +get_pinned_mem () +{ + return 0; +} + +#define mlockall(...) 0 +#endif + +#include + +int +main () +{ + // Sanity check + if (get_pinned_mem () == 0) + { + /* -foffload-memory=pinned has failed, but maybe that's because + isufficient pinned memory was available. */ + if (mlockall (MCL_CURRENT | MCL_FUTURE) == 0) + abort (); + } + + return 0; +} From patchwork Wed Jun 12 11:42:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Stubbs X-Patchwork-Id: 1946817 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=baylibre-com.20230601.gappssmtp.com header.i=@baylibre-com.20230601.gappssmtp.com header.a=rsa-sha256 header.s=20230601 header.b=fXIcR3e2; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=8.43.85.97; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VzkGQ2vD8z20Wd for ; Wed, 12 Jun 2024 21:44:14 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 50C1B3884565 for ; Wed, 12 Jun 2024 11:44:06 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by sourceware.org (Postfix) with ESMTPS id AED433877036 for ; Wed, 12 Jun 2024 11:42:41 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org AED433877036 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=baylibre.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=baylibre.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org AED433877036 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2a00:1450:4864:20::329 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1718192568; cv=none; b=bwYwd+SJJctUzSQ7YFXcjFvjYbcqcU6mCO8ya3tyk5ZqlX/eEUuxn+xAMRI/ZEZWwySE+Lz5tOEp+403OnSzcVr+XiXCDiUfkVwwmjG2BTgXl9sw3eqWszSl8pNJl2KOVkiTopDkyVSMxzPAfwsOiEZZXva7Idcl5LftASt9/t0= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1718192568; c=relaxed/simple; bh=ENltN6kKb9knjzOA2m1/joI73uFSwatfZheB5jtko/w=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=lxsLGISxD9VIhz1KRYd8LhnbpsbyI6K7Dao7lLa2LvSU0I0OBruDHdsDHQqmdq/QIJVqzvdlNn0J5OJSzbDPUVtqdNOCm1aL/2Wot0j2c9F0AHj6s7sjJOXqgItlB1Wl34yx/xMOjwN66xJnHQCpwVMqfSjD+QMfBnitIfGlIMU= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-wm1-x329.google.com with SMTP id 5b1f17b1804b1-421f4d1c057so24400875e9.3 for ; Wed, 12 Jun 2024 04:42:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1718192560; x=1718797360; darn=gcc.gnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sLy2ymI+jvNvvRjXHPV24xWhOKyd3hO4XDrc2eDgbzE=; b=fXIcR3e2uzo0umV3Ilw9scauTtbf0LhX/ym66VRNR3Eqbpr0Pu/JyxrOJLfufZs+JR NFUHvv6CCjTwXK5/47UcxWxRiMw7u62R6elBAf5PCOhgje/wBAdOMwfy/TW08hDLNGvf vkoVleX4Yj/KWRsFBwfhalkXf9ssy+k9OQffDEViXdVgUV4/dZLxyFcRxzpIb/pBjw9X vMpcyWA6r1hfBtHo68LYigJQI7KfQpgTzE1lO+mKsNytGUO1M1mt+nCQLDz2obWizUG+ NipCpbFI7XWOM3bx/9NLwgspPhB1I4j3vjMNz6V0iugtu6p+wcE2rH8itsBksXBcHRra a7HA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718192560; x=1718797360; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sLy2ymI+jvNvvRjXHPV24xWhOKyd3hO4XDrc2eDgbzE=; b=ACkOLcbkydYScKQ+u7BVjTW2L1lw1+LQZTE9o/YUPsgHFrnXDxlzuBTeoJNoAo98Wd ajqLT48Fi1xNrxRRzBjJF509FnmbvQURQNPfKPsSgEkj68QPIpkJIWdhzseuz3FZtOKP YWV5oKfjzvgJYuy+KVJ/t57aXSbFv1QCVJdFXHyLnJ9yNGAAwhan74ruP3uDIIIqWuYD MjDbprKfC66EHOmCuzAYo5vVmEJm4hgh1J+yh9tEnELf/8xauidZahGSb9IUJVknZDIX zl+pX89tA2OzAfs9Su1GWKpg18+2a60BadR3oVpUxhaDk2pF/K4mVbIxgyi2/+rSvXIh UcXQ== X-Gm-Message-State: AOJu0YwM1d2M36QRjKe44IAq/r1iGd9ATLhb7h3eEJJrBsXqnzZZeUjT xZpoSZwUdi6ZcaMJHn4fOuW8m45tr8Xc4Ttlm1XWg1x1qj0ANPgqoZIWnaCq0IuEeyajZMLVlTL RbNY= X-Google-Smtp-Source: AGHT+IGTPXbghfvE/GHL47Vk/AE9in7XFwlA8Sxecgqvl0HJzGZuj381NGpNVGtnMp66lcdeE3fEsg== X-Received: by 2002:a05:600c:138f:b0:421:7df5:656a with SMTP id 5b1f17b1804b1-422863b866bmr12763015e9.18.1718192559643; Wed, 12 Jun 2024 04:42:39 -0700 (PDT) Received: from arnold.baylibre (laubervilliers-658-1-213-31.w90-63.abo.wanadoo.fr. [90.63.244.31]) by smtp.googlemail.com with ESMTPSA id ffacd0b85a97d-35f1aa8d4f3sm11049246f8f.99.2024.06.12.04.42.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Jun 2024 04:42:39 -0700 (PDT) From: Andrew Stubbs To: gcc-patches@gcc.gnu.org Cc: tburnus@baylibre.com, jakub@redhat.com Subject: [PATCH v5 5/6] libgomp, nvptx: Cuda pinned memory Date: Wed, 12 Jun 2024 11:42:22 +0000 Message-ID: <20240612114223.2206735-6-ams@baylibre.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20240612114223.2206735-1-ams@baylibre.com> References: <20240612114223.2206735-1-ams@baylibre.com> MIME-Version: 1.0 X-Spam-Status: No, score=-10.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org This patch was already approved, in the v3 posting by Tobias Burnus (with one caveat about initialization location), but wasn't committed at that time as I didn't want to disentangle it from the textual dependencies on the other patches in the series. ------ Use Cuda to pin memory, instead of Linux mlock, when available. There are two advantages: firstly, this gives a significant speed boost for NVPTX offloading, and secondly, it side-steps the usual OS ulimit/rlimit setting. The design adds a device independent plugin API for allocating pinned memory, and then implements it for NVPTX. At present, the other supported devices do not have equivalent capabilities (or requirements). libgomp/ChangeLog: * config/linux/allocator.c: Include assert.h. (using_device_for_page_locked): New variable. (linux_memspace_alloc): Add init0 parameter. Support device pinning. (linux_memspace_calloc): Set init0 to true. (linux_memspace_free): Support device pinning. (linux_memspace_realloc): Support device pinning. (MEMSPACE_ALLOC): Set init0 to false. * libgomp-plugin.h (GOMP_OFFLOAD_page_locked_host_alloc): New prototype. (GOMP_OFFLOAD_page_locked_host_free): Likewise. * libgomp.h (gomp_page_locked_host_alloc): Likewise. (gomp_page_locked_host_free): Likewise. (struct gomp_device_descr): Add page_locked_host_alloc_func and page_locked_host_free_func. * libgomp.texi: Adjust the docs for the pinned trait. * libgomp_g.h (GOMP_enable_pinned_mode): New prototype. * plugin/plugin-nvptx.c (GOMP_OFFLOAD_page_locked_host_alloc): New function. (GOMP_OFFLOAD_page_locked_host_free): Likewise. * target.c (device_for_page_locked): New variable. (get_device_for_page_locked): New function. (gomp_page_locked_host_alloc): Likewise. (gomp_page_locked_host_free): Likewise. (gomp_load_plugin_for_device): Add page_locked_host_alloc and page_locked_host_free. * testsuite/libgomp.c/alloc-pinned-1.c: Change expectations for NVPTX devices. * testsuite/libgomp.c/alloc-pinned-2.c: Likewise. * testsuite/libgomp.c/alloc-pinned-3.c: Likewise. * testsuite/libgomp.c/alloc-pinned-4.c: Likewise. * testsuite/libgomp.c/alloc-pinned-5.c: Likewise. * testsuite/libgomp.c/alloc-pinned-6.c: Likewise. Co-Authored-By: Thomas Schwinge --- libgomp/config/linux/allocator.c | 141 ++++++++++++++----- libgomp/libgomp-plugin.h | 2 + libgomp/libgomp.h | 4 + libgomp/libgomp.texi | 11 +- libgomp/libgomp_g.h | 1 + libgomp/plugin/plugin-nvptx.c | 42 ++++++ libgomp/target.c | 136 ++++++++++++++++++ libgomp/testsuite/libgomp.c/alloc-pinned-1.c | 26 ++++ libgomp/testsuite/libgomp.c/alloc-pinned-2.c | 26 ++++ libgomp/testsuite/libgomp.c/alloc-pinned-3.c | 45 +++++- libgomp/testsuite/libgomp.c/alloc-pinned-4.c | 44 +++++- libgomp/testsuite/libgomp.c/alloc-pinned-5.c | 26 ++++ libgomp/testsuite/libgomp.c/alloc-pinned-6.c | 34 ++++- 13 files changed, 488 insertions(+), 50 deletions(-) diff --git a/libgomp/config/linux/allocator.c b/libgomp/config/linux/allocator.c index 7e09ba44b2f..063c46f972c 100644 --- a/libgomp/config/linux/allocator.c +++ b/libgomp/config/linux/allocator.c @@ -36,6 +36,11 @@ /* Implement malloc routines that can handle pinned memory on Linux. + Given that pinned memory is typically used to help host <-> device memory + transfers, we attempt to allocate such memory using a device (really: + libgomp plugin), but fall back to mmap plus mlock if no suitable device is + available. + It's possible to use mlock on any heap memory, but using munlock is problematic if there are multiple pinned allocations on the same page. Tracking all that manually would be possible, but adds overhead. This may @@ -49,6 +54,7 @@ #define _GNU_SOURCE #include #include +#include #include "libgomp.h" #ifdef HAVE_INTTYPES_H # include /* For PRIu64. */ @@ -68,50 +74,92 @@ GOMP_enable_pinned_mode () always_pinned_mode = true; } +static int using_device_for_page_locked + = /* uninitialized */ -1; + static void * -linux_memspace_alloc (omp_memspace_handle_t memspace, size_t size, int pin) +linux_memspace_alloc (omp_memspace_handle_t memspace, size_t size, int pin, + bool init0) { - (void)memspace; + gomp_debug (0, "%s: memspace=%llu, size=%llu, pin=%d, init0=%d\n", + __FUNCTION__, (unsigned long long) memspace, + (unsigned long long) size, pin, init0); + + void *addr; /* Explicit pinning may not be required. */ pin = pin && !always_pinned_mode; if (pin) { - /* Note that mmap always returns zeroed memory and is therefore also a - suitable implementation of calloc. */ - void *addr = mmap (NULL, size, PROT_READ | PROT_WRITE, - MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); - if (addr == MAP_FAILED) - return NULL; - - if (mlock (addr, size)) + int using_device + = __atomic_load_n (&using_device_for_page_locked, + MEMMODEL_RELAXED); + gomp_debug (0, " using_device=%d\n", + using_device); + if (using_device != 0) + { + using_device = gomp_page_locked_host_alloc (&addr, size); + int using_device_old + = __atomic_exchange_n (&using_device_for_page_locked, + using_device, MEMMODEL_RELAXED); + gomp_debug (0, " using_device=%d, using_device_old=%d\n", + using_device, using_device_old); + assert (using_device_old == -1 + /* We shouldn't have concurrently changed our mind. */ + || using_device_old == using_device); + } + if (using_device == 0) { + gomp_debug (0, " mmap\n"); + addr = mmap (NULL, size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (addr == MAP_FAILED) + addr = NULL; + else + { + /* 'mmap' zero-initializes. */ + init0 = false; + + gomp_debug (0, " mlock\n"); + if (mlock (addr, size)) + { #ifdef HAVE_INTTYPES_H - gomp_debug (0, "libgomp: failed to pin %"PRIu64" bytes of" - " memory (ulimit too low?)\n", (uint64_t) size); + gomp_debug (0, "libgomp: failed to pin %"PRIu64" bytes of" + " memory (ulimit too low?)\n", (uint64_t) size); #else - gomp_debug (0, "libgomp: failed to pin %lu bytes of" - " memory (ulimit too low?)\n", (unsigned long) size); + gomp_debug (0, "libgomp: failed to pin %lu bytes of memory" + " (ulimit too low?)\n", (unsigned long) size); #endif - munmap (addr, size); - return NULL; + munmap (addr, size); + addr = NULL; + } + } } - - return addr; } else - return malloc (size); + addr = malloc (size); + + if (addr && init0) + { + gomp_debug (0, " init0\n"); + memset (addr, 0, size); + } + + return addr; } static void * linux_memspace_calloc (omp_memspace_handle_t memspace, size_t size, int pin) { + gomp_debug (0, "%s: memspace=%llu, size=%llu, pin=%d\n", + __FUNCTION__, (unsigned long long) memspace, (unsigned long long) size, pin); + /* Explicit pinning may not be required. */ pin = pin && !always_pinned_mode; if (pin) - return linux_memspace_alloc (memspace, size, pin); + return linux_memspace_alloc (memspace, size, pin, true); else return calloc (1, size); } @@ -120,13 +168,25 @@ static void linux_memspace_free (omp_memspace_handle_t memspace, void *addr, size_t size, int pin) { - (void)memspace; + gomp_debug (0, "%s: memspace=%llu, addr=%p, size=%llu, pin=%d\n", + __FUNCTION__, (unsigned long long) memspace, addr, (unsigned long long) size, pin); /* Explicit pinning may not be required. */ pin = pin && !always_pinned_mode; if (pin) - munmap (addr, size); + { + int using_device + = __atomic_load_n (&using_device_for_page_locked, + MEMMODEL_RELAXED); + gomp_debug (0, " using_device=%d\n", + using_device); + if (using_device == 1) + gomp_page_locked_host_free (addr); + else + /* 'munlock'ing is implicit with following 'munmap'. */ + munmap (addr, size); + } else free (addr); } @@ -135,11 +195,25 @@ static void * linux_memspace_realloc (omp_memspace_handle_t memspace, void *addr, size_t oldsize, size_t size, int oldpin, int pin) { + gomp_debug (0, "%s: memspace=%llu, addr=%p, oldsize=%llu, size=%llu, oldpin=%d, pin=%d\n", + __FUNCTION__, (unsigned long long) memspace, addr, (unsigned long long) oldsize, (unsigned long long) size, oldpin, pin); + /* Explicit pinning may not be required. */ pin = pin && !always_pinned_mode; if (oldpin && pin) { + /* We can only expect to be able to just 'mremap' if not using a device + for page-locked memory. */ + int using_device + = __atomic_load_n (&using_device_for_page_locked, + MEMMODEL_RELAXED); + gomp_debug (0, " using_device=%d\n", + using_device); + if (using_device != 0) + goto manual_realloc; + + gomp_debug (0, " mremap\n"); void *newaddr = mremap (addr, oldsize, size, MREMAP_MAYMOVE); if (newaddr == MAP_FAILED) return NULL; @@ -147,18 +221,19 @@ linux_memspace_realloc (omp_memspace_handle_t memspace, void *addr, return newaddr; } else if (oldpin || pin) - { - void *newaddr = linux_memspace_alloc (memspace, size, pin); - if (newaddr) - { - memcpy (newaddr, addr, oldsize < size ? oldsize : size); - linux_memspace_free (memspace, addr, oldsize, oldpin); - } - - return newaddr; - } + goto manual_realloc; else return realloc (addr, size); + +manual_realloc:; + void *newaddr = linux_memspace_alloc (memspace, size, pin, false); + if (newaddr) + { + memcpy (newaddr, addr, oldsize < size ? oldsize : size); + linux_memspace_free (memspace, addr, oldsize, oldpin); + } + + return newaddr; } static int @@ -169,7 +244,7 @@ linux_memspace_validate (omp_memspace_handle_t, unsigned, int) } #define MEMSPACE_ALLOC(MEMSPACE, SIZE, PIN) \ - linux_memspace_alloc (MEMSPACE, SIZE, PIN) + linux_memspace_alloc (MEMSPACE, SIZE, PIN, false) #define MEMSPACE_CALLOC(MEMSPACE, SIZE, PIN) \ linux_memspace_calloc (MEMSPACE, SIZE, PIN) #define MEMSPACE_REALLOC(MEMSPACE, ADDR, OLDSIZE, SIZE, OLDPIN, PIN) \ diff --git a/libgomp/libgomp-plugin.h b/libgomp/libgomp-plugin.h index 0c9c28c65cf..0d2e3f0a6ec 100644 --- a/libgomp/libgomp-plugin.h +++ b/libgomp/libgomp-plugin.h @@ -139,6 +139,8 @@ extern int GOMP_OFFLOAD_load_image (int, unsigned, const void *, extern bool GOMP_OFFLOAD_unload_image (int, unsigned, const void *); extern void *GOMP_OFFLOAD_alloc (int, size_t); extern bool GOMP_OFFLOAD_free (int, void *); +extern bool GOMP_OFFLOAD_page_locked_host_alloc (void **, size_t); +extern bool GOMP_OFFLOAD_page_locked_host_free (void *); extern bool GOMP_OFFLOAD_dev2host (int, void *, const void *, size_t); extern bool GOMP_OFFLOAD_host2dev (int, void *, const void *, size_t); extern bool GOMP_OFFLOAD_dev2dev (int, void *, const void *, size_t); diff --git a/libgomp/libgomp.h b/libgomp/libgomp.h index 089393846d1..b8c6fc00de4 100644 --- a/libgomp/libgomp.h +++ b/libgomp/libgomp.h @@ -1128,6 +1128,8 @@ extern int gomp_get_num_devices (void); extern bool gomp_target_task_fn (void *); extern void gomp_target_rev (uint64_t, uint64_t, uint64_t, uint64_t, uint64_t, int, struct goacc_asyncqueue *); +extern bool gomp_page_locked_host_alloc (void **, size_t); +extern void gomp_page_locked_host_free (void *); /* Splay tree definitions. */ typedef struct splay_tree_node_s *splay_tree_node; @@ -1409,6 +1411,8 @@ struct gomp_device_descr __typeof (GOMP_OFFLOAD_unload_image) *unload_image_func; __typeof (GOMP_OFFLOAD_alloc) *alloc_func; __typeof (GOMP_OFFLOAD_free) *free_func; + __typeof (GOMP_OFFLOAD_page_locked_host_alloc) *page_locked_host_alloc_func; + __typeof (GOMP_OFFLOAD_page_locked_host_free) *page_locked_host_free_func; __typeof (GOMP_OFFLOAD_dev2host) *dev2host_func; __typeof (GOMP_OFFLOAD_host2dev) *host2dev_func; __typeof (GOMP_OFFLOAD_memcpy2d) *memcpy2d_func; diff --git a/libgomp/libgomp.texi b/libgomp/libgomp.texi index b3abb3dbdd9..eed95640001 100644 --- a/libgomp/libgomp.texi +++ b/libgomp/libgomp.texi @@ -6285,8 +6285,9 @@ a @code{nearest} allocation. Additional notes regarding the traits: @itemize -@item The @code{pinned} trait is supported on Linux hosts, but is subject to - the OS @code{ulimit}/@code{rlimit} locked memory settings. +@item The @code{pinned} trait is supported on Linux hosts, but is usually + subject to the OS @code{ulimit}/@code{rlimit} locked memory settings (see + @ref{Offload-Target Specifics} for exceptions). @item The default for the @code{pool_size} trait is no pool and for every (re)allocation the associated library routine is called, which might internally use a memory pool. @@ -6474,6 +6475,12 @@ The implementation remark: @code{omp_thread_mem_alloc}, all use low-latency memory as first preference, and fall back to main graphics memory when the low-latency pool is exhausted. +@item Pinned memory allocated using @code{omp_alloc} with the + @code{ompx_gnu_pinned_mem_alloc} allocator or the @code{pinned} trait is + obtained via the CUDA API when an NVPTX device is present. This provides + a performance boost for NVPTX offload code and also allows unlimited use + of pinned memory regardless of the OS @code{ulimit}/@code{rlimit} + settings. @end itemize diff --git a/libgomp/libgomp_g.h b/libgomp/libgomp_g.h index c0cc03ae61f..74c0fd17784 100644 --- a/libgomp/libgomp_g.h +++ b/libgomp/libgomp_g.h @@ -371,6 +371,7 @@ extern bool GOMP_is_alloc (void *); extern void *GOMP_alloc (size_t, size_t, uintptr_t); extern void GOMP_free (void *, uintptr_t); +extern void GOMP_enable_pinned_mode (void); /* error.c */ diff --git a/libgomp/plugin/plugin-nvptx.c b/libgomp/plugin/plugin-nvptx.c index 4cedc5390a3..5caf8fe5026 100644 --- a/libgomp/plugin/plugin-nvptx.c +++ b/libgomp/plugin/plugin-nvptx.c @@ -1625,6 +1625,48 @@ GOMP_OFFLOAD_free (int ord, void *ptr) && nvptx_free (ptr, ptx_devices[ord])); } +bool +GOMP_OFFLOAD_page_locked_host_alloc (void **ptr, size_t size) +{ + GOMP_PLUGIN_debug (0, "nvptx %s: ptr=%p, size=%llu\n", + __FUNCTION__, ptr, (unsigned long long) size); + + if (size == 0) + { + /* Special case to ensure omp_alloc specification compliance. */ + *ptr = NULL; + GOMP_PLUGIN_debug (0, " -> *ptr=null\n"); + return true; + } + + CUresult r; + + unsigned int flags = 0; + /* Given 'CU_DEVICE_ATTRIBUTE_UNIFIED_ADDRESSING', we don't need + 'flags |= CU_MEMHOSTALLOC_PORTABLE;' here. */ + r = CUDA_CALL_NOCHECK (cuMemHostAlloc, ptr, size, flags); + if (r == CUDA_ERROR_OUT_OF_MEMORY) + *ptr = NULL; + else if (r != CUDA_SUCCESS) + { + GOMP_PLUGIN_error ("cuMemHostAlloc error: %s", cuda_error (r)); + return false; + } + GOMP_PLUGIN_debug (0, " -> *ptr=%p\n", + *ptr); + return true; +} + +bool +GOMP_OFFLOAD_page_locked_host_free (void *ptr) +{ + GOMP_PLUGIN_debug (0, "nvptx %s: ptr=%p\n", + __FUNCTION__, ptr); + + CUDA_CALL (cuMemFreeHost, ptr); + return true; +} + void GOMP_OFFLOAD_openacc_exec (void (*fn) (void *), size_t mapnum __attribute__((unused)), diff --git a/libgomp/target.c b/libgomp/target.c index 48689920d4a..adbd18101f2 100644 --- a/libgomp/target.c +++ b/libgomp/target.c @@ -4437,6 +4437,140 @@ omp_target_free (void *device_ptr, int device_num) gomp_mutex_unlock (&devicep->lock); } +/* Device (really: libgomp plugin) to use for paged-locked memory. We + assume there is either none or exactly one such device for the lifetime of + the process. */ + +static struct gomp_device_descr *device_for_page_locked + = /* uninitialized */ (void *) -1; + +static struct gomp_device_descr * +get_device_for_page_locked (void) +{ + gomp_debug (0, "%s\n", + __FUNCTION__); + + struct gomp_device_descr *device; +#ifdef HAVE_SYNC_BUILTINS + device + = __atomic_load_n (&device_for_page_locked, MEMMODEL_RELAXED); + if (device == (void *) -1) + { + gomp_debug (0, " init\n"); + + gomp_init_targets_once (); + + device = NULL; + for (int i = 0; i < num_devices; ++i) + { + gomp_debug (0, " i=%d, target_id=%d\n", + i, devices[i].target_id); + + /* We consider only the first device of potentially several of the + same type as this functionality is not specific to an individual + offloading device, but instead relates to the host-side + implementation of the respective offloading implementation. */ + if (devices[i].target_id != 0) + continue; + + if (!devices[i].page_locked_host_alloc_func) + continue; + + gomp_debug (0, " found device: %p (%s)\n", + &devices[i], devices[i].name); + if (device) + gomp_fatal ("Unclear how %s and %s libgomp plugins may" + " simultaneously provide functionality" + " for page-locked memory", + device->name, devices[i].name); + else + device = &devices[i]; + } + + struct gomp_device_descr *device_old + = __atomic_exchange_n (&device_for_page_locked, device, + MEMMODEL_RELAXED); + gomp_debug (0, " old device_for_page_locked: %p\n", + device_old); + assert (device_old == (void *) -1 + /* We shouldn't have concurrently found a different or no + device. */ + || device_old == device); + } +#else /* !HAVE_SYNC_BUILTINS */ + gomp_debug (0, " not implemented for '!HAVE_SYNC_BUILTINS'\n"); + (void) &device_for_page_locked; + device = NULL; +#endif /* HAVE_SYNC_BUILTINS */ + + gomp_debug (0, " -> device=%p (%s)\n", + device, device ? device->name : "[none]"); + return device; +} + +/* Allocate page-locked host memory. + Returns whether we have a device capable of that. */ + +attribute_hidden bool +gomp_page_locked_host_alloc (void **ptr, size_t size) +{ + gomp_debug (0, "%s: ptr=%p, size=%llu\n", + __FUNCTION__, ptr, (unsigned long long) size); + + struct gomp_device_descr *device = get_device_for_page_locked (); + gomp_debug (0, " device=%p (%s)\n", + device, device ? device->name : "[none]"); + if (device) + { + gomp_mutex_lock (&device->lock); + if (device->state == GOMP_DEVICE_UNINITIALIZED) + gomp_init_device (device); + else if (device->state == GOMP_DEVICE_FINALIZED) + { + gomp_mutex_unlock (&device->lock); + gomp_fatal ("Device %s used for for page-locked memory is finalized", + device->name); + } + gomp_mutex_unlock (&device->lock); + + if (!device->page_locked_host_alloc_func (ptr, size)) + gomp_fatal ("Failed to allocate page-locked host memory" + " via %s libgomp plugin", + device->name); + } + return device != NULL; +} + +/* Free page-locked host memory. + This must only be called if 'gomp_page_locked_host_alloc' returned + 'true'. */ + +attribute_hidden void +gomp_page_locked_host_free (void *ptr) +{ + gomp_debug (0, "%s: ptr=%p\n", + __FUNCTION__, ptr); + + struct gomp_device_descr *device = get_device_for_page_locked (); + gomp_debug (0, " device=%p (%s)\n", + device, device ? device->name : "[none]"); + assert (device); + + gomp_mutex_lock (&device->lock); + assert (device->state != GOMP_DEVICE_UNINITIALIZED); + if (device->state == GOMP_DEVICE_FINALIZED) + { + gomp_mutex_unlock (&device->lock); + return; + } + gomp_mutex_unlock (&device->lock); + + if (!device->page_locked_host_free_func (ptr)) + gomp_fatal ("Failed to free page-locked host memory" + " via %s libgomp plugin", + device->name); +} + int omp_target_is_present (const void *ptr, int device_num) { @@ -5159,6 +5293,8 @@ gomp_load_plugin_for_device (struct gomp_device_descr *device, DLSYM (unload_image); DLSYM (alloc); DLSYM (free); + DLSYM_OPT (page_locked_host_alloc, page_locked_host_alloc); + DLSYM_OPT (page_locked_host_free, page_locked_host_free); DLSYM (dev2host); DLSYM (host2dev); DLSYM_OPT (memcpy2d, memcpy2d); diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-1.c b/libgomp/testsuite/libgomp.c/alloc-pinned-1.c index 672f2453a78..693f9032c8d 100644 --- a/libgomp/testsuite/libgomp.c/alloc-pinned-1.c +++ b/libgomp/testsuite/libgomp.c/alloc-pinned-1.c @@ -2,6 +2,8 @@ /* { dg-skip-if "Pinning not implemented on this host" { ! *-*-linux-gnu* } } */ +/* { dg-additional-options -DOFFLOAD_DEVICE_NVPTX { target offload_device_nvptx } } */ + /* Test that pinned memory works. */ #include @@ -63,10 +65,16 @@ verify0 (char *p, size_t s) int main () { +#ifdef OFFLOAD_DEVICE_NVPTX + /* Go big or go home. + The OS ulimit does not affect memory locked via CUDA for NVPTX devices. */ + const int SIZE = 40 * 1024 * 1024; +#else /* Allocate at least a page each time, allowing space for overhead, but stay within the ulimit. */ const int SIZE = PAGE_SIZE - 128; CHECK_SIZE (SIZE * 5); // This is intended to help diagnose failures +#endif const omp_alloctrait_t traits[] = { { omp_atk_pinned, 1 } @@ -88,21 +96,39 @@ main () abort (); int amount = get_pinned_mem (); +#ifdef OFFLOAD_DEVICE_NVPTX + /* This doesn't show up as process 'VmLck'ed memory. */ + if (amount != 0) + abort (); +#else if (amount == 0) abort (); +#endif p = omp_realloc (p, SIZE * 2, allocator, allocator); int amount2 = get_pinned_mem (); +#ifdef OFFLOAD_DEVICE_NVPTX + /* This doesn't show up as process 'VmLck'ed memory. */ + if (amount2 != 0) + abort (); +#else if (amount2 <= amount) abort (); +#endif /* SIZE*2 ensures that it doesn't slot into the space possibly vacated by realloc. */ p = omp_calloc (1, SIZE * 2, allocator); +#ifdef OFFLOAD_DEVICE_NVPTX + /* This doesn't show up as process 'VmLck'ed memory. */ + if (get_pinned_mem () != 0) + abort (); +#else if (get_pinned_mem () <= amount2) abort (); +#endif verify0 (p, SIZE * 2); diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-2.c b/libgomp/testsuite/libgomp.c/alloc-pinned-2.c index b6d1d83fb6f..e7ac64e911f 100644 --- a/libgomp/testsuite/libgomp.c/alloc-pinned-2.c +++ b/libgomp/testsuite/libgomp.c/alloc-pinned-2.c @@ -2,6 +2,8 @@ /* { dg-skip-if "Pinning not implemented on this host" { ! *-*-linux-gnu* } } */ +/* { dg-additional-options -DOFFLOAD_DEVICE_NVPTX { target offload_device_nvptx } } */ + /* Test that pinned memory works (pool_size code path). */ #include @@ -63,10 +65,16 @@ verify0 (char *p, size_t s) int main () { +#ifdef OFFLOAD_DEVICE_NVPTX + /* Go big or go home. + The OS ulimit does not affect memory locked via CUDA for NVPTX devices. */ + const int SIZE = 40 * 1024 * 1024; +#else /* Allocate at least a page each time, allowing space for overhead, but stay within the ulimit. */ const int SIZE = PAGE_SIZE - 128; CHECK_SIZE (SIZE * 5); // This is intended to help diagnose failures +#endif const omp_alloctrait_t traits[] = { { omp_atk_pinned, 1 }, @@ -89,16 +97,28 @@ main () abort (); int amount = get_pinned_mem (); +#ifdef OFFLOAD_DEVICE_NVPTX + /* This doesn't show up as process 'VmLck'ed memory. */ + if (amount != 0) + abort (); +#else if (amount == 0) abort (); +#endif p = omp_realloc (p, SIZE * 2, allocator, allocator); if (!p) abort (); int amount2 = get_pinned_mem (); +#ifdef OFFLOAD_DEVICE_NVPTX + /* This doesn't show up as process 'VmLck'ed memory. */ + if (amount2 != 0) + abort (); +#else if (amount2 <= amount) abort (); +#endif /* SIZE*2 ensures that it doesn't slot into the space possibly vacated by realloc. */ @@ -106,8 +126,14 @@ main () if (!p) abort (); +#ifdef OFFLOAD_DEVICE_NVPTX + /* This doesn't show up as process 'VmLck'ed memory. */ + if (get_pinned_mem () != 0) + abort (); +#else if (get_pinned_mem () <= amount2) abort (); +#endif verify0 (p, SIZE * 2); diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-3.c b/libgomp/testsuite/libgomp.c/alloc-pinned-3.c index 53e4720cc9c..b17daf7e222 100644 --- a/libgomp/testsuite/libgomp.c/alloc-pinned-3.c +++ b/libgomp/testsuite/libgomp.c/alloc-pinned-3.c @@ -1,5 +1,7 @@ /* { dg-do run } */ +/* { dg-additional-options -DOFFLOAD_DEVICE_NVPTX { target offload_device_nvptx } } */ + /* Test that pinned memory fails correctly. */ #include @@ -75,8 +77,15 @@ verify0 (char *p, size_t s) int main () { +#ifdef OFFLOAD_DEVICE_NVPTX + /* Go big or go home. + The OS ulimit does not affect memory locked via CUDA for NVPTX devices. */ + const int SIZE = 40 * 1024 * 1024; +#else /* This needs to be large enough to cover multiple pages. */ const int SIZE = PAGE_SIZE * 4; +#endif + const int PIN_LIMIT = PAGE_SIZE * 2; /* Pinned memory, no fallback. */ const omp_alloctrait_t traits1[] = { @@ -101,23 +110,34 @@ main () #endif /* Ensure that the limit is smaller than the allocation. */ - set_pin_limit (SIZE / 2); + set_pin_limit (PIN_LIMIT); // Sanity check if (get_pinned_mem () != 0) abort (); - // Should fail void *p1 = omp_alloc (SIZE, allocator1); +#ifdef OFFLOAD_DEVICE_NVPTX + // Doesn't care about 'set_pin_limit'. + if (!p1) + abort (); +#else + // Should fail if (p1) abort (); +#endif - // Should fail void *p2 = omp_calloc (1, SIZE, allocator1); +#ifdef OFFLOAD_DEVICE_NVPTX + // Doesn't care about 'set_pin_limit'. + if (!p2) + abort (); +#else + // Should fail if (p2) abort (); +#endif - // Should fall back void *p3 = omp_alloc (SIZE, allocator2); if (!p3) abort (); @@ -128,16 +148,29 @@ main () abort (); verify0 (p4, SIZE); - // Should fail to realloc void *notpinned = omp_alloc (SIZE, omp_default_mem_alloc); void *p5 = omp_realloc (notpinned, SIZE, allocator1, omp_default_mem_alloc); +#ifdef OFFLOAD_DEVICE_NVPTX + // Doesn't care about 'set_pin_limit'; does reallocate. + if (!notpinned || !p5 || p5 == notpinned) + abort (); +#else + // Should fail to realloc if (!notpinned || p5) abort (); +#endif - // Should fall back to no realloc needed +#ifdef OFFLOAD_DEVICE_NVPTX + void *p6 = omp_realloc (p5, SIZE, allocator2, allocator1); + // Does reallocate. + if (p5 == p6) + abort (); +#else void *p6 = omp_realloc (notpinned, SIZE, allocator2, omp_default_mem_alloc); + // Should fall back to no realloc needed if (p6 != notpinned) abort (); +#endif // No memory should have been pinned int amount = get_pinned_mem (); diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-4.c b/libgomp/testsuite/libgomp.c/alloc-pinned-4.c index 9d850c23e4b..aadd9249ba9 100644 --- a/libgomp/testsuite/libgomp.c/alloc-pinned-4.c +++ b/libgomp/testsuite/libgomp.c/alloc-pinned-4.c @@ -1,5 +1,7 @@ /* { dg-do run } */ +/* { dg-additional-options -DOFFLOAD_DEVICE_NVPTX { target offload_device_nvptx } } */ + /* Test that pinned memory fails correctly, pool_size code path. */ #include @@ -75,8 +77,15 @@ verify0 (char *p, size_t s) int main () { +#ifdef OFFLOAD_DEVICE_NVPTX + /* Go big or go home. + The OS ulimit does not affect memory locked via CUDA for NVPTX devices. */ + const int SIZE = 40 * 1024 * 1024; +#else /* This needs to be large enough to cover multiple pages. */ const int SIZE = PAGE_SIZE * 4; +#endif + const int PIN_LIMIT = PAGE_SIZE * 2; /* Pinned memory, no fallback. */ const omp_alloctrait_t traits1[] = { @@ -103,21 +112,33 @@ main () #endif /* Ensure that the limit is smaller than the allocation. */ - set_pin_limit (SIZE / 2); + set_pin_limit (PIN_LIMIT); // Sanity check if (get_pinned_mem () != 0) abort (); - // Should fail void *p = omp_alloc (SIZE, allocator1); +#ifdef OFFLOAD_DEVICE_NVPTX + // Doesn't care about 'set_pin_limit'. + if (!p) + abort (); +#else + // Should fail if (p) abort (); +#endif - // Should fail p = omp_calloc (1, SIZE, allocator1); +#ifdef OFFLOAD_DEVICE_NVPTX + // Doesn't care about 'set_pin_limit'. + if (!p) + abort (); +#else + // Should fail if (p) abort (); +#endif // Should fall back p = omp_alloc (SIZE, allocator2); @@ -130,16 +151,29 @@ main () abort (); verify0 (p, SIZE); - // Should fail to realloc void *notpinned = omp_alloc (SIZE, omp_default_mem_alloc); p = omp_realloc (notpinned, SIZE, allocator1, omp_default_mem_alloc); +#ifdef OFFLOAD_DEVICE_NVPTX + // Doesn't care about 'set_pin_limit'; does reallocate. + if (!notpinned || !p || p == notpinned) + abort (); +#else + // Should fail to realloc if (!notpinned || p) abort (); +#endif - // Should fall back to no realloc needed +#ifdef OFFLOAD_DEVICE_NVPTX + void *p_ = omp_realloc (p, SIZE, allocator2, allocator1); + // Does reallocate. + if (p_ == p) + abort (); +#else p = omp_realloc (notpinned, SIZE, allocator2, omp_default_mem_alloc); + // Should fall back to no realloc needed if (p != notpinned) abort (); +#endif // No memory should have been pinned int amount = get_pinned_mem (); diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-5.c b/libgomp/testsuite/libgomp.c/alloc-pinned-5.c index 0ba2feb8d96..cc7776409ff 100644 --- a/libgomp/testsuite/libgomp.c/alloc-pinned-5.c +++ b/libgomp/testsuite/libgomp.c/alloc-pinned-5.c @@ -2,6 +2,8 @@ /* { dg-skip-if "Pinning not implemented on this host" { ! *-*-linux-gnu* } } */ +/* { dg-additional-options -DOFFLOAD_DEVICE_NVPTX { target offload_device_nvptx } } */ + /* Test that ompx_gnu_pinned_mem_alloc works. */ #include @@ -63,10 +65,16 @@ verify0 (char *p, size_t s) int main () { +#ifdef OFFLOAD_DEVICE_NVPTX + /* Go big or go home. + The OS ulimit does not affect memory locked via CUDA for NVPTX devices. */ + const int SIZE = 40 * 1024 * 1024; +#else /* Allocate at least a page each time, allowing space for overhead, but stay within the ulimit. */ const int SIZE = PAGE_SIZE - 128; CHECK_SIZE (SIZE * 5); +#endif // Sanity check if (get_pinned_mem () != 0) @@ -77,22 +85,40 @@ main () abort (); int amount = get_pinned_mem (); +#ifdef OFFLOAD_DEVICE_NVPTX + /* This doesn't show up as process 'VmLck'ed memory. */ + if (amount != 0) + abort (); +#else if (amount == 0) abort (); +#endif p = omp_realloc (p, SIZE * 2, ompx_gnu_pinned_mem_alloc, ompx_gnu_pinned_mem_alloc); int amount2 = get_pinned_mem (); +#ifdef OFFLOAD_DEVICE_NVPTX + /* This doesn't show up as process 'VmLck'ed memory. */ + if (amount2 != 0) + abort (); +#else if (amount2 <= amount) abort (); +#endif /* SIZE*2 ensures that it doesn't slot into the space possibly vacated by realloc. */ p = omp_calloc (1, SIZE * 2, ompx_gnu_pinned_mem_alloc); +#ifdef OFFLOAD_DEVICE_NVPTX + /* This doesn't show up as process 'VmLck'ed memory. */ + if (get_pinned_mem () != 0) + abort (); +#else if (get_pinned_mem () <= amount2) abort (); +#endif verify0 (p, SIZE * 2); diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-6.c b/libgomp/testsuite/libgomp.c/alloc-pinned-6.c index 7ed68740ae6..44aca12bad1 100644 --- a/libgomp/testsuite/libgomp.c/alloc-pinned-6.c +++ b/libgomp/testsuite/libgomp.c/alloc-pinned-6.c @@ -1,4 +1,5 @@ /* { dg-do run } */ +/* { dg-additional-options -DOFFLOAD_DEVICE_NVPTX { target offload_device_nvptx } } */ /* Test that ompx_gnu_pinned_mem_alloc fails correctly. */ @@ -66,32 +67,57 @@ set_pin_limit () int main () { +#ifdef OFFLOAD_DEVICE_NVPTX + /* Go big or go home. + The OS ulimit does not affect memory locked via CUDA for NVPTX devices. */ + const int SIZE = 40 * 1024 * 1024; +#else /* Allocate at least a page each time, but stay within the ulimit. */ const int SIZE = PAGE_SIZE * 4; +#endif + const int PIN_LIMIT = PAGE_SIZE*2; /* Ensure that the limit is smaller than the allocation. */ - set_pin_limit (SIZE / 2); + set_pin_limit (PIN_LIMIT); // Sanity check if (get_pinned_mem () != 0) abort (); - // Should fail void *p = omp_alloc (SIZE, ompx_gnu_pinned_mem_alloc); +#ifdef OFFLOAD_DEVICE_NVPTX + // Doesn't care about 'set_pin_limit'. + if (!p) + abort (); +#else + // Should fail if (p) abort (); +#endif - // Should fail p = omp_calloc (1, SIZE, ompx_gnu_pinned_mem_alloc); +#ifdef OFFLOAD_DEVICE_NVPTX + // Doesn't care about 'set_pin_limit'. + if (!p) + abort (); +#else + // Should fail if (p) abort (); +#endif - // Should fail to realloc void *notpinned = omp_alloc (SIZE, omp_default_mem_alloc); p = omp_realloc (notpinned, SIZE, ompx_gnu_pinned_mem_alloc, omp_default_mem_alloc); +#ifdef OFFLOAD_DEVICE_NVPTX + // Doesn't care about 'set_pin_limit'; does reallocate. + if (!notpinned || !p || p == notpinned) + abort (); +#else + // Should fail to realloc if (!notpinned || p) abort (); +#endif // No memory should have been pinned int amount = get_pinned_mem (); From patchwork Wed Jun 12 11:42:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Stubbs X-Patchwork-Id: 1946819 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=baylibre-com.20230601.gappssmtp.com header.i=@baylibre-com.20230601.gappssmtp.com header.a=rsa-sha256 header.s=20230601 header.b=IgNsRqap; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VzkGT4t3wz20Wd for ; Wed, 12 Jun 2024 21:44:17 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id B537E38845AD for ; Wed, 12 Jun 2024 11:44:09 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail-lj1-x22d.google.com (mail-lj1-x22d.google.com [IPv6:2a00:1450:4864:20::22d]) by sourceware.org (Postfix) with ESMTPS id 58C9D3877015 for ; Wed, 12 Jun 2024 11:42:42 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 58C9D3877015 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=baylibre.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=baylibre.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 58C9D3877015 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2a00:1450:4864:20::22d ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1718192567; cv=none; b=D79EM4HGtZnvnnCOopOA/MK4J7dcUPgqT+0mEdlizq+G5FrwpQK2OAaZLTSc1jCYArUZks+RynwlOL3efqT2f5mrdghOOy+qalA+hqVI9898Vrg6F4oMhg9Qybbbrh0HONDEadFjdmVTMAPoPxzI7W+uN3rLvAc6Nee8nOO6lYk= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1718192567; c=relaxed/simple; bh=IGH8idELJUu4ErTCq0opDfad9MCBmdkJXCkdLNPo3iM=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=rvrD/KyxuxeQI8ZM2WEFRhh8aUrq2iAoo1wd6sNxSwN0CpniprSWePyIn7JUpzY4xCHVLHgIFTjn/T6jrYVQ9ARKd5ixZH0C7mFnf6hi6PFlSAEgbD8we9k/uoWfBEsCi76I4bjy/gu1wZAXFOMbyL7GPsHiFdbGLalTTGChw/Y= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-lj1-x22d.google.com with SMTP id 38308e7fff4ca-2eaea28868dso67028031fa.3 for ; Wed, 12 Jun 2024 04:42:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1718192560; x=1718797360; darn=gcc.gnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tziQM1DfvIERIB3Ek23QOsgCGCHCeX5j3KL3nfl3Lo0=; b=IgNsRqapACJDazIxemXBJ297IzPGD6PrEzR8fhQAXmLWiTpKliGMFTZJbt0QtmMdkT CQHyEdjt80ruoZeFtnai+pYtDq6qEGG7IsieK4AL16PnjO+tOGrIwZ0CjZQWJsYnaatm qBfTb5XzFGMVtyeb9zbZbRa7u9Sioi2de9qvijDcnKW025Wt3CakMIJnXAfigl61sMWa ohim92gJ+Aofp5ePwpqX5pGIA7zzYsX30G+PRp2eLQVPfiP9H7NYW/Pgukd3TmnKrU0f i8ULmy0/4yoAV1gxq0aXiaTlp777uGGFNTVMDai8z1C93UROeYFHsZiP2gbkhdZOZIl3 BAgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718192560; x=1718797360; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tziQM1DfvIERIB3Ek23QOsgCGCHCeX5j3KL3nfl3Lo0=; b=et+FJZT17EwTsfZS0g4L7P8FhS+myINQ/0i0nxIMGIBVo3nAf6OSLBN4g/JMzi76vG SXfyySzWpIeL/I5BpezVJ8tCdWPVuEpxGtwNYbaUs1twcvj0bCcGAdZHwWHuD0zY04/5 DTivi9dqxtNXLfm0x6Uk4Ykn/GI3cjsFosJnq0imx7bY1FZulv9ZyJ0IhMdwtCsNUV4m iZm3OJl0TDYQurRfVkYws0gberMYj1Ov+4atU4QDYPR1g2HjyZHJBgU+4zC1FfNUU92o SH0k7aYrGPTxfz/+1JRSxJrALF8LxRxehDfI377KLJ3yTtAvlYWgf28olJn9Pix90veq TMAA== X-Gm-Message-State: AOJu0YzbQ1j9nBC4u8pJR2zhAhZSDnZyGy+lho9H2/65UMUfDixg6ViQ aNJAky9dMus4pjTqYC3W75O/1UOnC94hRM2n6agkegVpVoJtOrSSyVOKLLiDb0mtmgCfrDVLQyY D9xM= X-Google-Smtp-Source: AGHT+IGRgJExnOmL6s6DCdE2MK4UHcGbk0JgfBgoOMI6wyX4qMaM2qj8zM214CWxB9OtR/Bipolz5A== X-Received: by 2002:a2e:a7d0:0:b0:2eb:fea7:4519 with SMTP id 38308e7fff4ca-2ebfea74620mr11091091fa.11.1718192560275; Wed, 12 Jun 2024 04:42:40 -0700 (PDT) Received: from arnold.baylibre (laubervilliers-658-1-213-31.w90-63.abo.wanadoo.fr. [90.63.244.31]) by smtp.googlemail.com with ESMTPSA id ffacd0b85a97d-35f1aa8d4f3sm11049246f8f.99.2024.06.12.04.42.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Jun 2024 04:42:39 -0700 (PDT) From: Andrew Stubbs To: gcc-patches@gcc.gnu.org Cc: tburnus@baylibre.com, jakub@redhat.com Subject: [PATCH v5 6/6] libgomp: fine-grained pinned memory allocator Date: Wed, 12 Jun 2024 11:42:23 +0000 Message-ID: <20240612114223.2206735-7-ams@baylibre.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20240612114223.2206735-1-ams@baylibre.com> References: <20240612114223.2206735-1-ams@baylibre.com> MIME-Version: 1.0 X-Spam-Status: No, score=-10.4 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org This patch introduces a new custom memory allocator for use with pinned memory (in the case where the Cuda allocator isn't available). In future, this allocator will also be used for Unified Shared Memory. Both memories are incompatible with the system malloc because allocated memory cannot share a page with memory allocated for other purposes. This means that small allocations will no longer consume an entire page of pinned memory. Unfortunately, it also means that pinned memory pages will never be unmapped (although they may be reused). The implementation is not perfect; there are various corner cases (especially related to extending onto new pages) where allocations and reallocations may be sub-optimal, but it should still be a step forward in support for small allocations. I have considered using libmemkind's "fixed" memory but rejected it for three reasons: 1) libmemkind may not always be present at runtime, 2) there's no currently documented means to extend a "fixed" kind one page at a time (although the code appears to have an undocumented function that may do the job, and/or extending libmemkind to support the MAP_LOCKED mmap flag with its regular kinds would be straight-forward), 3) USM benefits from having the metadata located in different memory and using an external implementation makes it hard to guarantee this. libgomp/ChangeLog: * Makefile.am (libgomp_la_SOURCES): Add usmpin-allocator.c. * Makefile.in: Regenerate. * config/linux/allocator.c: Include unistd.h. (pin_ctx): New variable. (ctxlock): New variable. (linux_init_pin_ctx): New function. (linux_memspace_alloc): Use usmpin-allocator for pinned memory. (linux_memspace_free): Likewise. (linux_memspace_realloc): Likewise. * libgomp.h (usmpin_init_context): New prototype. (usmpin_register_memory): New prototype. (usmpin_alloc): New prototype. (usmpin_free): New prototype. (usmpin_realloc): New prototype. * testsuite/libgomp.c/alloc-pinned-1.c: Adjust for new behaviour. * testsuite/libgomp.c/alloc-pinned-2.c: Likewise. * testsuite/libgomp.c/alloc-pinned-5.c: Likewise. * testsuite/libgomp.c/alloc-pinned-8.c: New test. * usmpin-allocator.c: New file. --- libgomp/Makefile.am | 2 +- libgomp/Makefile.in | 7 +- libgomp/config/linux/allocator.c | 97 ++++-- libgomp/libgomp.h | 10 + libgomp/testsuite/libgomp.c/alloc-pinned-8.c | 122 +++++++ libgomp/usmpin-allocator.c | 319 +++++++++++++++++++ 6 files changed, 522 insertions(+), 35 deletions(-) create mode 100644 libgomp/testsuite/libgomp.c/alloc-pinned-8.c create mode 100644 libgomp/usmpin-allocator.c diff --git a/libgomp/Makefile.am b/libgomp/Makefile.am index 855f0affddf..73c21699332 100644 --- a/libgomp/Makefile.am +++ b/libgomp/Makefile.am @@ -70,7 +70,7 @@ libgomp_la_SOURCES = alloc.c atomic.c barrier.c critical.c env.c error.c \ target.c splay-tree.c libgomp-plugin.c oacc-parallel.c oacc-host.c \ oacc-init.c oacc-mem.c oacc-async.c oacc-plugin.c oacc-cuda.c \ priority_queue.c affinity-fmt.c teams.c allocator.c oacc-profiling.c \ - oacc-target.c target-indirect.c + oacc-target.c target-indirect.c usmpin-allocator.c include $(top_srcdir)/plugin/Makefrag.am diff --git a/libgomp/Makefile.in b/libgomp/Makefile.in index da902f3daca..b74e39a1c2a 100644 --- a/libgomp/Makefile.in +++ b/libgomp/Makefile.in @@ -219,7 +219,8 @@ am_libgomp_la_OBJECTS = alloc.lo atomic.lo barrier.lo critical.lo \ oacc-parallel.lo oacc-host.lo oacc-init.lo oacc-mem.lo \ oacc-async.lo oacc-plugin.lo oacc-cuda.lo priority_queue.lo \ affinity-fmt.lo teams.lo allocator.lo oacc-profiling.lo \ - oacc-target.lo target-indirect.lo $(am__objects_1) + oacc-target.lo target-indirect.lo usmpin-allocator.lo \ + $(am__objects_1) libgomp_la_OBJECTS = $(am_libgomp_la_OBJECTS) AM_V_P = $(am__v_P_@AM_V@) am__v_P_ = $(am__v_P_@AM_DEFAULT_V@) @@ -552,7 +553,8 @@ libgomp_la_SOURCES = alloc.c atomic.c barrier.c critical.c env.c \ oacc-parallel.c oacc-host.c oacc-init.c oacc-mem.c \ oacc-async.c oacc-plugin.c oacc-cuda.c priority_queue.c \ affinity-fmt.c teams.c allocator.c oacc-profiling.c \ - oacc-target.c target-indirect.c $(am__append_3) + oacc-target.c target-indirect.c usmpin-allocator.c \ + $(am__append_3) # Nvidia PTX OpenACC plugin. @PLUGIN_NVPTX_TRUE@libgomp_plugin_nvptx_version_info = -version-info $(libtool_VERSION) @@ -786,6 +788,7 @@ distclean-compile: @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/team.Plo@am__quote@ @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/teams.Plo@am__quote@ @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/time.Plo@am__quote@ +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/usmpin-allocator.Plo@am__quote@ @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/work.Plo@am__quote@ .c.o: diff --git a/libgomp/config/linux/allocator.c b/libgomp/config/linux/allocator.c index 063c46f972c..04214973cc9 100644 --- a/libgomp/config/linux/allocator.c +++ b/libgomp/config/linux/allocator.c @@ -53,6 +53,7 @@ #define _GNU_SOURCE #include +#include #include #include #include "libgomp.h" @@ -77,6 +78,16 @@ GOMP_enable_pinned_mode () static int using_device_for_page_locked = /* uninitialized */ -1; + +static usmpin_ctx_p pin_ctx = NULL; +static pthread_once_t ctxlock = PTHREAD_ONCE_INIT; + +static void +linux_init_pin_ctx () +{ + pin_ctx = usmpin_init_context (); +} + static void * linux_memspace_alloc (omp_memspace_handle_t memspace, size_t size, int pin, bool init0) @@ -85,7 +96,7 @@ linux_memspace_alloc (omp_memspace_handle_t memspace, size_t size, int pin, __FUNCTION__, (unsigned long long) memspace, (unsigned long long) size, pin, init0); - void *addr; + void *addr = NULL; /* Explicit pinning may not be required. */ pin = pin && !always_pinned_mode; @@ -111,28 +122,51 @@ linux_memspace_alloc (omp_memspace_handle_t memspace, size_t size, int pin, } if (using_device == 0) { - gomp_debug (0, " mmap\n"); - addr = mmap (NULL, size, PROT_READ | PROT_WRITE, - MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); - if (addr == MAP_FAILED) - addr = NULL; - else - { - /* 'mmap' zero-initializes. */ - init0 = false; + static int pagesize = 0; + static void *addrhint = NULL; - gomp_debug (0, " mlock\n"); - if (mlock (addr, size)) + if (!pagesize) + pagesize = sysconf(_SC_PAGE_SIZE); + + while (1) + { + addr = usmpin_alloc (pin_ctx, size); + if (addr) + break; + + gomp_debug (0, " mmap\n"); + + /* Round up to a whole page. */ + size_t misalignment = size % pagesize; + size_t mmap_size = (misalignment > 0 + ? size + pagesize - misalignment + : size); + void *newpage = mmap (addrhint, mmap_size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (newpage == MAP_FAILED) + break; + else { + gomp_debug (0, " mlock\n"); + if (mlock (newpage, size)) + { #ifdef HAVE_INTTYPES_H - gomp_debug (0, "libgomp: failed to pin %"PRIu64" bytes of" - " memory (ulimit too low?)\n", (uint64_t) size); + gomp_debug (0, "libgomp: failed to pin %"PRIu64" bytes" + " of memory (ulimit too low?)\n", + (uint64_t) size); #else - gomp_debug (0, "libgomp: failed to pin %lu bytes of memory" - " (ulimit too low?)\n", (unsigned long) size); + gomp_debug (0, "libgomp: failed to pin %lu bytes of" + " memory (ulimit too low?)\n", + (unsigned long) size); #endif - munmap (addr, size); - addr = NULL; + munmap (newpage, size); + break; + } + + addrhint = newpage + mmap_size; + + pthread_once (&ctxlock, linux_init_pin_ctx); + usmpin_register_memory (pin_ctx, newpage, mmap_size); } } } @@ -184,8 +218,7 @@ linux_memspace_free (omp_memspace_handle_t memspace, void *addr, size_t size, if (using_device == 1) gomp_page_locked_host_free (addr); else - /* 'munlock'ing is implicit with following 'munmap'. */ - munmap (addr, size); + usmpin_free (pin_ctx, addr); } else free (addr); @@ -203,29 +236,29 @@ linux_memspace_realloc (omp_memspace_handle_t memspace, void *addr, if (oldpin && pin) { - /* We can only expect to be able to just 'mremap' if not using a device - for page-locked memory. */ int using_device = __atomic_load_n (&using_device_for_page_locked, MEMMODEL_RELAXED); gomp_debug (0, " using_device=%d\n", using_device); - if (using_device != 0) - goto manual_realloc; - - gomp_debug (0, " mremap\n"); - void *newaddr = mremap (addr, oldsize, size, MREMAP_MAYMOVE); - if (newaddr == MAP_FAILED) - return NULL; - return newaddr; + /* The device plugin API does not support realloc, + but the usmpin allocator does. */ + if (using_device == 0) + { + /* This can fail if there is insufficient pinned memory free. */ + void *newaddr = usmpin_realloc (pin_ctx, addr, size); + if (newaddr) + return newaddr; + } } else if (oldpin || pin) - goto manual_realloc; + /* Moving from pinned to unpinned memory cannot be done in-place. */ + ; else return realloc (addr, size); -manual_realloc:; + /* In-place reallocation failed. Fall back to copy. */ void *newaddr = linux_memspace_alloc (memspace, size, pin, false); if (newaddr) { diff --git a/libgomp/libgomp.h b/libgomp/libgomp.h index b8c6fc00de4..c3aabd4b7d3 100644 --- a/libgomp/libgomp.h +++ b/libgomp/libgomp.h @@ -1658,4 +1658,14 @@ gomp_thread_to_pthread_t (struct gomp_thread *thr) } #endif +/* usmpin-allocator.c */ + +typedef struct usmpin_context *usmpin_ctx_p; + +usmpin_ctx_p usmpin_init_context (); +void usmpin_register_memory (usmpin_ctx_p ctx, char *base, size_t size); +void *usmpin_alloc (usmpin_ctx_p ctx, size_t size); +void usmpin_free (usmpin_ctx_p ctx, void *addr); +void *usmpin_realloc (usmpin_ctx_p ctx, void *addr, size_t newsize); + #endif /* LIBGOMP_H */ diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-8.c b/libgomp/testsuite/libgomp.c/alloc-pinned-8.c new file mode 100644 index 00000000000..0fc737b1e2a --- /dev/null +++ b/libgomp/testsuite/libgomp.c/alloc-pinned-8.c @@ -0,0 +1,122 @@ +/* { dg-do run } */ + +/* { dg-skip-if "Pinning not implemented on this host" { ! *-*-linux-gnu* } } */ + +/* { dg-additional-options -DOFFLOAD_DEVICE_NVPTX { target offload_device_nvptx } } */ + +/* Test that pinned memory works for small allocations. */ + +#include +#include + +#ifdef __linux__ +#include +#include + +#include +#include + +#define PAGE_SIZE sysconf(_SC_PAGESIZE) +#define CHECK_SIZE(SIZE) { \ + struct rlimit limit; \ + if (getrlimit (RLIMIT_MEMLOCK, &limit) \ + || limit.rlim_cur <= SIZE) \ + fprintf (stderr, "insufficient lockable memory; please increase ulimit\n"); \ + } + +int +get_pinned_mem () +{ + int pid = getpid (); + char buf[100]; + sprintf (buf, "/proc/%d/status", pid); + + FILE *proc = fopen (buf, "r"); + if (!proc) + abort (); + while (fgets (buf, 100, proc)) + { + int val; + if (sscanf (buf, "VmLck: %d", &val)) + { + fclose (proc); + return val; + } + } + abort (); +} +#else +#error "OS unsupported" +#endif + +static void +verify0 (char *p, size_t s) +{ + for (size_t i = 0; i < s; ++i) + if (p[i] != 0) + abort (); +} + +#include + +int +main () +{ + /* Choose a small size where all our allocations fit on one page. */ + const int SIZE = 10; +#ifndef OFFLOAD_DEVICE_NVPTX + CHECK_SIZE (SIZE*4); +#endif + + const omp_alloctrait_t traits[] = { + { omp_atk_pinned, 1 } + }; + omp_allocator_handle_t allocator = omp_init_allocator (omp_default_mem_space, 1, traits); + + // Sanity check + if (get_pinned_mem () != 0) + abort (); + + void *p = omp_alloc (SIZE, allocator); + if (!p) + abort (); + + int amount = get_pinned_mem (); +#ifdef OFFLOAD_DEVICE_NVPTX + /* This doesn't show up as process 'VmLck'ed memory. */ + if (amount != 0) + abort (); +#else + if (amount == 0) + abort (); +#endif + + p = omp_realloc (p, SIZE * 2, allocator, allocator); + + int amount2 = get_pinned_mem (); +#ifdef OFFLOAD_DEVICE_NVPTX + /* This doesn't show up as process 'VmLck'ed memory. */ + if (amount2 != 0) + abort (); +#else + /* A small allocation should not allocate another page. */ + if (amount2 != amount) + abort (); +#endif + + p = omp_calloc (1, SIZE, allocator); + +#ifdef OFFLOAD_DEVICE_NVPTX + /* This doesn't show up as process 'VmLck'ed memory. */ + if (get_pinned_mem () != 0) + abort (); +#else + /* A small allocation should not allocate another page. */ + if (get_pinned_mem () != amount2) + abort (); +#endif + + verify0 (p, SIZE); + + return 0; +} diff --git a/libgomp/usmpin-allocator.c b/libgomp/usmpin-allocator.c new file mode 100644 index 00000000000..311bda5054e --- /dev/null +++ b/libgomp/usmpin-allocator.c @@ -0,0 +1,319 @@ +/* Copyright (C) 2023 Free Software Foundation, Inc. + + This file is part of the GNU Offloading and Multi Processing Library + (libgomp). + + Libgomp is free software; you can redistribute it and/or modify it + under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 3, or (at your option) + any later version. + + Libgomp is distributed in the hope that it will be useful, but WITHOUT ANY + WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS + FOR A PARTICULAR PURPOSE. See the GNU General Public License for + more details. + + Under Section 7 of GPL version 3, you are granted additional + permissions described in the GCC Runtime Library Exception, version + 3.1, as published by the Free Software Foundation. + + You should have received a copy of the GNU General Public License and + a copy of the GCC Runtime Library Exception along with this program; + see the files COPYING3 and COPYING.RUNTIME respectively. If not, see + . */ + +/* This is a simple "malloc" implementation intended for use with Unified + Shared Memory and Pinned Memory. It allocates memory from a pool allocated + and configured by the device plugin (for USM), or the OS-specific allocator + (for pinned). + + This implementation keeps the allocated/free chain in a side-table (splay + tree) to ensure that the allocation routine does not migrate all the USM + pages back into host memory. Keeping the meta-data elsewhere is also useful + for pinned memory, which is typically an extremely limited resource. */ + +#include +#include "libgomp.h" + +/* Use a splay tree to track allocations. */ + +typedef struct usmpin_splay_tree_node_s *usmpin_splay_tree_node; +typedef struct usmpin_splay_tree_s *usmpin_splay_tree; +typedef struct usmpin_splay_tree_key_s *usmpin_splay_tree_key; + +struct usmpin_splay_tree_key_s { + void *base; + size_t size; +}; + +static inline int +usmpin_splay_compare (usmpin_splay_tree_key x, usmpin_splay_tree_key y) +{ + return (x->base == y->base ? 0 + : x->base > y->base ? 1 + : -1); +} + +#define splay_tree_prefix usmpin +#include "splay-tree.h" + +/* 128-byte granularity means GPU cache-line aligned. */ +#define ALIGN(VAR) (((VAR) + 127) & ~127) + +/* The context data prevents the need for global state. */ +struct usmpin_context { + int lock; + struct usmpin_splay_tree_s allocations; + struct usmpin_splay_tree_s free_space; +}; + +usmpin_ctx_p +usmpin_init_context () +{ + return calloc (1, sizeof (struct usmpin_context)); +} + +/* Coalesce contiguous free space into one entry. This considers the entries + either side of the root node only, so it should be called each time a new + entry in inserted into the root. */ + +static void +usmpin_coalesce_free_space (usmpin_ctx_p ctx) +{ + usmpin_splay_tree_node prev, next, node = ctx->free_space.root; + + for (prev = node->left; prev && prev->right; prev = prev->right) + ; + for (next = node->right; next && next->left; next = next->left) + ; + + /* Coalesce adjacent free chunks. */ + if (next + && node->key.base + node->key.size == next->key.base) + { + /* Free chunk follows. */ + node->key.size += next->key.size; + usmpin_splay_tree_remove (&ctx->free_space, &next->key); + free (next); + } + if (prev + && prev->key.base + prev->key.size == node->key.base) + { + /* Free chunk precedes. */ + prev->key.size += node->key.size; + usmpin_splay_tree_remove (&ctx->free_space, &node->key); + free (node); + } +} + +/* Add a new memory region into the free chain. This is how the USM heap is + initialized and extended. If the new region is contiguous with an existing + region then any free space will be coalesced. */ + +void +usmpin_register_memory (usmpin_ctx_p ctx, char *base, size_t size) +{ + if (base == NULL || ctx == NULL) + return; + + while (__atomic_exchange_n (&ctx->lock, 1, MEMMODEL_ACQUIRE) == 1) + ; + + usmpin_splay_tree_node node; + node = malloc (sizeof (struct usmpin_splay_tree_node_s)); + node->key.base = base; + node->key.size = size; + node->left = NULL; + node->right = NULL; + usmpin_splay_tree_insert (&ctx->free_space, node); + usmpin_coalesce_free_space (ctx); + + __atomic_store_n (&ctx->lock, 0, MEMMODEL_RELEASE); +} + +/* This splay_tree_foreach callback selects the first free space large enough + to hold the allocation needed. Since the splay_tree walk may start in the + middle the "first" isn't necessarily the "leftmost" entry. */ + +struct usmpin_callback_data { + size_t size; + usmpin_splay_tree_node found; +}; + +static int +usmpin_alloc_callback (usmpin_splay_tree_key key, void *data) +{ + struct usmpin_callback_data *cbd = (struct usmpin_callback_data *)data; + + if (key->size >= cbd->size) + { + cbd->found = (usmpin_splay_tree_node)key; + return 1; + } + + return 0; +} + +/* USM "malloc". Selects and moves and address range from ctx->free_space to + ctx->allocations, while leaving any excess in ctx->free_space. */ + +void * +usmpin_alloc (usmpin_ctx_p ctx, size_t size) +{ + if (ctx == NULL) + return NULL; + + /* Memory is allocated in N-byte granularity. */ + size = ALIGN (size); + + /* Acquire the lock. */ + while (__atomic_exchange_n (&ctx->lock, 1, MEMMODEL_ACQUIRE) == 1) + ; + + if (!ctx->free_space.root) + { + /* No memory registered, or no free space. */ + __atomic_store_n (&ctx->lock, 0, MEMMODEL_RELEASE); + return NULL; + } + + /* Find a suitable free block. */ + struct usmpin_callback_data cbd = {size, NULL}; + usmpin_splay_tree_foreach_lazy (&ctx->free_space, usmpin_alloc_callback, + &cbd); + usmpin_splay_tree_node freenode = cbd.found; + + void *result = NULL; + if (freenode) + { + /* Allocation successful. */ + result = freenode->key.base; + usmpin_splay_tree_node allocnode = malloc (sizeof (*allocnode)); + allocnode->key.base = result; + allocnode->key.size = size; + allocnode->left = NULL; + allocnode->right = NULL; + usmpin_splay_tree_insert (&ctx->allocations, allocnode); + + /* Update the free chain. */ + size_t stillfree_size = freenode->key.size - size; + if (stillfree_size > 0) + { + freenode->key.base = freenode->key.base + size; + freenode->key.size = stillfree_size; + } + else + { + usmpin_splay_tree_remove (&ctx->free_space, &freenode->key); + free (freenode); + } + } + + /* Release the lock. */ + __atomic_store_n (&ctx->lock, 0, MEMMODEL_RELEASE); + + return result; +} + +/* USM "free". Moves an address range from ctx->allocations to + ctx->free_space and merges that record with any contiguous free memory. */ + +void +usmpin_free (usmpin_ctx_p ctx, void *addr) +{ + if (ctx == NULL) + return; + + /* Acquire the lock. */ + while (__atomic_exchange_n (&ctx->lock, 1, MEMMODEL_ACQUIRE) == 1) + ; + + /* Convert the memory map to free. */ + struct usmpin_splay_tree_key_s key = {addr}; + usmpin_splay_tree_key found = usmpin_splay_tree_lookup (&ctx->allocations, + &key); + if (!found) + GOMP_PLUGIN_fatal ("invalid free"); + usmpin_splay_tree_remove (&ctx->allocations, &key); + usmpin_splay_tree_insert (&ctx->free_space, (usmpin_splay_tree_node)found); + usmpin_coalesce_free_space (ctx); + + /* Release the lock. */ + __atomic_store_n (&ctx->lock, 0, MEMMODEL_RELEASE); +} + +/* USM "realloc". Works in-place, if possible; reallocates otherwise. */ + +void * +usmpin_realloc (usmpin_ctx_p ctx, void *addr, size_t newsize) +{ + if (ctx == NULL) + return NULL; + + newsize = ALIGN (newsize); + + /* Acquire the lock. */ + while (__atomic_exchange_n (&ctx->lock, 1, MEMMODEL_ACQUIRE) == 1) + ; + + /* Convert the memory map to free. */ + struct usmpin_splay_tree_key_s key = {addr}; + usmpin_splay_tree_key found = usmpin_splay_tree_lookup (&ctx->allocations, + &key); + if (!found) + GOMP_PLUGIN_fatal ("invalid realloc"); + + if (newsize == found->size) + ; /* Nothing to do. */ + else if (newsize < found->size) + { + /* We're reducing the allocation size. */ + usmpin_splay_tree_node newfree = malloc (sizeof (*newfree)); + newfree->key.base = found->base + newsize; + newfree->key.size = found->size - newsize; + newfree->left = NULL; + newfree->right = NULL; + usmpin_splay_tree_insert (&ctx->free_space, newfree); + usmpin_coalesce_free_space (ctx); + } + else + { + /* We're extending the allocation. */ + struct usmpin_splay_tree_key_s freekey = {addr + found->size}; + usmpin_splay_tree_key foundfree; + foundfree = usmpin_splay_tree_lookup (&ctx->free_space, &freekey); + if (foundfree && foundfree->size >= newsize - found->size) + { + /* Allocation can be expanded in place. */ + foundfree->base += found->size; + foundfree->size -= newsize - found->size; + found->size = newsize; + + if (foundfree->size == 0) + usmpin_splay_tree_remove (&ctx->free_space, &freekey); + } + else + { + /* Allocation must be relocated. + Release the lock and use alloc/free. */ + __atomic_store_n (&ctx->lock, 0, MEMMODEL_RELEASE); + + void *newaddr = usmpin_alloc (ctx, newsize); + if (!newaddr) + return NULL; + + memcpy (newaddr, addr, found->size); + usmpin_free (ctx, addr); + return newaddr; + } + } + + /* Release the lock. */ + __atomic_store_n (&ctx->lock, 0, MEMMODEL_RELEASE); + return addr; +} + +/* Include the splay tree code inline, with the prefixes added. */ +#define splay_tree_prefix usmpin +#define splay_tree_c +#include "splay-tree.h"