From patchwork Wed Jun 12 11:42:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Stubbs X-Patchwork-Id: 1946819 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=baylibre-com.20230601.gappssmtp.com header.i=@baylibre-com.20230601.gappssmtp.com header.a=rsa-sha256 header.s=20230601 header.b=IgNsRqap; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VzkGT4t3wz20Wd for ; Wed, 12 Jun 2024 21:44:17 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id B537E38845AD for ; Wed, 12 Jun 2024 11:44:09 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from mail-lj1-x22d.google.com (mail-lj1-x22d.google.com [IPv6:2a00:1450:4864:20::22d]) by sourceware.org (Postfix) with ESMTPS id 58C9D3877015 for ; Wed, 12 Jun 2024 11:42:42 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 58C9D3877015 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=baylibre.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=baylibre.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 58C9D3877015 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=2a00:1450:4864:20::22d ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1718192567; cv=none; b=D79EM4HGtZnvnnCOopOA/MK4J7dcUPgqT+0mEdlizq+G5FrwpQK2OAaZLTSc1jCYArUZks+RynwlOL3efqT2f5mrdghOOy+qalA+hqVI9898Vrg6F4oMhg9Qybbbrh0HONDEadFjdmVTMAPoPxzI7W+uN3rLvAc6Nee8nOO6lYk= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1718192567; c=relaxed/simple; bh=IGH8idELJUu4ErTCq0opDfad9MCBmdkJXCkdLNPo3iM=; h=DKIM-Signature:From:To:Subject:Date:Message-ID:MIME-Version; b=rvrD/KyxuxeQI8ZM2WEFRhh8aUrq2iAoo1wd6sNxSwN0CpniprSWePyIn7JUpzY4xCHVLHgIFTjn/T6jrYVQ9ARKd5ixZH0C7mFnf6hi6PFlSAEgbD8we9k/uoWfBEsCi76I4bjy/gu1wZAXFOMbyL7GPsHiFdbGLalTTGChw/Y= ARC-Authentication-Results: i=1; server2.sourceware.org Received: by mail-lj1-x22d.google.com with SMTP id 38308e7fff4ca-2eaea28868dso67028031fa.3 for ; Wed, 12 Jun 2024 04:42:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=baylibre-com.20230601.gappssmtp.com; s=20230601; t=1718192560; x=1718797360; darn=gcc.gnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tziQM1DfvIERIB3Ek23QOsgCGCHCeX5j3KL3nfl3Lo0=; b=IgNsRqapACJDazIxemXBJ297IzPGD6PrEzR8fhQAXmLWiTpKliGMFTZJbt0QtmMdkT CQHyEdjt80ruoZeFtnai+pYtDq6qEGG7IsieK4AL16PnjO+tOGrIwZ0CjZQWJsYnaatm qBfTb5XzFGMVtyeb9zbZbRa7u9Sioi2de9qvijDcnKW025Wt3CakMIJnXAfigl61sMWa ohim92gJ+Aofp5ePwpqX5pGIA7zzYsX30G+PRp2eLQVPfiP9H7NYW/Pgukd3TmnKrU0f i8ULmy0/4yoAV1gxq0aXiaTlp777uGGFNTVMDai8z1C93UROeYFHsZiP2gbkhdZOZIl3 BAgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718192560; x=1718797360; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tziQM1DfvIERIB3Ek23QOsgCGCHCeX5j3KL3nfl3Lo0=; b=et+FJZT17EwTsfZS0g4L7P8FhS+myINQ/0i0nxIMGIBVo3nAf6OSLBN4g/JMzi76vG SXfyySzWpIeL/I5BpezVJ8tCdWPVuEpxGtwNYbaUs1twcvj0bCcGAdZHwWHuD0zY04/5 DTivi9dqxtNXLfm0x6Uk4Ykn/GI3cjsFosJnq0imx7bY1FZulv9ZyJ0IhMdwtCsNUV4m iZm3OJl0TDYQurRfVkYws0gberMYj1Ov+4atU4QDYPR1g2HjyZHJBgU+4zC1FfNUU92o SH0k7aYrGPTxfz/+1JRSxJrALF8LxRxehDfI377KLJ3yTtAvlYWgf28olJn9Pix90veq TMAA== X-Gm-Message-State: AOJu0YzbQ1j9nBC4u8pJR2zhAhZSDnZyGy+lho9H2/65UMUfDixg6ViQ aNJAky9dMus4pjTqYC3W75O/1UOnC94hRM2n6agkegVpVoJtOrSSyVOKLLiDb0mtmgCfrDVLQyY D9xM= X-Google-Smtp-Source: AGHT+IGRgJExnOmL6s6DCdE2MK4UHcGbk0JgfBgoOMI6wyX4qMaM2qj8zM214CWxB9OtR/Bipolz5A== X-Received: by 2002:a2e:a7d0:0:b0:2eb:fea7:4519 with SMTP id 38308e7fff4ca-2ebfea74620mr11091091fa.11.1718192560275; Wed, 12 Jun 2024 04:42:40 -0700 (PDT) Received: from arnold.baylibre (laubervilliers-658-1-213-31.w90-63.abo.wanadoo.fr. [90.63.244.31]) by smtp.googlemail.com with ESMTPSA id ffacd0b85a97d-35f1aa8d4f3sm11049246f8f.99.2024.06.12.04.42.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Jun 2024 04:42:39 -0700 (PDT) From: Andrew Stubbs To: gcc-patches@gcc.gnu.org Cc: tburnus@baylibre.com, jakub@redhat.com Subject: [PATCH v5 6/6] libgomp: fine-grained pinned memory allocator Date: Wed, 12 Jun 2024 11:42:23 +0000 Message-ID: <20240612114223.2206735-7-ams@baylibre.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20240612114223.2206735-1-ams@baylibre.com> References: <20240612114223.2206735-1-ams@baylibre.com> MIME-Version: 1.0 X-Spam-Status: No, score=-10.4 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, SPF_HELO_NONE, SPF_PASS, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+incoming=patchwork.ozlabs.org@gcc.gnu.org This patch introduces a new custom memory allocator for use with pinned memory (in the case where the Cuda allocator isn't available). In future, this allocator will also be used for Unified Shared Memory. Both memories are incompatible with the system malloc because allocated memory cannot share a page with memory allocated for other purposes. This means that small allocations will no longer consume an entire page of pinned memory. Unfortunately, it also means that pinned memory pages will never be unmapped (although they may be reused). The implementation is not perfect; there are various corner cases (especially related to extending onto new pages) where allocations and reallocations may be sub-optimal, but it should still be a step forward in support for small allocations. I have considered using libmemkind's "fixed" memory but rejected it for three reasons: 1) libmemkind may not always be present at runtime, 2) there's no currently documented means to extend a "fixed" kind one page at a time (although the code appears to have an undocumented function that may do the job, and/or extending libmemkind to support the MAP_LOCKED mmap flag with its regular kinds would be straight-forward), 3) USM benefits from having the metadata located in different memory and using an external implementation makes it hard to guarantee this. libgomp/ChangeLog: * Makefile.am (libgomp_la_SOURCES): Add usmpin-allocator.c. * Makefile.in: Regenerate. * config/linux/allocator.c: Include unistd.h. (pin_ctx): New variable. (ctxlock): New variable. (linux_init_pin_ctx): New function. (linux_memspace_alloc): Use usmpin-allocator for pinned memory. (linux_memspace_free): Likewise. (linux_memspace_realloc): Likewise. * libgomp.h (usmpin_init_context): New prototype. (usmpin_register_memory): New prototype. (usmpin_alloc): New prototype. (usmpin_free): New prototype. (usmpin_realloc): New prototype. * testsuite/libgomp.c/alloc-pinned-1.c: Adjust for new behaviour. * testsuite/libgomp.c/alloc-pinned-2.c: Likewise. * testsuite/libgomp.c/alloc-pinned-5.c: Likewise. * testsuite/libgomp.c/alloc-pinned-8.c: New test. * usmpin-allocator.c: New file. --- libgomp/Makefile.am | 2 +- libgomp/Makefile.in | 7 +- libgomp/config/linux/allocator.c | 97 ++++-- libgomp/libgomp.h | 10 + libgomp/testsuite/libgomp.c/alloc-pinned-8.c | 122 +++++++ libgomp/usmpin-allocator.c | 319 +++++++++++++++++++ 6 files changed, 522 insertions(+), 35 deletions(-) create mode 100644 libgomp/testsuite/libgomp.c/alloc-pinned-8.c create mode 100644 libgomp/usmpin-allocator.c diff --git a/libgomp/Makefile.am b/libgomp/Makefile.am index 855f0affddf..73c21699332 100644 --- a/libgomp/Makefile.am +++ b/libgomp/Makefile.am @@ -70,7 +70,7 @@ libgomp_la_SOURCES = alloc.c atomic.c barrier.c critical.c env.c error.c \ target.c splay-tree.c libgomp-plugin.c oacc-parallel.c oacc-host.c \ oacc-init.c oacc-mem.c oacc-async.c oacc-plugin.c oacc-cuda.c \ priority_queue.c affinity-fmt.c teams.c allocator.c oacc-profiling.c \ - oacc-target.c target-indirect.c + oacc-target.c target-indirect.c usmpin-allocator.c include $(top_srcdir)/plugin/Makefrag.am diff --git a/libgomp/Makefile.in b/libgomp/Makefile.in index da902f3daca..b74e39a1c2a 100644 --- a/libgomp/Makefile.in +++ b/libgomp/Makefile.in @@ -219,7 +219,8 @@ am_libgomp_la_OBJECTS = alloc.lo atomic.lo barrier.lo critical.lo \ oacc-parallel.lo oacc-host.lo oacc-init.lo oacc-mem.lo \ oacc-async.lo oacc-plugin.lo oacc-cuda.lo priority_queue.lo \ affinity-fmt.lo teams.lo allocator.lo oacc-profiling.lo \ - oacc-target.lo target-indirect.lo $(am__objects_1) + oacc-target.lo target-indirect.lo usmpin-allocator.lo \ + $(am__objects_1) libgomp_la_OBJECTS = $(am_libgomp_la_OBJECTS) AM_V_P = $(am__v_P_@AM_V@) am__v_P_ = $(am__v_P_@AM_DEFAULT_V@) @@ -552,7 +553,8 @@ libgomp_la_SOURCES = alloc.c atomic.c barrier.c critical.c env.c \ oacc-parallel.c oacc-host.c oacc-init.c oacc-mem.c \ oacc-async.c oacc-plugin.c oacc-cuda.c priority_queue.c \ affinity-fmt.c teams.c allocator.c oacc-profiling.c \ - oacc-target.c target-indirect.c $(am__append_3) + oacc-target.c target-indirect.c usmpin-allocator.c \ + $(am__append_3) # Nvidia PTX OpenACC plugin. @PLUGIN_NVPTX_TRUE@libgomp_plugin_nvptx_version_info = -version-info $(libtool_VERSION) @@ -786,6 +788,7 @@ distclean-compile: @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/team.Plo@am__quote@ @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/teams.Plo@am__quote@ @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/time.Plo@am__quote@ +@AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/usmpin-allocator.Plo@am__quote@ @AMDEP_TRUE@@am__include@ @am__quote@./$(DEPDIR)/work.Plo@am__quote@ .c.o: diff --git a/libgomp/config/linux/allocator.c b/libgomp/config/linux/allocator.c index 063c46f972c..04214973cc9 100644 --- a/libgomp/config/linux/allocator.c +++ b/libgomp/config/linux/allocator.c @@ -53,6 +53,7 @@ #define _GNU_SOURCE #include +#include #include #include #include "libgomp.h" @@ -77,6 +78,16 @@ GOMP_enable_pinned_mode () static int using_device_for_page_locked = /* uninitialized */ -1; + +static usmpin_ctx_p pin_ctx = NULL; +static pthread_once_t ctxlock = PTHREAD_ONCE_INIT; + +static void +linux_init_pin_ctx () +{ + pin_ctx = usmpin_init_context (); +} + static void * linux_memspace_alloc (omp_memspace_handle_t memspace, size_t size, int pin, bool init0) @@ -85,7 +96,7 @@ linux_memspace_alloc (omp_memspace_handle_t memspace, size_t size, int pin, __FUNCTION__, (unsigned long long) memspace, (unsigned long long) size, pin, init0); - void *addr; + void *addr = NULL; /* Explicit pinning may not be required. */ pin = pin && !always_pinned_mode; @@ -111,28 +122,51 @@ linux_memspace_alloc (omp_memspace_handle_t memspace, size_t size, int pin, } if (using_device == 0) { - gomp_debug (0, " mmap\n"); - addr = mmap (NULL, size, PROT_READ | PROT_WRITE, - MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); - if (addr == MAP_FAILED) - addr = NULL; - else - { - /* 'mmap' zero-initializes. */ - init0 = false; + static int pagesize = 0; + static void *addrhint = NULL; - gomp_debug (0, " mlock\n"); - if (mlock (addr, size)) + if (!pagesize) + pagesize = sysconf(_SC_PAGE_SIZE); + + while (1) + { + addr = usmpin_alloc (pin_ctx, size); + if (addr) + break; + + gomp_debug (0, " mmap\n"); + + /* Round up to a whole page. */ + size_t misalignment = size % pagesize; + size_t mmap_size = (misalignment > 0 + ? size + pagesize - misalignment + : size); + void *newpage = mmap (addrhint, mmap_size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (newpage == MAP_FAILED) + break; + else { + gomp_debug (0, " mlock\n"); + if (mlock (newpage, size)) + { #ifdef HAVE_INTTYPES_H - gomp_debug (0, "libgomp: failed to pin %"PRIu64" bytes of" - " memory (ulimit too low?)\n", (uint64_t) size); + gomp_debug (0, "libgomp: failed to pin %"PRIu64" bytes" + " of memory (ulimit too low?)\n", + (uint64_t) size); #else - gomp_debug (0, "libgomp: failed to pin %lu bytes of memory" - " (ulimit too low?)\n", (unsigned long) size); + gomp_debug (0, "libgomp: failed to pin %lu bytes of" + " memory (ulimit too low?)\n", + (unsigned long) size); #endif - munmap (addr, size); - addr = NULL; + munmap (newpage, size); + break; + } + + addrhint = newpage + mmap_size; + + pthread_once (&ctxlock, linux_init_pin_ctx); + usmpin_register_memory (pin_ctx, newpage, mmap_size); } } } @@ -184,8 +218,7 @@ linux_memspace_free (omp_memspace_handle_t memspace, void *addr, size_t size, if (using_device == 1) gomp_page_locked_host_free (addr); else - /* 'munlock'ing is implicit with following 'munmap'. */ - munmap (addr, size); + usmpin_free (pin_ctx, addr); } else free (addr); @@ -203,29 +236,29 @@ linux_memspace_realloc (omp_memspace_handle_t memspace, void *addr, if (oldpin && pin) { - /* We can only expect to be able to just 'mremap' if not using a device - for page-locked memory. */ int using_device = __atomic_load_n (&using_device_for_page_locked, MEMMODEL_RELAXED); gomp_debug (0, " using_device=%d\n", using_device); - if (using_device != 0) - goto manual_realloc; - - gomp_debug (0, " mremap\n"); - void *newaddr = mremap (addr, oldsize, size, MREMAP_MAYMOVE); - if (newaddr == MAP_FAILED) - return NULL; - return newaddr; + /* The device plugin API does not support realloc, + but the usmpin allocator does. */ + if (using_device == 0) + { + /* This can fail if there is insufficient pinned memory free. */ + void *newaddr = usmpin_realloc (pin_ctx, addr, size); + if (newaddr) + return newaddr; + } } else if (oldpin || pin) - goto manual_realloc; + /* Moving from pinned to unpinned memory cannot be done in-place. */ + ; else return realloc (addr, size); -manual_realloc:; + /* In-place reallocation failed. Fall back to copy. */ void *newaddr = linux_memspace_alloc (memspace, size, pin, false); if (newaddr) { diff --git a/libgomp/libgomp.h b/libgomp/libgomp.h index b8c6fc00de4..c3aabd4b7d3 100644 --- a/libgomp/libgomp.h +++ b/libgomp/libgomp.h @@ -1658,4 +1658,14 @@ gomp_thread_to_pthread_t (struct gomp_thread *thr) } #endif +/* usmpin-allocator.c */ + +typedef struct usmpin_context *usmpin_ctx_p; + +usmpin_ctx_p usmpin_init_context (); +void usmpin_register_memory (usmpin_ctx_p ctx, char *base, size_t size); +void *usmpin_alloc (usmpin_ctx_p ctx, size_t size); +void usmpin_free (usmpin_ctx_p ctx, void *addr); +void *usmpin_realloc (usmpin_ctx_p ctx, void *addr, size_t newsize); + #endif /* LIBGOMP_H */ diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-8.c b/libgomp/testsuite/libgomp.c/alloc-pinned-8.c new file mode 100644 index 00000000000..0fc737b1e2a --- /dev/null +++ b/libgomp/testsuite/libgomp.c/alloc-pinned-8.c @@ -0,0 +1,122 @@ +/* { dg-do run } */ + +/* { dg-skip-if "Pinning not implemented on this host" { ! *-*-linux-gnu* } } */ + +/* { dg-additional-options -DOFFLOAD_DEVICE_NVPTX { target offload_device_nvptx } } */ + +/* Test that pinned memory works for small allocations. */ + +#include +#include + +#ifdef __linux__ +#include +#include + +#include +#include + +#define PAGE_SIZE sysconf(_SC_PAGESIZE) +#define CHECK_SIZE(SIZE) { \ + struct rlimit limit; \ + if (getrlimit (RLIMIT_MEMLOCK, &limit) \ + || limit.rlim_cur <= SIZE) \ + fprintf (stderr, "insufficient lockable memory; please increase ulimit\n"); \ + } + +int +get_pinned_mem () +{ + int pid = getpid (); + char buf[100]; + sprintf (buf, "/proc/%d/status", pid); + + FILE *proc = fopen (buf, "r"); + if (!proc) + abort (); + while (fgets (buf, 100, proc)) + { + int val; + if (sscanf (buf, "VmLck: %d", &val)) + { + fclose (proc); + return val; + } + } + abort (); +} +#else +#error "OS unsupported" +#endif + +static void +verify0 (char *p, size_t s) +{ + for (size_t i = 0; i < s; ++i) + if (p[i] != 0) + abort (); +} + +#include + +int +main () +{ + /* Choose a small size where all our allocations fit on one page. */ + const int SIZE = 10; +#ifndef OFFLOAD_DEVICE_NVPTX + CHECK_SIZE (SIZE*4); +#endif + + const omp_alloctrait_t traits[] = { + { omp_atk_pinned, 1 } + }; + omp_allocator_handle_t allocator = omp_init_allocator (omp_default_mem_space, 1, traits); + + // Sanity check + if (get_pinned_mem () != 0) + abort (); + + void *p = omp_alloc (SIZE, allocator); + if (!p) + abort (); + + int amount = get_pinned_mem (); +#ifdef OFFLOAD_DEVICE_NVPTX + /* This doesn't show up as process 'VmLck'ed memory. */ + if (amount != 0) + abort (); +#else + if (amount == 0) + abort (); +#endif + + p = omp_realloc (p, SIZE * 2, allocator, allocator); + + int amount2 = get_pinned_mem (); +#ifdef OFFLOAD_DEVICE_NVPTX + /* This doesn't show up as process 'VmLck'ed memory. */ + if (amount2 != 0) + abort (); +#else + /* A small allocation should not allocate another page. */ + if (amount2 != amount) + abort (); +#endif + + p = omp_calloc (1, SIZE, allocator); + +#ifdef OFFLOAD_DEVICE_NVPTX + /* This doesn't show up as process 'VmLck'ed memory. */ + if (get_pinned_mem () != 0) + abort (); +#else + /* A small allocation should not allocate another page. */ + if (get_pinned_mem () != amount2) + abort (); +#endif + + verify0 (p, SIZE); + + return 0; +} diff --git a/libgomp/usmpin-allocator.c b/libgomp/usmpin-allocator.c new file mode 100644 index 00000000000..311bda5054e --- /dev/null +++ b/libgomp/usmpin-allocator.c @@ -0,0 +1,319 @@ +/* Copyright (C) 2023 Free Software Foundation, Inc. + + This file is part of the GNU Offloading and Multi Processing Library + (libgomp). + + Libgomp is free software; you can redistribute it and/or modify it + under the terms of the GNU General Public License as published by + the Free Software Foundation; either version 3, or (at your option) + any later version. + + Libgomp is distributed in the hope that it will be useful, but WITHOUT ANY + WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS + FOR A PARTICULAR PURPOSE. See the GNU General Public License for + more details. + + Under Section 7 of GPL version 3, you are granted additional + permissions described in the GCC Runtime Library Exception, version + 3.1, as published by the Free Software Foundation. + + You should have received a copy of the GNU General Public License and + a copy of the GCC Runtime Library Exception along with this program; + see the files COPYING3 and COPYING.RUNTIME respectively. If not, see + . */ + +/* This is a simple "malloc" implementation intended for use with Unified + Shared Memory and Pinned Memory. It allocates memory from a pool allocated + and configured by the device plugin (for USM), or the OS-specific allocator + (for pinned). + + This implementation keeps the allocated/free chain in a side-table (splay + tree) to ensure that the allocation routine does not migrate all the USM + pages back into host memory. Keeping the meta-data elsewhere is also useful + for pinned memory, which is typically an extremely limited resource. */ + +#include +#include "libgomp.h" + +/* Use a splay tree to track allocations. */ + +typedef struct usmpin_splay_tree_node_s *usmpin_splay_tree_node; +typedef struct usmpin_splay_tree_s *usmpin_splay_tree; +typedef struct usmpin_splay_tree_key_s *usmpin_splay_tree_key; + +struct usmpin_splay_tree_key_s { + void *base; + size_t size; +}; + +static inline int +usmpin_splay_compare (usmpin_splay_tree_key x, usmpin_splay_tree_key y) +{ + return (x->base == y->base ? 0 + : x->base > y->base ? 1 + : -1); +} + +#define splay_tree_prefix usmpin +#include "splay-tree.h" + +/* 128-byte granularity means GPU cache-line aligned. */ +#define ALIGN(VAR) (((VAR) + 127) & ~127) + +/* The context data prevents the need for global state. */ +struct usmpin_context { + int lock; + struct usmpin_splay_tree_s allocations; + struct usmpin_splay_tree_s free_space; +}; + +usmpin_ctx_p +usmpin_init_context () +{ + return calloc (1, sizeof (struct usmpin_context)); +} + +/* Coalesce contiguous free space into one entry. This considers the entries + either side of the root node only, so it should be called each time a new + entry in inserted into the root. */ + +static void +usmpin_coalesce_free_space (usmpin_ctx_p ctx) +{ + usmpin_splay_tree_node prev, next, node = ctx->free_space.root; + + for (prev = node->left; prev && prev->right; prev = prev->right) + ; + for (next = node->right; next && next->left; next = next->left) + ; + + /* Coalesce adjacent free chunks. */ + if (next + && node->key.base + node->key.size == next->key.base) + { + /* Free chunk follows. */ + node->key.size += next->key.size; + usmpin_splay_tree_remove (&ctx->free_space, &next->key); + free (next); + } + if (prev + && prev->key.base + prev->key.size == node->key.base) + { + /* Free chunk precedes. */ + prev->key.size += node->key.size; + usmpin_splay_tree_remove (&ctx->free_space, &node->key); + free (node); + } +} + +/* Add a new memory region into the free chain. This is how the USM heap is + initialized and extended. If the new region is contiguous with an existing + region then any free space will be coalesced. */ + +void +usmpin_register_memory (usmpin_ctx_p ctx, char *base, size_t size) +{ + if (base == NULL || ctx == NULL) + return; + + while (__atomic_exchange_n (&ctx->lock, 1, MEMMODEL_ACQUIRE) == 1) + ; + + usmpin_splay_tree_node node; + node = malloc (sizeof (struct usmpin_splay_tree_node_s)); + node->key.base = base; + node->key.size = size; + node->left = NULL; + node->right = NULL; + usmpin_splay_tree_insert (&ctx->free_space, node); + usmpin_coalesce_free_space (ctx); + + __atomic_store_n (&ctx->lock, 0, MEMMODEL_RELEASE); +} + +/* This splay_tree_foreach callback selects the first free space large enough + to hold the allocation needed. Since the splay_tree walk may start in the + middle the "first" isn't necessarily the "leftmost" entry. */ + +struct usmpin_callback_data { + size_t size; + usmpin_splay_tree_node found; +}; + +static int +usmpin_alloc_callback (usmpin_splay_tree_key key, void *data) +{ + struct usmpin_callback_data *cbd = (struct usmpin_callback_data *)data; + + if (key->size >= cbd->size) + { + cbd->found = (usmpin_splay_tree_node)key; + return 1; + } + + return 0; +} + +/* USM "malloc". Selects and moves and address range from ctx->free_space to + ctx->allocations, while leaving any excess in ctx->free_space. */ + +void * +usmpin_alloc (usmpin_ctx_p ctx, size_t size) +{ + if (ctx == NULL) + return NULL; + + /* Memory is allocated in N-byte granularity. */ + size = ALIGN (size); + + /* Acquire the lock. */ + while (__atomic_exchange_n (&ctx->lock, 1, MEMMODEL_ACQUIRE) == 1) + ; + + if (!ctx->free_space.root) + { + /* No memory registered, or no free space. */ + __atomic_store_n (&ctx->lock, 0, MEMMODEL_RELEASE); + return NULL; + } + + /* Find a suitable free block. */ + struct usmpin_callback_data cbd = {size, NULL}; + usmpin_splay_tree_foreach_lazy (&ctx->free_space, usmpin_alloc_callback, + &cbd); + usmpin_splay_tree_node freenode = cbd.found; + + void *result = NULL; + if (freenode) + { + /* Allocation successful. */ + result = freenode->key.base; + usmpin_splay_tree_node allocnode = malloc (sizeof (*allocnode)); + allocnode->key.base = result; + allocnode->key.size = size; + allocnode->left = NULL; + allocnode->right = NULL; + usmpin_splay_tree_insert (&ctx->allocations, allocnode); + + /* Update the free chain. */ + size_t stillfree_size = freenode->key.size - size; + if (stillfree_size > 0) + { + freenode->key.base = freenode->key.base + size; + freenode->key.size = stillfree_size; + } + else + { + usmpin_splay_tree_remove (&ctx->free_space, &freenode->key); + free (freenode); + } + } + + /* Release the lock. */ + __atomic_store_n (&ctx->lock, 0, MEMMODEL_RELEASE); + + return result; +} + +/* USM "free". Moves an address range from ctx->allocations to + ctx->free_space and merges that record with any contiguous free memory. */ + +void +usmpin_free (usmpin_ctx_p ctx, void *addr) +{ + if (ctx == NULL) + return; + + /* Acquire the lock. */ + while (__atomic_exchange_n (&ctx->lock, 1, MEMMODEL_ACQUIRE) == 1) + ; + + /* Convert the memory map to free. */ + struct usmpin_splay_tree_key_s key = {addr}; + usmpin_splay_tree_key found = usmpin_splay_tree_lookup (&ctx->allocations, + &key); + if (!found) + GOMP_PLUGIN_fatal ("invalid free"); + usmpin_splay_tree_remove (&ctx->allocations, &key); + usmpin_splay_tree_insert (&ctx->free_space, (usmpin_splay_tree_node)found); + usmpin_coalesce_free_space (ctx); + + /* Release the lock. */ + __atomic_store_n (&ctx->lock, 0, MEMMODEL_RELEASE); +} + +/* USM "realloc". Works in-place, if possible; reallocates otherwise. */ + +void * +usmpin_realloc (usmpin_ctx_p ctx, void *addr, size_t newsize) +{ + if (ctx == NULL) + return NULL; + + newsize = ALIGN (newsize); + + /* Acquire the lock. */ + while (__atomic_exchange_n (&ctx->lock, 1, MEMMODEL_ACQUIRE) == 1) + ; + + /* Convert the memory map to free. */ + struct usmpin_splay_tree_key_s key = {addr}; + usmpin_splay_tree_key found = usmpin_splay_tree_lookup (&ctx->allocations, + &key); + if (!found) + GOMP_PLUGIN_fatal ("invalid realloc"); + + if (newsize == found->size) + ; /* Nothing to do. */ + else if (newsize < found->size) + { + /* We're reducing the allocation size. */ + usmpin_splay_tree_node newfree = malloc (sizeof (*newfree)); + newfree->key.base = found->base + newsize; + newfree->key.size = found->size - newsize; + newfree->left = NULL; + newfree->right = NULL; + usmpin_splay_tree_insert (&ctx->free_space, newfree); + usmpin_coalesce_free_space (ctx); + } + else + { + /* We're extending the allocation. */ + struct usmpin_splay_tree_key_s freekey = {addr + found->size}; + usmpin_splay_tree_key foundfree; + foundfree = usmpin_splay_tree_lookup (&ctx->free_space, &freekey); + if (foundfree && foundfree->size >= newsize - found->size) + { + /* Allocation can be expanded in place. */ + foundfree->base += found->size; + foundfree->size -= newsize - found->size; + found->size = newsize; + + if (foundfree->size == 0) + usmpin_splay_tree_remove (&ctx->free_space, &freekey); + } + else + { + /* Allocation must be relocated. + Release the lock and use alloc/free. */ + __atomic_store_n (&ctx->lock, 0, MEMMODEL_RELEASE); + + void *newaddr = usmpin_alloc (ctx, newsize); + if (!newaddr) + return NULL; + + memcpy (newaddr, addr, found->size); + usmpin_free (ctx, addr); + return newaddr; + } + } + + /* Release the lock. */ + __atomic_store_n (&ctx->lock, 0, MEMMODEL_RELEASE); + return addr; +} + +/* Include the splay tree code inline, with the prefixes added. */ +#define splay_tree_prefix usmpin +#define splay_tree_c +#include "splay-tree.h"