libgomp: Update OpenMP memory allocation doc, fix omp_high_bw_mem_space
libgomp/ChangeLog:
* allocator.c (omp_init_allocator): Use malloc for
omp_high_bw_mem_space when the memkind lib is unavailable instead
of returning omp_null_allocator.
* libgomp.texi (Memory allocation with libmemkind): Document
implementation in more details.
libgomp/allocator.c | 2 +-
libgomp/libgomp.texi | 26 +++++++++++++++++++++++++-
2 files changed, 26 insertions(+), 2 deletions(-)
@@ -301,7 +301,7 @@ omp_init_allocator (omp_memspace_handle_t memspace, int ntraits,
break;
}
#endif
- return omp_null_allocator;
+ break;
case omp_large_cap_mem_space:
#ifdef LIBGOMP_USE_MEMKIND
memkind_data = gomp_get_memkind ();
@@ -4634,6 +4634,17 @@ smaller number. On non-host devices, the value of the
@node Memory allocation with libmemkind
@section Memory allocation with libmemkind
+For the memory spaces, the following applies:
+@itemize
+@item @code{omp_default_mem_space} is supported
+@item @code{omp_const_mem_space} maps to @code{omp_default_mem_space}
+@item @code{omp_low_lat_mem_space} maps to @code{omp_default_mem_space}
+@item @code{omp_large_cap_mem_space} maps to @code{omp_default_mem_space},
+ unless the memkind library is available
+@item @code{omp_high_bw_mem_space} maps to @code{omp_default_mem_space},
+ unless the memkind library is available
+@end itemize
+
On Linux systems, where the @uref{https://github.com/memkind/memkind, memkind
library} (@code{libmemkind.so.0}) is available at runtime, it is used when
creating memory allocators requesting
@@ -4641,9 +4652,22 @@ creating memory allocators requesting
@itemize
@item the memory space @code{omp_high_bw_mem_space}
@item the memory space @code{omp_large_cap_mem_space}
-@item the partition trait @code{omp_atv_interleaved}
+@item the partition trait @code{omp_atv_interleaved}; note that for
+ @code{omp_large_cap_mem_space} the allocation will not be interleaved
@end itemize
+Additional notes:
+@itemize
+@item The @code{pinned} trait is unsupported.
+@item For the @code{partition} trait, the partition part size will be the same
+ as the requested size (i.e. @code{interleaved} or @code{blocked} has no
+ effect), except for @code{interleaved} when the memkind library is
+ available. Furthermore, @code{nearest} might not always return memory
+ on the node of the CPU that triggered an allocation.
+@item The @code{access} trait has no effect such that memory is always
+ accessible by all threads.
+@item The @code{sync_hint} trait has no effect.
+@end itemize
@c ---------------------------------------------------------------------
@c Offload-Target Specifics
I noted that all memory spaces are supported, some by falling back to the default ("malloc") - except for omp_high_bw_mem_space (unless the memkind lib is available). I think it makes more sense to fallback to 'malloc' also for omp_high_bw_mem_space. Additionally, I updated the documentation to more explicitly state what the current implementation is. Thoughts? Wording improvement suggestions? Tobias PS: I wonder whether it makes sense to use use libnuma besides libmemkind (which depends on libnuma); however, the question is when. libnuma provides numa_alloc_interleaved(_subset), numa_alloc_local and numa_alloc_onnode. In any case, something is odd here. I have two nodes, 0 and 1 (→ 'lscpu') and 'numactls --show' shows "preferred node: current". I allocate memory and then use the following to find the node: "get_mempolicy (&node, NULL, 0, ptr, MPOL_F_ADDR|MPOL_F_NODE)" Result: With malloc'ed data, it shows the same node as the node running the code (i.e. the same as 'getcpu (NULL, &node1);' == 'numa_node_of_cpu (sched_getcpu());'). But I get a constant result of 1 for numa_alloc_local and numa_alloc_onnode, independent of the passed node number (0 or 1) and on the CPU the thread runs on. ----------------- Siemens Electronic Design Automation GmbH; Anschrift: Arnulfstraße 201, 80634 München; Gesellschaft mit beschränkter Haftung; Geschäftsführer: Thomas Heurung, Frank Thürauf; Sitz der Gesellschaft: München; Registergericht München, HRB 106955