From patchwork Wed Jul 31 14:02:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Biener X-Patchwork-Id: 1967121 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=suse.de header.i=@suse.de header.a=rsa-sha256 header.s=susede2_rsa header.b=vANZHFGa; dkim=pass header.d=suse.de header.i=@suse.de header.a=ed25519-sha256 header.s=susede2_ed25519 header.b=RH/o6zp0; dkim=pass (1024-bit key) header.d=suse.de header.i=@suse.de header.a=rsa-sha256 header.s=susede2_rsa header.b=vANZHFGa; dkim=neutral header.d=suse.de header.i=@suse.de header.a=ed25519-sha256 header.s=susede2_ed25519 header.b=RH/o6zp0; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=8.43.85.97; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces~incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WYv1m2Lmbz1ybX for ; Thu, 1 Aug 2024 00:02:52 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 78E233858288 for ; Wed, 31 Jul 2024 14:02:50 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by sourceware.org (Postfix) with ESMTPS id 1E8233858C56 for ; Wed, 31 Jul 2024 14:02:24 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 1E8233858C56 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=suse.de Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=suse.de ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 1E8233858C56 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=195.135.223.131 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1722434547; cv=none; b=gtPq1KdMXHTLpTMFhSV/Q1DU4q33T/KDtZln544+WNUB/V+qmT1WYJpb+jr4nCG7w09zXB3BpZ9jdSRykVzHf/cRcDop6slK7+DxFx2BHdPKeBxdy4E7MY7CItjr/qYkAfOrzvh+nZcTdkZAVqyi0JxFuKbYKymSwRiVa5LNMmQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1722434547; c=relaxed/simple; bh=67r3OVEbS16r07RG3RWBepWPJpZbzME52qAafGe+nGU=; h=DKIM-Signature:DKIM-Signature:DKIM-Signature:DKIM-Signature:Date: From:To:Subject:MIME-Version; b=UC2N+WaRJqhyy4IMZ9xDsCRHG7kywUi9uYeQapSP5CLmDpBsde+QnBsaJ2WwwL8wUpGofsx9w88Y4eipu/2GYYpgcjbkYSaNQXb7i7nA7lI6mH9Yhp5bFH7E7cZhfgsVdPKTP5ZfsMVggMKZi1yFc76UeZL6gp+7NGTk6WZqTBw= ARC-Authentication-Results: i=1; server2.sourceware.org Received: from murzim.nue2.suse.org (unknown [10.168.4.243]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 044711F7E8; Wed, 31 Jul 2024 14:02:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1722434543; h=from:from:reply-to:date:date:to:to:cc:cc:mime-version:mime-version: content-type:content-type; bh=pbQHRNDR8wLFIpXZQKkozG4yukCdwJPnBEw/VwYGBT0=; b=vANZHFGaKX6FzL4JbPn60aCTcKcPi8q1ZjxvfxBXtLoEbVIiMkj7feaBXgste9PZNO16RN HwNKs1BzhP1OMCoH6IOzm+Qosk8fcQJhuy0OxEXk/zAIpfXvwKjmbA/veJm7ceGX/ENtKf +5FFwm5DFMEfYoe8QkPIbhQTPsPfEaQ= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1722434543; h=from:from:reply-to:date:date:to:to:cc:cc:mime-version:mime-version: content-type:content-type; bh=pbQHRNDR8wLFIpXZQKkozG4yukCdwJPnBEw/VwYGBT0=; b=RH/o6zp0ZlH05PXxoEZOy/96slzUgWMoq1Roxhjnn/tkaItAWIrqYOMmpZRgEkSafE+nZL iQcPNNsrhjv7TqBQ== Authentication-Results: smtp-out2.suse.de; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1722434543; h=from:from:reply-to:date:date:to:to:cc:cc:mime-version:mime-version: content-type:content-type; bh=pbQHRNDR8wLFIpXZQKkozG4yukCdwJPnBEw/VwYGBT0=; b=vANZHFGaKX6FzL4JbPn60aCTcKcPi8q1ZjxvfxBXtLoEbVIiMkj7feaBXgste9PZNO16RN HwNKs1BzhP1OMCoH6IOzm+Qosk8fcQJhuy0OxEXk/zAIpfXvwKjmbA/veJm7ceGX/ENtKf +5FFwm5DFMEfYoe8QkPIbhQTPsPfEaQ= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1722434543; h=from:from:reply-to:date:date:to:to:cc:cc:mime-version:mime-version: content-type:content-type; bh=pbQHRNDR8wLFIpXZQKkozG4yukCdwJPnBEw/VwYGBT0=; b=RH/o6zp0ZlH05PXxoEZOy/96slzUgWMoq1Roxhjnn/tkaItAWIrqYOMmpZRgEkSafE+nZL iQcPNNsrhjv7TqBQ== Date: Wed, 31 Jul 2024 16:02:22 +0200 (CEST) From: Richard Biener To: gcc-patches@gcc.gnu.org cc: ak@linux.intel.com, Jan Hubicka Subject: [PATCH] middle-end/114563 - improve release_pages MIME-Version: 1.0 X-Spamd-Result: default: False [1.49 / 50.00]; MISSING_MID(2.50)[]; NEURAL_HAM_LONG(-0.91)[-0.907]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; MISSING_XM_UA(0.00)[]; RCVD_COUNT_ZERO(0.00)[0]; FROM_HAS_DN(0.00)[]; DKIM_SIGNED(0.00)[suse.de:s=susede2_rsa,suse.de:s=susede2_ed25519]; FROM_EQ_ENVFROM(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FUZZY_BLOCKED(0.00)[rspamd.com]; RCPT_COUNT_THREE(0.00)[3] X-Spam-Score: 1.49 X-Spam-Status: No, score=-10.5 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, MISSING_MID, SPF_HELO_NONE, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~incoming=patchwork.ozlabs.org@gcc.gnu.org Message-Id: <20240731140250.78E233858288@sourceware.org> The following improves release_pages when using the madvise path to sort the freelist to get more page entries contiguous and possibly release them. This populates the unused prev pointer so the reclaim can then easily unlink from the freelist without re-ordering it. The paths not having madvise do not keep the memory allocated, so I left them untouched. Re-bootstrap and regtest running on x86_64-unknown-linux-gnu. I've CCed people messing with release_pages; This doesn't really address PR114563 but I thought I post this patch anyway - the actual issue we run into for the PR is the linear search of G.free_pages when that list becomes large but a requested allocation cannot be served from it. PR middle-end/114563 * ggc-page.cc (page_sort): New qsort comparator. (release_pages): Sort the free_pages list entries after their memory block virtual address to improve contiguous memory chunk release. --- gcc/ggc-page.cc | 68 ++++++++++++++++++++++++++++++++++--------------- 1 file changed, 48 insertions(+), 20 deletions(-) diff --git a/gcc/ggc-page.cc b/gcc/ggc-page.cc index 4245f843a29..c9d8a8cd8e9 100644 --- a/gcc/ggc-page.cc +++ b/gcc/ggc-page.cc @@ -1010,6 +1010,19 @@ free_page (page_entry *entry) G.free_pages = entry; } +/* Comparison function to sort page_entry after virtual address. */ + +static int +page_sort (const void *pa_, const void *pb_) +{ + const page_entry *pa = *(const page_entry * const *)pa_; + const page_entry *pb = *(const page_entry * const *)pb_; + if ((uintptr_t)pa->page < (uintptr_t)pb->page) + return -1; + else + return 1; +} + /* Release the free page cache to the system. */ static void @@ -1022,7 +1035,7 @@ release_pages (void) char *start; size_t len; size_t mapped_len; - page_entry *next, *prev, *newprev; + page_entry *prev; size_t free_unit = (GGC_QUIRE_SIZE/2) * G.pagesize; /* First free larger continuous areas to the OS. @@ -1031,41 +1044,56 @@ release_pages (void) This does not always work because the free_pages list is only approximately sorted. */ - p = G.free_pages; + auto_vec pages; prev = NULL; + p = G.free_pages; while (p) { + p->prev = prev; + pages.safe_push (p); + prev = p; + p = p->next; + } + pages.qsort (page_sort); + + for (unsigned i = 0; i < pages.length ();) + { + p = pages[i]; start = p->page; - start_p = p; + unsigned start_i = i; len = 0; mapped_len = 0; - newprev = prev; - while (p && p->page == start + len) + while (i < pages.length () && pages[i]->page == start + len) { + p = pages[i]; len += p->bytes; if (!p->discarded) - mapped_len += p->bytes; - newprev = p; - p = p->next; + mapped_len += p->bytes; + ++i; } if (len >= free_unit) { - while (start_p != p) - { - next = start_p->next; - free (start_p); - start_p = next; - } + for (unsigned j = start_i; j != i; ++j) + { + p = pages[j]; + if (!p->prev) + { + G.free_pages = p->next; + if (p->next) + p->next->prev = NULL; + } + else + { + p->prev->next = p->next; + if (p->next) + p->next->prev = p->prev; + } + free (pages[j]); + } munmap (start, len); - if (prev) - prev->next = p; - else - G.free_pages = p; G.bytes_mapped -= mapped_len; n1 += len; - continue; } - prev = newprev; } /* Now give back the fragmented pages to the OS, but keep the address