From patchwork Mon Jul 8 05:51:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: k4lizen X-Patchwork-Id: 1957802 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=proton.me header.i=@proton.me header.a=rsa-sha256 header.s=4n6jrujs3bc6rhn5pfg2eernge.protonmail header.b=FOVkVc23; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=8.43.85.97; helo=server2.sourceware.org; envelope-from=libc-alpha-bounces~incoming=patchwork.ozlabs.org@sourceware.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WHYDD2mxJz1xr9 for ; Mon, 8 Jul 2024 15:52:12 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 47D23385E82F for ; Mon, 8 Jul 2024 05:52:10 +0000 (GMT) X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from mail-4325.protonmail.ch (mail-4325.protonmail.ch [185.70.43.25]) by sourceware.org (Postfix) with ESMTPS id C83F1385E82F for ; Mon, 8 Jul 2024 05:51:54 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org C83F1385E82F Authentication-Results: sourceware.org; dmarc=pass (p=quarantine dis=none) header.from=proton.me Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=proton.me ARC-Filter: OpenARC Filter v1.0.0 sourceware.org C83F1385E82F Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=185.70.43.25 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1720417916; cv=none; b=Tl10Y7eelMnplT7zehxXDM7aV3xCGebcCnlBnklnjR8or5j0n+jMF+KVCpq82kZb5DBIDhrEqw2EAniD9x6H1igZZ0PeBPmFSI37E5UzY6Lfh+5xwI4bIXmYrdE8hJmUSK8WXyTSVihknxT8CELBXYbPprE2c3v9LsJ6QX8UmSA= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1720417916; c=relaxed/simple; bh=BFVZyvr6oWrAyBUDP013BZh0DSdZwMIYTaYuUCDrWEg=; h=DKIM-Signature:Date:To:From:Subject:Message-ID:MIME-Version; b=HeL6DWHM8bTlA8lkpaCBLcLmkgCD1YU7lDGtSBCapGjHVVgKlyU90WLCT9CS6WFrY84nABiMXcv93HsjeqpDkSWQSle/sfSTCb5U2n2bXj/yo1+d9ZCB7Le1rDUNEdAdWh6rv+7Qrih1m5vFsQtn4RGdsKne5JNyWEKckCFiURU= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=proton.me; s=4n6jrujs3bc6rhn5pfg2eernge.protonmail; t=1720417912; x=1720677112; bh=cuuui3yzIr03ZdyAxn02DSvjs7qY36y2BaFR+ugBTk8=; h=Date:To:From:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=FOVkVc23DkxeK0QVv1W4vXF5ISzv4A+nccp+hH5r5E2fub5kZ+vJntSoZmtF/vxgW 0/M2/Zc+KslvGqTWxAO+MgOduiDmwAlKDfwzXT0D0GEdWqfPtJf37HG7ZE6qLZOZQj 0BESn4woII+rNdQmW9tvT5IAjmBDrgLyYS2y9f6GU6fOiAUIQSeu7J8qLW4DqseUDJ 7Y6h9hNV2c5Qt9rvGGhWfs1QhdZbDcGBctisiRpZklAXok42n2ejN/d1nMLmJcupwD QTRpfCHlDiExCZjISdLvB2IcODSDsA06TQc9amb94yQo+XNdsoYyILHD/NTLMKCNff HAgfVBoLqtmIA== Date: Mon, 08 Jul 2024 05:51:50 +0000 To: "libc-alpha@sourceware.org" From: k4lizen Subject: [PATCH v2] malloc: send freed small chunks to smallbin Message-ID: <5a2_xklfnO_qoJgnmLAlYontjxrkxJif4jr1hVxo4MetQOhj-P3KF-1_NbHLMsBGCWL2AMGcZaueC_TGIEWZYhsbNlISdWwcPNeuqUv-oCw=@proton.me> In-Reply-To: References: Feedback-ID: 102038742:user:proton X-Pm-Message-ID: b13fec1e22ed52f21fe4ecc5dc8a0f2f20895bc1 MIME-Version: 1.0 X-Spam-Status: No, score=-9.1 required=5.0 tests=BAYES_00, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_INFOUSMEBIZ, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4, RCVD_IN_MSPIKE_WL, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libc-alpha-bounces~incoming=patchwork.ozlabs.org@sourceware.org Large chunks get added to the unsorted bin since sorting them takes time, for small chunks the benefit of adding them to the unsorted bin is non-existant, actually hurting performance. Splitting and malloc_consolidate still add small chunks to unsorted, but we can hint the compiler that that is a relatively rare occurance. Benchmarking shows this to be consistently good. --- malloc/malloc.c | 59 +++++++++++++++++++++++++++++++++---------------- 1 file changed, 40 insertions(+), 19 deletions(-) diff --git a/malloc/malloc.c b/malloc/malloc.c index bcb6e5b83c..ad77cd083e 100644 --- a/malloc/malloc.c +++ b/malloc/malloc.c @@ -4156,9 +4156,9 @@ _int_malloc (mstate av, size_t bytes) #endif } - /* place chunk in bin */ - - if (in_smallbin_range (size)) + /* Place chunk in bin. Only malloc_consolidate() and splitting can put + small chunks into the unsorted bin. */ + if (__glibc_unlikely (in_smallbin_range (size))) { victim_index = smallbin_index (size); bck = bin_at (av, victim_index); @@ -4723,23 +4723,45 @@ _int_free_create_chunk (mstate av, mchunkptr p, INTERNAL_SIZE_T size, } else clear_inuse_bit_at_offset(nextchunk, 0); - /* - Place the chunk in unsorted chunk list. Chunks are - not placed into regular bins until after they have - been given one chance to be used in malloc. - */ + mchunkptr bck, fwd; - mchunkptr bck = unsorted_chunks (av); - mchunkptr fwd = bck->fd; - if (__glibc_unlikely (fwd->bk != bck)) - malloc_printerr ("free(): corrupted unsorted chunks"); - p->fd = fwd; + if(!in_smallbin_range (size)) + { + /* + Place large chunks in unsorted chunk list. Large chunks are + not placed into regular bins until after they have + been given one chance to be used in malloc. + + This branch is first in the if-statement to help branch + prediction on consecutive adjacent frees. + */ + + bck = unsorted_chunks (av); + fwd = bck->fd; + if (__glibc_unlikely (fwd->bk != bck)) + malloc_printerr ("free(): corrupted unsorted chunks"); + p->fd_nextsize = NULL; + p->bk_nextsize = NULL; + } + else + { + /* + Place small chunks directly in their smallbin, so they + don't pollute the unsorted bin. + */ + + int chunk_index = smallbin_index (size); + bck = bin_at (av, chunk_index); + fwd = bck->fd; + + if (__glibc_unlikely (fwd->bk != bck)) + malloc_printerr ("free(): chunks in smallbin corrupted"); + + mark_bin (av, chunk_index); + } + p->bk = bck; - if (!in_smallbin_range(size)) - { - p->fd_nextsize = NULL; - p->bk_nextsize = NULL; - } + p->fd = fwd; bck->fd = p; fwd->bk = p; @@ -4748,7 +4770,6 @@ _int_free_create_chunk (mstate av, mchunkptr p, INTERNAL_SIZE_T size, check_free_chunk(av, p); } - else { /* If the chunk borders the current high end of memory,