From patchwork Tue Mar 15 22:31:44 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Henderson X-Patchwork-Id: 597885 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from sourceware.org (server1.sourceware.org [209.132.180.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3qPq9h6Pb2z9sRB for ; Wed, 16 Mar 2016 09:32:08 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=gcc.gnu.org header.i=@gcc.gnu.org header.b=deM/BexL; dkim-atps=neutral DomainKey-Signature: a=rsa-sha1; c=nofws; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender :subject:to:references:cc:from:message-id:date:mime-version :in-reply-to:content-type:content-transfer-encoding; q=dns; s= default; b=N5GK0dJKkedkh7JVNPzA1Vr3jJbeew/PV4ODJWAzjEZJf3mxauUz4 pCywAYuCDnStowC+WLQ+/UMZmHNo+Tqdpdrfg9BLfqlv5Yyuq7J4+w7hzIRobJVQ iFJ+xTt9SYb/bcZHhyAaL/c+9oNRTGEWeRe2cH7mJTaglMYS3YZeqY= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=gcc.gnu.org; h=list-id :list-unsubscribe:list-archive:list-post:list-help:sender :subject:to:references:cc:from:message-id:date:mime-version :in-reply-to:content-type:content-transfer-encoding; s=default; bh=L30TaT+35oXpkDc2f4yZ5JvTfZY=; b=deM/BexLzR2yg0CGeZhsgAFDfRiv zUNjyySJozAu7q7OfUbEwaXI31jr5t4amftRPh2T2Si1K+8/Y/JNr7h+XOrWiBBY 5MbeJJRRKN5/2kjpcnLz8wryos3DfPQvPpKFuIv4V7wCC2WtZt9cUzWIdkk0R27W 1aY3fligLXC9ahM= Received: (qmail 28871 invoked by alias); 15 Mar 2016 22:31:52 -0000 Mailing-List: contact gcc-patches-help@gcc.gnu.org; run by ezmlm Precedence: bulk List-Id: List-Unsubscribe: List-Archive: List-Post: List-Help: Sender: gcc-patches-owner@gcc.gnu.org Delivered-To: mailing list gcc-patches@gcc.gnu.org Received: (qmail 28820 invoked by uid 89); 15 Mar 2016 22:31:51 -0000 Authentication-Results: sourceware.org; auth=none X-Virus-Found: No X-Spam-SWARE-Status: No, score=-1.9 required=5.0 tests=BAYES_00, RP_MATCHES_RCVD, SPF_HELO_PASS autolearn=ham version=3.3.2 spammy=tops, Delorie, delorie, 40GB X-HELO: mx1.redhat.com Received: from mx1.redhat.com (HELO mx1.redhat.com) (209.132.183.28) by sourceware.org (qpsmtpd/0.93/v0.84-503-g423c35a) with (AES256-GCM-SHA384 encrypted) ESMTPS; Tue, 15 Mar 2016 22:31:47 +0000 Received: from int-mx11.intmail.prod.int.phx2.redhat.com (int-mx11.intmail.prod.int.phx2.redhat.com [10.5.11.24]) by mx1.redhat.com (Postfix) with ESMTPS id 178CC1F57F for ; Tue, 15 Mar 2016 22:31:46 +0000 (UTC) Received: from bigtime.twiddle.net (ovpn-113-67.phx2.redhat.com [10.3.113.67]) by int-mx11.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u2FMVj6o003842; Tue, 15 Mar 2016 18:31:45 -0400 Subject: [PATCH] Fix compiling large files To: DJ Delorie References: <201603110420.u2B4KQ70018134@greed.delorie.com> <56E88179.6010805@redhat.com> Cc: gcc-patches@gcc.gnu.org From: Richard Henderson Message-ID: <56E88D50.3000809@redhat.com> Date: Tue, 15 Mar 2016 15:31:44 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.6.0 MIME-Version: 1.0 In-Reply-To: <56E88179.6010805@redhat.com> X-IsSubscribed: yes On 03/10/2016 08:20 PM, DJ Delorie wrote: > I'm moving on to Plan C but I put a copy of the file on > .../dj/foo.c.gz (195Mb) if anyone wants to find out > why there's a 16Gb limit compiling it... With just the following, we successfully compile your file. It takes about 25 minutes and memory use tops out around 40GB. Which still seems insane for a 1.6GB input file consisting primarily of data for a static array, but that's a different problem. At this point we usually have a PR to go with all stage4 changes. But a meaningful PR is difficult to create, since the attachment would be too large. Perhaps a generator could be created, but since it wouldn't go in the testsuite it seems like a waste of time. Thoughts? r~ * line-map.c (new_linemap): Make alloc_size a size_t. diff --git a/libcpp/line-map.c b/libcpp/line-map.c index 1fb634a..80d4e6b 100644 --- a/libcpp/line-map.c +++ b/libcpp/line-map.c @@ -376,7 +376,7 @@ new_linemap (struct line_maps *set, if (LINEMAPS_USED (set, macro_map_p) == LINEMAPS_ALLOCATED (set, macro_map_p)) { /* We ran out of allocated line maps. Let's allocate more. */ - unsigned alloc_size; + size_t alloc_size; /* Cast away extern "C" from the type of xrealloc. */ line_map_realloc reallocator = (set->reallocator