Making it easy to use -flto for bootstrap

I've been playing with -flto bootstraps.

The first problem I ran into was that LTO sections may differ depending
on the presence of debug stmts, so compare-debug needed some tweaking.

Furthermore, simply adding -flto to BOOT_CFLAGS didn't quite work,
because Ada doesn't seem to be ready for LTO.  I got a crash during
linking and didn't pursue it any further.  Adding -fno-lto to
BOOT_ADAFLAGS was the work-around I ended up with.

To make my partly-automated testing easier, I added bootstrap-lto as a
BUILD_CONFIG option.

I didn't have much luck with -flto in target libraries (bad relocs
creating shared libs on x86_64-linux-gnu IIRC), so I left it out.  Was
-flto supposed to work for shared libs?

I ran into -fcompare-debug failures with -flto on x86_64-linux-gnu, and
also with bootstrap-O3 on ia64-lixnu-gnu, but I haven't looked into them
yet.

This is the patch I'm going to install if there aren't objections within
24 hours or so.

> Furthermore, simply adding -flto to BOOT_CFLAGS didn't quite work,
> because Ada doesn't seem to be ready for LTO.

No, it's supposed to work on mainline, there are LTO tests in gnat.dg.

On Thu, Jun 17, 2010 at 9:32 PM, Eric Botcazou <ebotcazou@adacore.com> wrote:
>> Furthermore, simply adding -flto to BOOT_CFLAGS didn't quite work,
>> because Ada doesn't seem to be ready for LTO.
>
> No, it's supposed to work on mainline, there are LTO tests in gnat.dg.

LTO bootstrap indeed fails with Ada (first you run into ICEs due
to -g, then if building without that you get incompatible types of
global syms inside the RTS for some Ada to C bindings if I
understand correctly).

To reproduce:

../configure --enable-stage1-languages=c,lto --enable-gold
--with-plugin-ld=/usr/bin/gold
make BOOT_CFLAGS="-O2 -g -fuse-linker-plugin -fwhopr -B
/path-to-build/prev-lto-plugin/.libs"

same with -flto, for convenience parallelize link with -fwhopr=N.  Add
STAGE1_CFLAGS="-O2 -g" for your convenience.

It would be nice to have small reproducers for the -g ICEs.

Richard.

> --
> Eric Botcazou
>

On Thu, Jun 17, 2010 at 9:18 PM, Alexandre Oliva <aoliva@redhat.com> wrote:
> I've been playing with -flto bootstraps.
>
> The first problem I ran into was that LTO sections may differ depending
> on the presence of debug stmts, so compare-debug needed some tweaking.
>
> Furthermore, simply adding -flto to BOOT_CFLAGS didn't quite work,
> because Ada doesn't seem to be ready for LTO.  I got a crash during
> linking and didn't pursue it any further.  Adding -fno-lto to
> BOOT_ADAFLAGS was the work-around I ended up with.

The build-config should use WHOPR, not LTO.  So -fwhopr=N
and -fno-whopr (but where and how to get N from?).

I thought about removing the current --enable-itermodule and
add --with-whopr=N or so.  Can the build-config get that N somehow?

Note that LTO bootstrap needs gold to do anything reasonable.
Note that BOOT_CFLAGs can also use -fwhole-program (if gold
and the linker-plugin is used).  Note that plugins defeat that ...

> To make my partly-automated testing easier, I added bootstrap-lto as a
> BUILD_CONFIG option.
>
> I didn't have much luck with -flto in target libraries (bad relocs
> creating shared libs on x86_64-linux-gnu IIRC), so I left it out.  Was
> -flto supposed to work for shared libs?

Yes.  But I never tried.  Note that it will be more useful to have
LTO sections in static libraries (in fact I think GNU ld will remove
all LTO sections from shared libs at link time).

> I ran into -fcompare-debug failures with -flto on x86_64-linux-gnu, and
> also with bootstrap-O3 on ia64-lixnu-gnu, but I haven't looked into them
> yet.

Hm, that works for me.

Richard.

> This is the patch I'm going to install if there aren't objections within
> 24 hours or so.
>
>
>
>
> --
> Alexandre Oliva, freedom fighter    http://FSFLA.org/~lxoliva/
> You must be the change you wish to see in the world. -- Gandhi
> Be Free! -- http://FSFLA.org/   FSF Latin America board member
> Free Software Evangelist      Red Hat Brazil Compiler Engineer
>
>

On Jun 18, 2010, Richard Guenther <richard.guenther@gmail.com> wrote:

> The build-config should use WHOPR, not LTO.

I'm not sure I fully understand the difference, even after looking at
the docs.  Clearly using -flto had some effect, and we have tests for
both -flto and -fwhopr, so I went ahead and installed this patch.  I
suppose it would make sense to introduce bootstrap-whopr as well.

> So -fwhopr=N and -fno-whopr (but where and how to get N from?).

Perhaps some --with-whopr configure option could optionally set a
Makefile variable, which a bootstrap-whopr configuration could use.  The
Makefile variable could always be overridden in the command line, of
course.

> On Jun 18, 2010, Richard Guenther <richard.guenther@gmail.com> wrote:
> 
> > The build-config should use WHOPR, not LTO.
> 
> I'm not sure I fully understand the difference, even after looking at
> the docs.  Clearly using -flto had some effect, and we have tests for
> both -flto and -fwhopr, so I went ahead and installed this patch.  I
> suppose it would make sense to introduce bootstrap-whopr as well.

-fwhopr allows parallel build, so it gets you faster to resulting binary and
use less memory. Result should be pretty much same as with -flto (because of
partitioning process the some of late IPA propagation might be lost, but
according to SPEC benchmark there is almost 0 difference).

So I am not sure we need both LTO and WHOPR make, WHOPR should be IMO sufficient.

The difference in between -fwhopr and -flto is indeed bit difficult to parse
for normal user. In future I would like to whopr to be enabled with -flto (or
be -flto subswitch), but I did not had chance to propose change like this.

I would also like to have way to combine profiling and whopr/lto.
What should be interface here?  Make profiledbootstrap-whopr?

Honza
> 
> > So -fwhopr=N and -fno-whopr (but where and how to get N from?).
> 
> Perhaps some --with-whopr configure option could optionally set a
> Makefile variable, which a bootstrap-whopr configuration could use.  The
> Makefile variable could always be overridden in the command line, of
> course.
> 
> -- 
> Alexandre Oliva, freedom fighter    http://FSFLA.org/~lxoliva/
> You must be the change you wish to see in the world. -- Gandhi
> Be Free! -- http://FSFLA.org/   FSF Latin America board member
> Free Software Evangelist      Red Hat Brazil Compiler Engineer

> +# This option enables LTO for stage2 and stage3.  It requires lto to
> +# be enabled for stage1 with --enable-stage1-languages.
> +
> +STAGE2_CFLAGS += -flto
> +STAGE3_CFLAGS += -flto

Also to make this useful (i.e. produce better binary), one needs -fwhole-program. This also need
to build with Gold plugin as used in Richard's example.

Honza
> +
> +# Ada fails to build with LTO, turn it off for now.
> +BOOT_ADAFLAGS += -fno-lto
> Index: gcc/doc/install.texi
> ===================================================================
> --- gcc/doc/install.texi.orig	2010-06-16 16:56:26.000000000 -0300
> +++ gcc/doc/install.texi	2010-06-16 16:58:10.000000000 -0300
> @@ -2150,6 +2150,11 @@ Removes any @option{-O}-started option f
>  @item @samp{bootstrap-O3}
>  Analogous to @code{bootstrap-O1}.
>  
> +@item @samp{bootstrap-lto}
> +Enables Link-Time Optimization for host tools during bootstrapping.
> +@samp{BUILD_CONFIG=bootstrap-lto} is equivalent to adding
> +@option{-flto} to @samp{BOOT_CFLAGS}.
> +
>  @item @samp{bootstrap-debug}
>  Verifies that the compiler generates the same executable code, whether
>  or not it is asked to emit debug information.  To this end, this

> 
> 
> -- 
> Alexandre Oliva, freedom fighter    http://FSFLA.org/~lxoliva/
> You must be the change you wish to see in the world. -- Gandhi
> Be Free! -- http://FSFLA.org/   FSF Latin America board member
> Free Software Evangelist      Red Hat Brazil Compiler Engineer