Disable ipa-reference with -fprofile-arcs

Hi,
this patch workarounds problems with ipa-reference and profile feedback.
Profile feedback adds static var access into every function in program.
Especially with LTO this makes ipa-reference tables to explode.  It is the case
for Mozilla build where LTO -fprofile-generate OOms for me after allocating 9GB
of bitmaps.

lto-Profilebootstrapped/regtested x86_64-linux, OK?

Honza

	PR tree-optimization/47233
	* opts.c (common_handle_option): Disable ipa-reference with profile feedback.

On Mon, 10 Jan 2011, Jan Hubicka wrote:

> Hi,
> this patch workarounds problems with ipa-reference and profile feedback.
> Profile feedback adds static var access into every function in program.
> Especially with LTO this makes ipa-reference tables to explode.  It is the case
> for Mozilla build where LTO -fprofile-generate OOms for me after allocating 9GB
> of bitmaps.
> 
> lto-Profilebootstrapped/regtested x86_64-linux, OK?

Can't we just identify those vars and not track them at all?  They
are DECL_ARTIFICIAL after all.  OTOH, why does one or two bits
per function blow up memory usage so much?

Richard.

> Honza
> 
> 	PR tree-optimization/47233
> 	* opts.c (common_handle_option): Disable ipa-reference with profile feedback.
> Index: opts.c
> ===================================================================
> --- opts.c	(revision 168596)
> +++ opts.c	(working copy)
> @@ -1561,6 +1561,11 @@ common_handle_option (struct gcc_options
>  	opts->x_flag_value_profile_transformations = value;
>        if (!opts_set->x_flag_inline_functions)
>  	opts->x_flag_inline_functions = value;
> +      /* FIXME: Instrumentation we insert makes ipa-reference bitmaps
> +	 quadratic.  Disable the pass until better memory representation
> +	 is done.  */
> +      if (!opts_set->x_flag_ipa_reference)
> +        opts->x_flag_ipa_reference = false;
>        break;
>  
>      case OPT_fshow_column:

> On Mon, 10 Jan 2011, Jan Hubicka wrote:
> 
> > Hi,
> > this patch workarounds problems with ipa-reference and profile feedback.
> > Profile feedback adds static var access into every function in program.
> > Especially with LTO this makes ipa-reference tables to explode.  It is the case
> > for Mozilla build where LTO -fprofile-generate OOms for me after allocating 9GB
> > of bitmaps.
> > 
> > lto-Profilebootstrapped/regtested x86_64-linux, OK?
> 
> Can't we just identify those vars and not track them at all?  They

I think we want to track orhter kinds of artifical vars - not sure what kinds
we produce however. We surely can do some name matching or pass down a flag.
Overall however, I think it is a lot easier to just disable the pass - we are
speaking of -fprofile-generate mode only and clearly ipa-reference needs to go
and be replaced by ipa-PTA (or more fine grained analysis) in longer run.

> are DECL_ARTIFICIAL after all.  OTOH, why does one or two bits
> per function blow up memory usage so much?

Because we build mask of vars known to be not set/used by the function.  This
usually contains all the counter arrays except the one used by the function.
So we end up with O(n*m) where n is number of functions and m is number of
arrays.  For Mozilla this is quite high. I think there are constant number of
arrays for every file, but Mozilla has many files.

We also might go for disabling with -fprofile-generate -fprofile-arcs -flto
only: I plan to drive users for not using LTO for training run anyway (and
modify profiledbootstrap accordingly)

Honza
> 
> Richard.
> 
> > Honza
> > 
> > 	PR tree-optimization/47233
> > 	* opts.c (common_handle_option): Disable ipa-reference with profile feedback.
> > Index: opts.c
> > ===================================================================
> > --- opts.c	(revision 168596)
> > +++ opts.c	(working copy)
> > @@ -1561,6 +1561,11 @@ common_handle_option (struct gcc_options
> >  	opts->x_flag_value_profile_transformations = value;
> >        if (!opts_set->x_flag_inline_functions)
> >  	opts->x_flag_inline_functions = value;
> > +      /* FIXME: Instrumentation we insert makes ipa-reference bitmaps
> > +	 quadratic.  Disable the pass until better memory representation
> > +	 is done.  */
> > +      if (!opts_set->x_flag_ipa_reference)
> > +        opts->x_flag_ipa_reference = false;
> >        break;
> >  
> >      case OPT_fshow_column:

On Mon, 10 Jan 2011, Jan Hubicka wrote:

> > On Mon, 10 Jan 2011, Jan Hubicka wrote:
> > 
> > > Hi,
> > > this patch workarounds problems with ipa-reference and profile feedback.
> > > Profile feedback adds static var access into every function in program.
> > > Especially with LTO this makes ipa-reference tables to explode.  It is the case
> > > for Mozilla build where LTO -fprofile-generate OOms for me after allocating 9GB
> > > of bitmaps.
> > > 
> > > lto-Profilebootstrapped/regtested x86_64-linux, OK?
> > 
> > Can't we just identify those vars and not track them at all?  They
> 
> I think we want to track orhter kinds of artifical vars - not sure what kinds
> we produce however. We surely can do some name matching or pass down a flag.
> Overall however, I think it is a lot easier to just disable the pass - we are
> speaking of -fprofile-generate mode only and clearly ipa-reference needs to go
> and be replaced by ipa-PTA (or more fine grained analysis) in longer run.
> 
> > are DECL_ARTIFICIAL after all.  OTOH, why does one or two bits
> > per function blow up memory usage so much?
> 
> Because we build mask of vars known to be not set/used by the function.  This
> usually contains all the counter arrays except the one used by the function.
> So we end up with O(n*m) where n is number of functions and m is number of
> arrays.  For Mozilla this is quite high. I think there are constant number of
> arrays for every file, but Mozilla has many files.

Ah, ok.  But we only have a constant number of global statics per
compilation unit, initialized by gimple_init_edge_profiler (at
least I hope so).  So are you seeing explosion because of a high
number of compilation units (we'd of course see number of CUs times
number of global statics bits in the unused bitmaps), or do we have
some statics that are per function?

> We also might go for disabling with -fprofile-generate -fprofile-arcs -flto
> only: I plan to drive users for not using LTO for training run anyway (and
> modify profiledbootstrap accordingly)

Yeah, that was the long-term plan - but of course that's nothing for 4.7.

I'm trying to understand the problem and if we have no per function
statics then your patch is ok for 4.6.

Thanks,
Richard.

> Honza
> > 
> > Richard.
> > 
> > > Honza
> > > 
> > > 	PR tree-optimization/47233
> > > 	* opts.c (common_handle_option): Disable ipa-reference with profile feedback.
> > > Index: opts.c
> > > ===================================================================
> > > --- opts.c	(revision 168596)
> > > +++ opts.c	(working copy)
> > > @@ -1561,6 +1561,11 @@ common_handle_option (struct gcc_options
> > >  	opts->x_flag_value_profile_transformations = value;
> > >        if (!opts_set->x_flag_inline_functions)
> > >  	opts->x_flag_inline_functions = value;
> > > +      /* FIXME: Instrumentation we insert makes ipa-reference bitmaps
> > > +	 quadratic.  Disable the pass until better memory representation
> > > +	 is done.  */
> > > +      if (!opts_set->x_flag_ipa_reference)
> > > +        opts->x_flag_ipa_reference = false;
> > >        break;
> > >  
> > >      case OPT_fshow_column:
> 
>

> 
> Ah, ok.  But we only have a constant number of global statics per
> compilation unit, initialized by gimple_init_edge_profiler (at
> least I hope so).  So are you seeing explosion because of a high
> number of compilation units (we'd of course see number of CUs times
> number of global statics bits in the unused bitmaps), or do we have
> some statics that are per function?

Yes, it is high number of units in Mozilla and the fact that the vectors tends
to be sparse (since the UIDs of those vars are no nicely grouped).  This is why
I was suggesting in previous mail perhaps disabling only in combination with
LTO.

Mozilla has 17978 units and 200000 functions.

Tere are no per funciton statics...

Honza
> 
> > We also might go for disabling with -fprofile-generate -fprofile-arcs -flto
> > only: I plan to drive users for not using LTO for training run anyway (and
> > modify profiledbootstrap accordingly)
> 
> Yeah, that was the long-term plan - but of course that's nothing for 4.7.
> 
> I'm trying to understand the problem and if we have no per function
> statics then your patch is ok for 4.6.
> 
> Thanks,
> Richard.
> 
> > Honza
> > > 
> > > Richard.
> > > 
> > > > Honza
> > > > 
> > > > 	PR tree-optimization/47233
> > > > 	* opts.c (common_handle_option): Disable ipa-reference with profile feedback.
> > > > Index: opts.c
> > > > ===================================================================
> > > > --- opts.c	(revision 168596)
> > > > +++ opts.c	(working copy)
> > > > @@ -1561,6 +1561,11 @@ common_handle_option (struct gcc_options
> > > >  	opts->x_flag_value_profile_transformations = value;
> > > >        if (!opts_set->x_flag_inline_functions)
> > > >  	opts->x_flag_inline_functions = value;
> > > > +      /* FIXME: Instrumentation we insert makes ipa-reference bitmaps
> > > > +	 quadratic.  Disable the pass until better memory representation
> > > > +	 is done.  */
> > > > +      if (!opts_set->x_flag_ipa_reference)
> > > > +        opts->x_flag_ipa_reference = false;
> > > >        break;
> > > >  
> > > >      case OPT_fshow_column:
> > 
> > 
> 
> -- 
> Richard Guenther <rguenther@suse.de>
> Novell / SUSE Labs
> SUSE LINUX Products GmbH - Nuernberg - AG Nuernberg - HRB 16746 - GF: Markus Rex