[ping,03/19,v2] Auto-generate libc-modules.h

Rebased on top of current master.  No changes to the patch.

Siddhesh

On Thu, Aug 28, 2014 at 11:41:28AM +0530, Siddhesh Poyarekar wrote:
> Here's an updated patch to autogenerate libc-modules.h.  I now include
> libc-symbols.h everywhere like before, so all of the %.v.i files can
> once again use the macros libc-symbols.h defines.  To break the
> circular dependency with shlib-versions, I include libc-modules.h in
> the commandline instead of in libc-symbols.h and I don't include it
> for shlib-versions.v.i and Versions.v.i.

From 1aefedff6666344d5368b82ca0f53f6dec9e7056 Mon Sep 17 00:00:00 2001
From: Siddhesh Poyarekar <siddhesh@redhat.com>
Date: Thu, 21 Aug 2014 11:44:54 +0530
Subject: [PATCH] Auto-generate libc-modules.h

Remove libc-modules.h from the tree and auto-generate it from
soversions.i and a new file build.list.  Macros generated from
soversions.i have lower values starting from 1, while those from
build.list start from a special value MODULE_OTHER_BEGIN, which has a
high enough value (1000).  This allows us to conveniently
differentiate between the versioned libraries and other built modules,
which is needed in errno.h and netdb.h to decide whether to use an
internal symbol or an external one.

Verified that generated code remains unchanged on x86_64.

	* Makeconfig (module-cppflags): Include libc-modules.h for
	everything except shlib-versions.v.i.
	(CPPFLAGS): Use it.
	(before-compile): Add libc-modules.h.
	($(common-objpfx)libc-modules.h,
	$(common-objpfx)libc-modules.stmp): New targets.
	(common-generated): Add libc-modules.h and libc-modules.stmp.
	($(common-objpfx)Versions.v.i): Depend on libc-modules.h.
	* build.list: New file.
	* include/libc-symbols.h: Don't include libc-modules.h.
	* include/libc-modules.h: Remove file.
	* scripts/gen-libc-modules.awk: New script to generate
	libc-modules.h.
	* sysdeps/unix/Makefile ($(common-objpfx)sysd-syscalls):
	Depend on libc-modules.stmp.
---
 Makeconfig                   | 44 +++++++++++++++++++++++++++++++++++++++-----
 build.list                   | 15 +++++++++++++++
 include/libc-modules.h       | 37 -------------------------------------
 include/libc-symbols.h       |  2 --
 scripts/gen-libc-modules.awk | 40 ++++++++++++++++++++++++++++++++++++++++
 sysdeps/unix/Makefile        |  3 ++-
 6 files changed, 96 insertions(+), 45 deletions(-)
 create mode 100644 build.list
 delete mode 100644 include/libc-modules.h
 create mode 100644 scripts/gen-libc-modules.awk

Ping!

On Wed, Sep 17, 2014 at 06:13:29PM +0530, Siddhesh Poyarekar wrote:
> Rebased on top of current master.  No changes to the patch.
> 
> Siddhesh
> 
> On Thu, Aug 28, 2014 at 11:41:28AM +0530, Siddhesh Poyarekar wrote:
> > Here's an updated patch to autogenerate libc-modules.h.  I now include
> > libc-symbols.h everywhere like before, so all of the %.v.i files can
> > once again use the macros libc-symbols.h defines.  To break the
> > circular dependency with shlib-versions, I include libc-modules.h in
> > the commandline instead of in libc-symbols.h and I don't include it
> > for shlib-versions.v.i and Versions.v.i.
> 
> From 1aefedff6666344d5368b82ca0f53f6dec9e7056 Mon Sep 17 00:00:00 2001
> From: Siddhesh Poyarekar <siddhesh@redhat.com>
> Date: Thu, 21 Aug 2014 11:44:54 +0530
> Subject: [PATCH] Auto-generate libc-modules.h
> 
> Remove libc-modules.h from the tree and auto-generate it from
> soversions.i and a new file build.list.  Macros generated from
> soversions.i have lower values starting from 1, while those from
> build.list start from a special value MODULE_OTHER_BEGIN, which has a
> high enough value (1000).  This allows us to conveniently
> differentiate between the versioned libraries and other built modules,
> which is needed in errno.h and netdb.h to decide whether to use an
> internal symbol or an external one.
> 
> Verified that generated code remains unchanged on x86_64.
> 
> 	* Makeconfig (module-cppflags): Include libc-modules.h for
> 	everything except shlib-versions.v.i.
> 	(CPPFLAGS): Use it.
> 	(before-compile): Add libc-modules.h.
> 	($(common-objpfx)libc-modules.h,
> 	$(common-objpfx)libc-modules.stmp): New targets.
> 	(common-generated): Add libc-modules.h and libc-modules.stmp.
> 	($(common-objpfx)Versions.v.i): Depend on libc-modules.h.
> 	* build.list: New file.
> 	* include/libc-symbols.h: Don't include libc-modules.h.
> 	* include/libc-modules.h: Remove file.
> 	* scripts/gen-libc-modules.awk: New script to generate
> 	libc-modules.h.
> 	* sysdeps/unix/Makefile ($(common-objpfx)sysd-syscalls):
> 	Depend on libc-modules.stmp.
> ---
>  Makeconfig                   | 44 +++++++++++++++++++++++++++++++++++++++-----
>  build.list                   | 15 +++++++++++++++
>  include/libc-modules.h       | 37 -------------------------------------
>  include/libc-symbols.h       |  2 --
>  scripts/gen-libc-modules.awk | 40 ++++++++++++++++++++++++++++++++++++++++
>  sysdeps/unix/Makefile        |  3 ++-
>  6 files changed, 96 insertions(+), 45 deletions(-)
>  create mode 100644 build.list
>  delete mode 100644 include/libc-modules.h
>  create mode 100644 scripts/gen-libc-modules.awk
> 
> diff --git a/Makeconfig b/Makeconfig
> index 5c85b81..9ce7287 100644
> --- a/Makeconfig
> +++ b/Makeconfig
> @@ -813,10 +813,22 @@ endif	# $(+cflags) == ""
>  # of many little headers in the include directory.
>  libio-include = -I$(..)libio
>  
> +# We don't include libc-modules.h when these targets are being built.  These
> +# targets don't (and will likely never need to) use the IS_IN facility.  In
> +# fact, shlib-versions should not use it because that will create a circular
> +# dependency as libc-modules.h is generated from shlib-versions.
> +skip-module-cppflags = shlib-versions.v.i Versions.v.i
> +
>  in-module = $(strip $(foreach lib,$(libof-$(basename $(@F))) $(libof-$(<F)) \
>  				  $(libof-$(@F)),-DIN_MODULE=MODULE_$(lib)))
>  
>  module-def = $(if $(in-module),$(in-module),-DIN_MODULE=MODULE_libc)
> +module-cppflags-real = -include $(common-objpfx)libc-modules.h $(module-def)
> +
> +# We don't need libc-modules.h and the IN_MODULE definition for
> +# shlib-version.v.i.
> +module-cppflags = $(if $(filter $(@F),$(skip-module-cppflags)), \
> +		       ,$(module-cppflags-real))
>  
>  # These are the variables that the implicit compilation rules use.
>  # Note that we can't use -std=* in CPPFLAGS, because it overrides
> @@ -824,9 +836,9 @@ module-def = $(if $(in-module),$(in-module),-DIN_MODULE=MODULE_libc)
>  # it causes cpp to stop predefining __ASSEMBLER__.
>  CPPFLAGS = $(config-extra-cppflags) $(CPPUNDEFS) $(CPPFLAGS-config) \
>  	   $($(subdir)-CPPFLAGS) \
> -	   $(+includes) $(defines) \
> +	   $(+includes) $(defines) $(module-cppflags) \
>  	   -include $(..)include/libc-symbols.h $(sysdep-CPPFLAGS) \
> -	   $(CPPFLAGS-$(suffix $@)) $(module-def) \
> +	   $(CPPFLAGS-$(suffix $@)) \
>  	   $(foreach lib,$(libof-$(basename $(@F))) \
>  			 $(libof-$(<F)) $(libof-$(@F)),$(CPPFLAGS-$(lib))) \
>  	   $(CPPFLAGS-$(<F)) $(CPPFLAGS-$(@F)) $(CPPFLAGS-$(basename $(@F)))
> @@ -986,9 +998,12 @@ endif
>  postclean-generated += soversions.mk soversions.i \
>  		       shlib-versions.v shlib-versions.v.i
>  
> -# Generate the header containing the names of all shared libraries.
> +# Generate a header containing the names of all shared libraries and another
> +# one containing macros that comprise valid values for the IN_MODULE and
> +# MODULE_NAME macros..
>  # We use a stamp file to avoid unnecessary recompilations.
> -before-compile += $(common-objpfx)gnu/lib-names.h
> +before-compile += $(common-objpfx)gnu/lib-names.h \
> +		  $(common-objpfx)libc-modules.h
>  ifeq ($(soversions.mk-done),t)
>  $(common-objpfx)gnu/lib-names.h: $(common-objpfx)gnu/lib-names.stmp; @:
>  $(common-objpfx)gnu/lib-names.stmp: $(..)scripts/lib-names.awk \
> @@ -1021,9 +1036,28 @@ $(common-objpfx)gnu/lib-names.stmp: $(..)scripts/lib-names.awk \
>  	} >  ${@:stmp=T}
>  	$(move-if-change) ${@:stmp=T} ${@:stmp=h}
>  	touch $@
> +
> +# Generate a header with macro definitions for use with the IS_IN macro.
> +# These are the possible values for the IN_MODULE macro defined when building
> +# sources, to identify which module the translation unit is going to be built
> +# into.  This needs to be one of the first headers to be generated since
> +# everything uses it.  We work around a one-time circular dependency with
> +# sysd-rules by touching an empty header file since the sysd-rules don't use
> +# the IN_MODULE macros even though it is defined in the compile command.  This
> +# is only necessary the first time, i.e. when there is no
> +# include/libc-modules.h.
> +$(common-objpfx)libc-modules.h: $(common-objpfx)libc-modules.stmp; @:
> +$(common-objpfx)libc-modules.stmp: $(..)scripts/gen-libc-modules.awk \
> +				   $(common-objpfx)soversions.i \
> +				   $(..)build.list
> +	$(AWK) -f $^ > ${@:stmp=T}
> +	$(move-if-change) ${@:stmp=T} ${@:stmp=h}
> +	touch $@
> +
>  endif
>  
> -common-generated += gnu/lib-names.h gnu/lib-names.stmp
> +common-generated += gnu/lib-names.h gnu/lib-names.stmp libc-modules.h \
> +		    libc-modules.stmp
>  
>  # The name under which the run-time dynamic linker is installed.
>  # We are currently going for the convention that `/lib/ld.so.1'
> diff --git a/build.list b/build.list
> new file mode 100644
> index 0000000..eab57c7
> --- /dev/null
> +++ b/build.list
> @@ -0,0 +1,15 @@
> +iconvprogs
> +iconvdata
> +ldconfig
> +lddlibc4
> +libmemusage
> +libSegFault
> +libpcprofile
> +librpcsvc
> +libutil
> +locale_programs
> +memusagestat
> +nonlib
> +nscd
> +extramodules
> +libnldbl
> diff --git a/include/libc-modules.h b/include/libc-modules.h
> deleted file mode 100644
> index d12fb90..0000000
> --- a/include/libc-modules.h
> +++ /dev/null
> @@ -1,37 +0,0 @@
> -#define MODULE_libc		1
> -#define MODULE_libpthread	2
> -#define MODULE_rtld		3
> -#define MODULE_libdl		4
> -#define MODULE_libm		5
> -#define MODULE_iconvprogs	6
> -#define MODULE_iconvdata	7
> -#define MODULE_lddlibc4		8
> -#define MODULE_locale_programs	9
> -#define MODULE_memusagestat	10
> -#define MODULE_libutil		12
> -#define MODULE_libBrokenLocale	13
> -#define MODULE_libmemusage	15
> -#define MODULE_libresolv	16
> -#define MODULE_libnss_db	17
> -#define MODULE_libnss_files	18
> -#define	MODULE_libnss_dns	19
> -#define MODULE_libnss_compat	20
> -#define MODULE_libnss_hesiod	21
> -#define MODULE_libnss_nis	22
> -#define MODULE_libnss_nisplus	23
> -#define MODULE_libanl		24
> -#define MODULE_librt		25
> -#define MODULE_libSegFault	26
> -#define MODULE_libthread_db	27
> -#define MODULE_libcidn		28
> -#define MODULE_libcrypt		29
> -#define MODULE_libnsl		30
> -#define MODULE_libpcprofile	31
> -#define MODULE_librpcsvc	32
> -#define MODULE_nscd		33
> -#define MODULE_ldconfig 	34
> -#define MODULE_libnldbl 	35
> -
> -/* Catch-all for test modules and other binaries.  */
> -#define MODULE_nonlib		98
> -#define MODULE_extramodules	99
> diff --git a/include/libc-symbols.h b/include/libc-symbols.h
> index 131d7eb..9b8208b 100644
> --- a/include/libc-symbols.h
> +++ b/include/libc-symbols.h
> @@ -20,8 +20,6 @@
>  #ifndef _LIBC_SYMBOLS_H
>  #define _LIBC_SYMBOLS_H	1
>  
> -#include "libc-modules.h"
> -
>  #define IS_IN(lib) (IN_MODULE == MODULE_##lib)
>  
>  /* This file's macros are included implicitly in the compilation of every
> diff --git a/scripts/gen-libc-modules.awk b/scripts/gen-libc-modules.awk
> new file mode 100644
> index 0000000..d3382b6
> --- /dev/null
> +++ b/scripts/gen-libc-modules.awk
> @@ -0,0 +1,40 @@
> +# Generate a header file that defines the MODULE_* macros for each library and
> +# module we build in glibc.  The library names are pulled in from soversions.i
> +# and the additional modules are mentioned one-per-line in build.list.
> +BEGIN {
> +  PROCINFO["sorted_in"] = "@val_type_asc"
> +  shlibs = 1
> +  others = 1000
> +  libs["OTHERS_BEGIN"] = others++
> +}
> +
> +# Skip over comments.
> +$1 == "#" {
> +  next
> +}
> +
> +# build.list is simply one module per line.
> +match (FILENAME, ".*build.list") {
> +  libs[$0] = others++
> +}
> +
> +# We have only one special case in soversions.i parsing, which is to replace ld
> +# with rtld since that's what we call it throughout the sources.
> +match (FILENAME, ".*soversions.i") {
> +  name = $2
> +  if (name == "ld")
> +    name = "rtld"
> +
> +  if (!(name in libs)) {
> +    libs[name] = shlibs++
> +  }
> +}
> +
> +# Finally, print out the header file.
> +END {
> +  printf ("/* AUTOGENERATED BY gen-libc-modules.awk, DO NOT EDIT.  */\n\n")
> +  i = 1
> +  for (l in libs) {
> +    printf ("#define MODULE_%s %d\n", l, libs[l])
> +  }
> +}
> diff --git a/sysdeps/unix/Makefile b/sysdeps/unix/Makefile
> index 0e535b6..573e90d 100644
> --- a/sysdeps/unix/Makefile
> +++ b/sysdeps/unix/Makefile
> @@ -79,7 +79,8 @@ compile-syscall = $(COMPILE.S) -o $@ -x assembler-with-cpp - \
>  
>  ifndef avoid-generated
>  $(common-objpfx)sysd-syscalls: $(..)sysdeps/unix/make-syscalls.sh \
> -			       $(wildcard $(+sysdep_dirs:%=%/syscalls.list))
> +			       $(wildcard $(+sysdep_dirs:%=%/syscalls.list)) \
> +			       $(common-objpfx)libc-modules.stmp
>  	for dir in $(+sysdep_dirs); do \
>  	  test -f $$dir/syscalls.list && \
>  	  { sysdirs='$(sysdirs)' \
> -- 
> 1.9.3
>