[PATCHv2] package/glibc: allow runing on kernels older than used for the headers

From: "Yann E. MORIN" <yann.morin@orange.com>

Currently, we configure glibc to not add compatibility support for
kernels older than the one used for the headers. This is on the
expectation that the system will never run on a kernel that is older
than the one used for the headers or, when Buildroot builds the kernel,
on another, older kernel.

However, in some situations, it is possible to build for a generic
system, where the kernel may be a different version. This can be the
case, for example, when Building an image that is to be used in a
container that can run on a range of machines each with different kernel
versions. In such a case, it is interesting to build glibc in a way as
to take better advantage of the newer kernels, and thus using newer
kernel headers, while still allowing running on older kernels, and thus
carrying more compatibility code.

We add an option to glibc to allow the user to enable compatibility
shims. To simplify the case, when that option is enabled, we just let
glibc enable as old compatibility shims as supported by the current
architecture.

The code size increase is very small. For an ARM Cortex-A7, with
gcc-10.3.0, the delta is as follows (other files installed by glibc had
no size delta; sizes in bytes):
    file                  | no compat | compat    | delta
    ----------------------+-----------+-----------+-------
    ld-linux-armhf.so.3   | 200216    | 200284    | +  68
    libc.so.6             | 1814496   | 1823120   | +8624
                                            ------+-------
                                            Total | +8692

No runtime overhead has been measured; the overhead is most probably
in the measurement noise. Indeed, the compatibility shims are very
lightweight. For example, there are 9 arch-generic shims:
    renameat2(), execveat(), mlock2(), statx(), faccessat2(),
    close_range(), time64-related syscall shenanigans, a waitid()
    feature, and a futex operation (LOCK_PI2)
and then each arch may define a few others. i386 has less than 20
(mostly related to socket options, and one for the ordering of the
clone() arguments), while ARM seems to have only two (mlock2() and a
configurable futex feature).

Note: however, as Arnout pointed out, some programs may still actually
fail to run even with such compatibility shim, if they really expect the
shimed syscalls to really exist and have no fallback (and/or no proper
error-handling). Still, in the vast majority of cases, those
compatibility shims are enough to have a system running.

Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Cc: Alexey Brodkin <Alexey.Brodkin@synopsys.com>
Cc: Arnout Vandecappelle <arnout@mind.be>

---
Changes v1 -> v2:
  - don't let user provide a version, enable all compat shims  (Arnout,
    Alexey)
  - add size measurements and explanations about no runtime overhead
    measurements  (Arnout)
---
 package/glibc/Config.in | 12 ++++++++++++
 package/glibc/glibc.mk  |  8 ++++++--
 2 files changed, 18 insertions(+), 2 deletions(-)

Hi Yann,

> We add an option to glibc to allow the user to enable compatibility
> shims. To simplify the case, when that option is enabled, we just let
> glibc enable as old compatibility shims as supported by the current
> architecture.

Looks reasonable to me. I mean to not have all the flexibility proposed
initially, instead adding an option to fall back to something much
older compared to used headers.

If more options are needed later, we'll get to know it at some point ;)

And thanks for doing that, as I meant to do something similar a while ago,
but never posted my local changes.

-Alexey

Alexey, All,

On 2022-02-02 17:00 +0000, Alexey Brodkin spake thusly:
> > We add an option to glibc to allow the user to enable compatibility
> > shims. To simplify the case, when that option is enabled, we just let
> > glibc enable as old compatibility shims as supported by the current
> > architecture.
> Looks reasonable to me. I mean to not have all the flexibility proposed
> initially, instead adding an option to fall back to something much
> older compared to used headers.
> 
> If more options are needed later, we'll get to know it at some point ;)
> 
> And thanks for doing that, as I meant to do something similar a while ago,
> but never posted my local changes.

Will that be followed up by a reviewed-by or maybe a tested-by tag? ;-)

Thanks for the feedback, by the way. 👍

Regards,
Yann E. MORIN.

Hi Yann,

> On 2022-02-02 17:00 +0000, Alexey Brodkin spake thusly:
> > > We add an option to glibc to allow the user to enable compatibility
> > > shims. To simplify the case, when that option is enabled, we just let
> > > glibc enable as old compatibility shims as supported by the current
> > > architecture.
> > Looks reasonable to me. I mean to not have all the flexibility proposed
> > initially, instead adding an option to fall back to something much
> > older compared to used headers.
> >
> > If more options are needed later, we'll get to know it at some point ;)
> >
> > And thanks for doing that, as I meant to do something similar a while ago,
> > but never posted my local changes.
> 
> Will that be followed up by a reviewed-by or maybe a tested-by tag? ;-)

Indeed the following config 
----------------------->8---------------------
BR2_arcle=y
BR2_archs38=y
BR2_TOOLCHAIN_BUILDROOT_GLIBC=y
BR2_KERNEL_HEADERS_5_15=y
BR2_BINUTILS_VERSION_2_37_X=y
BR2_GCC_VERSION_11_X=y
BR2_LINUX_KERNEL=y
BR2_LINUX_KERNEL_CUSTOM_VERSION=y
BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="5.4.176"
BR2_LINUX_KERNEL_DEFCONFIG="haps_hs"
BR2_LINUX_KERNEL_VMLINUX=y
BR2_TARGET_ROOTFS_INITRAMFS=y
----------------------->8---------------------

fails to boot with:
----------------------->8---------------------
Run /init as init process
FATAL: kernel too old
Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00
---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00007f00 ]---
----------------------->8---------------------

But if we add newly introduced "BR2_PACKAGE_GLIBC_KERNEL_COMPAT=y" all boots
well. With that...

Tested-by: Alexey Brodkin <abrodkin@synopsys.com>

> Thanks for the feedback, by the way. 

That's my pleasure!

-Alexey

On 02/02/2022 14:48, yann.morin@orange.com wrote:
> From: "Yann E. MORIN" <yann.morin@orange.com>
> 
> Currently, we configure glibc to not add compatibility support for
> kernels older than the one used for the headers. This is on the
> expectation that the system will never run on a kernel that is older
> than the one used for the headers or, when Buildroot builds the kernel,
> on another, older kernel.
> 
> However, in some situations, it is possible to build for a generic
> system, where the kernel may be a different version. This can be the
> case, for example, when Building an image that is to be used in a
> container that can run on a range of machines each with different kernel
> versions. In such a case, it is interesting to build glibc in a way as
> to take better advantage of the newer kernels, and thus using newer
> kernel headers, while still allowing running on older kernels, and thus
> carrying more compatibility code.
> 
> We add an option to glibc to allow the user to enable compatibility
> shims. To simplify the case, when that option is enabled, we just let
> glibc enable as old compatibility shims as supported by the current
> architecture.
> 
> The code size increase is very small. For an ARM Cortex-A7, with
> gcc-10.3.0, the delta is as follows (other files installed by glibc had
> no size delta; sizes in bytes):
>      file                  | no compat | compat    | delta
>      ----------------------+-----------+-----------+-------
>      ld-linux-armhf.so.3   | 200216    | 200284    | +  68
>      libc.so.6             | 1814496   | 1823120   | +8624
>                                              ------+-------
>                                              Total | +8692
> 
> No runtime overhead has been measured; the overhead is most probably
> in the measurement noise. Indeed, the compatibility shims are very
> lightweight. For example, there are 9 arch-generic shims:
>      renameat2(), execveat(), mlock2(), statx(), faccessat2(),
>      close_range(), time64-related syscall shenanigans, a waitid()
>      feature, and a futex operation (LOCK_PI2)
> and then each arch may define a few others. i386 has less than 20
> (mostly related to socket options, and one for the ordering of the
> clone() arguments), while ARM seems to have only two (mlock2() and a
> configurable futex feature).
> 
> Note: however, as Arnout pointed out, some programs may still actually
> fail to run even with such compatibility shim, if they really expect the
> shimed syscalls to really exist and have no fallback (and/or no proper
> error-handling). Still, in the vast majority of cases, those
> compatibility shims are enough to have a system running.
> 
> Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
> Cc: Alexey Brodkin <Alexey.Brodkin@synopsys.com>
> Cc: Arnout Vandecappelle <arnout@mind.be>


  Applied to master, thanks.

  Regards,
  Arnout

> 
> ---
> Changes v1 -> v2:
>    - don't let user provide a version, enable all compat shims  (Arnout,
>      Alexey)
>    - add size measurements and explanations about no runtime overhead
>      measurements  (Arnout)
> ---
>   package/glibc/Config.in | 12 ++++++++++++
>   package/glibc/glibc.mk  |  8 ++++++--
>   2 files changed, 18 insertions(+), 2 deletions(-)
> 
> diff --git a/package/glibc/Config.in b/package/glibc/Config.in
> index 5ecd058145..8e9ddac7d9 100644
> --- a/package/glibc/Config.in
> +++ b/package/glibc/Config.in
> @@ -10,6 +10,18 @@ config BR2_PACKAGE_GLIBC
>   	help
>   	  https://www.gnu.org/software/libc/
>   
> +config BR2_PACKAGE_GLIBC_KERNEL_COMPAT
> +	bool "Enable compatibiltiy shims to run on older kernels"
> +	help
> +	  Say 'y' here if you plan on running your system on a kernel
> +	  older than the version used for the toolchain headers.
> +
> +	  Enabling those compatibility shims may generate a slightly
> +	  bigger and slightly slower glibc library.
> +
> +	  The oldest supported kernel version depends on the
> +	  architecture.
> +
>   config BR2_PACKAGE_GLIBC_UTILS
>   	bool "Install glibc utilities"
>   	help
> diff --git a/package/glibc/glibc.mk b/package/glibc/glibc.mk
> index 5c26b0e6df..9ea9f27a2f 100644
> --- a/package/glibc/glibc.mk
> +++ b/package/glibc/glibc.mk
> @@ -98,6 +98,10 @@ endif
>   GLIBC_MAKE = $(BR2_MAKE)
>   GLIBC_CONF_ENV += ac_cv_prog_MAKE="$(BR2_MAKE)"
>   
> +ifeq ($(BR2_PACKAGE_GLIBC_KERNEL_COMPAT),)
> +GLIBC_CONF_OPTS += --enable-kernel=$(call qstrip,$(BR2_TOOLCHAIN_HEADERS_AT_LEAST))
> +endif
> +
>   # Even though we use the autotools-package infrastructure, we have to
>   # override the default configure commands for several reasons:
>   #
> @@ -128,8 +132,8 @@ define GLIBC_CONFIGURE_CMDS
>   		--disable-profile \
>   		--disable-werror \
>   		--without-gd \
> -		--enable-kernel=$(call qstrip,$(BR2_TOOLCHAIN_HEADERS_AT_LEAST)) \
> -		--with-headers=$(STAGING_DIR)/usr/include)
> +		--with-headers=$(STAGING_DIR)/usr/include \
> +		$(GLIBC_CONF_OPTS))
>   	$(GLIBC_ADD_MISSING_STUB_H)
>   endef
>