| Message ID | 20230210233658.3763549-1-romain.naour@gmail.com |
|---|---|
| State | Accepted |
| Headers | show |
| Series | package/busybox: disable stack optimization for i386 target | expand |
On Sat, 11 Feb 2023 00:36:58 +0100 Romain Naour <romain.naour@gmail.com> wrote: > The toolchain-builder project reported an issue with Qemu 7.2.0 for > x86-core2--glibc--bleeding-edge toolchain [1]: > > Run /sbin/init as init process > random: fast init done > EXT4-fs (vda): warning: mounting unchecked fs, running e2fsck is recommended > EXT4-fs (vda): re-mounted. Opts: (null). Quota mode: disabled. > Starting syslogd: OK > traps: syslogd[52] general protection fault ip:b7e21465 sp:bfe59e6c error:0 in libc.so.6[b7d9b000+123000] > Starting klogd: OK > traps: klogd[56] general protection fault ip:b7e94465 sp:bf8f069c error:0 in libc.so.6[b7e0e000+123000] > Running sysctl: traps: logger[62] general protection fault ip:b7e48b6c sp:bfd7d194 error:0 in libc.so.6[b7e05000+123000] > Segmentation fault > traps: logger[64] general protection fault ip:b7dd3b6c sp:bf9b8604 error:0 in libc.so.6[b7d90000+123000] > Segmentation fault > > (Followed by a kernel panic.) > > Testing with the pevious Qemu release (7.1.0) allows to boot the system without any problem. > > Building qemu sources between 7.1.0 and 7.2.0 allows to identify the first "bad" commit [2] and > report to the Qemu project [3]. > > Thanks to Qemu maintainers review, several issues was noticed: > > "The default i386 busybox build config does not respect glibc's requirements around stack alignment > (see [4] for previous discussions and a workaround)." > > Disabling CONFIG_STACK_OPTIMIZATION_386 option (as suggested in the Gentoo bug report) fixed the issue! > > This option has been added and enabled by default in buxybox 1_29_0, so it was used since then the for > Buildroot's qemu defconfig. > > Note: The x86-i686--glibc--bleeding-edge (generic x86) doesn't trigger the issue with > CONFIG_STACK_OPTIMIZATION_386 enabled. > > Fixes: > https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/3731683337 > > [1] https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/3731683337 > [2] https://gitlab.com/qemu-project/qemu/-/commit/958e1dd1300f37f18b2161dfb4eb806fc8c19b44 > [3] https://gitlab.com/qemu-project/qemu/-/issues/1478 > [4] https://bugs.gentoo.org/725674 Thanks a lo for the great investigation. Do we understand precisely what is happening? The link at [4] does not really have an explanation, it only has experimental observations that lead to the conclusion that disabling CONFIG_STACK_OPTIMIZATION_386 is a work-around, but it does not really explain what is happening. Thomas
Hello Thomas, Le 11/02/2023 à 10:49, Thomas Petazzoni via buildroot a écrit : > On Sat, 11 Feb 2023 00:36:58 +0100 > Romain Naour <romain.naour@gmail.com> wrote: > >> The toolchain-builder project reported an issue with Qemu 7.2.0 for >> x86-core2--glibc--bleeding-edge toolchain [1]: >> >> Run /sbin/init as init process >> random: fast init done >> EXT4-fs (vda): warning: mounting unchecked fs, running e2fsck is recommended >> EXT4-fs (vda): re-mounted. Opts: (null). Quota mode: disabled. >> Starting syslogd: OK >> traps: syslogd[52] general protection fault ip:b7e21465 sp:bfe59e6c error:0 in libc.so.6[b7d9b000+123000] >> Starting klogd: OK >> traps: klogd[56] general protection fault ip:b7e94465 sp:bf8f069c error:0 in libc.so.6[b7e0e000+123000] >> Running sysctl: traps: logger[62] general protection fault ip:b7e48b6c sp:bfd7d194 error:0 in libc.so.6[b7e05000+123000] >> Segmentation fault >> traps: logger[64] general protection fault ip:b7dd3b6c sp:bf9b8604 error:0 in libc.so.6[b7d90000+123000] >> Segmentation fault >> >> (Followed by a kernel panic.) >> >> Testing with the pevious Qemu release (7.1.0) allows to boot the system without any problem. >> >> Building qemu sources between 7.1.0 and 7.2.0 allows to identify the first "bad" commit [2] and >> report to the Qemu project [3]. >> >> Thanks to Qemu maintainers review, several issues was noticed: >> >> "The default i386 busybox build config does not respect glibc's requirements around stack alignment >> (see [4] for previous discussions and a workaround)." >> >> Disabling CONFIG_STACK_OPTIMIZATION_386 option (as suggested in the Gentoo bug report) fixed the issue! >> >> This option has been added and enabled by default in buxybox 1_29_0, so it was used since then the for >> Buildroot's qemu defconfig. >> >> Note: The x86-i686--glibc--bleeding-edge (generic x86) doesn't trigger the issue with >> CONFIG_STACK_OPTIMIZATION_386 enabled. >> >> Fixes: >> https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/3731683337 >> >> [1] https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/3731683337 >> [2] https://gitlab.com/qemu-project/qemu/-/commit/958e1dd1300f37f18b2161dfb4eb806fc8c19b44 >> [3] https://gitlab.com/qemu-project/qemu/-/issues/1478 >> [4] https://bugs.gentoo.org/725674 > > Thanks a lo for the great investigation. Do we understand precisely > what is happening? The link at [4] does not really have an explanation, > it only has experimental observations that lead to the conclusion that > disabling CONFIG_STACK_OPTIMIZATION_386 is a work-around, but it does > not really explain what is happening. Actually there are two different issues that contribute to this issue: 1) An existing latent Busybox bug on i386 (busybox compiled with -mpreferred-stack-boundary=2) https://lists.debian.org/debian-boot/2018/01/msg00352.html 2) A Qemu improvement that trigger an exception on unaligned memory accesses that require 16-byte alignment. https://gitlab.com/qemu-project/qemu/-/commit/958e1dd1300f37f18b2161dfb4eb806fc8c19b44 I didn't digging further the root cause of the issue. Best regards, Romain > > Thomas
Hello Thomas, All, Le 11/02/2023 à 11:10, Romain Naour a écrit : > Hello Thomas, > > Le 11/02/2023 à 10:49, Thomas Petazzoni via buildroot a écrit : >> On Sat, 11 Feb 2023 00:36:58 +0100 >> Romain Naour <romain.naour@gmail.com> wrote: >> >>> The toolchain-builder project reported an issue with Qemu 7.2.0 for >>> x86-core2--glibc--bleeding-edge toolchain [1]: >>> >>> Run /sbin/init as init process >>> random: fast init done >>> EXT4-fs (vda): warning: mounting unchecked fs, running e2fsck is recommended >>> EXT4-fs (vda): re-mounted. Opts: (null). Quota mode: disabled. >>> Starting syslogd: OK >>> traps: syslogd[52] general protection fault ip:b7e21465 sp:bfe59e6c error:0 in libc.so.6[b7d9b000+123000] >>> Starting klogd: OK >>> traps: klogd[56] general protection fault ip:b7e94465 sp:bf8f069c error:0 in libc.so.6[b7e0e000+123000] >>> Running sysctl: traps: logger[62] general protection fault ip:b7e48b6c sp:bfd7d194 error:0 in libc.so.6[b7e05000+123000] >>> Segmentation fault >>> traps: logger[64] general protection fault ip:b7dd3b6c sp:bf9b8604 error:0 in libc.so.6[b7d90000+123000] >>> Segmentation fault >>> >>> (Followed by a kernel panic.) >>> >>> Testing with the pevious Qemu release (7.1.0) allows to boot the system without any problem. >>> >>> Building qemu sources between 7.1.0 and 7.2.0 allows to identify the first "bad" commit [2] and >>> report to the Qemu project [3]. >>> >>> Thanks to Qemu maintainers review, several issues was noticed: >>> >>> "The default i386 busybox build config does not respect glibc's requirements around stack alignment >>> (see [4] for previous discussions and a workaround)." >>> >>> Disabling CONFIG_STACK_OPTIMIZATION_386 option (as suggested in the Gentoo bug report) fixed the issue! >>> >>> This option has been added and enabled by default in buxybox 1_29_0, so it was used since then the for >>> Buildroot's qemu defconfig. >>> >>> Note: The x86-i686--glibc--bleeding-edge (generic x86) doesn't trigger the issue with >>> CONFIG_STACK_OPTIMIZATION_386 enabled. >>> >>> Fixes: >>> https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/3731683337 >>> >>> [1] https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/3731683337 >>> [2] https://gitlab.com/qemu-project/qemu/-/commit/958e1dd1300f37f18b2161dfb4eb806fc8c19b44 >>> [3] https://gitlab.com/qemu-project/qemu/-/issues/1478 >>> [4] https://bugs.gentoo.org/725674 >> >> Thanks a lo for the great investigation. Do we understand precisely >> what is happening? The link at [4] does not really have an explanation, >> it only has experimental observations that lead to the conclusion that >> disabling CONFIG_STACK_OPTIMIZATION_386 is a work-around, but it does >> not really explain what is happening. > > Actually there are two different issues that contribute to this issue: > > 1) An existing latent Busybox bug on i386 (busybox compiled with > -mpreferred-stack-boundary=2) > > https://lists.debian.org/debian-boot/2018/01/msg00352.html > > 2) A Qemu improvement that trigger an exception on unaligned memory accesses > that require 16-byte alignment. > > https://gitlab.com/qemu-project/qemu/-/commit/958e1dd1300f37f18b2161dfb4eb806fc8c19b44 > > I didn't digging further the root cause of the issue. Several bug report are confirming a stack problem due to the i386 GCC ABI which assumes the stack is 16-byte aligned [1] [2]. The gcc's default ABI for i386-linux-gnu was quietly changed [3] (maybe between gcc 9 and gcc 10). Note: When the option was added to Busybox and enabled by default, the help text explains that this option may not work with some libc versions: "This option makes for smaller code, but some libc versions do not work with it (they use SSE instructions without ensuring stack alignment)." This problem break the test of the x86 core2 toolchain on toolchain-builder. [1] https://bugs.gentoo.org/725674#c30 [2] https://lists.debian.org/debian-boot/2018/01/msg00352.html [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=886506;msg=97 [4] https://git.busybox.net/busybox/commit/?id=2c9970281083a99acfa3aec8c6d41db955cb583d Best regards, Romain > > Best regards, > Romain > > >> >> Thomas >
Romain, All, On 2023-02-11 00:36 +0100, Romain Naour spake thusly: > The toolchain-builder project reported an issue with Qemu 7.2.0 for > x86-core2--glibc--bleeding-edge toolchain [1]: > > Run /sbin/init as init process > random: fast init done > EXT4-fs (vda): warning: mounting unchecked fs, running e2fsck is recommended > EXT4-fs (vda): re-mounted. Opts: (null). Quota mode: disabled. > Starting syslogd: OK > traps: syslogd[52] general protection fault ip:b7e21465 sp:bfe59e6c error:0 in libc.so.6[b7d9b000+123000] > Starting klogd: OK > traps: klogd[56] general protection fault ip:b7e94465 sp:bf8f069c error:0 in libc.so.6[b7e0e000+123000] > Running sysctl: traps: logger[62] general protection fault ip:b7e48b6c sp:bfd7d194 error:0 in libc.so.6[b7e05000+123000] > Segmentation fault > traps: logger[64] general protection fault ip:b7dd3b6c sp:bf9b8604 error:0 in libc.so.6[b7d90000+123000] > Segmentation fault > > (Followed by a kernel panic.) > > Testing with the pevious Qemu release (7.1.0) allows to boot the system without any problem. > > Building qemu sources between 7.1.0 and 7.2.0 allows to identify the first "bad" commit [2] and > report to the Qemu project [3]. > > Thanks to Qemu maintainers review, several issues was noticed: > > "The default i386 busybox build config does not respect glibc's requirements around stack alignment > (see [4] for previous discussions and a workaround)." > > Disabling CONFIG_STACK_OPTIMIZATION_386 option (as suggested in the Gentoo bug report) fixed the issue! > > This option has been added and enabled by default in buxybox 1_29_0, so it was used since then the for > Buildroot's qemu defconfig. > > Note: The x86-i686--glibc--bleeding-edge (generic x86) doesn't trigger the issue with > CONFIG_STACK_OPTIMIZATION_386 enabled. > > Fixes: > https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/3731683337 > > [1] https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/3731683337 > [2] https://gitlab.com/qemu-project/qemu/-/commit/958e1dd1300f37f18b2161dfb4eb806fc8c19b44 > [3] https://gitlab.com/qemu-project/qemu/-/issues/1478 > [4] https://bugs.gentoo.org/725674 > > Signed-off-by: Romain Naour <romain.naour@gmail.com> Applied to master, thanks. Regards, Yann E. MORIN. > --- > package/busybox/busybox-minimal.config | 2 +- > package/busybox/busybox.config | 2 +- > 2 files changed, 2 insertions(+), 2 deletions(-) > > diff --git a/package/busybox/busybox-minimal.config b/package/busybox/busybox-minimal.config > index 1fb4e39e21..1e7ad1d357 100644 > --- a/package/busybox/busybox-minimal.config > +++ b/package/busybox/busybox-minimal.config > @@ -53,7 +53,7 @@ CONFIG_EXTRA_CFLAGS="" > CONFIG_EXTRA_LDFLAGS="" > CONFIG_EXTRA_LDLIBS="" > # CONFIG_USE_PORTABLE_CODE is not set > -CONFIG_STACK_OPTIMIZATION_386=y > +# CONFIG_STACK_OPTIMIZATION_386 is not set > CONFIG_STATIC_LIBGCC=y > > # > diff --git a/package/busybox/busybox.config b/package/busybox/busybox.config > index 4826fb8242..6e8bd2799d 100644 > --- a/package/busybox/busybox.config > +++ b/package/busybox/busybox.config > @@ -53,7 +53,7 @@ CONFIG_EXTRA_CFLAGS="" > CONFIG_EXTRA_LDFLAGS="" > CONFIG_EXTRA_LDLIBS="" > # CONFIG_USE_PORTABLE_CODE is not set > -CONFIG_STACK_OPTIMIZATION_386=y > +# CONFIG_STACK_OPTIMIZATION_386 is not set > CONFIG_STATIC_LIBGCC=y > > # > -- > 2.34.3 > > _______________________________________________ > buildroot mailing list > buildroot@buildroot.org > https://lists.buildroot.org/mailman/listinfo/buildroot
Romain, All, On 2023-05-11 22:44 +0200, Yann E. MORIN spake thusly: > On 2023-02-11 00:36 +0100, Romain Naour spake thusly: > > Disabling CONFIG_STACK_OPTIMIZATION_386 option (as suggested in the Gentoo bug report) fixed the issue! > Applied to master, thanks. Additionally, I think we should also enforce that with something like: ifeq ($(BR2_i386),y) define BUSYBOX_NO_STACK_OPTIM $(call KCONDFIG_DISABLE_OPT,CONFIG_STACK_OPTIMIZATION_386) endef endif ... so that we are also covered when the user provides their custom config file. Honestly, I think that i386-based systems are rather on the big side of the spectrum, such that the gain from optimising the stack to a non- standard alignment would be hidden in the measurement noise... Regards, Yann E. MORIN.
>>>>> "Romain" == Romain Naour <romain.naour@smile.fr> writes: > Hello Thomas, All, > Le 11/02/2023 à 11:10, Romain Naour a écrit : >> Hello Thomas, >> >> Le 11/02/2023 à 10:49, Thomas Petazzoni via buildroot a écrit : >>> On Sat, 11 Feb 2023 00:36:58 +0100 >>> Romain Naour <romain.naour@gmail.com> wrote: >>> >>>> The toolchain-builder project reported an issue with Qemu 7.2.0 for >>>> x86-core2--glibc--bleeding-edge toolchain [1]: >>>> >>>> Run /sbin/init as init process >>>> random: fast init done >>>> EXT4-fs (vda): warning: mounting unchecked fs, running e2fsck is recommended >>>> EXT4-fs (vda): re-mounted. Opts: (null). Quota mode: disabled. >>>> Starting syslogd: OK >>>> traps: syslogd[52] general protection fault ip:b7e21465 >>>> sp:bfe59e6c error:0 in libc.so.6[b7d9b000+123000] >>>> Starting klogd: OK >>>> traps: klogd[56] general protection fault ip:b7e94465 >>>> sp:bf8f069c error:0 in libc.so.6[b7e0e000+123000] >>>> Running sysctl: traps: logger[62] general protection fault >>>> ip:b7e48b6c sp:bfd7d194 error:0 in libc.so.6[b7e05000+123000] >>>> Segmentation fault >>>> traps: logger[64] general protection fault ip:b7dd3b6c >>>> sp:bf9b8604 error:0 in libc.so.6[b7d90000+123000] >>>> Segmentation fault >>>> >>>> (Followed by a kernel panic.) >>>> >>>> Testing with the pevious Qemu release (7.1.0) allows to boot the >>>> system without any problem. >>>> >>>> Building qemu sources between 7.1.0 and 7.2.0 allows to identify >>>> the first "bad" commit [2] and >>>> report to the Qemu project [3]. >>>> >>>> Thanks to Qemu maintainers review, several issues was noticed: >>>> >>>> "The default i386 busybox build config does not respect glibc's >>>> requirements around stack alignment >>>> (see [4] for previous discussions and a workaround)." >>>> >>>> Disabling CONFIG_STACK_OPTIMIZATION_386 option (as suggested in >>>> the Gentoo bug report) fixed the issue! >>>> >>>> This option has been added and enabled by default in buxybox >>>> 1_29_0, so it was used since then the for >>>> Buildroot's qemu defconfig. >>>> >>>> Note: The x86-i686--glibc--bleeding-edge (generic x86) doesn't trigger the issue with >>>> CONFIG_STACK_OPTIMIZATION_386 enabled. >>>> >>>> Fixes: >>>> https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/3731683337 >>>> >>>> [1] https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/3731683337 >>>> [2] https://gitlab.com/qemu-project/qemu/-/commit/958e1dd1300f37f18b2161dfb4eb806fc8c19b44 >>>> [3] https://gitlab.com/qemu-project/qemu/-/issues/1478 >>>> [4] https://bugs.gentoo.org/725674 >>> >>> Thanks a lo for the great investigation. Do we understand precisely >>> what is happening? The link at [4] does not really have an explanation, >>> it only has experimental observations that lead to the conclusion that >>> disabling CONFIG_STACK_OPTIMIZATION_386 is a work-around, but it does >>> not really explain what is happening. >> >> Actually there are two different issues that contribute to this issue: >> >> 1) An existing latent Busybox bug on i386 (busybox compiled with >> -mpreferred-stack-boundary=2) >> >> https://lists.debian.org/debian-boot/2018/01/msg00352.html >> >> 2) A Qemu improvement that trigger an exception on unaligned memory accesses >> that require 16-byte alignment. >> >> https://gitlab.com/qemu-project/qemu/-/commit/958e1dd1300f37f18b2161dfb4eb806fc8c19b44 >> >> I didn't digging further the root cause of the issue. > Several bug report are confirming a stack problem due to the i386 GCC ABI which > assumes the stack is 16-byte aligned [1] [2]. The gcc's default ABI for > i386-linux-gnu was quietly changed [3] (maybe between gcc 9 and gcc 10). > Note: When the option was added to Busybox and enabled by default, the help text > explains that this option may not work with some libc versions: > "This option makes for smaller code, but some libc versions > do not work with it (they use SSE instructions without > ensuring stack alignment)." > This problem break the test of the x86 core2 toolchain on toolchain-builder. > [1] https://bugs.gentoo.org/725674#c30 > [2] https://lists.debian.org/debian-boot/2018/01/msg00352.html > [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=886506;msg=97 > [4] > https://git.busybox.net/busybox/commit/?id=2c9970281083a99acfa3aec8c6d41db955cb583d Committed to 2023.02.x, thanks.
diff --git a/package/busybox/busybox-minimal.config b/package/busybox/busybox-minimal.config index 1fb4e39e21..1e7ad1d357 100644 --- a/package/busybox/busybox-minimal.config +++ b/package/busybox/busybox-minimal.config @@ -53,7 +53,7 @@ CONFIG_EXTRA_CFLAGS="" CONFIG_EXTRA_LDFLAGS="" CONFIG_EXTRA_LDLIBS="" # CONFIG_USE_PORTABLE_CODE is not set -CONFIG_STACK_OPTIMIZATION_386=y +# CONFIG_STACK_OPTIMIZATION_386 is not set CONFIG_STATIC_LIBGCC=y # diff --git a/package/busybox/busybox.config b/package/busybox/busybox.config index 4826fb8242..6e8bd2799d 100644 --- a/package/busybox/busybox.config +++ b/package/busybox/busybox.config @@ -53,7 +53,7 @@ CONFIG_EXTRA_CFLAGS="" CONFIG_EXTRA_LDFLAGS="" CONFIG_EXTRA_LDLIBS="" # CONFIG_USE_PORTABLE_CODE is not set -CONFIG_STACK_OPTIMIZATION_386=y +# CONFIG_STACK_OPTIMIZATION_386 is not set CONFIG_STATIC_LIBGCC=y #
The toolchain-builder project reported an issue with Qemu 7.2.0 for x86-core2--glibc--bleeding-edge toolchain [1]: Run /sbin/init as init process random: fast init done EXT4-fs (vda): warning: mounting unchecked fs, running e2fsck is recommended EXT4-fs (vda): re-mounted. Opts: (null). Quota mode: disabled. Starting syslogd: OK traps: syslogd[52] general protection fault ip:b7e21465 sp:bfe59e6c error:0 in libc.so.6[b7d9b000+123000] Starting klogd: OK traps: klogd[56] general protection fault ip:b7e94465 sp:bf8f069c error:0 in libc.so.6[b7e0e000+123000] Running sysctl: traps: logger[62] general protection fault ip:b7e48b6c sp:bfd7d194 error:0 in libc.so.6[b7e05000+123000] Segmentation fault traps: logger[64] general protection fault ip:b7dd3b6c sp:bf9b8604 error:0 in libc.so.6[b7d90000+123000] Segmentation fault (Followed by a kernel panic.) Testing with the pevious Qemu release (7.1.0) allows to boot the system without any problem. Building qemu sources between 7.1.0 and 7.2.0 allows to identify the first "bad" commit [2] and report to the Qemu project [3]. Thanks to Qemu maintainers review, several issues was noticed: "The default i386 busybox build config does not respect glibc's requirements around stack alignment (see [4] for previous discussions and a workaround)." Disabling CONFIG_STACK_OPTIMIZATION_386 option (as suggested in the Gentoo bug report) fixed the issue! This option has been added and enabled by default in buxybox 1_29_0, so it was used since then the for Buildroot's qemu defconfig. Note: The x86-i686--glibc--bleeding-edge (generic x86) doesn't trigger the issue with CONFIG_STACK_OPTIMIZATION_386 enabled. Fixes: https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/3731683337 [1] https://gitlab.com/buildroot.org/toolchains-builder/-/jobs/3731683337 [2] https://gitlab.com/qemu-project/qemu/-/commit/958e1dd1300f37f18b2161dfb4eb806fc8c19b44 [3] https://gitlab.com/qemu-project/qemu/-/issues/1478 [4] https://bugs.gentoo.org/725674 Signed-off-by: Romain Naour <romain.naour@gmail.com> --- package/busybox/busybox-minimal.config | 2 +- package/busybox/busybox.config | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)