| Message ID | 1470868546-8237-2-git-send-email-fhunleth@troodon-software.com |
|---|---|
| State | Rejected |
| Headers | show |
Hello, On Wed, 10 Aug 2016 18:35:46 -0400, Frank Hunleth wrote: > Both libconfuse (via libintl) and libarchive have references to pthreads > but do not specify pthreads in their pkg-config descriptions. This > change adds pthreads to the fwup linker options so that the link step > succeeds. I've finally taken some time to look into this, and I believe your fix is not the right fix, and the bug is in fact in uClibc (and I have already posted a mail on the uClibc mailing list about this). The fact that libintl and libarchive do *not* link with libpthread is intentional and actually a feature. As explained in my mail to the uClibc mailing list [1], the idea is that those libraries don't need threads, they only need to be thread-safe if threads are used by the application using them. So the libc part of the C library contains a stub implementation of those functions, while the libpthread part of the C library contains the real implementation. So, libintl and libarchive are only linked against libc.so. If you make a regular, non-threaded application, linked against libintl and/or libarchive, then those libraries will use the stub version of the pthread functions. This is correct and actually good: your application is not multi-threaded, so why suffer the cost of acquiring/releasing a mutex? If on the other hand, your application is multi-threaded, it will link against libpthread. In this case, the pthread_mutex_*() calls from libarchive or libintl will use the real implementation from libpthread, and those libraries will have a thread-safe behavior. The problem is that this doesn't work with static linking with uClibc. I believe it's a uClibc bug, but I'm not entirely sure either. So, if fwup links fine dynamically (i.e without linking against pthread), there should be no reason to link it with pthread for static linking. [1] http://mailman.uclibc-ng.org/pipermail/devel/2016-August/001132.html Thomas
Hi Thomas, On Thu, Aug 11, 2016 at 6:06 PM, Thomas Petazzoni <thomas.petazzoni@free-electrons.com> wrote: > Hello, > > On Wed, 10 Aug 2016 18:35:46 -0400, Frank Hunleth wrote: >> Both libconfuse (via libintl) and libarchive have references to pthreads >> but do not specify pthreads in their pkg-config descriptions. This >> change adds pthreads to the fwup linker options so that the link step >> succeeds. > > I've finally taken some time to look into this, and I believe your fix > is not the right fix, and the bug is in fact in uClibc (and I have > already posted a mail on the uClibc mailing list about this). > > The fact that libintl and libarchive do *not* link with libpthread is > intentional and actually a feature. As explained in my mail to the > uClibc mailing list [1], the idea is that those libraries don't need > threads, they only need to be thread-safe if threads are used by the > application using them. > > So the libc part of the C library contains a stub implementation of > those functions, while the libpthread part of the C library contains > the real implementation. > > So, libintl and libarchive are only linked against libc.so. If you make > a regular, non-threaded application, linked against libintl and/or > libarchive, then those libraries will use the stub version of the > pthread functions. This is correct and actually good: your application > is not multi-threaded, so why suffer the cost of acquiring/releasing a > mutex? > > If on the other hand, your application is multi-threaded, it will link > against libpthread. In this case, the pthread_mutex_*() calls from > libarchive or libintl will use the real implementation from libpthread, > and those libraries will have a thread-safe behavior. > > The problem is that this doesn't work with static linking with uClibc. > I believe it's a uClibc bug, but I'm not entirely sure either. > > So, if fwup links fine dynamically (i.e without linking against > pthread), there should be no reason to link it with pthread for static > linking. Thank you so much for investigating this further and posting to the uclibc mailing list. I much prefer the behaviour you describe since fwup does not use pthreads. I'll monitor the responses to your post. Thanks! Frank > > [1] http://mailman.uclibc-ng.org/pipermail/devel/2016-August/001132.html > > Thomas > -- > Thomas Petazzoni, CTO, Free Electrons > Embedded Linux and Kernel engineering > http://free-electrons.com
> On Aug 11, 2016, at 3:06 PM, Thomas Petazzoni <thomas.petazzoni@free-electrons.com> wrote: > > Hello, > > On Wed, 10 Aug 2016 18:35:46 -0400, Frank Hunleth wrote: >> Both libconfuse (via libintl) and libarchive have references to pthreads >> but do not specify pthreads in their pkg-config descriptions. This >> change adds pthreads to the fwup linker options so that the link step >> succeeds. > > I've finally taken some time to look into this, and I believe your fix > is not the right fix, and the bug is in fact in uClibc (and I have > already posted a mail on the uClibc mailing list about this). > > The fact that libintl and libarchive do *not* link with libpthread is > intentional and actually a feature. As explained in my mail to the > uClibc mailing list [1], the idea is that those libraries don't need > threads, they only need to be thread-safe if threads are used by the > application using them. > > So the libc part of the C library contains a stub implementation of > those functions, while the libpthread part of the C library contains > the real implementation. > > So, libintl and libarchive are only linked against libc.so. If you make > a regular, non-threaded application, linked against libintl and/or > libarchive, then those libraries will use the stub version of the > pthread functions. This is correct and actually good: your application > is not multi-threaded, so why suffer the cost of acquiring/releasing a > mutex? > > If on the other hand, your application is multi-threaded, it will link > against libpthread. In this case, the pthread_mutex_*() calls from > libarchive or libintl will use the real implementation from libpthread, > and those libraries will have a thread-safe behavior. > > The problem is that this doesn't work with static linking with uClibc. > I believe it's a uClibc bug, but I'm not entirely sure either. As per elf specs. "A weak definition does not change the rules by which object files are selected from libraries. However, if a link set contains both a weak definition and a non-weak definition, the non-weak definition will always be used.” The "link set" is the set of objects that have been loaded by the linker. It does not include objects from libraries that are not required. So if libc.a appears first and provides the symbol it will be used even if its weak. If you exchange the order where libpthread appears first it will use the libpthread provided symbols. another option could be to use -Wl,--whole-archive -lpthread -Wl,--no-whole-archive This would link the right strong symbols into binary irrespective of link order but this will also include unused symbols into binary, probably not optimal, and may using -Wl,--gc-sections could help in letting linker remove the unused functions, maybe worth trying. > > So, if fwup links fine dynamically (i.e without linking against > pthread), there should be no reason to link it with pthread for static > linking. > > [1] http://mailman.uclibc-ng.org/pipermail/devel/2016-August/001132.html > > Thomas > -- > Thomas Petazzoni, CTO, Free Electrons > Embedded Linux and Kernel engineering > http://free-electrons.com > _______________________________________________ > buildroot mailing list > buildroot@busybox.net > http://lists.busybox.net/mailman/listinfo/buildroot
Hello, On Thu, 11 Aug 2016 21:54:24 -0700, Khem Raj wrote: > > The problem is that this doesn't work with static linking with uClibc. > > I believe it's a uClibc bug, but I'm not entirely sure either. > > As per elf specs. > > "A weak definition does not change the rules by which object files are > selected from libraries. However, if a link set contains both a weak > definition and a non-weak definition, the non-weak definition will always > be used.” > > The "link set" is the set of objects that have been loaded by the linker. > It does not include objects from libraries that are not required. > > So if libc.a appears first and provides the symbol it will be used > even if its weak. If you exchange the order where libpthread appears > first it will use the libpthread provided symbols. another option > could be to use > > -Wl,--whole-archive -lpthread -Wl,--no-whole-archive > > This would link the right strong symbols into binary irrespective of > link order > > but this will also include unused symbols into binary, probably not > optimal, and may using -Wl,--gc-sections could help in letting linker > remove the unused functions, maybe worth trying. Thanks for those details, but I'm not sure to understand what is the conclusion. Do you mean that it's impossible to implement this dual symbol definition mechanism with static libraries? Indeed, while shared libraries have the weak/non-weak mechanism that allows to be sure the pthread implementation of the symbols will be used if available, with static libraries, only the link order will decide if the libpthread of libc implementation is used. So, if this cannot be fixed in uClibc, how should this problem be addressed? Thanks, Thomas
On Fri, Aug 12, 2016 at 6:51 AM, Thomas Petazzoni <thomas.petazzoni@free-electrons.com> wrote: > Hello, > > On Thu, 11 Aug 2016 21:54:24 -0700, Khem Raj wrote: > >> > The problem is that this doesn't work with static linking with uClibc. >> > I believe it's a uClibc bug, but I'm not entirely sure either. >> >> As per elf specs. >> >> "A weak definition does not change the rules by which object files are >> selected from libraries. However, if a link set contains both a weak >> definition and a non-weak definition, the non-weak definition will always >> be used.” >> >> The "link set" is the set of objects that have been loaded by the linker. >> It does not include objects from libraries that are not required. >> >> So if libc.a appears first and provides the symbol it will be used >> even if its weak. If you exchange the order where libpthread appears >> first it will use the libpthread provided symbols. another option >> could be to use >> >> -Wl,--whole-archive -lpthread -Wl,--no-whole-archive >> >> This would link the right strong symbols into binary irrespective of >> link order >> >> but this will also include unused symbols into binary, probably not >> optimal, and may using -Wl,--gc-sections could help in letting linker >> remove the unused functions, maybe worth trying. > > Thanks for those details, but I'm not sure to understand what is the > conclusion. Do you mean that it's impossible to implement this dual > symbol definition mechanism with static libraries? Indeed, while shared > libraries have the weak/non-weak mechanism that allows to be sure the > pthread implementation of the symbols will be used if available, with > static libraries, only the link order will decide if the libpthread of > libc implementation is used. > > So, if this cannot be fixed in uClibc, how should this problem be > addressed? > may be using with above linking options in application > Thanks, > > Thomas > -- > Thomas Petazzoni, CTO, Free Electrons > Embedded Linux and Kernel engineering > http://free-electrons.com
Hello, On Wed, 10 Aug 2016 18:35:46 -0400, Frank Hunleth wrote: > Both libconfuse (via libintl) and libarchive have references to pthreads > but do not specify pthreads in their pkg-config descriptions. This > change adds pthreads to the fwup linker options so that the link step > succeeds. > > Fixes autobuilder failures: > > http://autobuild.buildroot.net/results/ce3793c79d5dc4e296d645c75f22e121f35030d3/ > http://autobuild.buildroot.net/results/0c5ffec5438f766dade951a64ebfa1b734a3c0aa/ > http://autobuild.buildroot.net/results/802/802b6d77e1b77b7c8fcb8f3b394cdabfd406de7a/ > > Signed-off-by: Frank Hunleth <fhunleth@troodon-software.com> Thanks to the recent uClibc-ng bump, which merges libpthread into libc, the problem that this patch was trying to fix no longer exists. I double checked: with an older uClibc-ng toolchain, I could reproduce the static linking issue. With the newer uClibc-ng toolchain, the built went fine. Therefore, I've marked your patch as Rejected in patchwork. Thanks! Thomas
On Sun, Oct 16, 2016 at 10:32 AM, Thomas Petazzoni <thomas.petazzoni@free-electrons.com> wrote: > Hello, > > On Wed, 10 Aug 2016 18:35:46 -0400, Frank Hunleth wrote: >> Both libconfuse (via libintl) and libarchive have references to pthreads >> but do not specify pthreads in their pkg-config descriptions. This >> change adds pthreads to the fwup linker options so that the link step >> succeeds. >> >> Fixes autobuilder failures: >> >> http://autobuild.buildroot.net/results/ce3793c79d5dc4e296d645c75f22e121f35030d3/ >> http://autobuild.buildroot.net/results/0c5ffec5438f766dade951a64ebfa1b734a3c0aa/ >> http://autobuild.buildroot.net/results/802/802b6d77e1b77b7c8fcb8f3b394cdabfd406de7a/ >> >> Signed-off-by: Frank Hunleth <fhunleth@troodon-software.com> > > Thanks to the recent uClibc-ng bump, which merges libpthread into libc, > the problem that this patch was trying to fix no longer exists. > > I double checked: with an older uClibc-ng toolchain, I could reproduce > the static linking issue. With the newer uClibc-ng toolchain, the built > went fine. > > Therefore, I've marked your patch as Rejected in patchwork. Great! This is a much better solution. Thanks, Frank > > Thanks! > > Thomas > -- > Thomas Petazzoni, CTO, Free Electrons > Embedded Linux and Kernel engineering > http://free-electrons.com
diff --git a/package/fwup/fwup.mk b/package/fwup/fwup.mk index 3d3e3be..c5accc6 100644 --- a/package/fwup/fwup.mk +++ b/package/fwup/fwup.mk @@ -13,5 +13,13 @@ HOST_FWUP_DEPENDENCIES = host-libconfuse host-libarchive host-libsodium FWUP_AUTORECONF = YES FWUP_CONF_ENV = ac_cv_path_HELP2MAN="" +# The following is needed to fix link errors in static configurations +# due to libconfuse requiring pthreads via libintl and libarchive +# using pthread_mutex_*. Neither list pthreads in their pkg-config +# description. +ifeq ($(BR2_STATIC_LIBS)$(BR2_TOOLCHAIN_HAS_THREADS),yy) +FWUP_CONF_ENV += LDFLAGS="$(TARGET_LDFLAGS) -pthread" +endif + $(eval $(autotools-package)) $(eval $(host-autotools-package))
Both libconfuse (via libintl) and libarchive have references to pthreads but do not specify pthreads in their pkg-config descriptions. This change adds pthreads to the fwup linker options so that the link step succeeds. Fixes autobuilder failures: http://autobuild.buildroot.net/results/ce3793c79d5dc4e296d645c75f22e121f35030d3/ http://autobuild.buildroot.net/results/0c5ffec5438f766dade951a64ebfa1b734a3c0aa/ http://autobuild.buildroot.net/results/802/802b6d77e1b77b7c8fcb8f3b394cdabfd406de7a/ Signed-off-by: Frank Hunleth <fhunleth@troodon-software.com> --- package/fwup/fwup.mk | 8 ++++++++ 1 file changed, 8 insertions(+)