Message ID | 20091119195216.GC11414@hmsreliant.think-freely.org (mailing list archive) |
---|---|
State | Superseded, archived |
Delegated to: | Paul Mackerras |
Headers | show |
Neil Horman writes: > Before anyone flames me for what a oddball solution this is, let me just > say I'm trying to get the ball rolling here. I think there may be better > solutions that can be impemented in reloc_64.S, but I've yet to make any of the > ones I've tried work successfully. I'm open to suggestions, but this solution > is the only one so far that I've been able to get to work. thanks :) I had thought that the CRCs would be the only things with reloc addends less than 4G, but it turns out that the toolchain folds expressions like __pa(&sym) into just a load from a doubleword (in the TOC) containing the physical address of sym. That needs to be relocated and its reloc addend will be less than 4GB. Hence the crashes early in boot that you were seeing with my proposed patch to reloc_64.S. Given that, your solution seems as reasonable as any. Should this go via the modules maintainer, Rusty Russell, perhaps? In any case, Acked-by: Paul Mackerras <paulus@samba.org> > Adjust crcs in __kcrctab_* sections if relocs are used with CONFIG_RELOCATABLE > > When CONFIG_MODVERSIONS and CONFIG_RELOCATABLE are enabled on powerpc platforms, > kdump has been failing in a rather odd way. specifically modules will not > install. This is because when validating the crcs of symbols that the module > needs, the crc of the module never matches the crc that is stored in the kernel. > > The underlying cause of this is how CONFIG_MODVERSIONS is implemented, and how > CONFIG_RELOCATABLE are implemented. with CONFIG_MODVERSIONS enabled, for every > exported symbol in the kernel we emit 2 symbols, __crc_#sym which is declared > extern and __kcrctab_##sym, which is placed in the __kcrctab section of the > binary. The latter has its value set equal to the address of the former > (recalling it is declared extern). After the object file is built, genksyms is > run on the processed source, and crcs are computed for each exported symbol. > genksyms then emits a linker script which defines each of the needed __crc_##sym > symbols, and sets their addresses euqal to their respective crcs. This script > is then used in a secondary link to the previously build object file, so that > the crcs of the missing symbol can be validated on module insert. > > The problem here is that because __kcrctab_sym is set equal to &__crc_##sym, a > relocation entry is emitted by the compiler for the __kcrctab__##sym. Normally > this is not a problem, since relocation on other arches is done without the aid > of .rel.dyn sections. PPC however uses these relocations when > CONFIG_RELOCATABLE is enabled. nominally, since addressing starts at 0 for ppc, > its irrelevant, but if we start at a non-zero address (like we do when booting > via kexec from reserved crashkernel memory), the ppc boot code iterates over the > relocation entries, and winds up adding that relocation offset to all symbols, > including the symbols that are actually the aforementioned crc values in the > __kcrctab_* sections. This effectively corrupts the crcs and prevents any > module loads from happening during a kdump. > > My solution is to 'undo' these relocations prior to boot up. If > ARCH_USES_RELOC_ENTRIES is defined, we add a symbol at address zero to the > linker script for that arch (I call it reloc_start, so that &reloc_start = 0). > This symbol will then indicate the relocation offset for any given boot. We > also add an initcall to the module code that, during boot, scans the __kcrctab_* > sections and subtracts &reloc_start from every entry in those sections, > restoring the appropriate crc value. > > I've verified that this allows kexec to work properly on ppc64 systems myself. > > Signed-off-by: Neil Horman <nhorman@tuxdriver.com> > > > arch/powerpc/include/asm/local.h | 6 ++++++ > arch/powerpc/kernel/vmlinux.lds.S | 4 ++++ > kernel/module.c | 30 ++++++++++++++++++++++++++++++ > 3 files changed, 40 insertions(+) > > diff --git a/arch/powerpc/include/asm/local.h b/arch/powerpc/include/asm/local.h > index 84b457a..9cc49e5 100644 > --- a/arch/powerpc/include/asm/local.h > +++ b/arch/powerpc/include/asm/local.h > @@ -4,6 +4,12 @@ > #include <linux/percpu.h> > #include <asm/atomic.h> > > +#ifdef CONFIG_MODVERSIONS > +#define ARCH_USES_RELOC_ENTRIES > + > +extern unsigned long reloc_start; > +#endif > + > typedef struct > { > atomic_long_t a; > diff --git a/arch/powerpc/kernel/vmlinux.lds.S b/arch/powerpc/kernel/vmlinux.lds.S > index 27735a7..2b9fb2e 100644 > --- a/arch/powerpc/kernel/vmlinux.lds.S > +++ b/arch/powerpc/kernel/vmlinux.lds.S > @@ -38,6 +38,10 @@ jiffies = jiffies_64 + 4; > #endif > SECTIONS > { > + . = 0; > + reloc_start = .; > + . = 0; > + > . = KERNELBASE; > > /* > diff --git a/kernel/module.c b/kernel/module.c > index 8b7d880..87a4928 100644 > --- a/kernel/module.c > +++ b/kernel/module.c > @@ -181,8 +181,11 @@ extern const struct kernel_symbol __stop___ksymtab_gpl_future[]; > extern const struct kernel_symbol __start___ksymtab_gpl_future[]; > extern const struct kernel_symbol __stop___ksymtab_gpl_future[]; > extern const unsigned long __start___kcrctab[]; > +extern const unsigned long __stop___kcrctab[]; > extern const unsigned long __start___kcrctab_gpl[]; > +extern const unsigned long __stop___kcrctab_gpl[]; > extern const unsigned long __start___kcrctab_gpl_future[]; > +extern const unsigned long __stop___kcrctab_gpl_future[]; > #ifdef CONFIG_UNUSED_SYMBOLS > extern const struct kernel_symbol __start___ksymtab_unused[]; > extern const struct kernel_symbol __stop___ksymtab_unused[]; > @@ -3144,3 +3147,30 @@ int module_get_iter_tracepoints(struct tracepoint_iter *iter) > return found; > } > #endif > + > +#ifdef ARCH_USES_RELOC_ENTRIES > +static __init int adjust_kcrctab(void) > +{ > + int i; > + int count; > + unsigned long *crc ; > + > + count = __stop___kcrctab - __start___kcrctab; > + crc = (unsigned long *)__start___kcrctab; > + for (i = 0; i < count; i++) { > + crc[i] -= (unsigned long)&reloc_start; > + } > + count = __stop___kcrctab_gpl - __start___kcrctab_gpl; > + crc = (unsigned long *)__start___kcrctab_gpl; > + for (i = 0; i < count; i++) { > + crc[i] -= (unsigned long)&reloc_start; > + } > + count = __stop___kcrctab_gpl_future - __start___kcrctab_gpl_future; > + crc = (unsigned long *)__start___kcrctab_gpl_future; > + for (i = 0; i< count; i++) { > + crc[i] -= (unsigned long)&reloc_start; > + } > + return 0; > +} > +early_initcall(adjust_kcrctab); > +#endif
On Thu, Nov 19, 2009 at 02:52:16PM -0500, Neil Horman wrote: > Hey there- > Before anyone flames me for what a oddball solution this is, let me just > say I'm trying to get the ball rolling here. I think there may be better > solutions that can be impemented in reloc_64.S, but I've yet to make any of the > ones I've tried work successfully. I'm open to suggestions, but this solution > is the only one so far that I've been able to get to work. thanks :) > > > Adjust crcs in __kcrctab_* sections if relocs are used with CONFIG_RELOCATABLE > > When CONFIG_MODVERSIONS and CONFIG_RELOCATABLE are enabled on powerpc platforms, > kdump has been failing in a rather odd way. specifically modules will not > install. This is because when validating the crcs of symbols that the module > needs, the crc of the module never matches the crc that is stored in the kernel. > > The underlying cause of this is how CONFIG_MODVERSIONS is implemented, and how > CONFIG_RELOCATABLE are implemented. with CONFIG_MODVERSIONS enabled, for every > exported symbol in the kernel we emit 2 symbols, __crc_#sym which is declared > extern and __kcrctab_##sym, which is placed in the __kcrctab section of the > binary. The latter has its value set equal to the address of the former > (recalling it is declared extern). After the object file is built, genksyms is > run on the processed source, and crcs are computed for each exported symbol. > genksyms then emits a linker script which defines each of the needed __crc_##sym > symbols, and sets their addresses euqal to their respective crcs. This script > is then used in a secondary link to the previously build object file, so that > the crcs of the missing symbol can be validated on module insert. > > The problem here is that because __kcrctab_sym is set equal to &__crc_##sym, a > relocation entry is emitted by the compiler for the __kcrctab__##sym. Normally > this is not a problem, since relocation on other arches is done without the aid > of .rel.dyn sections. PPC however uses these relocations when > CONFIG_RELOCATABLE is enabled. nominally, since addressing starts at 0 for ppc, > its irrelevant, but if we start at a non-zero address (like we do when booting > via kexec from reserved crashkernel memory), the ppc boot code iterates over the > relocation entries, and winds up adding that relocation offset to all symbols, > including the symbols that are actually the aforementioned crc values in the > __kcrctab_* sections. This effectively corrupts the crcs and prevents any > module loads from happening during a kdump. > > My solution is to 'undo' these relocations prior to boot up. If > ARCH_USES_RELOC_ENTRIES is defined, we add a symbol at address zero to the > linker script for that arch (I call it reloc_start, so that &reloc_start = 0). > This symbol will then indicate the relocation offset for any given boot. We > also add an initcall to the module code that, during boot, scans the __kcrctab_* > sections and subtracts &reloc_start from every entry in those sections, > restoring the appropriate crc value. > > I've verified that this allows kexec to work properly on ppc64 systems myself. > > Signed-off-by: Neil Horman <nhorman@tuxdriver.com> > Ping, any thoughts here? As I've been mulling this over, I'm beginning to like this solution better than a completely arch-specific section, as this approach makes common the bits that any arch is going to need if they implement CONFIG_RELOCATABLE with a .rel[a].* section set. The alternative is of course to simply skip the appropriate relocations, but thats something that every arch will need to discover on their own. This makes it a bit more clear whats happening I think. Regards Neil
Neil Horman writes: > Before anyone flames me for what a oddball solution this is, let me just > say I'm trying to get the ball rolling here. I think there may be better > solutions that can be impemented in reloc_64.S, but I've yet to make any of the > ones I've tried work successfully. I'm open to suggestions, but this solution > is the only one so far that I've been able to get to work. thanks :) I had thought that the CRCs would be the only things with reloc addends less than 4G, but it turns out that the toolchain folds expressions like __pa(&sym) into just a load from a doubleword (in the TOC) containing the physical address of sym. That needs to be relocated and its reloc addend will be less than 4GB. Hence the crashes early in boot that you were seeing with my proposed patch to reloc_64.S. Given that, your solution seems as reasonable as any. Should this go via the modules maintainer, Rusty Russell, perhaps? In any case, Acked-by: Paul Mackerras <paulus at samba.org> I'm not 100% sure, it does kind of split the middle in regards to what tree is should come through. Rusy, given the previous entries in this thread, do you have any thoughts on the patch, or which tree it should go through? I kind of think that the ppc tree is just fine here since its currently the only user of this code, but I'll defer to other opinions on the subject. Thanks & Regards Neil
On Thu, Nov 19, 2009 at 02:52:16PM -0500, Neil Horman wrote: > Hey there- > Before anyone flames me for what a oddball solution this is, let me just > say I'm trying to get the ball rolling here. I think there may be better > solutions that can be impemented in reloc_64.S, but I've yet to make any of the > ones I've tried work successfully. I'm open to suggestions, but this solution > is the only one so far that I've been able to get to work. thanks :) > > > Adjust crcs in __kcrctab_* sections if relocs are used with CONFIG_RELOCATABLE > > When CONFIG_MODVERSIONS and CONFIG_RELOCATABLE are enabled on powerpc platforms, > kdump has been failing in a rather odd way. specifically modules will not > install. This is because when validating the crcs of symbols that the module > needs, the crc of the module never matches the crc that is stored in the kernel. > > The underlying cause of this is how CONFIG_MODVERSIONS is implemented, and how > CONFIG_RELOCATABLE are implemented. with CONFIG_MODVERSIONS enabled, for every > exported symbol in the kernel we emit 2 symbols, __crc_#sym which is declared > extern and __kcrctab_##sym, which is placed in the __kcrctab section of the > binary. The latter has its value set equal to the address of the former > (recalling it is declared extern). After the object file is built, genksyms is > run on the processed source, and crcs are computed for each exported symbol. > genksyms then emits a linker script which defines each of the needed __crc_##sym > symbols, and sets their addresses euqal to their respective crcs. This script > is then used in a secondary link to the previously build object file, so that > the crcs of the missing symbol can be validated on module insert. > > The problem here is that because __kcrctab_sym is set equal to &__crc_##sym, a > relocation entry is emitted by the compiler for the __kcrctab__##sym. Normally > this is not a problem, since relocation on other arches is done without the aid > of .rel.dyn sections. PPC however uses these relocations when > CONFIG_RELOCATABLE is enabled. nominally, since addressing starts at 0 for ppc, > its irrelevant, but if we start at a non-zero address (like we do when booting > via kexec from reserved crashkernel memory), the ppc boot code iterates over the > relocation entries, and winds up adding that relocation offset to all symbols, > including the symbols that are actually the aforementioned crc values in the > __kcrctab_* sections. This effectively corrupts the crcs and prevents any > module loads from happening during a kdump. > > My solution is to 'undo' these relocations prior to boot up. If > ARCH_USES_RELOC_ENTRIES is defined, we add a symbol at address zero to the > linker script for that arch (I call it reloc_start, so that &reloc_start = 0). > This symbol will then indicate the relocation offset for any given boot. We > also add an initcall to the module code that, during boot, scans the __kcrctab_* > sections and subtracts &reloc_start from every entry in those sections, > restoring the appropriate crc value. > > I've verified that this allows kexec to work properly on ppc64 systems myself. > > Signed-off-by: Neil Horman <nhorman@tuxdriver.com> > Paul, Ben, given that Rusty hasn't come back with any opinion on this patch, do you feel comfortable merging it via the ppc tree? Currently the earlyinit routine is only compiled in and used for your arch, so I think its fairly benign. Regards Neil
>> Paul, Ben, given that Rusty hasn't come back with any opinion on this patch, >do you >> feel comfortable merging it via the ppc tree? Currently the earlyinit routine >> is only compiled in and used for your arch, so I think its fairly benign. > >Sorry, I *did* track down the archives for linuxppc-dev, then find your post, >then read your patch. But I didn't actually reply. > >Other than minor issues, there's one significant one: you shouldn't be trying >to change rodata. It might work on PPC today, but it's poor form at least. > >How's this? Untested on ppc. > I'll try grab a ppc64 system and test this soon. >Other changes: >1) I also changed reloc_start to an array; this is a good idea for any > linker-defined symbols so the compiler can't make assumptions about size. Ok, I'll certainly trust your linker skill over mine :) >2) local.h? How about module.h? Seems good to me >3) I don't think the extra ". = 0" is necessary. Its not, I was just trying to be clear about where reloc_start was to be placed. >4) ARCH_USES_RELOC_ENTRIES isn't clear enough for me; I prefer > ARCH_RELOCATE_KCRCTAB. Sounds good to me. I'll test this as soon as I'm able Thanks! Neil
>>> Paul, Ben, given that Rusty hasn't come back with any opinion on this patch, >>do you >>> feel comfortable merging it via the ppc tree? Currently the earlyinit >routine >>> is only compiled in and used for your arch, so I think its fairly benign. >> >>Sorry, I *did* track down the archives for linuxppc-dev, then find your post, >>then read your patch. But I didn't actually reply. >> >>Other than minor issues, there's one significant one: you shouldn't be trying >>to change rodata. It might work on PPC today, but it's poor form at least. >> >>How's this? Untested on ppc. >> >I'll try grab a ppc64 system and test this soon. > >>Other changes: >>1) I also changed reloc_start to an array; this is a good idea for any >> linker-defined symbols so the compiler can't make assumptions about size. >Ok, I'll certainly trust your linker skill over mine :) > >>2) local.h? How about module.h? >Seems good to me >>3) I don't think the extra ". = 0" is necessary. >Its not, I was just trying to be clear about where reloc_start was to be placed. > >>4) ARCH_USES_RELOC_ENTRIES isn't clear enough for me; I prefer >> ARCH_RELOCATE_KCRCTAB. >Sounds good to me. > > >I'll test this as soon as I'm able >Thanks! >Neil > Just finished testing your patch Rusty, it all works quite well, Thanks! Will this be going in via your tree, or the ppc tree? Acked-by: Neil Horman <nhorman@tuxdriver.com> Neil
On Fri, 4 Dec 2009 01:34:17 am Neil Horman wrote: > Just finished testing your patch Rusty, it all works quite well, Thanks! > Will this be going in via your tree, or the ppc tree? It's in my tree now. Since I didn't really alter the ppc parts, I kept Paul's ack attached too. Thanks, Rusty.
diff --git a/arch/powerpc/include/asm/local.h b/arch/powerpc/include/asm/local.h index 84b457a..9cc49e5 100644 --- a/arch/powerpc/include/asm/local.h +++ b/arch/powerpc/include/asm/local.h @@ -4,6 +4,12 @@ #include <linux/percpu.h> #include <asm/atomic.h> +#ifdef CONFIG_MODVERSIONS +#define ARCH_USES_RELOC_ENTRIES + +extern unsigned long reloc_start; +#endif + typedef struct { atomic_long_t a; diff --git a/arch/powerpc/kernel/vmlinux.lds.S b/arch/powerpc/kernel/vmlinux.lds.S index 27735a7..2b9fb2e 100644 --- a/arch/powerpc/kernel/vmlinux.lds.S +++ b/arch/powerpc/kernel/vmlinux.lds.S @@ -38,6 +38,10 @@ jiffies = jiffies_64 + 4; #endif SECTIONS { + . = 0; + reloc_start = .; + . = 0; + . = KERNELBASE; /* diff --git a/kernel/module.c b/kernel/module.c index 8b7d880..87a4928 100644 --- a/kernel/module.c +++ b/kernel/module.c @@ -181,8 +181,11 @@ extern const struct kernel_symbol __stop___ksymtab_gpl_future[]; extern const struct kernel_symbol __start___ksymtab_gpl_future[]; extern const struct kernel_symbol __stop___ksymtab_gpl_future[]; extern const unsigned long __start___kcrctab[]; +extern const unsigned long __stop___kcrctab[]; extern const unsigned long __start___kcrctab_gpl[]; +extern const unsigned long __stop___kcrctab_gpl[]; extern const unsigned long __start___kcrctab_gpl_future[]; +extern const unsigned long __stop___kcrctab_gpl_future[]; #ifdef CONFIG_UNUSED_SYMBOLS extern const struct kernel_symbol __start___ksymtab_unused[]; extern const struct kernel_symbol __stop___ksymtab_unused[]; @@ -3144,3 +3147,30 @@ int module_get_iter_tracepoints(struct tracepoint_iter *iter) return found; } #endif + +#ifdef ARCH_USES_RELOC_ENTRIES +static __init int adjust_kcrctab(void) +{ + int i; + int count; + unsigned long *crc ; + + count = __stop___kcrctab - __start___kcrctab; + crc = (unsigned long *)__start___kcrctab; + for (i = 0; i < count; i++) { + crc[i] -= (unsigned long)&reloc_start; + } + count = __stop___kcrctab_gpl - __start___kcrctab_gpl; + crc = (unsigned long *)__start___kcrctab_gpl; + for (i = 0; i < count; i++) { + crc[i] -= (unsigned long)&reloc_start; + } + count = __stop___kcrctab_gpl_future - __start___kcrctab_gpl_future; + crc = (unsigned long *)__start___kcrctab_gpl_future; + for (i = 0; i< count; i++) { + crc[i] -= (unsigned long)&reloc_start; + } + return 0; +} +early_initcall(adjust_kcrctab); +#endif
Hey there- Before anyone flames me for what a oddball solution this is, let me just say I'm trying to get the ball rolling here. I think there may be better solutions that can be impemented in reloc_64.S, but I've yet to make any of the ones I've tried work successfully. I'm open to suggestions, but this solution is the only one so far that I've been able to get to work. thanks :) Adjust crcs in __kcrctab_* sections if relocs are used with CONFIG_RELOCATABLE When CONFIG_MODVERSIONS and CONFIG_RELOCATABLE are enabled on powerpc platforms, kdump has been failing in a rather odd way. specifically modules will not install. This is because when validating the crcs of symbols that the module needs, the crc of the module never matches the crc that is stored in the kernel. The underlying cause of this is how CONFIG_MODVERSIONS is implemented, and how CONFIG_RELOCATABLE are implemented. with CONFIG_MODVERSIONS enabled, for every exported symbol in the kernel we emit 2 symbols, __crc_#sym which is declared extern and __kcrctab_##sym, which is placed in the __kcrctab section of the binary. The latter has its value set equal to the address of the former (recalling it is declared extern). After the object file is built, genksyms is run on the processed source, and crcs are computed for each exported symbol. genksyms then emits a linker script which defines each of the needed __crc_##sym symbols, and sets their addresses euqal to their respective crcs. This script is then used in a secondary link to the previously build object file, so that the crcs of the missing symbol can be validated on module insert. The problem here is that because __kcrctab_sym is set equal to &__crc_##sym, a relocation entry is emitted by the compiler for the __kcrctab__##sym. Normally this is not a problem, since relocation on other arches is done without the aid of .rel.dyn sections. PPC however uses these relocations when CONFIG_RELOCATABLE is enabled. nominally, since addressing starts at 0 for ppc, its irrelevant, but if we start at a non-zero address (like we do when booting via kexec from reserved crashkernel memory), the ppc boot code iterates over the relocation entries, and winds up adding that relocation offset to all symbols, including the symbols that are actually the aforementioned crc values in the __kcrctab_* sections. This effectively corrupts the crcs and prevents any module loads from happening during a kdump. My solution is to 'undo' these relocations prior to boot up. If ARCH_USES_RELOC_ENTRIES is defined, we add a symbol at address zero to the linker script for that arch (I call it reloc_start, so that &reloc_start = 0). This symbol will then indicate the relocation offset for any given boot. We also add an initcall to the module code that, during boot, scans the __kcrctab_* sections and subtracts &reloc_start from every entry in those sections, restoring the appropriate crc value. I've verified that this allows kexec to work properly on ppc64 systems myself. Signed-off-by: Neil Horman <nhorman@tuxdriver.com> arch/powerpc/include/asm/local.h | 6 ++++++ arch/powerpc/kernel/vmlinux.lds.S | 4 ++++ kernel/module.c | 30 ++++++++++++++++++++++++++++++ 3 files changed, 40 insertions(+)