Message ID | a662962e-e650-4d99-bed2-aa45f0b2cf19@app.fastmail.com |
---|---|
State | New |
Headers | show |
Series | [GIT,PULL] asm-generic updates for 6.11 | expand |
The pull request you sent on Mon, 15 Jul 2024 23:14:27 +0200:
> https://git.kernel.org/pub/scm/linux/kernel/git/arnd/asm-generic.git tags/asm-generic-6.11
has been merged into torvalds/linux.git:
https://git.kernel.org/torvalds/c/d80f2996b8502779c39221a9e7c9ea7e361c0ae4
Thank you!
On Mon, 15 Jul 2024 at 14:07, Arnd Bergmann <arnd@arndb.de> wrote: > > Most of this is part of my ongoing work to clean up the system call > tables. In this bit, all of the newer architectures are converted to > use the machine readable syscall.tbl format instead in place of complex > macros in include/uapi/asm-generic/unistd.h. I haven't bisected things, but I think this code is seriously and utterly broken. When I do a make allmodconfig make -j199 > ../makes on my arm64 machine, it keeps rebuilding pretty much the whole thing every time - whether I have made any changes or not. The second time it should be basically a no-op. Sure, a fairly slow no-op, because 'make' will go through the motions and check all the dependencies etc, but it shouldn't *build* anything. But that's not at all what I see. It rebuilds pretty much the whole tree (not quite everything, but at an estimate it rebuilds the majority of files). And the first things I see in the build log is SYSHDR arch/arm64/include/generated/uapi/asm/unistd_64.h SYSTBL arch/arm64/include/generated/asm/syscall_table_32.h SYSTBL arch/arm64/include/generated/asm/syscall_table_64.h SYSHDR arch/arm64/include/generated/asm/unistd_32.h SYSHDR arch/arm64/include/generated/asm/unistd_compat_32.h HDRINST usr/include/asm/unistd_64.h CC arch/arm64/kernel/asm-offsets.s CALL scripts/checksyscalls.sh ... which is why I'm suspecting your changes without having actually bisected the build time regression at all. And yes, I checked - it does update the timestamps of those four generated files: unistd_compat_32.h, unistd_32.h, syscall_table_64.h, and syscall_table_32.h Every time. So I'm pretty sure it's on you, even if I didn't do the bisection. This needs fixing, urgently. Because it turns a "small pull" into always taking 5+ minutes for me. I didn't notice immediately, because many of my core pulls I _expect_ to rebuild everything, but... Btw, I don't see the same effect on x86-64, so this is purely an arm64 issue (well, and presumably any other architecture that uses 'syscall-y'). You might want to do a build like this: make allmodconfig make twice, and then do find . -newer .config -name '*.h' to find things where the build has generated header files with new timestamps despite no changes. There are other bad cases, but the syscall ones seem to be the ones that cause problems. I'm adding Masahiro to the participants, because of some of the other files that *do* show up for me when I do the above thing: On x86: arch/x86/boot/voffset.h arch/x86/boot/zoffset.h security/apparmor/net_names.h On arm64 (ignoring the above and the syscall ones): include/generated/vdso-offsets.h That 'find' also shows that the install headers thing does the same to the ./usr/include/ directories, but the kernel build doesn't care about those. Linus
On Wed, Jul 17, 2024, at 06:02, Linus Torvalds wrote: > On Mon, 15 Jul 2024 at 14:07, Arnd Bergmann <arnd@arndb.de> wrote: > > But that's not at all what I see. It rebuilds pretty much the whole > tree (not quite everything, but at an estimate it rebuilds the > majority of files). > > And the first things I see in the build log is > > SYSHDR arch/arm64/include/generated/uapi/asm/unistd_64.h > SYSTBL arch/arm64/include/generated/asm/syscall_table_32.h > SYSTBL arch/arm64/include/generated/asm/syscall_table_64.h > SYSHDR arch/arm64/include/generated/asm/unistd_32.h > SYSHDR arch/arm64/include/generated/asm/unistd_compat_32.h > HDRINST usr/include/asm/unistd_64.h > CC arch/arm64/kernel/asm-offsets.s > CALL scripts/checksyscalls.sh > ... > > which is why I'm suspecting your changes without having actually > bisected the build time regression at all. Right, I can confirm that this is not supposed to happen, and it didn't do that for me during my testing. It should only recreate asm/unistd_64.h if scripts/syscall.tbl changed, as it does on the architectures that are using the "archheaders:" rule to get into the architecture specific arch/*/*/syscalls/Makefile/. > This needs fixing, urgently. Because it turns a "small pull" into > always taking 5+ minutes for me. I didn't notice immediately, because > many of my core pulls I _expect_ to rebuild everything, but... > > Btw, I don't see the same effect on x86-64, so this is purely an arm64 > issue (well, and presumably any other architecture that uses > 'syscall-y'). > > You might want to do a build like this: > > make allmodconfig > make > > twice, and then do > > find . -newer .config -name '*.h' > > to find things where the build has generated header files with new > timestamps despite no changes. I tried now both with my branch and with your latest commit 51835949dda3 ("Merge tag 'net-next-6.11' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next") but can't reproduce it yet. My first guess was that this might be related to building inside the source tree rather than a separate object tree (which I used for my own testing), but that did not make a difference either. > I'm adding Masahiro to the participants, because of some of the other > files that *do* show up for me when I do the above thing: > > On x86: > arch/x86/boot/voffset.h > arch/x86/boot/zoffset.h > security/apparmor/net_names.h > > On arm64 (ignoring the above and the syscall ones): > include/generated/vdso-offsets.h > > That 'find' also shows that the install headers thing does the same to > the ./usr/include/ directories, but the kernel build doesn't care > about those. I don't see those either (this is an x86 defconfig example): $ find . -newer .config -name '*.h' ./include/generated/autoconf.h So whatever caused the regression is probably a result of my changes, but so far I haven't been able to explain or reproduce it. I'll keep looking. Arnd
On Tue, 16 Jul 2024 at 21:44, Arnd Bergmann <arnd@arndb.de> wrote: > > I don't see those either (this is an x86 defconfig example): Note, it really might be just 'allmodconfig'. We've had things that depend on config entries in the past, eg the whole CONFIG_HEADERS_INSTALL etc could affect things. And yes, I build everything in the source tree, so $(obj) is $(src). I find the whole "separate object tree" model incomprehensible, and think it's a result of people using CVS or something like that where having multiple source trees is hard. Linus
On Tue, 16 Jul 2024 at 21:57, Linus Torvalds <torvalds@linux-foundation.org> wrote: > > Note, it really might be just 'allmodconfig'. We've had things that > depend on config entries in the past, eg the whole > CONFIG_HEADERS_INSTALL etc could affect things. Oh, and my "find" line was bogus, because it turns out ".config" itself is only updated if it changed, which explains my confusion about some of the other header files. So it turns out that to get the real list of rewritten headers, you need to do something like make allmodconfig touch .config make and then to make it faster, you just ^C after five seconds, and do that find . -newer .config -name '*.h' and _now_ it's meaningful, and on arm64 I see ./arch/arm64/include/generated/uapi/asm/unistd_64.h ./arch/arm64/include/generated/asm/syscall_table_32.h ./arch/arm64/include/generated/asm/syscall_table_64.h ./arch/arm64/include/generated/asm/unistd_32.h ./arch/arm64/include/generated/asm/unistd_compat_32.h ./include/generated/autoconf.h ./usr/include/asm/unistd_64.h while on x86, the only header that changes as part of the build is ./include/generated/autoconf.h and all the other files I listed were just because I hadn't noticed that "make allmodconfig" itself avoids the write of ".config". So that "touch .config" is needed. Or just use another file entirely to flag the "this is the state before I actually started the build". Which I probably should have done instead of thinking that "hey, I have this .config file that I can just use as a marker". Linus
On Wed, Jul 17, 2024, at 07:08, Linus Torvalds wrote: > On Tue, 16 Jul 2024 at 21:57, Linus Torvalds > <torvalds@linux-foundation.org> wrote: >> >> Note, it really might be just 'allmodconfig'. We've had things that >> depend on config entries in the past, eg the whole >> CONFIG_HEADERS_INSTALL etc could affect things. I had tried a partial allmodconfig build earlier to save time, did a full build again now, still nothing: arnd@studio:~/arm-soc/bisect$ make allmodconfig -sj20 arnd@studio:~/arm-soc/bisect$ make -sj20 arnd@studio:~/arm-soc/bisect$ touch .config arnd@studio:~/arm-soc/bisect$ make allmodconfig -sj20 arnd@studio:~/arm-soc/bisect$ make -j20 SYNC include/config/auto.conf.cmd CALL scripts/checksyscalls.sh CHK kernel/kheaders_data.tar.xz arnd@studio:~/arm-soc/bisect$ find . -newer .config . ./arch/arm64/kvm ./include/config ./include/config/auto.conf.cmd ./include/config/auto.conf ./include/generated ./include/generated/uapi/linux ./include/generated/autoconf.h ./include/generated/rustc_cfg ./init ./kernel ./lib ./scripts/mod ./.missing-syscalls.d This is in a fresh worktree, doing a native arm64 build in the source directory. In case the problem is related to changes in high-resolution timestamping, I am running a slightly older kernel (still 6.8) and I'm using btrfs with relatime for the build here. I also tried on tmpfs now, same results. ccache is disabled. Arnd
On Wed, Jul 17, 2024 at 08:01:43AM +0200, Arnd Bergmann wrote: > On Wed, Jul 17, 2024, at 07:08, Linus Torvalds wrote: > > On Tue, 16 Jul 2024 at 21:57, Linus Torvalds > > <torvalds@linux-foundation.org> wrote: > >> > >> Note, it really might be just 'allmodconfig'. We've had things that > >> depend on config entries in the past, eg the whole > >> CONFIG_HEADERS_INSTALL etc could affect things. > > I had tried a partial allmodconfig build earlier to save time, > did a full build again now, still nothing: FWIW, I noticed this last Friday as well when I did a few builds of linux-next and every change I made triggered what appeared to be a full rebuild of the tree. This was with a trimmed config [1] and separate build tree (tmpfs). Johan [1] https://github.com/jhovold/linux/commit/c3324059a67c7c63186107771eabc17a66772989
On Wed, Jul 17, 2024, at 07:08, Linus Torvalds wrote: > On Tue, 16 Jul 2024 at 21:57, Linus Torvalds <torvalds@linux-foundation.org> wrote: > > ./arch/arm64/include/generated/uapi/asm/unistd_64.h > ./arch/arm64/include/generated/asm/syscall_table_32.h > ./arch/arm64/include/generated/asm/syscall_table_64.h > ./arch/arm64/include/generated/asm/unistd_32.h > ./arch/arm64/include/generated/asm/unistd_compat_32.h > ./include/generated/autoconf.h > ./usr/include/asm/unistd_64.h > I've tried to come up with a patch that avoids including asm/unistd.h in most files, which would give some relief and hopefully let you get through the merge window in case we can't figure out my Makefile bug quickly. It's only a small drop in the ocean of excessive header inclusions, but it's still a step in the right direction I think. I'll do some more testing on other architectures with this patch so I can send you something that works. You can also just revert the three arm64 commits for now d2a4a07190f4 arm64: rework compat syscall macros e632bca07c8e arm64: generate 64-bit syscall.tbl 7fe33e9f662c arm64: convert unistd_32.h to syscall.tbl format since the patch to remove uapi/asm-generic/unistd.h wasn't part of the 6.11 series yet and nothing else depends on the arm64 conversion. Arnd extern const unsigned long sys_call_table[]; diff --git a/arch/arm64/kernel/sys.c b/arch/arm64/kernel/sys.c index f08408b6e826..274b67f02f3e 100644 --- a/arch/arm64/kernel/sys.c +++ b/arch/arm64/kernel/sys.c @@ -13,7 +13,7 @@ #include <linux/export.h> #include <linux/sched.h> #include <linux/slab.h> -#include <linux/syscalls.h> +#include <linux/syscalls_api.h> #include <asm/cpufeature.h> #include <asm/syscall.h> diff --git a/fs/proc/base.c b/fs/proc/base.c index 72a1acd03675..9a535916dc03 100644 --- a/fs/proc/base.c +++ b/fs/proc/base.c @@ -92,6 +92,7 @@ #include <linux/sched/coredump.h> #include <linux/sched/debug.h> #include <linux/sched/stat.h> +#include <linux/syscalls_api.h> #include <linux/posix-timers.h> #include <linux/time_namespace.h> #include <linux/resctrl.h> diff --git a/include/linux/binfmts.h b/include/linux/binfmts.h index e6c00e860951..5144b80027be 100644 --- a/include/linux/binfmts.h +++ b/include/linux/binfmts.h @@ -3,7 +3,6 @@ #define _LINUX_BINFMTS_H #include <linux/sched.h> -#include <linux/unistd.h> #include <asm/exec.h> #include <uapi/linux/binfmts.h> diff --git a/include/linux/compat.h b/include/linux/compat.h index 56cebaff0c91..89c307da6e5d 100644 --- a/include/linux/compat.h +++ b/include/linux/compat.h @@ -17,7 +17,6 @@ #include <linux/fs.h> #include <linux/aio_abi.h> /* for aio_context_t */ #include <linux/uaccess.h> -#include <linux/unistd.h> #include <asm/compat.h> #include <asm/siginfo.h> diff --git a/include/linux/ptrace.h b/include/linux/ptrace.h index 90507d4afcd6..f1ddc1eb9290 100644 --- a/include/linux/ptrace.h +++ b/include/linux/ptrace.h @@ -9,13 +9,6 @@ #include <linux/bug.h> /* For BUG_ON. */ #include <linux/pid_namespace.h> /* For task_active_pid_ns. */ #include <uapi/linux/ptrace.h> -#include <linux/seccomp.h> - -/* Add sp to seccomp_data, as seccomp is user API, we don't want to modify it */ -struct syscall_info { - __u64 sp; - struct seccomp_data data; -}; extern int ptrace_access_vm(struct task_struct *tsk, unsigned long addr, void *buf, int len, unsigned int gup_flags); @@ -397,8 +390,6 @@ static inline void user_single_step_report(struct pt_regs *regs) #define exception_ip(x) instruction_pointer(x) #endif -extern int task_current_syscall(struct task_struct *target, struct syscall_info *info); - extern void sigaction_compat_abi(struct k_sigaction *act, struct k_sigaction *oact); /* diff --git a/include/linux/syscalls.h b/include/linux/syscalls.h index fff820c3e93e..2613b8f264bb 100644 --- a/include/linux/syscalls.h +++ b/include/linux/syscalls.h @@ -86,7 +86,6 @@ struct mnt_id_req; #include <linux/bug.h> #include <linux/sem.h> #include <asm/siginfo.h> -#include <linux/unistd.h> #include <linux/quota.h> #include <linux/key.h> #include <linux/personality.h> diff --git a/include/linux/syscalls_api.h b/include/linux/syscalls_api.h index 23e012b04db4..bf997576453f 100644 --- a/include/linux/syscalls_api.h +++ b/include/linux/syscalls_api.h @@ -1 +1,17 @@ +// SPDX-License-Identifier: GPL-2.0 +#ifndef _LINUX_SYSCALLS_API_H +#define _LINUX_SYSCALLS_API_H + #include <linux/syscalls.h> +#include <linux/seccomp.h> +#include <asm/syscall.h> + +/* Add sp to seccomp_data, as seccomp is user API, we don't want to modify it */ +struct syscall_info { + __u64 sp; + struct seccomp_data data; +}; + +extern int task_current_syscall(struct task_struct *target, struct syscall_info *info); + +#endif diff --git a/include/trace/syscall.h b/include/trace/syscall.h index 8e193f3a33b3..0dfe926e70df 100644 --- a/include/trace/syscall.h +++ b/include/trace/syscall.h @@ -3,7 +3,6 @@ #define _TRACE_SYSCALL_H #include <linux/tracepoint.h> -#include <linux/unistd.h> #include <linux/trace_events.h> #include <linux/thread_info.h> diff --git a/include/uapi/linux/lsm.h b/include/uapi/linux/lsm.h index 33d8c9f4aa6b..523a53f12d4f 100644 --- a/include/uapi/linux/lsm.h +++ b/include/uapi/linux/lsm.h @@ -11,7 +11,6 @@ #include <linux/stddef.h> #include <linux/types.h> -#include <linux/unistd.h> /** * struct lsm_ctx - LSM context information diff --git a/kernel/exit.c b/kernel/exit.c index be81342caf1b..a78a6e97615a 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -13,6 +13,7 @@ #include <linux/sched/task.h> #include <linux/sched/task_stack.h> #include <linux/sched/cputime.h> +#include <linux/seccomp.h> #include <linux/interrupt.h> #include <linux/module.h> #include <linux/capability.h> diff --git a/kernel/ptrace.c b/kernel/ptrace.c index d5f89f9ef29f..1067da7a8409 100644 --- a/kernel/ptrace.c +++ b/kernel/ptrace.c @@ -14,6 +14,7 @@ #include <linux/sched/mm.h> #include <linux/sched/coredump.h> #include <linux/sched/task.h> +#include <linux/seccomp.h> #include <linux/errno.h> #include <linux/mm.h> #include <linux/highmem.h> diff --git a/lib/syscall.c b/lib/syscall.c index 006e256d2264..5fb67b3699b9 100644 --- a/lib/syscall.c +++ b/lib/syscall.c @@ -3,7 +3,7 @@ #include <linux/sched.h> #include <linux/sched/task_stack.h> #include <linux/export.h> -#include <asm/syscall.h> +#include <linux/syscalls_api.h> static int collect_syscall(struct task_struct *target, struct syscall_info *info) {
On Wed, Jul 17, 2024, at 08:41, Johan Hovold wrote: > On Wed, Jul 17, 2024 at 08:01:43AM +0200, Arnd Bergmann wrote: >> On Wed, Jul 17, 2024, at 07:08, Linus Torvalds wrote: >> > On Tue, 16 Jul 2024 at 21:57, Linus Torvalds <torvalds@linux-foundation.org> wrote: >> >> >> >> Note, it really might be just 'allmodconfig'. We've had things that >> >> depend on config entries in the past, eg the whole >> >> CONFIG_HEADERS_INSTALL etc could affect things. >> >> I had tried a partial allmodconfig build earlier to save time, >> did a full build again now, still nothing: > > FWIW, I noticed this last Friday as well when I did a few builds of > linux-next and every change I made triggered what appeared to be a full > rebuild of the tree. > > This was with a trimmed config [1] and separate build tree (tmpfs). Thanks, that makes it quicker to try out. I'm now using your config to do more testing. I still don't see it with a normal build though. I do see that setting the timestamp of syscall.tbl to a future date does result in always rebuilding everything, but I don't think that is what you are seeing, since that also produces a warning from make: arnd@studio:~/arm-soc/build/bisect$ touch -t 202501010000 arch/x86/entry/syscalls/syscall_64.tbl arnd@studio:~/arm-soc/build/bisect$ make ARCH=x86 CROSS_COMPILE=x86_64-linux- make[2]: Warning: File 'arch/x86/entry/syscalls/syscall_64.tbl' has modification time 14483017 s in the future SYSHDR arch/x86/include/generated/uapi/asm/unistd_64.h SYSHDR arch/x86/include/generated/uapi/asm/unistd_x32.h SYSHDR arch/x86/include/generated/asm/unistd_64_x32.h SYSTBL arch/x86/include/generated/asm/syscalls_64.h make[2]: warning: Clock skew detected. Your build may be incomplete. Arnd
On Wed, Jul 17, 2024 at 10:01:10AM +0200, Arnd Bergmann wrote: > On Wed, Jul 17, 2024, at 08:41, Johan Hovold wrote: > > FWIW, I noticed this last Friday as well when I did a few builds of > > linux-next and every change I made triggered what appeared to be a full > > rebuild of the tree. > > > > This was with a trimmed config [1] and separate build tree (tmpfs). > > Thanks, that makes it quicker to try out. I'm now using > your config to do more testing. I still don't see it with > a normal build though. > > I do see that setting the timestamp of syscall.tbl to > a future date does result in always rebuilding everything, > but I don't think that is what you are seeing, since that > also produces a warning from make: > > arnd@studio:~/arm-soc/build/bisect$ touch -t 202501010000 arch/x86/entry/syscalls/syscall_64.tbl > arnd@studio:~/arm-soc/build/bisect$ make ARCH=x86 CROSS_COMPILE=x86_64-linux- > make[2]: Warning: File 'arch/x86/entry/syscalls/syscall_64.tbl' has modification time 14483017 s in the future > SYSHDR arch/x86/include/generated/uapi/asm/unistd_64.h > SYSHDR arch/x86/include/generated/uapi/asm/unistd_x32.h > SYSHDR arch/x86/include/generated/asm/unistd_64_x32.h > SYSTBL arch/x86/include/generated/asm/syscalls_64.h > make[2]: warning: Clock skew detected. Your build may be incomplete. Yeah, that's not something I noticed at least (and I assume I would have). And I only did aarch64 builds on a 6.9 x86_64 host (make 4.4.1). Johan
On Wed, Jul 17, 2024, at 10:17, Johan Hovold wrote: > On Wed, Jul 17, 2024 at 10:01:10AM +0200, Arnd Bergmann wrote: > > Yeah, that's not something I noticed at least (and I assume I would > have). And I only did aarch64 builds on a 6.9 x86_64 host (make 4.4.1). Ok, I can reproduce the problem now: I installed a Fedora VM guest and chroot mount and I see the same issue in there. My normal Debian host has make 4.3, so I'll see if I can figure if a specific change in make does it. Arnd
On Wed, Jul 17, 2024, at 11:36, Arnd Bergmann wrote: > On Wed, Jul 17, 2024, at 10:17, Johan Hovold wrote: >> On Wed, Jul 17, 2024 at 10:01:10AM +0200, Arnd Bergmann wrote: >> >> Yeah, that's not something I noticed at least (and I assume I would >> have). And I only did aarch64 builds on a 6.9 x86_64 host (make 4.4.1). > > Ok, I can reproduce the problem now: I installed a Fedora > VM guest and chroot mount and I see the same issue in there. > > My normal Debian host has make 4.3, so I'll see if I can figure > if a specific change in make does it. I see that there is a version check in scripts/Makefile.include from commit 875ef1a57f32 ("kbuild: use .NOTINTERMEDIATE for future GNU Make versions") that detects Fedora's make 4.4.1 as newer than 4.4, so for the first time enables this logic that I did not see on Debian. In my scripts/Makefile.asm-headers, I had copied the 'FORCE' from the existing rules in arch/x86/entry/syscalls/Makefile etc without fully understanding what that does. It looks like this does not make a difference for make-4.3 but is actually wrong for make-4.4 on the generic rule. This makes it work for me with both versions of make: --- a/scripts/Makefile.asm-headers +++ b/scripts/Makefile.asm-headers @@ -77,14 +77,14 @@ all: $(generic-y) $(syscall-y) $(obj)/%.h: $(srctree)/$(generic)/%.h $(call cmd,wrap) -$(obj)/unistd_%.h: $(syscalltbl) $(syshdr) FORCE +$(obj)/unistd_%.h: $(syscalltbl) $(syshdr) $(call if_changed,syshdr) $(obj)/unistd_compat_%.h: syscall_compat:=1 -$(obj)/unistd_compat_%.h: $(syscalltbl) $(syshdr) FORCE +$(obj)/unistd_compat_%.h: $(syscalltbl) $(syshdr) $(call if_changed,syshdr) -$(obj)/syscall_table_%.h: $(syscalltbl) $(systbl) FORCE +$(obj)/syscall_table_%.h: $(syscalltbl) $(systbl) $(call if_changed,systbl) # Create output directory. Skip it if at least one old header exists Masahiro, does that make sense to you? I assume you can explain this properly, but I'll already send a patch with this version. Arnd
On Wed, Jul 17, 2024, at 12:54, Arnd Bergmann wrote: > > -$(obj)/syscall_table_%.h: $(syscalltbl) $(systbl) FORCE > +$(obj)/syscall_table_%.h: $(syscalltbl) $(systbl) > $(call if_changed,systbl) > > # Create output directory. Skip it if at least one old header exists > > Masahiro, does that make sense to you? I assume you can > explain this properly, but I'll already send a patch with > this version. This was not quite right either, but I sent a patch now that works with both old and new versions of make and has appears to have the intended behavior for incremental builds with or without changes to syscall.tbl: https://lore.kernel.org/lkml/20240717124253.2275084-1-arnd@kernel.org/T/ Arnd