scripts: create kernel configuration upgrade script

Date: Tue, 6 Feb 2024 17:16:41 -0800

Create a script for automating kernel version changes.  This
generates a pair of commits which cause history to remain attached to
all versioned configuration files.

Crucially this makes `git blame` work without needing
--find-copies-harder, which is too slow for routine use.  This also
updates *everything*, which greatly simplifies rebasing patches
which effect multiple devices.

Credit to Christian Marangi who knew of the technique:
<https://lists.openwrt.org/pipermail/openwrt-devel/2023-October/041672.html>

Signed-off-by: Elliott Mitchell <ehem+openwrt@m5p.com>
---
v3:
Dust off knowledge of PerlOO.  Confine the fast-importer interface
to an object.

Better layer the lowest I/O layer.  If fast-import grows a \0 command
separation mode, we should be mostly ready (issue will be the commit
message).

Switch to SPDX.  Try to match what the other scripts have.

I was kind of hoping for more review activity, the near-silence is
almost deafening.  Using a script to handle this job seems best.  I
feel what this script produces is rather easier for most developers
to handle.

v2:
Major tweaking.  No longer try to do `git merge --ff-only <hash>`,
but instead advise user to do so.

Add usage message.

Add statement about strategy.

Adjust commit messages.  Add advice to run `git bisect skip` if
someone ends up on that commit.
---
 scripts/kernel_upgrade.pl | 280 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 280 insertions(+)
 create mode 100755 scripts/kernel_upgrade.pl

Hi Elliott,

Op zondag 3 maart 2024 om 15:24:50 -08:00:00 schreef Elliott Mitchell 
<ehem+openwrt@m5p.com>:
> Date: Tue, 6 Feb 2024 17:16:41 -0800
> 
> Create a script for automating kernel version changes.  This
> generates a pair of commits which cause history to remain attached to
> all versioned configuration files.
> 
> Crucially this makes `git blame` work without needing
> --find-copies-harder, which is too slow for routine use.  This also
> updates *everything*, which greatly simplifies rebasing patches
> which effect multiple devices.
> 
> Credit to Christian Marangi who knew of the technique:
> <https://lists.openwrt.org/pipermail/openwrt-devel/2023-October/041672.html>
> 
> Signed-off-by: Elliott Mitchell <ehem+openwrt@m5p.com>


Is there a way to bump a specific target to a new kernel with your 
script? It doesn't look like it, but I might be mistaken. Would it be a 
lot of work to integrate that? With the shell equivalent being merged, 
I can understand any reluctance to adding this functionality, but it's 
worth asking.

Thanks

Stijn


> ---
> v3:
> Dust off knowledge of PerlOO.  Confine the fast-importer interface
> to an object.
> 
> Better layer the lowest I/O layer.  If fast-import grows a \0 command
> separation mode, we should be mostly ready (issue will be the commit
> message).
> 
> Switch to SPDX.  Try to match what the other scripts have.
> 
> I was kind of hoping for more review activity, the near-silence is
> almost deafening.  Using a script to handle this job seems best.  I
> feel what this script produces is rather easier for most developers
> to handle.
> 
> v2:
> Major tweaking.  No longer try to do `git merge --ff-only <hash>`,
> but instead advise user to do so.
> 
> Add usage message.
> 
> Add statement about strategy.
> 
> Adjust commit messages.  Add advice to run `git bisect skip` if
> someone ends up on that commit.
> ---
>  scripts/kernel_upgrade.pl | 280 
> ++++++++++++++++++++++++++++++++++++++
>  1 file changed, 280 insertions(+)
>  create mode 100755 scripts/kernel_upgrade.pl
> 
> diff --git a/scripts/kernel_upgrade.pl b/scripts/kernel_upgrade.pl
> new file mode 100755
> index 0000000000..c2e5fb6078
> --- /dev/null
> +++ b/scripts/kernel_upgrade.pl
> @@ -0,0 +1,280 @@
> +#!/usr/bin/env perl
> +#########################################################################
> +# SPDX-License-Identifier: GPL-3.0-or-later				#
> +#									#
> +# Copyright (C) 2024 Elliott Mitchell <ehem+openwrt@m5p.com>		#
> +#########################################################################
> +
> +use warnings;
> +use strict;
> +
> +#
> +# I'm not certain the technique originated here, but this comes from:
> +# <https://devblogs.microsoft.com/oldnewthing/20190919-00/?p=102904>
> +#
> +# Problem is copying a file in git causes the new file to be created
> +# without any history.  Files can move around without losing their
> +# history, but that only leaves the history on the new location.
> +#
> +# As such this can be solved with two commits.  The first commit 
> moves
> +# files from their old name to their new name.  The second merges the
> +# original commit with the rename commit.  The merge commit then has
> +# files in both locations with the full history.
> +#
> +#
> +# Note, git handles discarded data by garbage collection.  When doing
> +# development on this script, beware this script is an excellent
> +# garbage generator.  Frequent use of `git gc` and `git prune` may be
> +# needed.
> +#
> +
> +
> +sub gethead()
> +{
> +	open(my $fd, '-|', 'git', 'rev-parse', 'HEAD');
> +	$_=<$fd>;
> +	chop;
> +	return $_;
> +}
> +
> +sub getlist($$)
> +{
> +	my ($target, $from)=@_;
> +	my $ret=[];
> +
> +	local $/="\0";
> +	open(my $fd, '-| :raw :bytes', 'git', 'ls-tree', '-trz', 
> '--full-name',
> +'--name-only', 'HEAD', '--', $target)||die("failed to read git 
> tree");
> +
> +	while(<$fd>) {
> +		chop($_);
> +		push(@$ret, substr($_, 0, -length($from)))
> +if(substr($_, -length($from)) eq $from);
> +	}
> +
> +	@$ret=sort({length($b)-length($a)} @$ret);
> +
> +	return $ret;
> +}
> +
> +{ # start of interface to git fast-import
> +package GitImporter;
> +
> +# git fast-import's protocol uses *linefeeds*
> +local $/="\n";
> +
> +sub new()
> +{
> +	my $class=shift;
> +	my $self={};
> +	my @child;
> +	(pipe($child[0], $self->{out})&&pipe($self->{in}, $child[1])) ||
> +die("pipe() failed");
> +	binmode($self->{out});
> +	binmode($self->{in});
> +
> +	$self->{pid}=fork();
> +	if($self->{pid}==0) {
> +		close($self->{out});
> +		close($self->{in});
> +
> +		open(STDIN, '<&', $child[0]);
> +		close($child[0]);
> +
> +		open(STDOUT, '>&', $child[1]);
> +		close($child[1]);
> +
> +		exec('git', 'fast-import', '--done');
> +		die('exec() of git failed');
> +	} elsif(!$self->{pid}) {
> +		die('fork() failed');
> +	}
> +	close($child[0]);
> +	close($child[1]);
> +	$self->{out}->autoflush(1);
> +
> +	return bless($self, $class);
> +}
> +
> +sub send($)
> +{
> +	my $self=shift;
> +	my ($data)=@_;
> +	return print({$self->{out}} $data);
> +}
> +
> +sub putcmd($)
> +{
> +	my $self=shift;
> +	my ($data)=@_;
> +	return $self->send("$data\n");
> +}
> +
> +sub recv()
> +{
> +	my $self=shift;
> +	return $_=readline($self->{in});
> +}
> +
> +sub getres()
> +{
> +	my $self=shift;
> +	$_=$self->recv();
> +	chomp;
> +	return $_;
> +}
> +
> +sub ls($$)
> +{
> +	my $self=shift;
> +	my ($commit, $name)=@_;
> +
> +	$commit.=' ' if($commit);
> +	$self->putcmd("ls $commit$name");
> +	$self->getres();
> +
> +	die('git ls failed') 
> unless(/^([0-8]+)\s+[a-z]+\s+([0-9a-z]+)\s+.+$/);
> +
> +	return [$1, $2];
> +}
> +
> +sub commit($$$)
> +{
> +	my $self=shift;
> +	my ($dest, $message, $mark)=@_;
> +
> +	use feature 'state';
> +	use Digest::SHA qw(sha1_hex);
> +
> +	state $author=undef;
> +	unless($author) {
> +		$author=['', ''];
> +		open(my $user, '-|', 'git', 'config', '--get', 'user.name');
> +		while(<$user>) {
> +			chomp;
> +			$author->[0].=$_;
> +		}
> +		$author->[0]=[split(/,/, [getpwuid($<)]->[6])]->[0]
> +unless($author->[0]);
> +
> +		open(my $email, '-|', 'git', 'config', '--get', 'user.email');
> +		while(<$email>) {
> +			chomp;
> +			$author->[1].=$_;
> +		}
> +		$author->[1]='anonymous@example.com' unless($author->[1]);
> +
> +		$author=$author->[0].' <'.$author->[1].'>';
> +	}
> +
> +	$_=sha1_hex(time());
> +	$self->putcmd("commit $_");
> +	$self->putcmd("mark $mark");
> +	$self->putcmd("committer $author ".time()." +0000");
> +
> +	$_=length($message);
> +	$self->putcmd("data $_");
> +	$self->send($message);
> +	$self->putcmd("from $dest");
> +}
> +
> +sub DESTROY()
> +{
> +	my $self=shift;
> +
> +	$self->putcmd('done');
> +	close($self->{out});
> +	delete $self->{out};
> +
> +	0 while(waitpid($self->{pid}, 0)!=$self->{pid});
> +	delete $self->{pid};
> +	close($self->{in});
> +	delete $self->{in};
> +
> +	print(STDERR <<~"__GIT_STATUS__") if($?);
> +	WARNING: git returned error exit status: $?
> +
> +	This likely means `git gc` needs to be run, but the return codes of
> +	`git fast-import` are undocumented.
> +
> +	__GIT_STATUS__
> +}
> +} # end of interface to git fast-import
> +
> +
> +die(<<"__USAGE__") if(@ARGV!=2);
> +Usage: $0 <old-version> <new-version>
> +
> +Copies all kernel configuration files and patches from the old 
> version
> +to the new version.  Git history is preserved on the copies by using 
> a
> +move/merge strategy.  Must be run while somewhere inside the git
> +repository directory, but it does not matter where.
> +__USAGE__
> +
> +my ($from, $to)=@ARGV;
> +
> +
> +my $target='target/linux';
> +
> +my $start=gethead();
> +
> +my $list=getlist($target, $from);
> +
> +die("no files matching \"$from\" found") unless(@$list);
> +
> +
> +my $git=GitImporter->new();
> +
> +$git->commit($start, <<"__TMP__", ':1');
> +kernel: add configs and patches for $to
> +
> +This is a special tool-generated commit.
> +
> +Copy configuration and patches from $from to $to.
> +
> +If you see this commit during a `git bisect` session, the recommended
> +course of action is to run `git bisect skip`.
> +__TMP__
> +
> +foreach my $name (@$list) {
> +	my $new=$git->ls($start, "$name$from");
> +	$git->putcmd("M $new->[0] $new->[1] $name$to");
> +	$git->putcmd("D $name$from");
> +}
> +$git->putcmd('');
> +
> +
> +$git->commit(':1', <<"__TMP__", ':2');
> +kernel: finish update from $from to $to
> +
> +This is a special tool-generated commit.
> +
> +Merge the add commit into HEAD to create all files with full history.
> +__TMP__
> +
> +$git->putcmd("merge $start");
> +
> +foreach my $name (@$list) {
> +	my $new=$git->ls($start, "$name$from");
> +	$git->putcmd("M $new->[0] $new->[1] $name$from");
> +}
> +$git->putcmd('');
> +
> +
> +$git->putcmd('get-mark :2');
> +my $result=$git->getres();
> +
> +undef($git);
> +
> +print(<<"__END__");
> +Result is commit $result.
> +
> +Depending on the setup of your development environment, you now 
> likely
> +want to run one of two commands:
> +
> +	`git merge --ff-only $result`
> +or:
> +	`git branch linux-$to $result`
> +__END__
> +
> +exit(0);
> --
> 2.39.2
> 
> 
> _______________________________________________
> openwrt-devel mailing list
> openwrt-devel@lists.openwrt.org
> https://lists.openwrt.org/mailman/listinfo/openwrt-devel

On Sat, Mar 23, 2024 at 07:56:01PM +0100, Stijn Segers wrote:
> 
> Op zondag 3 maart 2024 om 15:24:50 -08:00:00 schreef Elliott Mitchell 
> <ehem+openwrt@m5p.com>:
> > Date: Tue, 6 Feb 2024 17:16:41 -0800
> > 
> > Create a script for automating kernel version changes.  This
> > generates a pair of commits which cause history to remain attached to
> > all versioned configuration files.
> > 
> > Crucially this makes `git blame` work without needing
> > --find-copies-harder, which is too slow for routine use.  This also
> > updates *everything*, which greatly simplifies rebasing patches
> > which effect multiple devices.
> > 
> > Credit to Christian Marangi who knew of the technique:
> > <https://lists.openwrt.org/pipermail/openwrt-devel/2023-October/041672.html>
> > 
> > Signed-off-by: Elliott Mitchell <ehem+openwrt@m5p.com>
> 
> 
> Is there a way to bump a specific target to a new kernel with your 
> script? It doesn't look like it, but I might be mistaken. Would it be a 
> lot of work to integrate that? With the shell equivalent being merged, 
> I can understand any reluctance to adding this functionality, but it's 
> worth asking.

As originally written, no.  I see significant advantages to that approach
and it really is starting to look like the best balance.

To add the ability to handle a single target, near trivial.  To add the
ability to do multiple target(s), very simple.  To do this efficiently,
still fairly simple.

It does seem this list has become useless for patch submission, so it has
been brought onto GitHub:

https://github.com/openwrt/openwrt/pull/14907

Sorry to hijack into this thread but about having a more friendly way to 
configure the kernel consistently across versions I wrote this little 
utility which I sue to configure both Linux Kernel and OpenWrt 
programmatically in a way which make it very very easy to upgrade versions

https://gitlab.com/g10h4ck/kconfig-utils

I have researched quite a bit into this topic because I need to deal 
with kconfig based stuff in many places, and finally implemented this 
which is quite sustainable even for one single person

A side note is that it seems quite dumb and frustrating to me that 
KConfig doesn't support a way to configure the things programmatically 
by it self, when it already support a way to do that from an interactive 
menu which is probably much more cumbersome to expose a CLI to just 
set/unset what is needed and report an error if something goes wrong... 
(well the tool i have implemented does just that)

Cheers

Gio


On 2024-03-24 20:00, Elliott Mitchell wrote:
> On Sat, Mar 23, 2024 at 07:56:01PM +0100, Stijn Segers wrote:
>> Op zondag 3 maart 2024 om 15:24:50 -08:00:00 schreef Elliott Mitchell
>> <ehem+openwrt@m5p.com>:
>>> Date: Tue, 6 Feb 2024 17:16:41 -0800
>>>
>>> Create a script for automating kernel version changes.  This
>>> generates a pair of commits which cause history to remain attached to
>>> all versioned configuration files.
>>>
>>> Crucially this makes `git blame` work without needing
>>> --find-copies-harder, which is too slow for routine use.  This also
>>> updates *everything*, which greatly simplifies rebasing patches
>>> which effect multiple devices.
>>>
>>> Credit to Christian Marangi who knew of the technique:
>>> <https://lists.openwrt.org/pipermail/openwrt-devel/2023-October/041672.html>
>>>
>>> Signed-off-by: Elliott Mitchell <ehem+openwrt@m5p.com>
>>
>> Is there a way to bump a specific target to a new kernel with your
>> script? It doesn't look like it, but I might be mistaken. Would it be a
>> lot of work to integrate that? With the shell equivalent being merged,
>> I can understand any reluctance to adding this functionality, but it's
>> worth asking.
> As originally written, no.  I see significant advantages to that approach
> and it really is starting to look like the best balance.
>
> To add the ability to handle a single target, near trivial.  To add the
> ability to do multiple target(s), very simple.  To do this efficiently,
> still fairly simple.
>
> It does seem this list has become useless for patch submission, so it has
> been brought onto GitHub:
>
> https://github.com/openwrt/openwrt/pull/14907
>
>