From patchwork Sun Mar 3 23:24:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elliott Mitchell X-Patchwork-Id: 1907354 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=yxX+Yaeq; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.openwrt.org (client-ip=2607:7c80:54:3::133; helo=bombadil.infradead.org; envelope-from=openwrt-devel-bounces+incoming=patchwork.ozlabs.org@lists.openwrt.org; receiver=patchwork.ozlabs.org) Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:3::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Tp26h0R6vz23l2 for ; Mon, 4 Mar 2024 13:03:37 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:Subject:Date:In-Reply-To: References:Cc:To:From:Message-Id:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=U7yoW+JjkTNnUzr9QDVVc7DoOyju78q5gXVVexRsTyM=; b=yxX+YaeqNk2WjU Fvc612MtmuqF9qBffywfb8Pg0z4jK7VyGH4FF96XkhkD/k+BvStx4AM9jfW6cSyPjvg8w43mTq2oz EKkM6TxigAUXCEjwWNJB58WcIVFHIEB4KOxNNXSwO58TK1K1md9VIBnt+7M96NWW8jJaU7eDedCPX 4lh/38Cs5LYdxf5Gl/nH3NSq7FCxvZFm3rZ5QGzJcPi38ksOiEKoMdN7VCHCiqtlLEx3IwhIZ5Tgg OA5hSRs4G6sb/IyXz4n3zJsCS5j7EZ8r0QptZyIH8x08eo6TlOqXxqYP7kJ4EhiC6WPrnuY74M6wj /+d/L+du/sBHt4TjS+uw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rgxeL-00000007G6i-0xES; Mon, 04 Mar 2024 02:01:53 +0000 Received: from mailhost.m5p.com ([74.104.188.4]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rgxeH-00000007G3f-2e5u for openwrt-devel@lists.openwrt.org; Mon, 04 Mar 2024 02:01:51 +0000 Received: from m5p.com (mailhost.m5p.com [IPv6:2001:470:1f07:15ff:0:0:0:f7]) by mailhost.m5p.com (8.17.1/8.15.2) with ESMTPS id 424218Cp021730 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sun, 3 Mar 2024 21:01:13 -0500 (EST) (envelope-from ehem@m5p.com) Received: (from ehem@localhost) by m5p.com (8.17.1/8.15.2/Submit) id 42421827021729; Sun, 3 Mar 2024 18:01:08 -0800 (PST) (envelope-from ehem) Message-Id: <8adc3e8e0590d3f803d30c77a16779773a391231.1709516615.git.ehem+openwrt@m5p.com> From: Elliott Mitchell To: Rafa? Mi?ecki , Hauke Mehrtens , Jonas Gorski , Felix Baumann Cc: openwrt-devel@lists.openwrt.org References: <1a419e7f-b323-4952-bd03-33dbe880be80@gmail.com> <4690477d-e1b7-48e5-83d4-39d8c8e93191@hauke-m.de> <41e9cda304814eca2066a87cf2477c83933358f7.1707269880.git.ehem+openwrt@m5p.com> <414511fcb30b883565ae724aaf17141b104164ff.1707337693.git.ehem+openwrt@m5p.com> In-Reply-To: <1a419e7f-b323-4952-bd03-33dbe880be80@gmail.com> Date: Sun, 3 Mar 2024 15:24:50 -0800 Subject: [PATCH] scripts: create kernel configuration upgrade script X-Spam-Status: No, score=0.0 required=10.0 tests=KHOP_HELO_FCRDNS autolearn=unavailable autolearn_force=no version=4.0.0 X-Spam-Checker-Version: SpamAssassin 4.0.0 (2022-12-14) on mattapan.m5p.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240303_180149_895878_B3F4DAFD X-CRM114-Status: GOOD ( 18.77 ) X-Spam-Score: -0.0 (/) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Date: Tue, 6 Feb 2024 17:16:41 -0800 Create a script for automating kernel version changes. This generates a pair of commits which cause history to remain attached to all versioned configuration files. Content analysis details: (-0.0 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 SPF_HELO_PASS SPF: HELO matches SPF record -0.0 SPF_PASS SPF: sender matches SPF record -0.0 T_SCC_BODY_TEXT_LINE No description available. X-BeenThere: openwrt-devel@lists.openwrt.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: OpenWrt Development List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "openwrt-devel" Errors-To: openwrt-devel-bounces+incoming=patchwork.ozlabs.org@lists.openwrt.org Date: Tue, 6 Feb 2024 17:16:41 -0800 Create a script for automating kernel version changes. This generates a pair of commits which cause history to remain attached to all versioned configuration files. Crucially this makes `git blame` work without needing --find-copies-harder, which is too slow for routine use. This also updates *everything*, which greatly simplifies rebasing patches which effect multiple devices. Credit to Christian Marangi who knew of the technique: Signed-off-by: Elliott Mitchell --- v3: Dust off knowledge of PerlOO. Confine the fast-importer interface to an object. Better layer the lowest I/O layer. If fast-import grows a \0 command separation mode, we should be mostly ready (issue will be the commit message). Switch to SPDX. Try to match what the other scripts have. I was kind of hoping for more review activity, the near-silence is almost deafening. Using a script to handle this job seems best. I feel what this script produces is rather easier for most developers to handle. v2: Major tweaking. No longer try to do `git merge --ff-only `, but instead advise user to do so. Add usage message. Add statement about strategy. Adjust commit messages. Add advice to run `git bisect skip` if someone ends up on that commit. --- scripts/kernel_upgrade.pl | 280 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 280 insertions(+) create mode 100755 scripts/kernel_upgrade.pl diff --git a/scripts/kernel_upgrade.pl b/scripts/kernel_upgrade.pl new file mode 100755 index 0000000000..c2e5fb6078 --- /dev/null +++ b/scripts/kernel_upgrade.pl @@ -0,0 +1,280 @@ +#!/usr/bin/env perl +######################################################################### +# SPDX-License-Identifier: GPL-3.0-or-later # +# # +# Copyright (C) 2024 Elliott Mitchell # +######################################################################### + +use warnings; +use strict; + +# +# I'm not certain the technique originated here, but this comes from: +# +# +# Problem is copying a file in git causes the new file to be created +# without any history. Files can move around without losing their +# history, but that only leaves the history on the new location. +# +# As such this can be solved with two commits. The first commit moves +# files from their old name to their new name. The second merges the +# original commit with the rename commit. The merge commit then has +# files in both locations with the full history. +# +# +# Note, git handles discarded data by garbage collection. When doing +# development on this script, beware this script is an excellent +# garbage generator. Frequent use of `git gc` and `git prune` may be +# needed. +# + + +sub gethead() +{ + open(my $fd, '-|', 'git', 'rev-parse', 'HEAD'); + $_=<$fd>; + chop; + return $_; +} + +sub getlist($$) +{ + my ($target, $from)=@_; + my $ret=[]; + + local $/="\0"; + open(my $fd, '-| :raw :bytes', 'git', 'ls-tree', '-trz', '--full-name', +'--name-only', 'HEAD', '--', $target)||die("failed to read git tree"); + + while(<$fd>) { + chop($_); + push(@$ret, substr($_, 0, -length($from))) +if(substr($_, -length($from)) eq $from); + } + + @$ret=sort({length($b)-length($a)} @$ret); + + return $ret; +} + +{ # start of interface to git fast-import +package GitImporter; + +# git fast-import's protocol uses *linefeeds* +local $/="\n"; + +sub new() +{ + my $class=shift; + my $self={}; + my @child; + (pipe($child[0], $self->{out})&&pipe($self->{in}, $child[1])) || +die("pipe() failed"); + binmode($self->{out}); + binmode($self->{in}); + + $self->{pid}=fork(); + if($self->{pid}==0) { + close($self->{out}); + close($self->{in}); + + open(STDIN, '<&', $child[0]); + close($child[0]); + + open(STDOUT, '>&', $child[1]); + close($child[1]); + + exec('git', 'fast-import', '--done'); + die('exec() of git failed'); + } elsif(!$self->{pid}) { + die('fork() failed'); + } + close($child[0]); + close($child[1]); + $self->{out}->autoflush(1); + + return bless($self, $class); +} + +sub send($) +{ + my $self=shift; + my ($data)=@_; + return print({$self->{out}} $data); +} + +sub putcmd($) +{ + my $self=shift; + my ($data)=@_; + return $self->send("$data\n"); +} + +sub recv() +{ + my $self=shift; + return $_=readline($self->{in}); +} + +sub getres() +{ + my $self=shift; + $_=$self->recv(); + chomp; + return $_; +} + +sub ls($$) +{ + my $self=shift; + my ($commit, $name)=@_; + + $commit.=' ' if($commit); + $self->putcmd("ls $commit$name"); + $self->getres(); + + die('git ls failed') unless(/^([0-8]+)\s+[a-z]+\s+([0-9a-z]+)\s+.+$/); + + return [$1, $2]; +} + +sub commit($$$) +{ + my $self=shift; + my ($dest, $message, $mark)=@_; + + use feature 'state'; + use Digest::SHA qw(sha1_hex); + + state $author=undef; + unless($author) { + $author=['', '']; + open(my $user, '-|', 'git', 'config', '--get', 'user.name'); + while(<$user>) { + chomp; + $author->[0].=$_; + } + $author->[0]=[split(/,/, [getpwuid($<)]->[6])]->[0] +unless($author->[0]); + + open(my $email, '-|', 'git', 'config', '--get', 'user.email'); + while(<$email>) { + chomp; + $author->[1].=$_; + } + $author->[1]='anonymous@example.com' unless($author->[1]); + + $author=$author->[0].' <'.$author->[1].'>'; + } + + $_=sha1_hex(time()); + $self->putcmd("commit $_"); + $self->putcmd("mark $mark"); + $self->putcmd("committer $author ".time()." +0000"); + + $_=length($message); + $self->putcmd("data $_"); + $self->send($message); + $self->putcmd("from $dest"); +} + +sub DESTROY() +{ + my $self=shift; + + $self->putcmd('done'); + close($self->{out}); + delete $self->{out}; + + 0 while(waitpid($self->{pid}, 0)!=$self->{pid}); + delete $self->{pid}; + close($self->{in}); + delete $self->{in}; + + print(STDERR <<~"__GIT_STATUS__") if($?); + WARNING: git returned error exit status: $? + + This likely means `git gc` needs to be run, but the return codes of + `git fast-import` are undocumented. + + __GIT_STATUS__ +} +} # end of interface to git fast-import + + +die(<<"__USAGE__") if(@ARGV!=2); +Usage: $0 + +Copies all kernel configuration files and patches from the old version +to the new version. Git history is preserved on the copies by using a +move/merge strategy. Must be run while somewhere inside the git +repository directory, but it does not matter where. +__USAGE__ + +my ($from, $to)=@ARGV; + + +my $target='target/linux'; + +my $start=gethead(); + +my $list=getlist($target, $from); + +die("no files matching \"$from\" found") unless(@$list); + + +my $git=GitImporter->new(); + +$git->commit($start, <<"__TMP__", ':1'); +kernel: add configs and patches for $to + +This is a special tool-generated commit. + +Copy configuration and patches from $from to $to. + +If you see this commit during a `git bisect` session, the recommended +course of action is to run `git bisect skip`. +__TMP__ + +foreach my $name (@$list) { + my $new=$git->ls($start, "$name$from"); + $git->putcmd("M $new->[0] $new->[1] $name$to"); + $git->putcmd("D $name$from"); +} +$git->putcmd(''); + + +$git->commit(':1', <<"__TMP__", ':2'); +kernel: finish update from $from to $to + +This is a special tool-generated commit. + +Merge the add commit into HEAD to create all files with full history. +__TMP__ + +$git->putcmd("merge $start"); + +foreach my $name (@$list) { + my $new=$git->ls($start, "$name$from"); + $git->putcmd("M $new->[0] $new->[1] $name$from"); +} +$git->putcmd(''); + + +$git->putcmd('get-mark :2'); +my $result=$git->getres(); + +undef($git); + +print(<<"__END__"); +Result is commit $result. + +Depending on the setup of your development environment, you now likely +want to run one of two commands: + + `git merge --ff-only $result` +or: + `git branch linux-$to $result` +__END__ + +exit(0);