From patchwork Thu Dec 8 07:17:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Juerg Haefliger X-Patchwork-Id: 1713582 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.ubuntu.com (client-ip=91.189.94.19; helo=huckleberry.canonical.com; envelope-from=kernel-team-bounces@lists.ubuntu.com; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=vDnBAAsA; dkim-atps=neutral Received: from huckleberry.canonical.com (huckleberry.canonical.com [91.189.94.19]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4NSQTm3YCyz2405 for ; Thu, 8 Dec 2022 18:17:47 +1100 (AEDT) Received: from localhost ([127.0.0.1] helo=huckleberry.canonical.com) by huckleberry.canonical.com with esmtp (Exim 4.86_2) (envelope-from ) id 1p3BA3-0004dQ-A9; Thu, 08 Dec 2022 07:17:39 +0000 Received: from smtp-relay-internal-0.internal ([10.131.114.225] helo=smtp-relay-internal-0.canonical.com) by huckleberry.canonical.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.86_2) (envelope-from ) id 1p3B9v-0004aS-Hg for kernel-team@lists.ubuntu.com; Thu, 08 Dec 2022 07:17:31 +0000 Received: from mail-ej1-f69.google.com (mail-ej1-f69.google.com [209.85.218.69]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-internal-0.canonical.com (Postfix) with ESMTPS id 4D4AE44347 for ; Thu, 8 Dec 2022 07:17:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1670483851; bh=hm/HO/Dbvaae/I+DVW/ScgwY5kZ5vWtbvvlg2Gbzovc=; h=From:To:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=vDnBAAsA9TEjArjfClb9USd62pgp36Ok3HFzvRcIwkzJulVaaIg94kcZEtm73y/Mj A4N6iAMBWnTgQIJloIg8W/pXNi2hFwV1Msp9YlfHE8b6oVprA5dqPXRJOCqzjnd8bl rZB27K9Oit6ckXtsCmT1041UceZwK+e/tPWljqtvDYNTQ81Ohh11alLohriVTAIqWp SPQhkfeRhX6Wp2716om+gT8aXyFpWUlVmt3gH9ZL6o1l+h8HDGLBi8qY23nFcdO2MJ ijj6o3j7NZYsFOm5DjOikJWyfj8/FLj7I/ZDcmwlV1tzygMRG7xHjT4U11dpRNX5Uo 3jExl+qMvR+eA== Received: by mail-ej1-f69.google.com with SMTP id gn36-20020a1709070d2400b007ba3374574dso512797ejc.23 for ; Wed, 07 Dec 2022 23:17:31 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=hm/HO/Dbvaae/I+DVW/ScgwY5kZ5vWtbvvlg2Gbzovc=; b=6vv6UrHhiu5udjwik8Vc2gd0ldagZq3K6Zhv1e/UvYK3TDBVji77ByIypOLl2J6tXD 7Jq2Qoyqfu9vWguGF0VaZPpBMV/mzhn7kcuSK7v4o1jLxb2AMtg/IlZFZtb/mQmnWBlW 8z7p18LCNA2xdrCkKka+ggjlUZmN+LYSqAUbAxfAy0pE062xC6XBIpimWB/I8ZAz+otu dDZF3A1oe/QP9d/G25bEULai6f92wHbfzR4SEnJ4HevX+BDKTIABfeiXL9KmEkyjY62p OdD6f5DOqcOQhrx3TUDTO6Mi+Xm2i1TboeCpy/UGwG4O99YL30F6IxPqiCLJdXhOLwNA 4wOA== X-Gm-Message-State: ANoB5pn0E/003hiH41rcMz+QX5bJLJ9dIqR/jAHbge/FWP0H02E4YVzw lsXyxbV5JXu9cDUdPqHzVpXIrr1a5w0R4aCZzoq7+4tStcJ/HxYsThLcbrYEspYXzXRv+HrNKcH Uymb8o4DZhp429Q7Sb487usdiz9AoecBKTgqFgT0/3Q== X-Received: by 2002:a17:907:d18:b0:78d:f604:2c01 with SMTP id gn24-20020a1709070d1800b0078df6042c01mr2346769ejc.9.1670483850881; Wed, 07 Dec 2022 23:17:30 -0800 (PST) X-Google-Smtp-Source: AA0mqf7pu9npYYVsiWzeUNixm/lM4uc9yLILPEqp8Ar2TEIBYOcDieCYy5mdowvLnEPLLoq7gOjXGA== X-Received: by 2002:a17:907:d18:b0:78d:f604:2c01 with SMTP id gn24-20020a1709070d1800b0078df6042c01mr2346758ejc.9.1670483850545; Wed, 07 Dec 2022 23:17:30 -0800 (PST) Received: from gollum.fritz.box ([194.191.244.86]) by smtp.gmail.com with ESMTPSA id 4-20020a170906308400b007bd9e683639sm9346891ejv.130.2022.12.07.23.17.29 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Dec 2022 23:17:30 -0800 (PST) From: Juerg Haefliger To: kernel-team@lists.ubuntu.com Subject: [Unstable][PATCH 1/4] UBUNTU: [Packaging] Rewrite debian/scripts/abi-check in Python Date: Thu, 8 Dec 2022 08:17:25 +0100 Message-Id: <20221208071728.193067-2-juerg.haefliger@canonical.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221208071728.193067-1-juerg.haefliger@canonical.com> References: <20221208071728.193067-1-juerg.haefliger@canonical.com> MIME-Version: 1.0 X-BeenThere: kernel-team@lists.ubuntu.com X-Mailman-Version: 2.1.20 Precedence: list List-Id: Kernel team discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: kernel-team-bounces@lists.ubuntu.com Sender: "kernel-team" Rewrite the ABI checker script in Python. While at it, drop the check for symbol hash changes since this has been ignored/disabled for ages now. Signed-off-by: Juerg Haefliger --- debian/rules.d/4-checks.mk | 4 +- debian/scripts/abi-check | 345 +++++++++++++++---------------------- 2 files changed, 139 insertions(+), 210 deletions(-) diff --git a/debian/rules.d/4-checks.mk b/debian/rules.d/4-checks.mk index 7eaae529e5c1..68ac5471c6e4 100644 --- a/debian/rules.d/4-checks.mk +++ b/debian/rules.d/4-checks.mk @@ -1,8 +1,8 @@ # Check ABI for package against last release (if not same abinum) abi-check-%: $(stampdir)/stamp-install-% @echo Debug: $@ - @perl -f $(DROOT)/scripts/abi-check "$*" "$(prev_abinum)" "$(abinum)" \ - "$(prev_abidir)" "$(abidir)" "$(skipabi)" + $(DROOT)/scripts/abi-check "$*" \ + "$(prev_abidir)" "$(abidir)" $(skipabi) # Check the module list against the last release (always) module-check-%: $(stampdir)/stamp-install-% diff --git a/debian/scripts/abi-check b/debian/scripts/abi-check index d065f3d698c0..5bc1c935d7e0 100755 --- a/debian/scripts/abi-check +++ b/debian/scripts/abi-check @@ -1,210 +1,139 @@ -#!/usr/bin/perl -w - -my $flavour = shift; -my $prev_abinum = shift; -my $abinum = shift; -my $prev_abidir = shift; -my $abidir = shift; -my $skipabi = shift; - -my $fail_exit = 1; -my $EE = "EE:"; -my $errors = 0; -my $abiskip = 0; - -my $count; - -print "II: Checking ABI for $flavour...\n"; - -if (-f "$prev_abidir/ignore" - or -f "$prev_abidir/$flavour.ignore" or "$skipabi" eq "true") { - print "WW: Explicitly asked to ignore ABI, running in no-fail mode\n"; - $fail_exit = 0; - $abiskip = 1; - $EE = "WW:"; -} - -if ($prev_abinum != $abinum) { - print "II: Different ABI's, running in no-fail mode\n"; - $fail_exit = 0; - $EE = "WW:"; -} - -if (not -f "$abidir/$flavour" or not -f "$prev_abidir/$flavour") { - print "EE: Previous or current ABI file missing!\n"; - print " $abidir/$flavour\n" if not -f "$abidir/$flavour"; - print " $prev_abidir/$flavour\n" if not -f "$prev_abidir/$flavour"; - - # Exit if the ABI files are missing, but return status based on whether - # skip ABI was indicated. - if ("$abiskip" eq "1") { - exit(0); - } else { - exit(1); - } -} - -my %symbols; -my %symbols_ignore; -my %modules_ignore; -my %module_syms; +#!/usr/bin/python3 +# +# Check ABI changes +# +# To skip the ABI checks, add a file +# debian./abi//ignore.abi +# or +# debian./abi//.ignore.abi +# +# To ignore a list of symbols, add the symbols to the file +# debian./abi/abi.ignore +# + +import os +import re +import sys + +def decode_symline(line): + comps = re.sub(r'\s+', ' ', line).split(' ') + if comps[0].startswith('EXPORT_SYMBOL'): + stype, sloc, shash, sname = comps + else: + stype, shash, sname, sloc = comps[1:] + return sname, {'type': stype, 'loc': sloc, 'hash': shash} + +if len(sys.argv) < 4 or len(sys.argv) > 5: + print('Usage: abi-check []') + sys.exit(2) + +flavor, prev_abidir, abidir = sys.argv[1:4] # pylint: disable=W0632 +if len(sys.argv) > 4: + skipabi = sys.argv[4].lower() in ['1', 'true', 'yes'] +else: + skipabi = False + +print('II: Checking ABI for {}...'.format(flavor), end='') + +if ((os.path.exists('{}/ignore.abi'.format(prev_abidir)) or + os.path.exists('{}/{}.ignore.abi'.format(prev_abidir, flavor)))): + print('WW: Explicitly ignoring ABI') + print('II: Done') + sys.exit(0) + +curr_abi = '{}/{}'.format(abidir, flavor) +prev_abi = '{}/{}'.format(prev_abidir, flavor) +if not os.path.exists(curr_abi) or not os.path.exists(prev_abi): + print('II: Previous or current ABI file missing!') + print(' {}'.format(curr_abi)) + print(' {}'.format(prev_abi)) + if skipabi: + print('WW: Explicitly asked to ignore failures') + print('II: Done') + sys.exit(0) + print('EE: Missing ABI file') + sys.exit(1) + +print() + +symbols = {} +symbols_ignore = {} # See if we have any ignores -my $ignore = 0; -print " Reading symbols/modules to ignore..."; - -for $file ("$prev_abidir/../blacklist") { - if (-f $file) { - open(IGNORE, "< $file") or - die "Could not open $file"; - while () { - chomp; - if ($_ =~ m/M: (.*)/) { - $modules_ignore{$1} = 1; - } else { - $symbols_ignore{$_} = 1; - } - $ignore++; - } - close(IGNORE); - } -} -print "read $ignore symbols/modules.\n"; - -sub is_ignored($$) { - my ($mod, $sym) = @_; - - die "Missing module name in is_ignored()" if not defined($mod); - die "Missing symbol name in is_ignored()" if not defined($sym); - - if (defined($symbols_ignore{$sym}) or defined($modules_ignore{$mod})) { - return 1; - } - return 0; -} - -# Read new syms first -print " Reading new symbols ($abinum)..."; -$count = 0; -open(NEW, "< $abidir/$flavour") or - die "Could not open $abidir/$flavour"; -while () { - chomp; - m/^(\S+)\s(.+)\s(0x[0-9a-f]+)\s(.+)$/; - $symbols{$4}{'type'} = $1; - $symbols{$4}{'loc'} = $2; - $symbols{$4}{'hash'} = $3; - $module_syms{$2} = 0; - $count++; -} -close(NEW); -print "read $count symbols.\n"; - -# Now the old symbols, checking for missing ones -print " Reading old symbols ($prev_abinum)..."; -$count = 0; -open(OLD, "< $prev_abidir/$flavour") or - die "Could not open $prev_abidir/$flavour"; -while () { - chomp; - m/^(\S+)\s(.+)\s(0x[0-9a-f]+)\s(.+)$/; - $symbols{$4}{'old_type'} = $1; - $symbols{$4}{'old_loc'} = $2; - $symbols{$4}{'old_hash'} = $3; - $count++; -} -close(OLD); - -print "read $count symbols.\n"; - -print "II: Checking for missing symbols in new ABI..."; -$count = 0; -foreach $sym (keys(%symbols)) { - if (!defined($symbols{$sym}{'type'})) { - print "\n" if not $count; - printf(" MISS : %s%s\n", $sym, - is_ignored($symbols{$sym}{'old_loc'}, $sym) ? " (ignored)" : ""); - $count++ if !is_ignored($symbols{$sym}{'old_loc'}, $sym); - } -} -print " " if $count; -print "found $count missing symbols\n"; -if ($count) { - print "$EE Symbols gone missing (what did you do!?!)\n"; - $errors++; -} - - -print "II: Checking for new symbols in new ABI..."; -$count = 0; -foreach $sym (keys(%symbols)) { - if (!defined($symbols{$sym}{'old_type'})) { - print "\n" if not $count; - print " NEW : $sym\n"; - $count++; - } -} -print " " if $count; -print "found $count new symbols\n"; -if ($count and $prev_abinum == $abinum) { - print "WW: Found new symbols within same ABI. Not recommended\n"; -} - -print "II: Checking for changes to ABI...\n"; -$count = 0; -my $moved = 0; -my $changed_type = 0; -my $changed_hash = 0; -foreach $sym (keys(%symbols)) { - if (!defined($symbols{$sym}{'old_type'}) or - !defined($symbols{$sym}{'type'})) { - next; - } - - # Changes in location don't hurt us, but log it anyway - if ($symbols{$sym}{'loc'} ne $symbols{$sym}{'old_loc'}) { - printf(" MOVE : %-40s : %s => %s\n", $sym, $symbols{$sym}{'old_loc'}, - $symbols{$sym}{'loc'}); - $moved++; - } - - # Changes to export type are only bad if new type isn't - # EXPORT_SYMBOL. Changing things to GPL are bad. - if ($symbols{$sym}{'type'} ne $symbols{$sym}{'old_type'}) { - printf(" TYPE : %-40s : %s => %s%s\n", $sym, $symbols{$sym}{'old_type'}. - $symbols{$sym}{'type'}, is_ignored($symbols{$sym}{'loc'}, $sym) - ? " (ignored)" : ""); - $changed_type++ if $symbols{$sym}{'type'} ne "EXPORT_SYMBOL" - and !is_ignored($symbols{$sym}{'loc'}, $sym); - } - - # Changes to the hash are always bad - if ($symbols{$sym}{'hash'} ne $symbols{$sym}{'old_hash'}) { - printf(" HASH : %-40s : %s => %s%s\n", $sym, $symbols{$sym}{'old_hash'}, - $symbols{$sym}{'hash'}, is_ignored($symbols{$sym}{'loc'}, $sym) - ? " (ignored)" : ""); - $changed_hash++ if !is_ignored($symbols{$sym}{'loc'}, $sym); - $module_syms{$symbols{$sym}{'loc'}}++; - } -} - -print "WW: $moved symbols changed location\n" if $moved; -print "$EE $changed_type symbols changed export type and weren't ignored\n" if $changed_type; -print "$EE $changed_hash symbols changed hash and weren't ignored\n" if $changed_hash; - -$errors++ if $changed_hash or $changed_type; -if ($changed_hash) { - print "II: Module hash change summary...\n"; - foreach $mod (sort { $module_syms{$b} <=> $module_syms{$a} } keys %module_syms) { - next if ! $module_syms{$mod}; - printf(" %-40s: %d\n", $mod, $module_syms{$mod}); - } -} - -print "II: Done\n"; - -if ($errors) { - exit($fail_exit); -} else { - exit(0); -} +print(' Reading symbols to ignore...', end='') +ignore = 0 +prev_abi_ignore = '{}/../abi.ignore'.format(prev_abidir) +if os.path.exists(prev_abi_ignore): + with open(prev_abi_ignore) as fh: + for sym in fh: + sym = sym.strip() + if sym.startswith('#'): + continue + symbols_ignore[sym] = 1 + ignore += 1 +print('read {} symbols.'.format(ignore)) + +# Read new symbols first +print(' Reading new symbols...', end='') +new_count = 0 +with open('{}/{}'.format(abidir, flavor)) as fh: + for line in fh: + sym, vals = decode_symline(line.strip()) + symbols[sym] = vals + new_count += 1 +print('read {} symbols.'.format(new_count)) + +# Now the old symbols +print(' Reading old symbols...', end='') +old_count = 0 +with open('{}/{}'.format(prev_abidir, flavor)) as fh: + for line in fh: + sym, vals = decode_symline(line.strip()) + if sym not in symbols: + symbols[sym] = {} + symbols[sym]['old'] = vals + old_count += 1 +print('read {} symbols.'.format(old_count)) + +print('II: Checking for ABI changes...') +changed_loc = 0 +changed_type = 0 +error = False +for sym, vals in symbols.items(): + # Ignore new symbols + if 'old' not in vals: + continue + + # Changes in location don't hurt us, but log it anyway + if vals['loc'] != vals['old']['loc']: + changed_loc += 1 + print(' MOVE : {:40} : {} => {}'.format(sym, vals['old']['loc'], + vals['loc'])) + + # Changes from GPL to non-GPL are bad + if ((vals['old']['type'] == 'EXPORT_SYMBOL_GPL' and + vals['type'] != 'EXPORT_SYMBOL_GPL')): + changed_type += 1 + if sym in symbols_ignore or vals['loc'] in symbols_ignore: + ignored = ' (ignore)' + else: + ignored = '' + error = True + print(' TYPE : {:40} : {} => {}{}'.format(sym, vals['old']['type'], + vals['type'], ignored)) + +if changed_loc > 0: + print('II: {} symbols changed location'.format(changed_loc)) + +if changed_type > 0: + print('II: {} symbols changed export type'.format(changed_type)) + +if error: + if skipabi: + print('WW: Explicitly asked to ignore failures') + else: + print('EE: Symbol types changed') + sys.exit(1) + +print('II: Done') +sys.exit(0)