From patchwork Fri Apr 28 11:48:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Carlos O'Donell X-Patchwork-Id: 1774901 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=sourceware.org (client-ip=2620:52:3:1:0:246e:9693:128c; helo=sourceware.org; envelope-from=libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; secure) header.d=sourceware.org header.i=@sourceware.org header.a=rsa-sha256 header.s=default header.b=slWHhjkJ; dkim-atps=neutral Received: from sourceware.org (server2.sourceware.org [IPv6:2620:52:3:1:0:246e:9693:128c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Q79qC4vKbz23tl for ; Fri, 28 Apr 2023 21:48:39 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 9AAAF3858C60 for ; Fri, 28 Apr 2023 11:48:37 +0000 (GMT) DKIM-Filter: OpenDKIM Filter v2.11.0 sourceware.org 9AAAF3858C60 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sourceware.org; s=default; t=1682682517; bh=G5fZkrQ7TY8ivp6SugPpvhgUhGVAIQMq3cxB7u4BkJY=; h=To:Cc:Subject:Date:In-Reply-To:References:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: From:Reply-To:From; b=slWHhjkJF/3JjEzsbVAb8mbQiqMUuOoPHTJcZkFpSPpOuabMAhT2/n37kpptTeOaM qglug/J75/3BxUgnlqHyKUivWCipY+Y9KUlDlWUvLfVq6rAwfQy9sfajsc48GOqV7A /Sv87xZ+lrFr0WEq8LnEzGlCySKf8jZ9vfuSjfoE= X-Original-To: libc-alpha@sourceware.org Delivered-To: libc-alpha@sourceware.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by sourceware.org (Postfix) with ESMTPS id 0CD203858C31 for ; Fri, 28 Apr 2023 11:48:21 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 0CD203858C31 Received: from mail-yb1-f200.google.com (mail-yb1-f200.google.com [209.85.219.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-114-IQcCImqZM92tNDjzRos4xg-1; Fri, 28 Apr 2023 07:48:19 -0400 X-MC-Unique: IQcCImqZM92tNDjzRos4xg-1 Received: by mail-yb1-f200.google.com with SMTP id 3f1490d57ef6-b9a7df507c5so3373596276.1 for ; Fri, 28 Apr 2023 04:48:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682682498; x=1685274498; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=G5fZkrQ7TY8ivp6SugPpvhgUhGVAIQMq3cxB7u4BkJY=; b=ijsi3WGBna0PQYskcbSeNNfE52UvfhF/BJz/n8hReJTDc5KAqWuhgO0qStrgKj323V FIRESjXxQ/uOC4Zdud5BVNPpMpsEQI8iy363VyhtO11o+wYHlcd14Si+G2e1IfT3mpUh OYMweWXj0rsVg8cD8NiIi7+Iq5m1AqCv0NbJFY3KE3aocI2OksngzuTlzgwljRe1wiP+ VOX/nsJuW2W+8bNfyVYrZs4OqIUJM7OrzzWIfD1UkjEk85Wtw9vH+ypSkXJNmgZhmyv6 pb4PPiAFx3iC+q3Sm5HjsRpG7aH0pDeKUTK/LBNGGXGB8LWstTyhBTIAYar8X76MXZgJ Gz5w== X-Gm-Message-State: AC+VfDz8q9de8gAuMzXCwntgpbZT3zXvNtBjfn9sxcrhn9ndPqwt9Cr5 xM4C91otKVcTWFIgLYOtRpXf5Mgm32RndXT1Y1vja3lpUhoT/15fNrerwlDK2HbAGhnxytBAVxb LODlEPkzhsBMjZwNgpPp7bWVTzKbZRcpkufkoQ66tEKrGeAyGtFtQZsee9Wgv/L0NZPLjBBP+EB l1Pw== X-Received: by 2002:a05:6902:102f:b0:b92:41a6:5637 with SMTP id x15-20020a056902102f00b00b9241a65637mr4145252ybt.14.1682682498296; Fri, 28 Apr 2023 04:48:18 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7DaVCU7G+ZxXbxhYEICk2+svLP2lQ9nx7cceQEwk2Tj0WydDNJ4nhvXesyZA4npGlLn/KGog== X-Received: by 2002:a05:6902:102f:b0:b92:41a6:5637 with SMTP id x15-20020a056902102f00b00b9241a65637mr4145238ybt.14.1682682497827; Fri, 28 Apr 2023 04:48:17 -0700 (PDT) Received: from athas.redhat.com ([198.48.244.52]) by smtp.gmail.com with ESMTPSA id i3-20020a258b03000000b00b8c08669033sm5064956ybl.40.2023.04.28.04.48.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 28 Apr 2023 04:48:17 -0700 (PDT) To: libc-alpha@sourceware.org Cc: Carlos O'Donell Subject: [PATCH 2/2] scripts: Add sort-makefile-lines.py to sort Makefile variables. Date: Fri, 28 Apr 2023 07:48:11 -0400 Message-Id: <20230428114811.4129539-3-carlos@redhat.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230428114811.4129539-1-carlos@redhat.com> References: <20230428114811.4129539-1-carlos@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain; charset=UTF-8 X-Spam-Status: No, score=-12.6 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, SPF_HELO_NONE, SPF_NONE, TXREP, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: libc-alpha@sourceware.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Libc-alpha mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Carlos O'Donell via Libc-alpha From: Carlos O'Donell Reply-To: Carlos O'Donell Errors-To: libc-alpha-bounces+incoming=patchwork.ozlabs.org@sourceware.org Sender: "Libc-alpha" The scripts/sort-makefile-lines.py script sorts Makefile variables according to project expected order. The script is used like this: $ scripts/sort-makefile-lines.py -i elf/Makefile -o elf/Makefile.tmp $ mv elf/Makefile.tmp elf/Makefile --- scripts/sort-makefile-lines.py | 217 +++++++++++++++++++++++++++++++++ 1 file changed, 217 insertions(+) create mode 100755 scripts/sort-makefile-lines.py diff --git a/scripts/sort-makefile-lines.py b/scripts/sort-makefile-lines.py new file mode 100755 index 0000000000..06c0b3b3a2 --- /dev/null +++ b/scripts/sort-makefile-lines.py @@ -0,0 +1,217 @@ +#!/usr/bin/python3 +# Sort Makefile lines as expected by project policy. +# Copyright (C) 2022-2023 Free Software Foundation, Inc. +# Copyright The GNU Toolchain Authors. +# This file is part of the GNU C Library. +# +# The GNU C Library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# The GNU C Library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with the GNU C Library; if not, see +# . + +# The project consensus is to split Makefile variable assignment +# across multiple lines with one value per line. The values are +# then sorted as described below, and terminated with a special +# list termination marker. This splitting makes it much easier +# to add new tests to the list since they become just a single +# line insertion. It also makes backports and merges easier +# since the new test may not conflict due to the ordering. +# +# Consensus discussion: +# https://public-inbox.org/libc-alpha/f6406204-84f5-adb1-d00e-979ebeebbbde@redhat.com/ +# +# To support cleaning up Makefiles we created this program to +# help sort existing lists converted to the new format. +# +# The program takes as input the Makefile to sort correctly, +# and the output file to write the correctly sorted output. +# +# Sorting is only carried out between two special markers: +# (a) Marker start is ' += \' +# (b) Marker end is ' # ' +# With everthing between (a) and (b) being sorted. +# +# You can use it like this: +# $ scripts/sort-makefile-lines.py -i elf/Makefile -o elf/Makefile.tmp +# $ mv elf/Makefile.tmp elf/Makefile +# +# The Makefile lines in the project are sorted using the +# following rules: +# - First all lines are sorted as-if `LC_COLLATE=C sort` +# - Then all entries by group are sorted against the last digits +# of the test. +# +# For example: +# ~~~ +# tests += \ +# test-a \ +# test-b \ +# test-b1 \ +# test-b2 \ +# test-b10 \ +# test-b20 \ +# test-b100 \ +# # tests +# ~~~ +# This example shows tests sorted alphabetically, followed +# by a numeric suffix sort in increasing numeric order. +# +# Required cleanups: +# - Tests that end in "a" or "b" variants must be renamed to +# end in just the numerical value. For example 'tst-mutex7robust' +# should be renamed to 'tst-mutex12' (the highest numbered test) +# or 'tst-robust11' (the highest numbered test). +# - Modules that end in "mod" or "mod1" should be renamed. For +# example 'tst-atfork2mod' should be renamed to 'tst-mod-atfork2' +# (test module for atfork2). If there are more than one module +# then they should be named with a suffix that uses [0-9] first +# then [A-Z] next for a total of 36 possible modules per test. +# No manually listed test currently uses more than that (though +# automatically generated tests may; they don't need sorting). +# - Avoid including another test and instead refactor into common +# code with all tests including hte common code, then give the +# tests unique names. +# +# If you have a Makefile that needs converting, then you can +# quickly split the values into one-per-line, ensure the start +# and end markers are in place, and then run the script to +# sort the values. + +import argparse +import sys +import locale +import re +import functools + +def numeric_key(line): + # Turn a line into a numeric sort value by fetching + # the ending number and using that as a key. + var = re.search(r'([0-9]+) \\$', line) + if var == None: + print ("Error: Test line is currently: \"", end='') + print (line, end='') + print ("\"") + print ( + ''' +Test name does not match expected pattern. +Rename to match pattern e.g. tst-name[0-9]+. + ''' + ) + raise Exception ("Invalid test name.") + # Return the numeric value as the key or throws because + # var is None. + return int(var.group(1)) + +def sort_lines(lines): + + # Use the C locale for language independent collation. + locale.setlocale (locale.LC_ALL, "C") + + # Sort with strcoll initially. The tests ending in numeric + # names will not sort correctly, but we will adjust that next. + lines = sorted(lines, key=functools.cmp_to_key(locale.strcoll)) + + # We could use a trie here, but since the problem is restricted + # to just numeric suffix we sort by group with a unique key + # function. + + # Build a list of all start markers (tuple includes prefix) + prefixes = [] + groups = [] + for i in range(len(lines)): + # Look for things like " tst-foo1 \" to start the numbered list. + var = re.search(r'([0-9]+) \\$', lines[i]) + if var: + prefix = lines[i][0:var.span()[0]] + if prefix in prefixes: + continue + prefixes.append(prefix) + groups.append((prefix, i)) + + # For each prefix find the range it covers that needs numeric sorting. + numgroups = [] + for group in groups: + for j in range(group[1] + 1,len(lines)): + if not lines[j].startswith(group[0]): + # If it doesn't start with the prefix, then we're on to + # to the next group so mark the last entry as the end + # of the group. + numgroups.append((group[0], group[1], j - 1)) + break + + # We now have a list of groups to sort. + for ng in numgroups: + # Note slices exclude nth element, so we must add one to right side. + lines[ng[1]:ng[2]+1] = sorted(lines[ng[1]:ng[2]+1], key=numeric_key) + + # Return a sorted list with numeric tests sorted by number. + return lines + +def sort_makefile_lines(infile, outfile): + + # Read the whole Makefile. + mfile = open(infile) + lines = mfile.readlines() + mfile.close() + + # We will output the Makefile here. Open it early to check + # for any errors. + ofile = open(outfile, "w") + + # Build a list of all start markers (tuple includes name). + startmarks = [] + for i in range(len(lines)): + # Look for things like "var += \" to start the sorted list. + var = re.search(r'^([a-zA-Z0-9]*) \+\= \\$', lines[i]) + if var: + # Remember the index and the name. + startmarks.append((i, var.group(1))) + + # For each start marker try to find a matching end mark + # and build a block that needs sorting. The end marker + # must have the matching comment name for it to be valid. + rangemarks = [] + for sm in startmarks: + # Look for things like " # var" to end the sorted list. + reg = r'^ # ' + sm[1] + r'$' + for j in range(sm[0] + 1, len(lines)): + if re.search(reg, lines[j]): + # Rembember the block to sort (inclusive). + rangemarks.append((sm[0] + 1, j - 1)) + break + + # We now have a list of all ranges that need sorting. + # Sort those ranges. + for r in rangemarks: + lines[r[0]:r[1]] = sort_lines(lines[r[0]:r[1]]) + + # Output the whole list with sorted lines. + for line in lines: + ofile.write(line) + + ofile.close() + +def get_parser(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('-i', dest='infile', + help='Input Makefile to read lines from') + parser.add_argument('-o', dest='outfile', + help='Output Makefile to write sorted lines to') + return parser + +def main(argv): + parser = get_parser() + opts = parser.parse_args(argv) + sort_makefile_lines (opts.infile, opts.outfile) + +if __name__ == '__main__': + main(sys.argv[1:])