[1/1] Add lint-makefiles Makefile linting test.

We add a `make check` test that lints all Makefiles in the source
directory of the glibc build. This linting test ensures that the
lines in all Makefiles will be sorted correctly as developers
creates patches.

The test adds ~3s to a test run.

No regressions on x86_64 and i686.
---
 Makefile                  |  6 +++++
 scripts/lint-makefiles.sh | 47 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 53 insertions(+)
 create mode 100644 scripts/lint-makefiles.sh

Hi Carlos,

On 5/19/23 14:13, Carlos O'Donell via Libc-alpha wrote:
> We add a `make check` test that lints all Makefiles in the source
> directory of the glibc build. This linting test ensures that the
> lines in all Makefiles will be sorted correctly as developers
> creates patches.

make check is usually to test the result of a build, according the
GNU guidelines in:

<https://www.gnu.org/software/make/manual/html_node/Standard-Targets.html>

I believe linting the source code is a different thing, and so I
used a different target in the Linux man-pages Makefile: 'lint'.

So, in the man-pages project, it goes like this:

lint -> build -> check -> install

Maybe you could follow a similar scheme?  Otherwise, it's a bit
weird to run the linters _after_ building (since `make check`
requires previously building).

If you want to have a look at the Linux man-pages' makefiles,
you should start by running `make help` and `make help-variables`,
and also read the README, CONTRIBUTING (see 'Lint & check' section),
and INSTALL files.

Cheers,
Alex

> 
> The test adds ~3s to a test run.
> 
> No regressions on x86_64 and i686.
> ---
>   Makefile                  |  6 +++++
>   scripts/lint-makefiles.sh | 47 +++++++++++++++++++++++++++++++++++++++
>   2 files changed, 53 insertions(+)
>   create mode 100644 scripts/lint-makefiles.sh
> 
> diff --git a/Makefile b/Makefile
> index 224c792185..93e5a60af9 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -564,6 +564,12 @@ $(objpfx)check-wrapper-headers.out: scripts/check-wrapper-headers.py $(headers)
>   	  --generated $(common-generated) > $@; $(evaluate-test)
>   endif # $(headers)
>   
> +# Lint all Makefiles including this one to check for correctly sorted lines.
> +tests-special += $(objpfx)lint-makefiles.out
> +$(objpfx)lint-makefiles.out: scripts/lint-makefiles.sh
> +	$(SHELL) $< "$(PYTHON)" "$(srcdir)" > $@ ; \
> +	$(evaluate-test)
> +
>   define summarize-tests
>   @grep -E -v '^(PASS|XFAIL):' $(objpfx)$1 || true
>   @echo "Summary of test results$2:"
> diff --git a/scripts/lint-makefiles.sh b/scripts/lint-makefiles.sh
> new file mode 100644
> index 0000000000..cf63a4ab9f
> --- /dev/null
> +++ b/scripts/lint-makefiles.sh
> @@ -0,0 +1,47 @@
> +#!/bin/bash
> +# Copyright (C) 2023 Free Software Foundation, Inc.
> +# This file is part of the GNU C Library.
> +
> +# The GNU C Library is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU Lesser General Public
> +# License as published by the Free Software Foundation; either
> +# version 2.1 of the License, or (at your option) any later version.
> +
> +# The GNU C Library is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> +# Lesser General Public License for more details.
> +
> +# You should have received a copy of the GNU Lesser General Public
> +# License along with the GNU C Library; if not, see
> +# <https://www.gnu.org/licenses/>.
> +
> +# This script checks to see that all Makefiles in the source tree
> +# conform to the sorted variable rules as defined by:
> +# scripts/sort-makefile-lines.py.
> +# Any difference is an error and should be corrected e.g. the lines
> +# reordered to sort correctly.
> +# The intent with this check is to ensure that changes made by
> +# developers match the expected format for the project.
> +
> +set -e
> +export LC_ALL=C
> +
> +tmpfile="$(mktemp)"
> +
> +cleanup () {
> +  rm -f -- "$tmpfile"
> +}
> +
> +trap cleanup 0
> +
> +PYTHON=$1
> +srcdir=$2
> +
> +echo "Linting Makefiles:"
> +echo "Check: Are all lines sorted correctly?"
> +for mfile in `find "$srcdir" -name Makefile`; do
> +    echo "$mfile"
> +    $PYTHON "${srcdir}scripts/sort-makefile-lines.py" < "$mfile" > "$tmpfile"
> +    diff -q "$mfile" "$tmpfile"
> +done

* Alejandro Colomar:

> Hi Carlos,
>
> On 5/19/23 14:13, Carlos O'Donell via Libc-alpha wrote:
>> We add a `make check` test that lints all Makefiles in the source
>> directory of the glibc build. This linting test ensures that the
>> lines in all Makefiles will be sorted correctly as developers
>> creates patches.
>
> make check is usually to test the result of a build, according the
> GNU guidelines in:
>
> <https://www.gnu.org/software/make/manual/html_node/Standard-Targets.html>
>
> I believe linting the source code is a different thing, and so I
> used a different target in the Linux man-pages Makefile: 'lint'.
>
> So, in the man-pages project, it goes like this:
>
> lint -> build -> check -> install
>
> Maybe you could follow a similar scheme?  Otherwise, it's a bit
> weird to run the linters _after_ building (since `make check`
> requires previously building).

I think the consistency with the existing tests is more important.  The
way Carlos implemented it, it becomes part of developer workflows
automatically.

Thanks,
Florian

On 5/19/23 08:45, Florian Weimer wrote:
> * Alejandro Colomar:
> 
>> Hi Carlos,
>>
>> On 5/19/23 14:13, Carlos O'Donell via Libc-alpha wrote:
>>> We add a `make check` test that lints all Makefiles in the source
>>> directory of the glibc build. This linting test ensures that the
>>> lines in all Makefiles will be sorted correctly as developers
>>> creates patches.
>>
>> make check is usually to test the result of a build, according the
>> GNU guidelines in:



>> <https://www.gnu.org/software/make/manual/html_node/Standard-Targets.html>
>>
>> I believe linting the source code is a different thing, and so I
>> used a different target in the Linux man-pages Makefile: 'lint'.
>>
>> So, in the man-pages project, it goes like this:
>>
>> lint -> build -> check -> install
>>
>> Maybe you could follow a similar scheme?  Otherwise, it's a bit
>> weird to run the linters _after_ building (since `make check`
>> requires previously building).
> 
> I think the consistency with the existing tests is more important.  The
> way Carlos implemented it, it becomes part of developer workflows
> automatically.

Correct.

The goal here is to improve the developer workflow.

The current workflow is:
* Edit sources.
* make
* make check

And adding to `make check` means we are part of that workflow and improve the
new developer experience.

Let me digress a little bit here. glibc has an 'xcheck' target which is
conceptually flawed. I say it is flawed because having two ways to test
glibc is not simple and requires using a non-standard target to add these
extra tests. We should work to merge every 'xcheck' test into 'check'
and make a single 'check' that must be run every time you are ready to
check glibc, either after developing, or before pushing to a CI/CD system.

If a build system cannot run 'check' (hardened) then the binary artifacts
should be re-testable outside of the build system in a CI/CD system that
can run those tests. We don't easily support this today, but it is IMO the
right direction to go in e.g. test existing binary artifacts + sources.

In summary:
- I do not want to complicate the developer workflow by adding another target.
- I want to reduce the number of targets and eventually get rid of xcheck.
- Expensive checks can be delegated to scripts like scripts/build-many-glibcs.py
  which can be run in CI/CD systems.

On 5/19/23 08:45, Florian Weimer wrote:
> * Alejandro Colomar:
> 
>> Hi Carlos,
>>
>> On 5/19/23 14:13, Carlos O'Donell via Libc-alpha wrote:
>>> We add a `make check` test that lints all Makefiles in the source
>>> directory of the glibc build. This linting test ensures that the
>>> lines in all Makefiles will be sorted correctly as developers
>>> creates patches.
>>
>> make check is usually to test the result of a build, according the
>> GNU guidelines in:
>>
>> <https://www.gnu.org/software/make/manual/html_node/Standard-Targets.html>
>>
>> I believe linting the source code is a different thing, and so I
>> used a different target in the Linux man-pages Makefile: 'lint'.
>>
>> So, in the man-pages project, it goes like this:
>>
>> lint -> build -> check -> install
>>
>> Maybe you could follow a similar scheme?  Otherwise, it's a bit
>> weird to run the linters _after_ building (since `make check`
>> requires previously building).
> 
> I think the consistency with the existing tests is more important.  The
> way Carlos implemented it, it becomes part of developer workflows
> automatically.

In the intervening time we've already regressed sorting in elf/Makefile, which to me
indicates that this needs to be a light-weight linting check during 'make check.'

I've reposted the light-weight linting with addtional comments:
https://sourceware.org/pipermail/libc-alpha/2023-May/148634.html
https://sourceware.org/pipermail/libc-alpha/2023-May/148635.html