[RFC,2/2] make: Add test target

Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
 Makefile           |  3 +++
 lib/tests/Makefile |  3 +++
 lib/tests/test.sh  | 32 ++++++++++++++++++++++++++++++++
 3 files changed, 38 insertions(+)
 create mode 100755 lib/tests/test.sh

----- Original Message -----
> Signed-off-by: Petr Vorel <pvorel@suse.cz>
> ---
>  Makefile           |  3 +++
>  lib/tests/Makefile |  3 +++
>  lib/tests/test.sh  | 32 ++++++++++++++++++++++++++++++++
>  3 files changed, 38 insertions(+)
>  create mode 100755 lib/tests/test.sh
> +
> +for i in $@; do
> +    echo "=== Testing '$i' ==="
> +    case $i in
> +        tst_checkpoint_wake_timeout|tst_record_childstatus)
> +        if [ "$i" = "tst_record_childstatus" ]; then
> +            echo "NOTE: expecting fail the test"
> +            ./$i || [ $? -eq 1 ]
> +        fi

I'd prefer we fix the tests and make them return 0 if everything goes as expected.

Regards,
Jan

Hi!
> diff --git a/Makefile b/Makefile
> index b0368a472..f886ac350 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -89,6 +89,9 @@ MAKE_TARGETS		:= $(addsuffix -all,$(filter-out lib,$(COMMON_TARGETS)))
>  # overtaxed one, or one where -j => 1 was specified.
>  all: $(addsuffix -all,$(COMMON_TARGETS)) Version
>  
> +test:
> +	cd lib/tests && make $@
> +
>  $(MAKE_TARGETS): lib-all
>  
>  .PHONY: include-all include-install
> diff --git a/lib/tests/Makefile b/lib/tests/Makefile
> index 73a0f1655..732fee2c4 100644
> --- a/lib/tests/Makefile
> +++ b/lib/tests/Makefile
> @@ -7,4 +7,7 @@ LDLIBS			+= -lltp
>  
>  tst_cleanup_once: CFLAGS += -pthread
>  
> +test:
> +	./test.sh $(MAKE_TARGETS)
> +
>  include $(top_srcdir)/include/mk/generic_leaf_target.mk
> diff --git a/lib/tests/test.sh b/lib/tests/test.sh
> new file mode 100755
> index 000000000..81d57a81b
> --- /dev/null
> +++ b/lib/tests/test.sh
> @@ -0,0 +1,32 @@
> +#!/bin/sh
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +# Copyright (c) 2016-2018 Petr Vorel <pvorel@suse.cz>
> +
> +set -e
> +
> +for i in $@; do
> +    echo "=== Testing '$i' ==="
> +    case $i in
> +        tst_checkpoint_wake_timeout|tst_record_childstatus)
> +        if [ "$i" = "tst_record_childstatus" ]; then
> +            echo "NOTE: expecting fail the test"
> +            ./$i || [ $? -eq 1 ]
> +        fi
> +        ;;
> +        tst_device)
> +            if [ $(id -u) -ne 0 ]; then
> +                echo "WARN: not root, skip the test"
> +            else
> +                ./$i
> +            fi
> +        ;;

Listing the testcases does not scale that much. I wonder how we can do
better.

Also new library tests are stored in newlib_tests directory.

> +        *)
> +        ./$i
> +        ;;
> +    esac
> +    echo
> +done
> +
> +echo "END OF TESTING"
> +
> +# vim: set ft=sh ts=4 sts=4 sw=4 expandtab :

I wonder if we can try to check output of these tests against expected
output. Most of the testcases have nice and stable output that is always
the same. Some of the include pid in the test output that will change
with each iteration and some, for instance the thread safety related
tests for the new library, produce the output lines in random order.

So maybe the solution would be grepping for pre-defined patterns in the
test output in the case that the test output is not stable.

Hi Jan,

> ----- Original Message -----
> > Signed-off-by: Petr Vorel <pvorel@suse.cz>
> > ---
> >  Makefile           |  3 +++
> >  lib/tests/Makefile |  3 +++
> >  lib/tests/test.sh  | 32 ++++++++++++++++++++++++++++++++
> >  3 files changed, 38 insertions(+)
> >  create mode 100755 lib/tests/test.sh
> > +
> > +for i in $@; do
> > +    echo "=== Testing '$i' ==="
> > +    case $i in
> > +        tst_checkpoint_wake_timeout|tst_record_childstatus)
> > +        if [ "$i" = "tst_record_childstatus" ]; then
> > +            echo "NOTE: expecting fail the test"
> > +            ./$i || [ $? -eq 1 ]
> > +        fi

> I'd prefer we fix the tests and make them return 0 if everything goes as expected.
Agree. But I took different folder. As Cyril pointed out, tests in lib/tests/
are for legacy API. I'm going to test new ones from lib/newlib_tests/.

As Christian works on #312 "Unit testing the shell library" [1], I'd like if
both shell and C tests had the same approach of testing.

I wonder whether to keep these tests in lib/tests/. At least some of them aren't
meant to be for unit testing (due expecting failing), nobody runs them, ...

[1] https://github.com/linux-test-project/ltp/issues/312

> Regards,
> Jan