[1/1] support/testing: new ltp-testsuite runtime test

Signed-off-by: Julien Olivain <ju.o@free.fr>
---
Patch tested in:
https://gitlab.com/jolivain/buildroot/-/jobs/8388435290
---
 DEVELOPERS                                    |  1 +
 .../tests/package/test_ltp_testsuite.py       | 39 +++++++++++++++++++
 2 files changed, 40 insertions(+)
 create mode 100644 support/testing/tests/package/test_ltp_testsuite.py

On Fri, Nov 15, 2024 at 11:47:53PM UTC, Julien Olivain wrote:
> Signed-off-by: Julien Olivain <ju.o@free.fr>
> ---
> Patch tested in:
> https://gitlab.com/jolivain/buildroot/-/jobs/8388435290

It is a very good idea to run LTP. How can it be extended to be run with
more CPU architecture ? To run some wider set of tests ?

> ---
>  DEVELOPERS                                    |  1 +
>  .../tests/package/test_ltp_testsuite.py       | 39 +++++++++++++++++++
>  2 files changed, 40 insertions(+)
>  create mode 100644 support/testing/tests/package/test_ltp_testsuite.py
> 
> diff --git a/DEVELOPERS b/DEVELOPERS
> index 3945803d53..1ff269b362 100644
> --- a/DEVELOPERS
> +++ b/DEVELOPERS
> @@ -1931,6 +1931,7 @@ F:	support/testing/tests/package/test_links.py
>  F:	support/testing/tests/package/test_links/
>  F:	support/testing/tests/package/test_lrzip.py
>  F:	support/testing/tests/package/test_lrzsz.py
> +F:	support/testing/tests/package/test_ltp_testsuite.py
>  F:	support/testing/tests/package/test_ltrace.py
>  F:	support/testing/tests/package/test_lvm2.py
>  F:	support/testing/tests/package/test_lzip.py
> diff --git a/support/testing/tests/package/test_ltp_testsuite.py b/support/testing/tests/package/test_ltp_testsuite.py
> new file mode 100644
> index 0000000000..a85ceb35a6
> --- /dev/null
> +++ b/support/testing/tests/package/test_ltp_testsuite.py
> @@ -0,0 +1,39 @@
> +import os
> +
> +import infra.basetest
> +
> +
> +class TestLtpTestsuite(infra.basetest.BRTest):
> +    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
> +        """
> +        BR2_PACKAGE_LTP_TESTSUITE=y
> +        BR2_TARGET_ROOTFS_EXT2=y
> +        BR2_TARGET_ROOTFS_EXT2_4=y
> +        BR2_TARGET_ROOTFS_EXT2_SIZE="600M"
> +        # BR2_TARGET_ROOTFS_TAR is not set
> +        """
> +
> +    def test_run(self):
> +        drive = os.path.join(self.builddir, "images", "rootfs.ext4")
> +        self.emulator.boot(arch="armv5",
> +                           kernel="builtin",
> +                           kernel_cmdline=["rootwait", "root=/dev/sda"],
> +                           options=["-drive", f"file={drive},if=scsi,format=raw"])
> +        self.emulator.login()
> +
> +        # We run a reduced number of tests (read syscall tests) for a
> +        # fast execution. See "runltp --help" for option details.
> +        cmd = "/usr/lib/ltp-testsuite/runltp"
> +        cmd += " -p -q"
> +        cmd += " -s ^read0[0-9]*"
> +        cmd += " -l /tmp/ltp.log"
> +        cmd += " -o /tmp/ltp.output"
> +        cmd += " -C /tmp/ltp.failed"
> +        cmd += " -T /tmp/ltp.tconf"
> +        self.assertRunOk(cmd)
> +
> +        # We print the LTP run log and check there was zero failure in
> +        # our test selection.
> +        out, ret = self.emulator.run("cat /tmp/ltp.log")
> +        self.assertEqual(ret, 0)
> +        self.assertIn("Total Failures: 0", out)
> -- 
> 2.47.0
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

Hi Vincent,

On 16/11/2024 00:25, Vincent Jardin wrote:
> On Fri, Nov 15, 2024 at 11:47:53PM UTC, Julien Olivain wrote:
>> Signed-off-by: Julien Olivain <ju.o@free.fr>
>> ---
>> Patch tested in:
>> https://gitlab.com/jolivain/buildroot/-/jobs/8388435290
> 
> It is a very good idea to run LTP. How can it be extended to be run 
> with
> more CPU architecture ? To run some wider set of tests ?

The primary intent of this test is to check the Buildroot package recipe
is able to compile the ltp-testsuite correctly and install it on target
properly to be functional. It will be also useful when updating the
package.

This is for this reason I propose only one architecture (the default
armv5), with a very limited test set, to make sure compilation and
execution will be fast.

Conversely, this test is NOT aiming to fully test a given Kernel
version on Qemu. Buildroot is already testing that qemu_*_defconfig
are booting and reaching the user-space login.

What did you have in mind exactly? Testing more extensively a
specific defconfig? This could be done outside the Buildroot Gitlab CI,
for example, by a defconfig maintainer?

Best regards,

Julien.

On Sat, Nov 16, 2024 at 11:28:56AM UTC, Julien Olivain wrote:
> The primary intent of this test is to check the Buildroot package recipe
> is able to compile the ltp-testsuite correctly and install it on target
> properly to be functional. It will be also useful when updating the
> package.

OK, it did not inderstand that it was the primary intend. I was guessing
that the intends were to have a broader LTP scope of tests.

> What did you have in mind exactly? Testing more extensively a
> specific defconfig? This could be done outside the Buildroot Gitlab CI,
> for example, by a defconfig maintainer?

I understand your point. We can limit the time allocated to running the Buildroot
test suite, while LTP+Buildroot could be managed as a separate GitLab CI/project.

So the current proposal for LTP check is OK.

Reviewed-by: Vincent Jardin <vjardin@free.fr>

Best regards,
  Vincent