[1/1] support/testing: new ltp-testsuite runtime test

Signed-off-by: Julien Olivain <ju.o@free.fr>
---
Patch tested in:
https://gitlab.com/jolivain/buildroot/-/jobs/8388435290
---
 DEVELOPERS                                    |  1 +
 .../tests/package/test_ltp_testsuite.py       | 39 +++++++++++++++++++
 2 files changed, 40 insertions(+)
 create mode 100644 support/testing/tests/package/test_ltp_testsuite.py

On Fri, Nov 15, 2024 at 11:47:53PM UTC, Julien Olivain wrote:
> Signed-off-by: Julien Olivain <ju.o@free.fr>
> ---
> Patch tested in:
> https://gitlab.com/jolivain/buildroot/-/jobs/8388435290

It is a very good idea to run LTP. How can it be extended to be run with
more CPU architecture ? To run some wider set of tests ?

> ---
>  DEVELOPERS                                    |  1 +
>  .../tests/package/test_ltp_testsuite.py       | 39 +++++++++++++++++++
>  2 files changed, 40 insertions(+)
>  create mode 100644 support/testing/tests/package/test_ltp_testsuite.py
> 
> diff --git a/DEVELOPERS b/DEVELOPERS
> index 3945803d53..1ff269b362 100644
> --- a/DEVELOPERS
> +++ b/DEVELOPERS
> @@ -1931,6 +1931,7 @@ F:	support/testing/tests/package/test_links.py
>  F:	support/testing/tests/package/test_links/
>  F:	support/testing/tests/package/test_lrzip.py
>  F:	support/testing/tests/package/test_lrzsz.py
> +F:	support/testing/tests/package/test_ltp_testsuite.py
>  F:	support/testing/tests/package/test_ltrace.py
>  F:	support/testing/tests/package/test_lvm2.py
>  F:	support/testing/tests/package/test_lzip.py
> diff --git a/support/testing/tests/package/test_ltp_testsuite.py b/support/testing/tests/package/test_ltp_testsuite.py
> new file mode 100644
> index 0000000000..a85ceb35a6
> --- /dev/null
> +++ b/support/testing/tests/package/test_ltp_testsuite.py
> @@ -0,0 +1,39 @@
> +import os
> +
> +import infra.basetest
> +
> +
> +class TestLtpTestsuite(infra.basetest.BRTest):
> +    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
> +        """
> +        BR2_PACKAGE_LTP_TESTSUITE=y
> +        BR2_TARGET_ROOTFS_EXT2=y
> +        BR2_TARGET_ROOTFS_EXT2_4=y
> +        BR2_TARGET_ROOTFS_EXT2_SIZE="600M"
> +        # BR2_TARGET_ROOTFS_TAR is not set
> +        """
> +
> +    def test_run(self):
> +        drive = os.path.join(self.builddir, "images", "rootfs.ext4")
> +        self.emulator.boot(arch="armv5",
> +                           kernel="builtin",
> +                           kernel_cmdline=["rootwait", "root=/dev/sda"],
> +                           options=["-drive", f"file={drive},if=scsi,format=raw"])
> +        self.emulator.login()
> +
> +        # We run a reduced number of tests (read syscall tests) for a
> +        # fast execution. See "runltp --help" for option details.
> +        cmd = "/usr/lib/ltp-testsuite/runltp"
> +        cmd += " -p -q"
> +        cmd += " -s ^read0[0-9]*"
> +        cmd += " -l /tmp/ltp.log"
> +        cmd += " -o /tmp/ltp.output"
> +        cmd += " -C /tmp/ltp.failed"
> +        cmd += " -T /tmp/ltp.tconf"
> +        self.assertRunOk(cmd)
> +
> +        # We print the LTP run log and check there was zero failure in
> +        # our test selection.
> +        out, ret = self.emulator.run("cat /tmp/ltp.log")
> +        self.assertEqual(ret, 0)
> +        self.assertIn("Total Failures: 0", out)
> -- 
> 2.47.0
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

Hi Vincent,

On 16/11/2024 00:25, Vincent Jardin wrote:
> On Fri, Nov 15, 2024 at 11:47:53PM UTC, Julien Olivain wrote:
>> Signed-off-by: Julien Olivain <ju.o@free.fr>
>> ---
>> Patch tested in:
>> https://gitlab.com/jolivain/buildroot/-/jobs/8388435290
> 
> It is a very good idea to run LTP. How can it be extended to be run 
> with
> more CPU architecture ? To run some wider set of tests ?

The primary intent of this test is to check the Buildroot package recipe
is able to compile the ltp-testsuite correctly and install it on target
properly to be functional. It will be also useful when updating the
package.

This is for this reason I propose only one architecture (the default
armv5), with a very limited test set, to make sure compilation and
execution will be fast.

Conversely, this test is NOT aiming to fully test a given Kernel
version on Qemu. Buildroot is already testing that qemu_*_defconfig
are booting and reaching the user-space login.

What did you have in mind exactly? Testing more extensively a
specific defconfig? This could be done outside the Buildroot Gitlab CI,
for example, by a defconfig maintainer?

Best regards,

Julien.

On Sat, Nov 16, 2024 at 11:28:56AM UTC, Julien Olivain wrote:
> The primary intent of this test is to check the Buildroot package recipe
> is able to compile the ltp-testsuite correctly and install it on target
> properly to be functional. It will be also useful when updating the
> package.

OK, it did not inderstand that it was the primary intend. I was guessing
that the intends were to have a broader LTP scope of tests.

> What did you have in mind exactly? Testing more extensively a
> specific defconfig? This could be done outside the Buildroot Gitlab CI,
> for example, by a defconfig maintainer?

I understand your point. We can limit the time allocated to running the Buildroot
test suite, while LTP+Buildroot could be managed as a separate GitLab CI/project.

So the current proposal for LTP check is OK.

Reviewed-by: Vincent Jardin <vjardin@free.fr>

Best regards,
  Vincent

Hi Julien,

> Signed-off-by: Julien Olivain <ju.o@free.fr>
> ---
> Patch tested in:
> https://gitlab.com/jolivain/buildroot/-/jobs/8388435290
> ---
>  DEVELOPERS                                    |  1 +
>  .../tests/package/test_ltp_testsuite.py       | 39 +++++++++++++++++++
>  2 files changed, 40 insertions(+)
>  create mode 100644 support/testing/tests/package/test_ltp_testsuite.py

> diff --git a/DEVELOPERS b/DEVELOPERS
> index 3945803d53..1ff269b362 100644
> --- a/DEVELOPERS
> +++ b/DEVELOPERS
> @@ -1931,6 +1931,7 @@ F:	support/testing/tests/package/test_links.py
>  F:	support/testing/tests/package/test_links/
>  F:	support/testing/tests/package/test_lrzip.py
>  F:	support/testing/tests/package/test_lrzsz.py
> +F:	support/testing/tests/package/test_ltp_testsuite.py
>  F:	support/testing/tests/package/test_ltrace.py
>  F:	support/testing/tests/package/test_lvm2.py
>  F:	support/testing/tests/package/test_lzip.py
> diff --git a/support/testing/tests/package/test_ltp_testsuite.py b/support/testing/tests/package/test_ltp_testsuite.py
> new file mode 100644
> index 0000000000..a85ceb35a6
> --- /dev/null
> +++ b/support/testing/tests/package/test_ltp_testsuite.py
> @@ -0,0 +1,39 @@
> +import os
> +
> +import infra.basetest
> +
> +
> +class TestLtpTestsuite(infra.basetest.BRTest):
> +    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
> +        """
> +        BR2_PACKAGE_LTP_TESTSUITE=y

Thanks a lot for taking care about LTP. Feel free to Cc me on LTP related
patches and issues.

> +        BR2_TARGET_ROOTFS_EXT2=y
> +        BR2_TARGET_ROOTFS_EXT2_4=y
> +        BR2_TARGET_ROOTFS_EXT2_SIZE="600M"
> +        # BR2_TARGET_ROOTFS_TAR is not set
> +        """
> +
> +    def test_run(self):
> +        drive = os.path.join(self.builddir, "images", "rootfs.ext4")
> +        self.emulator.boot(arch="armv5",
> +                           kernel="builtin",
> +                           kernel_cmdline=["rootwait", "root=/dev/sda"],
> +                           options=["-drive", f"file={drive},if=scsi,format=raw"])
> +        self.emulator.login()
> +
> +        # We run a reduced number of tests (read syscall tests) for a
> +        # fast execution. See "runltp --help" for option details.
> +        cmd = "/usr/lib/ltp-testsuite/runltp"
> +        cmd += " -p -q"
> +        cmd += " -s ^read0[0-9]*"
> +        cmd += " -l /tmp/ltp.log"
> +        cmd += " -o /tmp/ltp.output"
> +        cmd += " -C /tmp/ltp.failed"
> +        cmd += " -T /tmp/ltp.tconf"

While this above is OK, some notes:

1) Many LTP tests require LTPROOT environment variable (to access some data files),
e.g.:

export LTPROOT="/usr/lib/ltp-testsuite"

See https://linux-test-project.readthedocs.io/en/latest/users/setup_tests.html

It looks to me that infra.emulator.Emulator does not support inserting
environment variables via os.environ.

2) runltp old script has been deprecated, kirk is the replacement:

https://github.com/linux-test-project/kirk

Currently kirk supports few testing frameworks: LTP, kselftests and liburing.

While kirk can be run o SUT (like runltp), the best usage is to run it on the
host. It supports testing via starting QEMU VM, SSH and not yet stable LTX
backend.

Due the above I'd like to add kirk host package to Buildroot.

Kind regards,
Petr


> +        self.assertRunOk(cmd)
> +
> +        # We print the LTP run log and check there was zero failure in
> +        # our test selection.
> +        out, ret = self.emulator.run("cat /tmp/ltp.log")
> +        self.assertEqual(ret, 0)
> +        self.assertIn("Total Failures: 0", out)

On 15/11/2024 23:47, Julien Olivain wrote:
> Signed-off-by: Julien Olivain <ju.o@free.fr>

  Applied to master, thanks.

  Regards,
  Arnout

> ---
> Patch tested in:
> https://gitlab.com/jolivain/buildroot/-/jobs/8388435290
> ---
>   DEVELOPERS                                    |  1 +
>   .../tests/package/test_ltp_testsuite.py       | 39 +++++++++++++++++++
>   2 files changed, 40 insertions(+)
>   create mode 100644 support/testing/tests/package/test_ltp_testsuite.py
> 
> diff --git a/DEVELOPERS b/DEVELOPERS
> index 3945803d53..1ff269b362 100644
> --- a/DEVELOPERS
> +++ b/DEVELOPERS
> @@ -1931,6 +1931,7 @@ F:	support/testing/tests/package/test_links.py
>   F:	support/testing/tests/package/test_links/
>   F:	support/testing/tests/package/test_lrzip.py
>   F:	support/testing/tests/package/test_lrzsz.py
> +F:	support/testing/tests/package/test_ltp_testsuite.py
>   F:	support/testing/tests/package/test_ltrace.py
>   F:	support/testing/tests/package/test_lvm2.py
>   F:	support/testing/tests/package/test_lzip.py
> diff --git a/support/testing/tests/package/test_ltp_testsuite.py b/support/testing/tests/package/test_ltp_testsuite.py
> new file mode 100644
> index 0000000000..a85ceb35a6
> --- /dev/null
> +++ b/support/testing/tests/package/test_ltp_testsuite.py
> @@ -0,0 +1,39 @@
> +import os
> +
> +import infra.basetest
> +
> +
> +class TestLtpTestsuite(infra.basetest.BRTest):
> +    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
> +        """
> +        BR2_PACKAGE_LTP_TESTSUITE=y
> +        BR2_TARGET_ROOTFS_EXT2=y
> +        BR2_TARGET_ROOTFS_EXT2_4=y
> +        BR2_TARGET_ROOTFS_EXT2_SIZE="600M"
> +        # BR2_TARGET_ROOTFS_TAR is not set
> +        """
> +
> +    def test_run(self):
> +        drive = os.path.join(self.builddir, "images", "rootfs.ext4")
> +        self.emulator.boot(arch="armv5",
> +                           kernel="builtin",
> +                           kernel_cmdline=["rootwait", "root=/dev/sda"],
> +                           options=["-drive", f"file={drive},if=scsi,format=raw"])
> +        self.emulator.login()
> +
> +        # We run a reduced number of tests (read syscall tests) for a
> +        # fast execution. See "runltp --help" for option details.
> +        cmd = "/usr/lib/ltp-testsuite/runltp"
> +        cmd += " -p -q"
> +        cmd += " -s ^read0[0-9]*"
> +        cmd += " -l /tmp/ltp.log"
> +        cmd += " -o /tmp/ltp.output"
> +        cmd += " -C /tmp/ltp.failed"
> +        cmd += " -T /tmp/ltp.tconf"
> +        self.assertRunOk(cmd)
> +
> +        # We print the LTP run log and check there was zero failure in
> +        # our test selection.
> +        out, ret = self.emulator.run("cat /tmp/ltp.log")
> +        self.assertEqual(ret, 0)
> +        self.assertIn("Total Failures: 0", out)

Hi Petr,

On 31/01/2025 23:36, Petr Vorel wrote:
[snip]
>> +        cmd = "/usr/lib/ltp-testsuite/runltp"
>> +        cmd += " -p -q"
>> +        cmd += " -s ^read0[0-9]*"
>> +        cmd += " -l /tmp/ltp.log"
>> +        cmd += " -o /tmp/ltp.output"
>> +        cmd += " -C /tmp/ltp.failed"
>> +        cmd += " -T /tmp/ltp.tconf"
> 
> While this above is OK, some notes:
> 
> 1) Many LTP tests require LTPROOT environment variable (to access some data files),
> e.g.:
> 
> export LTPROOT="/usr/lib/ltp-testsuite"
> 
> See https://linux-test-project.readthedocs.io/en/latest/users/setup_tests.html

  Since only a limited set of tests are run, it's apparently not needed.

> It looks to me that infra.emulator.Emulator does not support inserting
> environment variables via os.environ.

  No, the environment can be set as part of the command itself:

cmd = ("LTPROOT='...' "
        "/usr/lib/ltp-testsute/runltp "
        ...
       )

> 2) runltp old script has been deprecated, kirk is the replacement:

  Ah, annoying. I think we'd probably prefer to run some tests directly then, 
without runltp at all. The point of this test is not to run the actual LTP 
tests, but just to verify that ltp-testsuite has been installed correctly.


> https://github.com/linux-test-project/kirk
> 
> Currently kirk supports few testing frameworks: LTP, kselftests and liburing.
> 
> While kirk can be run o SUT (like runltp), the best usage is to run it on the
> host. It supports testing via starting QEMU VM, SSH and not yet stable LTX
> backend.

  Starting via Qemu is possibly annoying because then we no longer control how 
QEMU is booted. SSH could be interesting, but currently the image doesn't have 
an SSH server by default. So I think running kirk in the test environment 
directly is probably the easiest for us.

  Regards,
  Arnout



> 
> Due the above I'd like to add kirk host package to Buildroot.
> 
> Kind regards,
> Petr
> 
> 
>> +        self.assertRunOk(cmd)
>> +
>> +        # We print the LTP run log and check there was zero failure in
>> +        # our test selection.
>> +        out, ret = self.emulator.run("cat /tmp/ltp.log")
>> +        self.assertEqual(ret, 0)
>> +        self.assertIn("Total Failures: 0", out)
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot