diff mbox series

[v3,2/8] support/testing: disable tests using edk2 on aarch64 architecture

Message ID 20241031213348.2057705-3-ju.o@free.fr
State Accepted
Headers show
Series *** Fixes for qemu_aarch64_sbsa_defconfig *** | expand

Commit Message

Julien Olivain Oct. 31, 2024, 9:33 p.m. UTC
EDK2 version edk2-stable202405 (not yet in Buildroot) has dependencies
on specific Qemu version.

Buildroot guidelines also recommend that "each patch must be complete".
See: [1]. This means, a patch in a series is not allowed to break
something knowingly, even in the same patch series. This rule is
mainly for using git bisect.

We also want to avoid bumping EDK2, update the runtime test infra,
and few tests in a big single commit, for readability. This would mix
too many changes in the same commit.

As a trade off, this commit disables the runtime tests which are known
to be broken by the upcoming edk2 update. Then, those tests will be
individually fixed and re-enabled in followup commits.

This EDK2/Qemu incompatibility only affects runtime tests using EDK2
on the Arm Aarch64 architecture. This commit disable the runtime
tests by adding a "skipTest()" call in:
- tests.boot.test_edk2
- tests.boot.test_grub.TestGrubAArch64EFI
- tests.package.test_fwts

[1] https://buildroot.org/downloads/manual/manual.html#submitting-patches

Signed-off-by: Julien Olivain <ju.o@free.fr>
---
 support/testing/tests/boot/test_edk2.py    | 4 ++++
 support/testing/tests/boot/test_grub.py    | 4 ++++
 support/testing/tests/package/test_fwts.py | 4 ++++
 3 files changed, 12 insertions(+)

Comments

Romain Naour Oct. 31, 2024, 10:02 p.m. UTC | #1
Hello Julien,

Le 31/10/2024 à 22:33, Julien Olivain a écrit :
> EDK2 version edk2-stable202405 (not yet in Buildroot) has dependencies
> on specific Qemu version.
> 
> Buildroot guidelines also recommend that "each patch must be complete".
> See: [1]. This means, a patch in a series is not allowed to break
> something knowingly, even in the same patch series. This rule is
> mainly for using git bisect.
> 
> We also want to avoid bumping EDK2, update the runtime test infra,
> and few tests in a big single commit, for readability. This would mix
> too many changes in the same commit.
> 
> As a trade off, this commit disables the runtime tests which are known
> to be broken by the upcoming edk2 update. Then, those tests will be
> individually fixed and re-enabled in followup commits.

With Julien, we did our best to avoid breaking tests. This Qemu version bump
break things in several subtile way...

I don't have another suggestion.

Reviewed-by: Romain Naour <romain.naour@smile.fr>

Yet, this qemu 9.1 commit [1] broke qemu_arm_ebbr_defconfig [2].
The issue is currently investigated [3].

[1]
https://gitlab.com/qemu-project/qemu/-/commit/4c2c0474693229c1f533239bb983495c5427784d
[2] https://gitlab.com/buildroot.org/buildroot/-/jobs/8233227337
[3] https://gitlab.com/qemu-project/qemu/-/issues/2588#note_2181296618

Best regards,
Romain


> 
> This EDK2/Qemu incompatibility only affects runtime tests using EDK2
> on the Arm Aarch64 architecture. This commit disable the runtime
> tests by adding a "skipTest()" call in:
> - tests.boot.test_edk2
> - tests.boot.test_grub.TestGrubAArch64EFI
> - tests.package.test_fwts
> 
> [1] https://buildroot.org/downloads/manual/manual.html#submitting-patches
> 
> Signed-off-by: Julien Olivain <ju.o@free.fr>
> ---
>  support/testing/tests/boot/test_edk2.py    | 4 ++++
>  support/testing/tests/boot/test_grub.py    | 4 ++++
>  support/testing/tests/package/test_fwts.py | 4 ++++
>  3 files changed, 12 insertions(+)
> 
> diff --git a/support/testing/tests/boot/test_edk2.py b/support/testing/tests/boot/test_edk2.py
> index 16fa10f8c9..7d350f52d9 100644
> --- a/support/testing/tests/boot/test_edk2.py
> +++ b/support/testing/tests/boot/test_edk2.py
> @@ -33,6 +33,10 @@ class TestEdk2(infra.basetest.BRTest):
>          BR2_PACKAGE_HOST_MTOOLS=y
>          """
>  
> +    def setUp(self):
> +        self.skipTest("Incompatible QEmu 5.2.0 (Docker image) / "
> +                      "EDK2 (stable202405) versions.")
> +
>      def test_run(self):
>          hda = os.path.join(self.builddir, "images", "disk.img")
>          flash0 = os.path.join(self.builddir, "images", "SBSA_FLASH0.fd")
> diff --git a/support/testing/tests/boot/test_grub.py b/support/testing/tests/boot/test_grub.py
> index 5315fa23ba..41234de2b9 100644
> --- a/support/testing/tests/boot/test_grub.py
> +++ b/support/testing/tests/boot/test_grub.py
> @@ -156,6 +156,10 @@ class TestGrubAArch64EFI(infra.basetest.BRTest):
>          """.format(post_image=infra.filepath("tests/boot/test_grub/post-image-aarch64-efi.sh"),
>                     linux_fragment=infra.filepath("tests/boot/test_grub/linux-aarch64-efi.config"))
>  
> +    def setUp(self):
> +        self.skipTest("Incompatible QEmu 5.2.0 (Docker image) / "
> +                      "EDK2 (stable202405) versions.")
> +
>      def test_run(self):
>          hda = os.path.join(self.builddir, "images", "disk.img")
>          bios = os.path.join(self.builddir, "images", "QEMU_EFI.fd")
> diff --git a/support/testing/tests/package/test_fwts.py b/support/testing/tests/package/test_fwts.py
> index ef792136c2..2c9873ee54 100644
> --- a/support/testing/tests/package/test_fwts.py
> +++ b/support/testing/tests/package/test_fwts.py
> @@ -36,6 +36,10 @@ class TestFwts(infra.basetest.BRTest):
>          BR2_PACKAGE_HOST_MTOOLS=y
>          """
>  
> +    def setUp(self):
> +        self.skipTest("Incompatible QEmu 5.2.0 (Docker image) / "
> +                      "EDK2 (stable202405) versions.")
> +
>      def test_run(self):
>          hda = os.path.join(self.builddir, "images", "disk.img")
>          flash0 = os.path.join(self.builddir, "images", "SBSA_FLASH0.fd")
diff mbox series

Patch

diff --git a/support/testing/tests/boot/test_edk2.py b/support/testing/tests/boot/test_edk2.py
index 16fa10f8c9..7d350f52d9 100644
--- a/support/testing/tests/boot/test_edk2.py
+++ b/support/testing/tests/boot/test_edk2.py
@@ -33,6 +33,10 @@  class TestEdk2(infra.basetest.BRTest):
         BR2_PACKAGE_HOST_MTOOLS=y
         """
 
+    def setUp(self):
+        self.skipTest("Incompatible QEmu 5.2.0 (Docker image) / "
+                      "EDK2 (stable202405) versions.")
+
     def test_run(self):
         hda = os.path.join(self.builddir, "images", "disk.img")
         flash0 = os.path.join(self.builddir, "images", "SBSA_FLASH0.fd")
diff --git a/support/testing/tests/boot/test_grub.py b/support/testing/tests/boot/test_grub.py
index 5315fa23ba..41234de2b9 100644
--- a/support/testing/tests/boot/test_grub.py
+++ b/support/testing/tests/boot/test_grub.py
@@ -156,6 +156,10 @@  class TestGrubAArch64EFI(infra.basetest.BRTest):
         """.format(post_image=infra.filepath("tests/boot/test_grub/post-image-aarch64-efi.sh"),
                    linux_fragment=infra.filepath("tests/boot/test_grub/linux-aarch64-efi.config"))
 
+    def setUp(self):
+        self.skipTest("Incompatible QEmu 5.2.0 (Docker image) / "
+                      "EDK2 (stable202405) versions.")
+
     def test_run(self):
         hda = os.path.join(self.builddir, "images", "disk.img")
         bios = os.path.join(self.builddir, "images", "QEMU_EFI.fd")
diff --git a/support/testing/tests/package/test_fwts.py b/support/testing/tests/package/test_fwts.py
index ef792136c2..2c9873ee54 100644
--- a/support/testing/tests/package/test_fwts.py
+++ b/support/testing/tests/package/test_fwts.py
@@ -36,6 +36,10 @@  class TestFwts(infra.basetest.BRTest):
         BR2_PACKAGE_HOST_MTOOLS=y
         """
 
+    def setUp(self):
+        self.skipTest("Incompatible QEmu 5.2.0 (Docker image) / "
+                      "EDK2 (stable202405) versions.")
+
     def test_run(self):
         hda = os.path.join(self.builddir, "images", "disk.img")
         flash0 = os.path.join(self.builddir, "images", "SBSA_FLASH0.fd")