diff mbox series

[v2,2/2] Boot Linux Console Test: add a test for the Raspberry Pi 2

Message ID 20190312234541.2887-3-philmd@redhat.com
State New
Headers show
Series Acceptance Tests: Test the Raspberry Pi 2 board | expand

Commit Message

Philippe Mathieu-Daudé March 12, 2019, 11:45 p.m. UTC 
From: Philippe Mathieu-Daudé <f4bug@amsat.org>

Similar to the x86_64/pc test, it boots a Linux kernel on a raspi2
board and verify the serial is working.

If ARM is a target being built, "make check-acceptance" will
automatically include this test by the use of the "arch:arm" tags.

Alternatively, this test can be run using:

    $ avocado run -t arch:arm tests/acceptance
    $ avocado run -t machine:raspi2 tests/acceptance

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
 tests/acceptance/boot_linux_console.py | 29 ++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

Comments

Cleber Rosa March 13, 2019, 12:09 a.m. UTC | #1 
On Wed, Mar 13, 2019 at 12:45:41AM +0100, Philippe Mathieu-Daudé wrote:
> From: Philippe Mathieu-Daudé <f4bug@amsat.org>
> 
> Similar to the x86_64/pc test, it boots a Linux kernel on a raspi2
> board and verify the serial is working.
> 
> If ARM is a target being built, "make check-acceptance" will
> automatically include this test by the use of the "arch:arm" tags.
> 
> Alternatively, this test can be run using:
> 
>     $ avocado run -t arch:arm tests/acceptance
>     $ avocado run -t machine:raspi2 tests/acceptance
> 
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
>  tests/acceptance/boot_linux_console.py | 29 ++++++++++++++++++++++++++
>  1 file changed, 29 insertions(+)
> 
> diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
> index 0936589fd8..dd2733b55f 100644
> --- a/tests/acceptance/boot_linux_console.py
> +++ b/tests/acceptance/boot_linux_console.py
> @@ -197,6 +197,35 @@ class BootLinuxConsole(Test):
>          console_pattern = 'Kernel command line: %s' % kernel_command_line
>          self.wait_for_console_pattern(console_pattern)
>  
> +    def test_arm_raspi2(self):
> +        """
> +        The kernel can be rebuilt using the kernel source referenced
> +        and following the instructions on the on:
> +        https://www.raspberrypi.org/documentation/linux/kernel/building.md
> +
> +        :avocado: tags=arch:arm
> +        :avocado: tags=machine:raspi2
> +        """
> +        deb_url = ('http://archive.raspberrypi.org/debian/'
> +                   'pool/main/r/raspberrypi-firmware/'
> +                   'raspberrypi-kernel_1.20190215-1_armhf.deb')
> +        deb_hash = 'cd284220b32128c5084037553db3c482426f3972'
> +        deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
> +        print("deb_path: ", deb_path)

This looks like a left over you forgot to remove.

> +        kernel_path = self.extract_from_deb(deb_path, '/boot/kernel7.img')
> +        dtb_path = self.extract_from_deb(deb_path, '/boot/bcm2709-rpi-2-b.dtb')
> +
> +        self.vm.set_machine('raspi2')
> +        self.vm.set_console()
> +        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
> +                               'console=ttyAMA0')
> +        self.vm.add_args('-kernel', kernel_path,
> +                         '-dtb', dtb_path,
> +                         '-append', kernel_command_line)
> +        self.vm.launch()
> +        console_pattern = 'Kernel command line: %s' % kernel_command_line
> +        self.wait_for_console_pattern(console_pattern)
> +
>      def test_s390x_s390_ccw_virtio(self):
>          """
>          :avocado: tags=arch:s390x
> -- 
> 2.20.1
> 

Other than that, it looks good.

- Cleber.
Eduardo Habkost May 22, 2019, 8:52 p.m. UTC | #2 
On Wed, Mar 13, 2019 at 12:45:41AM +0100, Philippe Mathieu-Daudé wrote:
> From: Philippe Mathieu-Daudé <f4bug@amsat.org>
> 
> Similar to the x86_64/pc test, it boots a Linux kernel on a raspi2
> board and verify the serial is working.
> 
> If ARM is a target being built, "make check-acceptance" will
> automatically include this test by the use of the "arch:arm" tags.
> 
> Alternatively, this test can be run using:
> 
>     $ avocado run -t arch:arm tests/acceptance
>     $ avocado run -t machine:raspi2 tests/acceptance
> 
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>

We're getting timeouts on travis-ci:
https://travis-ci.org/ehabkost/qemu/jobs/535468057#L3289

I don't know yet if the guest is hanging, or if we just need to
increase the timeout.  I could reproduce the timeout locally,
too.
Cleber Rosa May 22, 2019, 9:05 p.m. UTC | #3 
----- Original Message -----
> From: "Eduardo Habkost" <ehabkost@redhat.com>
> To: "Philippe Mathieu-Daudé" <philmd@redhat.com>
> Cc: qemu-devel@nongnu.org, "Cleber Rosa" <crosa@redhat.com>, "Peter Maydell" <peter.maydell@linaro.org>,
> qemu-arm@nongnu.org, "Philippe Mathieu-Daudé" <f4bug@amsat.org>, "Andrew Baumann" <Andrew.Baumann@microsoft.com>
> Sent: Wednesday, May 22, 2019 4:52:34 PM
> Subject: Re: [Qemu-devel] [PATCH v2 2/2] Boot Linux Console Test: add a test for the Raspberry Pi 2
> 
> On Wed, Mar 13, 2019 at 12:45:41AM +0100, Philippe Mathieu-Daudé wrote:
> > From: Philippe Mathieu-Daudé <f4bug@amsat.org>
> > 
> > Similar to the x86_64/pc test, it boots a Linux kernel on a raspi2
> > board and verify the serial is working.
> > 
> > If ARM is a target being built, "make check-acceptance" will
> > automatically include this test by the use of the "arch:arm" tags.
> > 
> > Alternatively, this test can be run using:
> > 
> >     $ avocado run -t arch:arm tests/acceptance
> >     $ avocado run -t machine:raspi2 tests/acceptance
> > 
> > Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> 
> We're getting timeouts on travis-ci:
> https://travis-ci.org/ehabkost/qemu/jobs/535468057#L3289
> 
> I don't know yet if the guest is hanging, or if we just need to
> increase the timeout.  I could reproduce the timeout locally,
> too.
> 
> --
> Eduardo
> 

It may be related to:

 https://bugs.launchpad.net/qemu/+bug/1829779

And this is a proof that we urgently need to have a better
way of presenting/storing test artifacts.  The brief output
is nice when everything goes well, but makes the test results
close to useless once a failure happens.

Philippe showed us how GitLab allows CI jobs to preserve
artifacts, so maybe the solution is to move the loads there.

- Cleber.
Philippe Mathieu-Daudé May 23, 2019, 9:29 a.m. UTC | #4 
On 5/22/19 11:05 PM, Cleber Rosa wrote:
> ----- Original Message -----
>> From: "Eduardo Habkost" <ehabkost@redhat.com>
>> On Wed, Mar 13, 2019 at 12:45:41AM +0100, Philippe Mathieu-Daudé wrote:
>>> From: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>>
>>> Similar to the x86_64/pc test, it boots a Linux kernel on a raspi2
>>> board and verify the serial is working.
>>>
>>> If ARM is a target being built, "make check-acceptance" will
>>> automatically include this test by the use of the "arch:arm" tags.
>>>
>>> Alternatively, this test can be run using:
>>>
>>>     $ avocado run -t arch:arm tests/acceptance
>>>     $ avocado run -t machine:raspi2 tests/acceptance
>>>
>>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>>
>> We're getting timeouts on travis-ci:
>> https://travis-ci.org/ehabkost/qemu/jobs/535468057#L3289
>>
>> I don't know yet if the guest is hanging, or if we just need to
>> increase the timeout.  I could reproduce the timeout locally,
>> too.

That's odd, I can't reproduce (this test is quicker than the following
test_arm_emcraft_sf2 which start u-boot then timeouts and start the kernel).

My guess is network issues, since this test use a different mirror:
archive.raspberrypi.org

Gerd already raised this problem (timeout reached while fetching
artifacts) to Cleber.
Cleber, can we add test_setup() methods that use different timeouts?

Regards,

Phil.

> It may be related to:
> 
>  https://bugs.launchpad.net/qemu/+bug/1829779
> 
> And this is a proof that we urgently need to have a better
> way of presenting/storing test artifacts.  The brief output
> is nice when everything goes well, but makes the test results
> close to useless once a failure happens.
> 
> Philippe showed us how GitLab allows CI jobs to preserve
> artifacts, so maybe the solution is to move the loads there.
> 
> - Cleber.
>
Cleber Rosa May 23, 2019, 1:40 p.m. UTC | #5 
----- Original Message -----
> From: "Philippe Mathieu-Daudé" <philmd@redhat.com>
> To: "Cleber Rosa" <crosa@redhat.com>, "Eduardo Habkost" <ehabkost@redhat.com>
> Cc: qemu-devel@nongnu.org, "Peter Maydell" <peter.maydell@linaro.org>, qemu-arm@nongnu.org, "Philippe Mathieu-Daudé"
> <f4bug@amsat.org>, "Andrew Baumann" <Andrew.Baumann@microsoft.com>, "Gerd Hoffmann" <kraxel@redhat.com>
> Sent: Thursday, May 23, 2019 5:29:22 AM
> Subject: Re: [Qemu-devel] [PATCH v2 2/2] Boot Linux Console Test: add a test for the Raspberry Pi 2
> 
> On 5/22/19 11:05 PM, Cleber Rosa wrote:
> > ----- Original Message -----
> >> From: "Eduardo Habkost" <ehabkost@redhat.com>
> >> On Wed, Mar 13, 2019 at 12:45:41AM +0100, Philippe Mathieu-Daudé wrote:
> >>> From: Philippe Mathieu-Daudé <f4bug@amsat.org>
> >>>
> >>> Similar to the x86_64/pc test, it boots a Linux kernel on a raspi2
> >>> board and verify the serial is working.
> >>>
> >>> If ARM is a target being built, "make check-acceptance" will
> >>> automatically include this test by the use of the "arch:arm" tags.
> >>>
> >>> Alternatively, this test can be run using:
> >>>
> >>>     $ avocado run -t arch:arm tests/acceptance
> >>>     $ avocado run -t machine:raspi2 tests/acceptance
> >>>
> >>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> >>
> >> We're getting timeouts on travis-ci:
> >> https://travis-ci.org/ehabkost/qemu/jobs/535468057#L3289
> >>
> >> I don't know yet if the guest is hanging, or if we just need to
> >> increase the timeout.  I could reproduce the timeout locally,
> >> too.
> 
> That's odd, I can't reproduce (this test is quicker than the following
> test_arm_emcraft_sf2 which start u-boot then timeouts and start the kernel).
> 
> My guess is network issues, since this test use a different mirror:
> archive.raspberrypi.org
> 

It could be a network issue, it could be something else.  I think the very
first step, and I'd urge us to get that on master ASAP, is to show the
entire logs in CI, I mean:

---

diff --git a/.travis.yml b/.travis.yml
index 6fd89b3d91..fd8f6ca2d2 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -225,7 +225,7 @@ matrix:
     # Acceptance (Functional) tests
     - env:
         - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu"
-        - TEST_CMD="make check-acceptance"
+        - TEST_CMD="make AVOCADO_SHOW=test check-acceptance"
       addons:
         apt:
           packages:

---

That way we can know for sure what's going on. 

> Gerd already raised this problem (timeout reached while fetching
> artifacts) to Cleber.
> Cleber, can we add test_setup() methods that use different timeouts?
> 

Not in a released Avocado version.  Interestingly enough, I wrote a PoC
that adds different timeouts to tearDown()[1], that can be used the same
way for setUp(), and it looks like Intel DAOS is already using those
patches[2].

So, I guess I can work on a non-PoC version for that.

- Cleber.

[1] - https://github.com/avocado-framework/avocado/pull/3076
[2] - https://github.com/daos-stack/daos/commit/084ec23461e7bd9b997d4b6f5e8095a4eaffc685

> Regards,
> 
> Phil.
> 
> > It may be related to:
> > 
> >  https://bugs.launchpad.net/qemu/+bug/1829779
> > 
> > And this is a proof that we urgently need to have a better
> > way of presenting/storing test artifacts.  The brief output
> > is nice when everything goes well, but makes the test results
> > close to useless once a failure happens.
> > 
> > Philippe showed us how GitLab allows CI jobs to preserve
> > artifacts, so maybe the solution is to move the loads there.
> > 
> > - Cleber.
> > 
>
Eduardo Habkost June 5, 2019, 8:33 p.m. UTC | #6 
On Thu, May 23, 2019 at 09:40:06AM -0400, Cleber Rosa wrote:
> ----- Original Message -----
> > From: "Philippe Mathieu-Daudé" <philmd@redhat.com>
> > To: "Cleber Rosa" <crosa@redhat.com>, "Eduardo Habkost" <ehabkost@redhat.com>
> > Cc: qemu-devel@nongnu.org, "Peter Maydell" <peter.maydell@linaro.org>, qemu-arm@nongnu.org, "Philippe Mathieu-Daudé"
> > <f4bug@amsat.org>, "Andrew Baumann" <Andrew.Baumann@microsoft.com>, "Gerd Hoffmann" <kraxel@redhat.com>
> > Sent: Thursday, May 23, 2019 5:29:22 AM
> > Subject: Re: [Qemu-devel] [PATCH v2 2/2] Boot Linux Console Test: add a test for the Raspberry Pi 2
> > 
> > On 5/22/19 11:05 PM, Cleber Rosa wrote:
> > > ----- Original Message -----
> > >> From: "Eduardo Habkost" <ehabkost@redhat.com>
> > >> On Wed, Mar 13, 2019 at 12:45:41AM +0100, Philippe Mathieu-Daudé wrote:
> > >>> From: Philippe Mathieu-Daudé <f4bug@amsat.org>
> > >>>
> > >>> Similar to the x86_64/pc test, it boots a Linux kernel on a raspi2
> > >>> board and verify the serial is working.
> > >>>
> > >>> If ARM is a target being built, "make check-acceptance" will
> > >>> automatically include this test by the use of the "arch:arm" tags.
> > >>>
> > >>> Alternatively, this test can be run using:
> > >>>
> > >>>     $ avocado run -t arch:arm tests/acceptance
> > >>>     $ avocado run -t machine:raspi2 tests/acceptance
> > >>>
> > >>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> > >>
> > >> We're getting timeouts on travis-ci:
> > >> https://travis-ci.org/ehabkost/qemu/jobs/535468057#L3289
> > >>
> > >> I don't know yet if the guest is hanging, or if we just need to
> > >> increase the timeout.  I could reproduce the timeout locally,
> > >> too.
> > 
> > That's odd, I can't reproduce (this test is quicker than the following
> > test_arm_emcraft_sf2 which start u-boot then timeouts and start the kernel).
> > 
> > My guess is network issues, since this test use a different mirror:
> > archive.raspberrypi.org
> > 
> 
> It could be a network issue, it could be something else.  I think the very
> first step, and I'd urge us to get that on master ASAP, is to show the
> entire logs in CI, I mean:
> 
> ---
> 
> diff --git a/.travis.yml b/.travis.yml
> index 6fd89b3d91..fd8f6ca2d2 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -225,7 +225,7 @@ matrix:
>      # Acceptance (Functional) tests
>      - env:
>          - CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu,mips-softmmu,mips64el-softmmu,aarch64-softmmu,arm-softmmu,s390x-softmmu,alpha-softmmu"
> -        - TEST_CMD="make check-acceptance"
> +        - TEST_CMD="make AVOCADO_SHOW=test check-acceptance"
>        addons:
>          apt:
>            packages:

I have applied this change on python-next.  The failure can be
seen here:
https://travis-ci.org/ehabkost/qemu/jobs/541758418#L4033

While we don't fix this, I'm removing this test case from the
queue, so I can send a pull request.  Sorry for taking so long to
act on this.
diff mbox series

Patch

diff --git a/tests/acceptance/boot_linux_console.py b/tests/acceptance/boot_linux_console.py
index 0936589fd8..dd2733b55f 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -197,6 +197,35 @@  class BootLinuxConsole(Test):
         console_pattern = 'Kernel command line: %s' % kernel_command_line
         self.wait_for_console_pattern(console_pattern)
 
+    def test_arm_raspi2(self):
+        """
+        The kernel can be rebuilt using the kernel source referenced
+        and following the instructions on the on:
+        https://www.raspberrypi.org/documentation/linux/kernel/building.md
+
+        :avocado: tags=arch:arm
+        :avocado: tags=machine:raspi2
+        """
+        deb_url = ('http://archive.raspberrypi.org/debian/'
+                   'pool/main/r/raspberrypi-firmware/'
+                   'raspberrypi-kernel_1.20190215-1_armhf.deb')
+        deb_hash = 'cd284220b32128c5084037553db3c482426f3972'
+        deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
+        print("deb_path: ", deb_path)
+        kernel_path = self.extract_from_deb(deb_path, '/boot/kernel7.img')
+        dtb_path = self.extract_from_deb(deb_path, '/boot/bcm2709-rpi-2-b.dtb')
+
+        self.vm.set_machine('raspi2')
+        self.vm.set_console()
+        kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
+                               'console=ttyAMA0')
+        self.vm.add_args('-kernel', kernel_path,
+                         '-dtb', dtb_path,
+                         '-append', kernel_command_line)
+        self.vm.launch()
+        console_pattern = 'Kernel command line: %s' % kernel_command_line
+        self.wait_for_console_pattern(console_pattern)
+
     def test_s390x_s390_ccw_virtio(self):
         """
         :avocado: tags=arch:s390x