Acceptance Test BootLinuxAarch64.test_virt_tcg execution times

TL;DR: This is a followup from an IRC chat about the
tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg test
taking many orders of magnitute longer than other acceptance (and even
similar boot) tests.  I could not find an easy way for this specific
test (aarch64+tcg) to have a significant execution time improvement.
The best solution may be to filter out tests that are known to be
slow, and create a specific "test job" that includes them.

Fisrt, if it's not clear, this specific test runs QEMU with TCG and
boots a Fedora 31 "cloud image" and waits until the cloud-init agent
notifies the test that the boot is over.

Out of the four archtiectures tested with the same approach under
"tests/acceptance/boot_linux.py", aarch64 was special, in the sense
that many Linux "cloud images" got stuck very late in the boot
process.  What seemed to be a disk activity within the guest that
seemed to make the kernel drain its random number sources if my memory
serves me right.  Giving the machine a RNG device fixed it.  This can
still be verified Today if you comment the virtio-rng lines in the
aarch64 test.

So, even with the RNG device and the boot process not getting stuck, a
lot of the test time is spent with QEMU actively using CPU time
produced by the guest boot process.  This may or may not be the cause
for the slowness.

One approach to have a shorter test time, would be to reduce the
things that happen during the guest boot process.  Choosing a minimal
guest, such as CirrOS, would be an example of such a solution, but:

* With less things happening during the guest boot, less things
  get tested within QEMU;

* CirrOS can not make use of the same boot cloud-init configuration
  and boot verification the test currently uses;

So that leaves other non-minimal Linux "cloud images" as options.  But
still, the following things are required or nice to have:

* Support for cloud-init;

* Support for as many as possible architectures;

* Wide user base;

* Be thoroughly tested with this "boot_linux.py" test

So in the end, I picked Fedora 31, which was available and behaved
well for four different architectures with and without KVM. Today, I
verified if switching distros would provide an "easy fix", but the
results were negative.  Any ideas on how to improve the test execution
times are appreciated.

For the record, one of the ways we're trying to improve the overall
test experience is to allow tests to run in parallel (expected to be
fully supported on the upcoming version 81.0).

For those interested, these are the numbers I got, and how I tested
with other distros.  I'm using QEMU e1d322c405 with a vanilla
configure under a x86_64 Fedora 32 host.

Fedora 31 (baseline):
====================

  $ make check-venv
  $ ./tests/venv/bin/avocado run -t arch:aarch64,accel:tcg --keep-tmp on -- tests/acceptance/boot_linux.py{,,,,}
  JOB ID     : 14802f9d5016a44d2937ed7b1fec63b2eaa06e89
  JOB LOG    : /home/cleber/avocado/job-results/job-2020-08-06T13.19-14802f9/job.log
   (1/5) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (153.12 s)
   (2/5) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (149.57 s)
   (3/5) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (154.45 s)
   (4/5) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (148.97 s)
   (5/5) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (150.70 s)
  RESULTS    : PASS 5 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
  JOB TIME   : 757.50 s

Fedora 32:
==========

1. Tweak version and image hash:

---
---

2. Download the image before the test:

  $ ./tests/venv/bin/avocado vmimage get --distro=ubuntu --arch aarch64 --distro-version=20.04
  The image was downloaded:
  Provider Version Architecture File
  ubuntu   20.04   arm64        /tmp/data/cache/by_location/19db8c6d910a3f2660c4109ffb85d73d43e5cdf2/ubuntu-20.04-server-cloudimg-arm64.img

3. Run the tests:

  $ ./tests/venv/bin/avocado run -t arch:aarch64,accel:tcg --keep-tmp on -- tests/acceptance/boot_linux.py{,,,,}
  JOB ID     : 92a1bdbb5e933e6dff8b882808a191f1de3c2600
  JOB LOG    : /home/cleber/avocado/job-results/job-2020-08-06T12.13-92a1bdb/job.log
   (1/5) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (341.40 s)
   (2/5) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (345.82 s)
   (3/5) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (335.91 s)
   (4/5) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (320.32 s)
   (5/5) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (319.79 s)
  RESULTS    : PASS 5 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
  JOB TIME   : 1663.92 s

On 8/6/20 9:35 PM, Cleber Rosa wrote:
> TL;DR: This is a followup from an IRC chat about the
> tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg test
> taking many orders of magnitute longer than other acceptance (and even
> similar boot) tests.  I could not find an easy way for this specific
> test (aarch64+tcg) to have a significant execution time improvement.
> The best solution may be to filter out tests that are known to be
> slow, and create a specific "test job" that includes them.
> 
> Fisrt, if it's not clear, this specific test runs QEMU with TCG and
> boots a Fedora 31 "cloud image" and waits until the cloud-init agent
> notifies the test that the boot is over.
> 
> Out of the four archtiectures tested with the same approach under
> "tests/acceptance/boot_linux.py", aarch64 was special, in the sense
> that many Linux "cloud images" got stuck very late in the boot
> process.  What seemed to be a disk activity within the guest that
> seemed to make the kernel drain its random number sources if my memory
> serves me right.  Giving the machine a RNG device fixed it.  This can
> still be verified Today if you comment the virtio-rng lines in the
> aarch64 test.
> 
> So, even with the RNG device and the boot process not getting stuck, a
> lot of the test time is spent with QEMU actively using CPU time
> produced by the guest boot process.  This may or may not be the cause
> for the slowness.

I believe this is related to the issue Richard recently addressed:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg729216.html

> 
> One approach to have a shorter test time, would be to reduce the
> things that happen during the guest boot process.  Choosing a minimal
> guest, such as CirrOS, would be an example of such a solution, but:
> 
> * With less things happening during the guest boot, less things
>   get tested within QEMU;
> 
> * CirrOS can not make use of the same boot cloud-init configuration
>   and boot verification the test currently uses;
> 
> So that leaves other non-minimal Linux "cloud images" as options.  But
> still, the following things are required or nice to have:
> 
> * Support for cloud-init;
> 
> * Support for as many as possible architectures;
> 
> * Wide user base;
> 
> * Be thoroughly tested with this "boot_linux.py" test
> 
> So in the end, I picked Fedora 31, which was available and behaved
> well for four different architectures with and without KVM. Today, I
> verified if switching distros would provide an "easy fix", but the
> results were negative.  Any ideas on how to improve the test execution
> times are appreciated.
> 
> For the record, one of the ways we're trying to improve the overall
> test experience is to allow tests to run in parallel (expected to be
> fully supported on the upcoming version 81.0).
> 
> For those interested, these are the numbers I got, and how I tested
> with other distros.  I'm using QEMU e1d322c405 with a vanilla
> configure under a x86_64 Fedora 32 host.
> 
> Fedora 31 (baseline):
> ====================
> 
>   $ make check-venv
>   $ ./tests/venv/bin/avocado run -t arch:aarch64,accel:tcg --keep-tmp on -- tests/acceptance/boot_linux.py{,,,,}
>   JOB ID     : 14802f9d5016a44d2937ed7b1fec63b2eaa06e89
>   JOB LOG    : /home/cleber/avocado/job-results/job-2020-08-06T13.19-14802f9/job.log
>    (1/5) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (153.12 s)
>    (2/5) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (149.57 s)
>    (3/5) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (154.45 s)
>    (4/5) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (148.97 s)
>    (5/5) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (150.70 s)
>   RESULTS    : PASS 5 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
>   JOB TIME   : 757.50 s
> 
> Fedora 32:
> ==========
> 
> 1. Tweak version and image hash:
> 
> ---
> diff --git a/tests/acceptance/boot_linux.py b/tests/acceptance/boot_linux.py
> index 0055dc7cee..44c62bd4a2 100644
> --- a/tests/acceptance/boot_linux.py
> +++ b/tests/acceptance/boot_linux.py
> @@ -48,7 +48,7 @@ class BootLinuxBase(Test):
>              image_arch = 'ppc64le'
>          try:
>              boot = vmimage.get(
> -                'fedora', arch=image_arch, version='31',
> +                'fedora', arch=image_arch, version='32',
>                  checksum=self.chksum,
>                  algorithm='sha256',
>                  cache_dir=self.cache_dirs[0],
> @@ -160,7 +160,7 @@ class BootLinuxAarch64(BootLinux):
>      :avocado: tags=machine:gic-version=2
>      """
>  
> -    chksum = '1e18d9c0cf734940c4b5d5ec592facaed2af0ad0329383d5639c997fdf16fe49'
> +    chksum = 'b367755c664a2d7a26955bbfff985855adfa2ca15e908baf15b4b176d68d3967'
>  
>      def add_common_args(self):
>          self.vm.add_args('-bios',
> ---
> 
> 2. Download the image before the test:
> 
>   $ ./tests/venv/bin/avocado vmimage get --distro=fedora --arch aarch64 --distro-version=32
>   The image was downloaded:
>   Provider Version Architecture File
>   fedora   32      aarch64      /tmp/data/cache/by_location/7049001631a4b2eabf5766cc110e66d486e09821/Fedora-Cloud-Base-32-1.6.aarch64.qcow2
> 
> 3. Run the tests:
> 
>   $ ./tests/venv/bin/avocado run -t arch:aarch64,accel:tcg --keep-tmp on -- tests/acceptance/boot_linux.py{,,,,}
> JOB ID     : 09e740a41dc400f9fcbb9253f613734597fe0efc
> JOB LOG    : /home/cleber/avocado/job-results/job-2020-08-06T13.53-09e740a/job.log
>  (1/5) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (162.06 s)
>  (2/5) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (167.78 s)
>  (3/5) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (166.98 s)
>  (4/5) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (171.13 s)
>  (5/5) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (167.43 s)
> RESULTS    : PASS 5 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
> JOB TIME   : 836.05 s
> 
> Ubuntu 20.04:
> =============
> 
> 1. Tweak version and image hash:
> 
> ---
> diff --git a/tests/acceptance/boot_linux.py b/tests/acceptance/boot_linux.py
> index 0055dc7cee..03c0e1bee9 100644
> --- a/tests/acceptance/boot_linux.py
> +++ b/tests/acceptance/boot_linux.py
> @@ -48,7 +48,7 @@ class BootLinuxBase(Test):
>              image_arch = 'ppc64le'
>          try:
>              boot = vmimage.get(
> -                'fedora', arch=image_arch, version='31',
> +                'ubuntu', arch=image_arch, version='20.04',
>                  checksum=self.chksum,
>                  algorithm='sha256',
>                  cache_dir=self.cache_dirs[0],
> @@ -160,7 +160,7 @@ class BootLinuxAarch64(BootLinux):
>      :avocado: tags=machine:gic-version=2
>      """
>  
> -    chksum = '1e18d9c0cf734940c4b5d5ec592facaed2af0ad0329383d5639c997fdf16fe49'
> +    chksum = '1d9e50f3381145835b11911adf611f455d674a570814086b7d6581ecc0718770'
>  
>      def add_common_args(self):
>          self.vm.add_args('-bios',
> ---
> 
> 2. Download the image before the test:
> 
>   $ ./tests/venv/bin/avocado vmimage get --distro=ubuntu --arch aarch64 --distro-version=20.04
>   The image was downloaded:
>   Provider Version Architecture File
>   ubuntu   20.04   arm64        /tmp/data/cache/by_location/19db8c6d910a3f2660c4109ffb85d73d43e5cdf2/ubuntu-20.04-server-cloudimg-arm64.img
> 
> 3. Run the tests:
> 
>   $ ./tests/venv/bin/avocado run -t arch:aarch64,accel:tcg --keep-tmp on -- tests/acceptance/boot_linux.py{,,,,}
>   JOB ID     : 92a1bdbb5e933e6dff8b882808a191f1de3c2600
>   JOB LOG    : /home/cleber/avocado/job-results/job-2020-08-06T12.13-92a1bdb/job.log
>    (1/5) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (341.40 s)
>    (2/5) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (345.82 s)
>    (3/5) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (335.91 s)
>    (4/5) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (320.32 s)
>    (5/5) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (319.79 s)
>   RESULTS    : PASS 5 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
>   JOB TIME   : 1663.92 s
>