[v9,2/3] Acceptance test: add "boot_linux" tests

This acceptance test, validates that a full blown Linux guest can
successfully boot in QEMU.  In this specific case, the guest chosen is
Fedora version 31.

 * x86_64, pc-i440fx and pc-q35 machine types, with TCG and KVM as
   accelerators

 * aarch64 and virt machine type, with TCG and KVM as accelerators

 * ppc64 and pseries machine type with TCG as accelerator

 * s390x and s390-ccw-virtio machine type with TCG as accelerator

The Avocado vmimage utils library is used to download and cache the
Linux guest images, and from those images a snapshot image is created
and given to QEMU.  If a qemu-img binary is available in the build
directory, it's used to create the snapshot image, so that matching
qemu-system-* and qemu-img are used in the same test run.  If qemu-img
is not available in the build tree, one is attempted to be found
installed system-wide (in the $PATH).  If qemu-img is not found in the
build dir or in the $PATH, the test is canceled.

The method for checking the successful boot is based on "cloudinit"
and its "phone home" feature.  The guest is given an ISO image with
the location of the phone home server, and the information to post
(the instance ID).  Upon receiving the correct information, from the
guest, the test is considered to have PASSed.

This test is currently limited to user mode networking only, and
instructs the guest to connect to the "router" address that is hard
coded in QEMU.

To create the cloudinit ISO image that will be used to configure the
guest, the pycdlib library is also required and has been added as
requirement to the virtual environment created by "check-venv".

The console output is read by a separate thread, by means of the
Avocado datadrainer utility module.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 .travis.yml                    |   2 +-
 tests/acceptance/boot_linux.py | 215 +++++++++++++++++++++++++++++++++
 tests/requirements.txt         |   3 +-
 3 files changed, 218 insertions(+), 2 deletions(-)
 create mode 100644 tests/acceptance/boot_linux.py

On 2/19/20 11:06 PM, Cleber Rosa wrote:
> This acceptance test, validates that a full blown Linux guest can
> successfully boot in QEMU.  In this specific case, the guest chosen is
> Fedora version 31.
>
>   * x86_64, pc-i440fx and pc-q35 machine types, with TCG and KVM as
>     accelerators
>
>   * aarch64 and virt machine type, with TCG and KVM as accelerators
>
>   * ppc64 and pseries machine type with TCG as accelerator
>
>   * s390x and s390-ccw-virtio machine type with TCG as accelerator
>
> The Avocado vmimage utils library is used to download and cache the
> Linux guest images, and from those images a snapshot image is created
> and given to QEMU.  If a qemu-img binary is available in the build
> directory, it's used to create the snapshot image, so that matching
> qemu-system-* and qemu-img are used in the same test run.  If qemu-img
> is not available in the build tree, one is attempted to be found
> installed system-wide (in the $PATH).  If qemu-img is not found in the
> build dir or in the $PATH, the test is canceled.
>
> The method for checking the successful boot is based on "cloudinit"
> and its "phone home" feature.  The guest is given an ISO image with
> the location of the phone home server, and the information to post
> (the instance ID).  Upon receiving the correct information, from the
> guest, the test is considered to have PASSed.
>
> This test is currently limited to user mode networking only, and
> instructs the guest to connect to the "router" address that is hard
> coded in QEMU.
>
> To create the cloudinit ISO image that will be used to configure the
> guest, the pycdlib library is also required and has been added as
> requirement to the virtual environment created by "check-venv".
>
> The console output is read by a separate thread, by means of the
> Avocado datadrainer utility module.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>   .travis.yml                    |   2 +-
>   tests/acceptance/boot_linux.py | 215 +++++++++++++++++++++++++++++++++
>   tests/requirements.txt         |   3 +-
>   3 files changed, 218 insertions(+), 2 deletions(-)
>   create mode 100644 tests/acceptance/boot_linux.py
>
> diff --git a/.travis.yml b/.travis.yml
> index 5887055951..0c54cdf40f 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -313,7 +313,7 @@ matrix:
>       # Acceptance (Functional) tests
>       - name: "GCC check-acceptance"
>         env:
> -        - CONFIG="--target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
> +        - CONFIG="--enable-tools --target-list=aarch64-softmmu,alpha-softmmu,arm-softmmu,m68k-softmmu,microblaze-softmmu,mips-softmmu,mips64el-softmmu,nios2-softmmu,or1k-softmmu,ppc-softmmu,ppc64-softmmu,s390x-softmmu,sparc-softmmu,x86_64-softmmu,xtensa-softmmu"
>           - TEST_CMD="make check-acceptance"
>         after_script:
>           - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in r["tests"] if t["status"] not in ("PASS", "SKIP")]' | xargs cat
> diff --git a/tests/acceptance/boot_linux.py b/tests/acceptance/boot_linux.py
> new file mode 100644
> index 0000000000..6787e79aea
> --- /dev/null
> +++ b/tests/acceptance/boot_linux.py
> @@ -0,0 +1,215 @@
> +# Functional test that boots a complete Linux system via a cloud image
> +#
> +# Copyright (c) 2018-2020 Red Hat, Inc.
> +#
> +# Author:
> +#  Cleber Rosa <crosa@redhat.com>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or
> +# later.  See the COPYING file in the top-level directory.
> +
> +import os
> +
> +from avocado_qemu import Test, BUILD_DIR
> +
> +from qemu.accel import kvm_available
> +from qemu.accel import tcg_available
> +
> +from avocado.utils import cloudinit
> +from avocado.utils import network
> +from avocado.utils import vmimage
> +from avocado.utils import datadrainer
> +
> +ACCEL_NOT_AVAILABLE_FMT = "%s accelerator does not seem to be available"
> +KVM_NOT_AVAILABLE = ACCEL_NOT_AVAILABLE_FMT % "KVM"
> +TCG_NOT_AVAILABLE = ACCEL_NOT_AVAILABLE_FMT % "TCG"
> +
> +
> +class BootLinux(Test):
> +    """
> +    Boots a Linux system, checking for a successful initialization
> +    """
> +
> +    timeout = 900
> +    chksum = None
> +
> +    def setUp(self):
> +        super(BootLinux, self).setUp()
> +        self.prepare_boot()
> +        self.vm.add_args('-smp', '2')
> +        self.vm.add_args('-m', '1024')
> +        self.vm.add_args('-drive', 'file=%s' % self.boot.path)
> +        self.prepare_cloudinit()
> +
> +    def prepare_boot(self):
> +        self.log.info('Downloading/preparing boot image')
> +        # Fedora 31 only provides ppc64le images
> +        image_arch = self.arch
> +        if image_arch == 'ppc64':
> +            image_arch = 'ppc64le'
> +        # If qemu-img has been built, use it, otherwise the system wide one
> +        # will be used.  If none is available, the test will cancel.
> +        qemu_img = os.path.join(BUILD_DIR, 'qemu-img')
> +        if os.path.exists(qemu_img):
> +            vmimage.QEMU_IMG = qemu_img
> +        try:
> +            self.boot = vmimage.get(
> +                'fedora', arch=image_arch, version='31',
> +                checksum=self.chksum,
> +                algorithm='sha256',
> +                cache_dir=self.cache_dirs[0],
> +                snapshot_dir=self.workdir)
> +        except:
> +            self.cancel('Failed to download/prepare boot image')
> +
> +    def prepare_cloudinit(self):
> +        self.log.info('Preparing cloudinit image')
> +        try:
> +            cloudinit_iso = os.path.join(self.workdir, 'cloudinit.iso')
> +            self.phone_home_port = network.find_free_port()
> +            cloudinit.iso(cloudinit_iso, self.name,
> +                          username='root',
> +                          password='password',
> +                          # QEMU's hard coded usermode router address
> +                          phone_home_host='10.0.2.2',
> +                          phone_home_port=self.phone_home_port)
> +            self.vm.add_args('-drive', 'file=%s,format=raw' % cloudinit_iso)
> +        except Exception:
> +            self.cancel('Failed to prepared cloudinit image')
> +
> +    def launch_and_wait(self):
> +        self.vm.set_console()
> +        self.vm.launch()
> +        console_drainer = datadrainer.LineLogger(self.vm.console_socket.fileno(),
> +                                                 logger=self.log.getChild('console'))
> +        console_drainer.start()
> +        self.log.info('VM launched, waiting for boot confirmation from guest')
> +        cloudinit.wait_for_phone_home(('0.0.0.0', self.phone_home_port), self.name)
> +
> +
> +class BootLinuxX8664(BootLinux):
> +    """
> +    :avocado: tags=arch:x86_64
> +    """
> +
> +    chksum = 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0'
> +
> +    def test_pc_i440fx_tcg(self):
> +        """
> +        :avocado: tags=machine:pc
> +        :avocado: tags=machine:pc-i440fx
> +        :avocado: tags=accel:tcg
> +        """
> +        if not tcg_available(self.qemu_bin):
> +            self.cancel(TCG_NOT_AVAILABLE)
> +        self.vm.add_args("-accel", "tcg")
> +        self.launch_and_wait()
> +
> +    def test_pc_i440fx_kvm(self):
> +        """
> +        :avocado: tags=machine:pc
> +        :avocado: tags=machine:pc-i440fx
> +        :avocado: tags=accel:kvm
> +        """
> +        if not kvm_available(self.arch, self.qemu_bin):
> +            self.cancel(KVM_NOT_AVAILABLE)
> +        self.vm.add_args("-accel", "kvm")
> +        self.launch_and_wait()
> +
> +    def test_pc_q35_tcg(self):
> +        """
> +        :avocado: tags=machine:pc
> +        :avocado: tags=machine:pc-q35
> +        :avocado: tags=accel:tcg
> +        """
> +        if not tcg_available(self.qemu_bin):
> +            self.cancel(TCG_NOT_AVAILABLE)
> +        self.vm.add_args("-accel", "tcg")
> +        self.launch_and_wait()
> +
> +    def test_pc_q35_kvm(self):
> +        """
> +        :avocado: tags=machine:q35
> +        :avocado: tags=machine:pc-q35
> +        :avocado: tags=accel:kvm
> +        """
> +        if not kvm_available(self.arch, self.qemu_bin):
> +            self.cancel(KVM_NOT_AVAILABLE)
> +        self.vm.add_args("-accel", "kvm")
> +        self.launch_and_wait()
> +
> +
> +class BootLinuxAarch64(BootLinux):
> +    """
> +    :avocado: tags=arch:aarch64
> +    :avocado: tags=machine:virt
> +    """
> +
> +    chksum = '1e18d9c0cf734940c4b5d5ec592facaed2af0ad0329383d5639c997fdf16fe49'
> +
> +    def add_common_args(self):
> +        self.vm.add_args('-bios',
> +                         os.path.join(BUILD_DIR, 'pc-bios',
> +                                      'edk2-aarch64-code.fd'))
> +        self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0')
> +        self.vm.add_args('-object', 'rng-random,id=rng0,filename=/dev/urandom')
> +
> +    def test_virt_tcg(self):
> +        """
> +        :avocado: tags=accel:tcg
> +        :avocado: tags=cpu:cortex-a53
> +        """
> +        if not tcg_available(self.qemu_bin):
> +            self.cancel(TCG_NOT_AVAILABLE)
> +        self.vm.add_args("-accel", "tcg")
> +        self.vm.add_args('-cpu', 'cortex-a53')
> +        self.add_common_args()
> +        self.launch_and_wait()
> +
> +    def test_virt_kvm(self):
> +        """
> +        :avocado: tags=accel:kvm
> +        :avocado: tags=cpu:host
> +        """
> +        if not kvm_available(self.arch, self.qemu_bin):
> +            self.cancel(KVM_NOT_AVAILABLE)
> +        self.vm.add_args("-accel", "kvm")
> +        self.vm.add_args("-cpu", "host")
> +        self.add_common_args()
> +        self.launch_and_wait()


For aarch64 tests it seems '-cpu max' is the best choice. See in 
https://www.mail-archive.com/qemu-devel@nongnu.org/msg672755.html


> +
> +
> +class BootLinuxPPC64(BootLinux):
> +    """
> +    :avocado: tags=arch:ppc64
> +    """
> +
> +    chksum = '7c3528b85a3df4b2306e892199a9e1e43f991c506f2cc390dc4efa2026ad2f58'
> +
> +    def test_pseries_tcg(self):
> +        """
> +        :avocado: tags=machine:pseries
> +        :avocado: tags=accel:tcg
> +        """
> +        if not tcg_available(self.qemu_bin):
> +            self.cancel(TCG_NOT_AVAILABLE)
> +        self.vm.add_args("-accel", "tcg")
> +        self.launch_and_wait()
> +
> +
> +class BootLinuxS390X(BootLinux):
> +    """
> +    :avocado: tags=arch:s390x
> +    """
> +
> +    chksum = '4caaab5a434fd4d1079149a072fdc7891e354f834d355069ca982fdcaf5a122d'
> +
> +    def test_s390_ccw_virtio_tcg(self):
> +        """
> +        :avocado: tags=machine:s390-ccw-virtio
> +        :avocado: tags=accel:tcg
> +        """
> +        if not tcg_available(self.qemu_bin):
> +            self.cancel(TCG_NOT_AVAILABLE)
> +        self.vm.add_args("-accel", "tcg")
> +        self.launch_and_wait()
> diff --git a/tests/requirements.txt b/tests/requirements.txt
> index a2a587223a..a3b5fe4159 100644
> --- a/tests/requirements.txt
> +++ b/tests/requirements.txt
> @@ -1,4 +1,5 @@
>   # Add Python module requirements, one per line, to be installed
>   # in the tests/venv Python virtual environment. For more info,
>   # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> -avocado-framework==72.0
> +avocado-framework==74.0
> +pycdlib==1.9.0


Tested on x86_64 machine, the tests behave correctly with following 
configurations:

1. ---target-list=x86_64-softmmu --disable-tcg

2. ---target-list=x86_64-softmmu --disable-kvm

3. --target-list=x86_64-softmmu,aarch64-softmmu,ppc64-softmmu,s390x-softmmu

But failed if:

3. ---target-list=x86_64-softmmu --disable-tools.

And the error message is:

(01/32) 
tests/acceptance/boot_linux.py:BootLinuxX8664.test_pc_i440fx_tcg: ERROR: 
Command 'qemu-img' could not be found in any of the PATH dirs: 
['/usr/bin', '/usr/sbin', '/usr/lib64/ccache', '/bin', '/root/bin', 
'/sbin', '/usr/local/sbin', '/usr/local/bin', '/usr/libexec'] (1.58 s)

Thanks!

- Wainer

On Thu, Feb 20, 2020 at 01:49:40PM -0300, Wainer dos Santos Moschetta wrote:
> On 2/19/20 11:06 PM, Cleber Rosa wrote:
> > +
> > +    def test_virt_tcg(self):
> > +        """
> > +        :avocado: tags=accel:tcg
> > +        :avocado: tags=cpu:cortex-a53
> > +        """
> > +        if not tcg_available(self.qemu_bin):
> > +            self.cancel(TCG_NOT_AVAILABLE)
> > +        self.vm.add_args("-accel", "tcg")
> > +        self.vm.add_args('-cpu', 'cortex-a53')
> > +        self.add_common_args()
> > +        self.launch_and_wait()
> > +
> > +    def test_virt_kvm(self):
> > +        """
> > +        :avocado: tags=accel:kvm
> > +        :avocado: tags=cpu:host
> > +        """
> > +        if not kvm_available(self.arch, self.qemu_bin):
> > +            self.cancel(KVM_NOT_AVAILABLE)
> > +        self.vm.add_args("-accel", "kvm")
> > +        self.vm.add_args("-cpu", "host")
> > +        self.add_common_args()
> > +        self.launch_and_wait()
> 
> 
> For aarch64 tests it seems '-cpu max' is the best choice. See in
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg672755.html
> 
>

+drew

Thanks for pointing that out.  There's one thing, though, which I can
not agree on.  And I know that Drew is an expert on the matter, which
makes it harder to disagree on... but, I've got results which clearly
indicate that *not using* the gic-version machine parameter still gets
me KVM:

./tests/venv/bin/avocado run tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_kvm
JOB ID     : 21a394b884b474ceee0a045b3e74f98da0aee023
JOB LOG    : /home/cleber/avocado/job-results/job-2020-02-20T14.28-21a394b/job.log
 (1/1) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_kvm: PASS (35.10 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 35.87 s

VM launch command:
   aarch64-softmmu/qemu-system-aarch64
   -display none
   -vga none
   -chardev socket,id=mon,path=/var/tmp/tmpntz_r_h7/qemu-18331-monitor.sock
   -mon chardev=mon,mode=control
   -machine virt
   -chardev socket,id=console,path=/var/tmp/tmpntz_r_h7/qemu-18331-console.sock,server,nowait
   -serial chardev:console
   -smp 2
   -m 1024
   -drive file=/var/tmp/avocado_u9jm04di/avocado_job_28oth9kk/1-tests_acceptance_boot_linux.py_BootLinuxAarch64.test_virt_kvm/Fedora-Cloud-Base-31-1.9.aarch64-05265df5.qcow2 -drive file=/var/tmp/avocado_u9jm04di/avocado_job_28oth9kk/1-tests_acceptance_boot_linux.py_BootLinuxAarch64.test_virt_kvm/cloudinit.iso,format=raw
   -accel kvm
   -cpu host
   -bios /home/cleber/build/qemu/pc-bios/edk2-aarch64-code.fd
   -device virtio-rng-pci,rng=rng0
   -object rng-random,id=rng0,filename=/dev/urandom

Guest boot messages shows:
[    1.538955] systemd[1]: Detected virtualization kvm.
[    1.539828] systemd[1]: Detected architecture arm64.

This is in contrast with:

./tests/venv/bin/avocado run tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg 
JOB ID     : 90b9412f700e52428b59e97719496c30b4f54435
JOB LOG    : /home/cleber/avocado/job-results/job-2020-02-20T14.32-90b9412/job.log
 (1/1) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (581.14 s)
RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 581.93 s

VM launch command:
   aarch64-softmmu/qemu-system-aarch64
   -display none
   -vga none
   -chardev socket,id=mon,path=/var/tmp/tmpa6i4livg/qemu-18498-monitor.sock
   -mon chardev=mon,mode=control
   -machine virt
   -chardev socket,id=console,path=/var/tmp/tmpa6i4livg/qemu-18498-console.sock,server,nowait
   -serial chardev:console
   -smp 2
   -m 1024
   -drive file=/var/tmp/avocado_slcj2x9e/avocado_job_x5u__309/1-tests_acceptance_boot_linux.py_BootLinuxAarch64.test_virt_tcg/Fedora-Cloud-Base-31-1.9.aarch64-5b006a2f.qcow2 -drive file=/var/tmp/avocado_slcj2x9e/avocado_job_x5u__309/1-tests_acceptance_boot_linux.py_BootLinuxAarch64.test_virt_tcg/cloudinit.iso,format=raw
   -accel tcg
   -cpu cortex-a53
   -bios /home/cleber/build/qemu/pc-bios/edk2-aarch64-code.fd
   -device virtio-rng-pci,rng=rng0
   -object rng-random,id=rng0,filename=/dev/urandom'

Guest boot messages shows:
[   28.606310] systemd[1]: Detected virtualization qemu.
[   28.607861] systemd[1]: Detected architecture arm64.

And with regards to the CPU type, IIRC, "max" will fallback to the
best CPU on TCG mode.  As a general best practice in testing, I'd
rather not have this dynamic aspect where we can avoid it.  Looks like
with TCG we can set it to one CPU and validate that the guests work on
that configuration.

IIUC, by using either "-cpu host" or "-cpu max" for KVM, we may end up
having the same test PASS or FAIL because of the (dynamic) host CPU.
That's not ideal for testing purposes, but given it's outside of our
control, do best we can do is keep track of the host CPU (via Avocado's
sysinfo collection).

Also, I've used the same CPU model that has been used on
boot_linux_console.py:BootLinuxConsole.test_aarch64_virt, which may be
a plus.

> > +
> > +
> > +class BootLinuxPPC64(BootLinux):
> > +    """
> > +    :avocado: tags=arch:ppc64
> > +    """
> > +
> > +    chksum = '7c3528b85a3df4b2306e892199a9e1e43f991c506f2cc390dc4efa2026ad2f58'
> > +
> > +    def test_pseries_tcg(self):
> > +        """
> > +        :avocado: tags=machine:pseries
> > +        :avocado: tags=accel:tcg
> > +        """
> > +        if not tcg_available(self.qemu_bin):
> > +            self.cancel(TCG_NOT_AVAILABLE)
> > +        self.vm.add_args("-accel", "tcg")
> > +        self.launch_and_wait()
> > +
> > +
> > +class BootLinuxS390X(BootLinux):
> > +    """
> > +    :avocado: tags=arch:s390x
> > +    """
> > +
> > +    chksum = '4caaab5a434fd4d1079149a072fdc7891e354f834d355069ca982fdcaf5a122d'
> > +
> > +    def test_s390_ccw_virtio_tcg(self):
> > +        """
> > +        :avocado: tags=machine:s390-ccw-virtio
> > +        :avocado: tags=accel:tcg
> > +        """
> > +        if not tcg_available(self.qemu_bin):
> > +            self.cancel(TCG_NOT_AVAILABLE)
> > +        self.vm.add_args("-accel", "tcg")
> > +        self.launch_and_wait()
> > diff --git a/tests/requirements.txt b/tests/requirements.txt
> > index a2a587223a..a3b5fe4159 100644
> > --- a/tests/requirements.txt
> > +++ b/tests/requirements.txt
> > @@ -1,4 +1,5 @@
> >   # Add Python module requirements, one per line, to be installed
> >   # in the tests/venv Python virtual environment. For more info,
> >   # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> > -avocado-framework==72.0
> > +avocado-framework==74.0
> > +pycdlib==1.9.0
> 
> 
> Tested on x86_64 machine, the tests behave correctly with following
> configurations:
> 
> 1. ---target-list=x86_64-softmmu --disable-tcg
> 
> 2. ---target-list=x86_64-softmmu --disable-kvm
> 
> 3. --target-list=x86_64-softmmu,aarch64-softmmu,ppc64-softmmu,s390x-softmmu
> 
> But failed if:
> 
> 3. ---target-list=x86_64-softmmu --disable-tools.
> 
> And the error message is:
> 
> (01/32) tests/acceptance/boot_linux.py:BootLinuxX8664.test_pc_i440fx_tcg:
> ERROR: Command 'qemu-img' could not be found in any of the PATH dirs:
> ['/usr/bin', '/usr/sbin', '/usr/lib64/ccache', '/bin', '/root/bin', '/sbin',
> '/usr/local/sbin', '/usr/local/bin', '/usr/libexec'] (1.58 s)

This is what I call comprehensive testing! Thanks!

It looks like I was not paying that much attention to what happens during
the "self.boot.path" attribute access, and had left it "unprotected" in
the QEMU command line arguments assignment.  But there's where a lazy
snapshot image creation is attempted.

I've adjusted the code and will have that on a v10.

> 
> Thanks!
> 
> - Wainer
> 
> 

Thanks a lot for the review!
- Cleber.

On Thu, Feb 20, 2020 at 02:52:45PM -0500, Cleber Rosa wrote:
> On Thu, Feb 20, 2020 at 01:49:40PM -0300, Wainer dos Santos Moschetta wrote:
> > On 2/19/20 11:06 PM, Cleber Rosa wrote:
> > > +
> > > +    def test_virt_tcg(self):
> > > +        """
> > > +        :avocado: tags=accel:tcg
> > > +        :avocado: tags=cpu:cortex-a53
> > > +        """
> > > +        if not tcg_available(self.qemu_bin):
> > > +            self.cancel(TCG_NOT_AVAILABLE)
> > > +        self.vm.add_args("-accel", "tcg")
> > > +        self.vm.add_args('-cpu', 'cortex-a53')

Should use '-cpu max'. gic-version will be '2' by default, which is good
for tcg, but I would probably add an explicit '-machine gic-version=2'
anyway.

> > > +        self.add_common_args()
> > > +        self.launch_and_wait()
> > > +
> > > +    def test_virt_kvm(self):
> > > +        """
> > > +        :avocado: tags=accel:kvm
> > > +        :avocado: tags=cpu:host
> > > +        """
> > > +        if not kvm_available(self.arch, self.qemu_bin):
> > > +            self.cancel(KVM_NOT_AVAILABLE)
> > > +        self.vm.add_args("-accel", "kvm")
> > > +        self.vm.add_args("-cpu", "host")

kvm needs '-machine gic-version=max' and could use '-cpu max' too,
because, for kvm, CPU::max == CPU::host

> > > +        self.add_common_args()
> > > +        self.launch_and_wait()
> > 
> > 
> > For aarch64 tests it seems '-cpu max' is the best choice. See in
> > https://www.mail-archive.com/qemu-devel@nongnu.org/msg672755.html
> > 
> >
> 
> +drew
> 
> Thanks for pointing that out.  There's one thing, though, which I can
> not agree on.  And I know that Drew is an expert on the matter, which
> makes it harder to disagree on... but, I've got results which clearly
> indicate that *not using* the gic-version machine parameter still gets
> me KVM:

The gic-version machine property is completely independent of whether kvm
or tcg is selected. However, if you select kvm and a gic-version that is
incompatible with the host then the guest will not start. If gic-version
is not specified it defaults to '2'. Below the kvm guest started
successfully because it happened to be started on a host with gicv2 or
on a host with gicv3 that supports gicv2-on-v3 (which is an optional
feature that doesn't appear to be getting implemented in modern AArch64
servers). The test below would have failed to start the guest on a host
with only gicv3.

When kvm is in use, one must use gic-version=max in order to automatically
select the latest host-compatible gic version, or the guest will not start
on all hosts.

> 
> ./tests/venv/bin/avocado run tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_kvm
> JOB ID     : 21a394b884b474ceee0a045b3e74f98da0aee023
> JOB LOG    : /home/cleber/avocado/job-results/job-2020-02-20T14.28-21a394b/job.log
>  (1/1) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_kvm: PASS (35.10 s)
> RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
> JOB TIME   : 35.87 s
> 
> VM launch command:
>    aarch64-softmmu/qemu-system-aarch64
>    -display none
>    -vga none
>    -chardev socket,id=mon,path=/var/tmp/tmpntz_r_h7/qemu-18331-monitor.sock
>    -mon chardev=mon,mode=control
>    -machine virt
>    -chardev socket,id=console,path=/var/tmp/tmpntz_r_h7/qemu-18331-console.sock,server,nowait
>    -serial chardev:console
>    -smp 2
>    -m 1024
>    -drive file=/var/tmp/avocado_u9jm04di/avocado_job_28oth9kk/1-tests_acceptance_boot_linux.py_BootLinuxAarch64.test_virt_kvm/Fedora-Cloud-Base-31-1.9.aarch64-05265df5.qcow2 -drive file=/var/tmp/avocado_u9jm04di/avocado_job_28oth9kk/1-tests_acceptance_boot_linux.py_BootLinuxAarch64.test_virt_kvm/cloudinit.iso,format=raw
>    -accel kvm
>    -cpu host
>    -bios /home/cleber/build/qemu/pc-bios/edk2-aarch64-code.fd
>    -device virtio-rng-pci,rng=rng0
>    -object rng-random,id=rng0,filename=/dev/urandom
> 
> Guest boot messages shows:
> [    1.538955] systemd[1]: Detected virtualization kvm.
> [    1.539828] systemd[1]: Detected architecture arm64.
> 
> This is in contrast with:
> 
> ./tests/venv/bin/avocado run tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg 
> JOB ID     : 90b9412f700e52428b59e97719496c30b4f54435
> JOB LOG    : /home/cleber/avocado/job-results/job-2020-02-20T14.32-90b9412/job.log
>  (1/1) tests/acceptance/boot_linux.py:BootLinuxAarch64.test_virt_tcg: PASS (581.14 s)
> RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
> JOB TIME   : 581.93 s
> 
> VM launch command:
>    aarch64-softmmu/qemu-system-aarch64
>    -display none
>    -vga none
>    -chardev socket,id=mon,path=/var/tmp/tmpa6i4livg/qemu-18498-monitor.sock
>    -mon chardev=mon,mode=control
>    -machine virt
>    -chardev socket,id=console,path=/var/tmp/tmpa6i4livg/qemu-18498-console.sock,server,nowait
>    -serial chardev:console
>    -smp 2
>    -m 1024
>    -drive file=/var/tmp/avocado_slcj2x9e/avocado_job_x5u__309/1-tests_acceptance_boot_linux.py_BootLinuxAarch64.test_virt_tcg/Fedora-Cloud-Base-31-1.9.aarch64-5b006a2f.qcow2 -drive file=/var/tmp/avocado_slcj2x9e/avocado_job_x5u__309/1-tests_acceptance_boot_linux.py_BootLinuxAarch64.test_virt_tcg/cloudinit.iso,format=raw
>    -accel tcg
>    -cpu cortex-a53
>    -bios /home/cleber/build/qemu/pc-bios/edk2-aarch64-code.fd
>    -device virtio-rng-pci,rng=rng0
>    -object rng-random,id=rng0,filename=/dev/urandom'
> 
> Guest boot messages shows:
> [   28.606310] systemd[1]: Detected virtualization qemu.
> [   28.607861] systemd[1]: Detected architecture arm64.
> 
> And with regards to the CPU type, IIRC, "max" will fallback to the
> best CPU on TCG mode.

No, it doesn't fallback. It implements the cortex-a57 and enables all
optional CPU features. Why was the cortex-a53 chosen for the tests?

> As a general best practice in testing, I'd
> rather not have this dynamic aspect where we can avoid it.  Looks like
> with TCG we can set it to one CPU and validate that the guests work on
> that configuration.

The only dynamic aspect is that as new CPU features are implemented
they'll get enabled. Personally I'd prefer the tests run with the latest
code enabled in order to find the latest bugs.

> 
> IIUC, by using either "-cpu host" or "-cpu max" for KVM, we may end up
> having the same test PASS or FAIL because of the (dynamic) host CPU.
> That's not ideal for testing purposes, but given it's outside of our
> control, do best we can do is keep track of the host CPU (via Avocado's
> sysinfo collection).

Right. AArch64 KVM must use '-cpu host' (or equivalently '-cpu max'). That
can't be changed. Capturing the host CPU type is a good idea.

> 
> Also, I've used the same CPU model that has been used on
> boot_linux_console.py:BootLinuxConsole.test_aarch64_virt, which may be
> a plus.

Is that the cortex-a53? Was some justification given for its use there?
If not, then maybe it should be changed there too.

Thanks,
drew