| Message ID | 20181009232607.15521-7-crosa@redhat.com |
|---|---|
| State | New |
| Headers | show |
| Series | Acceptance Tests: basic architecture support | expand |
Hi Cleber, On 10/10/2018 01:26, Cleber Rosa wrote: > One of the Avocado features relevant to virtualization testing is the > ability to reuse tests in different scenarios, known as variants. > This adds a JSON based variants file, that can be used to run most > tests in a number of different architectures. It can be run with: > > $ avocado run \ > --json-variants-load=tests/acceptance/variants/arch.json \ > --filter-by-tags='-x86_64' -- tests/acceptance/ > > Currently this covers 5 architectures, resulting in the execution > of 25 different combinations. Can you provide here the command you used to generate this JSON? Thanks, Phil. > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > --- > tests/acceptance/variants/arch.json | 92 +++++++++++++++++++++++++++++ > 1 file changed, 92 insertions(+) > create mode 100644 tests/acceptance/variants/arch.json > > diff --git a/tests/acceptance/variants/arch.json b/tests/acceptance/variants/arch.json > new file mode 100644 > index 0000000000..64ff6ad278 > --- /dev/null > +++ b/tests/acceptance/variants/arch.json > @@ -0,0 +1,92 @@ > +[ > + { > + "paths": [ > + "/run/*" > + ], > + "variant": [ > + [ > + "/run/aarch64", > + [ > + [ > + "/run/aarch64", > + "arch", > + "aarch64" > + ] > + ] > + ] > + ], > + "variant_id": "aarch64" > + }, > + { > + "paths": [ > + "/run/*" > + ], > + "variant": [ > + [ > + "/run/ppc", > + [ > + [ > + "/run/ppc", > + "arch", > + "ppc" > + ] > + ] > + ] > + ], > + "variant_id": "ppc" > + }, > + { > + "paths": [ > + "/run/*" > + ], > + "variant": [ > + [ > + "/run/ppc64", > + [ > + [ > + "/run/ppc64", > + "arch", > + "ppc64" > + ] > + ] > + ] > + ], > + "variant_id": "ppc64" > + }, > + { > + "paths": [ > + "/run/*" > + ], > + "variant": [ > + [ > + "/run/s390x", > + [ > + [ > + "/run/s390x", > + "arch", > + "s390x" > + ] > + ] > + ] > + ], > + "variant_id": "s390x" > + }, > + { > + "paths": [ > + "/run/*" > + ], > + "variant": [ > + [ > + "/run/x86_64", > + [ > + [ > + "/run/x86_64", > + "arch", > + "x86_64" > + ] > + ] > + ] > + ], > + "variant_id": "x86_64" > + } > +] >
On 10/10/2018 01:26, Cleber Rosa wrote: > One of the Avocado features relevant to virtualization testing is the > ability to reuse tests in different scenarios, known as variants. > This adds a JSON based variants file, that can be used to run most > tests in a number of different architectures. It can be run with: > > $ avocado run \ > --json-variants-load=tests/acceptance/variants/arch.json \ > --filter-by-tags='-x86_64' -- tests/acceptance/ > > Currently this covers 5 architectures, resulting in the execution > of 25 different combinations. IMHO we should add this file with all supported archs, and CANCEL a job if there is no preferred machine in Qemu::MACHINE_TYPES. > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > --- > tests/acceptance/variants/arch.json | 92 +++++++++++++++++++++++++++++ > 1 file changed, 92 insertions(+) > create mode 100644 tests/acceptance/variants/arch.json
On 10/10/18 6:59 AM, Philippe Mathieu-Daudé wrote: > On 10/10/2018 01:26, Cleber Rosa wrote: >> One of the Avocado features relevant to virtualization testing is the >> ability to reuse tests in different scenarios, known as variants. >> This adds a JSON based variants file, that can be used to run most >> tests in a number of different architectures. It can be run with: >> >> $ avocado run \ >> --json-variants-load=tests/acceptance/variants/arch.json \ >> --filter-by-tags='-x86_64' -- tests/acceptance/ >> >> Currently this covers 5 architectures, resulting in the execution >> of 25 different combinations. > > IMHO we should add this file with all supported archs, and CANCEL a job > if there is no preferred machine in Qemu::MACHINE_TYPES. > While your proposal works, I see two issues: 1) It defines a policy, which will affect all tests, existing and new - even for tests that don't launch a VM and don't care about the arch. 2) CANCEL is mostly silent, and the amount of noise in test results would be quite large. So, the real problem is that there are other issues when making basic use of QEMUMachine with other archs. The idea is to incrementally test other archs, make the modifications necessary to scripts/qemu.py, and end up with all architectures covered. What do you think? - Cleber. >> >> Signed-off-by: Cleber Rosa <crosa@redhat.com> >> --- >> tests/acceptance/variants/arch.json | 92 +++++++++++++++++++++++++++++ >> 1 file changed, 92 insertions(+) >> create mode 100644 tests/acceptance/variants/arch.json
On 10/10/18 6:51 AM, Philippe Mathieu-Daudé wrote: > Hi Cleber, > > On 10/10/2018 01:26, Cleber Rosa wrote: >> One of the Avocado features relevant to virtualization testing is the >> ability to reuse tests in different scenarios, known as variants. >> This adds a JSON based variants file, that can be used to run most >> tests in a number of different architectures. It can be run with: >> >> $ avocado run \ >> --json-variants-load=tests/acceptance/variants/arch.json \ >> --filter-by-tags='-x86_64' -- tests/acceptance/ >> >> Currently this covers 5 architectures, resulting in the execution >> of 25 different combinations. > > Can you provide here the command you used to generate this JSON? > Yep, there are many ways to do it, one is to have the varianter-yaml-to-mux plugin installed, and create a YAML file such as: ---- archs.yaml START ----- !mux x86_64: arch: x86_64 aarch64: arch: aarch64 ppc64: arch: ppc64 ppc: arch: ppc s390x: arch: s390x ---- archs.yaml END ----- And run: $ avocado run -m archs.yaml -- /bin/true $ python -m json.tool ~/avocado/job-results/latest/jobdata/variants.json I did tweak the "variant_id" value of the resulting JSON file, because the hashes are somewhat specific to yaml-to-mux, and don't add anything to the meaning of the variants. - Cleber. > Thanks, > > Phil. > >> >> Signed-off-by: Cleber Rosa <crosa@redhat.com> >> --- >> tests/acceptance/variants/arch.json | 92 +++++++++++++++++++++++++++++ >> 1 file changed, 92 insertions(+) >> create mode 100644 tests/acceptance/variants/arch.json >> >> diff --git a/tests/acceptance/variants/arch.json b/tests/acceptance/variants/arch.json >> new file mode 100644 >> index 0000000000..64ff6ad278 >> --- /dev/null >> +++ b/tests/acceptance/variants/arch.json >> @@ -0,0 +1,92 @@ >> +[ >> + { >> + "paths": [ >> + "/run/*" >> + ], >> + "variant": [ >> + [ >> + "/run/aarch64", >> + [ >> + [ >> + "/run/aarch64", >> + "arch", >> + "aarch64" >> + ] >> + ] >> + ] >> + ], >> + "variant_id": "aarch64" >> + }, >> + { >> + "paths": [ >> + "/run/*" >> + ], >> + "variant": [ >> + [ >> + "/run/ppc", >> + [ >> + [ >> + "/run/ppc", >> + "arch", >> + "ppc" >> + ] >> + ] >> + ] >> + ], >> + "variant_id": "ppc" >> + }, >> + { >> + "paths": [ >> + "/run/*" >> + ], >> + "variant": [ >> + [ >> + "/run/ppc64", >> + [ >> + [ >> + "/run/ppc64", >> + "arch", >> + "ppc64" >> + ] >> + ] >> + ] >> + ], >> + "variant_id": "ppc64" >> + }, >> + { >> + "paths": [ >> + "/run/*" >> + ], >> + "variant": [ >> + [ >> + "/run/s390x", >> + [ >> + [ >> + "/run/s390x", >> + "arch", >> + "s390x" >> + ] >> + ] >> + ] >> + ], >> + "variant_id": "s390x" >> + }, >> + { >> + "paths": [ >> + "/run/*" >> + ], >> + "variant": [ >> + [ >> + "/run/x86_64", >> + [ >> + [ >> + "/run/x86_64", >> + "arch", >> + "x86_64" >> + ] >> + ] >> + ] >> + ], >> + "variant_id": "x86_64" >> + } >> +] >>
diff --git a/tests/acceptance/variants/arch.json b/tests/acceptance/variants/arch.json new file mode 100644 index 0000000000..64ff6ad278 --- /dev/null +++ b/tests/acceptance/variants/arch.json @@ -0,0 +1,92 @@ +[ + { + "paths": [ + "/run/*" + ], + "variant": [ + [ + "/run/aarch64", + [ + [ + "/run/aarch64", + "arch", + "aarch64" + ] + ] + ] + ], + "variant_id": "aarch64" + }, + { + "paths": [ + "/run/*" + ], + "variant": [ + [ + "/run/ppc", + [ + [ + "/run/ppc", + "arch", + "ppc" + ] + ] + ] + ], + "variant_id": "ppc" + }, + { + "paths": [ + "/run/*" + ], + "variant": [ + [ + "/run/ppc64", + [ + [ + "/run/ppc64", + "arch", + "ppc64" + ] + ] + ] + ], + "variant_id": "ppc64" + }, + { + "paths": [ + "/run/*" + ], + "variant": [ + [ + "/run/s390x", + [ + [ + "/run/s390x", + "arch", + "s390x" + ] + ] + ] + ], + "variant_id": "s390x" + }, + { + "paths": [ + "/run/*" + ], + "variant": [ + [ + "/run/x86_64", + [ + [ + "/run/x86_64", + "arch", + "x86_64" + ] + ] + ] + ], + "variant_id": "x86_64" + } +]
One of the Avocado features relevant to virtualization testing is the ability to reuse tests in different scenarios, known as variants. This adds a JSON based variants file, that can be used to run most tests in a number of different architectures. It can be run with: $ avocado run \ --json-variants-load=tests/acceptance/variants/arch.json \ --filter-by-tags='-x86_64' -- tests/acceptance/ Currently this covers 5 architectures, resulting in the execution of 25 different combinations. Signed-off-by: Cleber Rosa <crosa@redhat.com> --- tests/acceptance/variants/arch.json | 92 +++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+) create mode 100644 tests/acceptance/variants/arch.json