diff mbox series

[v4,11/35] tests/functional: Prepare the meson build system for the functional tests

Message ID 20240821082748.65853-12-thuth@redhat.com
State New
Headers show
Series Convert avocado tests to normal Python unittests | expand

Commit Message

Thomas Huth Aug. 21, 2024, 8:27 a.m. UTC
Provide a meson.build file for the upcoming python-based functional
tests, and add some wrapper glue targets to the tests/Makefile.include
file. We are going to use two "speed" modes for the functional tests:
The "quick" tests can be run at any time (i.e. also during "make check"),
while the "thorough" tests should only be run when running a
"make check-functional" test run (since these tests might download
additional assets from the internet).

The changes to the meson.build files are partly based on an earlier
patch by Ani Sinha.

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 tests/Makefile.include       | 11 ++++++
 tests/functional/meson.build | 66 ++++++++++++++++++++++++++++++++++++
 tests/meson.build            |  1 +
 3 files changed, 78 insertions(+)
 create mode 100644 tests/functional/meson.build

Comments

Philippe Mathieu-Daudé Aug. 21, 2024, 2:30 p.m. UTC | #1
On 21/8/24 10:27, Thomas Huth wrote:
> Provide a meson.build file for the upcoming python-based functional
> tests, and add some wrapper glue targets to the tests/Makefile.include
> file. We are going to use two "speed" modes for the functional tests:
> The "quick" tests can be run at any time (i.e. also during "make check"),
> while the "thorough" tests should only be run when running a
> "make check-functional" test run (since these tests might download
> additional assets from the internet).
> 
> The changes to the meson.build files are partly based on an earlier
> patch by Ani Sinha.
> 
> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>   tests/Makefile.include       | 11 ++++++
>   tests/functional/meson.build | 66 ++++++++++++++++++++++++++++++++++++
>   tests/meson.build            |  1 +
>   3 files changed, 78 insertions(+)
>   create mode 100644 tests/functional/meson.build
> 
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index fbb1a4b211..66c8cc3123 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -9,6 +9,8 @@ check-help:
>   	@echo "Individual test suites:"
>   	@echo " $(MAKE) check-qtest-TARGET       Run qtest tests for given target"
>   	@echo " $(MAKE) check-qtest              Run qtest tests"
> +	@echo " $(MAKE) check-functional         Run python-based functional tests"
> +	@echo " $(MAKE) check-functional-TARGET  Run functional tests for a given target"
>   	@echo " $(MAKE) check-unit               Run qobject tests"
>   	@echo " $(MAKE) check-qapi-schema        Run QAPI schema tests"
>   	@echo " $(MAKE) check-block              Run block tests"
> @@ -152,6 +154,15 @@ check-acceptance-deprecated-warning:
>   
>   check-acceptance: check-acceptance-deprecated-warning | check-avocado
>   
> +FUNCTIONAL_TARGETS=$(patsubst %-softmmu,check-functional-%, $(filter %-softmmu,$(TARGETS)))

This only handles system binaries.

> +.PHONY: $(FUNCTIONAL_TARGETS)
> +$(FUNCTIONAL_TARGETS):
> +	@$(MAKE) SPEED=thorough $(subst -functional,-func,$@)
> +
> +.PHONY: check-functional
> +check-functional:
> +	@$(MAKE) SPEED=thorough check-func check-func-quick
> +
>   # Consolidated targets
>   
>   .PHONY: check check-clean get-vm-images
> diff --git a/tests/functional/meson.build b/tests/functional/meson.build
> new file mode 100644
> index 0000000000..7fc9c4e513
> --- /dev/null
> +++ b/tests/functional/meson.build
> @@ -0,0 +1,66 @@
> +# QEMU functional tests:
> +# Tests that are put in the 'quick' category are run by default during
> +# 'make check'. Everything that should not be run during 'make check'
> +# (e.g. tests that fetch assets from the internet) should be put into
> +# the 'thorough' category instead.
> +
> +# Most tests run too slow with TCI enabled, so skip the functional tests there
> +if get_option('tcg_interpreter')
> +  subdir_done()
> +endif
> +
> +# Timeouts for individual tests that can be slow e.g. with debugging enabled
> +test_timeouts = {
> +}
> +
> +tests_generic = [
> +]
> +
> +tests_x86_64_quick = [

Hard to distinct system/user tests.

> +]
> +
> +tests_x86_64_thorough = [

It is a bit unfortunate to have to modify meson.build to add a test:
this triggers meson to regenerate build files. Good enough for now,
hopefully we will improve.

> +]
> +
> +foreach speed : ['quick', 'thorough']
> +  foreach dir : target_dirs
> +    if not dir.endswith('-softmmu')
> +      continue
> +    endif
> +
> +    target_base = dir.split('-')[0]
> +    test_emulator = emulators['qemu-system-' + target_base]
> +
> +    if speed == 'quick'
> +      suites = ['func-quick', 'func-' + target_base]
> +      target_tests = get_variable('tests_' + target_base + '_quick', []) + tests_generic
> +    else
> +      suites = ['func-' + speed, 'func-' + target_base + '-' + speed, speed]
> +      target_tests = get_variable('tests_' + target_base + '_' + speed, [])
> +    endif
> +
> +    test_deps = roms
> +    test_env = environment()
> +    if have_tools
> +      test_env.set('QEMU_TEST_QEMU_IMG', meson.global_build_root() / 'qemu-img')
> +      test_deps += [qemu_img]
> +    endif
> +    test_env.set('QEMU_TEST_QEMU_BINARY',
> +                 meson.global_build_root() / 'qemu-system-' + target_base)
> +    test_env.set('QEMU_BUILD_ROOT', meson.project_build_root())
> +    test_env.set('PYTHONPATH', meson.project_source_root() / 'python:' +
> +                               meson.current_source_dir())
> +
> +    foreach test : target_tests
> +      test('func-@0@/@1@'.format(target_base, test),
> +           python,
> +           depends: [test_deps, test_emulator, emulator_modules],
> +           env: test_env,
> +           args: [meson.current_source_dir() / 'test_' + test + '.py'],
> +           protocol: 'tap',
> +           timeout: test_timeouts.get(test, 60),
> +           priority: test_timeouts.get(test, 60),
> +           suite: suites)
> +    endforeach
> +  endforeach
> +endforeach
> diff --git a/tests/meson.build b/tests/meson.build
> index 80dd3029cf..907a4c1c98 100644
> --- a/tests/meson.build
> +++ b/tests/meson.build
> @@ -85,3 +85,4 @@ subdir('unit')
>   subdir('qapi-schema')
>   subdir('qtest')
>   subdir('migration')
> +subdir('functional')
Philippe Mathieu-Daudé Aug. 23, 2024, 12:54 p.m. UTC | #2
On 21/8/24 10:27, Thomas Huth wrote:
> Provide a meson.build file for the upcoming python-based functional
> tests, and add some wrapper glue targets to the tests/Makefile.include
> file. We are going to use two "speed" modes for the functional tests:
> The "quick" tests can be run at any time (i.e. also during "make check"),
> while the "thorough" tests should only be run when running a
> "make check-functional" test run (since these tests might download
> additional assets from the internet).
> 
> The changes to the meson.build files are partly based on an earlier
> patch by Ani Sinha.
> 
> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>   tests/Makefile.include       | 11 ++++++
>   tests/functional/meson.build | 66 ++++++++++++++++++++++++++++++++++++
>   tests/meson.build            |  1 +
>   3 files changed, 78 insertions(+)
>   create mode 100644 tests/functional/meson.build


> +# Timeouts for individual tests that can be slow e.g. with debugging enabled
> +test_timeouts = {
> +}


> +    foreach test : target_tests
> +      test('func-@0@/@1@'.format(target_base, test),
> +           python,
> +           depends: [test_deps, test_emulator, emulator_modules],
> +           env: test_env,
> +           args: [meson.current_source_dir() / 'test_' + test + '.py'],
> +           protocol: 'tap',
> +           timeout: test_timeouts.get(test, 60),
> +           priority: test_timeouts.get(test, 60),

IIUC with Avocado the timeout was for each test_func in a TestClass.
Now this is only per TestClass. Hopefully I'm wrong...

> +           suite: suites)
> +    endforeach
> +  endforeach
> +endforeach
> diff --git a/tests/meson.build b/tests/meson.build
> index 80dd3029cf..907a4c1c98 100644
> --- a/tests/meson.build
> +++ b/tests/meson.build
> @@ -85,3 +85,4 @@ subdir('unit')
>   subdir('qapi-schema')
>   subdir('qtest')
>   subdir('migration')
> +subdir('functional')
Thomas Huth Aug. 26, 2024, 8:18 a.m. UTC | #3
On 23/08/2024 14.54, Philippe Mathieu-Daudé wrote:
> On 21/8/24 10:27, Thomas Huth wrote:
>> Provide a meson.build file for the upcoming python-based functional
>> tests, and add some wrapper glue targets to the tests/Makefile.include
>> file. We are going to use two "speed" modes for the functional tests:
>> The "quick" tests can be run at any time (i.e. also during "make check"),
>> while the "thorough" tests should only be run when running a
>> "make check-functional" test run (since these tests might download
>> additional assets from the internet).
>>
>> The changes to the meson.build files are partly based on an earlier
>> patch by Ani Sinha.
>>
>> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
>> Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>   tests/Makefile.include       | 11 ++++++
>>   tests/functional/meson.build | 66 ++++++++++++++++++++++++++++++++++++
>>   tests/meson.build            |  1 +
>>   3 files changed, 78 insertions(+)
>>   create mode 100644 tests/functional/meson.build
> 
> 
>> +# Timeouts for individual tests that can be slow e.g. with debugging enabled
>> +test_timeouts = {
>> +}
> 
> 
>> +    foreach test : target_tests
>> +      test('func-@0@/@1@'.format(target_base, test),
>> +           python,
>> +           depends: [test_deps, test_emulator, emulator_modules],
>> +           env: test_env,
>> +           args: [meson.current_source_dir() / 'test_' + test + '.py'],
>> +           protocol: 'tap',
>> +           timeout: test_timeouts.get(test, 60),
>> +           priority: test_timeouts.get(test, 60),
> 
> IIUC with Avocado the timeout was for each test_func in a TestClass.
> Now this is only per TestClass. Hopefully I'm wrong...

I think you're right ... we might need to adjust the meson timeouts here and 
there in case they are causing problems, but that's just business as usual 
(we had to do the same when enabling the meson timeouts for the qtests for 
example).

  Thomas
Daniel P. Berrangé Aug. 29, 2024, 9:54 a.m. UTC | #4
On Mon, Aug 26, 2024 at 10:18:44AM +0200, Thomas Huth wrote:
> On 23/08/2024 14.54, Philippe Mathieu-Daudé wrote:
> > On 21/8/24 10:27, Thomas Huth wrote:
> > > Provide a meson.build file for the upcoming python-based functional
> > > tests, and add some wrapper glue targets to the tests/Makefile.include
> > > file. We are going to use two "speed" modes for the functional tests:
> > > The "quick" tests can be run at any time (i.e. also during "make check"),
> > > while the "thorough" tests should only be run when running a
> > > "make check-functional" test run (since these tests might download
> > > additional assets from the internet).
> > > 
> > > The changes to the meson.build files are partly based on an earlier
> > > patch by Ani Sinha.
> > > 
> > > Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> > > Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org>
> > > Signed-off-by: Thomas Huth <thuth@redhat.com>
> > > ---
> > >   tests/Makefile.include       | 11 ++++++
> > >   tests/functional/meson.build | 66 ++++++++++++++++++++++++++++++++++++
> > >   tests/meson.build            |  1 +
> > >   3 files changed, 78 insertions(+)
> > >   create mode 100644 tests/functional/meson.build
> > 
> > 
> > > +# Timeouts for individual tests that can be slow e.g. with debugging enabled
> > > +test_timeouts = {
> > > +}
> > 
> > 
> > > +    foreach test : target_tests
> > > +      test('func-@0@/@1@'.format(target_base, test),
> > > +           python,
> > > +           depends: [test_deps, test_emulator, emulator_modules],
> > > +           env: test_env,
> > > +           args: [meson.current_source_dir() / 'test_' + test + '.py'],
> > > +           protocol: 'tap',
> > > +           timeout: test_timeouts.get(test, 60),
> > > +           priority: test_timeouts.get(test, 60),
> > 
> > IIUC with Avocado the timeout was for each test_func in a TestClass.
> > Now this is only per TestClass. Hopefully I'm wrong...
> 
> I think you're right ... we might need to adjust the meson timeouts here and
> there in case they are causing problems, but that's just business as usual
> (we had to do the same when enabling the meson timeouts for the qtests for
> example).

Meson timeouts are around the lifetime of whatever child process
we tell it to execute.  So, yes, the timeout is for the .py file
as a whole, not the individual TestClass within it.

This is different from Avocado, but not a big problem. Worst case
we'll adjust some meson level timeouts, or split overly long running
.py test files into multiple test .py files.

With regards,
Daniel
diff mbox series

Patch

diff --git a/tests/Makefile.include b/tests/Makefile.include
index fbb1a4b211..66c8cc3123 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -9,6 +9,8 @@  check-help:
 	@echo "Individual test suites:"
 	@echo " $(MAKE) check-qtest-TARGET       Run qtest tests for given target"
 	@echo " $(MAKE) check-qtest              Run qtest tests"
+	@echo " $(MAKE) check-functional         Run python-based functional tests"
+	@echo " $(MAKE) check-functional-TARGET  Run functional tests for a given target"
 	@echo " $(MAKE) check-unit               Run qobject tests"
 	@echo " $(MAKE) check-qapi-schema        Run QAPI schema tests"
 	@echo " $(MAKE) check-block              Run block tests"
@@ -152,6 +154,15 @@  check-acceptance-deprecated-warning:
 
 check-acceptance: check-acceptance-deprecated-warning | check-avocado
 
+FUNCTIONAL_TARGETS=$(patsubst %-softmmu,check-functional-%, $(filter %-softmmu,$(TARGETS)))
+.PHONY: $(FUNCTIONAL_TARGETS)
+$(FUNCTIONAL_TARGETS):
+	@$(MAKE) SPEED=thorough $(subst -functional,-func,$@)
+
+.PHONY: check-functional
+check-functional:
+	@$(MAKE) SPEED=thorough check-func check-func-quick
+
 # Consolidated targets
 
 .PHONY: check check-clean get-vm-images
diff --git a/tests/functional/meson.build b/tests/functional/meson.build
new file mode 100644
index 0000000000..7fc9c4e513
--- /dev/null
+++ b/tests/functional/meson.build
@@ -0,0 +1,66 @@ 
+# QEMU functional tests:
+# Tests that are put in the 'quick' category are run by default during
+# 'make check'. Everything that should not be run during 'make check'
+# (e.g. tests that fetch assets from the internet) should be put into
+# the 'thorough' category instead.
+
+# Most tests run too slow with TCI enabled, so skip the functional tests there
+if get_option('tcg_interpreter')
+  subdir_done()
+endif
+
+# Timeouts for individual tests that can be slow e.g. with debugging enabled
+test_timeouts = {
+}
+
+tests_generic = [
+]
+
+tests_x86_64_quick = [
+]
+
+tests_x86_64_thorough = [
+]
+
+foreach speed : ['quick', 'thorough']
+  foreach dir : target_dirs
+    if not dir.endswith('-softmmu')
+      continue
+    endif
+
+    target_base = dir.split('-')[0]
+    test_emulator = emulators['qemu-system-' + target_base]
+
+    if speed == 'quick'
+      suites = ['func-quick', 'func-' + target_base]
+      target_tests = get_variable('tests_' + target_base + '_quick', []) + tests_generic
+    else
+      suites = ['func-' + speed, 'func-' + target_base + '-' + speed, speed]
+      target_tests = get_variable('tests_' + target_base + '_' + speed, [])
+    endif
+
+    test_deps = roms
+    test_env = environment()
+    if have_tools
+      test_env.set('QEMU_TEST_QEMU_IMG', meson.global_build_root() / 'qemu-img')
+      test_deps += [qemu_img]
+    endif
+    test_env.set('QEMU_TEST_QEMU_BINARY',
+                 meson.global_build_root() / 'qemu-system-' + target_base)
+    test_env.set('QEMU_BUILD_ROOT', meson.project_build_root())
+    test_env.set('PYTHONPATH', meson.project_source_root() / 'python:' +
+                               meson.current_source_dir())
+
+    foreach test : target_tests
+      test('func-@0@/@1@'.format(target_base, test),
+           python,
+           depends: [test_deps, test_emulator, emulator_modules],
+           env: test_env,
+           args: [meson.current_source_dir() / 'test_' + test + '.py'],
+           protocol: 'tap',
+           timeout: test_timeouts.get(test, 60),
+           priority: test_timeouts.get(test, 60),
+           suite: suites)
+    endforeach
+  endforeach
+endforeach
diff --git a/tests/meson.build b/tests/meson.build
index 80dd3029cf..907a4c1c98 100644
--- a/tests/meson.build
+++ b/tests/meson.build
@@ -85,3 +85,4 @@  subdir('unit')
 subdir('qapi-schema')
 subdir('qtest')
 subdir('migration')
+subdir('functional')