| Message ID | 20220710170014.1673480-11-ani@anisinha.ca |
|---|---|
| State | New |
| Headers | show |
| Series | Introduce new acpi/smbios python tests using biosbits | expand |
On Sun, Jul 10, 2022 at 1:01 PM Ani Sinha <ani@anisinha.ca> wrote: > > Integrate the pytest framework with the meson build system. This will make meson > run all the pytests under the pytest directory. > > Signed-off-by: Ani Sinha <ani@anisinha.ca> > --- > tests/Makefile.include | 4 +++- > tests/meson.build | 1 + > tests/pytest/meson.build | 49 ++++++++++++++++++++++++++++++++++++++++ > 3 files changed, 53 insertions(+), 1 deletion(-) > create mode 100644 tests/pytest/meson.build > > diff --git a/tests/Makefile.include b/tests/Makefile.include > index 3accb83b13..40755a6bd1 100644 > --- a/tests/Makefile.include > +++ b/tests/Makefile.include > @@ -3,12 +3,14 @@ > .PHONY: check-help > check-help: > @echo "Regression testing targets:" > - @echo " $(MAKE) check Run block, qapi-schema, unit, softfloat, qtest and decodetree tests" > + @echo " $(MAKE) check Run block, qapi-schema, unit, softfloat, qtest, pytest and decodetree tests" Does this mean that "make check" *requires* an internet connection? If so, I'm afraid that introduces some complications for downstreams which require that "make check" can run without an internet connection. It's something I've been trying to wrestle with as I split the qemu.qmp library out of the QEMU tree, and I have been working (slowly) on remedying it with some additional Makefile logic. I have been looking into making a testing "dummy package" for python with optional dependency groups that use a venv building script to either pull from PyPI in online mode, or use system packages in offline mode. In the case of offline mode, I am working on augmenting the configure script to test that dependencies are met, and disabling test groups when the correct dependencies cannot be located. I hope to have another version of that series soon; it should be trivial to add a new optional dependency group to it. I'll CC you on the next version of the series. --js > @echo " $(MAKE) bench Run speed tests" > @echo > @echo "Individual test suites:" > @echo " $(MAKE) check-qtest-TARGET Run qtest tests for given target" > @echo " $(MAKE) check-qtest Run qtest tests" > + @echo " $(MAKE) check-pytest Run pytest tests" > + @echo " $(MAKE) check-pytest-TARGET Run pytest for a given target" > @echo " $(MAKE) check-unit Run qobject tests" > @echo " $(MAKE) check-qapi-schema Run QAPI schema tests" > @echo " $(MAKE) check-block Run block tests" > diff --git a/tests/meson.build b/tests/meson.build > index 8e318ec513..f344cbdc6c 100644 > --- a/tests/meson.build > +++ b/tests/meson.build > @@ -93,3 +93,4 @@ subdir('unit') > subdir('qapi-schema') > subdir('qtest') > subdir('migration') > +subdir('pytest') > diff --git a/tests/pytest/meson.build b/tests/pytest/meson.build > new file mode 100644 > index 0000000000..e60d481ae4 > --- /dev/null > +++ b/tests/pytest/meson.build > @@ -0,0 +1,49 @@ > +slow_pytests = { > + 'acpi-bits' : 120, > +} > + > +pytests_generic = [] > + > +# biosbits tests are currenly only supported on x86_64 platforms. > +pytests_x86_64 = ['acpi-bits-test'] > + > +pytest_executables = {} > +other_deps = [] > + > +subdir('acpi-bits') > + > +foreach dir : target_dirs > + if not dir.endswith('-softmmu') > + continue > + endif > + > + target_base = dir.split('-')[0] > + pytest_emulator = emulators['qemu-system-' + target_base] > + target_pytests = get_variable('pytests_' + target_base, []) + pytests_generic > + > + test_deps = roms > + pytest_env = environment() > + if have_tools > + pytest_env.set('QTEST_QEMU_IMG', './qemu-img') > + test_deps += [qemu_img] > + endif > + pytest_env.set('G_TEST_DBUS_DAEMON', meson.project_source_root() / 'tests/dbus-vmstate-daemon.sh') > + pytest_env.set('PYTEST_QEMU_BINARY', './qemu-system-' + target_base) > + pytest_env.set('PYTEST_SOURCE_ROOT', meson.project_source_root()) > + if have_tools and have_vhost_user_blk_server > + pytest_env.set('PYTEST_QEMU_STORAGE_DAEMON_BINARY', './storage-daemon/qemu-storage-daemon') > + test_deps += [qsd] > + endif > + > + foreach test : target_pytests > + test('pytest-@0@/@1@'.format(target_base, test), > + pytest_executables[test], > + depends: [test_deps, pytest_emulator, emulator_modules, other_deps], > + env: pytest_env, > + args: ['--tap', '-k'], > + protocol: 'tap', > + timeout: slow_pytests.get(test, 30), > + priority: slow_pytests.get(test, 30), > + suite: ['pytest', 'pytest-' + target_base]) > + endforeach > +endforeach > -- > 2.25.1 >
On Mon, 11 Jul 2022, John Snow wrote: > On Sun, Jul 10, 2022 at 1:01 PM Ani Sinha <ani@anisinha.ca> wrote: > > > > Integrate the pytest framework with the meson build system. This will make meson > > run all the pytests under the pytest directory. > > > > Signed-off-by: Ani Sinha <ani@anisinha.ca> > > --- > > tests/Makefile.include | 4 +++- > > tests/meson.build | 1 + > > tests/pytest/meson.build | 49 ++++++++++++++++++++++++++++++++++++++++ > > 3 files changed, 53 insertions(+), 1 deletion(-) > > create mode 100644 tests/pytest/meson.build > > > > diff --git a/tests/Makefile.include b/tests/Makefile.include > > index 3accb83b13..40755a6bd1 100644 > > --- a/tests/Makefile.include > > +++ b/tests/Makefile.include > > @@ -3,12 +3,14 @@ > > .PHONY: check-help > > check-help: > > @echo "Regression testing targets:" > > - @echo " $(MAKE) check Run block, qapi-schema, unit, softfloat, qtest and decodetree tests" > > + @echo " $(MAKE) check Run block, qapi-schema, unit, softfloat, qtest, pytest and decodetree tests" > > Does this mean that "make check" *requires* an internet connection? No. My test will be skipped if it is unable to download the artifacts it requires due to lack of Internet connectivity. If > so, I'm afraid that introduces some complications for downstreams > which require that "make check" can run without an internet > connection. It's something I've been trying to wrestle with as I split > the qemu.qmp library out of the QEMU tree, and I have been working > (slowly) on remedying it with some additional Makefile logic. > > I have been looking into making a testing "dummy package" for python > with optional dependency groups that use a venv building script to > either pull from PyPI in online mode, or use system packages in > offline mode. In the case of offline mode, I am working on augmenting > the configure script to test that dependencies are met, and disabling > test groups when the correct dependencies cannot be located. > > I hope to have another version of that series soon; it should be > trivial to add a new optional dependency group to it. I'll CC you on > the next version of the series. > > --js > > > @echo " $(MAKE) bench Run speed tests" > > @echo > > @echo "Individual test suites:" > > @echo " $(MAKE) check-qtest-TARGET Run qtest tests for given target" > > @echo " $(MAKE) check-qtest Run qtest tests" > > + @echo " $(MAKE) check-pytest Run pytest tests" > > + @echo " $(MAKE) check-pytest-TARGET Run pytest for a given target" > > @echo " $(MAKE) check-unit Run qobject tests" > > @echo " $(MAKE) check-qapi-schema Run QAPI schema tests" > > @echo " $(MAKE) check-block Run block tests" > > diff --git a/tests/meson.build b/tests/meson.build > > index 8e318ec513..f344cbdc6c 100644 > > --- a/tests/meson.build > > +++ b/tests/meson.build > > @@ -93,3 +93,4 @@ subdir('unit') > > subdir('qapi-schema') > > subdir('qtest') > > subdir('migration') > > +subdir('pytest') > > diff --git a/tests/pytest/meson.build b/tests/pytest/meson.build > > new file mode 100644 > > index 0000000000..e60d481ae4 > > --- /dev/null > > +++ b/tests/pytest/meson.build > > @@ -0,0 +1,49 @@ > > +slow_pytests = { > > + 'acpi-bits' : 120, > > +} > > + > > +pytests_generic = [] > > + > > +# biosbits tests are currenly only supported on x86_64 platforms. > > +pytests_x86_64 = ['acpi-bits-test'] > > + > > +pytest_executables = {} > > +other_deps = [] > > + > > +subdir('acpi-bits') > > + > > +foreach dir : target_dirs > > + if not dir.endswith('-softmmu') > > + continue > > + endif > > + > > + target_base = dir.split('-')[0] > > + pytest_emulator = emulators['qemu-system-' + target_base] > > + target_pytests = get_variable('pytests_' + target_base, []) + pytests_generic > > + > > + test_deps = roms > > + pytest_env = environment() > > + if have_tools > > + pytest_env.set('QTEST_QEMU_IMG', './qemu-img') > > + test_deps += [qemu_img] > > + endif > > + pytest_env.set('G_TEST_DBUS_DAEMON', meson.project_source_root() / 'tests/dbus-vmstate-daemon.sh') > > + pytest_env.set('PYTEST_QEMU_BINARY', './qemu-system-' + target_base) > > + pytest_env.set('PYTEST_SOURCE_ROOT', meson.project_source_root()) > > + if have_tools and have_vhost_user_blk_server > > + pytest_env.set('PYTEST_QEMU_STORAGE_DAEMON_BINARY', './storage-daemon/qemu-storage-daemon') > > + test_deps += [qsd] > > + endif > > + > > + foreach test : target_pytests > > + test('pytest-@0@/@1@'.format(target_base, test), > > + pytest_executables[test], > > + depends: [test_deps, pytest_emulator, emulator_modules, other_deps], > > + env: pytest_env, > > + args: ['--tap', '-k'], > > + protocol: 'tap', > > + timeout: slow_pytests.get(test, 30), > > + priority: slow_pytests.get(test, 30), > > + suite: ['pytest', 'pytest-' + target_base]) > > + endforeach > > +endforeach > > -- > > 2.25.1 > > > >
On Tue, Jul 12, 2022 at 12:22:10PM +0530, Ani Sinha wrote: > > > On Mon, 11 Jul 2022, John Snow wrote: > > > On Sun, Jul 10, 2022 at 1:01 PM Ani Sinha <ani@anisinha.ca> wrote: > > > > > > Integrate the pytest framework with the meson build system. This will make meson > > > run all the pytests under the pytest directory. > > > > > > Signed-off-by: Ani Sinha <ani@anisinha.ca> > > > --- > > > tests/Makefile.include | 4 +++- > > > tests/meson.build | 1 + > > > tests/pytest/meson.build | 49 ++++++++++++++++++++++++++++++++++++++++ > > > 3 files changed, 53 insertions(+), 1 deletion(-) > > > create mode 100644 tests/pytest/meson.build > > > > > > diff --git a/tests/Makefile.include b/tests/Makefile.include > > > index 3accb83b13..40755a6bd1 100644 > > > --- a/tests/Makefile.include > > > +++ b/tests/Makefile.include > > > @@ -3,12 +3,14 @@ > > > .PHONY: check-help > > > check-help: > > > @echo "Regression testing targets:" > > > - @echo " $(MAKE) check Run block, qapi-schema, unit, softfloat, qtest and decodetree tests" > > > + @echo " $(MAKE) check Run block, qapi-schema, unit, softfloat, qtest, pytest and decodetree tests" > > > > Does this mean that "make check" *requires* an internet connection? > > No. My test will be skipped if it is unable to download the artifacts it > requires due to lack of Internet connectivity. That's not the only concern, there are also people who have metered internet connections, or whose connections are slow and thus have long download times. Any test that downloads should be opt-in only. With regards, Daniel
On Tue, Sep 06, 2022 at 02:10:56PM +0100, Daniel P. Berrangé wrote: > On Tue, Jul 12, 2022 at 12:22:10PM +0530, Ani Sinha wrote: > > > > > > On Mon, 11 Jul 2022, John Snow wrote: > > > > > On Sun, Jul 10, 2022 at 1:01 PM Ani Sinha <ani@anisinha.ca> wrote: > > > > > > > > Integrate the pytest framework with the meson build system. This will make meson > > > > run all the pytests under the pytest directory. > > > > > > > > Signed-off-by: Ani Sinha <ani@anisinha.ca> > > > > --- > > > > tests/Makefile.include | 4 +++- > > > > tests/meson.build | 1 + > > > > tests/pytest/meson.build | 49 ++++++++++++++++++++++++++++++++++++++++ > > > > 3 files changed, 53 insertions(+), 1 deletion(-) > > > > create mode 100644 tests/pytest/meson.build > > > > > > > > diff --git a/tests/Makefile.include b/tests/Makefile.include > > > > index 3accb83b13..40755a6bd1 100644 > > > > --- a/tests/Makefile.include > > > > +++ b/tests/Makefile.include > > > > @@ -3,12 +3,14 @@ > > > > .PHONY: check-help > > > > check-help: > > > > @echo "Regression testing targets:" > > > > - @echo " $(MAKE) check Run block, qapi-schema, unit, softfloat, qtest and decodetree tests" > > > > + @echo " $(MAKE) check Run block, qapi-schema, unit, softfloat, qtest, pytest and decodetree tests" > > > > > > Does this mean that "make check" *requires* an internet connection? > > > > No. My test will be skipped if it is unable to download the artifacts it > > requires due to lack of Internet connectivity. > > That's not the only concern, there are also people who have metered > internet connections, or whose connections are slow and thus have > long download times. Any test that downloads should be opt-in only. > > > With regards, > Daniel <rant> This is why I wanted git submodules. A well understood decentralized model. Now we are reinventing them badly. I asked on the maintainers summit what issues people have with submodules, no one volunteered any information. It might make sense to figure out if there's a way to use submodules sanely. </rant> Anyway, download should just be done separately, make check should just verify it has the correct binary and if not fail. And I'd like to have a target that fails if it can not run the tests a opposed to skipping. > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On 27/09/2022 23.26, Michael S. Tsirkin wrote: > On Tue, Sep 06, 2022 at 02:10:56PM +0100, Daniel P. Berrangé wrote: >> On Tue, Jul 12, 2022 at 12:22:10PM +0530, Ani Sinha wrote: >>> >>> >>> On Mon, 11 Jul 2022, John Snow wrote: >>> >>>> On Sun, Jul 10, 2022 at 1:01 PM Ani Sinha <ani@anisinha.ca> wrote: >>>>> >>>>> Integrate the pytest framework with the meson build system. This will make meson >>>>> run all the pytests under the pytest directory. >>>>> >>>>> Signed-off-by: Ani Sinha <ani@anisinha.ca> >>>>> --- >>>>> tests/Makefile.include | 4 +++- >>>>> tests/meson.build | 1 + >>>>> tests/pytest/meson.build | 49 ++++++++++++++++++++++++++++++++++++++++ >>>>> 3 files changed, 53 insertions(+), 1 deletion(-) >>>>> create mode 100644 tests/pytest/meson.build >>>>> >>>>> diff --git a/tests/Makefile.include b/tests/Makefile.include >>>>> index 3accb83b13..40755a6bd1 100644 >>>>> --- a/tests/Makefile.include >>>>> +++ b/tests/Makefile.include >>>>> @@ -3,12 +3,14 @@ >>>>> .PHONY: check-help >>>>> check-help: >>>>> @echo "Regression testing targets:" >>>>> - @echo " $(MAKE) check Run block, qapi-schema, unit, softfloat, qtest and decodetree tests" >>>>> + @echo " $(MAKE) check Run block, qapi-schema, unit, softfloat, qtest, pytest and decodetree tests" >>>> >>>> Does this mean that "make check" *requires* an internet connection? >>> >>> No. My test will be skipped if it is unable to download the artifacts it >>> requires due to lack of Internet connectivity. >> >> That's not the only concern, there are also people who have metered >> internet connections, or whose connections are slow and thus have >> long download times. Any test that downloads should be opt-in only. >> >> >> With regards, >> Daniel > > <rant> > This is why I wanted git submodules. A well understood decentralized > model. Now we are reinventing them badly. I don't see much of a difference here with submodules with regards to Daniel's concerns: If you clone the QEMU repo with a good internet connection, you also don't get the submodules by default. If you then move to a bad internet connection or offline and the build system tries to pull in the submodule, you've also lost. OTOH, with downloads, you can also make sure to have all downloads cached while you're still on the good internet connection. The cache should continue working if you're then moving offline. > I asked on the maintainers summit what issues people have with > submodules, no one volunteered any information. You know my position already (https://lore.kernel.org/qemu-devel/d7a7b28f-a665-2567-0fb6-e31e7ecbb5c8@redhat.com/), so at least I did not feel the need to repeat that again (and we were pretty short in time anyway). Thomas
On Wed, Sep 28, 2022 at 09:32:57AM +0200, Thomas Huth wrote: > On 27/09/2022 23.26, Michael S. Tsirkin wrote: > > On Tue, Sep 06, 2022 at 02:10:56PM +0100, Daniel P. Berrangé wrote: > > > On Tue, Jul 12, 2022 at 12:22:10PM +0530, Ani Sinha wrote: > > > > > > > > > > > > On Mon, 11 Jul 2022, John Snow wrote: > > > > > > > > > On Sun, Jul 10, 2022 at 1:01 PM Ani Sinha <ani@anisinha.ca> wrote: > > > > > > > > > > > > Integrate the pytest framework with the meson build system. This will make meson > > > > > > run all the pytests under the pytest directory. > > > > > > > > > > > > Signed-off-by: Ani Sinha <ani@anisinha.ca> > > > > > > --- > > > > > > tests/Makefile.include | 4 +++- > > > > > > tests/meson.build | 1 + > > > > > > tests/pytest/meson.build | 49 ++++++++++++++++++++++++++++++++++++++++ > > > > > > 3 files changed, 53 insertions(+), 1 deletion(-) > > > > > > create mode 100644 tests/pytest/meson.build > > > > > > > > > > > > diff --git a/tests/Makefile.include b/tests/Makefile.include > > > > > > index 3accb83b13..40755a6bd1 100644 > > > > > > --- a/tests/Makefile.include > > > > > > +++ b/tests/Makefile.include > > > > > > @@ -3,12 +3,14 @@ > > > > > > .PHONY: check-help > > > > > > check-help: > > > > > > @echo "Regression testing targets:" > > > > > > - @echo " $(MAKE) check Run block, qapi-schema, unit, softfloat, qtest and decodetree tests" > > > > > > + @echo " $(MAKE) check Run block, qapi-schema, unit, softfloat, qtest, pytest and decodetree tests" > > > > > > > > > > Does this mean that "make check" *requires* an internet connection? > > > > > > > > No. My test will be skipped if it is unable to download the artifacts it > > > > requires due to lack of Internet connectivity. > > > > > > That's not the only concern, there are also people who have metered > > > internet connections, or whose connections are slow and thus have > > > long download times. Any test that downloads should be opt-in only. > > > > > > > > > With regards, > > > Daniel > > > > <rant> > > This is why I wanted git submodules. A well understood decentralized > > model. Now we are reinventing them badly. > > I don't see much of a difference here with submodules with regards to > Daniel's concerns: If you clone the QEMU repo with a good internet > connection, you also don't get the submodules by default. If you then move > to a bad internet connection or offline and the build system tries to pull > in the submodule, you've also lost. > OTOH, with downloads, you can also make sure to have all downloads cached > while you're still on the good internet connection. The cache should > continue working if you're then moving offline. submodules as a cache include checksumming, a way to make local changes without too much pain, documentation .... > > I asked on the maintainers summit what issues people have with > > submodules, no one volunteered any information. > > You know my position already (https://lore.kernel.org/qemu-devel/d7a7b28f-a665-2567-0fb6-e31e7ecbb5c8@redhat.com/), > so at least I did not feel the need to repeat that again (and we were pretty > short in time anyway). Pity, but ok, I will revive that thread. > > Thomas
diff --git a/tests/Makefile.include b/tests/Makefile.include index 3accb83b13..40755a6bd1 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -3,12 +3,14 @@ .PHONY: check-help check-help: @echo "Regression testing targets:" - @echo " $(MAKE) check Run block, qapi-schema, unit, softfloat, qtest and decodetree tests" + @echo " $(MAKE) check Run block, qapi-schema, unit, softfloat, qtest, pytest and decodetree tests" @echo " $(MAKE) bench Run speed tests" @echo @echo "Individual test suites:" @echo " $(MAKE) check-qtest-TARGET Run qtest tests for given target" @echo " $(MAKE) check-qtest Run qtest tests" + @echo " $(MAKE) check-pytest Run pytest tests" + @echo " $(MAKE) check-pytest-TARGET Run pytest for a given target" @echo " $(MAKE) check-unit Run qobject tests" @echo " $(MAKE) check-qapi-schema Run QAPI schema tests" @echo " $(MAKE) check-block Run block tests" diff --git a/tests/meson.build b/tests/meson.build index 8e318ec513..f344cbdc6c 100644 --- a/tests/meson.build +++ b/tests/meson.build @@ -93,3 +93,4 @@ subdir('unit') subdir('qapi-schema') subdir('qtest') subdir('migration') +subdir('pytest') diff --git a/tests/pytest/meson.build b/tests/pytest/meson.build new file mode 100644 index 0000000000..e60d481ae4 --- /dev/null +++ b/tests/pytest/meson.build @@ -0,0 +1,49 @@ +slow_pytests = { + 'acpi-bits' : 120, +} + +pytests_generic = [] + +# biosbits tests are currenly only supported on x86_64 platforms. +pytests_x86_64 = ['acpi-bits-test'] + +pytest_executables = {} +other_deps = [] + +subdir('acpi-bits') + +foreach dir : target_dirs + if not dir.endswith('-softmmu') + continue + endif + + target_base = dir.split('-')[0] + pytest_emulator = emulators['qemu-system-' + target_base] + target_pytests = get_variable('pytests_' + target_base, []) + pytests_generic + + test_deps = roms + pytest_env = environment() + if have_tools + pytest_env.set('QTEST_QEMU_IMG', './qemu-img') + test_deps += [qemu_img] + endif + pytest_env.set('G_TEST_DBUS_DAEMON', meson.project_source_root() / 'tests/dbus-vmstate-daemon.sh') + pytest_env.set('PYTEST_QEMU_BINARY', './qemu-system-' + target_base) + pytest_env.set('PYTEST_SOURCE_ROOT', meson.project_source_root()) + if have_tools and have_vhost_user_blk_server + pytest_env.set('PYTEST_QEMU_STORAGE_DAEMON_BINARY', './storage-daemon/qemu-storage-daemon') + test_deps += [qsd] + endif + + foreach test : target_pytests + test('pytest-@0@/@1@'.format(target_base, test), + pytest_executables[test], + depends: [test_deps, pytest_emulator, emulator_modules, other_deps], + env: pytest_env, + args: ['--tap', '-k'], + protocol: 'tap', + timeout: slow_pytests.get(test, 30), + priority: slow_pytests.get(test, 30), + suite: ['pytest', 'pytest-' + target_base]) + endforeach +endforeach
Integrate the pytest framework with the meson build system. This will make meson run all the pytests under the pytest directory. Signed-off-by: Ani Sinha <ani@anisinha.ca> --- tests/Makefile.include | 4 +++- tests/meson.build | 1 + tests/pytest/meson.build | 49 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 53 insertions(+), 1 deletion(-) create mode 100644 tests/pytest/meson.build