| Message ID | 20231018140736.3618-1-farosas@suse.de |
|---|---|
| Headers | show |
| Series | tests/migration-test: Allow testing older machine types | expand |
Fabiano Rosas <farosas@suse.de> writes: > This adds support for running migration-test with two different QEMU > versions to test migration compatibility. The tests automatically > choose the latest machine type supported by both QEMU versions. Let me point out two situations that came up while developing this: 1) There was a generic command line change in QEMU (-audio option) introduced this cycle that was ported to the command line used by libqtest.c. This means that when using the same command line to start both guests, the older one will not recognize the option and will break. Ideally, we would keep command lines out of the library and require each test to provide their own. Then maybe the migration code could cope with the change somehow. 2) When fixing an issue during a release cycle, that issue can still show up on the next one when using this series to test compatibility. There's probably not much we can do here, but remember the issues that were already fixed and ignore them when they appear in these compatibiliy tests. I don't see how to avoid the situations above, but I think their existence implies we cannot enable by default any sort of migration compatibility testing that uses a live guest.
On Wed, Oct 18, 2023 at 11:19:01AM -0300, Fabiano Rosas wrote: > Fabiano Rosas <farosas@suse.de> writes: > > > This adds support for running migration-test with two different QEMU > > versions to test migration compatibility. The tests automatically > > choose the latest machine type supported by both QEMU versions. > > Let me point out two situations that came up while developing this: > > 1) There was a generic command line change in QEMU (-audio option) > introduced this cycle that was ported to the command line used by > libqtest.c. This means that when using the same command line to > start both guests, the older one will not recognize the option and > will break. > > Ideally, we would keep command lines out of the library and require > each test to provide their own. Then maybe the migration code could > cope with the change somehow. Right the problem is that QEMU doesn't provide a long term stable API, but has finite sliding windows where QEMU versions can be compatible in their CLI, determined by our minimal deprecation period of 2 releases. eg you can come up with a CLI that is compatible for all of 7.1, 7.2 and 8.0 but you can't neccessarily come up with a CLI that is compatible for all of 7.1, 7.2, 8.0 and 8.1 because the lifespan of that set of versions is greater than our deprecation period. If we did't use -audio in libqtest, we could be compatible with current QEMU and some old QEMUs, but not future QEMU. If we do use -audio in libqtest, we could be compatible with current QEMU and some future QEMUs, but not old QEMUs. This is a no win scenario. Now, libvirt provides a long term stable configuration for QEMU that transcends changes in QEMU CLI. So we could come up with a live migration test between QEMU's of an arbitrary age difference, if we used libvirt to drive the process. That wouldn't really be qtest though. Could perhaps we done in the context of avocado, if we used the libvirt-python API binding, but this would be different from other avocado tests by virtue of not using the QEMU python classes for launch. > 2) When fixing an issue during a release cycle, that issue can still > show up on the next one when using this series to test > compatibility. There's probably not much we can do here, but > remember the issues that were already fixed and ignore them when > they appear in these compatibiliy tests. > > I don't see how to avoid the situations above, but I think their > existence implies we cannot enable by default any sort of migration > compatibility testing that uses a live guest. With regards, Daniel
On Wed, Oct 18, 2023 at 11:19:01AM -0300, Fabiano Rosas wrote: > Let me point out two situations that came up while developing this: > > 1) There was a generic command line change in QEMU (-audio option) > introduced this cycle that was ported to the command line used by > libqtest.c. This means that when using the same command line to > start both guests, the older one will not recognize the option and > will break. > > Ideally, we would keep command lines out of the library and require > each test to provide their own. Then maybe the migration code could > cope with the change somehow. Yeah this is unfortunate, and I hit it as soon as trying this out with an old binary. One other thing on top of this: the cross binary tests for different devices, where we used to hit issues when some specific pci device or a pcie root port failed to be compatible with old binaries, iirc. I'm still not sure what's the best way to do that, though, especially if the migration failure can only trigger when a device driver setup the device first. Maybe we can start with migrating a device with the reset status between binaries. Having this series is definitely better than nothing already. Thanks,
This adds support for running migration-test with two different QEMU versions to test migration compatibility. The tests automatically choose the latest machine type supported by both QEMU versions. changes: - added a cleanup function for the allocated strings in the machine list. The suggestion of making a copy of the list won't work because keeping the structure around to avoid freeing memory, and caching the list are two separate things that only make sense together at qtest_get_machines(). Outside of it, code can just free memory normally and caching doesn't make sense because there's no QMP call; - kept strdups for the reasons mentioned above; - added a message when we're unable to find a common machine type (should never happen); - changed the default x86_64 machine to q35; - new patch to fix booting with the a-b bootsector on q35; - added machine_opts for the options after the comma; - used machine_alias and machine for the variable names; CI run: https://gitlab.com/farosas/qemu/-/pipelines/1041238172 v2: https://lore.kernel.org/r/20231006123910.17759-1-farosas@suse.de changes: - introduce *_with_env variants of the relevant functions [Daniel, Juan] - keep the requirement for having the QTEST_QEMU_BINARY always present. qtest_get_arch() is used extensively in the qtest_add* functions. It would be too much churn to pass a different binary into it. - with this^ we also need to keep the requirement for using only one of SRC|DST. Otherwise it would be confusing to have three binaries listed. - query the alias to find out the machine types [Daniel] I haven't looked into the docker part for now. I think Daniel's suggestion of QTEST_QEMU_BINARY_SRC='podman run ... qemu-system-foo' looks interesting. Do we have the latest release already built in the registry at any given point? Thanks v1: https://lore.kernel.org/r/20231003141932.2367-1-farosas@suse.de Hi, I had this WIP patch laying around that seems to fit Juan's vision about testing older machine types. It is a very rough draft for now, but it may be useful for kickstarting the discussion. With this we can give the tests two different QEMU versions. The test picks the older machine type between the two and runs the whole migration-test suite. We'd just need a way to provide the older build. Currently I'm doing this by hand. sample output: # Using two different QEMU binaries. Common machine type: pc-i440fx-8.1 ... # Using ./qemu-system-x86_64 (v8.1.0-952-g8a940312a2-dirty) as migration source ... # Using ../build-8.1.0/qemu-system-x86_64 (v8.1.0-dirty) as migration destination Let me know what you think. Fabiano Rosas (11): tests/qtest: Allow qtest_qemu_binary to use a custom environment variable tests/qtest: Introduce qtest_init_with_env tests/qtest: Allow qtest_get_machines to use an alternate QEMU binary tests/qtest: Introduce qtest_has_machine_with_env tests/qtest: Introduce qtest_resolve_machine_alias tests/qtest/migration: Introduce find_common_machine_version tests/qtest/migration: Define a machine for all architectures tests/qtest/migration: Specify the geometry of the bootsector tests/qtest/migration: Set q35 as the default machine for x86_86 tests/qtest/migration: Support more than one QEMU binary tests/qtest: Don't print messages from query instances tests/qtest/libqtest.c | 86 +++++++++++++++++++++++++++------ tests/qtest/libqtest.h | 32 ++++++++++++ tests/qtest/migration-helpers.c | 24 +++++++++ tests/qtest/migration-helpers.h | 2 + tests/qtest/migration-test.c | 44 +++++++++++++++-- 5 files changed, 169 insertions(+), 19 deletions(-)