mbox series

[RFC,0/2] tests/qtest: Run migration subtests via meson

Message ID 20230703144700.13536-1-farosas@suse.de
Headers show
Series tests/qtest: Run migration subtests via meson | expand

Message

Fabiano Rosas July 3, 2023, 2:46 p.m. UTC 
We've had some issues with the migration-test failing in CI and giving
no clue as to which migration sub-test has failed. I think it would be
an improvement to have each subtest listed individually.

With this we can see which migration test has failed, can set timeouts
individually and can run 'make check-migration' to run only the
migration tests.

It does however increase the amount of lines emitted with make check,
so let me know if this is an issue.

I had to include some hacking due to meson, but I think it is simple
enough and worth the benefits.

CI run: https://gitlab.com/farosas/qemu/-/pipelines/919324510

Fabiano Rosas (2):
  tests/qtest: Add a script to gather migration tests list
  tests/qtest: Pass migration tests individually to meson

 .gitlab-ci.d/windows.yml                |  2 +-
 tests/qtest/gen_migration_tests_list.py | 12 ++++++++
 tests/qtest/meson.build                 | 40 +++++++++++++++++++------
 3 files changed, 44 insertions(+), 10 deletions(-)
 create mode 100644 tests/qtest/gen_migration_tests_list.py

Comments

Daniel P. Berrangé July 3, 2023, 3:17 p.m. UTC | #1 
On Mon, Jul 03, 2023 at 11:46:58AM -0300, Fabiano Rosas wrote:
> We've had some issues with the migration-test failing in CI and giving
> no clue as to which migration sub-test has failed. I think it would be
> an improvement to have each subtest listed individually.

If the migration-test (or any test) actually fails, the meson test
log should show us exactly which test case failed already. We had
some jobs which didn't publish the log as an artifact, but I think
we finally have those all fixed.

If the migration-test hangs, however, then we get stuck until the
GitLab CI job timeout and get no useful info. For this scenario
we need to enable timeouts at the meson level, which are currently
all disabled. I've got a series here to address this:

  https://lists.gnu.org/archive/html/qemu-devel/2023-06/msg00305.html

It needs 

> 
> With this we can see which migration test has failed, can set timeouts
> individually and can run 'make check-migration' to run only the
> migration tests.

IMHO focusing just on migration-test is undesirable, as while this
has been one of the least reliable ones, other tests do also exhibit
hard to debug failures for much the same reasons. IOW, we need
improved debugging across the board rather than specialcased solutions.

With regards,
Daniel