Message ID | 20230303160727.3977246-8-berrange@redhat.com |
---|---|
State | New |
Headers | show |
Series | iotests: make meson aware of individual I/O tests | expand |
On 03/03/2023 17.07, Daniel P. Berrangé wrote: > Currently meson registers a single test that invokes an entire group of > I/O tests, hiding the test granularity from meson. There are various > downsides of doing this > > * You cannot ask 'meson test' to invoke a single I/O test > * The meson test timeout can't be applied to the individual > tests > * Meson only gets a pass/fail for the overall I/O test group > not individual tests > * If a CI job gets killed by the GitLab timeout, we don't > get visibility into how far through the I/O tests > execution got. > > This switches meson to perform test discovery by invoking 'check' in > dry-run mode. It then registers one meson test case for each I/O > test. Parallel execution remains disabled since the I/O tests do not > use self contained execution environments and thus conflict with > each other. > > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> > --- > tests/qemu-iotests/meson.build | 35 ++++++++++++++++++++++++++++------ > 1 file changed, 29 insertions(+), 6 deletions(-) Reviewed-by: Thomas Huth <thuth@redhat.com>
On 03/03/2023 17.07, Daniel P. Berrangé wrote: > Currently meson registers a single test that invokes an entire group of > I/O tests, hiding the test granularity from meson. There are various > downsides of doing this > > * You cannot ask 'meson test' to invoke a single I/O test > * The meson test timeout can't be applied to the individual > tests > * Meson only gets a pass/fail for the overall I/O test group > not individual tests > * If a CI job gets killed by the GitLab timeout, we don't > get visibility into how far through the I/O tests > execution got. > > This switches meson to perform test discovery by invoking 'check' in > dry-run mode. It then registers one meson test case for each I/O > test. Parallel execution remains disabled since the I/O tests do not > use self contained execution environments and thus conflict with > each other. > > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> > --- > tests/qemu-iotests/meson.build | 35 ++++++++++++++++++++++++++++------ > 1 file changed, 29 insertions(+), 6 deletions(-) > > diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build > index 323a4acb6a..a162f683ef 100644 > --- a/tests/qemu-iotests/meson.build > +++ b/tests/qemu-iotests/meson.build > @@ -32,16 +32,39 @@ foreach k, v : emulators > endif > endforeach > > +qemu_iotests_check_cmd = files('check') > + > foreach format, speed: qemu_iotests_formats > if speed == 'quick' > suites = 'block' > else > suites = ['block-' + speed, speed] > endif > - test('qemu-iotests ' + format, sh, args: [files('../check-block.sh'), format], > - depends: qemu_iotests_binaries, env: qemu_iotests_env, > - protocol: 'tap', > - suite: suites, > - timeout: 0, > - is_parallel: false) > + > + args = ['-tap', '-' + format] > + if speed == 'quick' > + args += ['-g', 'auto'] > + endif > + > + rc = run_command( > + [qemu_iotests_check_cmd] + args + ['-n'], > + check: true, > + ) > + > + foreach item: rc.stdout().strip().split() > + args = ['-tap', '-' + format, item, > + '--source-dir', meson.current_source_dir(), > + '--build-dir', meson.current_build_dir()] > + # Some individual tests take as long as 45 seconds > + # Bump the timeout to 3 minutes for some headroom > + # on slow machines to minimize spurious failures > + test('io-' + format + '-' + item, > + qemu_iotests_check_cmd, > + args: args, > + depends: qemu_iotests_binaries, > + env: qemu_iotests_env, > + protocol: 'tap', > + timeout: 180, > + suite: suites) > + endforeach > endforeach Seems like this somehow broke compilation on NetBSD: https://gitlab.com/thuth/qemu/-/jobs/4021584713#L2980 ? Thomas
On Wed, Mar 29, 2023 at 12:47:51PM +0200, Thomas Huth wrote: > On 03/03/2023 17.07, Daniel P. Berrangé wrote: > > Currently meson registers a single test that invokes an entire group of > > I/O tests, hiding the test granularity from meson. There are various > > downsides of doing this > > > > * You cannot ask 'meson test' to invoke a single I/O test > > * The meson test timeout can't be applied to the individual > > tests > > * Meson only gets a pass/fail for the overall I/O test group > > not individual tests > > * If a CI job gets killed by the GitLab timeout, we don't > > get visibility into how far through the I/O tests > > execution got. > > > > This switches meson to perform test discovery by invoking 'check' in > > dry-run mode. It then registers one meson test case for each I/O > > test. Parallel execution remains disabled since the I/O tests do not > > use self contained execution environments and thus conflict with > > each other. > > > > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> > > --- > > tests/qemu-iotests/meson.build | 35 ++++++++++++++++++++++++++++------ > > 1 file changed, 29 insertions(+), 6 deletions(-) > > > > diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build > > index 323a4acb6a..a162f683ef 100644 > > --- a/tests/qemu-iotests/meson.build > > +++ b/tests/qemu-iotests/meson.build > > @@ -32,16 +32,39 @@ foreach k, v : emulators > > endif > > endforeach > > +qemu_iotests_check_cmd = files('check') > > + > > foreach format, speed: qemu_iotests_formats > > if speed == 'quick' > > suites = 'block' > > else > > suites = ['block-' + speed, speed] > > endif > > - test('qemu-iotests ' + format, sh, args: [files('../check-block.sh'), format], > > - depends: qemu_iotests_binaries, env: qemu_iotests_env, > > - protocol: 'tap', > > - suite: suites, > > - timeout: 0, > > - is_parallel: false) > > + > > + args = ['-tap', '-' + format] > > + if speed == 'quick' > > + args += ['-g', 'auto'] > > + endif > > + > > + rc = run_command( > > + [qemu_iotests_check_cmd] + args + ['-n'], > > + check: true, > > + ) > > + > > + foreach item: rc.stdout().strip().split() > > + args = ['-tap', '-' + format, item, > > + '--source-dir', meson.current_source_dir(), > > + '--build-dir', meson.current_build_dir()] > > + # Some individual tests take as long as 45 seconds > > + # Bump the timeout to 3 minutes for some headroom > > + # on slow machines to minimize spurious failures > > + test('io-' + format + '-' + item, > > + qemu_iotests_check_cmd, > > + args: args, > > + depends: qemu_iotests_binaries, > > + env: qemu_iotests_env, > > + protocol: 'tap', > > + timeout: 180, > > + suite: suites) > > + endforeach > > endforeach > > Seems like this somehow broke compilation on NetBSD: > > https://gitlab.com/thuth/qemu/-/jobs/4021584713#L2980 I ran it locally and got the meson-log.txt file which reports env: python3: No such file or directory and indeed there is no python3 binary present in our netbsd VM. our tests/vm/netbsd script works around this by passing an explicit --python=python3.7 arg to configure, but the way we invoke the 'check' script means it is just using the "#!/usr/bin/env python3" logic instead. With regards, Daniel
On 29/03/2023 13.18, Daniel P. Berrangé wrote: > On Wed, Mar 29, 2023 at 12:47:51PM +0200, Thomas Huth wrote: >> On 03/03/2023 17.07, Daniel P. Berrangé wrote: >>> Currently meson registers a single test that invokes an entire group of >>> I/O tests, hiding the test granularity from meson. There are various >>> downsides of doing this >>> >>> * You cannot ask 'meson test' to invoke a single I/O test >>> * The meson test timeout can't be applied to the individual >>> tests >>> * Meson only gets a pass/fail for the overall I/O test group >>> not individual tests >>> * If a CI job gets killed by the GitLab timeout, we don't >>> get visibility into how far through the I/O tests >>> execution got. >>> >>> This switches meson to perform test discovery by invoking 'check' in >>> dry-run mode. It then registers one meson test case for each I/O >>> test. Parallel execution remains disabled since the I/O tests do not >>> use self contained execution environments and thus conflict with >>> each other. >>> >>> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> >>> --- >>> tests/qemu-iotests/meson.build | 35 ++++++++++++++++++++++++++++------ >>> 1 file changed, 29 insertions(+), 6 deletions(-) >>> >>> diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build >>> index 323a4acb6a..a162f683ef 100644 >>> --- a/tests/qemu-iotests/meson.build >>> +++ b/tests/qemu-iotests/meson.build >>> @@ -32,16 +32,39 @@ foreach k, v : emulators >>> endif >>> endforeach >>> +qemu_iotests_check_cmd = files('check') >>> + >>> foreach format, speed: qemu_iotests_formats >>> if speed == 'quick' >>> suites = 'block' >>> else >>> suites = ['block-' + speed, speed] >>> endif >>> - test('qemu-iotests ' + format, sh, args: [files('../check-block.sh'), format], >>> - depends: qemu_iotests_binaries, env: qemu_iotests_env, >>> - protocol: 'tap', >>> - suite: suites, >>> - timeout: 0, >>> - is_parallel: false) >>> + >>> + args = ['-tap', '-' + format] >>> + if speed == 'quick' >>> + args += ['-g', 'auto'] >>> + endif >>> + >>> + rc = run_command( >>> + [qemu_iotests_check_cmd] + args + ['-n'], >>> + check: true, >>> + ) >>> + >>> + foreach item: rc.stdout().strip().split() >>> + args = ['-tap', '-' + format, item, >>> + '--source-dir', meson.current_source_dir(), >>> + '--build-dir', meson.current_build_dir()] >>> + # Some individual tests take as long as 45 seconds >>> + # Bump the timeout to 3 minutes for some headroom >>> + # on slow machines to minimize spurious failures >>> + test('io-' + format + '-' + item, >>> + qemu_iotests_check_cmd, >>> + args: args, >>> + depends: qemu_iotests_binaries, >>> + env: qemu_iotests_env, >>> + protocol: 'tap', >>> + timeout: 180, >>> + suite: suites) >>> + endforeach >>> endforeach >> >> Seems like this somehow broke compilation on NetBSD: >> >> https://gitlab.com/thuth/qemu/-/jobs/4021584713#L2980 > > I ran it locally and got the meson-log.txt file which reports > > env: python3: No such file or directory > > and indeed there is no python3 binary present in our netbsd > VM. > > our tests/vm/netbsd script works around this by passing an > explicit --python=python3.7 arg to configure, but the way > we invoke the 'check' script means it is just using the > "#!/usr/bin/env python3" logic instead. Ah, that rings a bell - python scripts must not have the executable flags, otherwise meson tries to run them directly instead of using the interpreter that has been specified with the --python option. Thomas
diff --git a/tests/qemu-iotests/meson.build b/tests/qemu-iotests/meson.build index 323a4acb6a..a162f683ef 100644 --- a/tests/qemu-iotests/meson.build +++ b/tests/qemu-iotests/meson.build @@ -32,16 +32,39 @@ foreach k, v : emulators endif endforeach +qemu_iotests_check_cmd = files('check') + foreach format, speed: qemu_iotests_formats if speed == 'quick' suites = 'block' else suites = ['block-' + speed, speed] endif - test('qemu-iotests ' + format, sh, args: [files('../check-block.sh'), format], - depends: qemu_iotests_binaries, env: qemu_iotests_env, - protocol: 'tap', - suite: suites, - timeout: 0, - is_parallel: false) + + args = ['-tap', '-' + format] + if speed == 'quick' + args += ['-g', 'auto'] + endif + + rc = run_command( + [qemu_iotests_check_cmd] + args + ['-n'], + check: true, + ) + + foreach item: rc.stdout().strip().split() + args = ['-tap', '-' + format, item, + '--source-dir', meson.current_source_dir(), + '--build-dir', meson.current_build_dir()] + # Some individual tests take as long as 45 seconds + # Bump the timeout to 3 minutes for some headroom + # on slow machines to minimize spurious failures + test('io-' + format + '-' + item, + qemu_iotests_check_cmd, + args: args, + depends: qemu_iotests_binaries, + env: qemu_iotests_env, + protocol: 'tap', + timeout: 180, + suite: suites) + endforeach endforeach
Currently meson registers a single test that invokes an entire group of I/O tests, hiding the test granularity from meson. There are various downsides of doing this * You cannot ask 'meson test' to invoke a single I/O test * The meson test timeout can't be applied to the individual tests * Meson only gets a pass/fail for the overall I/O test group not individual tests * If a CI job gets killed by the GitLab timeout, we don't get visibility into how far through the I/O tests execution got. This switches meson to perform test discovery by invoking 'check' in dry-run mode. It then registers one meson test case for each I/O test. Parallel execution remains disabled since the I/O tests do not use self contained execution environments and thus conflict with each other. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- tests/qemu-iotests/meson.build | 35 ++++++++++++++++++++++++++++------ 1 file changed, 29 insertions(+), 6 deletions(-)