@@ -1147,6 +1147,34 @@ variable as shown on the code snippet below to skip the test:
def test(self):
do_something()
+Disabling tests
+---------------
+
+If you want to have a test temporarily disabled (from ``make
+check-acceptance``), you can add an Avocado ``issue`` tag to the test. Example::
+
+ class Feature(Test):
+
+ def test(self):
+ """
+ :avocado: tags=issue
+ """
+ do_something()
+
+Rather than simply adding an ``issue`` tag, it may be helpful to point
+to an actual issue on the project's issue tracker::
+
+ class Feature(Test):
+
+ def test(self):
+ """
+ Reason for disabling this test is documented and tracked at
+ https://gitlab.com/qemu-project/qemu/-/issues/9999
+
+ :avocado: tags=issue:9999
+ """
+ do_something()
+
Uninstalling Avocado
--------------------
@@ -94,7 +94,8 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
# information please refer to "avocado --help".
AVOCADO_SHOW=app
AVOCADO_RUNNER=runner
-AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGETS)))
+comma := ,
+AVOCADO_TAGS=$(patsubst %-softmmu,--filter-by-tags=-issue$(comma)arch:%, $(filter %-softmmu,$(TARGETS)))
$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
$(call quiet-command, \
@@ -485,6 +485,9 @@ def test_arm_raspi2_initrd(self):
"""
:avocado: tags=arch:arm
:avocado: tags=machine:raspi2b
+
+ # https://gitlab.com/qemu-project/qemu/-/issues/636
+ :avocado: tags=issue:636
"""
deb_url = ('http://archive.raspberrypi.org/debian/'
'pool/main/r/raspberrypi-firmware/'
@@ -54,6 +54,9 @@ def test_linux_sash(self):
:avocado: tags=arch:rx
:avocado: tags=machine:gdbsim-r5f562n7
:avocado: tags=endian:little
+
+ # https://gitlab.com/qemu-project/qemu/-/issues/507
+ :avocado: tags=issue:507
"""
dtb_url = ('https://acc.dl.osdn.jp/users/23/23887/rx-virt.dtb')
dtb_hash = '7b4e4e2c71905da44e86ce47adee2210b026ac18'
This introduces a convention, under which tests with known open issues, will be automatically excluded from execution via "make check-acceptance". It should by itself improve the reliability of test jobs, even those with rare failures. Those tests can still be run with: ./tests/venv/bin/avocado run $TEST_ID With the test identification are obtainable via: ./tests/venv/bin/avocado list tests/acceptance And for tests with known issues: ./tests/venv/bin/avocado list -t issue tests/acceptance A list of tests that are enabled (that is, match the currently configured targets, and do not have known issues) can be obtained with: make list-acceptance As a future improvement, a tool can be added to sync the "issue" tags with issues that end up being resolved closed. Signed-off-by: Cleber Rosa <crosa@redhat.com> --- docs/devel/testing.rst | 28 ++++++++++++++++++++++++++ tests/Makefile.include | 3 ++- tests/acceptance/boot_linux_console.py | 3 +++ tests/acceptance/machine_rx_gdbsim.py | 3 +++ 4 files changed, 36 insertions(+), 1 deletion(-)