| Message ID | 20210225232122.1254879-2-crosa@redhat.com |
|---|---|
| State | New |
| Headers | show |
| Series | Acceptance Tests: restore the use of target related information | expand |
On 2/26/21 12:21 AM, Cleber Rosa wrote: > The "get-vm-images" target defined in tests/Makefile.include is a > prerequisite for "check-acceptance", so that those files get > downloaded before the Avocado job even starts. > > It looks like on c401c058a1c a TARGETS variable was introduced with a > different content than it was previously coming from the main > Makefile. From that point on, the "get-vm-images" succeed without > doing anything because there was no matching architecture to download. Any idea about how to detect such side effects (tests silently disabled) automatically?
On Fri, Feb 26, 2021 at 01:01:28AM +0100, Philippe Mathieu-Daudé wrote: > On 2/26/21 12:21 AM, Cleber Rosa wrote: > > The "get-vm-images" target defined in tests/Makefile.include is a > > prerequisite for "check-acceptance", so that those files get > > downloaded before the Avocado job even starts. > > > > It looks like on c401c058a1c a TARGETS variable was introduced with a > > different content than it was previously coming from the main > > Makefile. From that point on, the "get-vm-images" succeed without > > doing anything because there was no matching architecture to download. > > Any idea about how to detect such side effects (tests silently > disabled) automatically? > It wasn't really that any tests were disabled... they all continued to run. In this case it was a broken make rule that caused the download of the images, ahead of time, to not be performed. But your question is still valid and something that could happen. The best answer I have is that all job results could and should also be persisted in a structured way that is succeptible to being queried. Then on top of that, you can build queries to show stability metrics, regressions, etc. To that regards, I can speak about three possibilities: 1) Avocado has support for Fedora's resultsdb[1][2] 2) Because the Acceptance tests are already communicating the test results to GitLab (via junit), using the GitLab API that lets you query the detailed test results 3) In addition to that, Marcelo (cc'd here) has written an Avocado plugin that will export test resutls suitable to be used on a datawarehouse tool developed by the Continuous Kernel Integration project[3]. This is not generally available at the moment, but should be available soon. Regards, - Cleber. [1] - https://taskotron.fedoraproject.org/resultsdb/results [2] - https://avocado-framework.readthedocs.io/en/85.0/plugins/optional/results.html#resultsdb-plugin [3] - https://cki-project.org
On 2/25/21 8:21 PM, Cleber Rosa wrote: > The "get-vm-images" target defined in tests/Makefile.include is a > prerequisite for "check-acceptance", so that those files get > downloaded before the Avocado job even starts. > > It looks like on c401c058a1c a TARGETS variable was introduced with a > different content than it was previously coming from the main > Makefile. From that point on, the "get-vm-images" succeed without > doing anything because there was no matching architecture to download. > > This restores the download of images (that match targets to be built) > before the job starts, eliminating downloads and their associated > failures during the tests. > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > --- > tests/Makefile.include | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com> > > diff --git a/tests/Makefile.include b/tests/Makefile.include > index d34254fb29..dbd53a9de6 100644 > --- a/tests/Makefile.include > +++ b/tests/Makefile.include > @@ -109,7 +109,8 @@ $(TESTS_RESULTS_DIR): > > check-venv: $(TESTS_VENV_DIR) > > -FEDORA_31_ARCHES_CANDIDATES=$(patsubst ppc64,ppc64le,$(TARGETS)) > +FEDORA_31_ARCHES_TARGETS=$(patsubst %-softmmu,%, $(filter %-softmmu,$(TARGETS))) > +FEDORA_31_ARCHES_CANDIDATES=$(patsubst ppc64,ppc64le,$(FEDORA_31_ARCHES_TARGETS)) > FEDORA_31_ARCHES := x86_64 aarch64 ppc64le s390x > FEDORA_31_DOWNLOAD=$(filter $(FEDORA_31_ARCHES),$(FEDORA_31_ARCHES_CANDIDATES)) >
On 2/27/21 12:04 AM, Cleber Rosa wrote: > On Fri, Feb 26, 2021 at 01:01:28AM +0100, Philippe Mathieu-Daudé wrote: >> On 2/26/21 12:21 AM, Cleber Rosa wrote: >>> The "get-vm-images" target defined in tests/Makefile.include is a >>> prerequisite for "check-acceptance", so that those files get >>> downloaded before the Avocado job even starts. >>> >>> It looks like on c401c058a1c a TARGETS variable was introduced with a >>> different content than it was previously coming from the main >>> Makefile. From that point on, the "get-vm-images" succeed without >>> doing anything because there was no matching architecture to download. >> >> Any idea about how to detect such side effects (tests silently >> disabled) automatically? >> > > It wasn't really that any tests were disabled... they all continued to > run. In this case it was a broken make rule that caused the download > of the images, ahead of time, to not be performed. > > But your question is still valid and something that could happen. The > best answer I have is that all job results could and should also be > persisted in a structured way that is succeptible to being queried. > Then on top of that, you can build queries to show stability metrics, > regressions, etc. > > To that regards, I can speak about three possibilities: > > 1) Avocado has support for Fedora's resultsdb[1][2] > > 2) Because the Acceptance tests are already communicating the test > results to GitLab (via junit), using the GitLab API that lets you > query the detailed test results > > 3) In addition to that, Marcelo (cc'd here) has written an Avocado plugin > that will export test resutls suitable to be used on a datawarehouse > tool developed by the Continuous Kernel Integration project[3]. This > is not generally available at the moment, but should be available > soon. > > Regards, > - Cleber. > > [1] - https://taskotron.fedoraproject.org/resultsdb/results > [2] - https://avocado-framework.readthedocs.io/en/85.0/plugins/optional/results.html#resultsdb-plugin > [3] - https://cki-project.org Wow this is thrilling! Maybe we could use fosshost to run a resultsdb VM.
On Thu, Feb 25, 2021 at 8:21 PM Cleber Rosa <crosa@redhat.com> wrote: > > The "get-vm-images" target defined in tests/Makefile.include is a > prerequisite for "check-acceptance", so that those files get > downloaded before the Avocado job even starts. > > It looks like on c401c058a1c a TARGETS variable was introduced with a > different content than it was previously coming from the main > Makefile. From that point on, the "get-vm-images" succeed without > doing anything because there was no matching architecture to download. > > This restores the download of images (that match targets to be built) > before the job starts, eliminating downloads and their associated > failures during the tests. > > Signed-off-by: Cleber Rosa <crosa@redhat.com> > --- > tests/Makefile.include | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > Reviewed-by: Willian Rampazzo <willianr@redhat.com>
diff --git a/tests/Makefile.include b/tests/Makefile.include index d34254fb29..dbd53a9de6 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -109,7 +109,8 @@ $(TESTS_RESULTS_DIR): check-venv: $(TESTS_VENV_DIR) -FEDORA_31_ARCHES_CANDIDATES=$(patsubst ppc64,ppc64le,$(TARGETS)) +FEDORA_31_ARCHES_TARGETS=$(patsubst %-softmmu,%, $(filter %-softmmu,$(TARGETS))) +FEDORA_31_ARCHES_CANDIDATES=$(patsubst ppc64,ppc64le,$(FEDORA_31_ARCHES_TARGETS)) FEDORA_31_ARCHES := x86_64 aarch64 ppc64le s390x FEDORA_31_DOWNLOAD=$(filter $(FEDORA_31_ARCHES),$(FEDORA_31_ARCHES_CANDIDATES))
The "get-vm-images" target defined in tests/Makefile.include is a prerequisite for "check-acceptance", so that those files get downloaded before the Avocado job even starts. It looks like on c401c058a1c a TARGETS variable was introduced with a different content than it was previously coming from the main Makefile. From that point on, the "get-vm-images" succeed without doing anything because there was no matching architecture to download. This restores the download of images (that match targets to be built) before the job starts, eliminating downloads and their associated failures during the tests. Signed-off-by: Cleber Rosa <crosa@redhat.com> --- tests/Makefile.include | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)