| Message ID | 20221020123506.26363-1-ani@anisinha.ca |
|---|---|
| State | New |
| Headers | show |
On Thu, Oct 20, 2022 at 06:04:56PM +0530, Ani Sinha wrote: > Please see the rst doc file in patch 9 for more details. > Sample runs are as follows: > > Passed tests: > > $ ./tests/venv/bin/avocado run -t acpi tests/avocado > Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits > JOB ID : e95c7413e996bfb59389839e5ca5105464ef098f > JOB LOG : /home/anisinha/avocado/job-results/job-2022-10-14T19.15-e95c741/job.log > (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: PASS (33.08 s) > RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 > JOB TIME : 39.18 s > > > $ ./tests/venv/bin/avocado run ./tests/avocado/acpi-bits.py > Fetching asset from ./tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits > JOB ID : c6e588594bceb2c5fdbdf45095559c58e4030f9c > JOB LOG : /home/anisinha/avocado/job-results/job-2022-10-14T19.13-c6e5885/job.log > (1/1) ./tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: PASS (32.29 s) > RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 > JOB TIME : 39.27 s > > When PSS tests are enabled: > > $ git diff > diff --git a/tests/avocado/acpi-bits/bits-tests/testacpi.py2 b/tests/avocado/acpi-bits/bits-tests/testacpi.py2 > index f818a9cce6..18dc818d62 100644 > --- a/tests/avocado/acpi-bits/bits-tests/testacpi.py2 > +++ b/tests/avocado/acpi-bits/bits-tests/testacpi.py2 > @@ -40,8 +40,8 @@ import time > > def register_tests(): > testsuite.add_test("ACPI _MAT (Multiple APIC Table Entry) under Processor objects", test_mat, submenu="ACPI Tests") > -# testsuite.add_test("ACPI _PSS (Pstate) table conformance tests", test_pss, submenu="ACPI Tests") > -# testsuite.add_test("ACPI _PSS (Pstate) runtime tests", test_pstates, submenu="ACPI Tests") > + testsuite.add_test("ACPI _PSS (Pstate) table conformance tests", test_pss, submenu="ACPI Tests") > + testsuite.add_test("ACPI _PSS (Pstate) runtime tests", test_pstates, submenu="ACPI Tests") > testsuite.add_test("ACPI DSDT (Differentiated System Description Table)", test_dsdt, submenu="ACPI Tests") > testsuite.add_test("ACPI FACP (Fixed ACPI Description Table)", test_facp, submenu="ACPI Tests") > testsuite.add_test("ACPI HPET (High Precision Event Timer Table)", test_hpet, submenu="ACPI Tests") > > $ ./tests/venv/bin/avocado run -t acpi tests/avocado > Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits > JOB ID : 16a8b8c7f8dc87c4b84f8a2f95135bb63464a71e > JOB LOG : /home/anisinha/avocado/job-results/job-2022-10-14T22.43-16a8b8c/job.log > (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: FAIL: '1' != '0'\n- 1\n+ 0\n : Some bits tests seems to have failed. Please check the test logs for more info. (33.01 s) > RESULTS : PASS 0 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 > JOB TIME : 39.17 s > > > Changelog: > v6: > - skip test when dependencies (xorriso for example) are not installed. > - skip test when run on a platform other than x86_64. Hmm why is that btw? > - move README as an rst file under docs/devel. > - updated tags. > v5: > - fixed bits so that grub-mkrescue is built as a static binary. > - updated the test so that it points to the new bits artifact. > - V=1 now spits out more meaningful information on failure. > - added a check to see if xorriso is installed. If not, the test skips with > a meaningful message instructing the user to install it. > - fixed README by removing references to stale instructions and adding > updated instructions. > - tags updated. > v4: > - renamed the bits test files as .py2. > - fixed a bug with missing log file parsing. dump bits logs automatically > for test failures (no need to re-run with V=1). > - fixed issues with README file and enhanced it. > - cosmetic comment updates. > v3: > - converted the existing test to avocado tests as per the popular > recommendation. Added appropriate tags. > - artifact download URL modified based on gitlab-CI URL. > > For biosbits repo: > - moved to a gitlab repo with me being the maintainer. > - added .gitlab-ci.yml file to generate the artifacts. > v2: > - a new class of python based tests introduced that is separate from avocado > tests or qtests. Can be run by using "make check-pytest". > - acpi biosbits tests are the first tests to use pytest environment. > - bios bits tests now download the bits binary archives from a remote > repository if they are not found locally. The test skips if download > fails. > - A new environment variable is introduced that can be passed by the tester > to specify the location of the bits archives locally. test skips if the > bits binaries are not found in that location. > - if pip install of python module fails for whatever reaoson, the test skips. > - misc code fixes including spell check of the README doc. README has been > updated as well. > - addition of SPDX license headers to bits test files. > - update MAINTAINERS to reflect the new pytest test class. > > For biosbits repo: > - added Dockerfile and build script. Made bios bits build on gcc 11. > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/build-artifacts.sh > The build script generates the zip archive and tarball used by the test. > > v1: initial patchset. uses qtest to implement the bios bits tests. > > Cc: Qemu Devel <qemu-devel@nongnu.org> > Cc: Daniel P. Berrangé <berrange@redhat.com> > Cc: Paolo Bonzini <pbonzini@redhat.com> > Cc: Maydell Peter <peter.maydell@linaro.org> > Cc: John Snow <jsnow@redhat.com> > Cc: Thomas Huth <thuth@redhat.com> > Cc: Alex Bennée <alex.bennee@linaro.org> > Cc: Igor Mammedov <imammedo@redhat.com> > Cc: Michael Tsirkin <mst@redhat.com> > > Ani Sinha (10): > acpi/tests/avocado/bits: initial commit of test scripts that are run > by biosbits > acpi/tests/avocado/bits: add SPDX license identifiers for bios bits > tests > acpi/tests/avocado/bits: disable acpi PSS tests that are failing in > biosbits > acpi/tests/avocado/bits: add smilatency test suite from bits in order > to disable it > acpi/tests/avocado/bits: add SPDX license identifiers for bios bits > smilatency tests > acpi/tests/avocado/bits: disable smilatency test since it does not > pass everytime > acpi/tests/avocado/bits: add biosbits config file for running bios > tests > acpi/tests/avocado/bits: add acpi and smbios avocado tests that uses > biosbits > acpi/tests/avocado/bits/doc: add a doc file to describe the acpi bits > test > MAINTAINERS: add myself as the maintainer for acpi biosbits avocado > tests > > MAINTAINERS | 6 + > docs/devel/acpi-bits.rst | 148 + > tests/avocado/acpi-bits.py | 389 +++ > .../acpi-bits/bits-config/bits-cfg.txt | 18 + > tests/avocado/acpi-bits/bits-tests/smbios.py2 | 2434 +++++++++++++++++ > .../acpi-bits/bits-tests/smilatency.py2 | 107 + > .../avocado/acpi-bits/bits-tests/testacpi.py2 | 287 ++ > .../acpi-bits/bits-tests/testcpuid.py2 | 87 + > 8 files changed, 3476 insertions(+) > create mode 100644 docs/devel/acpi-bits.rst > create mode 100644 tests/avocado/acpi-bits.py > create mode 100644 tests/avocado/acpi-bits/bits-config/bits-cfg.txt > create mode 100644 tests/avocado/acpi-bits/bits-tests/smbios.py2 > create mode 100644 tests/avocado/acpi-bits/bits-tests/smilatency.py2 > create mode 100644 tests/avocado/acpi-bits/bits-tests/testacpi.py2 > create mode 100644 tests/avocado/acpi-bits/bits-tests/testcpuid.py2 > > -- > 2.34.1
On Thu, Oct 20, 2022 at 6:08 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Thu, Oct 20, 2022 at 06:04:56PM +0530, Ani Sinha wrote: > > > > > > Changelog: > > v6: > > - skip test when dependencies (xorriso for example) are not installed. > > - skip test when run on a platform other than x86_64. > > Hmm why is that btw? The bits binaries that generate the iso (grub-mkrescue etc) are built for and are known to work only on x86_64 platform. They might also work on amd64 but I do not have one at my disposal at the moment to check. On other platforms, for example 32 bit x86 and non-x86, those binaries will likely not work. The test will fail.
On Thu, Oct 20, 2022 at 06:12:10PM +0530, Ani Sinha wrote: > On Thu, Oct 20, 2022 at 6:08 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Thu, Oct 20, 2022 at 06:04:56PM +0530, Ani Sinha wrote: > > > > > > > > > Changelog: > > > v6: > > > - skip test when dependencies (xorriso for example) are not installed. > > > - skip test when run on a platform other than x86_64. > > > > Hmm why is that btw? > > The bits binaries that generate the iso (grub-mkrescue etc) are built > for and are known to work only on x86_64 platform. They might also > work on amd64 but I do not have one at my disposal at the moment to > check. > On other platforms, for example 32 bit x86 and non-x86, those binaries > will likely not work. The test will fail. confused. I thought we are distributing the iso?
On Thu, Oct 20, 2022 at 6:15 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Thu, Oct 20, 2022 at 06:12:10PM +0530, Ani Sinha wrote: > > On Thu, Oct 20, 2022 at 6:08 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > On Thu, Oct 20, 2022 at 06:04:56PM +0530, Ani Sinha wrote: > > > > > > > > > > > > Changelog: > > > > v6: > > > > - skip test when dependencies (xorriso for example) are not installed. > > > > - skip test when run on a platform other than x86_64. > > > > > > Hmm why is that btw? > > > > The bits binaries that generate the iso (grub-mkrescue etc) are built > > for and are known to work only on x86_64 platform. They might also > > work on amd64 but I do not have one at my disposal at the moment to > > check. > > On other platforms, for example 32 bit x86 and non-x86, those binaries > > will likely not work. The test will fail. > > confused. I thought we are distributing the iso? No, the test builds the iso after adding the modified test scripts and then spawns the vm with it. It is all part of the test itself. We need to do that so that the iso contains the newly added tests etc.
On Thu, Oct 20, 2022 at 06:20:20PM +0530, Ani Sinha wrote: > On Thu, Oct 20, 2022 at 6:15 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Thu, Oct 20, 2022 at 06:12:10PM +0530, Ani Sinha wrote: > > > On Thu, Oct 20, 2022 at 6:08 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > On Thu, Oct 20, 2022 at 06:04:56PM +0530, Ani Sinha wrote: > > > > > > > > > > > > > > > Changelog: > > > > > v6: > > > > > - skip test when dependencies (xorriso for example) are not installed. > > > > > - skip test when run on a platform other than x86_64. > > > > > > > > Hmm why is that btw? > > > > > > The bits binaries that generate the iso (grub-mkrescue etc) are built > > > for and are known to work only on x86_64 platform. They might also > > > work on amd64 but I do not have one at my disposal at the moment to > > > check. > > > On other platforms, for example 32 bit x86 and non-x86, those binaries > > > will likely not work. The test will fail. > > > > confused. I thought we are distributing the iso? > > No, the test builds the iso after adding the modified test scripts and > then spawns the vm with it. It is all part of the test itself. > We need to do that so that the iso contains the newly added tests etc. It's good to have for people developing tests, but for most qemu developers please just have a ready iso and have avocado fetch it. It's important to make tests run on all platforms. We can think about moving iso generation into a VM too but as a first step I guess we can live with a container.
On Fri, Oct 21, 2022 at 12:43 AM Michael S. Tsirkin <mst@redhat.com> wrote: > On Thu, Oct 20, 2022 at 06:20:20PM +0530, Ani Sinha wrote: > > On Thu, Oct 20, 2022 at 6:15 PM Michael S. Tsirkin <mst@redhat.com> > wrote: > > > > > > On Thu, Oct 20, 2022 at 06:12:10PM +0530, Ani Sinha wrote: > > > > On Thu, Oct 20, 2022 at 6:08 PM Michael S. Tsirkin <mst@redhat.com> > wrote: > > > > > > > > > > On Thu, Oct 20, 2022 at 06:04:56PM +0530, Ani Sinha wrote: > > > > > > > > > > > > > > > > > > Changelog: > > > > > > v6: > > > > > > - skip test when dependencies (xorriso for example) are not > installed. > > > > > > - skip test when run on a platform other than x86_64. > > > > > > > > > > Hmm why is that btw? > > > > > > > > The bits binaries that generate the iso (grub-mkrescue etc) are built > > > > for and are known to work only on x86_64 platform. They might also > > > > work on amd64 but I do not have one at my disposal at the moment to > > > > check. > > > > On other platforms, for example 32 bit x86 and non-x86, those > binaries > > > > will likely not work. The test will fail. > > > > > > confused. I thought we are distributing the iso? > > > > No, the test builds the iso after adding the modified test scripts and > > then spawns the vm with it. It is all part of the test itself. > > We need to do that so that the iso contains the newly added tests etc. > > It's good to have for people developing tests, but for most qemu > developers please just have a ready iso and have avocado fetch it. > It's important to make tests run on all platforms. This changes things a lot and goes down the path of where do we check in test code changes? Do we deal with multiple repos? Where do we keep iso? Etc. having a static iso limits us also. If we can think through and come up with a consensus, I can write a second test. Else let's just go with this first. If we are really keen on running tests on multiple platforms, we can have binaries built for those. > > We can think about moving iso generation into a VM too > but as a first step I guess we can live with a container. > > -- > MST > >
On Fri, Oct 21, 2022 at 5:10 AM Ani Sinha <ani@anisinha.ca> wrote: > > > On Fri, Oct 21, 2022 at 12:43 AM Michael S. Tsirkin <mst@redhat.com> > wrote: > >> On Thu, Oct 20, 2022 at 06:20:20PM +0530, Ani Sinha wrote: >> > On Thu, Oct 20, 2022 at 6:15 PM Michael S. Tsirkin <mst@redhat.com> >> wrote: >> > > >> > > On Thu, Oct 20, 2022 at 06:12:10PM +0530, Ani Sinha wrote: >> > > > On Thu, Oct 20, 2022 at 6:08 PM Michael S. Tsirkin <mst@redhat.com> >> wrote: >> > > > > >> > > > > On Thu, Oct 20, 2022 at 06:04:56PM +0530, Ani Sinha wrote: >> > > > > > >> > > > > > >> > > > > > Changelog: >> > > > > > v6: >> > > > > > - skip test when dependencies (xorriso for example) are not >> installed. >> > > > > > - skip test when run on a platform other than x86_64. >> > > > > >> > > > > Hmm why is that btw? >> > > > >> > > > The bits binaries that generate the iso (grub-mkrescue etc) are >> built >> > > > for and are known to work only on x86_64 platform. They might also >> > > > work on amd64 but I do not have one at my disposal at the moment to >> > > > check. >> > > > On other platforms, for example 32 bit x86 and non-x86, those >> binaries >> > > > will likely not work. The test will fail. >> > > >> > > confused. I thought we are distributing the iso? >> > >> > No, the test builds the iso after adding the modified test scripts and >> > then spawns the vm with it. It is all part of the test itself. >> > We need to do that so that the iso contains the newly added tests etc. >> >> It's good to have for people developing tests, but for most qemu >> developers please just have a ready iso and have avocado fetch it. >> It's important to make tests run on all platforms. > > > This changes things a lot and goes down the path of where do we check in > test code changes? Do we deal with multiple repos? Where do we keep iso? > Etc. having a static iso limits us also. > > If we can think through and come up with a consensus, I can write a second > test. Else let's just go with this first. If we are really keen on running > tests on multiple platforms, we can have binaries built for those. > And have multiple platform specific branches in bits that have fixes for those platforms so that bits can run there. Plus the existing test can be enhanced to pull in binaries from those branches based on the platform on which it is being run.
On Fri, Oct 21, 2022 at 05:10:43AM +0530, Ani Sinha wrote: > > > On Fri, Oct 21, 2022 at 12:43 AM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Thu, Oct 20, 2022 at 06:20:20PM +0530, Ani Sinha wrote: > > On Thu, Oct 20, 2022 at 6:15 PM Michael S. Tsirkin <mst@redhat.com> > wrote: > > > > > > On Thu, Oct 20, 2022 at 06:12:10PM +0530, Ani Sinha wrote: > > > > On Thu, Oct 20, 2022 at 6:08 PM Michael S. Tsirkin <mst@redhat.com> > wrote: > > > > > > > > > > On Thu, Oct 20, 2022 at 06:04:56PM +0530, Ani Sinha wrote: > > > > > > > > > > > > > > > > > > Changelog: > > > > > > v6: > > > > > > - skip test when dependencies (xorriso for example) are not > installed. > > > > > > - skip test when run on a platform other than x86_64. > > > > > > > > > > Hmm why is that btw? > > > > > > > > The bits binaries that generate the iso (grub-mkrescue etc) are built > > > > for and are known to work only on x86_64 platform. They might also > > > > work on amd64 but I do not have one at my disposal at the moment to > > > > check. > > > > On other platforms, for example 32 bit x86 and non-x86, those > binaries > > > > will likely not work. The test will fail. > > > > > > confused. I thought we are distributing the iso? > > > > No, the test builds the iso after adding the modified test scripts and > > then spawns the vm with it. It is all part of the test itself. > > We need to do that so that the iso contains the newly added tests etc. > > It's good to have for people developing tests, but for most qemu > developers please just have a ready iso and have avocado fetch it. > It's important to make tests run on all platforms. > > > This changes things a lot and goes down the path of where do we check in test > code changes? Do we deal with multiple repos? Where do we keep iso? Etc. having > a static iso limits us also. It's the same as any firmware really. I don't see much of a difference. > If we can think through and come up with a consensus, I can write a second > test. Else let's just go with this first. We can go ahead for now. > If we are really keen on running > tests on multiple platforms, we can have binaries built for those. > Oh my. Not dealing with this is *exactly* what people use virtualization for. > > > We can think about moving iso generation into a VM too > but as a first step I guess we can live with a container. > > -- > MST > >
On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: > And have multiple platform specific branches in bits that have fixes for those > platforms so that bits can run there. Plus the existing test can be enhanced to > pull in binaries from those branches based on the platform on which it is being > run. > What a mess. Who is going to be testing all these million platforms? All this does nothing at all to help developers avoid bugs and when they do trigger debug the issue. Which is after all why we have testing. Yes once in a very long while we are going to tweak something in the tests, and for that rare occurence it makes sense to periodically rebuild everything, otherwise code bitrots. But the test is supposed to run within a VM anyway, let's have an image and be done with it.
On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: > > And have multiple platform specific branches in bits that have fixes for those > > platforms so that bits can run there. Plus the existing test can be enhanced to > > pull in binaries from those branches based on the platform on which it is being > > run. > > > > What a mess. > Who is going to be testing all these million platforms? I am not talking about branches in QEMU but branches in bits. If you are going to test multiple platforms, you do need to build bits binaries for them. There is no way around it. bits is not all platform independent python. It does have binary executables. Currently bits is built only for the x86 platform. Other platforms are not tested. I doubt if anyone even tried building bits for arm or mips. It makes sense to try things incrementally once we have something going. Lets discuss this on a separate thread. > All this does nothing at all to help developers avoid > bugs and when they do trigger debug the issue. Which is > after all why we have testing. > Yes once in a very long while we are going to tweak > something in the tests, and for that rare occurence > it makes sense to periodically rebuild everything, > otherwise code bitrots. > > But the test is supposed to run within a VM anyway, let's > have an image and be done with it. > > -- > MST >
Ani Sinha <ani@anisinha.ca> writes: > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin <mst@redhat.com> wrote: >> >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: >> > And have multiple platform specific branches in bits that have fixes for those >> > platforms so that bits can run there. Plus the existing test can be enhanced to >> > pull in binaries from those branches based on the platform on which it is being >> > run. >> > >> >> What a mess. >> Who is going to be testing all these million platforms? > > I am not talking about branches in QEMU but branches in bits. > If you are going to test multiple platforms, you do need to build bits > binaries for them. There is no way around it. > bits is not all platform independent python. It does have binary executables. > > Currently bits is built only for the x86 platform. Other platforms are > not tested. I doubt if anyone even tried building bits for arm or > mips. I'm not worried about test bits on other targets, but we do run x86 targets on a number of hosts. The current reliance on a special patched host build tool for only one architecture is the problem. If we just download the iso that problem goes away. > It makes sense to try things incrementally once we have something going. > > Lets discuss this on a separate thread. > >> All this does nothing at all to help developers avoid >> bugs and when they do trigger debug the issue. Which is >> after all why we have testing. >> Yes once in a very long while we are going to tweak >> something in the tests, and for that rare occurence >> it makes sense to periodically rebuild everything, >> otherwise code bitrots. >> >> But the test is supposed to run within a VM anyway, let's >> have an image and be done with it. >> >> -- >> MST >>
On Fri, Oct 21, 2022 at 10:30:09AM +0100, Alex Bennée wrote: > > Ani Sinha <ani@anisinha.ca> writes: > > > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin <mst@redhat.com> wrote: > >> > >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: > >> > And have multiple platform specific branches in bits that have fixes for those > >> > platforms so that bits can run there. Plus the existing test can be enhanced to > >> > pull in binaries from those branches based on the platform on which it is being > >> > run. > >> > > >> > >> What a mess. > >> Who is going to be testing all these million platforms? > > > > I am not talking about branches in QEMU but branches in bits. > > If you are going to test multiple platforms, you do need to build bits > > binaries for them. There is no way around it. > > bits is not all platform independent python. It does have binary executables. > > > > Currently bits is built only for the x86 platform. Other platforms are > > not tested. I doubt if anyone even tried building bits for arm or > > mips. > > I'm not worried about test bits on other targets, but we do run x86 > targets on a number of hosts. The current reliance on a special patched > host build tool for only one architecture is the problem. If we just > download the iso that problem goes away. 👍what he said. > > It makes sense to try things incrementally once we have something going. > > > > Lets discuss this on a separate thread. > > > >> All this does nothing at all to help developers avoid > >> bugs and when they do trigger debug the issue. Which is > >> after all why we have testing. > >> Yes once in a very long while we are going to tweak > >> something in the tests, and for that rare occurence > >> it makes sense to periodically rebuild everything, > >> otherwise code bitrots. > >> > >> But the test is supposed to run within a VM anyway, let's > >> have an image and be done with it. > >> > >> -- > >> MST > >> > > > -- > Alex Bennée
On Fri, Oct 21, 2022 at 3:10 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Fri, Oct 21, 2022 at 10:30:09AM +0100, Alex Bennée wrote: > > > > Ani Sinha <ani@anisinha.ca> writes: > > > > > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > >> > > >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: > > >> > And have multiple platform specific branches in bits that have fixes for those > > >> > platforms so that bits can run there. Plus the existing test can be enhanced to > > >> > pull in binaries from those branches based on the platform on which it is being > > >> > run. > > >> > > > >> > > >> What a mess. > > >> Who is going to be testing all these million platforms? > > > > > > I am not talking about branches in QEMU but branches in bits. > > > If you are going to test multiple platforms, you do need to build bits > > > binaries for them. There is no way around it. > > > bits is not all platform independent python. It does have binary executables. > > > > > > Currently bits is built only for the x86 platform. Other platforms are > > > not tested. I doubt if anyone even tried building bits for arm or > > > mips. > > > > I'm not worried about test bits on other targets, but we do run x86 > > targets on a number of hosts. The current reliance on a special patched > > host build tool for only one architecture is the problem. If we just > > download the iso that problem goes away. > > 👍what he said. Yes, in that case the problem is that upstream bits does not pass all the test out of the box. Hence we are taking this approach of keeping some test scripts in QEMU repo and modifying them. Then generating the iso with the modified scripts. It also helps developers who want to write new tests or make enhancements to existing tests. If modifications need to be made to tests, they need to be versioned. We have gone through the route of not using submodules and I am not going to open that can of worms again. We also have no consensus on where to keep the one time built iso that we can download for this test you are proposing. So let's figure out the above first. Programmatically downloading an iso and running tests within a VM would be a much simpler test than the one I wrote. We can add a subtest or a brand new test anytime if we can figure out the above logistics. > > > > It makes sense to try things incrementally once we have something going. > > > > > > Lets discuss this on a separate thread. > > > > > >> All this does nothing at all to help developers avoid > > >> bugs and when they do trigger debug the issue. Which is > > >> after all why we have testing. > > >> Yes once in a very long while we are going to tweak > > >> something in the tests, and for that rare occurence > > >> it makes sense to periodically rebuild everything, > > >> otherwise code bitrots. > > >> > > >> But the test is supposed to run within a VM anyway, let's > > >> have an image and be done with it. > > >> > > >> -- > > >> MST > > >> > > > > > > -- > > Alex Bennée >
On Fri, Oct 21, 2022 at 13:50 Michael S. Tsirkin <mst@redhat.com> wrote: > On Fri, Oct 21, 2022 at 05:10:43AM +0530, Ani Sinha wrote: > > > > > > On Fri, Oct 21, 2022 at 12:43 AM Michael S. Tsirkin <mst@redhat.com> > wrote: > > > > On Thu, Oct 20, 2022 at 06:20:20PM +0530, Ani Sinha wrote: > > > On Thu, Oct 20, 2022 at 6:15 PM Michael S. Tsirkin <mst@redhat.com > > > > wrote: > > > > > > > > On Thu, Oct 20, 2022 at 06:12:10PM +0530, Ani Sinha wrote: > > > > > On Thu, Oct 20, 2022 at 6:08 PM Michael S. Tsirkin < > mst@redhat.com> > > wrote: > > > > > > > > > > > > On Thu, Oct 20, 2022 at 06:04:56PM +0530, Ani Sinha wrote: > > > > > > > > > > > > > > > > > > > > > Changelog: > > > > > > > v6: > > > > > > > - skip test when dependencies (xorriso for example) are > not > > installed. > > > > > > > - skip test when run on a platform other than x86_64. > > > > > > > > > > > > Hmm why is that btw? > > > > > > > > > > The bits binaries that generate the iso (grub-mkrescue etc) > are built > > > > > for and are known to work only on x86_64 platform. They might > also > > > > > work on amd64 but I do not have one at my disposal at the > moment to > > > > > check. > > > > > On other platforms, for example 32 bit x86 and non-x86, those > > binaries > > > > > will likely not work. The test will fail. > > > > > > > > confused. I thought we are distributing the iso? > > > > > > No, the test builds the iso after adding the modified test scripts > and > > > then spawns the vm with it. It is all part of the test itself. > > > We need to do that so that the iso contains the newly added tests > etc. > > > > It's good to have for people developing tests, but for most qemu > > developers please just have a ready iso and have avocado fetch it. > > It's important to make tests run on all platforms. > > > > > > This changes things a lot and goes down the path of where do we check in > test > > code changes? Do we deal with multiple repos? Where do we keep iso? Etc. > having > > a static iso limits us also. > > It's the same as any firmware really. I don't see much of a difference. Except that the iso is much larger than the current edk2 binaries and there are serious objections to keeping them along with other bios binary blobs in qemu repo.
Ani Sinha <ani@anisinha.ca> writes: > On Fri, Oct 21, 2022 at 3:10 PM Michael S. Tsirkin <mst@redhat.com> wrote: >> >> On Fri, Oct 21, 2022 at 10:30:09AM +0100, Alex Bennée wrote: >> > >> > Ani Sinha <ani@anisinha.ca> writes: >> > >> > > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin <mst@redhat.com> wrote: >> > >> >> > >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: >> > >> > And have multiple platform specific branches in bits that have fixes for those >> > >> > platforms so that bits can run there. Plus the existing test can be enhanced to >> > >> > pull in binaries from those branches based on the platform on which it is being >> > >> > run. >> > >> > >> > >> >> > >> What a mess. >> > >> Who is going to be testing all these million platforms? >> > > >> > > I am not talking about branches in QEMU but branches in bits. >> > > If you are going to test multiple platforms, you do need to build bits >> > > binaries for them. There is no way around it. >> > > bits is not all platform independent python. It does have binary executables. >> > > >> > > Currently bits is built only for the x86 platform. Other platforms are >> > > not tested. I doubt if anyone even tried building bits for arm or >> > > mips. >> > >> > I'm not worried about test bits on other targets, but we do run x86 >> > targets on a number of hosts. The current reliance on a special patched >> > host build tool for only one architecture is the problem. If we just >> > download the iso that problem goes away. >> >> 👍what he said. > > Yes, in that case the problem is that upstream bits does not pass all > the test out of the box. Hence we are taking this approach of keeping > some test scripts in QEMU repo and modifying them. Then generating the > iso with the modified scripts. It also helps developers who want to > write new tests or make enhancements to existing tests. > If modifications need to be made to tests, they need to be versioned. > We have gone through the route of not using submodules and I am not > going to open that can of worms again. We have added a mirror of biosbits to the QEMU project so there is no reason why we can't track changes and modifications there (we do this for TestFloat which is forked from the upstream SoftFloat code). Maintaining and occasionally re-based "vendor" branch shouldn't be too hard and would track the changes we've made for QEMU's purposes. > We also have no consensus on where to keep the one time built iso that > we can download for this test you are proposing. How big is the eventual ISO? If it's small we could just enable some CI steps and serve the ISOs directly as tagged build artefacts from GitLab. > So let's figure out the above first. Programmatically downloading an > iso and running tests within a VM would be a much simpler test than > the one I wrote. We can add a subtest or a brand new test anytime if > we can figure out the above logistics. > >> >> > > It makes sense to try things incrementally once we have something going. >> > > >> > > Lets discuss this on a separate thread. >> > > >> > >> All this does nothing at all to help developers avoid >> > >> bugs and when they do trigger debug the issue. Which is >> > >> after all why we have testing. >> > >> Yes once in a very long while we are going to tweak >> > >> something in the tests, and for that rare occurence >> > >> it makes sense to periodically rebuild everything, >> > >> otherwise code bitrots. >> > >> >> > >> But the test is supposed to run within a VM anyway, let's >> > >> have an image and be done with it. >> > >> >> > >> -- >> > >> MST >> > >> >> > >> > >> > -- >> > Alex Bennée >>
On Fri, 21 Oct, 2022, 5:26 pm Alex Bennée, <alex.bennee@linaro.org> wrote: > > Ani Sinha <ani@anisinha.ca> writes: > > > On Fri, Oct 21, 2022 at 3:10 PM Michael S. Tsirkin <mst@redhat.com> > wrote: > >> > >> On Fri, Oct 21, 2022 at 10:30:09AM +0100, Alex Bennée wrote: > >> > > >> > Ani Sinha <ani@anisinha.ca> writes: > >> > > >> > > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin <mst@redhat.com> > wrote: > >> > >> > >> > >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: > >> > >> > And have multiple platform specific branches in bits that have > fixes for those > >> > >> > platforms so that bits can run there. Plus the existing test can > be enhanced to > >> > >> > pull in binaries from those branches based on the platform on > which it is being > >> > >> > run. > >> > >> > > >> > >> > >> > >> What a mess. > >> > >> Who is going to be testing all these million platforms? > >> > > > >> > > I am not talking about branches in QEMU but branches in bits. > >> > > If you are going to test multiple platforms, you do need to build > bits > >> > > binaries for them. There is no way around it. > >> > > bits is not all platform independent python. It does have binary > executables. > >> > > > >> > > Currently bits is built only for the x86 platform. Other platforms > are > >> > > not tested. I doubt if anyone even tried building bits for arm or > >> > > mips. > >> > > >> > I'm not worried about test bits on other targets, but we do run x86 > >> > targets on a number of hosts. The current reliance on a special > patched > >> > host build tool for only one architecture is the problem. If we just > >> > download the iso that problem goes away. > >> > >> 👍what he said. > > > > Yes, in that case the problem is that upstream bits does not pass all > > the test out of the box. Hence we are taking this approach of keeping > > some test scripts in QEMU repo and modifying them. Then generating the > > iso with the modified scripts. It also helps developers who want to > > write new tests or make enhancements to existing tests. > > If modifications need to be made to tests, they need to be versioned. > > We have gone through the route of not using submodules and I am not > > going to open that can of worms again. > > We have added a mirror of biosbits to the QEMU project so there is no > reason why we can't track changes and modifications there (we do this > for TestFloat which is forked from the upstream SoftFloat code). > The whole idea was that say an acpi developer added support for a new table in QEMU, he should write a corresponding test for bits so that the same table is exercised during run time. Making those changes from a single repo (either directly or through a submodule) makes things lit simpler and also keeps things in sync with each other. If we use separate repos for acpi bits test, it will be another mess when comes to developers adding changes and keeping things in sync. Anyways these things should have been brought up earlier. I'm out of the debate. I've sent v7 , incremental work over the last 6 months in my spare time without getting any pay. So take it or scrap it. 😊 > Maintaining and occasionally re-based "vendor" branch shouldn't be too > hard and would track the changes we've made for QEMU's purposes. > > > We also have no consensus on where to keep the one time built iso that > > we can download for this test you are proposing. > > How big is the eventual ISO? If it's small we could just enable some CI > steps and serve the ISOs directly as tagged build artefacts from GitLab. > > > So let's figure out the above first. Programmatically downloading an > > iso and running tests within a VM would be a much simpler test than > > the one I wrote. We can add a subtest or a brand new test anytime if > > we can figure out the above logistics. > > > >> > >> > > It makes sense to try things incrementally once we have something > going. > >> > > > >> > > Lets discuss this on a separate thread. > >> > > > >> > >> All this does nothing at all to help developers avoid > >> > >> bugs and when they do trigger debug the issue. Which is > >> > >> after all why we have testing. > >> > >> Yes once in a very long while we are going to tweak > >> > >> something in the tests, and for that rare occurence > >> > >> it makes sense to periodically rebuild everything, > >> > >> otherwise code bitrots. > >> > >> > >> > >> But the test is supposed to run within a VM anyway, let's > >> > >> have an image and be done with it. > >> > >> > >> > >> -- > >> > >> MST > >> > >> > >> > > >> > > >> > -- > >> > Alex Bennée > >> > > > -- > Alex Bennée >
On Fri, 21 Oct, 2022, 5:52 pm Ani Sinha, <ani@anisinha.ca> wrote: > > > On Fri, 21 Oct, 2022, 5:26 pm Alex Bennée, <alex.bennee@linaro.org> wrote: > >> >> Ani Sinha <ani@anisinha.ca> writes: >> >> > On Fri, Oct 21, 2022 at 3:10 PM Michael S. Tsirkin <mst@redhat.com> >> wrote: >> >> >> >> On Fri, Oct 21, 2022 at 10:30:09AM +0100, Alex Bennée wrote: >> >> > >> >> > Ani Sinha <ani@anisinha.ca> writes: >> >> > >> >> > > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin <mst@redhat.com> >> wrote: >> >> > >> >> >> > >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: >> >> > >> > And have multiple platform specific branches in bits that have >> fixes for those >> >> > >> > platforms so that bits can run there. Plus the existing test >> can be enhanced to >> >> > >> > pull in binaries from those branches based on the platform on >> which it is being >> >> > >> > run. >> >> > >> > >> >> > >> >> >> > >> What a mess. >> >> > >> Who is going to be testing all these million platforms? >> >> > > >> >> > > I am not talking about branches in QEMU but branches in bits. >> >> > > If you are going to test multiple platforms, you do need to build >> bits >> >> > > binaries for them. There is no way around it. >> >> > > bits is not all platform independent python. It does have binary >> executables. >> >> > > >> >> > > Currently bits is built only for the x86 platform. Other platforms >> are >> >> > > not tested. I doubt if anyone even tried building bits for arm or >> >> > > mips. >> >> > >> >> > I'm not worried about test bits on other targets, but we do run x86 >> >> > targets on a number of hosts. The current reliance on a special >> patched >> >> > host build tool for only one architecture is the problem. If we just >> >> > download the iso that problem goes away. >> >> >> >> 👍what he said. >> > >> > Yes, in that case the problem is that upstream bits does not pass all >> > the test out of the box. Hence we are taking this approach of keeping >> > some test scripts in QEMU repo and modifying them. Then generating the >> > iso with the modified scripts. It also helps developers who want to >> > write new tests or make enhancements to existing tests. >> > If modifications need to be made to tests, they need to be versioned. >> > We have gone through the route of not using submodules and I am not >> > going to open that can of worms again. >> >> We have added a mirror of biosbits to the QEMU project so there is no >> reason why we can't track changes and modifications there (we do this >> for TestFloat which is forked from the upstream SoftFloat code). >> > > The whole idea was that say an acpi developer added support for a new > table in QEMU, he should write a corresponding test for bits so that the > same table is exercised during run time. Making those changes from a single > repo (either directly or through a submodule) makes things lit simpler and > also keeps things in sync with each other. If we use separate repos for > acpi bits test, it will be another mess when comes to developers adding > changes and keeping things in sync. > Not only this. let's look at the developers workflow. (A) check out bits repo. (B) write new test. (C) build the bits iso. (D) get back to QEMU repo. (E) point the test to the new iso so that the test gets executed. (F) oops something failed. So let's rinse and repeat. (G) once ready, send a PR for bits repo. update tags and figure out how gitlab ci works so that the QEMU test can point to it. To do that figure out the artefact hash and other parameters. (H) send a patch for QEMU repo to update the test to point to new iso. How complicated is that? How complicated will it be for the reviewer? Right now the developer can simply make changes from a single repo and run a avocado test and check logs for failures. Once test is fixed, they can run the test again to make sure everything passes. Once done, commit the test in QEMU repo. If the test exercises a new table we make sure that the commits adding the new table is already present before the test that exercises it is committed. Send a patch for review. The reviewer applies the patch and simply runs the avocado test from QEMU repo. Everything is in one place. No back and forth between two repos. A lot like "make check".
Ani Sinha <ani@anisinha.ca> writes: > On Fri, 21 Oct, 2022, 5:52 pm Ani Sinha, <ani@anisinha.ca> wrote: > > On Fri, 21 Oct, 2022, 5:26 pm Alex Bennée, <alex.bennee@linaro.org> wrote: > > Ani Sinha <ani@anisinha.ca> writes: > > > On Fri, Oct 21, 2022 at 3:10 PM Michael S. Tsirkin <mst@redhat.com> wrote: > >> > >> On Fri, Oct 21, 2022 at 10:30:09AM +0100, Alex Bennée wrote: > >> > > >> > Ani Sinha <ani@anisinha.ca> writes: > >> > > >> > > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin <mst@redhat.com> wrote: > >> > >> > >> > >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: > >> > >> > And have multiple platform specific branches in bits that have fixes for those > >> > >> > platforms so that bits can run there. Plus the existing test can be enhanced to > >> > >> > pull in binaries from those branches based on the platform on which it is being > >> > >> > run. > >> > >> > > >> > >> > >> > >> What a mess. > >> > >> Who is going to be testing all these million platforms? > >> > > > >> > > I am not talking about branches in QEMU but branches in bits. > >> > > If you are going to test multiple platforms, you do need to build bits > >> > > binaries for them. There is no way around it. > >> > > bits is not all platform independent python. It does have binary executables. > >> > > > >> > > Currently bits is built only for the x86 platform. Other platforms are > >> > > not tested. I doubt if anyone even tried building bits for arm or > >> > > mips. > >> > > >> > I'm not worried about test bits on other targets, but we do run x86 > >> > targets on a number of hosts. The current reliance on a special patched > >> > host build tool for only one architecture is the problem. If we just > >> > download the iso that problem goes away. > >> > >> 👍what he said. > > > > Yes, in that case the problem is that upstream bits does not pass all > > the test out of the box. Hence we are taking this approach of keeping > > some test scripts in QEMU repo and modifying them. Then generating the > > iso with the modified scripts. It also helps developers who want to > > write new tests or make enhancements to existing tests. > > If modifications need to be made to tests, they need to be versioned. > > We have gone through the route of not using submodules and I am not > > going to open that can of worms again. > > We have added a mirror of biosbits to the QEMU project so there is no > reason why we can't track changes and modifications there (we do this > for TestFloat which is forked from the upstream SoftFloat code). > > The whole idea was that say an acpi developer added support for a new table in QEMU, he should write a > corresponding test for bits so that the same table is exercised during run time. Making those changes from a single > repo (either directly or through a submodule) makes things lit simpler and also keeps things in sync with each > other. If we use separate repos for acpi bits test, it will be another mess when comes to developers adding changes > and keeping things in sync. For people that care about ACPI it shouldn't be that hard. Most QEMU developers have separate repos of test cases that aren't directly integrated into QEMU for various things (e.g. RISU, semihosting, baremetal, kvm-unit-tests, LTP). > Not only this. let's look at the developers workflow. > > (A) check out bits repo. > (B) write new test. > (C) build the bits iso. > (D) get back to QEMU repo. > (E) point the test to the new iso so that the test gets executed. This seems like a long winded workflow. Usually you test your binaries before integrating them into the acceptance tests. All you need is a script to launch qemu (either system or point at a developer binary) and run directly. Only once you are happy with the final ISO would you upload and then integrate into check-acceptance. > (F) oops something failed. So let's rinse and repeat. > (G) once ready, send a PR for bits repo. update tags and figure out how gitlab ci works so that the QEMU test can point to > it. To do that figure out the artefact hash and other parameters. > (H) send a patch for QEMU repo to update the test to point to new iso. > > How complicated is that? How complicated will it be for the reviewer? Right now the developer can simply make changes > from a single repo and run a avocado test and check logs for failures. Once test is fixed, they can run the test again to > make sure everything passes. Once done, commit the test in QEMU repo. If the test exercises a new table we make sure > that the commits adding the new table is already present before the test that exercises it is committed. Send a patch for > review. The reviewer applies the patch and simply runs the avocado test from QEMU repo. Everything is in one place. No > back and forth between two repos. A lot like "make check". We do indeed build tests for a lot of make check (unit, qtest, tcg) but they build on all our host architectures and have configure machinery to make them optional if host binaries are missing. For avocado tests we typically are using other peoples binaries so this series is a departure from that model.
Ani Sinha <ani@anisinha.ca> writes: <snip> > > We have added a mirror of biosbits to the QEMU project so there is no > reason why we can't track changes and modifications there (we do this > for TestFloat which is forked from the upstream SoftFloat code). > > The whole idea was that say an acpi developer added support for a new table in QEMU, he should write a corresponding > test for bits so that the same table is exercised during run time. Making those changes from a single repo (either directly > or through a submodule) makes things lit simpler and also keeps things in sync with each other. If we use separate > repos for acpi bits test, it will be another mess when comes to developers adding changes and keeping things in sync. > > Anyways these things should have been brought up earlier. I'm out of the debate. > > I've sent v7 , incremental work over the last 6 months in my spare time without getting any pay. So take it or scrap it. > > 😊 I'm sorry you feel that way but you asked for people to spend their time reviewing your code and giving feedback. If you don't have time to take it forward then fine but I'm afraid a "take it or leave it" attitude will just result in it not getting merged. Hopefully someone else who has the time to look into it will be able to pick it up later.
On Fri, Oct 21, 2022 at 21:32 Alex Bennée <alex.bennee@linaro.org> wrote: > > Ani Sinha <ani@anisinha.ca> writes: > > > On Fri, 21 Oct, 2022, 5:52 pm Ani Sinha, <ani@anisinha.ca> wrote: > > > > On Fri, 21 Oct, 2022, 5:26 pm Alex Bennée, <alex.bennee@linaro.org> > wrote: > > > > Ani Sinha <ani@anisinha.ca> writes: > > > > > On Fri, Oct 21, 2022 at 3:10 PM Michael S. Tsirkin <mst@redhat.com> > wrote: > > >> > > >> On Fri, Oct 21, 2022 at 10:30:09AM +0100, Alex Bennée wrote: > > >> > > > >> > Ani Sinha <ani@anisinha.ca> writes: > > >> > > > >> > > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin < > mst@redhat.com> wrote: > > >> > >> > > >> > >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: > > >> > >> > And have multiple platform specific branches in bits that > have fixes for those > > >> > >> > platforms so that bits can run there. Plus the existing test > can be enhanced to > > >> > >> > pull in binaries from those branches based on the platform on > which it is being > > >> > >> > run. > > >> > >> > > > >> > >> > > >> > >> What a mess. > > >> > >> Who is going to be testing all these million platforms? > > >> > > > > >> > > I am not talking about branches in QEMU but branches in bits. > > >> > > If you are going to test multiple platforms, you do need to > build bits > > >> > > binaries for them. There is no way around it. > > >> > > bits is not all platform independent python. It does have binary > executables. > > >> > > > > >> > > Currently bits is built only for the x86 platform. Other > platforms are > > >> > > not tested. I doubt if anyone even tried building bits for arm or > > >> > > mips. > > >> > > > >> > I'm not worried about test bits on other targets, but we do run x86 > > >> > targets on a number of hosts. The current reliance on a special > patched > > >> > host build tool for only one architecture is the problem. If we > just > > >> > download the iso that problem goes away. > > >> > > >> 👍what he said. > > > > > > Yes, in that case the problem is that upstream bits does not pass all > > > the test out of the box. Hence we are taking this approach of keeping > > > some test scripts in QEMU repo and modifying them. Then generating the > > > iso with the modified scripts. It also helps developers who want to > > > write new tests or make enhancements to existing tests. > > > If modifications need to be made to tests, they need to be versioned. > > > We have gone through the route of not using submodules and I am not > > > going to open that can of worms again. > > > > We have added a mirror of biosbits to the QEMU project so there is no > > reason why we can't track changes and modifications there (we do this > > for TestFloat which is forked from the upstream SoftFloat code). > > > > The whole idea was that say an acpi developer added support for a new > table in QEMU, he should write a > > corresponding test for bits so that the same table is exercised during > run time. Making those changes from a single > > repo (either directly or through a submodule) makes things lit simpler > and also keeps things in sync with each > > other. If we use separate repos for acpi bits test, it will be another > mess when comes to developers adding changes > > and keeping things in sync. > > For people that care about ACPI it shouldn't be that hard. People who submit patches for acpi come from all over the place and they mostly care about the qemu source tree and not any other repos. Most QEMU > developers have separate repos of test cases that aren't directly > integrated into QEMU for various things (e.g. RISU, semihosting, > baremetal, kvm-unit-tests, LTP). > > > Not only this. let's look at the developers workflow. > > > > (A) check out bits repo. > > (B) write new test. > > (C) build the bits iso. > > (D) get back to QEMU repo. > > (E) point the test to the new iso so that the test gets executed. > > This seems like a long winded workflow. Usually you test your binaries > before integrating them into the acceptance tests. All you need is a > script to launch qemu (either system or point at a developer binary) and > run directly. Only once you are happy with the final ISO would you > upload and then integrate into check-acceptance. > > > (F) oops something failed. So let's rinse and repeat. > > (G) once ready, send a PR for bits repo. update tags and figure out how > gitlab ci works so that the QEMU test can point to > > it. To do that figure out the artefact hash and other parameters. > > (H) send a patch for QEMU repo to update the test to point to new iso. > > > > How complicated is that? How complicated will it be for the reviewer? > Right now the developer can simply make changes > > from a single repo and run a avocado test and check logs for failures. > Once test is fixed, they can run the test again to > > make sure everything passes. Once done, commit the test in QEMU repo. If > the test exercises a new table we make sure > > that the commits adding the new table is already present before the test > that exercises it is committed. Send a patch for > > review. The reviewer applies the patch and simply runs the avocado test > from QEMU repo. Everything is in one place. No > > back and forth between two repos. A lot like "make check". > > We do indeed build tests for a lot of make check (unit, qtest, tcg) but > they build on all our host architectures and have configure machinery to > make them optional if host binaries are missing. For avocado tests we > typically are using other peoples binaries so this series is a departure > from that model. Yea so if you are using other peoples binaries you should not assume that they will work on all host architectures.
On Fri, Oct 21, 2022 at 21:36 Alex Bennée <alex.bennee@linaro.org> wrote: > > Ani Sinha <ani@anisinha.ca> writes: > > <snip> > > > > We have added a mirror of biosbits to the QEMU project so there is no > > reason why we can't track changes and modifications there (we do this > > for TestFloat which is forked from the upstream SoftFloat code). > > > > The whole idea was that say an acpi developer added support for a new > table in QEMU, he should write a corresponding > > test for bits so that the same table is exercised during run time. > Making those changes from a single repo (either directly > > or through a submodule) makes things lit simpler and also keeps things > in sync with each other. If we use separate > > repos for acpi bits test, it will be another mess when comes to > developers adding changes and keeping things in sync. > > > > Anyways these things should have been brought up earlier. I'm out of the > debate. > > > > I've sent v7 , incremental work over the last 6 months in my spare time > without getting any pay. So take it or scrap it. > > > > 😊 > > I'm sorry you feel that way but you asked for people to spend their time > reviewing your code and giving feedback. I’m sorry but you don’t understand. Giving feedback is one thing. Asking people to make major changes to their design late in the game is completely not acceptable imho and outright rude. . I’ve been sending out the patch series with cc to anyone who should remotely care about this stuff. So if major rework is needed it should be pointed out earlier and not later. I’ve begged people on irc even to take look at this stuff including Igor and others. What else do you expect me to do? If you don't have time to take > it forward then fine I’m extremely discouraged by this and have lost all motivation. It’s not that I have no time. I squeeze all my spare time to work on this beyond my paid responsibilities. Please be respectful of that. but I'm afraid a "take it or leave it" attitude > will just result in it not getting merged. > > Hopefully someone else who has the time to look into it will be able to > pick it up later. > > -- > Alex Bennée >
On Fri, Oct 21, 2022 at 21:32 Alex Bennée <alex.bennee@linaro.org> wrote: > > Ani Sinha <ani@anisinha.ca> writes: > > > On Fri, 21 Oct, 2022, 5:52 pm Ani Sinha, <ani@anisinha.ca> wrote: > > > > On Fri, 21 Oct, 2022, 5:26 pm Alex Bennée, <alex.bennee@linaro.org> > wrote: > > > > Ani Sinha <ani@anisinha.ca> writes: > > > > > On Fri, Oct 21, 2022 at 3:10 PM Michael S. Tsirkin <mst@redhat.com> > wrote: > > >> > > >> On Fri, Oct 21, 2022 at 10:30:09AM +0100, Alex Bennée wrote: > > >> > > > >> > Ani Sinha <ani@anisinha.ca> writes: > > >> > > > >> > > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin < > mst@redhat.com> wrote: > > >> > >> > > >> > >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: > > >> > >> > And have multiple platform specific branches in bits that > have fixes for those > > >> > >> > platforms so that bits can run there. Plus the existing test > can be enhanced to > > >> > >> > pull in binaries from those branches based on the platform on > which it is being > > >> > >> > run. > > >> > >> > > > >> > >> > > >> > >> What a mess. > > >> > >> Who is going to be testing all these million platforms? > > >> > > > > >> > > I am not talking about branches in QEMU but branches in bits. > > >> > > If you are going to test multiple platforms, you do need to > build bits > > >> > > binaries for them. There is no way around it. > > >> > > bits is not all platform independent python. It does have binary > executables. > > >> > > > > >> > > Currently bits is built only for the x86 platform. Other > platforms are > > >> > > not tested. I doubt if anyone even tried building bits for arm or > > >> > > mips. > > >> > > > >> > I'm not worried about test bits on other targets, but we do run x86 > > >> > targets on a number of hosts. The current reliance on a special > patched > > >> > host build tool for only one architecture is the problem. If we > just > > >> > download the iso that problem goes away. > > >> > > >> 👍what he said. > > > > > > Yes, in that case the problem is that upstream bits does not pass all > > > the test out of the box. Hence we are taking this approach of keeping > > > some test scripts in QEMU repo and modifying them. Then generating the > > > iso with the modified scripts. It also helps developers who want to > > > write new tests or make enhancements to existing tests. > > > If modifications need to be made to tests, they need to be versioned. > > > We have gone through the route of not using submodules and I am not > > > going to open that can of worms again. > > > > We have added a mirror of biosbits to the QEMU project so there is no > > reason why we can't track changes and modifications there (we do this > > for TestFloat which is forked from the upstream SoftFloat code). One last option. Commit this patch set but also double commit patch 3 to the bits repo so that we can build an iso that would successfully run all tests for a separate platform independent test to be written later. Then we will have two tests: - this one for developers writing new test. - platform independent one for a basic sanity. I’m just documenting the fact that I have proposed ideas that can work where all can be happy. It’s up to others to take it or keep objecting and killing motivations for freelance contributors. > > > >
On Fri, Oct 21, 2022 at 9:32 PM Alex Bennée <alex.bennee@linaro.org> wrote: > > Ani Sinha <ani@anisinha.ca> writes: > > > On Fri, 21 Oct, 2022, 5:52 pm Ani Sinha, <ani@anisinha.ca> wrote: > > > > On Fri, 21 Oct, 2022, 5:26 pm Alex Bennée, <alex.bennee@linaro.org> > wrote: > > > > Ani Sinha <ani@anisinha.ca> writes: > > > > > On Fri, Oct 21, 2022 at 3:10 PM Michael S. Tsirkin <mst@redhat.com> > wrote: > > >> > > >> On Fri, Oct 21, 2022 at 10:30:09AM +0100, Alex Bennée wrote: > > >> > > > >> > Ani Sinha <ani@anisinha.ca> writes: > > >> > > > >> > > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin < > mst@redhat.com> wrote: > > >> > >> > > >> > >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: > > >> > >> > And have multiple platform specific branches in bits that > have fixes for those > > >> > >> > platforms so that bits can run there. Plus the existing test > can be enhanced to > > >> > >> > pull in binaries from those branches based on the platform on > which it is being > > >> > >> > run. > > >> > >> > > > >> > >> > > >> > >> What a mess. > > >> > >> Who is going to be testing all these million platforms? > > >> > > > > >> > > I am not talking about branches in QEMU but branches in bits. > > >> > > If you are going to test multiple platforms, you do need to > build bits > > >> > > binaries for them. There is no way around it. > > >> > > bits is not all platform independent python. It does have binary > executables. > > >> > > > > >> > > Currently bits is built only for the x86 platform. Other > platforms are > > >> > > not tested. I doubt if anyone even tried building bits for arm or > > >> > > mips. > > >> > > > >> > I'm not worried about test bits on other targets, but we do run x86 > > >> > targets on a number of hosts. The current reliance on a special > patched > > >> > host build tool for only one architecture is the problem. If we > just > > >> > download the iso that problem goes away. > > >> > > >> 👍what he said. > > > > > > Yes, in that case the problem is that upstream bits does not pass all > > > the test out of the box. Hence we are taking this approach of keeping > > > some test scripts in QEMU repo and modifying them. Then generating the > > > iso with the modified scripts. It also helps developers who want to > > > write new tests or make enhancements to existing tests. > > > If modifications need to be made to tests, they need to be versioned. > > > We have gone through the route of not using submodules and I am not > > > going to open that can of worms again. > > > > We have added a mirror of biosbits to the QEMU project so there is no > > reason why we can't track changes and modifications there (we do this > > for TestFloat which is forked from the upstream SoftFloat code). > > > > The whole idea was that say an acpi developer added support for a new > table in QEMU, he should write a > > corresponding test for bits so that the same table is exercised during > run time. Making those changes from a single > > repo (either directly or through a submodule) makes things lit simpler > and also keeps things in sync with each > > other. If we use separate repos for acpi bits test, it will be another > mess when comes to developers adding changes > > and keeping things in sync. > > For people that care about ACPI it shouldn't be that hard. Most QEMU > developers have separate repos of test cases that aren't directly > integrated into QEMU for various things (e.g. RISU, semihosting, > baremetal, kvm-unit-tests, LTP). > > > Not only this. let's look at the developers workflow. > > > > (A) check out bits repo. > > (B) write new test. > > (C) build the bits iso. > > (D) get back to QEMU repo. > > (E) point the test to the new iso so that the test gets executed. > > This seems like a long winded workflow. Usually you test your binaries > before integrating them into the acceptance tests. All you need is a > script to launch qemu (either system or point at a developer binary) and > run directly. Only once you are happy with the final ISO And how would you know which iso is the final iso? You need several trial runs with added test and building bits iso with it. Each time an iso is generated, it must be launched with a qemu using the right command line which the test automatically generates for the developer. Hence you either need to examine what command line the test uses or use the test script directly with the iso you generated and collect the logs. The test script is in one repo, the bits is in one repo. Since bits is not a sub module, the developer has to check out both repos separately and go back and forth. And how are you supposed to explain all this to a guy who submitted a acpi patch? You don't need to because you don't review acpi patches but I do. My last 2 cents. >
On Sat, Oct 22, 2022 at 06:28:32AM +0530, Ani Sinha wrote: > > > On Fri, Oct 21, 2022 at 21:32 Alex Bennée <alex.bennee@linaro.org> wrote: > > > Ani Sinha <ani@anisinha.ca> writes: > > > On Fri, 21 Oct, 2022, 5:52 pm Ani Sinha, <ani@anisinha.ca> wrote: > > > > On Fri, 21 Oct, 2022, 5:26 pm Alex Bennée, <alex.bennee@linaro.org> > wrote: > > > > Ani Sinha <ani@anisinha.ca> writes: > > > > > On Fri, Oct 21, 2022 at 3:10 PM Michael S. Tsirkin <mst@redhat.com> > wrote: > > >> > > >> On Fri, Oct 21, 2022 at 10:30:09AM +0100, Alex Bennée wrote: > > >> > > > >> > Ani Sinha <ani@anisinha.ca> writes: > > >> > > > >> > > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin < > mst@redhat.com> wrote: > > >> > >> > > >> > >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: > > >> > >> > And have multiple platform specific branches in bits that have > fixes for those > > >> > >> > platforms so that bits can run there. Plus the existing test > can be enhanced to > > >> > >> > pull in binaries from those branches based on the platform on > which it is being > > >> > >> > run. > > >> > >> > > > >> > >> > > >> > >> What a mess. > > >> > >> Who is going to be testing all these million platforms? > > >> > > > > >> > > I am not talking about branches in QEMU but branches in bits. > > >> > > If you are going to test multiple platforms, you do need to build > bits > > >> > > binaries for them. There is no way around it. > > >> > > bits is not all platform independent python. It does have binary > executables. > > >> > > > > >> > > Currently bits is built only for the x86 platform. Other > platforms are > > >> > > not tested. I doubt if anyone even tried building bits for arm or > > >> > > mips. > > >> > > > >> > I'm not worried about test bits on other targets, but we do run x86 > > >> > targets on a number of hosts. The current reliance on a special > patched > > >> > host build tool for only one architecture is the problem. If we > just > > >> > download the iso that problem goes away. > > >> > > >> 👍what he said. > > > > > > Yes, in that case the problem is that upstream bits does not pass all > > > the test out of the box. Hence we are taking this approach of keeping > > > some test scripts in QEMU repo and modifying them. Then generating the > > > iso with the modified scripts. It also helps developers who want to > > > write new tests or make enhancements to existing tests. > > > If modifications need to be made to tests, they need to be versioned. > > > We have gone through the route of not using submodules and I am not > > > going to open that can of worms again. > > > > We have added a mirror of biosbits to the QEMU project so there is no > > reason why we can't track changes and modifications there (we do this > > for TestFloat which is forked from the upstream SoftFloat code). > > > One last option. Commit this patch set but also double commit patch 3 to the > bits repo so that we can build an iso that would successfully run all tests for > a separate platform independent test to be written later. > > Then we will have two tests: > > - this one for developers writing new test. > - platform independent one for a basic sanity. > > I’m just documenting the fact that I have proposed ideas that can work where > all can be happy. It’s up to others to take it or keep objecting and killing > motivations for freelance contributors. I think it's ok to apply this as is for starters. Anyone has objections? Down the road I think things should be refactored slightly to work as follows: - test developers can check out biosbits repo to create the iso - everyone else gets iso downloaded Objections to this plan? > > > > > > > >
On Fri, Oct 21, 2022 at 05:12:07PM +0530, Ani Sinha wrote: > > > On Fri, Oct 21, 2022 at 13:50 Michael S. Tsirkin <mst@redhat.com> wrote: > > On Fri, Oct 21, 2022 at 05:10:43AM +0530, Ani Sinha wrote: > > > > > > On Fri, Oct 21, 2022 at 12:43 AM Michael S. Tsirkin <mst@redhat.com> > wrote: > > > > On Thu, Oct 20, 2022 at 06:20:20PM +0530, Ani Sinha wrote: > > > On Thu, Oct 20, 2022 at 6:15 PM Michael S. Tsirkin <mst@redhat.com> > > wrote: > > > > > > > > On Thu, Oct 20, 2022 at 06:12:10PM +0530, Ani Sinha wrote: > > > > > On Thu, Oct 20, 2022 at 6:08 PM Michael S. Tsirkin < > mst@redhat.com> > > wrote: > > > > > > > > > > > > On Thu, Oct 20, 2022 at 06:04:56PM +0530, Ani Sinha wrote: > > > > > > > > > > > > > > > > > > > > > Changelog: > > > > > > > v6: > > > > > > > - skip test when dependencies (xorriso for example) are > not > > installed. > > > > > > > - skip test when run on a platform other than x86_64. > > > > > > > > > > > > Hmm why is that btw? > > > > > > > > > > The bits binaries that generate the iso (grub-mkrescue etc) are > built > > > > > for and are known to work only on x86_64 platform. They might > also > > > > > work on amd64 but I do not have one at my disposal at the > moment to > > > > > check. > > > > > On other platforms, for example 32 bit x86 and non-x86, those > > binaries > > > > > will likely not work. The test will fail. > > > > > > > > confused. I thought we are distributing the iso? > > > > > > No, the test builds the iso after adding the modified test scripts > and > > > then spawns the vm with it. It is all part of the test itself. > > > We need to do that so that the iso contains the newly added tests > etc. > > > > It's good to have for people developing tests, but for most qemu > > developers please just have a ready iso and have avocado fetch it. > > It's important to make tests run on all platforms. > > > > > > This changes things a lot and goes down the path of where do we check in > test > > code changes? Do we deal with multiple repos? Where do we keep iso? Etc. > having > > a static iso limits us also. > > It's the same as any firmware really. I don't see much of a difference. > > > Except that the iso is much larger than the current edk2 binaries and there are > serious objections to keeping them along with other bios binary blobs in qemu > repo. > Right, I get that. So avocado was supposed to know how to download them automatically, no?
On Sat, Oct 22, 2022 at 22:05 Michael S. Tsirkin <mst@redhat.com> wrote: > On Sat, Oct 22, 2022 at 06:28:32AM +0530, Ani Sinha wrote: > > > > > > On Fri, Oct 21, 2022 at 21:32 Alex Bennée <alex.bennee@linaro.org> > wrote: > > > > > > Ani Sinha <ani@anisinha.ca> writes: > > > > > On Fri, 21 Oct, 2022, 5:52 pm Ani Sinha, <ani@anisinha.ca> wrote: > > > > > > On Fri, 21 Oct, 2022, 5:26 pm Alex Bennée, < > alex.bennee@linaro.org> > > wrote: > > > > > > Ani Sinha <ani@anisinha.ca> writes: > > > > > > > On Fri, Oct 21, 2022 at 3:10 PM Michael S. Tsirkin < > mst@redhat.com> > > wrote: > > > >> > > > >> On Fri, Oct 21, 2022 at 10:30:09AM +0100, Alex Bennée wrote: > > > >> > > > > >> > Ani Sinha <ani@anisinha.ca> writes: > > > >> > > > > >> > > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin < > > mst@redhat.com> wrote: > > > >> > >> > > > >> > >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: > > > >> > >> > And have multiple platform specific branches in bits > that have > > fixes for those > > > >> > >> > platforms so that bits can run there. Plus the existing > test > > can be enhanced to > > > >> > >> > pull in binaries from those branches based on the > platform on > > which it is being > > > >> > >> > run. > > > >> > >> > > > > >> > >> > > > >> > >> What a mess. > > > >> > >> Who is going to be testing all these million platforms? > > > >> > > > > > >> > > I am not talking about branches in QEMU but branches in > bits. > > > >> > > If you are going to test multiple platforms, you do need > to build > > bits > > > >> > > binaries for them. There is no way around it. > > > >> > > bits is not all platform independent python. It does have > binary > > executables. > > > >> > > > > > >> > > Currently bits is built only for the x86 platform. Other > > platforms are > > > >> > > not tested. I doubt if anyone even tried building bits for > arm or > > > >> > > mips. > > > >> > > > > >> > I'm not worried about test bits on other targets, but we do > run x86 > > > >> > targets on a number of hosts. The current reliance on a > special > > patched > > > >> > host build tool for only one architecture is the problem. > If we > > just > > > >> > download the iso that problem goes away. > > > >> > > > >> 👍what he said. > > > > > > > > Yes, in that case the problem is that upstream bits does not > pass all > > > > the test out of the box. Hence we are taking this approach of > keeping > > > > some test scripts in QEMU repo and modifying them. Then > generating the > > > > iso with the modified scripts. It also helps developers who > want to > > > > write new tests or make enhancements to existing tests. > > > > If modifications need to be made to tests, they need to be > versioned. > > > > We have gone through the route of not using submodules and I am > not > > > > going to open that can of worms again. > > > > > > We have added a mirror of biosbits to the QEMU project so there > is no > > > reason why we can't track changes and modifications there (we do > this > > > for TestFloat which is forked from the upstream SoftFloat code). > > > > > > One last option. Commit this patch set but also double commit patch 3 to > the > > bits repo so that we can build an iso that would successfully run all > tests for > > a separate platform independent test to be written later. > > > > Then we will have two tests: > > > > - this one for developers writing new test. > > - platform independent one for a basic sanity. > > > > I’m just documenting the fact that I have proposed ideas that can work > where > > all can be happy. It’s up to others to take it or keep objecting and > killing > > motivations for freelance contributors. > > I think it's ok to apply this as is for starters. > Anyone has objections? > > Down the road I think things should be refactored slightly to work as > follows: > - test developers can check out biosbits repo to create the iso > - everyone else gets iso downloaded It will be difficult to convince test developers to check out another repo and go back and forth between two repos. If the bits repo was a sub module that’s another story. Test developers should use the test scripts from qemu repo. Someone then later can incrementally commit these new tests into bits repo and generate newer iso at some periodic intervals. Since I am the maintainer of the bits repo I can do the second part. > > Objections to this plan? > > > > > > > > > > > > > > > > > >
On Sat, Oct 22, 2022 at 22:11 Michael S. Tsirkin <mst@redhat.com> wrote: > On Fri, Oct 21, 2022 at 05:12:07PM +0530, Ani Sinha wrote: > > > > > > On Fri, Oct 21, 2022 at 13:50 Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Fri, Oct 21, 2022 at 05:10:43AM +0530, Ani Sinha wrote: > > > > > > > > > On Fri, Oct 21, 2022 at 12:43 AM Michael S. Tsirkin < > mst@redhat.com> > > wrote: > > > > > > On Thu, Oct 20, 2022 at 06:20:20PM +0530, Ani Sinha wrote: > > > > On Thu, Oct 20, 2022 at 6:15 PM Michael S. Tsirkin < > mst@redhat.com> > > > wrote: > > > > > > > > > > On Thu, Oct 20, 2022 at 06:12:10PM +0530, Ani Sinha wrote: > > > > > > On Thu, Oct 20, 2022 at 6:08 PM Michael S. Tsirkin < > > mst@redhat.com> > > > wrote: > > > > > > > > > > > > > > On Thu, Oct 20, 2022 at 06:04:56PM +0530, Ani Sinha > wrote: > > > > > > > > > > > > > > > > > > > > > > > > Changelog: > > > > > > > > v6: > > > > > > > > - skip test when dependencies (xorriso for > example) are > > not > > > installed. > > > > > > > > - skip test when run on a platform other than > x86_64. > > > > > > > > > > > > > > Hmm why is that btw? > > > > > > > > > > > > The bits binaries that generate the iso (grub-mkrescue > etc) are > > built > > > > > > for and are known to work only on x86_64 platform. They > might > > also > > > > > > work on amd64 but I do not have one at my disposal at the > > moment to > > > > > > check. > > > > > > On other platforms, for example 32 bit x86 and non-x86, > those > > > binaries > > > > > > will likely not work. The test will fail. > > > > > > > > > > confused. I thought we are distributing the iso? > > > > > > > > No, the test builds the iso after adding the modified test > scripts > > and > > > > then spawns the vm with it. It is all part of the test > itself. > > > > We need to do that so that the iso contains the newly added > tests > > etc. > > > > > > It's good to have for people developing tests, but for most > qemu > > > developers please just have a ready iso and have avocado fetch > it. > > > It's important to make tests run on all platforms. > > > > > > > > > This changes things a lot and goes down the path of where do we > check in > > test > > > code changes? Do we deal with multiple repos? Where do we keep > iso? Etc. > > having > > > a static iso limits us also. > > > > It's the same as any firmware really. I don't see much of a > difference. > > > > > > Except that the iso is much larger than the current edk2 binaries and > there are > > serious objections to keeping them along with other bios binary blobs in > qemu > > repo. > > > > Right, I get that. So avocado was supposed to know how to download them > automatically, no? Yea that’s the reason why we are using CI artifacts and telling avocado to download them from a specific URL. > > -- > MST > >
On Fri, Oct 21, 2022 at 09:47:04PM +0530, Ani Sinha wrote: > > > On Fri, Oct 21, 2022 at 21:36 Alex Bennée <alex.bennee@linaro.org> wrote: > > > Ani Sinha <ani@anisinha.ca> writes: > > <snip> > > > > We have added a mirror of biosbits to the QEMU project so there is no > > reason why we can't track changes and modifications there (we do this > > for TestFloat which is forked from the upstream SoftFloat code). > > > > The whole idea was that say an acpi developer added support for a new > table in QEMU, he should write a corresponding > > test for bits so that the same table is exercised during run time. Making > those changes from a single repo (either directly > > or through a submodule) makes things lit simpler and also keeps things > in sync with each other. If we use separate > > repos for acpi bits test, it will be another mess when comes to > developers adding changes and keeping things in sync. > > > > Anyways these things should have been brought up earlier. I'm out of the > debate. > > > > I've sent v7 , incremental work over the last 6 months in my spare time > without getting any pay. So take it or scrap it. > > > > 😊 > > I'm sorry you feel that way but you asked for people to spend their time > reviewing your code and giving feedback. > > > > I’m sorry but you don’t understand. Giving feedback is one thing. Asking people > to make major changes to their design late in the game is completely not > acceptable imho and outright rude. . I’ve been sending out the patch series > with cc to anyone who should remotely care about this stuff. So if major rework > is needed it should be pointed out earlier and not later. I’ve begged people on > irc even to take look at this stuff including Igor and others. What else do you > expect me to do? > > > If you don't have time to take > it forward then fine > > > I’m extremely discouraged by this and have lost all motivation. It’s not that I > have no time. I squeeze all my spare time to work on this beyond my paid > responsibilities. Please be respectful of that. > Woa I hope you'll get the motivation back. I feel more testing for acpi can only help, even if it is limited to specific hosts at the moment. Let's start small, and build out. So IMHO major changes aren't strictly necessary to get this merged. Alex sent a bunch of code review comments - do you think you can address them? The limitations and TODO should just IMHO be mentioned in the cover letter and probably in comments in the code. Thanks! > > but I'm afraid a "take it or leave it" attitude > will just result in it not getting merged. > > Hopefully someone else who has the time to look into it will be able to > pick it up later. > > -- > Alex Bennée >
On Sat, Oct 22, 2022 at 10:13:13PM +0530, Ani Sinha wrote: > > > On Sat, Oct 22, 2022 at 22:05 Michael S. Tsirkin <mst@redhat.com> wrote: > > On Sat, Oct 22, 2022 at 06:28:32AM +0530, Ani Sinha wrote: > > > > > > On Fri, Oct 21, 2022 at 21:32 Alex Bennée <alex.bennee@linaro.org> wrote: > > > > > > Ani Sinha <ani@anisinha.ca> writes: > > > > > On Fri, 21 Oct, 2022, 5:52 pm Ani Sinha, <ani@anisinha.ca> wrote: > > > > > > On Fri, 21 Oct, 2022, 5:26 pm Alex Bennée, <alex.bennee@linaro.org > > > > wrote: > > > > > > Ani Sinha <ani@anisinha.ca> writes: > > > > > > > On Fri, Oct 21, 2022 at 3:10 PM Michael S. Tsirkin < > mst@redhat.com> > > wrote: > > > >> > > > >> On Fri, Oct 21, 2022 at 10:30:09AM +0100, Alex Bennée wrote: > > > >> > > > > >> > Ani Sinha <ani@anisinha.ca> writes: > > > >> > > > > >> > > On Fri, Oct 21, 2022 at 2:02 PM Michael S. Tsirkin < > > mst@redhat.com> wrote: > > > >> > >> > > > >> > >> On Fri, Oct 21, 2022 at 05:45:15AM +0530, Ani Sinha wrote: > > > >> > >> > And have multiple platform specific branches in bits > that have > > fixes for those > > > >> > >> > platforms so that bits can run there. Plus the existing > test > > can be enhanced to > > > >> > >> > pull in binaries from those branches based on the > platform on > > which it is being > > > >> > >> > run. > > > >> > >> > > > > >> > >> > > > >> > >> What a mess. > > > >> > >> Who is going to be testing all these million platforms? > > > >> > > > > > >> > > I am not talking about branches in QEMU but branches in > bits. > > > >> > > If you are going to test multiple platforms, you do need to > build > > bits > > > >> > > binaries for them. There is no way around it. > > > >> > > bits is not all platform independent python. It does have > binary > > executables. > > > >> > > > > > >> > > Currently bits is built only for the x86 platform. Other > > platforms are > > > >> > > not tested. I doubt if anyone even tried building bits for > arm or > > > >> > > mips. > > > >> > > > > >> > I'm not worried about test bits on other targets, but we do > run x86 > > > >> > targets on a number of hosts. The current reliance on a > special > > patched > > > >> > host build tool for only one architecture is the problem. If > we > > just > > > >> > download the iso that problem goes away. > > > >> > > > >> 👍what he said. > > > > > > > > Yes, in that case the problem is that upstream bits does not > pass all > > > > the test out of the box. Hence we are taking this approach of > keeping > > > > some test scripts in QEMU repo and modifying them. Then > generating the > > > > iso with the modified scripts. It also helps developers who want > to > > > > write new tests or make enhancements to existing tests. > > > > If modifications need to be made to tests, they need to be > versioned. > > > > We have gone through the route of not using submodules and I am > not > > > > going to open that can of worms again. > > > > > > We have added a mirror of biosbits to the QEMU project so there is > no > > > reason why we can't track changes and modifications there (we do > this > > > for TestFloat which is forked from the upstream SoftFloat code). > > > > > > One last option. Commit this patch set but also double commit patch 3 to > the > > bits repo so that we can build an iso that would successfully run all > tests for > > a separate platform independent test to be written later. > > > > Then we will have two tests: > > > > - this one for developers writing new test. > > - platform independent one for a basic sanity. > > > > I’m just documenting the fact that I have proposed ideas that can work > where > > all can be happy. It’s up to others to take it or keep objecting and > killing > > motivations for freelance contributors. > > I think it's ok to apply this as is for starters. > Anyone has objections? > > Down the road I think things should be refactored slightly to work as > follows: > - test developers can check out biosbits repo to create the iso > - everyone else gets iso downloaded > > > It will be difficult to convince test developers to check out another repo and > go back and forth between two repos. If the bits repo was a sub module that’s > another story. > > Test developers should use the test scripts from qemu repo. Someone then later > can incrementally commit these new tests into bits repo and generate newer iso > at some periodic intervals. Since I am the maintainer of the bits repo I can do > the second part. Sounds like a plan. Anyway, I think we can worry about that down the road after something is merged. > > > > > Objections to this plan? > > > > > > > > > > > > > > > > > >
On Sat, Oct 22, 2022 at 22:15 Michael S. Tsirkin <mst@redhat.com> wrote: > On Fri, Oct 21, 2022 at 09:47:04PM +0530, Ani Sinha wrote: > > > > > > On Fri, Oct 21, 2022 at 21:36 Alex Bennée <alex.bennee@linaro.org> > wrote: > > > > > > Ani Sinha <ani@anisinha.ca> writes: > > > > <snip> > > > > > > We have added a mirror of biosbits to the QEMU project so there > is no > > > reason why we can't track changes and modifications there (we do > this > > > for TestFloat which is forked from the upstream SoftFloat code). > > > > > > The whole idea was that say an acpi developer added support for a > new > > table in QEMU, he should write a corresponding > > > test for bits so that the same table is exercised during run time. > Making > > those changes from a single repo (either directly > > > or through a submodule) makes things lit simpler and also keeps > things > > in sync with each other. If we use separate > > > repos for acpi bits test, it will be another mess when comes to > > developers adding changes and keeping things in sync. > > > > > > Anyways these things should have been brought up earlier. I'm out > of the > > debate. > > > > > > I've sent v7 , incremental work over the last 6 months in my spare > time > > without getting any pay. So take it or scrap it. > > > > > > 😊 > > > > I'm sorry you feel that way but you asked for people to spend their > time > > reviewing your code and giving feedback. > > > > > > > > I’m sorry but you don’t understand. Giving feedback is one thing. Asking > people > > to make major changes to their design late in the game is completely not > > acceptable imho and outright rude. . I’ve been sending out the patch > series > > with cc to anyone who should remotely care about this stuff. So if major > rework > > is needed it should be pointed out earlier and not later. I’ve begged > people on > > irc even to take look at this stuff including Igor and others. What else > do you > > expect me to do? > > > > > > If you don't have time to take > > it forward then fine > > > > > > I’m extremely discouraged by this and have lost all motivation. It’s not > that I > > have no time. I squeeze all my spare time to work on this beyond my paid > > responsibilities. Please be respectful of that. > > > > > Woa I hope you'll get the motivation back. > > I feel more testing for acpi can only help, even if it is > limited to specific hosts at the moment. > > Let's start small, and build out. > > So IMHO major changes aren't strictly necessary to get this merged. > Alex sent a bunch of code review comments - do you think you can > address them? I’ve addressed them in v7. > The limitations and TODO should just IMHO be mentioned in the > cover letter and probably in comments in the code. > > Thanks! > > > > > > but I'm afraid a "take it or leave it" attitude > > will just result in it not getting merged. > > > > Hopefully someone else who has the time to look into it will be able > to > > pick it up later. > > > > -- > > Alex Bennée > > > >
diff --git a/tests/avocado/acpi-bits/bits-tests/testacpi.py2 b/tests/avocado/acpi-bits/bits-tests/testacpi.py2 index f818a9cce6..18dc818d62 100644 --- a/tests/avocado/acpi-bits/bits-tests/testacpi.py2 +++ b/tests/avocado/acpi-bits/bits-tests/testacpi.py2 @@ -40,8 +40,8 @@ import time def register_tests(): testsuite.add_test("ACPI _MAT (Multiple APIC Table Entry) under Processor objects", test_mat, submenu="ACPI Tests") -# testsuite.add_test("ACPI _PSS (Pstate) table conformance tests", test_pss, submenu="ACPI Tests") -# testsuite.add_test("ACPI _PSS (Pstate) runtime tests", test_pstates, submenu="ACPI Tests") + testsuite.add_test("ACPI _PSS (Pstate) table conformance tests", test_pss, submenu="ACPI Tests") + testsuite.add_test("ACPI _PSS (Pstate) runtime tests", test_pstates, submenu="ACPI Tests") testsuite.add_test("ACPI DSDT (Differentiated System Description Table)", test_dsdt, submenu="ACPI Tests") testsuite.add_test("ACPI FACP (Fixed ACPI Description Table)", test_facp, submenu="ACPI Tests") testsuite.add_test("ACPI HPET (High Precision Event Timer Table)", test_hpet, submenu="ACPI Tests")
Please see the rst doc file in patch 9 for more details. Sample runs are as follows: Passed tests: $ ./tests/venv/bin/avocado run -t acpi tests/avocado Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits JOB ID : e95c7413e996bfb59389839e5ca5105464ef098f JOB LOG : /home/anisinha/avocado/job-results/job-2022-10-14T19.15-e95c741/job.log (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: PASS (33.08 s) RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 39.18 s $ ./tests/venv/bin/avocado run ./tests/avocado/acpi-bits.py Fetching asset from ./tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits JOB ID : c6e588594bceb2c5fdbdf45095559c58e4030f9c JOB LOG : /home/anisinha/avocado/job-results/job-2022-10-14T19.13-c6e5885/job.log (1/1) ./tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: PASS (32.29 s) RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 39.27 s When PSS tests are enabled: $ git diff $ ./tests/venv/bin/avocado run -t acpi tests/avocado Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits JOB ID : 16a8b8c7f8dc87c4b84f8a2f95135bb63464a71e JOB LOG : /home/anisinha/avocado/job-results/job-2022-10-14T22.43-16a8b8c/job.log (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: FAIL: '1' != '0'\n- 1\n+ 0\n : Some bits tests seems to have failed. Please check the test logs for more info. (33.01 s) RESULTS : PASS 0 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 39.17 s Changelog: v6: - skip test when dependencies (xorriso for example) are not installed. - skip test when run on a platform other than x86_64. - move README as an rst file under docs/devel. - updated tags. v5: - fixed bits so that grub-mkrescue is built as a static binary. - updated the test so that it points to the new bits artifact. - V=1 now spits out more meaningful information on failure. - added a check to see if xorriso is installed. If not, the test skips with a meaningful message instructing the user to install it. - fixed README by removing references to stale instructions and adding updated instructions. - tags updated. v4: - renamed the bits test files as .py2. - fixed a bug with missing log file parsing. dump bits logs automatically for test failures (no need to re-run with V=1). - fixed issues with README file and enhanced it. - cosmetic comment updates. v3: - converted the existing test to avocado tests as per the popular recommendation. Added appropriate tags. - artifact download URL modified based on gitlab-CI URL. For biosbits repo: - moved to a gitlab repo with me being the maintainer. - added .gitlab-ci.yml file to generate the artifacts. v2: - a new class of python based tests introduced that is separate from avocado tests or qtests. Can be run by using "make check-pytest". - acpi biosbits tests are the first tests to use pytest environment. - bios bits tests now download the bits binary archives from a remote repository if they are not found locally. The test skips if download fails. - A new environment variable is introduced that can be passed by the tester to specify the location of the bits archives locally. test skips if the bits binaries are not found in that location. - if pip install of python module fails for whatever reaoson, the test skips. - misc code fixes including spell check of the README doc. README has been updated as well. - addition of SPDX license headers to bits test files. - update MAINTAINERS to reflect the new pytest test class. For biosbits repo: - added Dockerfile and build script. Made bios bits build on gcc 11. https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile https://github.com/ani-sinha/bits/blob/bits-qemu-logging/build-artifacts.sh The build script generates the zip archive and tarball used by the test. v1: initial patchset. uses qtest to implement the bios bits tests. Cc: Qemu Devel <qemu-devel@nongnu.org> Cc: Daniel P. Berrangé <berrange@redhat.com> Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: Maydell Peter <peter.maydell@linaro.org> Cc: John Snow <jsnow@redhat.com> Cc: Thomas Huth <thuth@redhat.com> Cc: Alex Bennée <alex.bennee@linaro.org> Cc: Igor Mammedov <imammedo@redhat.com> Cc: Michael Tsirkin <mst@redhat.com> Ani Sinha (10): acpi/tests/avocado/bits: initial commit of test scripts that are run by biosbits acpi/tests/avocado/bits: add SPDX license identifiers for bios bits tests acpi/tests/avocado/bits: disable acpi PSS tests that are failing in biosbits acpi/tests/avocado/bits: add smilatency test suite from bits in order to disable it acpi/tests/avocado/bits: add SPDX license identifiers for bios bits smilatency tests acpi/tests/avocado/bits: disable smilatency test since it does not pass everytime acpi/tests/avocado/bits: add biosbits config file for running bios tests acpi/tests/avocado/bits: add acpi and smbios avocado tests that uses biosbits acpi/tests/avocado/bits/doc: add a doc file to describe the acpi bits test MAINTAINERS: add myself as the maintainer for acpi biosbits avocado tests MAINTAINERS | 6 + docs/devel/acpi-bits.rst | 148 + tests/avocado/acpi-bits.py | 389 +++ .../acpi-bits/bits-config/bits-cfg.txt | 18 + tests/avocado/acpi-bits/bits-tests/smbios.py2 | 2434 +++++++++++++++++ .../acpi-bits/bits-tests/smilatency.py2 | 107 + .../avocado/acpi-bits/bits-tests/testacpi.py2 | 287 ++ .../acpi-bits/bits-tests/testcpuid.py2 | 87 + 8 files changed, 3476 insertions(+) create mode 100644 docs/devel/acpi-bits.rst create mode 100644 tests/avocado/acpi-bits.py create mode 100644 tests/avocado/acpi-bits/bits-config/bits-cfg.txt create mode 100644 tests/avocado/acpi-bits/bits-tests/smbios.py2 create mode 100644 tests/avocado/acpi-bits/bits-tests/smilatency.py2 create mode 100644 tests/avocado/acpi-bits/bits-tests/testacpi.py2 create mode 100644 tests/avocado/acpi-bits/bits-tests/testcpuid.py2