| Message ID | 20220710170014.1673480-1-ani@anisinha.ca |
|---|---|
| Headers | show |
| Series | Introduce new acpi/smbios python tests using biosbits | expand |
On Sun, Jul 10, 2022 at 10:30:03PM +0530, Ani Sinha wrote: > Changelog: > v2: > - a new class of python based tests introduced that is separate from avocado > tests or qtests. Can be run by using "make check-pytest". > - acpi biosbits tests are the first tests to use pytest environment. > - bios bits tests now download the bits binary archives from a remote > repository if they are not found locally. The test skips if download > fails. > - A new environment variable is introduced that can be passed by the tester > to specify the location of the bits archives locally. test skips if the > bits binaries are not found in that location. > - if pip install of python module fails for whatever reaoson, the test skips. > - misc code fixes including spell check of the README doc. README has been > updated as well. > - addition of SPDX license headers to bits test files. > - update MAINTAINERS to reflect the new pytest test class. > > For biosbits repo: > - added Dockerfile and build script. Made bios bits build on gcc 11. > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/build-artifacts.sh > The build script generates the zip archive and tarball used by the test. So far so good, I think it's ok for a start. It's probably a good idea to host the source on qemu.org. Peter - any objection to this? > v1: initial patchset. uses qtest to implement the bios bits tests. > > Ani Sinha (11): > acpi/tests/bits: initial commit of test scripts that are run by > biosbits > acpi/tests/bits: add SPDX license identifiers for bios bits tests > acpi/tests/bits: disable acpi PSS tests that are failing in biosbits > acpi/tests/bits: add smilatency test suite from bits in order to > disable it > acpi/tests/bits: add SPDX license identifiers for bios bits smilatency > tests > acpi/tests/bits: disable smilatency test since it does not pass > everytime > acpi/tests/bits: add python test that exercizes QEMU bios tables using > biosbits > acpi/tests/bits: add biosbits config file for running bios tests > acpi/tests/bits: add a README file describing bits pytests > pytest: add pytest to the meson build system > MAINTAINERS: add myself as the maintainer for acpi biosbits pytests > > MAINTAINERS | 5 + > tests/Makefile.include | 4 +- > tests/meson.build | 1 + > tests/pytest/acpi-bits/README | 129 + > tests/pytest/acpi-bits/acpi-bits-test-venv.sh | 59 + > tests/pytest/acpi-bits/acpi-bits-test.py | 382 +++ > .../pytest/acpi-bits/bits-config/bits-cfg.txt | 18 + > .../pytest/acpi-bits/bits-config/meson.build | 11 + > tests/pytest/acpi-bits/bits-tests/meson.build | 11 + > tests/pytest/acpi-bits/bits-tests/smbios.py | 2432 +++++++++++++++++ > .../pytest/acpi-bits/bits-tests/smilatency.py | 105 + > tests/pytest/acpi-bits/bits-tests/testacpi.py | 285 ++ > .../pytest/acpi-bits/bits-tests/testcpuid.py | 85 + > tests/pytest/acpi-bits/meson.build | 33 + > tests/pytest/acpi-bits/requirements.txt | 1 + > tests/pytest/meson.build | 49 + > 16 files changed, 3609 insertions(+), 1 deletion(-) > create mode 100644 tests/pytest/acpi-bits/README > create mode 100644 tests/pytest/acpi-bits/acpi-bits-test-venv.sh > create mode 100644 tests/pytest/acpi-bits/acpi-bits-test.py > create mode 100644 tests/pytest/acpi-bits/bits-config/bits-cfg.txt > create mode 100644 tests/pytest/acpi-bits/bits-config/meson.build > create mode 100644 tests/pytest/acpi-bits/bits-tests/meson.build > create mode 100644 tests/pytest/acpi-bits/bits-tests/smbios.py > create mode 100644 tests/pytest/acpi-bits/bits-tests/smilatency.py > create mode 100644 tests/pytest/acpi-bits/bits-tests/testacpi.py > create mode 100644 tests/pytest/acpi-bits/bits-tests/testcpuid.py > create mode 100644 tests/pytest/acpi-bits/meson.build > create mode 100644 tests/pytest/acpi-bits/requirements.txt > create mode 100644 tests/pytest/meson.build > > -- > 2.25.1
On Mon, 11 Jul 2022 at 10:34, Michael S. Tsirkin <mst@redhat.com> wrote: > > On Sun, Jul 10, 2022 at 10:30:03PM +0530, Ani Sinha wrote: > > Changelog: > > v2: > > - a new class of python based tests introduced that is separate from avocado > > tests or qtests. Can be run by using "make check-pytest". > > - acpi biosbits tests are the first tests to use pytest environment. > > - bios bits tests now download the bits binary archives from a remote > > repository if they are not found locally. The test skips if download > > fails. > > - A new environment variable is introduced that can be passed by the tester > > to specify the location of the bits archives locally. test skips if the > > bits binaries are not found in that location. > > - if pip install of python module fails for whatever reaoson, the test skips. > > - misc code fixes including spell check of the README doc. README has been > > updated as well. > > - addition of SPDX license headers to bits test files. > > - update MAINTAINERS to reflect the new pytest test class. > > > > For biosbits repo: > > - added Dockerfile and build script. Made bios bits build on gcc 11. > > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile > > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/build-artifacts.sh > > The build script generates the zip archive and tarball used by the test. > > So far so good, I think it's ok for a start. It's probably a good idea > to host the source on qemu.org. Peter - any objection to this? Dan was looking at v1 from the point of view of how we handle the guest binary blobs for these tests -- I'd rather defer to him rather than taking the time to get up to speed on the issue myself. thanks -- PMM
On Thu, Jul 14, 2022 at 6:54 PM Peter Maydell <peter.maydell@linaro.org> wrote: > > On Mon, 11 Jul 2022 at 10:34, Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Sun, Jul 10, 2022 at 10:30:03PM +0530, Ani Sinha wrote: > > > Changelog: > > > v2: > > > - a new class of python based tests introduced that is separate from avocado > > > tests or qtests. Can be run by using "make check-pytest". > > > - acpi biosbits tests are the first tests to use pytest environment. > > > - bios bits tests now download the bits binary archives from a remote > > > repository if they are not found locally. The test skips if download > > > fails. > > > - A new environment variable is introduced that can be passed by the tester > > > to specify the location of the bits archives locally. test skips if the > > > bits binaries are not found in that location. > > > - if pip install of python module fails for whatever reaoson, the test skips. > > > - misc code fixes including spell check of the README doc. README has been > > > updated as well. > > > - addition of SPDX license headers to bits test files. > > > - update MAINTAINERS to reflect the new pytest test class. > > > > > > For biosbits repo: > > > - added Dockerfile and build script. Made bios bits build on gcc 11. > > > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile > > > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/build-artifacts.sh > > > The build script generates the zip archive and tarball used by the test. > > > > So far so good, I think it's ok for a start. It's probably a good idea > > to host the source on qemu.org. Peter - any objection to this? > > Dan was looking at v1 from the point of view of how we handle the > guest binary blobs for these tests -- I'd rather defer to him rather > than taking the time to get up to speed on the issue myself. Ok let's resurrect this discussion again. What are we going to do with bios bits? Put it in git.qemu.org then? I have put a lot of time and effort into this work and I believe this will add another valuable tool to test acpi stuff, so I am not going away :-)
On Tue, Sep 06, 2022 at 11:56:33AM +0530, Ani Sinha wrote: > On Thu, Jul 14, 2022 at 6:54 PM Peter Maydell <peter.maydell@linaro.org> wrote: > > > > On Mon, 11 Jul 2022 at 10:34, Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > On Sun, Jul 10, 2022 at 10:30:03PM +0530, Ani Sinha wrote: > > > > Changelog: > > > > v2: > > > > - a new class of python based tests introduced that is separate from avocado > > > > tests or qtests. Can be run by using "make check-pytest". > > > > - acpi biosbits tests are the first tests to use pytest environment. > > > > - bios bits tests now download the bits binary archives from a remote > > > > repository if they are not found locally. The test skips if download > > > > fails. > > > > - A new environment variable is introduced that can be passed by the tester > > > > to specify the location of the bits archives locally. test skips if the > > > > bits binaries are not found in that location. > > > > - if pip install of python module fails for whatever reaoson, the test skips. > > > > - misc code fixes including spell check of the README doc. README has been > > > > updated as well. > > > > - addition of SPDX license headers to bits test files. > > > > - update MAINTAINERS to reflect the new pytest test class. > > > > > > > > For biosbits repo: > > > > - added Dockerfile and build script. Made bios bits build on gcc 11. > > > > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile > > > > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/build-artifacts.sh > > > > The build script generates the zip archive and tarball used by the test. > > > > > > So far so good, I think it's ok for a start. It's probably a good idea > > > to host the source on qemu.org. Peter - any objection to this? > > > > Dan was looking at v1 from the point of view of how we handle the > > guest binary blobs for these tests -- I'd rather defer to him rather > > than taking the time to get up to speed on the issue myself. > > Ok let's resurrect this discussion again. What are we going to do with > bios bits? Put it in git.qemu.org then? > I have put a lot of time and effort into this work and I believe this > will add another valuable tool to test acpi stuff, so I am not going > away :-) This makes sense to me. Peter, ok? Could you create a repo for Ani if yes?
On Thu, Jul 14, 2022 at 02:24:18PM +0100, Peter Maydell wrote: > On Mon, 11 Jul 2022 at 10:34, Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Sun, Jul 10, 2022 at 10:30:03PM +0530, Ani Sinha wrote: > > > Changelog: > > > v2: > > > - a new class of python based tests introduced that is separate from avocado > > > tests or qtests. Can be run by using "make check-pytest". > > > - acpi biosbits tests are the first tests to use pytest environment. > > > - bios bits tests now download the bits binary archives from a remote > > > repository if they are not found locally. The test skips if download > > > fails. > > > - A new environment variable is introduced that can be passed by the tester > > > to specify the location of the bits archives locally. test skips if the > > > bits binaries are not found in that location. > > > - if pip install of python module fails for whatever reaoson, the test skips. > > > - misc code fixes including spell check of the README doc. README has been > > > updated as well. > > > - addition of SPDX license headers to bits test files. > > > - update MAINTAINERS to reflect the new pytest test class. > > > > > > For biosbits repo: > > > - added Dockerfile and build script. Made bios bits build on gcc 11. > > > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile > > > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/build-artifacts.sh > > > The build script generates the zip archive and tarball used by the test. > > > > So far so good, I think it's ok for a start. It's probably a good idea > > to host the source on qemu.org. Peter - any objection to this? > > Dan was looking at v1 from the point of view of how we handle the > guest binary blobs for these tests -- I'd rather defer to him rather > than taking the time to get up to speed on the issue myself. Storing the *source* git repo for biosbits on gitlab.com/qemu-project is sensible, as that's what we've done for other 3rd party bits that we bundle/depend on git repo access for. The above git repo, however, has extra branches that also store the binary builds, and I'm not convinced that is a good idea. I feel like the source git repo should have a .gitlab-ci.yml file that builds the binary and publishes it as an CI artifact. This lets us keep the binary outside of GIT, have a CI job that periodically refreshes the binary (eg so when the Ubuntu version that Dockerfile uses goes EOL we can use something else). The test can access the CI artifact directly. With regards, Daniel
On Tue, Sep 6, 2022 at 18:45 Daniel P. Berrangé <berrange@redhat.com> wrote: > On Thu, Jul 14, 2022 at 02:24:18PM +0100, Peter Maydell wrote: > > On Mon, 11 Jul 2022 at 10:34, Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > On Sun, Jul 10, 2022 at 10:30:03PM +0530, Ani Sinha wrote: > > > > Changelog: > > > > v2: > > > > - a new class of python based tests introduced that is separate > from avocado > > > > tests or qtests. Can be run by using "make check-pytest". > > > > - acpi biosbits tests are the first tests to use pytest environment. > > > > - bios bits tests now download the bits binary archives from a > remote > > > > repository if they are not found locally. The test skips if > download > > > > fails. > > > > - A new environment variable is introduced that can be passed by > the tester > > > > to specify the location of the bits archives locally. test skips > if the > > > > bits binaries are not found in that location. > > > > - if pip install of python module fails for whatever reaoson, the > test skips. > > > > - misc code fixes including spell check of the README doc. README > has been > > > > updated as well. > > > > - addition of SPDX license headers to bits test files. > > > > - update MAINTAINERS to reflect the new pytest test class. > > > > > > > > For biosbits repo: > > > > - added Dockerfile and build script. Made bios bits build on gcc 11. > > > > > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile > > > > > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/build-artifacts.sh > > > > The build script generates the zip archive and tarball used by > the test. > > > > > > So far so good, I think it's ok for a start. It's probably a good idea > > > to host the source on qemu.org. Peter - any objection to this? > > > > Dan was looking at v1 from the point of view of how we handle the > > guest binary blobs for these tests -- I'd rather defer to him rather > > than taking the time to get up to speed on the issue myself. > > Storing the *source* git repo for biosbits on gitlab.com/qemu-project > is sensible, as that's what we've done for other 3rd party bits that > we bundle/depend on git repo access for. Great. Can you or Peter please create a git repo cloned from the official bios bits repo please? You don’t have to clone mine. Please provide me with push access so that I can push fixes that I have made in order to build it. > > > The above git repo, however, has extra branches that also store the > binary builds, and I'm not convinced that is a good idea. > > I feel like the source git repo should have a .gitlab-ci.yml file > that builds the binary and publishes it as an CI artifact. This > lets us keep the binary outside of GIT, have a CI job that periodically > refreshes the binary (eg so when the Ubuntu version that Dockerfile > uses goes EOL we can use something else). The test can access the CI > artifact directly. I can look into this incrementally. There is already a dockerfile for the build. I can look into pushing the yaml file as well. As for the binaries yes we need to decide where to keep thrm. But one step at a time . As for your other complaints like making the test opt-in, I’ll look into addressing them in v3.
On Tue, Sep 06, 2022 at 06:58:02PM +0530, Ani Sinha wrote: > On Tue, Sep 6, 2022 at 18:45 Daniel P. Berrangé <berrange@redhat.com> wrote: > > > On Thu, Jul 14, 2022 at 02:24:18PM +0100, Peter Maydell wrote: > > > On Mon, 11 Jul 2022 at 10:34, Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > On Sun, Jul 10, 2022 at 10:30:03PM +0530, Ani Sinha wrote: > > > > > Changelog: > > > > > v2: > > > > > - a new class of python based tests introduced that is separate > > from avocado > > > > > tests or qtests. Can be run by using "make check-pytest". > > > > > - acpi biosbits tests are the first tests to use pytest environment. > > > > > - bios bits tests now download the bits binary archives from a > > remote > > > > > repository if they are not found locally. The test skips if > > download > > > > > fails. > > > > > - A new environment variable is introduced that can be passed by > > the tester > > > > > to specify the location of the bits archives locally. test skips > > if the > > > > > bits binaries are not found in that location. > > > > > - if pip install of python module fails for whatever reaoson, the > > test skips. > > > > > - misc code fixes including spell check of the README doc. README > > has been > > > > > updated as well. > > > > > - addition of SPDX license headers to bits test files. > > > > > - update MAINTAINERS to reflect the new pytest test class. > > > > > > > > > > For biosbits repo: > > > > > - added Dockerfile and build script. Made bios bits build on gcc 11. > > > > > > > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile > > > > > > > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/build-artifacts.sh > > > > > The build script generates the zip archive and tarball used by > > the test. > > > > > > > > So far so good, I think it's ok for a start. It's probably a good idea > > > > to host the source on qemu.org. Peter - any objection to this? > > > > > > Dan was looking at v1 from the point of view of how we handle the > > > guest binary blobs for these tests -- I'd rather defer to him rather > > > than taking the time to get up to speed on the issue myself. > > > > Storing the *source* git repo for biosbits on gitlab.com/qemu-project > > is sensible, as that's what we've done for other 3rd party bits that > > we bundle/depend on git repo access for. > > > Great. Can you or Peter please create a git repo cloned from the official > bios bits repo please? You don’t have to clone mine. Please provide me with > push access so that I can push fixes that I have made in order to build it. I don't have any admin privileges for qemu infra, so can't do this myself. With regards, Daniel
On Tue, Sep 06, 2022 at 02:15:41PM +0100, Daniel P. Berrangé wrote: > On Thu, Jul 14, 2022 at 02:24:18PM +0100, Peter Maydell wrote: > > On Mon, 11 Jul 2022 at 10:34, Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > On Sun, Jul 10, 2022 at 10:30:03PM +0530, Ani Sinha wrote: > > > > Changelog: > > > > v2: > > > > - a new class of python based tests introduced that is separate from avocado > > > > tests or qtests. Can be run by using "make check-pytest". > > > > - acpi biosbits tests are the first tests to use pytest environment. > > > > - bios bits tests now download the bits binary archives from a remote > > > > repository if they are not found locally. The test skips if download > > > > fails. > > > > - A new environment variable is introduced that can be passed by the tester > > > > to specify the location of the bits archives locally. test skips if the > > > > bits binaries are not found in that location. > > > > - if pip install of python module fails for whatever reaoson, the test skips. > > > > - misc code fixes including spell check of the README doc. README has been > > > > updated as well. > > > > - addition of SPDX license headers to bits test files. > > > > - update MAINTAINERS to reflect the new pytest test class. > > > > > > > > For biosbits repo: > > > > - added Dockerfile and build script. Made bios bits build on gcc 11. > > > > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile > > > > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/build-artifacts.sh > > > > The build script generates the zip archive and tarball used by the test. > > > > > > So far so good, I think it's ok for a start. It's probably a good idea > > > to host the source on qemu.org. Peter - any objection to this? > > > > Dan was looking at v1 from the point of view of how we handle the > > guest binary blobs for these tests -- I'd rather defer to him rather > > than taking the time to get up to speed on the issue myself. > > Storing the *source* git repo for biosbits on gitlab.com/qemu-project > is sensible, as that's what we've done for other 3rd party bits that > we bundle/depend on git repo access for. > > The above git repo, however, has extra branches that also store the > binary builds, and I'm not convinced that is a good idea. > > I feel like the source git repo should have a .gitlab-ci.yml file > that builds the binary and publishes it as an CI artifact. This > lets us keep the binary outside of GIT, have a CI job that periodically > refreshes the binary (eg so when the Ubuntu version that Dockerfile > uses goes EOL we can use something else). The test can access the CI > artifact directly. > > > With regards, > Daniel CI is for biosbits developers, bits that everyone uses is production. I don't think our CI is yet robust enough - I think when it comes to pushing bits that all other developers are going to use, it is wise to have them pushed manually. Random CI failures are not unheard of, and if these cause a corrupted binary to be downloaded to all developers, it's going to be bad. > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Sep 06, 2022 at 10:03:55AM -0400, Michael S. Tsirkin wrote: > On Tue, Sep 06, 2022 at 02:15:41PM +0100, Daniel P. Berrangé wrote: > > On Thu, Jul 14, 2022 at 02:24:18PM +0100, Peter Maydell wrote: > > > On Mon, 11 Jul 2022 at 10:34, Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > On Sun, Jul 10, 2022 at 10:30:03PM +0530, Ani Sinha wrote: > > > > > Changelog: > > > > > v2: > > > > > - a new class of python based tests introduced that is separate from avocado > > > > > tests or qtests. Can be run by using "make check-pytest". > > > > > - acpi biosbits tests are the first tests to use pytest environment. > > > > > - bios bits tests now download the bits binary archives from a remote > > > > > repository if they are not found locally. The test skips if download > > > > > fails. > > > > > - A new environment variable is introduced that can be passed by the tester > > > > > to specify the location of the bits archives locally. test skips if the > > > > > bits binaries are not found in that location. > > > > > - if pip install of python module fails for whatever reaoson, the test skips. > > > > > - misc code fixes including spell check of the README doc. README has been > > > > > updated as well. > > > > > - addition of SPDX license headers to bits test files. > > > > > - update MAINTAINERS to reflect the new pytest test class. > > > > > > > > > > For biosbits repo: > > > > > - added Dockerfile and build script. Made bios bits build on gcc 11. > > > > > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile > > > > > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/build-artifacts.sh > > > > > The build script generates the zip archive and tarball used by the test. > > > > > > > > So far so good, I think it's ok for a start. It's probably a good idea > > > > to host the source on qemu.org. Peter - any objection to this? > > > > > > Dan was looking at v1 from the point of view of how we handle the > > > guest binary blobs for these tests -- I'd rather defer to him rather > > > than taking the time to get up to speed on the issue myself. > > > > Storing the *source* git repo for biosbits on gitlab.com/qemu-project > > is sensible, as that's what we've done for other 3rd party bits that > > we bundle/depend on git repo access for. > > > > The above git repo, however, has extra branches that also store the > > binary builds, and I'm not convinced that is a good idea. > > > > I feel like the source git repo should have a .gitlab-ci.yml file > > that builds the binary and publishes it as an CI artifact. This > > lets us keep the binary outside of GIT, have a CI job that periodically > > refreshes the binary (eg so when the Ubuntu version that Dockerfile > > uses goes EOL we can use something else). The test can access the CI > > artifact directly. > > > > > > With regards, > > Daniel > > CI is for biosbits developers, bits that everyone uses > is production. I don't think our CI is yet robust enough - > I think when it comes to pushing bits that all > other developers are going to use, it is wise to have them > pushed manually. Random CI failures are not unheard of, > and if these cause a corrupted binary to be downloaded > to all developers, it's going to be bad. The artifacts would only be published if the CI job succeeds. If it fails the artifacts from the previous successful job remain available. Second, we don't need to run the CI continuously. It could be done once a month, or could be triggered manually on demand. Or it could run frequently, but we only publish artifacts from a manual invokation. With regards, Daniel
On Tue, Sep 06, 2022 at 03:10:05PM +0100, Daniel P. Berrangé wrote: > On Tue, Sep 06, 2022 at 10:03:55AM -0400, Michael S. Tsirkin wrote: > > On Tue, Sep 06, 2022 at 02:15:41PM +0100, Daniel P. Berrangé wrote: > > > On Thu, Jul 14, 2022 at 02:24:18PM +0100, Peter Maydell wrote: > > > > On Mon, 11 Jul 2022 at 10:34, Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > > > On Sun, Jul 10, 2022 at 10:30:03PM +0530, Ani Sinha wrote: > > > > > > Changelog: > > > > > > v2: > > > > > > - a new class of python based tests introduced that is separate from avocado > > > > > > tests or qtests. Can be run by using "make check-pytest". > > > > > > - acpi biosbits tests are the first tests to use pytest environment. > > > > > > - bios bits tests now download the bits binary archives from a remote > > > > > > repository if they are not found locally. The test skips if download > > > > > > fails. > > > > > > - A new environment variable is introduced that can be passed by the tester > > > > > > to specify the location of the bits archives locally. test skips if the > > > > > > bits binaries are not found in that location. > > > > > > - if pip install of python module fails for whatever reaoson, the test skips. > > > > > > - misc code fixes including spell check of the README doc. README has been > > > > > > updated as well. > > > > > > - addition of SPDX license headers to bits test files. > > > > > > - update MAINTAINERS to reflect the new pytest test class. > > > > > > > > > > > > For biosbits repo: > > > > > > - added Dockerfile and build script. Made bios bits build on gcc 11. > > > > > > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile > > > > > > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/build-artifacts.sh > > > > > > The build script generates the zip archive and tarball used by the test. > > > > > > > > > > So far so good, I think it's ok for a start. It's probably a good idea > > > > > to host the source on qemu.org. Peter - any objection to this? > > > > > > > > Dan was looking at v1 from the point of view of how we handle the > > > > guest binary blobs for these tests -- I'd rather defer to him rather > > > > than taking the time to get up to speed on the issue myself. > > > > > > Storing the *source* git repo for biosbits on gitlab.com/qemu-project > > > is sensible, as that's what we've done for other 3rd party bits that > > > we bundle/depend on git repo access for. > > > > > > The above git repo, however, has extra branches that also store the > > > binary builds, and I'm not convinced that is a good idea. > > > > > > I feel like the source git repo should have a .gitlab-ci.yml file > > > that builds the binary and publishes it as an CI artifact. This > > > lets us keep the binary outside of GIT, have a CI job that periodically > > > refreshes the binary (eg so when the Ubuntu version that Dockerfile > > > uses goes EOL we can use something else). The test can access the CI > > > artifact directly. > > > > > > > > > With regards, > > > Daniel > > > > CI is for biosbits developers, bits that everyone uses > > is production. I don't think our CI is yet robust enough - > > I think when it comes to pushing bits that all > > other developers are going to use, it is wise to have them > > pushed manually. Random CI failures are not unheard of, > > and if these cause a corrupted binary to be downloaded > > to all developers, it's going to be bad. > > The artifacts would only be published if the CI job succeeds. If it fails > the artifacts from the previous successful job remain available. > > Second, we don't need to run the CI continuously. It could be done once > a month, or could be triggered manually on demand. Or it could run > frequently, but we only publish artifacts from a manual invokation. > > > With regards, > Daniel I think publishing should happen manually after maintainer decides it's worth it. In particular, I think we want to keep old qemu source working wrt tests and that means we don't ever erase old binaries, instead adding new ones on demand. IIUC at the moment no other subproject within qemu uses CI for production, we just use it for testing, and it seems somewhat weird to start with a new subproject as opposed to an established one. The new kid and a new maintainer will have problems enough as it is. > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
+alexb On Tue, Sep 6, 2022 at 7:00 PM Daniel P. Berrangé <berrange@redhat.com> wrote: > > On Tue, Sep 06, 2022 at 06:58:02PM +0530, Ani Sinha wrote: > > On Tue, Sep 6, 2022 at 18:45 Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > > On Thu, Jul 14, 2022 at 02:24:18PM +0100, Peter Maydell wrote: > > > > On Mon, 11 Jul 2022 at 10:34, Michael S. Tsirkin <mst@redhat.com> wrote: > > > > > > > > > > On Sun, Jul 10, 2022 at 10:30:03PM +0530, Ani Sinha wrote: > > > > > > Changelog: > > > > > > v2: > > > > > > - a new class of python based tests introduced that is separate > > > from avocado > > > > > > tests or qtests. Can be run by using "make check-pytest". > > > > > > - acpi biosbits tests are the first tests to use pytest environment. > > > > > > - bios bits tests now download the bits binary archives from a > > > remote > > > > > > repository if they are not found locally. The test skips if > > > download > > > > > > fails. > > > > > > - A new environment variable is introduced that can be passed by > > > the tester > > > > > > to specify the location of the bits archives locally. test skips > > > if the > > > > > > bits binaries are not found in that location. > > > > > > - if pip install of python module fails for whatever reaoson, the > > > test skips. > > > > > > - misc code fixes including spell check of the README doc. README > > > has been > > > > > > updated as well. > > > > > > - addition of SPDX license headers to bits test files. > > > > > > - update MAINTAINERS to reflect the new pytest test class. > > > > > > > > > > > > For biosbits repo: > > > > > > - added Dockerfile and build script. Made bios bits build on gcc 11. > > > > > > > > > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile > > > > > > > > > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/build-artifacts.sh > > > > > > The build script generates the zip archive and tarball used by > > > the test. > > > > > > > > > > So far so good, I think it's ok for a start. It's probably a good idea > > > > > to host the source on qemu.org. Peter - any objection to this? > > > > > > > > Dan was looking at v1 from the point of view of how we handle the > > > > guest binary blobs for these tests -- I'd rather defer to him rather > > > > than taking the time to get up to speed on the issue myself. > > > > > > Storing the *source* git repo for biosbits on gitlab.com/qemu-project > > > is sensible, as that's what we've done for other 3rd party bits that > > > we bundle/depend on git repo access for. > > > > > > Great. Can you or Peter please create a git repo cloned from the official > > bios bits repo please? You don’t have to clone mine. Please provide me with > > push access so that I can push fixes that I have made in order to build it. > > I don't have any admin privileges for qemu infra, so can't do this > myself. > > With regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| >
Changelog: v2: - a new class of python based tests introduced that is separate from avocado tests or qtests. Can be run by using "make check-pytest". - acpi biosbits tests are the first tests to use pytest environment. - bios bits tests now download the bits binary archives from a remote repository if they are not found locally. The test skips if download fails. - A new environment variable is introduced that can be passed by the tester to specify the location of the bits archives locally. test skips if the bits binaries are not found in that location. - if pip install of python module fails for whatever reaoson, the test skips. - misc code fixes including spell check of the README doc. README has been updated as well. - addition of SPDX license headers to bits test files. - update MAINTAINERS to reflect the new pytest test class. For biosbits repo: - added Dockerfile and build script. Made bios bits build on gcc 11. https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile https://github.com/ani-sinha/bits/blob/bits-qemu-logging/build-artifacts.sh The build script generates the zip archive and tarball used by the test. v1: initial patchset. uses qtest to implement the bios bits tests. Ani Sinha (11): acpi/tests/bits: initial commit of test scripts that are run by biosbits acpi/tests/bits: add SPDX license identifiers for bios bits tests acpi/tests/bits: disable acpi PSS tests that are failing in biosbits acpi/tests/bits: add smilatency test suite from bits in order to disable it acpi/tests/bits: add SPDX license identifiers for bios bits smilatency tests acpi/tests/bits: disable smilatency test since it does not pass everytime acpi/tests/bits: add python test that exercizes QEMU bios tables using biosbits acpi/tests/bits: add biosbits config file for running bios tests acpi/tests/bits: add a README file describing bits pytests pytest: add pytest to the meson build system MAINTAINERS: add myself as the maintainer for acpi biosbits pytests MAINTAINERS | 5 + tests/Makefile.include | 4 +- tests/meson.build | 1 + tests/pytest/acpi-bits/README | 129 + tests/pytest/acpi-bits/acpi-bits-test-venv.sh | 59 + tests/pytest/acpi-bits/acpi-bits-test.py | 382 +++ .../pytest/acpi-bits/bits-config/bits-cfg.txt | 18 + .../pytest/acpi-bits/bits-config/meson.build | 11 + tests/pytest/acpi-bits/bits-tests/meson.build | 11 + tests/pytest/acpi-bits/bits-tests/smbios.py | 2432 +++++++++++++++++ .../pytest/acpi-bits/bits-tests/smilatency.py | 105 + tests/pytest/acpi-bits/bits-tests/testacpi.py | 285 ++ .../pytest/acpi-bits/bits-tests/testcpuid.py | 85 + tests/pytest/acpi-bits/meson.build | 33 + tests/pytest/acpi-bits/requirements.txt | 1 + tests/pytest/meson.build | 49 + 16 files changed, 3609 insertions(+), 1 deletion(-) create mode 100644 tests/pytest/acpi-bits/README create mode 100644 tests/pytest/acpi-bits/acpi-bits-test-venv.sh create mode 100644 tests/pytest/acpi-bits/acpi-bits-test.py create mode 100644 tests/pytest/acpi-bits/bits-config/bits-cfg.txt create mode 100644 tests/pytest/acpi-bits/bits-config/meson.build create mode 100644 tests/pytest/acpi-bits/bits-tests/meson.build create mode 100644 tests/pytest/acpi-bits/bits-tests/smbios.py create mode 100644 tests/pytest/acpi-bits/bits-tests/smilatency.py create mode 100644 tests/pytest/acpi-bits/bits-tests/testacpi.py create mode 100644 tests/pytest/acpi-bits/bits-tests/testcpuid.py create mode 100644 tests/pytest/acpi-bits/meson.build create mode 100644 tests/pytest/acpi-bits/requirements.txt create mode 100644 tests/pytest/meson.build