| Message ID | 20221019025828.683113-10-ani@anisinha.ca |
|---|---|
| State | New |
| Headers | show |
| Series | [v5,01/10] acpi/tests/avocado/bits: initial commit of test scripts that are run by biosbits | expand |
Ani Sinha <ani@anisinha.ca> writes: > Add a README file that describes the purpose of the various test files and gives > guidance to developers on where and how to make changes. > > Cc: Daniel P. Berrange" <berrange@redhat.com> > Cc: Paolo Bonzini <pbonzini@redhat.com> > Cc: Maydell Peter <peter.maydell@linaro.org> > Cc: John Snow <jsnow@redhat.com> > Cc: Thomas Huth <thuth@redhat.com> > Cc: Alex Bennée <alex.bennee@linaro.org> > Cc: Igor Mammedov <imammedo@redhat.com> > Cc: Michael Tsirkin <mst@redhat.com> > Signed-off-by: Ani Sinha <ani@anisinha.ca> > --- > tests/avocado/acpi-bits/README | 133 +++++++++++++++++++++++++++++++++ > 1 file changed, 133 insertions(+) > create mode 100644 tests/avocado/acpi-bits/README > > diff --git a/tests/avocado/acpi-bits/README b/tests/avocado/acpi-bits/README > new file mode 100644 > index 0000000000..4945dfc1f2 > --- /dev/null > +++ b/tests/avocado/acpi-bits/README > @@ -0,0 +1,133 @@ > +============================================================================= > +ACPI/SMBIOS AVOCADO TESTS USING BIOSBITS > +============================================================================= > + > +Biosbits is a software written by Josh Triplett that can be downloaded > +from https://biosbits.org/. The github codebase can be found here: > +https://github.com/biosbits/bits/tree/master. It is a software that executes > +the bios components such as acpi and smbios tables directly through acpica > +bios interpreter (a freely available C based library written by Intel, > +downloadable from https://acpica.org/ and is included with biosbits) without an > +operating system getting involved in between. > +There are several advantages to directly testing the bios in a real physical > +machine or VM as opposed to indirectly discovering bios issues through the > +operating system. For one thing, the OSes tend to hide bios problems from the > +end user. The other is that we have more control of what we wanted to test > +and how by directly using acpica interpreter on top of the bios on a running > +system. More details on the inspiration for developing biosbits and its real > +life uses can be found in (a) and (b). > +This directory contains tests written in python using avocado framework that > +exercizes the QEMU bios components using biosbits and reports test failures. > +For QEMU, we maintain a fork of bios bits in gitlab along with all the > +dependent submodules: > +https://gitlab.com/qemu-project/biosbits-bits > +This fork contains numerous fixes, a newer acpica and changes specific to > +running this avocado QEMU tests using bits. The author of this document > +is the sole maintainer of the QEMU fork of bios bits repo. > + > +Under the directory tests/avocado/, acpi-bits.py is a QEMU avocado test that > +drives all this. > + > +A brief description of the various test files follows. > + > +Under tests/avocado/ as the root we have: > + > +├── acpi-bits > +│ ├── bits-config > +│ │ └── bits-cfg.txt > +│ ├── bits-tests > +│ │ ├── smbios.py2 > +│ │ ├── smilatency.py2 > +│ │ ├── testacpi.py2 > +│ │ └── testcpuid.py2 > +│ └── README > +├── acpi-bits.py > + > +tests/avocado: > + - acpi-bits.py: This is the main python avocado test script that generates a > + biosbits iso. It then spawns a QEMU VM with it, collects the log and reports > + test failures. This is the script one would be interested in if they wanted > + to add or change some component of the log parsing, add a new command line > + to alter how QEMU is spawned etc. Test writers typically would not need to > + modify this script unless they wanted to enhance or change the log parsing > + for their tests. Following environment variables are used in this test: > + - V=1 : This enables verbose mode for the test. It dumps the entire log > + from bios bits and also more details in case failure happens. It is > + useful for debugging the test failures or tests themselves. > + > + In order to run this test, please perform the following steps from the QEMU > + build directory: > + > + $ make check-venv (needed only the first time to create the venv) > + $ ./tests/venv/bin/avocado run -t acpi tests/avocado > + > + The above will run all acpi avocado tests including this one. > + In order to run the individual tests, perform the following: > + > + $ ./tests/venv/bin/avocado run tests/avocado/acpi-bits.py --tap - > + > + The above will produce output in tap format. You can omit "--tap -" in the > + end and it will produce output like the following: > + > + $ ./tests/venv/bin/avocado run tests/avocado/acpi-bits.py > + Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits > + JOB ID : eab225724da7b64c012c65705dc2fa14ab1defef > + JOB LOG : /home/anisinha/avocado/job-results/job-2022-10-10T17.58-eab2257/job.log > + (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: PASS (33.09 s) > + RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 > + JOB TIME : 39.22 s > + > + You can inspect the log file for more information about the run or in order > + to diagnoze issues. If you pass V=1 in the environment, more diagnostic logs > + would be found in the test log. > + > +tests/avocado/acpi-bits: > + - README: This text file. > + > +tests/avocado/acpi-bits/bits-config: > + This location contains biosbits config files that determine how the software > + runs the tests. > + - bits-config.txt: this is the biosbits config file that determines what tests > + or actions are performed by bits. The description of the config options are > + provided in the file itself. > + > +tests/avocado/acpi-bits/bits-tests: > + This directory contains biosbits python based tests that are run from within > + the biosbits environment in the spawned VM. New additions of test cases can > + be made in the appropriate test file. For example, new acpi tests can go > + into testacpi.py2 and one would call testsuite.add_test() to register the new > + test so that it gets executed as a part of the ACPI tests. > + It might be occasionally necessary to disable some subtests or add a new > + test that belongs to a test suite not already present in this directory. To > + do this, please clone the bits source from > + https://gitlab.com/qemu-project/biosbits-bits/-/tree/qemu-bits. > + Note that this is the "qemu-bits" branch and not the "bits" branch of the > + repository. "qemu-bits" is the branch where we have made all the QEMU > + specific enhancements and we must use the source from this branch only. > + Copy the test suite/script that needs modification (addition of new tests > + or disabling them) from python directory into this directory. For > + example, in order to change cpuid related tests, copy the following > + file into this directory and rename it with .py2 extension: > + https://gitlab.com/qemu-project/biosbits-bits/-/blob/qemu-bits/python/testcpuid.py > + Then make your additions and changes here. Hence, the steps are: > + > + step (a): copy unmodified test script to this directory from bits source. > + step (b): perform modifications to the test. > + Commit (a) and (b) should go under separate commits so that the original > + test script and the changes we have made are separated and clear. > + > + The test framework will then use your modified test script to run the test. > + No further changes would be needed. Please check the logs to make sure that > + appropriate changes have taken effect. > + > + The tests have an extension .py2 in order to indicate that > + (a) they are python2.7 based scripts and not python 3 scripts. > + (b) they are run from within the bios bits VM and is not subjected to QEMU > + build/test python script maintainance and dependency resolutions. > + (c) They need not be loaded by avocado framework when running tests. > + > +Author: Ani Sinha <ani@anisinha.ca> > + > +References: > +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf > +(b) https://www.youtube.com/watch?v=36QIepyUuhg This might me better as an RST under docs/devel so it can be included in the growing developer guide.
On Wed, Oct 19, 2022 at 9:04 PM Alex Bennée <alex.bennee@linaro.org> wrote: > > > Ani Sinha <ani@anisinha.ca> writes: > > > Add a README file that describes the purpose of the various test files and gives > > guidance to developers on where and how to make changes. > > > > Cc: Daniel P. Berrange" <berrange@redhat.com> > > Cc: Paolo Bonzini <pbonzini@redhat.com> > > Cc: Maydell Peter <peter.maydell@linaro.org> > > Cc: John Snow <jsnow@redhat.com> > > Cc: Thomas Huth <thuth@redhat.com> > > Cc: Alex Bennée <alex.bennee@linaro.org> > > Cc: Igor Mammedov <imammedo@redhat.com> > > Cc: Michael Tsirkin <mst@redhat.com> > > Signed-off-by: Ani Sinha <ani@anisinha.ca> > > --- > > tests/avocado/acpi-bits/README | 133 +++++++++++++++++++++++++++++++++ > > 1 file changed, 133 insertions(+) > > create mode 100644 tests/avocado/acpi-bits/README > > > > diff --git a/tests/avocado/acpi-bits/README b/tests/avocado/acpi-bits/README > > new file mode 100644 > > index 0000000000..4945dfc1f2 > > --- /dev/null > > +++ b/tests/avocado/acpi-bits/README > > @@ -0,0 +1,133 @@ > > +============================================================================= > > +ACPI/SMBIOS AVOCADO TESTS USING BIOSBITS > > +============================================================================= > > + > > +Biosbits is a software written by Josh Triplett that can be downloaded > > +from https://biosbits.org/. The github codebase can be found here: > > +https://github.com/biosbits/bits/tree/master. It is a software that executes > > +the bios components such as acpi and smbios tables directly through acpica > > +bios interpreter (a freely available C based library written by Intel, > > +downloadable from https://acpica.org/ and is included with biosbits) without an > > +operating system getting involved in between. > > +There are several advantages to directly testing the bios in a real physical > > +machine or VM as opposed to indirectly discovering bios issues through the > > +operating system. For one thing, the OSes tend to hide bios problems from the > > +end user. The other is that we have more control of what we wanted to test > > +and how by directly using acpica interpreter on top of the bios on a running > > +system. More details on the inspiration for developing biosbits and its real > > +life uses can be found in (a) and (b). > > +This directory contains tests written in python using avocado framework that > > +exercizes the QEMU bios components using biosbits and reports test failures. > > +For QEMU, we maintain a fork of bios bits in gitlab along with all the > > +dependent submodules: > > +https://gitlab.com/qemu-project/biosbits-bits > > +This fork contains numerous fixes, a newer acpica and changes specific to > > +running this avocado QEMU tests using bits. The author of this document > > +is the sole maintainer of the QEMU fork of bios bits repo. > > + > > +Under the directory tests/avocado/, acpi-bits.py is a QEMU avocado test that > > +drives all this. > > + > > +A brief description of the various test files follows. > > + > > +Under tests/avocado/ as the root we have: > > + > > +├── acpi-bits > > +│ ├── bits-config > > +│ │ └── bits-cfg.txt > > +│ ├── bits-tests > > +│ │ ├── smbios.py2 > > +│ │ ├── smilatency.py2 > > +│ │ ├── testacpi.py2 > > +│ │ └── testcpuid.py2 > > +│ └── README > > +├── acpi-bits.py > > + > > +tests/avocado: > > + - acpi-bits.py: This is the main python avocado test script that generates a > > + biosbits iso. It then spawns a QEMU VM with it, collects the log and reports > > + test failures. This is the script one would be interested in if they wanted > > + to add or change some component of the log parsing, add a new command line > > + to alter how QEMU is spawned etc. Test writers typically would not need to > > + modify this script unless they wanted to enhance or change the log parsing > > + for their tests. Following environment variables are used in this test: > > + - V=1 : This enables verbose mode for the test. It dumps the entire log > > + from bios bits and also more details in case failure happens. It is > > + useful for debugging the test failures or tests themselves. > > + > > + In order to run this test, please perform the following steps from the QEMU > > + build directory: > > + > > + $ make check-venv (needed only the first time to create the venv) > > + $ ./tests/venv/bin/avocado run -t acpi tests/avocado > > + > > + The above will run all acpi avocado tests including this one. > > + In order to run the individual tests, perform the following: > > + > > + $ ./tests/venv/bin/avocado run tests/avocado/acpi-bits.py --tap - > > + > > + The above will produce output in tap format. You can omit "--tap -" in the > > + end and it will produce output like the following: > > + > > + $ ./tests/venv/bin/avocado run tests/avocado/acpi-bits.py > > + Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits > > + JOB ID : eab225724da7b64c012c65705dc2fa14ab1defef > > + JOB LOG : /home/anisinha/avocado/job-results/job-2022-10-10T17.58-eab2257/job.log > > + (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: PASS (33.09 s) > > + RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 > > + JOB TIME : 39.22 s > > + > > + You can inspect the log file for more information about the run or in order > > + to diagnoze issues. If you pass V=1 in the environment, more diagnostic logs > > + would be found in the test log. > > + > > +tests/avocado/acpi-bits: > > + - README: This text file. > > + > > +tests/avocado/acpi-bits/bits-config: > > + This location contains biosbits config files that determine how the software > > + runs the tests. > > + - bits-config.txt: this is the biosbits config file that determines what tests > > + or actions are performed by bits. The description of the config options are > > + provided in the file itself. > > + > > +tests/avocado/acpi-bits/bits-tests: > > + This directory contains biosbits python based tests that are run from within > > + the biosbits environment in the spawned VM. New additions of test cases can > > + be made in the appropriate test file. For example, new acpi tests can go > > + into testacpi.py2 and one would call testsuite.add_test() to register the new > > + test so that it gets executed as a part of the ACPI tests. > > + It might be occasionally necessary to disable some subtests or add a new > > + test that belongs to a test suite not already present in this directory. To > > + do this, please clone the bits source from > > + https://gitlab.com/qemu-project/biosbits-bits/-/tree/qemu-bits. > > + Note that this is the "qemu-bits" branch and not the "bits" branch of the > > + repository. "qemu-bits" is the branch where we have made all the QEMU > > + specific enhancements and we must use the source from this branch only. > > + Copy the test suite/script that needs modification (addition of new tests > > + or disabling them) from python directory into this directory. For > > + example, in order to change cpuid related tests, copy the following > > + file into this directory and rename it with .py2 extension: > > + https://gitlab.com/qemu-project/biosbits-bits/-/blob/qemu-bits/python/testcpuid.py > > + Then make your additions and changes here. Hence, the steps are: > > + > > + step (a): copy unmodified test script to this directory from bits source. > > + step (b): perform modifications to the test. > > + Commit (a) and (b) should go under separate commits so that the original > > + test script and the changes we have made are separated and clear. > > + > > + The test framework will then use your modified test script to run the test. > > + No further changes would be needed. Please check the logs to make sure that > > + appropriate changes have taken effect. > > + > > + The tests have an extension .py2 in order to indicate that > > + (a) they are python2.7 based scripts and not python 3 scripts. > > + (b) they are run from within the bios bits VM and is not subjected to QEMU > > + build/test python script maintainance and dependency resolutions. > > + (c) They need not be loaded by avocado framework when running tests. > > + > > +Author: Ani Sinha <ani@anisinha.ca> > > + > > +References: > > +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf > > +(b) https://www.youtube.com/watch?v=36QIepyUuhg > > This might me better as an RST under docs/devel so it can be included in > the growing developer guide. Ok if there are no objects from others, will move it there. Sounds like a better idea.
diff --git a/tests/avocado/acpi-bits/README b/tests/avocado/acpi-bits/README new file mode 100644 index 0000000000..4945dfc1f2 --- /dev/null +++ b/tests/avocado/acpi-bits/README @@ -0,0 +1,133 @@ +============================================================================= +ACPI/SMBIOS AVOCADO TESTS USING BIOSBITS +============================================================================= + +Biosbits is a software written by Josh Triplett that can be downloaded +from https://biosbits.org/. The github codebase can be found here: +https://github.com/biosbits/bits/tree/master. It is a software that executes +the bios components such as acpi and smbios tables directly through acpica +bios interpreter (a freely available C based library written by Intel, +downloadable from https://acpica.org/ and is included with biosbits) without an +operating system getting involved in between. +There are several advantages to directly testing the bios in a real physical +machine or VM as opposed to indirectly discovering bios issues through the +operating system. For one thing, the OSes tend to hide bios problems from the +end user. The other is that we have more control of what we wanted to test +and how by directly using acpica interpreter on top of the bios on a running +system. More details on the inspiration for developing biosbits and its real +life uses can be found in (a) and (b). +This directory contains tests written in python using avocado framework that +exercizes the QEMU bios components using biosbits and reports test failures. +For QEMU, we maintain a fork of bios bits in gitlab along with all the +dependent submodules: +https://gitlab.com/qemu-project/biosbits-bits +This fork contains numerous fixes, a newer acpica and changes specific to +running this avocado QEMU tests using bits. The author of this document +is the sole maintainer of the QEMU fork of bios bits repo. + +Under the directory tests/avocado/, acpi-bits.py is a QEMU avocado test that +drives all this. + +A brief description of the various test files follows. + +Under tests/avocado/ as the root we have: + +├── acpi-bits +│ ├── bits-config +│ │ └── bits-cfg.txt +│ ├── bits-tests +│ │ ├── smbios.py2 +│ │ ├── smilatency.py2 +│ │ ├── testacpi.py2 +│ │ └── testcpuid.py2 +│ └── README +├── acpi-bits.py + +tests/avocado: + - acpi-bits.py: This is the main python avocado test script that generates a + biosbits iso. It then spawns a QEMU VM with it, collects the log and reports + test failures. This is the script one would be interested in if they wanted + to add or change some component of the log parsing, add a new command line + to alter how QEMU is spawned etc. Test writers typically would not need to + modify this script unless they wanted to enhance or change the log parsing + for their tests. Following environment variables are used in this test: + - V=1 : This enables verbose mode for the test. It dumps the entire log + from bios bits and also more details in case failure happens. It is + useful for debugging the test failures or tests themselves. + + In order to run this test, please perform the following steps from the QEMU + build directory: + + $ make check-venv (needed only the first time to create the venv) + $ ./tests/venv/bin/avocado run -t acpi tests/avocado + + The above will run all acpi avocado tests including this one. + In order to run the individual tests, perform the following: + + $ ./tests/venv/bin/avocado run tests/avocado/acpi-bits.py --tap - + + The above will produce output in tap format. You can omit "--tap -" in the + end and it will produce output like the following: + + $ ./tests/venv/bin/avocado run tests/avocado/acpi-bits.py + Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits + JOB ID : eab225724da7b64c012c65705dc2fa14ab1defef + JOB LOG : /home/anisinha/avocado/job-results/job-2022-10-10T17.58-eab2257/job.log + (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: PASS (33.09 s) + RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 + JOB TIME : 39.22 s + + You can inspect the log file for more information about the run or in order + to diagnoze issues. If you pass V=1 in the environment, more diagnostic logs + would be found in the test log. + +tests/avocado/acpi-bits: + - README: This text file. + +tests/avocado/acpi-bits/bits-config: + This location contains biosbits config files that determine how the software + runs the tests. + - bits-config.txt: this is the biosbits config file that determines what tests + or actions are performed by bits. The description of the config options are + provided in the file itself. + +tests/avocado/acpi-bits/bits-tests: + This directory contains biosbits python based tests that are run from within + the biosbits environment in the spawned VM. New additions of test cases can + be made in the appropriate test file. For example, new acpi tests can go + into testacpi.py2 and one would call testsuite.add_test() to register the new + test so that it gets executed as a part of the ACPI tests. + It might be occasionally necessary to disable some subtests or add a new + test that belongs to a test suite not already present in this directory. To + do this, please clone the bits source from + https://gitlab.com/qemu-project/biosbits-bits/-/tree/qemu-bits. + Note that this is the "qemu-bits" branch and not the "bits" branch of the + repository. "qemu-bits" is the branch where we have made all the QEMU + specific enhancements and we must use the source from this branch only. + Copy the test suite/script that needs modification (addition of new tests + or disabling them) from python directory into this directory. For + example, in order to change cpuid related tests, copy the following + file into this directory and rename it with .py2 extension: + https://gitlab.com/qemu-project/biosbits-bits/-/blob/qemu-bits/python/testcpuid.py + Then make your additions and changes here. Hence, the steps are: + + step (a): copy unmodified test script to this directory from bits source. + step (b): perform modifications to the test. + Commit (a) and (b) should go under separate commits so that the original + test script and the changes we have made are separated and clear. + + The test framework will then use your modified test script to run the test. + No further changes would be needed. Please check the logs to make sure that + appropriate changes have taken effect. + + The tests have an extension .py2 in order to indicate that + (a) they are python2.7 based scripts and not python 3 scripts. + (b) they are run from within the bios bits VM and is not subjected to QEMU + build/test python script maintainance and dependency resolutions. + (c) They need not be loaded by avocado framework when running tests. + +Author: Ani Sinha <ani@anisinha.ca> + +References: +(a) https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf +(b) https://www.youtube.com/watch?v=36QIepyUuhg
Add a README file that describes the purpose of the various test files and gives guidance to developers on where and how to make changes. Cc: Daniel P. Berrange" <berrange@redhat.com> Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: Maydell Peter <peter.maydell@linaro.org> Cc: John Snow <jsnow@redhat.com> Cc: Thomas Huth <thuth@redhat.com> Cc: Alex Bennée <alex.bennee@linaro.org> Cc: Igor Mammedov <imammedo@redhat.com> Cc: Michael Tsirkin <mst@redhat.com> Signed-off-by: Ani Sinha <ani@anisinha.ca> --- tests/avocado/acpi-bits/README | 133 +++++++++++++++++++++++++++++++++ 1 file changed, 133 insertions(+) create mode 100644 tests/avocado/acpi-bits/README