| Message ID | 20181012165347.2117-1-crosa@redhat.com |
|---|---|
| Headers | show |
| Series | Bootstrap Python venv and acceptance/functional tests | expand |
On 12/10/2018 18:53, Cleber Rosa wrote: > TL;DR > ===== > > Allow acceptance tests to be run with `make check-acceptance`. > > Details > ======= > > This introduces a Python virtual environment that will be setup within > the QEMU build directory, that will contain the exact environment that > tests may require. > > There's one current caveat: it requires Python 3, as it's based on the > venv module. This was based on some discussions and perception about > standardizing on Python 3, but can easily be made to accommodate Python > 2 as well. > > Example of bootstrap and test execution on Travis-CI: > > https://travis-ci.org/qemu/qemu/jobs/439331028#L2508 If you activate Travis on your github account, you can test that in your namespace without having to open zombie pull requests there... A simple push to your repository will trigger a full Travis build. This is how I use it btw, canceling the jobs I'm not interested in, to quickly run the others. i.e. https://travis-ci.org/philmd/qemu/jobs/439573299#L5600 > > ... > VENV /home/travis/build/qemu/qemu/tests/venv > MKDIR /home/travis/build/qemu/qemu/tests/results > PIP /home/travis/build/qemu/qemu/tests/venv-requirements.txt > AVOCADO tests/acceptance > JOB ID : 920e4fcf55a1782f1ae77bee64b20ccdc2e1111d > JOB LOG : /home/travis/build/qemu/qemu/tests/results/job-2018-10-09T21.42-920e4fc/job.log > (1/6) /home/travis/build/qemu/qemu/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test: PASS (3.57 s) > (2/6) /home/travis/build/qemu/qemu/tests/acceptance/version.py:Version.test_qmp_human_info_version: PASS (0.04 s) > (3/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc: PASS (0.04 s) > (4/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: PASS (0.04 s) > (5/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: PASS (0.04 s) > (6/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password: PASS (0.04 s) > RESULTS : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 > JOB TIME : 3.90 s > ... > > Changes from v3: > ================ > > * Fixed typo in commit message (s/requiment/requirement/). (Eric) > > Changes from v2: > ================ > > * Make the $(TESTS_VENV_DIR) target depend on the > venv-requirements.txt file, and touch $(TESTS_VENV_DIR) after venv > runs. With this, updates on the file are reflected on the > venv. (Philippe) > > * Run pip with "python -m pip". It may have been installed reusing > the system wide packages, and then the script may not be available > on the venv. (Philippe) > > * Dropped Python version on Travis, and using the version supplied > by the distro (3.4). (Philippe) > > * Added "python3.4-venv" package requirement on Travis. (Philippe) > > * Added variable (AVOCADO_SHOW) with logging streams to be shown > while running the acceptance tests. By default it's set to none, > the equivalent of the quiet mode used on previous versions. > (Philippe) > > * On Travis, set the AVOCADO_SHOW variable to "app", so that the > individual test results can be easily seen. (Philippe) > > Ideas discussed, but not implemented: > > * Run pip with "$(PYTHON) -m pip -q install ..." because it points > to the system wide Python installation. (Philippe) > > * Drop the "--system-site-packages" flag. Waiting on another round > of tests to determine if they are really the cause of some package > installation problems. > > Changes from v1: > ================ > > * TESTS_VENV_REQ (the path of "venv-requirements.txt") now points to > the source path ($SRC_PATH instead of $BUILD_DIR) > > * Create the venv with "--system-site-packages", which allows the > reuse of packages (and no additional downloads) in case there's a > package installed system wide providing the same package and > version. > > * Run Avocado with "python -m avocado". It may have been installed > reusing the system wide packages, and then the script may not > be available on the venv. > > * Improved documentation describing the Python 3, venv and pip > requirements. > > * Updated avocado-framework requirement to latest released version > (65.0) > > * (New commit) Added support for running the acceptance tests on > Travis. > > Ideas discussed, but not implemented: > > * Install external packages such as python3-pip on Debian based > systems, deemed too invasive on developer's systems. > > * Allow the use of Python 2, and consequently the "virtualenv" > module. > > Cleber Rosa (3): > Bootstrap Python venv for tests > Acceptance tests: add make rule for running them > Travis support for the acceptance tests > > .travis.yml | 5 +++++ > docs/devel/testing.rst | 35 ++++++++++++++++++++++++++++++----- > tests/Makefile.include | 37 +++++++++++++++++++++++++++++++++++++ > tests/venv-requirements.txt | 4 ++++ > 4 files changed, 76 insertions(+), 5 deletions(-) > create mode 100644 tests/venv-requirements.txt >
On 10/12/18 5:44 PM, Philippe Mathieu-Daudé wrote: > On 12/10/2018 18:53, Cleber Rosa wrote: >> TL;DR >> ===== >> >> Allow acceptance tests to be run with `make check-acceptance`. >> >> Details >> ======= >> >> This introduces a Python virtual environment that will be setup within >> the QEMU build directory, that will contain the exact environment that >> tests may require. >> >> There's one current caveat: it requires Python 3, as it's based on the >> venv module. This was based on some discussions and perception about >> standardizing on Python 3, but can easily be made to accommodate Python >> 2 as well. >> >> Example of bootstrap and test execution on Travis-CI: >> >> https://travis-ci.org/qemu/qemu/jobs/439331028#L2508 > > If you activate Travis on your github account, you can test that in your > namespace without having to open zombie pull requests there... A simple > push to your repository will trigger a full Travis build. > > This is how I use it btw, canceling the jobs I'm not interested in, to > quickly run the others. > i.e. https://travis-ci.org/philmd/qemu/jobs/439573299#L5600 > Yep, I've done that shortly after sending this. :) Thanks for the tip, though. One thing I miss is the ability, without editing the .travis.yaml, to "choose" just one/few jobs in a build. Or maybe even the order they run (Of course I want the acceptance tests job first). Do you know anything about that? - Cleber. >> >> ... >> VENV /home/travis/build/qemu/qemu/tests/venv >> MKDIR /home/travis/build/qemu/qemu/tests/results >> PIP /home/travis/build/qemu/qemu/tests/venv-requirements.txt >> AVOCADO tests/acceptance >> JOB ID : 920e4fcf55a1782f1ae77bee64b20ccdc2e1111d >> JOB LOG : /home/travis/build/qemu/qemu/tests/results/job-2018-10-09T21.42-920e4fc/job.log >> (1/6) /home/travis/build/qemu/qemu/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test: PASS (3.57 s) >> (2/6) /home/travis/build/qemu/qemu/tests/acceptance/version.py:Version.test_qmp_human_info_version: PASS (0.04 s) >> (3/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc: PASS (0.04 s) >> (4/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: PASS (0.04 s) >> (5/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: PASS (0.04 s) >> (6/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password: PASS (0.04 s) >> RESULTS : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 >> JOB TIME : 3.90 s >> ... >> >> Changes from v3: >> ================ >> >> * Fixed typo in commit message (s/requiment/requirement/). (Eric) >> >> Changes from v2: >> ================ >> >> * Make the $(TESTS_VENV_DIR) target depend on the >> venv-requirements.txt file, and touch $(TESTS_VENV_DIR) after venv >> runs. With this, updates on the file are reflected on the >> venv. (Philippe) >> >> * Run pip with "python -m pip". It may have been installed reusing >> the system wide packages, and then the script may not be available >> on the venv. (Philippe) >> >> * Dropped Python version on Travis, and using the version supplied >> by the distro (3.4). (Philippe) >> >> * Added "python3.4-venv" package requirement on Travis. (Philippe) >> >> * Added variable (AVOCADO_SHOW) with logging streams to be shown >> while running the acceptance tests. By default it's set to none, >> the equivalent of the quiet mode used on previous versions. >> (Philippe) >> >> * On Travis, set the AVOCADO_SHOW variable to "app", so that the >> individual test results can be easily seen. (Philippe) >> >> Ideas discussed, but not implemented: >> >> * Run pip with "$(PYTHON) -m pip -q install ..." because it points >> to the system wide Python installation. (Philippe) >> >> * Drop the "--system-site-packages" flag. Waiting on another round >> of tests to determine if they are really the cause of some package >> installation problems. >> >> Changes from v1: >> ================ >> >> * TESTS_VENV_REQ (the path of "venv-requirements.txt") now points to >> the source path ($SRC_PATH instead of $BUILD_DIR) >> >> * Create the venv with "--system-site-packages", which allows the >> reuse of packages (and no additional downloads) in case there's a >> package installed system wide providing the same package and >> version. >> >> * Run Avocado with "python -m avocado". It may have been installed >> reusing the system wide packages, and then the script may not >> be available on the venv. >> >> * Improved documentation describing the Python 3, venv and pip >> requirements. >> >> * Updated avocado-framework requirement to latest released version >> (65.0) >> >> * (New commit) Added support for running the acceptance tests on >> Travis. >> >> Ideas discussed, but not implemented: >> >> * Install external packages such as python3-pip on Debian based >> systems, deemed too invasive on developer's systems. >> >> * Allow the use of Python 2, and consequently the "virtualenv" >> module. >> >> Cleber Rosa (3): >> Bootstrap Python venv for tests >> Acceptance tests: add make rule for running them >> Travis support for the acceptance tests >> >> .travis.yml | 5 +++++ >> docs/devel/testing.rst | 35 ++++++++++++++++++++++++++++++----- >> tests/Makefile.include | 37 +++++++++++++++++++++++++++++++++++++ >> tests/venv-requirements.txt | 4 ++++ >> 4 files changed, 76 insertions(+), 5 deletions(-) >> create mode 100644 tests/venv-requirements.txt >> >
On 16/10/2018 16:27, Cleber Rosa wrote: > On 10/12/18 5:44 PM, Philippe Mathieu-Daudé wrote: >> On 12/10/2018 18:53, Cleber Rosa wrote: >>> TL;DR >>> ===== >>> >>> Allow acceptance tests to be run with `make check-acceptance`. >>> >>> Details >>> ======= >>> >>> This introduces a Python virtual environment that will be setup within >>> the QEMU build directory, that will contain the exact environment that >>> tests may require. >>> >>> There's one current caveat: it requires Python 3, as it's based on the >>> venv module. This was based on some discussions and perception about >>> standardizing on Python 3, but can easily be made to accommodate Python >>> 2 as well. >>> >>> Example of bootstrap and test execution on Travis-CI: >>> >>> https://travis-ci.org/qemu/qemu/jobs/439331028#L2508 >> >> If you activate Travis on your github account, you can test that in your >> namespace without having to open zombie pull requests there... A simple >> push to your repository will trigger a full Travis build. >> >> This is how I use it btw, canceling the jobs I'm not interested in, to >> quickly run the others. >> i.e. https://travis-ci.org/philmd/qemu/jobs/439573299#L5600 >> > > Yep, I've done that shortly after sending this. :) Thanks for the tip, > though. > > One thing I miss is the ability, without editing the .travis.yaml, to > "choose" just one/few jobs in a build. Or maybe even the order they run > (Of course I want the acceptance tests job first). > > Do you know anything about that? I just add an extra commit that drop every tests but the ones I'm interested to run and keep this patch until I'm happy with the series. I push as 'myseries_only' then I simply do "git branch myseries_only~ myseries && git push series" to run against everything, go to check my mailbox and review patches until it finishes, then I redact the cover and post. > > - Cleber. > >>> >>> ... >>> VENV /home/travis/build/qemu/qemu/tests/venv >>> MKDIR /home/travis/build/qemu/qemu/tests/results >>> PIP /home/travis/build/qemu/qemu/tests/venv-requirements.txt >>> AVOCADO tests/acceptance >>> JOB ID : 920e4fcf55a1782f1ae77bee64b20ccdc2e1111d >>> JOB LOG : /home/travis/build/qemu/qemu/tests/results/job-2018-10-09T21.42-920e4fc/job.log >>> (1/6) /home/travis/build/qemu/qemu/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test: PASS (3.57 s) >>> (2/6) /home/travis/build/qemu/qemu/tests/acceptance/version.py:Version.test_qmp_human_info_version: PASS (0.04 s) >>> (3/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc: PASS (0.04 s) >>> (4/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: PASS (0.04 s) >>> (5/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: PASS (0.04 s) >>> (6/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password: PASS (0.04 s) >>> RESULTS : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 >>> JOB TIME : 3.90 s >>> ... >>> >>> Changes from v3: >>> ================ >>> >>> * Fixed typo in commit message (s/requiment/requirement/). (Eric) >>> >>> Changes from v2: >>> ================ >>> >>> * Make the $(TESTS_VENV_DIR) target depend on the >>> venv-requirements.txt file, and touch $(TESTS_VENV_DIR) after venv >>> runs. With this, updates on the file are reflected on the >>> venv. (Philippe) >>> >>> * Run pip with "python -m pip". It may have been installed reusing >>> the system wide packages, and then the script may not be available >>> on the venv. (Philippe) >>> >>> * Dropped Python version on Travis, and using the version supplied >>> by the distro (3.4). (Philippe) >>> >>> * Added "python3.4-venv" package requirement on Travis. (Philippe) >>> >>> * Added variable (AVOCADO_SHOW) with logging streams to be shown >>> while running the acceptance tests. By default it's set to none, >>> the equivalent of the quiet mode used on previous versions. >>> (Philippe) >>> >>> * On Travis, set the AVOCADO_SHOW variable to "app", so that the >>> individual test results can be easily seen. (Philippe) >>> >>> Ideas discussed, but not implemented: >>> >>> * Run pip with "$(PYTHON) -m pip -q install ..." because it points >>> to the system wide Python installation. (Philippe) >>> >>> * Drop the "--system-site-packages" flag. Waiting on another round >>> of tests to determine if they are really the cause of some package >>> installation problems. >>> >>> Changes from v1: >>> ================ >>> >>> * TESTS_VENV_REQ (the path of "venv-requirements.txt") now points to >>> the source path ($SRC_PATH instead of $BUILD_DIR) >>> >>> * Create the venv with "--system-site-packages", which allows the >>> reuse of packages (and no additional downloads) in case there's a >>> package installed system wide providing the same package and >>> version. >>> >>> * Run Avocado with "python -m avocado". It may have been installed >>> reusing the system wide packages, and then the script may not >>> be available on the venv. >>> >>> * Improved documentation describing the Python 3, venv and pip >>> requirements. >>> >>> * Updated avocado-framework requirement to latest released version >>> (65.0) >>> >>> * (New commit) Added support for running the acceptance tests on >>> Travis. >>> >>> Ideas discussed, but not implemented: >>> >>> * Install external packages such as python3-pip on Debian based >>> systems, deemed too invasive on developer's systems. >>> >>> * Allow the use of Python 2, and consequently the "virtualenv" >>> module. >>> >>> Cleber Rosa (3): >>> Bootstrap Python venv for tests >>> Acceptance tests: add make rule for running them >>> Travis support for the acceptance tests >>> >>> .travis.yml | 5 +++++ >>> docs/devel/testing.rst | 35 ++++++++++++++++++++++++++++++----- >>> tests/Makefile.include | 37 +++++++++++++++++++++++++++++++++++++ >>> tests/venv-requirements.txt | 4 ++++ >>> 4 files changed, 76 insertions(+), 5 deletions(-) >>> create mode 100644 tests/venv-requirements.txt >>> >> >
TL;DR ===== Allow acceptance tests to be run with `make check-acceptance`. Details ======= This introduces a Python virtual environment that will be setup within the QEMU build directory, that will contain the exact environment that tests may require. There's one current caveat: it requires Python 3, as it's based on the venv module. This was based on some discussions and perception about standardizing on Python 3, but can easily be made to accommodate Python 2 as well. Example of bootstrap and test execution on Travis-CI: https://travis-ci.org/qemu/qemu/jobs/439331028#L2508 ... VENV /home/travis/build/qemu/qemu/tests/venv MKDIR /home/travis/build/qemu/qemu/tests/results PIP /home/travis/build/qemu/qemu/tests/venv-requirements.txt AVOCADO tests/acceptance JOB ID : 920e4fcf55a1782f1ae77bee64b20ccdc2e1111d JOB LOG : /home/travis/build/qemu/qemu/tests/results/job-2018-10-09T21.42-920e4fc/job.log (1/6) /home/travis/build/qemu/qemu/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test: PASS (3.57 s) (2/6) /home/travis/build/qemu/qemu/tests/acceptance/version.py:Version.test_qmp_human_info_version: PASS (0.04 s) (3/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc: PASS (0.04 s) (4/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: PASS (0.04 s) (5/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: PASS (0.04 s) (6/6) /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password: PASS (0.04 s) RESULTS : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 JOB TIME : 3.90 s ... Changes from v3: ================ * Fixed typo in commit message (s/requiment/requirement/). (Eric) Changes from v2: ================ * Make the $(TESTS_VENV_DIR) target depend on the venv-requirements.txt file, and touch $(TESTS_VENV_DIR) after venv runs. With this, updates on the file are reflected on the venv. (Philippe) * Run pip with "python -m pip". It may have been installed reusing the system wide packages, and then the script may not be available on the venv. (Philippe) * Dropped Python version on Travis, and using the version supplied by the distro (3.4). (Philippe) * Added "python3.4-venv" package requirement on Travis. (Philippe) * Added variable (AVOCADO_SHOW) with logging streams to be shown while running the acceptance tests. By default it's set to none, the equivalent of the quiet mode used on previous versions. (Philippe) * On Travis, set the AVOCADO_SHOW variable to "app", so that the individual test results can be easily seen. (Philippe) Ideas discussed, but not implemented: * Run pip with "$(PYTHON) -m pip -q install ..." because it points to the system wide Python installation. (Philippe) * Drop the "--system-site-packages" flag. Waiting on another round of tests to determine if they are really the cause of some package installation problems. Changes from v1: ================ * TESTS_VENV_REQ (the path of "venv-requirements.txt") now points to the source path ($SRC_PATH instead of $BUILD_DIR) * Create the venv with "--system-site-packages", which allows the reuse of packages (and no additional downloads) in case there's a package installed system wide providing the same package and version. * Run Avocado with "python -m avocado". It may have been installed reusing the system wide packages, and then the script may not be available on the venv. * Improved documentation describing the Python 3, venv and pip requirements. * Updated avocado-framework requirement to latest released version (65.0) * (New commit) Added support for running the acceptance tests on Travis. Ideas discussed, but not implemented: * Install external packages such as python3-pip on Debian based systems, deemed too invasive on developer's systems. * Allow the use of Python 2, and consequently the "virtualenv" module. Cleber Rosa (3): Bootstrap Python venv for tests Acceptance tests: add make rule for running them Travis support for the acceptance tests .travis.yml | 5 +++++ docs/devel/testing.rst | 35 ++++++++++++++++++++++++++++++----- tests/Makefile.include | 37 +++++++++++++++++++++++++++++++++++++ tests/venv-requirements.txt | 4 ++++ 4 files changed, 76 insertions(+), 5 deletions(-) create mode 100644 tests/venv-requirements.txt