| Message ID | 20240623203213.1571666-20-sjg@chromium.org |
|---|---|
| State | Changes Requested |
| Delegated to: | Tom Rini |
| Headers | show |
| Series | labgrid: Provide an integration with Labgrid | expand |
On Sun, Jun 23, 2024 at 02:32:13PM -0600, Simon Glass wrote: > Add a way to run tests on a real hardware lab. This is in the very early > experimental stages. There are only 23 boards and 3 of those are broken! > (bob, ff3399, samus). A fourth fails due to problems with the TPM tests. > > To try this, assuming you have gitlab access, set SJG_LAB=1, e.g.: > > git push -o ci.variable="SJG_LAB=1" dm HEAD:try > > This relies on the two previous series targeted at -next as well as the > bugfix series for -master > > Signed-off-by: Simon Glass <sjg@chromium.org> Reviewed-by: Andrejs Cainikovs <andrejs.cainikovs@toradex.com> > --- > > Changes in v3: > - Split out most patches into two new series and update cover letter > > Changes in v2: > - Avoid running a docker image for skipped lab tests > > .gitlab-ci.yml | 153 +++++++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 153 insertions(+) > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml > index 165f765a833..75c18a0f2f7 100644 > --- a/.gitlab-ci.yml > +++ b/.gitlab-ci.yml > @@ -17,6 +17,7 @@ stages: > - testsuites > - test.py > - world build > + - sjg-lab > > .buildman_and_testpy_template: &buildman_and_testpy_dfn > stage: test.py > @@ -482,3 +483,155 @@ coreboot test.py: > TEST_PY_TEST_SPEC: "not sleep" > TEST_PY_ID: "--id qemu" > <<: *buildman_and_testpy_dfn > + > +.lab_template: &lab_dfn > + stage: sjg-lab > + rules: > + - if: $SJG_LAB == "1" > + when: always > + - when: manual > + tags: [ 'lab' ] > + script: > + - if [[ -z "${SJG_LAB}" ]]; then > + exit 0; > + fi > + # Environment: > + # SRC - source tree > + # OUT - output directory for builds > + - export SRC="$(pwd)" > + - export OUT="${SRC}/build/${BOARD}" > + - export PATH=$PATH:~/bin > + - export PATH=$PATH:/vid/software/devel/ubtest/u-boot-test-hooks/bin > + > + # Load it on the device > + - ret=0 > + - echo "role ${ROLE}" > + - export strategy="-s uboot -e off" > + # export verbose="-v" > + - ${SRC}/test/py/test.py --role ${ROLE} --build-dir "${OUT}" > + --capture=tee-sys -k "not bootstd"|| ret=$? > + - U_BOOT_BOARD_IDENTITY="${ROLE}" u-boot-test-release || true > + - if [[ $ret -ne 0 ]]; then > + exit $ret; > + fi > + artifacts: > + when: always > + paths: > + - "build/${BOARD}/test-log.html" > + - "build/${BOARD}/multiplexed_log.css" > + expire_in: 1 week > + > +rpi3: > + variables: > + ROLE: rpi3 > + <<: *lab_dfn > + > +opi_pc: > + variables: > + ROLE: opi_pc > + <<: *lab_dfn > + > +pcduino3_nano: > + variables: > + ROLE: pcduino3_nano > + <<: *lab_dfn > + > +samus: > + variables: > + ROLE: samus > + <<: *lab_dfn > + > +link: > + variables: > + ROLE: link > + <<: *lab_dfn > + > +jerry: > + variables: > + ROLE: jerry > + <<: *lab_dfn > + > +minnowmax: > + variables: > + ROLE: minnowmax > + <<: *lab_dfn > + > +opi_pc2: > + variables: > + ROLE: opi_pc2 > + <<: *lab_dfn > + > +bpi: > + variables: > + ROLE: bpi > + <<: *lab_dfn > + > +rpi2: > + variables: > + ROLE: rpi2 > + <<: *lab_dfn > + > +bob: > + variables: > + ROLE: bob > + <<: *lab_dfn > + > +ff3399: > + variables: > + ROLE: ff3399 > + <<: *lab_dfn > + > +coral: > + variables: > + ROLE: coral > + <<: *lab_dfn > + > +rpi3z: > + variables: > + ROLE: rpi3z > + <<: *lab_dfn > + > +bbb: > + variables: > + ROLE: bbb > + <<: *lab_dfn > + > +kevin: > + variables: > + ROLE: kevin > + <<: *lab_dfn > + > +pine64: > + variables: > + ROLE: pine64 > + <<: *lab_dfn > + > +c4: > + variables: > + ROLE: c4 > + <<: *lab_dfn > + > +rpi4: > + variables: > + ROLE: rpi4 > + <<: *lab_dfn > + > +rpi0: > + variables: > + ROLE: rpi0 > + <<: *lab_dfn > + > +snow: > + variables: > + ROLE: snow > + <<: *lab_dfn > + > +pcduino3: > + variables: > + ROLE: pcduino3 > + <<: *lab_dfn > + > +nyan-big: > + variables: > + ROLE: nyan-big > + <<: *lab_dfn > -- > 2.34.1 >
Hi Simon On Mon, Jun 24, 2024 at 2:52 PM Andrejs Cainikovs <andrejs.cainikovs@toradex.com> wrote: > > On Sun, Jun 23, 2024 at 02:32:13PM -0600, Simon Glass wrote: > > Add a way to run tests on a real hardware lab. This is in the very early > > experimental stages. There are only 23 boards and 3 of those are broken! > > (bob, ff3399, samus). A fourth fails due to problems with the TPM tests. > > > > To try this, assuming you have gitlab access, set SJG_LAB=1, e.g.: > > > > git push -o ci.variable="SJG_LAB=1" dm HEAD:try > > > > This relies on the two previous series targeted at -next as well as the > > bugfix series for -master > > > > Signed-off-by: Simon Glass <sjg@chromium.org> > > Reviewed-by: Andrejs Cainikovs <andrejs.cainikovs@toradex.com> > Do you have documentation on how to set it? We would like to do in our company too Michael > > --- > > > > Changes in v3: > > - Split out most patches into two new series and update cover letter > > > > Changes in v2: > > - Avoid running a docker image for skipped lab tests > > > > .gitlab-ci.yml | 153 +++++++++++++++++++++++++++++++++++++++++++++++++ > > 1 file changed, 153 insertions(+) > > > > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml > > index 165f765a833..75c18a0f2f7 100644 > > --- a/.gitlab-ci.yml > > +++ b/.gitlab-ci.yml > > @@ -17,6 +17,7 @@ stages: > > - testsuites > > - test.py > > - world build > > + - sjg-lab > > > > .buildman_and_testpy_template: &buildman_and_testpy_dfn > > stage: test.py > > @@ -482,3 +483,155 @@ coreboot test.py: > > TEST_PY_TEST_SPEC: "not sleep" > > TEST_PY_ID: "--id qemu" > > <<: *buildman_and_testpy_dfn > > + > > +.lab_template: &lab_dfn > > + stage: sjg-lab > > + rules: > > + - if: $SJG_LAB == "1" > > + when: always > > + - when: manual > > + tags: [ 'lab' ] > > + script: > > + - if [[ -z "${SJG_LAB}" ]]; then > > + exit 0; > > + fi > > + # Environment: > > + # SRC - source tree > > + # OUT - output directory for builds > > + - export SRC="$(pwd)" > > + - export OUT="${SRC}/build/${BOARD}" > > + - export PATH=$PATH:~/bin > > + - export PATH=$PATH:/vid/software/devel/ubtest/u-boot-test-hooks/bin > > + > > + # Load it on the device > > + - ret=0 > > + - echo "role ${ROLE}" > > + - export strategy="-s uboot -e off" > > + # export verbose="-v" > > + - ${SRC}/test/py/test.py --role ${ROLE} --build-dir "${OUT}" > > + --capture=tee-sys -k "not bootstd"|| ret=$? > > + - U_BOOT_BOARD_IDENTITY="${ROLE}" u-boot-test-release || true > > + - if [[ $ret -ne 0 ]]; then > > + exit $ret; > > + fi > > + artifacts: > > + when: always > > + paths: > > + - "build/${BOARD}/test-log.html" > > + - "build/${BOARD}/multiplexed_log.css" > > + expire_in: 1 week > > + > > +rpi3: > > + variables: > > + ROLE: rpi3 > > + <<: *lab_dfn > > + > > +opi_pc: > > + variables: > > + ROLE: opi_pc > > + <<: *lab_dfn > > + > > +pcduino3_nano: > > + variables: > > + ROLE: pcduino3_nano > > + <<: *lab_dfn > > + > > +samus: > > + variables: > > + ROLE: samus > > + <<: *lab_dfn > > + > > +link: > > + variables: > > + ROLE: link > > + <<: *lab_dfn > > + > > +jerry: > > + variables: > > + ROLE: jerry > > + <<: *lab_dfn > > + > > +minnowmax: > > + variables: > > + ROLE: minnowmax > > + <<: *lab_dfn > > + > > +opi_pc2: > > + variables: > > + ROLE: opi_pc2 > > + <<: *lab_dfn > > + > > +bpi: > > + variables: > > + ROLE: bpi > > + <<: *lab_dfn > > + > > +rpi2: > > + variables: > > + ROLE: rpi2 > > + <<: *lab_dfn > > + > > +bob: > > + variables: > > + ROLE: bob > > + <<: *lab_dfn > > + > > +ff3399: > > + variables: > > + ROLE: ff3399 > > + <<: *lab_dfn > > + > > +coral: > > + variables: > > + ROLE: coral > > + <<: *lab_dfn > > + > > +rpi3z: > > + variables: > > + ROLE: rpi3z > > + <<: *lab_dfn > > + > > +bbb: > > + variables: > > + ROLE: bbb > > + <<: *lab_dfn > > + > > +kevin: > > + variables: > > + ROLE: kevin > > + <<: *lab_dfn > > + > > +pine64: > > + variables: > > + ROLE: pine64 > > + <<: *lab_dfn > > + > > +c4: > > + variables: > > + ROLE: c4 > > + <<: *lab_dfn > > + > > +rpi4: > > + variables: > > + ROLE: rpi4 > > + <<: *lab_dfn > > + > > +rpi0: > > + variables: > > + ROLE: rpi0 > > + <<: *lab_dfn > > + > > +snow: > > + variables: > > + ROLE: snow > > + <<: *lab_dfn > > + > > +pcduino3: > > + variables: > > + ROLE: pcduino3 > > + <<: *lab_dfn > > + > > +nyan-big: > > + variables: > > + ROLE: nyan-big > > + <<: *lab_dfn > > -- > > 2.34.1 > > >
On Mon, Jun 24, 2024 at 04:56:02PM +0200, Michael Nazzareno Trimarchi wrote: > Hi Simon > > On Mon, Jun 24, 2024 at 2:52 PM Andrejs Cainikovs > <andrejs.cainikovs@toradex.com> wrote: > > > > On Sun, Jun 23, 2024 at 02:32:13PM -0600, Simon Glass wrote: > > > Add a way to run tests on a real hardware lab. This is in the very early > > > experimental stages. There are only 23 boards and 3 of those are broken! > > > (bob, ff3399, samus). A fourth fails due to problems with the TPM tests. > > > > > > To try this, assuming you have gitlab access, set SJG_LAB=1, e.g.: > > > > > > git push -o ci.variable="SJG_LAB=1" dm HEAD:try > > > > > > This relies on the two previous series targeted at -next as well as the > > > bugfix series for -master > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org> > > > > Reviewed-by: Andrejs Cainikovs <andrejs.cainikovs@toradex.com> > > > > Do you have documentation on how to set it? We would like to do in our > company too I _think_ from some talking with Simon before the biggest sticking point might be changes needed on the labgrid side of things. However, that might also be most easily remedied if there's a few people showing up in the GitHub issue(s) showing interest in getting changes made/merged and to use the overall feature.
Hi, On Mon, 24 Jun 2024 at 19:01, Tom Rini <trini@konsulko.com> wrote: > > On Mon, Jun 24, 2024 at 04:56:02PM +0200, Michael Nazzareno Trimarchi wrote: > > Hi Simon > > > > On Mon, Jun 24, 2024 at 2:52 PM Andrejs Cainikovs > > <andrejs.cainikovs@toradex.com> wrote: > > > > > > On Sun, Jun 23, 2024 at 02:32:13PM -0600, Simon Glass wrote: > > > > Add a way to run tests on a real hardware lab. This is in the very early > > > > experimental stages. There are only 23 boards and 3 of those are broken! > > > > (bob, ff3399, samus). A fourth fails due to problems with the TPM tests. > > > > > > > > To try this, assuming you have gitlab access, set SJG_LAB=1, e.g.: > > > > > > > > git push -o ci.variable="SJG_LAB=1" dm HEAD:try > > > > > > > > This relies on the two previous series targeted at -next as well as the > > > > bugfix series for -master > > > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org> > > > > > > Reviewed-by: Andrejs Cainikovs <andrejs.cainikovs@toradex.com> > > > > > > > Do you have documentation on how to set it? We would like to do in our > > company too > > I _think_ from some talking with Simon before the biggest sticking point > might be changes needed on the labgrid side of things. However, that > might also be most easily remedied if there's a few people showing up in > the GitHub issue(s) showing interest in getting changes made/merged and > to use the overall feature. The documentation is in the PR [1] mostly in the last commit [2]. Yes it would really help for you to try it out and comment on the PR. I may end up splitting it into a few separate PRs, but code review on the project is very limited, from what I have seen so far. You will see an example of my lab (devices and environment file). I also have a few minor updates to the PR which I just uploaded, to work on top of the grpc branch and to support QEMU. [1] https://github.com/labgrid-project/labgrid/pull/1411 [2] https://github.com/labgrid-project/labgrid/pull/1411/commits/c4b13af0e6169228c9adef03d4b66401201edd23 Regards, Simon
Hi Tom, On Tue, 25 Jun 2024 at 06:30, Simon Glass <sjg@chromium.org> wrote: > > Hi, > > On Mon, 24 Jun 2024 at 19:01, Tom Rini <trini@konsulko.com> wrote: > > > > On Mon, Jun 24, 2024 at 04:56:02PM +0200, Michael Nazzareno Trimarchi wrote: > > > Hi Simon > > > > > > On Mon, Jun 24, 2024 at 2:52 PM Andrejs Cainikovs > > > <andrejs.cainikovs@toradex.com> wrote: > > > > > > > > On Sun, Jun 23, 2024 at 02:32:13PM -0600, Simon Glass wrote: > > > > > Add a way to run tests on a real hardware lab. This is in the very early > > > > > experimental stages. There are only 23 boards and 3 of those are broken! > > > > > (bob, ff3399, samus). A fourth fails due to problems with the TPM tests. > > > > > > > > > > To try this, assuming you have gitlab access, set SJG_LAB=1, e.g.: > > > > > > > > > > git push -o ci.variable="SJG_LAB=1" dm HEAD:try > > > > > > > > > > This relies on the two previous series targeted at -next as well as the > > > > > bugfix series for -master > > > > > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org> > > > > > > > > Reviewed-by: Andrejs Cainikovs <andrejs.cainikovs@toradex.com> > > > > > > > > > > Do you have documentation on how to set it? We would like to do in our > > > company too > > > > I _think_ from some talking with Simon before the biggest sticking point > > might be changes needed on the labgrid side of things. However, that > > might also be most easily remedied if there's a few people showing up in > > the GitHub issue(s) showing interest in getting changes made/merged and > > to use the overall feature. > > The documentation is in the PR [1] mostly in the last commit [2]. > > Yes it would really help for you to try it out and comment on the PR. > I may end up splitting it into a few separate PRs, but code review on > the project is very limited, from what I have seen so far. You will > see an example of my lab (devices and environment file). > > I also have a few minor updates to the PR which I just uploaded, to > work on top of the grpc branch and to support QEMU. Just to mention that I updated the Labgrid integration to support beagleplay (which as you know combines the U-Boot builds for two boards). It resulted in no changes at all to this series. So perhaps this series can be reviewed and some of it applied? Documentation is below. > > [1] https://github.com/labgrid-project/labgrid/pull/1411 > [2] https://github.com/labgrid-project/labgrid/pull/1411/commits/c4b13af0e6169228c9adef03d4b66401201edd23 Regards, Simon
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 165f765a833..75c18a0f2f7 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -17,6 +17,7 @@ stages: - testsuites - test.py - world build + - sjg-lab .buildman_and_testpy_template: &buildman_and_testpy_dfn stage: test.py @@ -482,3 +483,155 @@ coreboot test.py: TEST_PY_TEST_SPEC: "not sleep" TEST_PY_ID: "--id qemu" <<: *buildman_and_testpy_dfn + +.lab_template: &lab_dfn + stage: sjg-lab + rules: + - if: $SJG_LAB == "1" + when: always + - when: manual + tags: [ 'lab' ] + script: + - if [[ -z "${SJG_LAB}" ]]; then + exit 0; + fi + # Environment: + # SRC - source tree + # OUT - output directory for builds + - export SRC="$(pwd)" + - export OUT="${SRC}/build/${BOARD}" + - export PATH=$PATH:~/bin + - export PATH=$PATH:/vid/software/devel/ubtest/u-boot-test-hooks/bin + + # Load it on the device + - ret=0 + - echo "role ${ROLE}" + - export strategy="-s uboot -e off" + # export verbose="-v" + - ${SRC}/test/py/test.py --role ${ROLE} --build-dir "${OUT}" + --capture=tee-sys -k "not bootstd"|| ret=$? + - U_BOOT_BOARD_IDENTITY="${ROLE}" u-boot-test-release || true + - if [[ $ret -ne 0 ]]; then + exit $ret; + fi + artifacts: + when: always + paths: + - "build/${BOARD}/test-log.html" + - "build/${BOARD}/multiplexed_log.css" + expire_in: 1 week + +rpi3: + variables: + ROLE: rpi3 + <<: *lab_dfn + +opi_pc: + variables: + ROLE: opi_pc + <<: *lab_dfn + +pcduino3_nano: + variables: + ROLE: pcduino3_nano + <<: *lab_dfn + +samus: + variables: + ROLE: samus + <<: *lab_dfn + +link: + variables: + ROLE: link + <<: *lab_dfn + +jerry: + variables: + ROLE: jerry + <<: *lab_dfn + +minnowmax: + variables: + ROLE: minnowmax + <<: *lab_dfn + +opi_pc2: + variables: + ROLE: opi_pc2 + <<: *lab_dfn + +bpi: + variables: + ROLE: bpi + <<: *lab_dfn + +rpi2: + variables: + ROLE: rpi2 + <<: *lab_dfn + +bob: + variables: + ROLE: bob + <<: *lab_dfn + +ff3399: + variables: + ROLE: ff3399 + <<: *lab_dfn + +coral: + variables: + ROLE: coral + <<: *lab_dfn + +rpi3z: + variables: + ROLE: rpi3z + <<: *lab_dfn + +bbb: + variables: + ROLE: bbb + <<: *lab_dfn + +kevin: + variables: + ROLE: kevin + <<: *lab_dfn + +pine64: + variables: + ROLE: pine64 + <<: *lab_dfn + +c4: + variables: + ROLE: c4 + <<: *lab_dfn + +rpi4: + variables: + ROLE: rpi4 + <<: *lab_dfn + +rpi0: + variables: + ROLE: rpi0 + <<: *lab_dfn + +snow: + variables: + ROLE: snow + <<: *lab_dfn + +pcduino3: + variables: + ROLE: pcduino3 + <<: *lab_dfn + +nyan-big: + variables: + ROLE: nyan-big + <<: *lab_dfn
Add a way to run tests on a real hardware lab. This is in the very early experimental stages. There are only 23 boards and 3 of those are broken! (bob, ff3399, samus). A fourth fails due to problems with the TPM tests. To try this, assuming you have gitlab access, set SJG_LAB=1, e.g.: git push -o ci.variable="SJG_LAB=1" dm HEAD:try This relies on the two previous series targeted at -next as well as the bugfix series for -master Signed-off-by: Simon Glass <sjg@chromium.org> --- Changes in v3: - Split out most patches into two new series and update cover letter Changes in v2: - Avoid running a docker image for skipped lab tests .gitlab-ci.yml | 153 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 153 insertions(+)