| Message ID | 20200709024657.2500558-1-crosa@redhat.com |
|---|---|
| Headers | show |
| Series | QEMU Gating CI | expand |
On Thu, 9 Jul 2020 at 03:47, Cleber Rosa <crosa@redhat.com> wrote: > > The idea about a public facing Gating CI for QEMU was summarized in an > RFC[1]. Since then, it was decided that a simpler version should be > attempted first. > > At this point, there are two specific runners (an aarch64 and an s390) > registered with GitLab, at https://gitlab.com/qemu-project, currently > setup to the "qemu" repository. Could I have a simple "this is the command you need to run to test your staging branch with the project gitlab" set of getting-started instructions, please? thanks -- PMM
On Mon, Jul 20, 2020 at 05:18:54PM +0100, Peter Maydell wrote: > On Thu, 9 Jul 2020 at 03:47, Cleber Rosa <crosa@redhat.com> wrote: > > > > The idea about a public facing Gating CI for QEMU was summarized in an > > RFC[1]. Since then, it was decided that a simpler version should be > > attempted first. > > > > At this point, there are two specific runners (an aarch64 and an s390) > > registered with GitLab, at https://gitlab.com/qemu-project, currently > > setup to the "qemu" repository. > > Could I have a simple "this is the command you need to run to > test your staging branch with the project gitlab" set of > getting-started instructions, please? > > thanks > -- PMM > Hi Peter, Sure. It's important that PATCH 2/2 in this series is included in a branch that you need to push to the "staging" branch on the https://gitlab.com/qemu-project/qemu repo (it could be just that one patch). Then, you can run: ./scripts/ci/gitlab-pipeline-status --verbose -w And that should be it. You can drop '--verbose' if you just want the final outcome as the result. A longer explanation: that command defaults to checking for the pipeline associated with the commit for the *local* staging branch. You can check the pipeline results on other commits, such as the latest master (by the time I write this): $ ./scripts/ci/gitlab-pipeline-status --verbose -c 1e6c50ad8559c18b21041ef69d8fff781a8db0bb success Please let me if anything goes wrong, as I wasn't able to submit jobs to those two machines (did so on emulated ones) and the devil lies in the details. Thanks, - Cleber.
On Mon, 20 Jul 2020 at 18:22, Cleber Rosa <crosa@redhat.com> wrote: > Sure. It's important that PATCH 2/2 in this series is included in a > branch that you need to push to the "staging" branch on the > https://gitlab.com/qemu-project/qemu repo (it could be just that one > patch). Then, you can run: > > ./scripts/ci/gitlab-pipeline-status --verbose -w > > And that should be it. You can drop '--verbose' if you just want the > final outcome as the result. I tried this (local branch named "staging", pushed to gitlab remote "staging" branch), but it said: e104462:bionic:qemu$ ./scripts/ci/gitlab-pipeline-status --verbose -w ERROR: No pipeline found failure It does seem to have kicked off the pipeline on gitlab though: https://gitlab.com/qemu-project/qemu/-/pipelines/171671136/builds OTOH I can't see anything on that web page that suggests that it's submitting jobs to the s390 or aarch64 boxes -- is it intended to? thanks -- PMM
On Tue, Jul 28, 2020 at 03:48:38PM +0100, Peter Maydell wrote: > On Mon, 20 Jul 2020 at 18:22, Cleber Rosa <crosa@redhat.com> wrote: > > Sure. It's important that PATCH 2/2 in this series is included in a > > branch that you need to push to the "staging" branch on the > > https://gitlab.com/qemu-project/qemu repo (it could be just that one > > patch). Then, you can run: > > > > ./scripts/ci/gitlab-pipeline-status --verbose -w > > > > And that should be it. You can drop '--verbose' if you just want the > > final outcome as the result. > > I tried this (local branch named "staging", pushed to gitlab > remote "staging" branch), but it said: > > e104462:bionic:qemu$ ./scripts/ci/gitlab-pipeline-status --verbose -w > ERROR: No pipeline found > failure > > It does seem to have kicked off the pipeline on gitlab though: > https://gitlab.com/qemu-project/qemu/-/pipelines/171671136/builds > OTOH I can't see anything on that web page that suggests that > it's submitting jobs to the s390 or aarch64 boxes -- is it > intended to? It looks like those jobs are all in the "test" stage of the pipeline, so it is waiting for the earlier stages to complete before running the jobs. Regards, Daniel
On Tue, Jul 28, 2020 at 03:48:38PM +0100, Peter Maydell wrote: > On Mon, 20 Jul 2020 at 18:22, Cleber Rosa <crosa@redhat.com> wrote: > > Sure. It's important that PATCH 2/2 in this series is included in a > > branch that you need to push to the "staging" branch on the > > https://gitlab.com/qemu-project/qemu repo (it could be just that one > > patch). Then, you can run: > > > > ./scripts/ci/gitlab-pipeline-status --verbose -w > > > > And that should be it. You can drop '--verbose' if you just want the > > final outcome as the result. > > I tried this (local branch named "staging", pushed to gitlab > remote "staging" branch), but it said: > > e104462:bionic:qemu$ ./scripts/ci/gitlab-pipeline-status --verbose -w > ERROR: No pipeline found > failure > Hi Peter, I think this may just have been a timing issue. GitLab usually does take a few seconds after it receives a branch push to create a pipeline. Let me know if you'd like to see this within the script, or if you'd rather put a sleep between your push and the "gitlab-pipeline-status" execution. > It does seem to have kicked off the pipeline on gitlab though: > https://gitlab.com/qemu-project/qemu/-/pipelines/171671136/builds There's already new content on the staging branch, but supposing my local staging branch contained commit 6e7c2dcb50907aa6be0cbc37f81801d2fa67f7b4 (https://gitlab.com/qemu-project/qemu/-/commit/6e7c2dcb50907aa6be0cbc37f81801d2fa67f7b4), the command you ran: ./scripts/ci/gitlab-pipeline-status --verbose -w Should have behaved as this (output from my machine): /scripts/ci/gitlab-pipeline-status --verbose -w -c 6e7c2dcb50907aa6be0cbc37f81801d2fa67f7b4 running... > OTOH I can't see anything on that web page that suggests that > it's submitting jobs to the s390 or aarch64 boxes -- is it > intended to? > All the jobs for that pipeline have been created as expected, for instance: https://gitlab.com/qemu-project/qemu/-/jobs/659874849 But given the recent changes to the GitLab YAML adding other phases, it's waiting for the previous phases. The jobs introduced with this patch are in the test phase because they do builds and tests. But IIRC there's a way to tell jobs to avoid being blocked by previous phases. I'll look into that. Until then, I hope to see the job results soon from the s390 and aarch64 boxes. Thanks, - Cleber. > thanks > -- PMM >
On Tue, 28 Jul 2020 at 16:51, Cleber Rosa <crosa@redhat.com> wrote: > > On Tue, Jul 28, 2020 at 03:48:38PM +0100, Peter Maydell wrote: > > On Mon, 20 Jul 2020 at 18:22, Cleber Rosa <crosa@redhat.com> wrote: > > > Sure. It's important that PATCH 2/2 in this series is included in a > > > branch that you need to push to the "staging" branch on the > > > https://gitlab.com/qemu-project/qemu repo (it could be just that one > > > patch). Then, you can run: > > > > > > ./scripts/ci/gitlab-pipeline-status --verbose -w > > > > > > And that should be it. You can drop '--verbose' if you just want the > > > final outcome as the result. > > > > I tried this (local branch named "staging", pushed to gitlab > > remote "staging" branch), but it said: > > > > e104462:bionic:qemu$ ./scripts/ci/gitlab-pipeline-status --verbose -w > > ERROR: No pipeline found > > failure > > > > Hi Peter, > > I think this may just have been a timing issue. GitLab usually does > take a few seconds after it receives a branch push to create a > pipeline. Let me know if you'd like to see this within the script, or > if you'd rather put a sleep between your push and the > "gitlab-pipeline-status" execution. Ah, right. I ran the command again and it does (eventually) print "running...". I think the ideal behaviour would be for the script to have some kind of "waiting for pipeline to start..." phase where it sits and polls for the pipeline to appear, with a pretty long timeout (minutes?). > > It does seem to have kicked off the pipeline on gitlab though: > > https://gitlab.com/qemu-project/qemu/-/pipelines/171671136/builds > > There's already new content on the staging branch, but supposing my local > staging branch contained commit 6e7c2dcb50907aa6be0cbc37f81801d2fa67f7b4 > (https://gitlab.com/qemu-project/qemu/-/commit/6e7c2dcb50907aa6be0cbc37f81801d2fa67f7b4), > the command you ran: > > ./scripts/ci/gitlab-pipeline-status --verbose -w > > Should have behaved as this (output from my machine): > > /scripts/ci/gitlab-pipeline-status --verbose -w -c 6e7c2dcb50907aa6be0cbc37f81801d2fa67f7b4 > running... > > > OTOH I can't see anything on that web page that suggests that > > it's submitting jobs to the s390 or aarch64 boxes -- is it > > intended to? > > > > All the jobs for that pipeline have been created as expected, for > instance: > > https://gitlab.com/qemu-project/qemu/-/jobs/659874849 > > But given the recent changes to the GitLab YAML adding other phases, > it's waiting for the previous phases. The page now says "This job has been skipped"... thanks -- PMM
On Tue, Jul 28, 2020 at 03:51:34PM +0100, Daniel P. Berrangé wrote: > On Tue, Jul 28, 2020 at 03:48:38PM +0100, Peter Maydell wrote: > > On Mon, 20 Jul 2020 at 18:22, Cleber Rosa <crosa@redhat.com> wrote: > > > Sure. It's important that PATCH 2/2 in this series is included in a > > > branch that you need to push to the "staging" branch on the > > > https://gitlab.com/qemu-project/qemu repo (it could be just that one > > > patch). Then, you can run: > > > > > > ./scripts/ci/gitlab-pipeline-status --verbose -w > > > > > > And that should be it. You can drop '--verbose' if you just want the > > > final outcome as the result. > > > > I tried this (local branch named "staging", pushed to gitlab > > remote "staging" branch), but it said: > > > > e104462:bionic:qemu$ ./scripts/ci/gitlab-pipeline-status --verbose -w > > ERROR: No pipeline found > > failure > > > > It does seem to have kicked off the pipeline on gitlab though: > > https://gitlab.com/qemu-project/qemu/-/pipelines/171671136/builds > > OTOH I can't see anything on that web page that suggests that > > it's submitting jobs to the s390 or aarch64 boxes -- is it > > intended to? > > It looks like those jobs are all in the "test" stage of the pipeline, so > it is waiting for the earlier stages to complete before running the jobs. > Hi Daniel, Right. IIUC the criteria for putting jobs in the test stage at the moment is "if they include running tests (in addition to builds) they should be in test". Looking at that from this perspective, they're in the right place. But, these jobs don't depend on anything else, including container builds, so there's no reason to have them wait for so long to run. The solution may be to rename the first stage (containers) to something more generic (and unfortunately less descriptive) so that all of them will run concurrently and earlier. Thoughts? - Cleber. > Regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Jul 28, 2020 at 12:13:06PM -0400, Cleber Rosa wrote: > On Tue, Jul 28, 2020 at 03:51:34PM +0100, Daniel P. Berrangé wrote: > > On Tue, Jul 28, 2020 at 03:48:38PM +0100, Peter Maydell wrote: > > > On Mon, 20 Jul 2020 at 18:22, Cleber Rosa <crosa@redhat.com> wrote: > > > > Sure. It's important that PATCH 2/2 in this series is included in a > > > > branch that you need to push to the "staging" branch on the > > > > https://gitlab.com/qemu-project/qemu repo (it could be just that one > > > > patch). Then, you can run: > > > > > > > > ./scripts/ci/gitlab-pipeline-status --verbose -w > > > > > > > > And that should be it. You can drop '--verbose' if you just want the > > > > final outcome as the result. > > > > > > I tried this (local branch named "staging", pushed to gitlab > > > remote "staging" branch), but it said: > > > > > > e104462:bionic:qemu$ ./scripts/ci/gitlab-pipeline-status --verbose -w > > > ERROR: No pipeline found > > > failure > > > > > > It does seem to have kicked off the pipeline on gitlab though: > > > https://gitlab.com/qemu-project/qemu/-/pipelines/171671136/builds > > > OTOH I can't see anything on that web page that suggests that > > > it's submitting jobs to the s390 or aarch64 boxes -- is it > > > intended to? > > > > It looks like those jobs are all in the "test" stage of the pipeline, so > > it is waiting for the earlier stages to complete before running the jobs. > > > > Hi Daniel, > > Right. IIUC the criteria for putting jobs in the test stage at the > moment is "if they include running tests (in addition to builds) they > should be in test". Looking at that from this perspective, they're in > the right place. > > But, these jobs don't depend on anything else, including container > builds, so there's no reason to have them wait for so long to run. > The solution may be to rename the first stage (containers) to something > more generic (and unfortunately less descriptive) so that all of them > will run concurrently and earlier. Just add 'needs: []' to any jobs that don't depend on earlier jobs. Regards, Daniel
On Tue, Jul 28, 2020 at 05:15:17PM +0100, Daniel P. Berrangé wrote: > On Tue, Jul 28, 2020 at 12:13:06PM -0400, Cleber Rosa wrote: > > On Tue, Jul 28, 2020 at 03:51:34PM +0100, Daniel P. Berrangé wrote: > > > On Tue, Jul 28, 2020 at 03:48:38PM +0100, Peter Maydell wrote: > > > > On Mon, 20 Jul 2020 at 18:22, Cleber Rosa <crosa@redhat.com> wrote: > > > > > Sure. It's important that PATCH 2/2 in this series is included in a > > > > > branch that you need to push to the "staging" branch on the > > > > > https://gitlab.com/qemu-project/qemu repo (it could be just that one > > > > > patch). Then, you can run: > > > > > > > > > > ./scripts/ci/gitlab-pipeline-status --verbose -w > > > > > > > > > > And that should be it. You can drop '--verbose' if you just want the > > > > > final outcome as the result. > > > > > > > > I tried this (local branch named "staging", pushed to gitlab > > > > remote "staging" branch), but it said: > > > > > > > > e104462:bionic:qemu$ ./scripts/ci/gitlab-pipeline-status --verbose -w > > > > ERROR: No pipeline found > > > > failure > > > > > > > > It does seem to have kicked off the pipeline on gitlab though: > > > > https://gitlab.com/qemu-project/qemu/-/pipelines/171671136/builds > > > > OTOH I can't see anything on that web page that suggests that > > > > it's submitting jobs to the s390 or aarch64 boxes -- is it > > > > intended to? > > > > > > It looks like those jobs are all in the "test" stage of the pipeline, so > > > it is waiting for the earlier stages to complete before running the jobs. > > > > > > > Hi Daniel, > > > > Right. IIUC the criteria for putting jobs in the test stage at the > > moment is "if they include running tests (in addition to builds) they > > should be in test". Looking at that from this perspective, they're in > > the right place. > > > > But, these jobs don't depend on anything else, including container > > builds, so there's no reason to have them wait for so long to run. > > The solution may be to rename the first stage (containers) to something > > more generic (and unfortunately less descriptive) so that all of them > > will run concurrently and earlier. > > Just add 'needs: []' to any jobs that don't depend on earlier jobs. > Great, thanks! Long version: I remember something like this was possible (which I mentioned in the other reply), but the documentation about stages (https://docs.gitlab.com/ee/ci/yaml/#stages) made me loose hope when writing this reply. Thanks again! - Cleber. > Regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Tue, Jul 28, 2020 at 05:08:50PM +0100, Peter Maydell wrote: > On Tue, 28 Jul 2020 at 16:51, Cleber Rosa <crosa@redhat.com> wrote: > > > > On Tue, Jul 28, 2020 at 03:48:38PM +0100, Peter Maydell wrote: > > > On Mon, 20 Jul 2020 at 18:22, Cleber Rosa <crosa@redhat.com> wrote: > > > > Sure. It's important that PATCH 2/2 in this series is included in a > > > > branch that you need to push to the "staging" branch on the > > > > https://gitlab.com/qemu-project/qemu repo (it could be just that one > > > > patch). Then, you can run: > > > > > > > > ./scripts/ci/gitlab-pipeline-status --verbose -w > > > > > > > > And that should be it. You can drop '--verbose' if you just want the > > > > final outcome as the result. > > > > > > I tried this (local branch named "staging", pushed to gitlab > > > remote "staging" branch), but it said: > > > > > > e104462:bionic:qemu$ ./scripts/ci/gitlab-pipeline-status --verbose -w > > > ERROR: No pipeline found > > > failure > > > > > > > Hi Peter, > > > > I think this may just have been a timing issue. GitLab usually does > > take a few seconds after it receives a branch push to create a > > pipeline. Let me know if you'd like to see this within the script, or > > if you'd rather put a sleep between your push and the > > "gitlab-pipeline-status" execution. > > Ah, right. I ran the command again and it does (eventually) > print "running...". I think the ideal behaviour would be for > the script to have some kind of "waiting for pipeline to start..." > phase where it sits and polls for the pipeline to appear, > with a pretty long timeout (minutes?). > Fair enough. I'll send a patch to change the script behavior. > > > It does seem to have kicked off the pipeline on gitlab though: > > > https://gitlab.com/qemu-project/qemu/-/pipelines/171671136/builds > > > > There's already new content on the staging branch, but supposing my local > > staging branch contained commit 6e7c2dcb50907aa6be0cbc37f81801d2fa67f7b4 > > (https://gitlab.com/qemu-project/qemu/-/commit/6e7c2dcb50907aa6be0cbc37f81801d2fa67f7b4), > > the command you ran: > > > > ./scripts/ci/gitlab-pipeline-status --verbose -w > > > > Should have behaved as this (output from my machine): > > > > /scripts/ci/gitlab-pipeline-status --verbose -w -c 6e7c2dcb50907aa6be0cbc37f81801d2fa67f7b4 > > running... > > > > > OTOH I can't see anything on that web page that suggests that > > > it's submitting jobs to the s390 or aarch64 boxes -- is it > > > intended to? > > > > > > > All the jobs for that pipeline have been created as expected, for > > instance: > > > > https://gitlab.com/qemu-project/qemu/-/jobs/659874849 > > > > But given the recent changes to the GitLab YAML adding other phases, > > it's waiting for the previous phases. > > The page now says "This job has been skipped"... > I saw that, and I was very disappointed... I double checked the machines, the runners status, tag names and they all seem to be OK. So, I think the reason for the skip (there's an open issue on GitLab itself about not communicating to users the reason) is that GitLab does a late evaluation of the job condition. For those jobs the condition is: rules: - if: '$CI_COMMIT_REF_NAME == "staging"' Which by the time the job was evaluated it was no longer true (there was new content on the staging branch). There are multiple ways to solve the problem, including (and in my order of preference): 1. using '$CI_COMMIT_BRANCH' instead of '$CI_COMMIT_REF_NAME', given that the pushed branch name should be kept stable even if the content (thus reference name) changes 2. not changing anything if you believe that under normal circunstances one pipeline for the staging will be running at a time. I'll prepare a new version with #1, unless you have a strong feeling against it. - Cleber. > thanks > -- PMM >
On 7/28/20 6:33 PM, Cleber Rosa wrote: > On Tue, Jul 28, 2020 at 05:08:50PM +0100, Peter Maydell wrote: >> On Tue, 28 Jul 2020 at 16:51, Cleber Rosa <crosa@redhat.com> wrote: >>> ... >>>> OTOH I can't see anything on that web page that suggests that >>>> it's submitting jobs to the s390 or aarch64 boxes -- is it >>>> intended to? >>>> >>> >>> All the jobs for that pipeline have been created as expected, for >>> instance: >>> >>> https://gitlab.com/qemu-project/qemu/-/jobs/659874849 >>> >>> But given the recent changes to the GitLab YAML adding other phases, >>> it's waiting for the previous phases. >> >> The page now says "This job has been skipped"... >> > > I saw that, and I was very disappointed... I double checked the > machines, the runners status, tag names and they all seem to be OK. > > So, I think the reason for the skip (there's an open issue on GitLab > itself about not communicating to users the reason) is that GitLab > does a late evaluation of the job condition. For those jobs the > condition is: > > rules: > - if: '$CI_COMMIT_REF_NAME == "staging"' > > Which by the time the job was evaluated it was no longer true (there > was new content on the staging branch). There are multiple ways to > solve the problem, including (and in my order of preference): > > 1. using '$CI_COMMIT_BRANCH' instead of '$CI_COMMIT_REF_NAME', given > that the pushed branch name should be kept stable even if the content > (thus reference name) changes > > 2. not changing anything if you believe that under normal > circunstances one pipeline for the staging will be running at a > time. For mainstream, usually one at a time is enough, because if you tests various and one is merged, then you need to rerun on top of the merged one, so it is not very useful. For other users, it is useful. I'd certainly finish a patch, run the script, switch branch in another console, do some other work, run another instance of the script concurrently with the 1st one, etc... > > I'll prepare a new version with #1, unless you have a strong feeling > against it. > > - Cleber. > >> thanks >> -- PMM >>
On Tue, 28 Jul 2020 at 17:34, Cleber Rosa <crosa@redhat.com> wrote: > So, I think the reason for the skip (there's an open issue on GitLab > itself about not communicating to users the reason) is that GitLab > does a late evaluation of the job condition. For those jobs the > condition is: > > rules: > - if: '$CI_COMMIT_REF_NAME == "staging"' > > Which by the time the job was evaluated it was no longer true (there > was new content on the staging branch). What caused the new content on the staging branch? I only pushed the once (to go to commit 6e7c2dcb50907aa) -- is some other mechanism updating it? Looking at what it now seems to point to it looks like it's being mirrored from the git.qemu.org repo... Either we should turn that off, or tests need to be pushed to git.qemu.org:staging (which might introduce a delay if the mirror isn't immediate)... thanks -- PMM