Message ID | 20230912133829.61352-1-stefanha@redhat.com |
---|---|
State | New |
Headers | show |
Series | gitlab-ci/cirrus: Increase timeout to 100 minutes | expand |
On Tue, Sep 12, 2023 at 09:38:29AM -0400, Stefan Hajnoczi wrote: > The 80m timeout is not enough: > > 672/832 qemu:block / io-qcow2-041 OK 39.77s 1 subtests passed > Timed out! IIUC, that 'timed out' message is coming from Cirrus CI logs, which we can see over on the cirrus task: https://cirrus-ci.com/task/6462328380588032 > https://gitlab.com/qemu-project/qemu/-/jobs/5058610599 This reports duration "64 minutes", vs a GitLab timeout of 1hr20. IOW, we're not hitting the gitlab timeout, we're hitting hte Cirrus CI timeout, which defaults to 60 minutes. The other 4 minuts gitlab reports is likely because Cirrus queued the job for 4 minutes before starting execution. > > Buglink: https://gitlab.com/qemu-project/qemu/-/issues/1882 > Fixes: d06f3bf92267 ("gitlab-ci/cirrus: Increase timeout to 80 minutes") > Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> > --- > .gitlab-ci.d/cirrus.yml | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml > index 41d64d6680..d19633f758 100644 > --- a/.gitlab-ci.d/cirrus.yml > +++ b/.gitlab-ci.d/cirrus.yml > @@ -15,7 +15,7 @@ > stage: build > image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master > needs: [] > - timeout: 80m > + timeout: 100m > allow_failure: true > script: > - source .gitlab-ci.d/cirrus/$NAME.vars IIUC, we need to put a 'timeout_in' setting someone in .gitlab-ci.d/cirrus/build.yml instead, to override Cirrus 60 minute limit: https://cirrus-ci.org/faq/#instance-timed-out With regards, Daniel
On Tue, 12 Sept 2023 at 09:53, Daniel P. Berrangé <berrange@redhat.com> wrote: > > On Tue, Sep 12, 2023 at 09:38:29AM -0400, Stefan Hajnoczi wrote: > > The 80m timeout is not enough: > > > > 672/832 qemu:block / io-qcow2-041 OK 39.77s 1 subtests passed > > Timed out! > > IIUC, that 'timed out' message is coming from Cirrus CI logs, which > we can see over on the cirrus task: > > https://cirrus-ci.com/task/6462328380588032 > > > https://gitlab.com/qemu-project/qemu/-/jobs/5058610599 > > This reports duration "64 minutes", vs a GitLab timeout of 1hr20. > > IOW, we're not hitting the gitlab timeout, we're hitting hte > Cirrus CI timeout, which defaults to 60 minutes. The other > 4 minuts gitlab reports is likely because Cirrus queued the > job for 4 minutes before starting execution. I'm glad you spotted that. I'm not familiar with Cirrus. Could you send a patch that sets 'timeout_in'? Thanks, Stefan > > > > > Buglink: https://gitlab.com/qemu-project/qemu/-/issues/1882 > > Fixes: d06f3bf92267 ("gitlab-ci/cirrus: Increase timeout to 80 minutes") > > Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> > > --- > > .gitlab-ci.d/cirrus.yml | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml > > index 41d64d6680..d19633f758 100644 > > --- a/.gitlab-ci.d/cirrus.yml > > +++ b/.gitlab-ci.d/cirrus.yml > > @@ -15,7 +15,7 @@ > > stage: build > > image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master > > needs: [] > > - timeout: 80m > > + timeout: 100m > > allow_failure: true > > script: > > - source .gitlab-ci.d/cirrus/$NAME.vars > > IIUC, we need to put a 'timeout_in' setting someone in > .gitlab-ci.d/cirrus/build.yml instead, to override > Cirrus 60 minute limit: > > https://cirrus-ci.org/faq/#instance-timed-out > > > With regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| > >
On Tue, Sep 12, 2023 at 10:02:17AM -0400, Stefan Hajnoczi wrote: > On Tue, 12 Sept 2023 at 09:53, Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > On Tue, Sep 12, 2023 at 09:38:29AM -0400, Stefan Hajnoczi wrote: > > > The 80m timeout is not enough: > > > > > > 672/832 qemu:block / io-qcow2-041 OK 39.77s 1 subtests passed > > > Timed out! > > > > IIUC, that 'timed out' message is coming from Cirrus CI logs, which > > we can see over on the cirrus task: > > > > https://cirrus-ci.com/task/6462328380588032 > > > > > https://gitlab.com/qemu-project/qemu/-/jobs/5058610599 > > > > This reports duration "64 minutes", vs a GitLab timeout of 1hr20. > > > > IOW, we're not hitting the gitlab timeout, we're hitting hte > > Cirrus CI timeout, which defaults to 60 minutes. The other > > 4 minuts gitlab reports is likely because Cirrus queued the > > job for 4 minutes before starting execution. > > I'm glad you spotted that. I'm not familiar with Cirrus. Could you > send a patch that sets 'timeout_in'? Yes, testing now https://gitlab.com/berrange/qemu/-/commit/c15d677de5ed2965464bc6212f049ed9785c4434 https://gitlab.com/berrange/qemu/-/jobs/5069195895 https://cirrus-ci.com/task/5135339078025216 The cirrus CI job page looks to be picking up the elevated timeout. With regards, Daniel
Thank you! Stefan On Tue, 12 Sept 2023 at 10:15, Daniel P. Berrangé <berrange@redhat.com> wrote: > > On Tue, Sep 12, 2023 at 10:02:17AM -0400, Stefan Hajnoczi wrote: > > On Tue, 12 Sept 2023 at 09:53, Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > > > On Tue, Sep 12, 2023 at 09:38:29AM -0400, Stefan Hajnoczi wrote: > > > > The 80m timeout is not enough: > > > > > > > > 672/832 qemu:block / io-qcow2-041 OK 39.77s 1 subtests passed > > > > Timed out! > > > > > > IIUC, that 'timed out' message is coming from Cirrus CI logs, which > > > we can see over on the cirrus task: > > > > > > https://cirrus-ci.com/task/6462328380588032 > > > > > > > https://gitlab.com/qemu-project/qemu/-/jobs/5058610599 > > > > > > This reports duration "64 minutes", vs a GitLab timeout of 1hr20. > > > > > > IOW, we're not hitting the gitlab timeout, we're hitting hte > > > Cirrus CI timeout, which defaults to 60 minutes. The other > > > 4 minuts gitlab reports is likely because Cirrus queued the > > > job for 4 minutes before starting execution. > > > > I'm glad you spotted that. I'm not familiar with Cirrus. Could you > > send a patch that sets 'timeout_in'? > > Yes, testing now > > https://gitlab.com/berrange/qemu/-/commit/c15d677de5ed2965464bc6212f049ed9785c4434 > > https://gitlab.com/berrange/qemu/-/jobs/5069195895 > > https://cirrus-ci.com/task/5135339078025216 > > The cirrus CI job page looks to be picking up the elevated timeout. > > > With regards, > Daniel > -- > |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o- https://fstop138.berrange.com :| > |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| >
On Tue, Sep 12, 2023 at 03:15:28PM +0100, Daniel P. Berrangé wrote: > On Tue, Sep 12, 2023 at 10:02:17AM -0400, Stefan Hajnoczi wrote: > > On Tue, 12 Sept 2023 at 09:53, Daniel P. Berrangé <berrange@redhat.com> wrote: > > > > > > On Tue, Sep 12, 2023 at 09:38:29AM -0400, Stefan Hajnoczi wrote: > > > > The 80m timeout is not enough: > > > > > > > > 672/832 qemu:block / io-qcow2-041 OK 39.77s 1 subtests passed > > > > Timed out! > > > > > > IIUC, that 'timed out' message is coming from Cirrus CI logs, which > > > we can see over on the cirrus task: > > > > > > https://cirrus-ci.com/task/6462328380588032 > > > > > > > https://gitlab.com/qemu-project/qemu/-/jobs/5058610599 > > > > > > This reports duration "64 minutes", vs a GitLab timeout of 1hr20. > > > > > > IOW, we're not hitting the gitlab timeout, we're hitting hte > > > Cirrus CI timeout, which defaults to 60 minutes. The other > > > 4 minuts gitlab reports is likely because Cirrus queued the > > > job for 4 minutes before starting execution. > > > > I'm glad you spotted that. I'm not familiar with Cirrus. Could you > > send a patch that sets 'timeout_in'? > > Yes, testing now > > https://gitlab.com/berrange/qemu/-/commit/c15d677de5ed2965464bc6212f049ed9785c4434 > > https://gitlab.com/berrange/qemu/-/jobs/5069195895 > > https://cirrus-ci.com/task/5135339078025216 > > The cirrus CI job page looks to be picking up the elevated timeout. Still fails, and given it previously worked at 35 minutes, needing a 1h30 timeout would be incredulous. This appears to be a genuine hang in QEMU's test suite on FreeBSD 13.2. I'm spinning up a VM to debug further With regards, Daniel
diff --git a/.gitlab-ci.d/cirrus.yml b/.gitlab-ci.d/cirrus.yml index 41d64d6680..d19633f758 100644 --- a/.gitlab-ci.d/cirrus.yml +++ b/.gitlab-ci.d/cirrus.yml @@ -15,7 +15,7 @@ stage: build image: registry.gitlab.com/libvirt/libvirt-ci/cirrus-run:master needs: [] - timeout: 80m + timeout: 100m allow_failure: true script: - source .gitlab-ci.d/cirrus/$NAME.vars
The 80m timeout is not enough: 672/832 qemu:block / io-qcow2-041 OK 39.77s 1 subtests passed Timed out! https://gitlab.com/qemu-project/qemu/-/jobs/5058610599 Buglink: https://gitlab.com/qemu-project/qemu/-/issues/1882 Fixes: d06f3bf92267 ("gitlab-ci/cirrus: Increase timeout to 80 minutes") Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com> --- .gitlab-ci.d/cirrus.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)