| Message ID | 20240620133346.1135479-1-sjg@chromium.org |
|---|---|
| State | RFC |
| Delegated to: | Tom Rini |
| Headers | show |
| Series | RFC: Add a tag for the world builds | expand |
On Thu, Jun 20, 2024 at 07:33:46AM -0600, Simon Glass wrote: > Currently the world builds run on all runners, including faster and > slower ones. > > The difference can be quite dramatic, with some builders 4x as fast as > others, resulting in just one world build taking between 20 minutes and > an hour and 20 minutes. > > Add a tag so that we can select which builders run these CPU-intensive > jobs. > > With this tag we can also increase CPU utilisation by running multiple > QEMU tests in parallel. Currently these tests leave most machines fairly > idle, since we cannot run more than one world build on a machine. > > Signed-off-by: Simon Glass <sjg@chromium.org> This conflicts I think with Jiaxun's desire to make our GitLab job runnable on the public runners too, and where we'll end up with 10 world build jobs ala Azure.
Hi Tom, On Thu, 20 Jun 2024 at 08:39, Tom Rini <trini@konsulko.com> wrote: > > On Thu, Jun 20, 2024 at 07:33:46AM -0600, Simon Glass wrote: > > > Currently the world builds run on all runners, including faster and > > slower ones. > > > > The difference can be quite dramatic, with some builders 4x as fast as > > others, resulting in just one world build taking between 20 minutes and > > an hour and 20 minutes. > > > > Add a tag so that we can select which builders run these CPU-intensive > > jobs. > > > > With this tag we can also increase CPU utilisation by running multiple > > QEMU tests in parallel. Currently these tests leave most machines fairly > > idle, since we cannot run more than one world build on a machine. > > > > Signed-off-by: Simon Glass <sjg@chromium.org> > > This conflicts I think with Jiaxun's desire to make our GitLab job > runnable on the public runners too, and where we'll end up with 10 world > build jobs ala Azure. It probably doesn't actually conflict, although I am not sure if one can add a tag to jobs that run on public runners. Regards, Simon
On Thu, Jun 20, 2024 at 05:05:26PM -0600, Simon Glass wrote: > Hi Tom, > > On Thu, 20 Jun 2024 at 08:39, Tom Rini <trini@konsulko.com> wrote: > > > > On Thu, Jun 20, 2024 at 07:33:46AM -0600, Simon Glass wrote: > > > > > Currently the world builds run on all runners, including faster and > > > slower ones. > > > > > > The difference can be quite dramatic, with some builders 4x as fast as > > > others, resulting in just one world build taking between 20 minutes and > > > an hour and 20 minutes. > > > > > > Add a tag so that we can select which builders run these CPU-intensive > > > jobs. > > > > > > With this tag we can also increase CPU utilisation by running multiple > > > QEMU tests in parallel. Currently these tests leave most machines fairly > > > idle, since we cannot run more than one world build on a machine. > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org> > > > > This conflicts I think with Jiaxun's desire to make our GitLab job > > runnable on the public runners too, and where we'll end up with 10 world > > build jobs ala Azure. > > It probably doesn't actually conflict, although I am not sure if one > can add a tag to jobs that run on public runners. I mean conceptually at least as it will likely be slower to build the world as 10 jobs than as 4 jobs.
Hi Tom, On Thu, 20 Jun 2024 at 17:15, Tom Rini <trini@konsulko.com> wrote: > > On Thu, Jun 20, 2024 at 05:05:26PM -0600, Simon Glass wrote: > > Hi Tom, > > > > On Thu, 20 Jun 2024 at 08:39, Tom Rini <trini@konsulko.com> wrote: > > > > > > On Thu, Jun 20, 2024 at 07:33:46AM -0600, Simon Glass wrote: > > > > > > > Currently the world builds run on all runners, including faster and > > > > slower ones. > > > > > > > > The difference can be quite dramatic, with some builders 4x as fast as > > > > others, resulting in just one world build taking between 20 minutes and > > > > an hour and 20 minutes. > > > > > > > > Add a tag so that we can select which builders run these CPU-intensive > > > > jobs. > > > > > > > > With this tag we can also increase CPU utilisation by running multiple > > > > QEMU tests in parallel. Currently these tests leave most machines fairly > > > > idle, since we cannot run more than one world build on a machine. > > > > > > > > Signed-off-by: Simon Glass <sjg@chromium.org> > > > > > > This conflicts I think with Jiaxun's desire to make our GitLab job > > > runnable on the public runners too, and where we'll end up with 10 world > > > build jobs ala Azure. > > > > It probably doesn't actually conflict, although I am not sure if one > > can add a tag to jobs that run on public runners. > > I mean conceptually at least as it will likely be slower to build the > world as 10 jobs than as 4 jobs. Yes I've noticed that the per-board build time reduces as the number of boards increases. For example moa varies between about 0.53 builds per second (for ~100 boards), to 0.59 (for ~400 or ~600). But there are 64 threads, so we need at least a few hundred to get to a steady state. Regards, Simon > > -- > Tom
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 165f765a833..750c4ff5f4d 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -2,6 +2,7 @@ variables: DEFAULT_TAG: "" + BUILD_TAG: "build" MIRROR_DOCKER: docker.io default: @@ -92,6 +93,8 @@ stages: .world_build: stage: world build + tags: + - ${BUILD_TAG} rules: - when: always
Currently the world builds run on all runners, including faster and slower ones. The difference can be quite dramatic, with some builders 4x as fast as others, resulting in just one world build taking between 20 minutes and an hour and 20 minutes. Add a tag so that we can select which builders run these CPU-intensive jobs. With this tag we can also increase CPU utilisation by running multiple QEMU tests in parallel. Currently these tests leave most machines fairly idle, since we cannot run more than one world build on a machine. Signed-off-by: Simon Glass <sjg@chromium.org> --- .gitlab-ci.yml | 3 +++ 1 file changed, 3 insertions(+)