diff mbox series

[3/9] GitLab CI: Make wider use of tags

Message ID 20241208190741.4192601-4-trini@konsulko.com
State Accepted
Commit 77026080054702dbb40f8f04815620f15c7ac593
Delegated to: Tom Rini
Headers show
Series Enable arm64 host support in Gitlab | expand

Commit Message

Tom Rini Dec. 8, 2024, 5:07 p.m. UTC
First, introduce DEFAULT_ALL_TAG, DEFAULT_ARM64_TAG, DEFAULT_AMD64_TAG
and DEFAULT_FAST_AMD64_TAG and remove the previous DEFAULT_TAG (as
anyone making use of that will need to adjust their jobs). This allows
us to say that some jobs can run on amd64 or arm64 hosts under the all
tag, while some jobs must run on amd64 (the Xtensa jobs due to
binary-only toolchains and sandbox for now) Then we rework the world
build stage to only run on our very fast amd64 hosts, or our arm64 hosts
(which are also very fast). This should result in a similar overall
build time but also a much more consistent overall build time as we
won't have the two big world jobs possibly run on our slower build
nodes.

Signed-off-by: Tom Rini <trini@konsulko.com>
---
 .gitlab-ci.yml | 74 ++++++++++++++++++--------------------------------
 1 file changed, 27 insertions(+), 47 deletions(-)

Comments

Simon Glass Dec. 9, 2024, 3 p.m. UTC | #1
Hi Tom,

On Sun, 8 Dec 2024 at 12:08, Tom Rini <trini@konsulko.com> wrote:
>
> First, introduce DEFAULT_ALL_TAG, DEFAULT_ARM64_TAG, DEFAULT_AMD64_TAG
> and DEFAULT_FAST_AMD64_TAG and remove the previous DEFAULT_TAG (as
> anyone making use of that will need to adjust their jobs). This allows
> us to say that some jobs can run on amd64 or arm64 hosts under the all
> tag, while some jobs must run on amd64 (the Xtensa jobs due to
> binary-only toolchains and sandbox for now) Then we rework the world
> build stage to only run on our very fast amd64 hosts, or our arm64 hosts
> (which are also very fast). This should result in a similar overall
> build time but also a much more consistent overall build time as we
> won't have the two big world jobs possibly run on our slower build
> nodes.
>
> Signed-off-by: Tom Rini <trini@konsulko.com>
> ---
>  .gitlab-ci.yml | 74 ++++++++++++++++++--------------------------------
>  1 file changed, 27 insertions(+), 47 deletions(-)

I'd prefer fast_amd64 to avoid spaces

What is the run time for this new, unified 'world' build? I just
wonder whether splitting it might make the runs go faster?

Regards,
SImon
Tom Rini Dec. 9, 2024, 3:51 p.m. UTC | #2
On Mon, Dec 09, 2024 at 08:00:42AM -0700, Simon Glass wrote:
> Hi Tom,
> 
> On Sun, 8 Dec 2024 at 12:08, Tom Rini <trini@konsulko.com> wrote:
> >
> > First, introduce DEFAULT_ALL_TAG, DEFAULT_ARM64_TAG, DEFAULT_AMD64_TAG
> > and DEFAULT_FAST_AMD64_TAG and remove the previous DEFAULT_TAG (as
> > anyone making use of that will need to adjust their jobs). This allows
> > us to say that some jobs can run on amd64 or arm64 hosts under the all
> > tag, while some jobs must run on amd64 (the Xtensa jobs due to
> > binary-only toolchains and sandbox for now) Then we rework the world
> > build stage to only run on our very fast amd64 hosts, or our arm64 hosts
> > (which are also very fast). This should result in a similar overall
> > build time but also a much more consistent overall build time as we
> > won't have the two big world jobs possibly run on our slower build
> > nodes.
> >
> > Signed-off-by: Tom Rini <trini@konsulko.com>
> > ---
> >  .gitlab-ci.yml | 74 ++++++++++++++++++--------------------------------
> >  1 file changed, 27 insertions(+), 47 deletions(-)
> 
> I'd prefer fast_amd64 to avoid spaces

I don't see a reason to avoid spaces here.

> What is the run time for this new, unified 'world' build? I just
> wonder whether splitting it might make the runs go faster?

It's about the same at the worst case of ~40-45min, but it's more
consistent now, by excluding the older / slower amd64 hosts.
diff mbox series

Patch

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 2671c3bb1061..7912eeee4401 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,14 +1,17 @@ 
 # SPDX-License-Identifier: GPL-2.0+
 
 variables:
-  DEFAULT_TAG: ""
+  DEFAULT_ALL_TAG: "all"
+  DEFAULT_ARM64_TAG: "arm64"
+  DEFAULT_AMD64_TAG: "amd64"
+  DEFAULT_FAST_AMD64_TAG: "fast amd64"
   MIRROR_DOCKER: docker.io
   SJG_LAB: ""
   PLATFORM: linux/amd64,linux/arm64
 
 default:
   tags:
-    - ${DEFAULT_TAG}
+    - ${DEFAULT_ALL_TAG}
 
 # Grab our configured image.  The source for this is found
 # in the u-boot tree at tools/docker/Dockerfile
@@ -102,56 +105,21 @@  stages:
       junit: results.xml
     expire_in: 1 week
 
-.world_build:
+build all platforms in a single job:
   stage: world build
   rules:
     - when: always
-
-build all 32bit ARM platforms:
-  extends: .world_build
-  script:
-    - ret=0;
-      git config --global --add safe.directory "${CI_PROJECT_DIR}";
-      pip install -r tools/buildman/requirements.txt;
-      ./tools/buildman/buildman -o /tmp -PEWM arm -x aarch64 || ret=$?;
-      if [[ $ret -ne 0 ]]; then
-        ./tools/buildman/buildman -o /tmp -seP;
-        exit $ret;
-      fi;
-
-build all 64bit ARM platforms:
-  extends: .world_build
+  parallel:
+    matrix:
+      - HOST: "arm64"
+      - HOST: "fast amd64"
+  tags:
+    - ${HOST}
   script:
-    - virtualenv -p /usr/bin/python3 /tmp/venv
-    - . /tmp/venv/bin/activate
     - ret=0;
       git config --global --add safe.directory "${CI_PROJECT_DIR}";
       pip install -r tools/buildman/requirements.txt;
-      ./tools/buildman/buildman -o /tmp -PEWM aarch64 || ret=$?;
-      if [[ $ret -ne 0 ]]; then
-        ./tools/buildman/buildman -o /tmp -seP;
-        exit $ret;
-      fi;
-
-build all PowerPC platforms:
-  extends: .world_build
-  script:
-    - ret=0;
-      git config --global --add safe.directory "${CI_PROJECT_DIR}";
-      ./tools/buildman/buildman -o /tmp -P -E -W powerpc || ret=$?;
-      if [[ $ret -ne 0 ]]; then
-        ./tools/buildman/buildman -o /tmp -seP;
-        exit $ret;
-      fi;
-
-# We exclude xtensa here due to not being able to build on aarch64
-# hosts but covering all platforms in the pytest section.
-build all other platforms:
-  extends: .world_build
-  script:
-    - ret=0;
-      git config --global --add safe.directory "${CI_PROJECT_DIR}";
-      ./tools/buildman/buildman -o /tmp -PEWM -x arm,powerpc,xtensa || ret=$?;
+      ./tools/buildman/buildman -o /tmp -PEWM -x xtensa || ret=$?;
       if [[ $ret -ne 0 ]]; then
         ./tools/buildman/buildman -o /tmp -seP;
         exit $ret;
@@ -200,6 +168,8 @@  Build tools-only and envtools:
 
 Run binman, buildman, dtoc, Kconfig and patman testsuites:
   extends: .testsuites
+  tags:
+    - ${DEFAULT_AMD64_TAG}
   script:
     - git config --global user.name "GitLab CI Runner";
       git config --global user.email trini@konsulko.com;
@@ -259,22 +229,30 @@  Check packing of Python tools:
 
 # Test sandbox with test.py
 sandbox test.py:
+  tags:
+    - ${DEFAULT_AMD64_TAG}
   variables:
     TEST_PY_BD: "sandbox"
   <<: *buildman_and_testpy_dfn
 
 sandbox with clang test.py:
+  tags:
+    - ${DEFAULT_AMD64_TAG}
   variables:
     TEST_PY_BD: "sandbox"
     OVERRIDE: "-O clang-17"
   <<: *buildman_and_testpy_dfn
 
 sandbox64 test.py:
+  tags:
+    - ${DEFAULT_AMD64_TAG}
   variables:
     TEST_PY_BD: "sandbox64"
   <<: *buildman_and_testpy_dfn
 
 sandbox64 with clang test.py:
+  tags:
+    - ${DEFAULT_AMD64_TAG}
   variables:
     TEST_PY_BD: "sandbox64"
     OVERRIDE: "-O clang-17"
@@ -329,6 +307,8 @@  evb-ast2600 test.py:
   <<: *buildman_and_testpy_dfn
 
 sandbox_flattree test.py:
+  tags:
+    - ${DEFAULT_AMD64_TAG}
   variables:
     TEST_PY_BD: "sandbox_flattree"
   <<: *buildman_and_testpy_dfn
@@ -454,7 +434,7 @@  qemu-xtensa-dc233c test.py:
     TEST_PY_BD: "qemu-xtensa-dc233c"
     TEST_PY_TEST_SPEC: "not sleep and not efi"
   tags:
-    - all
+    - ${DEFAULT_AMD64_TAG}
   <<: *buildman_and_testpy_dfn
 
 r2dplus_i82557c test.py:
@@ -519,7 +499,7 @@  xtfpga test.py:
     TEST_PY_TEST_SPEC: "not sleep"
     TEST_PY_ID: "--id qemu"
   tags:
-    - all
+    - ${DEFAULT_AMD64_TAG}
   <<: *buildman_and_testpy_dfn
 
 coreboot test.py: