| Message ID | 20230517135448.262483-2-berrange@redhat.com |
|---|---|
| State | New |
| Headers | show |
| Series | gitlab: improvements to handling of stable staging branches | expand |
17.05.2023 16:54, Daniel P. Berrangé wrote: > We use a fixed container tag of 'latest' so that contributors' forks > don't end up with an ever growing number of containers as they work > on throwaway feature branches. > > This fixed tag causes problems running CI upstream in stable staging > branches, however, because the stable staging branch will publish old > container content that clashes with that needed by primary staging > branch. This makes it impossible to reliably run CI pipelines in > parallel in upstream for different staging branches. > > This introduces $QEMU_CI_CONTAINER_TAG global variable as a way to > change which tag container publishing uses. Initially it can be set > by contributors as a git push option if they want to override the > default use of 'latest' eg > > git push gitlab <branch> -o ci.variable=QEMU_CONTAINER_TAG=fish > > this is useful if contributors need to run pipelines for different > branches concurrently in their forks. Reviewed-by: Michael Tokarev <mjt@tls.msk.ru>
On 5/17/23 06:54, Daniel P. Berrangé wrote: > We use a fixed container tag of 'latest' so that contributors' forks > don't end up with an ever growing number of containers as they work > on throwaway feature branches. > > This fixed tag causes problems running CI upstream in stable staging > branches, however, because the stable staging branch will publish old > container content that clashes with that needed by primary staging > branch. This makes it impossible to reliably run CI pipelines in > parallel in upstream for different staging branches. > > This introduces $QEMU_CI_CONTAINER_TAG global variable as a way to > change which tag container publishing uses. Initially it can be set > by contributors as a git push option if they want to override the > default use of 'latest' eg > > git push gitlab <branch> -o ci.variable=QEMU_CONTAINER_TAG=fish > > this is useful if contributors need to run pipelines for different > branches concurrently in their forks. > > Signed-off-by: Daniel P. Berrangé<berrange@redhat.com> > --- > .gitlab-ci.d/base.yml | 6 ++++++ > .gitlab-ci.d/buildtest-template.yml | 4 ++-- > .gitlab-ci.d/buildtest.yml | 4 ++-- > .gitlab-ci.d/container-template.yml | 3 ++- > .gitlab-ci.d/crossbuild-template.yml | 6 +++--- > .gitlab-ci.d/static_checks.yml | 4 ++-- > docs/devel/ci-jobs.rst.inc | 5 +++++ > 7 files changed, 22 insertions(+), 10 deletions(-) Reviewed-by: Richard Henderson <richard.henderson@linaro.org> r~
On 17/05/2023 15.54, Daniel P. Berrangé wrote: > We use a fixed container tag of 'latest' so that contributors' forks > don't end up with an ever growing number of containers as they work > on throwaway feature branches. > > This fixed tag causes problems running CI upstream in stable staging > branches, however, because the stable staging branch will publish old > container content that clashes with that needed by primary staging > branch. This makes it impossible to reliably run CI pipelines in > parallel in upstream for different staging branches. > > This introduces $QEMU_CI_CONTAINER_TAG global variable as a way to > change which tag container publishing uses. Initially it can be set > by contributors as a git push option if they want to override the > default use of 'latest' eg > > git push gitlab <branch> -o ci.variable=QEMU_CONTAINER_TAG=fish > > this is useful if contributors need to run pipelines for different > branches concurrently in their forks. > > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> > --- > .gitlab-ci.d/base.yml | 6 ++++++ > .gitlab-ci.d/buildtest-template.yml | 4 ++-- > .gitlab-ci.d/buildtest.yml | 4 ++-- > .gitlab-ci.d/container-template.yml | 3 ++- > .gitlab-ci.d/crossbuild-template.yml | 6 +++--- > .gitlab-ci.d/static_checks.yml | 4 ++-- > docs/devel/ci-jobs.rst.inc | 5 +++++ > 7 files changed, 22 insertions(+), 10 deletions(-) > > diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml > index 2fbb58d2a3..fba9d31cc6 100644 > --- a/.gitlab-ci.d/base.yml > +++ b/.gitlab-ci.d/base.yml > @@ -1,4 +1,10 @@ > > +variables: > + # On stable branches this needs changing. Should also be > + # overridden per pipeline if running pipelines concurrently > + # for different branches in contributor forks. > + QEMU_CI_CONTAINER_TAG: latest > + > # The order of rules defined here is critically important. > # They are evaluated in order and first match wins. > # > diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml > index a6cfe9be97..094ae485b8 100644 > --- a/.gitlab-ci.d/buildtest-template.yml > +++ b/.gitlab-ci.d/buildtest-template.yml > @@ -1,7 +1,7 @@ > .native_build_job_template: > extends: .base_job_template > stage: build > - image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest > + image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG > before_script: > - JOBS=$(expr $(nproc) + 1) > script: > @@ -44,7 +44,7 @@ > .common_test_job_template: > extends: .base_job_template > stage: test > - image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest > + image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG > script: > - scripts/git-submodule.sh update > $(sed -n '/GIT_SUBMODULES=/ s/.*=// p' build/config-host.mak) > diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml > index bb3650a51c..35867dfe77 100644 > --- a/.gitlab-ci.d/buildtest.yml > +++ b/.gitlab-ci.d/buildtest.yml > @@ -532,7 +532,7 @@ build-without-defaults: > build-libvhost-user: > extends: .base_job_template > stage: build > - image: $CI_REGISTRY_IMAGE/qemu/fedora:latest > + image: $CI_REGISTRY_IMAGE/qemu/fedora:$QEMU_CI_CONTAINER_TAG > needs: > job: amd64-fedora-container > script: > @@ -572,7 +572,7 @@ build-tools-and-docs-debian: > # of what topic branch they're currently using > pages: > extends: .base_job_template > - image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:latest > + image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:$QEMU_CI_CONTAINER_TAG > stage: test > needs: > - job: build-tools-and-docs-debian > diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml > index 519b8a9482..83c693c807 100644 > --- a/.gitlab-ci.d/container-template.yml > +++ b/.gitlab-ci.d/container-template.yml > @@ -5,7 +5,8 @@ > services: > - docker:dind > before_script: > - - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest" > + - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:$QEMU_CI_CONTAINER_TAG" > + # Always ':latest' because we always use upstream as a common cache source > - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest" > - apk add python3 > - docker info This patch no longer applies ... could you please rebase and resend a v2? Thanks! Thomas
On Fri, May 26, 2023 at 09:25:39AM +0200, Thomas Huth wrote: > On 17/05/2023 15.54, Daniel P. Berrangé wrote: > > We use a fixed container tag of 'latest' so that contributors' forks > > don't end up with an ever growing number of containers as they work > > on throwaway feature branches. > > > > This fixed tag causes problems running CI upstream in stable staging > > branches, however, because the stable staging branch will publish old > > container content that clashes with that needed by primary staging > > branch. This makes it impossible to reliably run CI pipelines in > > parallel in upstream for different staging branches. > > > > This introduces $QEMU_CI_CONTAINER_TAG global variable as a way to > > change which tag container publishing uses. Initially it can be set > > by contributors as a git push option if they want to override the > > default use of 'latest' eg > > > > git push gitlab <branch> -o ci.variable=QEMU_CONTAINER_TAG=fish > > > > this is useful if contributors need to run pipelines for different > > branches concurrently in their forks. > > This patch no longer applies ... could you please rebase and resend a v2? > Thanks! I've rebased and sent a v2, but I didn't get any conflits when rebasing, so v1 should have applied OK. With regards, Daniel
On 26/05/2023 12.20, Daniel P. Berrangé wrote: > On Fri, May 26, 2023 at 09:25:39AM +0200, Thomas Huth wrote: >> On 17/05/2023 15.54, Daniel P. Berrangé wrote: >>> We use a fixed container tag of 'latest' so that contributors' forks >>> don't end up with an ever growing number of containers as they work >>> on throwaway feature branches. >>> >>> This fixed tag causes problems running CI upstream in stable staging >>> branches, however, because the stable staging branch will publish old >>> container content that clashes with that needed by primary staging >>> branch. This makes it impossible to reliably run CI pipelines in >>> parallel in upstream for different staging branches. >>> >>> This introduces $QEMU_CI_CONTAINER_TAG global variable as a way to >>> change which tag container publishing uses. Initially it can be set >>> by contributors as a git push option if they want to override the >>> default use of 'latest' eg >>> >>> git push gitlab <branch> -o ci.variable=QEMU_CONTAINER_TAG=fish >>> >>> this is useful if contributors need to run pipelines for different >>> branches concurrently in their forks. > >> >> This patch no longer applies ... could you please rebase and resend a v2? >> Thanks! > > I've rebased and sent a v2, but I didn't get any conflits when > rebasing, so v1 should have applied OK. git-am is sometimes more picky than git-rebase ... I think there were some contextual conflicts with the patches from Camilla (commit 5f63a67adb5847 and b105ce60ca8bdee3) ... I should have maybe tried with "--3way" first before complaining, sorry. Thomas
On Fri, May 26, 2023 at 12:31:21PM +0200, Thomas Huth wrote: > On 26/05/2023 12.20, Daniel P. Berrangé wrote: > > On Fri, May 26, 2023 at 09:25:39AM +0200, Thomas Huth wrote: > > > On 17/05/2023 15.54, Daniel P. Berrangé wrote: > > > > We use a fixed container tag of 'latest' so that contributors' forks > > > > don't end up with an ever growing number of containers as they work > > > > on throwaway feature branches. > > > > > > > > This fixed tag causes problems running CI upstream in stable staging > > > > branches, however, because the stable staging branch will publish old > > > > container content that clashes with that needed by primary staging > > > > branch. This makes it impossible to reliably run CI pipelines in > > > > parallel in upstream for different staging branches. > > > > > > > > This introduces $QEMU_CI_CONTAINER_TAG global variable as a way to > > > > change which tag container publishing uses. Initially it can be set > > > > by contributors as a git push option if they want to override the > > > > default use of 'latest' eg > > > > > > > > git push gitlab <branch> -o ci.variable=QEMU_CONTAINER_TAG=fish > > > > > > > > this is useful if contributors need to run pipelines for different > > > > branches concurrently in their forks. > > > > > > > > This patch no longer applies ... could you please rebase and resend a v2? > > > Thanks! > > > > I've rebased and sent a v2, but I didn't get any conflits when > > rebasing, so v1 should have applied OK. > > git-am is sometimes more picky than git-rebase ... I think there were some > contextual conflicts with the patches from Camilla (commit 5f63a67adb5847 > and b105ce60ca8bdee3) ... I should have maybe tried with "--3way" first > before complaining, sorry. No worries, it is annoying that git am doesn't default to enabling -3, as that's basically always what you want :-( With regards, Daniel
diff --git a/.gitlab-ci.d/base.yml b/.gitlab-ci.d/base.yml index 2fbb58d2a3..fba9d31cc6 100644 --- a/.gitlab-ci.d/base.yml +++ b/.gitlab-ci.d/base.yml @@ -1,4 +1,10 @@ +variables: + # On stable branches this needs changing. Should also be + # overridden per pipeline if running pipelines concurrently + # for different branches in contributor forks. + QEMU_CI_CONTAINER_TAG: latest + # The order of rules defined here is critically important. # They are evaluated in order and first match wins. # diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml index a6cfe9be97..094ae485b8 100644 --- a/.gitlab-ci.d/buildtest-template.yml +++ b/.gitlab-ci.d/buildtest-template.yml @@ -1,7 +1,7 @@ .native_build_job_template: extends: .base_job_template stage: build - image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest + image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG before_script: - JOBS=$(expr $(nproc) + 1) script: @@ -44,7 +44,7 @@ .common_test_job_template: extends: .base_job_template stage: test - image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest + image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG script: - scripts/git-submodule.sh update $(sed -n '/GIT_SUBMODULES=/ s/.*=// p' build/config-host.mak) diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml index bb3650a51c..35867dfe77 100644 --- a/.gitlab-ci.d/buildtest.yml +++ b/.gitlab-ci.d/buildtest.yml @@ -532,7 +532,7 @@ build-without-defaults: build-libvhost-user: extends: .base_job_template stage: build - image: $CI_REGISTRY_IMAGE/qemu/fedora:latest + image: $CI_REGISTRY_IMAGE/qemu/fedora:$QEMU_CI_CONTAINER_TAG needs: job: amd64-fedora-container script: @@ -572,7 +572,7 @@ build-tools-and-docs-debian: # of what topic branch they're currently using pages: extends: .base_job_template - image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:latest + image: $CI_REGISTRY_IMAGE/qemu/debian-amd64:$QEMU_CI_CONTAINER_TAG stage: test needs: - job: build-tools-and-docs-debian diff --git a/.gitlab-ci.d/container-template.yml b/.gitlab-ci.d/container-template.yml index 519b8a9482..83c693c807 100644 --- a/.gitlab-ci.d/container-template.yml +++ b/.gitlab-ci.d/container-template.yml @@ -5,7 +5,8 @@ services: - docker:dind before_script: - - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:latest" + - export TAG="$CI_REGISTRY_IMAGE/qemu/$NAME:$QEMU_CI_CONTAINER_TAG" + # Always ':latest' because we always use upstream as a common cache source - export COMMON_TAG="$CI_REGISTRY/qemu-project/qemu/qemu/$NAME:latest" - apk add python3 - docker info diff --git a/.gitlab-ci.d/crossbuild-template.yml b/.gitlab-ci.d/crossbuild-template.yml index 4f93b9e4e5..6efb0d2a54 100644 --- a/.gitlab-ci.d/crossbuild-template.yml +++ b/.gitlab-ci.d/crossbuild-template.yml @@ -1,7 +1,7 @@ .cross_system_build_job: extends: .base_job_template stage: build - image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest + image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG timeout: 80m script: - mkdir build @@ -27,7 +27,7 @@ .cross_accel_build_job: extends: .base_job_template stage: build - image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest + image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG timeout: 30m script: - mkdir build @@ -39,7 +39,7 @@ .cross_user_build_job: extends: .base_job_template stage: build - image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:latest + image: $CI_REGISTRY_IMAGE/qemu/$IMAGE:$QEMU_CI_CONTAINER_TAG script: - mkdir build - cd build diff --git a/.gitlab-ci.d/static_checks.yml b/.gitlab-ci.d/static_checks.yml index b4cbdbce2a..ad9f426a52 100644 --- a/.gitlab-ci.d/static_checks.yml +++ b/.gitlab-ci.d/static_checks.yml @@ -26,7 +26,7 @@ check-dco: check-python-minreqs: extends: .base_job_template stage: test - image: $CI_REGISTRY_IMAGE/qemu/python:latest + image: $CI_REGISTRY_IMAGE/qemu/python:$QEMU_CI_CONTAINER_TAG script: - make -C python check-minreqs variables: @@ -37,7 +37,7 @@ check-python-minreqs: check-python-tox: extends: .base_job_template stage: test - image: $CI_REGISTRY_IMAGE/qemu/python:latest + image: $CI_REGISTRY_IMAGE/qemu/python:$QEMU_CI_CONTAINER_TAG script: - make -C python check-tox variables: diff --git a/docs/devel/ci-jobs.rst.inc b/docs/devel/ci-jobs.rst.inc index 1f28fec0d0..f72537853b 100644 --- a/docs/devel/ci-jobs.rst.inc +++ b/docs/devel/ci-jobs.rst.inc @@ -70,6 +70,11 @@ in a handful of namespaces repository CI settings, or as git push variables, to influence which jobs get run in a pipeline + * QEMU_CI_CONTAINER_TAG - the tag used to publish containers + in stage 1, for use by build jobs in stage 2. Defaults to + 'latest', but if running pipelines for different branches + concurrently, it should be overridden per pipeline. + * nnn - other misc variables not falling into the above categories, or using different names for historical reasons and not yet converted.
We use a fixed container tag of 'latest' so that contributors' forks don't end up with an ever growing number of containers as they work on throwaway feature branches. This fixed tag causes problems running CI upstream in stable staging branches, however, because the stable staging branch will publish old container content that clashes with that needed by primary staging branch. This makes it impossible to reliably run CI pipelines in parallel in upstream for different staging branches. This introduces $QEMU_CI_CONTAINER_TAG global variable as a way to change which tag container publishing uses. Initially it can be set by contributors as a git push option if they want to override the default use of 'latest' eg git push gitlab <branch> -o ci.variable=QEMU_CONTAINER_TAG=fish this is useful if contributors need to run pipelines for different branches concurrently in their forks. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> --- .gitlab-ci.d/base.yml | 6 ++++++ .gitlab-ci.d/buildtest-template.yml | 4 ++-- .gitlab-ci.d/buildtest.yml | 4 ++-- .gitlab-ci.d/container-template.yml | 3 ++- .gitlab-ci.d/crossbuild-template.yml | 6 +++--- .gitlab-ci.d/static_checks.yml | 4 ++-- docs/devel/ci-jobs.rst.inc | 5 +++++ 7 files changed, 22 insertions(+), 10 deletions(-)