| Message ID | 20231122132847.118673-3-amusil@redhat.com |
|---|---|
| State | Superseded |
| Headers | show |
| Series | Allow to use different container images per branch | expand |
| Context | Check | Description |
|---|---|---|
| ovsrobot/apply-robot | success | apply and check: success |
| ovsrobot/github-robot-_Build_and_Test | success | github build: passed |
| ovsrobot/github-robot-_ovn-kubernetes | success | github build: passed |
On Wed, Nov 22, 2023 at 2:28 PM Ales Musil <amusil@redhat.com> wrote: > Build the image before every job to allow more > fine-grained dependency pinning. This is especially > useful for stable branches as they might need to stay > on specific distribution or Python version. > > Signed-off-by: Ales Musil <amusil@redhat.com> > --- > .cirrus.yml | 40 ++++++++++++++++++++++++++++----- > .github/workflows/test.yml | 46 +++++++++++++++++++++++++++++++++++--- > 2 files changed, 77 insertions(+), 9 deletions(-) > > diff --git a/.cirrus.yml b/.cirrus.yml > index 71e9afbb6..894f953f3 100644 > --- a/.cirrus.yml > +++ b/.cirrus.yml > @@ -1,9 +1,25 @@ > -arm_unit_tests_task: > +compute_engine_instance: > + image_project: ubuntu-os-cloud > + image: family/ubuntu-2304-amd64 > there is still one issues, it should be: image: family/ubuntu-2304-arm64 architecture: arm64 I'll wait with v3 for some reviews/next week. > + platform: linux > + memory: 4G > + > +build_image_task: > + install_dependencies_script: > + - sudo apt update > + - sudo apt install -y podman make > + > + build_container_script: > + - cd utilities/containers > + - make ubuntu > + - podman save -o /tmp/image.tar ovn-org/ovn-tests:ubuntu > + > + upload_image_script: > + - curl -s -X POST -T /tmp/image.tar http:// > $CIRRUS_HTTP_CACHE_HOST/${CIRRUS_CHANGE_IN_REPO} > > - arm_container: > - image: ghcr.io/ovn-org/ovn-tests:ubuntu > - memory: 4G > - cpu: 2 > +arm_unit_tests_task: > + depends_on: > + - build_image > > env: > ARCH: aarch64 > @@ -11,6 +27,7 @@ arm_unit_tests_task: > PATH: ${HOME}/bin:${HOME}/.local/bin:${PATH} > RECHECK: yes > LANG: C > + IMAGE_NAME: ovn-org/ovn-tests:ubuntu > matrix: > - CC: gcc > TESTSUITE: test > @@ -35,5 +52,16 @@ arm_unit_tests_task: > > name: ARM64 ${CC} ${TESTSUITE} ${TEST_RANGE} > > + install_dependencies_script: > + - sudo apt update > + - sudo apt install -y podman > + > + download_cache_script: > + - curl http://$CIRRUS_HTTP_CACHE_HOST/${CIRRUS_CHANGE_IN_REPO} -o > /tmp/image.tar > + > + load_image_script: > + - podman load -i /tmp/image.tar > + - rm -rf /tmp/image.tar > + > build_script: > - - ./.ci/linux-build.sh > + - ./.ci/ci.sh --archive-logs > diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml > index b3fcb57fc..fdbc8f5f5 100644 > --- a/.github/workflows/test.yml > +++ b/.github/workflows/test.yml > @@ -80,10 +80,38 @@ jobs: > if: steps.dpdk_cache.outputs.cache-hit != 'true' > run: ./.ci/dpdk-build.sh > > + prepare-container: > + env: > + DEPENDENCIES: podman > + name: Prepare container > + runs-on: ubuntu-22.04 > + > + steps: > + - uses: actions/checkout@v3 > + > + - name: Update APT cache > + run: sudo apt update > + > + - name: Install dependencies > + run: sudo apt install -y ${{ env.DEPENDENCIES }} > + > + - name: Build container > + run: make ubuntu > + working-directory: utilities/containers > + > + - name: Export image > + run: podman save -o /tmp/image.tar ovn-org/ovn-tests > + > + - name: Cache image > + id: image_cache > + uses: actions/cache@v3 > + with: > + path: /tmp/image.tar > + key: ${{ github.sha }} > + > build-linux: > - needs: build-dpdk > + needs: [build-dpdk, prepare-container] > env: > - IMAGE_NAME: ghcr.io/ovn-org/ovn-tests:ubuntu > ARCH: ${{ matrix.cfg.arch }} > CC: ${{ matrix.cfg.compiler }} > DPDK: ${{ matrix.cfg.dpdk }} > @@ -152,13 +180,25 @@ jobs: > sort -V | tail -1) > working-directory: ovs > > - - name: cache > + - name: cache dpdk > if: matrix.cfg.dpdk != '' > uses: actions/cache@v3 > with: > path: dpdk-dir > key: ${{ needs.build-dpdk.outputs.dpdk_key }} > > + - name: image cache > + uses: actions/cache@v3 > + with: > + path: /tmp/image.tar > + key: ${{ github.sha }} > + > + - name: load image > + run: | > + sudo podman load -i /tmp/image.tar > + podman load -i /tmp/image.tar > + rm -rf /tmp/image.tar > + > - name: build > if: ${{ startsWith(matrix.cfg.testsuite, 'system-test') }} > run: sudo -E ./.ci/ci.sh --archive-logs > -- > 2.41.0 > >
On 11/22/23 14:28, Ales Musil wrote: > Build the image before every job to allow more > fine-grained dependency pinning. This is especially > useful for stable branches as they might need to stay > on specific distribution or Python version. > > Signed-off-by: Ales Musil <amusil@redhat.com> > --- > .cirrus.yml | 40 ++++++++++++++++++++++++++++----- > .github/workflows/test.yml | 46 +++++++++++++++++++++++++++++++++++--- > 2 files changed, 77 insertions(+), 9 deletions(-) > > diff --git a/.cirrus.yml b/.cirrus.yml > index 71e9afbb6..894f953f3 100644 > --- a/.cirrus.yml > +++ b/.cirrus.yml > @@ -1,9 +1,25 @@ > -arm_unit_tests_task: > +compute_engine_instance: > + image_project: ubuntu-os-cloud > + image: family/ubuntu-2304-amd64 > + platform: linux > + memory: 4G > + > +build_image_task: > + install_dependencies_script: > + - sudo apt update > + - sudo apt install -y podman make > + > + build_container_script: > + - cd utilities/containers > + - make ubuntu > + - podman save -o /tmp/image.tar ovn-org/ovn-tests:ubuntu > + > + upload_image_script: > + - curl -s -X POST -T /tmp/image.tar http://$CIRRUS_HTTP_CACHE_HOST/${CIRRUS_CHANGE_IN_REPO} This seemed a bit strange initially but I understand now (thanks for the offline clarification) that it's because we want to avoid building for each instance of the matrix in the unit test task. I think a comment explaining that here would be good as others might also wonder why we need the cache. Thanks, Dumitru > > - arm_container: > - image: ghcr.io/ovn-org/ovn-tests:ubuntu > - memory: 4G > - cpu: 2 > +arm_unit_tests_task: > + depends_on: > + - build_image > > env: > ARCH: aarch64 > @@ -11,6 +27,7 @@ arm_unit_tests_task: > PATH: ${HOME}/bin:${HOME}/.local/bin:${PATH} > RECHECK: yes > LANG: C > + IMAGE_NAME: ovn-org/ovn-tests:ubuntu > matrix: > - CC: gcc > TESTSUITE: test > @@ -35,5 +52,16 @@ arm_unit_tests_task: > > name: ARM64 ${CC} ${TESTSUITE} ${TEST_RANGE} > > + install_dependencies_script: > + - sudo apt update > + - sudo apt install -y podman > + > + download_cache_script: > + - curl http://$CIRRUS_HTTP_CACHE_HOST/${CIRRUS_CHANGE_IN_REPO} -o /tmp/image.tar > + > + load_image_script: > + - podman load -i /tmp/image.tar > + - rm -rf /tmp/image.tar > + > build_script: > - - ./.ci/linux-build.sh > + - ./.ci/ci.sh --archive-logs > diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml > index b3fcb57fc..fdbc8f5f5 100644 > --- a/.github/workflows/test.yml > +++ b/.github/workflows/test.yml > @@ -80,10 +80,38 @@ jobs: > if: steps.dpdk_cache.outputs.cache-hit != 'true' > run: ./.ci/dpdk-build.sh > > + prepare-container: > + env: > + DEPENDENCIES: podman > + name: Prepare container > + runs-on: ubuntu-22.04 > + > + steps: > + - uses: actions/checkout@v3 > + > + - name: Update APT cache > + run: sudo apt update > + > + - name: Install dependencies > + run: sudo apt install -y ${{ env.DEPENDENCIES }} > + > + - name: Build container > + run: make ubuntu > + working-directory: utilities/containers > + > + - name: Export image > + run: podman save -o /tmp/image.tar ovn-org/ovn-tests > + > + - name: Cache image > + id: image_cache > + uses: actions/cache@v3 > + with: > + path: /tmp/image.tar > + key: ${{ github.sha }} > + > build-linux: > - needs: build-dpdk > + needs: [build-dpdk, prepare-container] > env: > - IMAGE_NAME: ghcr.io/ovn-org/ovn-tests:ubuntu > ARCH: ${{ matrix.cfg.arch }} > CC: ${{ matrix.cfg.compiler }} > DPDK: ${{ matrix.cfg.dpdk }} > @@ -152,13 +180,25 @@ jobs: > sort -V | tail -1) > working-directory: ovs > > - - name: cache > + - name: cache dpdk > if: matrix.cfg.dpdk != '' > uses: actions/cache@v3 > with: > path: dpdk-dir > key: ${{ needs.build-dpdk.outputs.dpdk_key }} > > + - name: image cache > + uses: actions/cache@v3 > + with: > + path: /tmp/image.tar > + key: ${{ github.sha }} > + > + - name: load image > + run: | > + sudo podman load -i /tmp/image.tar > + podman load -i /tmp/image.tar > + rm -rf /tmp/image.tar > + > - name: build > if: ${{ startsWith(matrix.cfg.testsuite, 'system-test') }} > run: sudo -E ./.ci/ci.sh --archive-logs
On Thu, Nov 23, 2023 at 11:13 AM Dumitru Ceara <dceara@redhat.com> wrote: > On 11/22/23 14:28, Ales Musil wrote: > > Build the image before every job to allow more > > fine-grained dependency pinning. This is especially > > useful for stable branches as they might need to stay > > on specific distribution or Python version. > > > > Signed-off-by: Ales Musil <amusil@redhat.com> > > --- > > .cirrus.yml | 40 ++++++++++++++++++++++++++++----- > > .github/workflows/test.yml | 46 +++++++++++++++++++++++++++++++++++--- > > 2 files changed, 77 insertions(+), 9 deletions(-) > > > > diff --git a/.cirrus.yml b/.cirrus.yml > > index 71e9afbb6..894f953f3 100644 > > --- a/.cirrus.yml > > +++ b/.cirrus.yml > > @@ -1,9 +1,25 @@ > > -arm_unit_tests_task: > > +compute_engine_instance: > > + image_project: ubuntu-os-cloud > > + image: family/ubuntu-2304-amd64 > > + platform: linux > > + memory: 4G > > + > > +build_image_task: > > + install_dependencies_script: > > + - sudo apt update > > + - sudo apt install -y podman make > > + > > + build_container_script: > > + - cd utilities/containers > > + - make ubuntu > > + - podman save -o /tmp/image.tar ovn-org/ovn-tests:ubuntu > > + > > + upload_image_script: > > + - curl -s -X POST -T /tmp/image.tar http:// > $CIRRUS_HTTP_CACHE_HOST/${CIRRUS_CHANGE_IN_REPO} > > This seemed a bit strange initially but I understand now (thanks for the > offline clarification) that it's because we want to avoid building for > each instance of the matrix in the unit test task. > > I think a comment explaining that here would be good as others might > also wonder why we need the cache. > Thank you for the review. I have added a comment in v3. > > Thanks, > Dumitru > > > > > - arm_container: > > - image: ghcr.io/ovn-org/ovn-tests:ubuntu > > - memory: 4G > > - cpu: 2 > > +arm_unit_tests_task: > > + depends_on: > > + - build_image > > > > env: > > ARCH: aarch64 > > @@ -11,6 +27,7 @@ arm_unit_tests_task: > > PATH: ${HOME}/bin:${HOME}/.local/bin:${PATH} > > RECHECK: yes > > LANG: C > > + IMAGE_NAME: ovn-org/ovn-tests:ubuntu > > matrix: > > - CC: gcc > > TESTSUITE: test > > @@ -35,5 +52,16 @@ arm_unit_tests_task: > > > > name: ARM64 ${CC} ${TESTSUITE} ${TEST_RANGE} > > > > + install_dependencies_script: > > + - sudo apt update > > + - sudo apt install -y podman > > + > > + download_cache_script: > > + - curl http://$CIRRUS_HTTP_CACHE_HOST/${CIRRUS_CHANGE_IN_REPO} -o > /tmp/image.tar > > + > > + load_image_script: > > + - podman load -i /tmp/image.tar > > + - rm -rf /tmp/image.tar > > + > > build_script: > > - - ./.ci/linux-build.sh > > + - ./.ci/ci.sh --archive-logs > > diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml > > index b3fcb57fc..fdbc8f5f5 100644 > > --- a/.github/workflows/test.yml > > +++ b/.github/workflows/test.yml > > @@ -80,10 +80,38 @@ jobs: > > if: steps.dpdk_cache.outputs.cache-hit != 'true' > > run: ./.ci/dpdk-build.sh > > > > + prepare-container: > > + env: > > + DEPENDENCIES: podman > > + name: Prepare container > > + runs-on: ubuntu-22.04 > > + > > + steps: > > + - uses: actions/checkout@v3 > > + > > + - name: Update APT cache > > + run: sudo apt update > > + > > + - name: Install dependencies > > + run: sudo apt install -y ${{ env.DEPENDENCIES }} > > + > > + - name: Build container > > + run: make ubuntu > > + working-directory: utilities/containers > > + > > + - name: Export image > > + run: podman save -o /tmp/image.tar ovn-org/ovn-tests > > + > > + - name: Cache image > > + id: image_cache > > + uses: actions/cache@v3 > > + with: > > + path: /tmp/image.tar > > + key: ${{ github.sha }} > > + > > build-linux: > > - needs: build-dpdk > > + needs: [build-dpdk, prepare-container] > > env: > > - IMAGE_NAME: ghcr.io/ovn-org/ovn-tests:ubuntu > > ARCH: ${{ matrix.cfg.arch }} > > CC: ${{ matrix.cfg.compiler }} > > DPDK: ${{ matrix.cfg.dpdk }} > > @@ -152,13 +180,25 @@ jobs: > > sort -V | tail -1) > > working-directory: ovs > > > > - - name: cache > > + - name: cache dpdk > > if: matrix.cfg.dpdk != '' > > uses: actions/cache@v3 > > with: > > path: dpdk-dir > > key: ${{ needs.build-dpdk.outputs.dpdk_key }} > > > > + - name: image cache > > + uses: actions/cache@v3 > > + with: > > + path: /tmp/image.tar > > + key: ${{ github.sha }} > > + > > + - name: load image > > + run: | > > + sudo podman load -i /tmp/image.tar > > + podman load -i /tmp/image.tar > > + rm -rf /tmp/image.tar > > + > > - name: build > > if: ${{ startsWith(matrix.cfg.testsuite, 'system-test') }} > > run: sudo -E ./.ci/ci.sh --archive-log Thanks, Ales
diff --git a/.cirrus.yml b/.cirrus.yml index 71e9afbb6..894f953f3 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -1,9 +1,25 @@ -arm_unit_tests_task: +compute_engine_instance: + image_project: ubuntu-os-cloud + image: family/ubuntu-2304-amd64 + platform: linux + memory: 4G + +build_image_task: + install_dependencies_script: + - sudo apt update + - sudo apt install -y podman make + + build_container_script: + - cd utilities/containers + - make ubuntu + - podman save -o /tmp/image.tar ovn-org/ovn-tests:ubuntu + + upload_image_script: + - curl -s -X POST -T /tmp/image.tar http://$CIRRUS_HTTP_CACHE_HOST/${CIRRUS_CHANGE_IN_REPO} - arm_container: - image: ghcr.io/ovn-org/ovn-tests:ubuntu - memory: 4G - cpu: 2 +arm_unit_tests_task: + depends_on: + - build_image env: ARCH: aarch64 @@ -11,6 +27,7 @@ arm_unit_tests_task: PATH: ${HOME}/bin:${HOME}/.local/bin:${PATH} RECHECK: yes LANG: C + IMAGE_NAME: ovn-org/ovn-tests:ubuntu matrix: - CC: gcc TESTSUITE: test @@ -35,5 +52,16 @@ arm_unit_tests_task: name: ARM64 ${CC} ${TESTSUITE} ${TEST_RANGE} + install_dependencies_script: + - sudo apt update + - sudo apt install -y podman + + download_cache_script: + - curl http://$CIRRUS_HTTP_CACHE_HOST/${CIRRUS_CHANGE_IN_REPO} -o /tmp/image.tar + + load_image_script: + - podman load -i /tmp/image.tar + - rm -rf /tmp/image.tar + build_script: - - ./.ci/linux-build.sh + - ./.ci/ci.sh --archive-logs diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index b3fcb57fc..fdbc8f5f5 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -80,10 +80,38 @@ jobs: if: steps.dpdk_cache.outputs.cache-hit != 'true' run: ./.ci/dpdk-build.sh + prepare-container: + env: + DEPENDENCIES: podman + name: Prepare container + runs-on: ubuntu-22.04 + + steps: + - uses: actions/checkout@v3 + + - name: Update APT cache + run: sudo apt update + + - name: Install dependencies + run: sudo apt install -y ${{ env.DEPENDENCIES }} + + - name: Build container + run: make ubuntu + working-directory: utilities/containers + + - name: Export image + run: podman save -o /tmp/image.tar ovn-org/ovn-tests + + - name: Cache image + id: image_cache + uses: actions/cache@v3 + with: + path: /tmp/image.tar + key: ${{ github.sha }} + build-linux: - needs: build-dpdk + needs: [build-dpdk, prepare-container] env: - IMAGE_NAME: ghcr.io/ovn-org/ovn-tests:ubuntu ARCH: ${{ matrix.cfg.arch }} CC: ${{ matrix.cfg.compiler }} DPDK: ${{ matrix.cfg.dpdk }} @@ -152,13 +180,25 @@ jobs: sort -V | tail -1) working-directory: ovs - - name: cache + - name: cache dpdk if: matrix.cfg.dpdk != '' uses: actions/cache@v3 with: path: dpdk-dir key: ${{ needs.build-dpdk.outputs.dpdk_key }} + - name: image cache + uses: actions/cache@v3 + with: + path: /tmp/image.tar + key: ${{ github.sha }} + + - name: load image + run: | + sudo podman load -i /tmp/image.tar + podman load -i /tmp/image.tar + rm -rf /tmp/image.tar + - name: build if: ${{ startsWith(matrix.cfg.testsuite, 'system-test') }} run: sudo -E ./.ci/ci.sh --archive-logs
Build the image before every job to allow more fine-grained dependency pinning. This is especially useful for stable branches as they might need to stay on specific distribution or Python version. Signed-off-by: Ales Musil <amusil@redhat.com> --- .cirrus.yml | 40 ++++++++++++++++++++++++++++----- .github/workflows/test.yml | 46 +++++++++++++++++++++++++++++++++++--- 2 files changed, 77 insertions(+), 9 deletions(-)