From patchwork Mon Jun 24 14:33:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ales Musil X-Patchwork-Id: 1951598 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=PEmOGoUF; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4W79St2wxqz20X6 for ; Tue, 25 Jun 2024 00:34:06 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 0457B608F9; Mon, 24 Jun 2024 14:34:04 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id spz2lQVDIrAh; Mon, 24 Jun 2024 14:34:03 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.9.56; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org D8146608D8 Authentication-Results: smtp3.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=PEmOGoUF Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id D8146608D8; Mon, 24 Jun 2024 14:34:02 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 65B83C0012; Mon, 24 Jun 2024 14:34:02 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id F2BADC0011 for ; Mon, 24 Jun 2024 14:34:00 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id E09CF80DE3 for ; Mon, 24 Jun 2024 14:34:00 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id pGliMt-BYE-Q for ; Mon, 24 Jun 2024 14:34:00 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=170.10.133.124; helo=us-smtp-delivery-124.mimecast.com; envelope-from=amusil@redhat.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp1.osuosl.org D477080D84 Authentication-Results: smtp1.osuosl.org; dmarc=pass (p=none dis=none) header.from=redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org D477080D84 Authentication-Results: smtp1.osuosl.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=PEmOGoUF Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id D477080D84 for ; Mon, 24 Jun 2024 14:33:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719239638; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=MhvwhjecCkzxA5LhTxIQE1j3J5FRkNzBUWSNab/SooY=; b=PEmOGoUFSD5sIS1irkp7gVVlX3ZPSDGnhNCR5CXIaqmzadccAvIc1HE83JviBcqpfapdwK belVdkLhnwGg7LJ/bvXAepNHTUt9C2QkZ5GzRD+s/lm2NJY5/5+m6BXyK+Cv84qMppO6DB 53ongTDcgoNsyAFucLQB/NHLuScstak= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-332-VX0KNIskOZOuEtXVqcD79w-1; Mon, 24 Jun 2024 10:33:57 -0400 X-MC-Unique: VX0KNIskOZOuEtXVqcD79w-1 Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 1FFDB19560B2 for ; Mon, 24 Jun 2024 14:33:56 +0000 (UTC) Received: from amusil.redhat.com (unknown [10.45.224.204]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 2D3311956048; Mon, 24 Jun 2024 14:33:53 +0000 (UTC) From: Ales Musil To: dev@openvswitch.org Date: Mon, 24 Jun 2024 16:33:51 +0200 Message-ID: <20240624143352.370620-1-amusil@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH ovn v2 1/2] ci: Move common build steps into script. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: dceara@redhat.com Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Move common preparation steps into script that can be invoked by both container builds. This will ensure that any update will be reflected in both containers, and it reduces the duplication between both containers. At the same time use the --user argument which avoids the error below and allows pip to upgrade itself: ERROR: Cannot uninstall pip 24.0, RECORD file not found. Hint: The package was installed by debian. Signed-off-by: Ales Musil Acked-by: Eelco Chaudron --- v2: Use --user for pip update. Add ack from Eelco. --- utilities/automake.mk | 1 + utilities/containers/fedora/Dockerfile | 35 ++++-------------------- utilities/containers/prepare.sh | 37 ++++++++++++++++++++++++++ utilities/containers/ubuntu/Dockerfile | 37 +++++--------------------- 4 files changed, 49 insertions(+), 61 deletions(-) create mode 100755 utilities/containers/prepare.sh diff --git a/utilities/automake.mk b/utilities/automake.mk index de4f6efb5..03e9096fa 100644 --- a/utilities/automake.mk +++ b/utilities/automake.mk @@ -42,6 +42,7 @@ EXTRA_DIST += \ utilities/containers/Makefile \ utilities/containers/openbfdd.patch \ utilities/containers/py-requirements.txt \ + utilities/containers/prepare.sh \ utilities/containers/fedora/Dockerfile \ utilities/containers/ubuntu/Dockerfile \ utilities/docker/Makefile \ diff --git a/utilities/containers/fedora/Dockerfile b/utilities/containers/fedora/Dockerfile index 019e9f138..f28c00b5d 100755 --- a/utilities/containers/fedora/Dockerfile +++ b/utilities/containers/fedora/Dockerfile @@ -45,41 +45,16 @@ RUN dnf -y update \ && \ dnf clean all -# Compile sparse from source -WORKDIR /workspace/sparse +ENV PATH "${PATH}:${HOME}/bin:${HOME}/.local/bin" -RUN git clone git://git.kernel.org/pub/scm/devel/sparse/sparse.git \ - /workspace/sparse \ - && \ - make -j4 PREFIX=/usr HAVE_LLVM= HAVE_SQLITE= install - -# Compile OpenBFDD from source -WORKDIR /workspace/OpenBFDD +WORKDIR /workspace COPY $CONTAINERS_PATH/openbfdd.patch /tmp/openbfdd.patch -RUN git clone https://github.com/dyninc/OpenBFDD.git \ - /workspace/OpenBFDD \ - && \ - git apply /tmp/openbfdd.patch \ - && \ - ./autogen.sh \ - && \ - ./configure --enable-silent-rules \ - && \ - make \ - && \ - make install - -WORKDIR /workspace - COPY $CONTAINERS_PATH/py-requirements.txt /tmp/py-requirements.txt -# Update and install pip dependencies -RUN python3 -m pip install --upgrade pip \ - && \ - python3 -m pip install wheel \ - && \ - python3 -m pip install -r /tmp/py-requirements.txt +COPY $CONTAINERS_PATH/prepare.sh /tmp/prepare.sh + +RUN /tmp/prepare.sh CMD ["/usr/sbin/init"] diff --git a/utilities/containers/prepare.sh b/utilities/containers/prepare.sh new file mode 100755 index 000000000..b3baa4345 --- /dev/null +++ b/utilities/containers/prepare.sh @@ -0,0 +1,37 @@ +#!/bin/bash -xe + +function compile_sparse() +{ + git clone git://git.kernel.org/pub/scm/devel/sparse/sparse.git \ + /workspace/sparse + + pushd sparse + make -j4 PREFIX=/usr HAVE_LLVM= HAVE_SQLITE= install + popd +} + +function compile_openbfdd() +{ + git clone https://github.com/dyninc/OpenBFDD.git \ + /workspace/OpenBFDD + + pushd OpenBFDD + git apply /tmp/openbfdd.patch + ./autogen.sh + ./configure --enable-silent-rules + make + make install + popd +} + +function install_python_dep() +{ + # The --user should be removed once pip can be upgraded on Ubuntu. + python3 -m pip install --user --upgrade pip + python3 -m pip install wheel + python3 -m pip install -r /tmp/py-requirements.txt +} + +compile_sparse +compile_openbfdd +install_python_dep diff --git a/utilities/containers/ubuntu/Dockerfile b/utilities/containers/ubuntu/Dockerfile index ce7ce16c6..49ba861ac 100755 --- a/utilities/containers/ubuntu/Dockerfile +++ b/utilities/containers/ubuntu/Dockerfile @@ -45,48 +45,23 @@ RUN apt update -y \ && \ apt clean -# Compile sparse from source -WORKDIR /workspace/sparse +ENV PATH "${PATH}:${HOME}/bin:${HOME}/.local/bin" -RUN git clone git://git.kernel.org/pub/scm/devel/sparse/sparse.git \ - /workspace/sparse \ - && \ - make -j4 PREFIX=/usr HAVE_LLVM= HAVE_SQLITE= install - -# Compile OpenBFDD from source -WORKDIR /workspace/OpenBFDD +WORKDIR /workspace COPY $CONTAINERS_PATH/openbfdd.patch /tmp/openbfdd.patch -RUN git clone https://github.com/dyninc/OpenBFDD.git \ - /workspace/OpenBFDD \ - && \ - git apply /tmp/openbfdd.patch \ - && \ - ./autogen.sh \ - && \ - ./configure --enable-silent-rules \ - && \ - make \ - && \ - make install - -WORKDIR /workspace - COPY $CONTAINERS_PATH/py-requirements.txt /tmp/py-requirements.txt +COPY $CONTAINERS_PATH/prepare.sh /tmp/prepare.sh + # Ubuntu 24.04 marks the Python installation as externally managed, allow pip # to install the packages despite that. ENV PIP_BREAK_SYSTEM_PACKAGES 1 -# Update and install pip dependencies -RUN python3 -m pip install --upgrade pip \ - && \ - python3 -m pip install wheel \ - && \ - python3 -m pip install -r /tmp/py-requirements.txt - # The Python Babel fails to detect timezone when it is set to UTC only. ENV TZ Etc/UTC +RUN /tmp/prepare.sh + CMD ["/sbin/init"] From patchwork Mon Jun 24 14:33:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ales Musil X-Patchwork-Id: 1951599 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=bT+idZM7; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4W79Sz16Cmz20X6 for ; Tue, 25 Jun 2024 00:34:10 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id DF12240A44; Mon, 24 Jun 2024 14:34:08 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id eCXODst0X778; Mon, 24 Jun 2024 14:34:06 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.9.56; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 922A6409A2 Authentication-Results: smtp2.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=bT+idZM7 Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id 922A6409A2; Mon, 24 Jun 2024 14:34:06 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 72D91C0012; Mon, 24 Jun 2024 14:34:06 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 3C60AC0011 for ; Mon, 24 Jun 2024 14:34:05 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 28F20608F7 for ; Mon, 24 Jun 2024 14:34:05 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id ZhlLsnBgQCoU for ; Mon, 24 Jun 2024 14:34:03 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=170.10.129.124; helo=us-smtp-delivery-124.mimecast.com; envelope-from=amusil@redhat.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp3.osuosl.org 32D49608A7 Authentication-Results: smtp3.osuosl.org; dmarc=pass (p=none dis=none) header.from=redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 32D49608A7 Authentication-Results: smtp3.osuosl.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=bT+idZM7 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id 32D49608A7 for ; Mon, 24 Jun 2024 14:34:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719239641; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LWHKYDEckd2FyJGJxaoz9hMAuKzFDk4NlgthhoA1lqs=; b=bT+idZM7oS11YyW3x9VaJMnlWlj2ss7TamWzFsklnUrpaQdYqH4+w4e8FYg1z6aRekO2Sh jzPVRspPOxEyLCKd2QF/Z/yYGiADx9ATKG7QO6yMQ0l/KsxWvSh/cJjTcN1jYNGj6rdxaU Q0Yxq1GjCNcjwCPW9ShiVCJ94lL92GI= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-265-mCrzsBwkMcqevy-9TVrfNA-1; Mon, 24 Jun 2024 10:33:59 -0400 X-MC-Unique: mCrzsBwkMcqevy-9TVrfNA-1 Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 623211956050 for ; Mon, 24 Jun 2024 14:33:58 +0000 (UTC) Received: from amusil.redhat.com (unknown [10.45.224.204]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 7B48A1956059; Mon, 24 Jun 2024 14:33:56 +0000 (UTC) From: Ales Musil To: dev@openvswitch.org Date: Mon, 24 Jun 2024 16:33:52 +0200 Message-ID: <20240624143352.370620-2-amusil@redhat.com> In-Reply-To: <20240624143352.370620-1-amusil@redhat.com> References: <20240624143352.370620-1-amusil@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH ovn v2 2/2] ci: Move DPDK build into container. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: dceara@redhat.com Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The DPDK was built as extra step in the CI, however it is useful to have it inside the container prepared. This also helps with reproduction of failures with DPDK by having the exact version inside the container already. Also bump the Ubuntu version for container builds to 24.04 to get Podman 4. This is required to avoid the tar permissions error: tar: dpdk-23.11/.ci: Cannot change mode to rwxrwxr-x: Operation not permitted Signed-off-by: Ales Musil Acked-by: Eelco Chaudron --- v2: Bump the meson version. Add ack from Eelco. --- .ci/ci.sh | 6 -- .ci/dpdk-build.sh | 62 ------------------ .ci/dpdk-prepare.sh | 11 ---- .ci/linux-build.sh | 7 +- .github/workflows/containers.yml | 2 +- .github/workflows/test.yml | 81 +----------------------- Makefile.am | 2 - utilities/containers/fedora/Dockerfile | 3 +- utilities/containers/prepare.sh | 56 ++++++++++++++++ utilities/containers/py-requirements.txt | 1 + utilities/containers/ubuntu/Dockerfile | 3 +- 11 files changed, 66 insertions(+), 168 deletions(-) delete mode 100755 .ci/dpdk-build.sh delete mode 100755 .ci/dpdk-prepare.sh diff --git a/.ci/ci.sh b/.ci/ci.sh index 6beeace84..f543967dc 100755 --- a/.ci/ci.sh +++ b/.ci/ci.sh @@ -16,7 +16,6 @@ OVN_PATH=${OVN_PATH:-$PWD} OVS_PATH=${OVS_PATH:-$OVN_PATH/ovs} -DPDK_PATH=${DPDK_PATH:-$OVN_PATH/dpdk-dir} CONTAINER_CMD=${CONTAINER_CMD:-podman} CONTAINER_WORKSPACE="/workspace" CONTAINER_WORKDIR="/workspace/ovn-tmp" @@ -163,17 +162,12 @@ if [ "$ARCH" = "aarch64" ] && ! check_clang_version_ge "16.0.0"; then ASAN_OPTIONS="detect_leaks=0" fi -if [ -z "$DPDK" ]; then - mkdir -p "$DPDK_PATH" -fi - CONTAINER_ID="$($CONTAINER_CMD run --privileged -d \ --pids-limit=-1 \ --env ASAN_OPTIONS=$ASAN_OPTIONS \ -v /lib/modules/$(uname -r):/lib/modules/$(uname -r):ro \ -v $OVN_PATH:$CONTAINER_WORKSPACE/ovn:Z \ -v $OVS_PATH:$CONTAINER_WORKSPACE/ovs:Z \ - -v $DPDK_PATH:$CONTAINER_WORKSPACE/dpdk-dir:Z \ $IMAGE_NAME)" trap remove_container EXIT diff --git a/.ci/dpdk-build.sh b/.ci/dpdk-build.sh deleted file mode 100755 index 0c13c98c9..000000000 --- a/.ci/dpdk-build.sh +++ /dev/null @@ -1,62 +0,0 @@ -#!/bin/bash - -set -o errexit -set -x - -function build_dpdk() -{ - local DPDK_VER=$1 - local DPDK_OPTS="" - local DPDK_INSTALL_DIR="$(pwd)/dpdk-dir" - local VERSION_FILE="$DPDK_INSTALL_DIR/cached-version" - - rm -rf dpdk-src - rm -rf $DPDK_INSTALL_DIR - - if [ "${DPDK_VER##refs/*/}" != "${DPDK_VER}" ]; then - git clone --single-branch $DPDK_GIT dpdk-src -b "${DPDK_VER##refs/*/}" - pushd dpdk-src - git log -1 --oneline - else - wget https://fast.dpdk.org/rel/dpdk-$1.tar.xz - tar xvf dpdk-$1.tar.xz > /dev/null - DIR_NAME=$(tar -tf dpdk-$1.tar.xz | head -1 | cut -f1 -d"/") - mv ${DIR_NAME} dpdk-src - pushd dpdk-src - fi - - # Switching to 'default' machine to make the dpdk cache usable on - # different CPUs. We can't be sure that all CI machines are exactly same. - DPDK_OPTS="$DPDK_OPTS -Dmachine=default" - - # Disable building DPDK unit tests. Not needed for OVS build or tests. - DPDK_OPTS="$DPDK_OPTS -Dtests=false" - - # Disable DPDK developer mode, this results in less build checks and less - # meson verbose outputs. - DPDK_OPTS="$DPDK_OPTS -Ddeveloper_mode=disabled" - - # OVS compilation and the "ovn-system-dpdk" unit tests (run in the CI) - # only depend on virtio/tap drivers. - # We can disable all remaining drivers to save compilation time. - DPDK_OPTS="$DPDK_OPTS -Denable_drivers=net/null,net/tap,net/virtio" - # OVS depends on the vhost library (and its dependencies). - # net/tap depends on the gso library. - DPDK_OPTS="$DPDK_OPTS -Denable_libs=cryptodev,dmadev,gso,vhost" - - # Install DPDK using prefix. - DPDK_OPTS="$DPDK_OPTS --prefix=$DPDK_INSTALL_DIR" - - meson $DPDK_OPTS build - ninja -C build - ninja -C build install - popd - - # Remove examples sources. - rm -rf $DPDK_INSTALL_DIR/share/dpdk/examples - - echo "Installed DPDK in $DPDK_INSTALL_DIR" - echo "${DPDK_VER}" > ${VERSION_FILE} -} - -build_dpdk $DPDK_VER diff --git a/.ci/dpdk-prepare.sh b/.ci/dpdk-prepare.sh deleted file mode 100755 index 5543da90a..000000000 --- a/.ci/dpdk-prepare.sh +++ /dev/null @@ -1,11 +0,0 @@ -#!/bin/bash - -set -ev - -# Installing wheel separately because it may be needed to build some -# of the packages during dependency backtracking and pip >= 22.0 will -# stop backtracking on build failures: -# https://github.com/pypa/pip/issues/10655 -pip3 install --disable-pip-version-check --user wheel -pip3 install --disable-pip-version-check --user pyelftools -pip3 install --user 'meson==0.53.2' diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh index d4c57d32c..75a9480f9 100755 --- a/.ci/linux-build.sh +++ b/.ci/linux-build.sh @@ -15,10 +15,10 @@ function install_dpdk() { local DPDK_INSTALL_DIR="$(pwd)/dpdk-dir" local VERSION_FILE="${DPDK_INSTALL_DIR}/cached-version" - local DPDK_LIB=${DPDK_INSTALL_DIR}/lib/x86_64-linux-gnu + local DPDK_PC=$(find $DPDK_INSTALL_DIR -type f -name libdpdk-libs.pc) # Export the following path for pkg-config to find the .pc file. - export PKG_CONFIG_PATH=$DPDK_LIB/pkgconfig/:$PKG_CONFIG_PATH + export PKG_CONFIG_PATH="$(dirname $DPDK_PC):$PKG_CONFIG_PATH" if [ ! -f "${VERSION_FILE}" ]; then echo "Could not find DPDK in $DPDK_INSTALL_DIR" @@ -26,8 +26,7 @@ function install_dpdk() fi # As we build inside a container we need to update the prefix. - sed -i -E "s|^prefix=.*|prefix=${DPDK_INSTALL_DIR}|" \ - "$DPDK_LIB/pkgconfig/libdpdk-libs.pc" + sed -i -E "s|^prefix=.*|prefix=${DPDK_INSTALL_DIR}|" $DPDK_PC # Update the library paths. sudo ldconfig diff --git a/.github/workflows/containers.yml b/.github/workflows/containers.yml index 87e28d645..4cce255a8 100644 --- a/.github/workflows/containers.yml +++ b/.github/workflows/containers.yml @@ -15,7 +15,7 @@ env: jobs: container: - runs-on: ubuntu-22.04 + runs-on: ubuntu-24.04 strategy: matrix: distro: [ fedora, ubuntu ] diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index efe2dac25..befa0bfac 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -12,74 +12,6 @@ concurrency: cancel-in-progress: true jobs: - build-dpdk: - env: - dependencies: gcc libnuma-dev ninja-build - CC: gcc - DPDK_GIT: https://dpdk.org/git/dpdk - DPDK_VER: 23.11 - name: dpdk gcc - outputs: - dpdk_key: ${{ steps.gen_dpdk_key.outputs.key }} - runs-on: ubuntu-20.04 - timeout-minutes: 30 - - steps: - - name: checkout - uses: actions/checkout@v4 - - - name: update PATH - run: | - echo "$HOME/bin" >> $GITHUB_PATH - echo "$HOME/.local/bin" >> $GITHUB_PATH - - - name: create ci signature file for the dpdk cache key - # This will collect most of DPDK related lines, so hash will be different - # if something changed in a way we're building DPDK including DPDK_VER. - # This also allows us to use cache from any branch as long as version - # and a way we're building DPDK stays the same. - run: | - grep -irE 'RTE_|DPDK|meson|ninja' .ci/dpdk-* > dpdk-ci-signature - grep -rwE 'DPDK_GIT|DPDK_VER' .github/ >> dpdk-ci-signature - if [ "${DPDK_VER##refs/*/}" != "${DPDK_VER}" ]; then - git ls-remote --heads $DPDK_GIT $DPDK_VER >> dpdk-ci-signature - fi - cat dpdk-ci-signature - - - name: generate ci DPDK key - id: gen_dpdk_key - env: - ci_key: ${{ hashFiles('dpdk-ci-signature') }} - run: echo 'key=dpdk-${{ env.ci_key }}' >> $GITHUB_OUTPUT - - - name: cache - id: dpdk_cache - uses: actions/cache@v4 - with: - path: dpdk-dir - key: ${{ steps.gen_dpdk_key.outputs.key }} - - - name: set up python - if: steps.dpdk_cache.outputs.cache-hit != 'true' - uses: actions/setup-python@v5 - with: - python-version: '3.9' - - - name: update APT cache - if: steps.dpdk_cache.outputs.cache-hit != 'true' - run: sudo apt update || true - - name: install common dependencies - if: steps.dpdk_cache.outputs.cache-hit != 'true' - run: sudo apt install -y ${{ env.dependencies }} - - - name: prepare - if: steps.dpdk_cache.outputs.cache-hit != 'true' - run: ./.ci/dpdk-prepare.sh - - - name: build - if: steps.dpdk_cache.outputs.cache-hit != 'true' - run: ./.ci/dpdk-build.sh - prepare-container: # This job has the following matrix, x: Job trigger, y: Branch # (scheduled jobs run only on main): @@ -93,7 +25,7 @@ jobs: env: DEPENDENCIES: podman name: Prepare container - runs-on: ubuntu-22.04 + runs-on: ubuntu-24.04 steps: - uses: actions/checkout@v4 @@ -124,7 +56,7 @@ jobs: run: podman pull ghcr.io/ovn-org/ovn-tests:${{ env.IMAGE_DISTRO }} - name: Export image - run: podman save -o /tmp/image.tar ovn-org/ovn-tests + run: podman save -o /tmp/image.tar ovn-org/ovn-tests:${{ env.IMAGE_DISTRO }} - name: Cache image id: image_cache @@ -134,7 +66,7 @@ jobs: key: ${{ github.sha }}/${{ github.event_name }} build-linux: - needs: [build-dpdk, prepare-container] + needs: [prepare-container] env: ARCH: ${{ matrix.cfg.arch }} CC: ${{ matrix.cfg.compiler }} @@ -204,13 +136,6 @@ jobs: sort -V | tail -1) working-directory: ovs - - name: cache dpdk - if: matrix.cfg.dpdk != '' - uses: actions/cache@v4 - with: - path: dpdk-dir - key: ${{ needs.build-dpdk.outputs.dpdk_key }} - - name: image cache id: image_cache uses: actions/cache@v4 diff --git a/Makefile.am b/Makefile.am index bfc9565e8..6b0f1913a 100644 --- a/Makefile.am +++ b/Makefile.am @@ -87,8 +87,6 @@ EXTRA_DIST = \ NOTICE \ .cirrus.yml \ .ci/ci.sh \ - .ci/dpdk-build.sh \ - .ci/dpdk-prepare.sh \ .ci/linux-build.sh \ .ci/linux-util.sh \ .ci/osx-build.sh \ diff --git a/utilities/containers/fedora/Dockerfile b/utilities/containers/fedora/Dockerfile index f28c00b5d..078180cff 100755 --- a/utilities/containers/fedora/Dockerfile +++ b/utilities/containers/fedora/Dockerfile @@ -27,6 +27,7 @@ RUN dnf -y update \ libcap-ng-devel \ libtool \ net-tools \ + ninja-build \ nmap-ncat \ numactl-devel \ openssl \ @@ -45,8 +46,6 @@ RUN dnf -y update \ && \ dnf clean all -ENV PATH "${PATH}:${HOME}/bin:${HOME}/.local/bin" - WORKDIR /workspace COPY $CONTAINERS_PATH/openbfdd.patch /tmp/openbfdd.patch diff --git a/utilities/containers/prepare.sh b/utilities/containers/prepare.sh index b3baa4345..460d34a72 100755 --- a/utilities/containers/prepare.sh +++ b/utilities/containers/prepare.sh @@ -1,5 +1,8 @@ #!/bin/bash -xe +DPDK_GIT=https://dpdk.org/git/dpdk +DPDK_VER=23.11 + function compile_sparse() { git clone git://git.kernel.org/pub/scm/devel/sparse/sparse.git \ @@ -32,6 +35,59 @@ function install_python_dep() python3 -m pip install -r /tmp/py-requirements.txt } +function build_dpdk() +{ + local DPDK_OPTS="" + local DPDK_INSTALL_DIR="$(pwd)/dpdk-dir" + local VERSION_FILE="$DPDK_INSTALL_DIR/cached-version" + + if [ "${DPDK_VER##refs/*/}" != "${DPDK_VER}" ]; then + git clone --single-branch $DPDK_GIT dpdk-src \ + -b "${DPDK_VER##refs/*/}" + pushd dpdk-src + git log -1 --oneline + else + curl -O https://fast.dpdk.org/rel/dpdk-$DPDK_VER.tar.xz + tar --no-same-owner -xvf dpdk-$DPDK_VER.tar.xz > /dev/null + DIR_NAME=$(tar -tf dpdk-$DPDK_VER.tar.xz | head -1 | cut -f1 -d"/") + mv ${DIR_NAME} dpdk-src + pushd dpdk-src + fi + + # Switching to 'default' machine to make the dpdk cache usable on + # different CPUs. We can't be sure that all CI machines are exactly same. + DPDK_OPTS="$DPDK_OPTS -Dmachine=default" + + # Disable building DPDK unit tests. Not needed for OVS build or tests. + DPDK_OPTS="$DPDK_OPTS -Dtests=false" + + # Disable DPDK developer mode, this results in less build checks and less + # meson verbose outputs. + DPDK_OPTS="$DPDK_OPTS -Ddeveloper_mode=disabled" + + # OVS compilation and the "ovn-system-dpdk" unit tests (run in the CI) + # only depend on virtio/tap drivers. + # We can disable all remaining drivers to save compilation time. + DPDK_OPTS="$DPDK_OPTS -Denable_drivers=net/null,net/tap,net/virtio" + # OVS depends on the vhost library (and its dependencies). + # net/tap depends on the gso library. + DPDK_OPTS="$DPDK_OPTS -Denable_libs=cryptodev,dmadev,gso,vhost" + + # Install DPDK using prefix. + DPDK_OPTS="$DPDK_OPTS --prefix=$DPDK_INSTALL_DIR" + + meson $DPDK_OPTS build + ninja -C build + ninja -C build install + popd + + # Remove examples sources. + rm -rf $DPDK_INSTALL_DIR/share/dpdk/examples + + echo "${DPDK_VER}" > ${VERSION_FILE} +} + compile_sparse compile_openbfdd install_python_dep +build_dpdk diff --git a/utilities/containers/py-requirements.txt b/utilities/containers/py-requirements.txt index a8e8f17da..2a310c687 100644 --- a/utilities/containers/py-requirements.txt +++ b/utilities/containers/py-requirements.txt @@ -1,4 +1,5 @@ flake8>=6.1.0 +meson>=1.4,<1.5 scapy sphinx setuptools diff --git a/utilities/containers/ubuntu/Dockerfile b/utilities/containers/ubuntu/Dockerfile index 49ba861ac..7cf075122 100755 --- a/utilities/containers/ubuntu/Dockerfile +++ b/utilities/containers/ubuntu/Dockerfile @@ -33,6 +33,7 @@ RUN apt update -y \ llvm-dev \ ncat \ net-tools \ + ninja-build \ python3-dev \ python3-pip \ selinux-policy-dev \ @@ -45,8 +46,6 @@ RUN apt update -y \ && \ apt clean -ENV PATH "${PATH}:${HOME}/bin:${HOME}/.local/bin" - WORKDIR /workspace COPY $CONTAINERS_PATH/openbfdd.patch /tmp/openbfdd.patch