From patchwork Thu Jun 20 12:57:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ales Musil X-Patchwork-Id: 1950143 X-Patchwork-Delegate: dceara@redhat.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=b0juUkJU; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4W4gWz0Psxz20X8 for ; Thu, 20 Jun 2024 22:58:07 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 72C1161623; Thu, 20 Jun 2024 12:58:05 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id xRnO-dA2Bkib; Thu, 20 Jun 2024 12:58:04 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.9.56; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 5602C60ECE Authentication-Results: smtp3.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=b0juUkJU Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 5602C60ECE; Thu, 20 Jun 2024 12:58:04 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 28CF5C0013; Thu, 20 Jun 2024 12:58:04 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id B2E5CC0011 for ; Thu, 20 Jun 2024 12:58:02 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 1F49343546 for ; Thu, 20 Jun 2024 12:57:12 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id tq-RECfCOetS for ; Thu, 20 Jun 2024 12:57:11 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=170.10.133.124; helo=us-smtp-delivery-124.mimecast.com; envelope-from=amusil@redhat.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp2.osuosl.org B4D6D43568 Authentication-Results: smtp2.osuosl.org; dmarc=pass (p=none dis=none) header.from=redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org B4D6D43568 Authentication-Results: smtp2.osuosl.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=b0juUkJU Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id B4D6D43568 for ; Thu, 20 Jun 2024 12:57:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1718888229; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Epm0t1a3CLMg08OqyT5l0r/6M0y85luJxcVgoSu854s=; b=b0juUkJUB8sDyMqg4+SjHpp4bVQWZ2qPR4KAkDm9Flm071qlDX4O2ZAF0QYdgQZh/G+gq3 lWCMEoOQVi9diP17yDmPy0UPb6sNrNCn9oNhqsvZmwnVv6v0X+fFVzhbKbFjI0qgcezR3h vGfR/h9odalAccYY/bxVccLvlX8VKNQ= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-640-FQh6-DOtP-6qwrzKEkKRuQ-1; Thu, 20 Jun 2024 08:57:08 -0400 X-MC-Unique: FQh6-DOtP-6qwrzKEkKRuQ-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 7DEC01956065 for ; Thu, 20 Jun 2024 12:57:07 +0000 (UTC) Received: from amusil.redhat.com (unknown [10.45.226.145]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id A9C5119560AF; Thu, 20 Jun 2024 12:57:05 +0000 (UTC) From: Ales Musil To: dev@openvswitch.org Date: Thu, 20 Jun 2024 14:57:03 +0200 Message-ID: <20240620125704.729797-1-amusil@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH ovn 1/2] ci: Move common build steps into script. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: dceara@redhat.com Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Move common preparation steps into script that can be invoked by both container builds. This will ensure that any update will be reflected in both containers, and it reduces the duplication between both containers. Signed-off-by: Ales Musil Acked-by: Eelco Chaudron --- utilities/automake.mk | 1 + utilities/containers/fedora/Dockerfile | 35 +++--------------------- utilities/containers/prepare.sh | 36 +++++++++++++++++++++++++ utilities/containers/ubuntu/Dockerfile | 37 ++++---------------------- 4 files changed, 46 insertions(+), 63 deletions(-) create mode 100755 utilities/containers/prepare.sh diff --git a/utilities/automake.mk b/utilities/automake.mk index de4f6efb5..03e9096fa 100644 --- a/utilities/automake.mk +++ b/utilities/automake.mk @@ -42,6 +42,7 @@ EXTRA_DIST += \ utilities/containers/Makefile \ utilities/containers/openbfdd.patch \ utilities/containers/py-requirements.txt \ + utilities/containers/prepare.sh \ utilities/containers/fedora/Dockerfile \ utilities/containers/ubuntu/Dockerfile \ utilities/docker/Makefile \ diff --git a/utilities/containers/fedora/Dockerfile b/utilities/containers/fedora/Dockerfile index 019e9f138..9e17a6b20 100755 --- a/utilities/containers/fedora/Dockerfile +++ b/utilities/containers/fedora/Dockerfile @@ -45,41 +45,14 @@ RUN dnf -y update \ && \ dnf clean all -# Compile sparse from source -WORKDIR /workspace/sparse - -RUN git clone git://git.kernel.org/pub/scm/devel/sparse/sparse.git \ - /workspace/sparse \ - && \ - make -j4 PREFIX=/usr HAVE_LLVM= HAVE_SQLITE= install - -# Compile OpenBFDD from source -WORKDIR /workspace/OpenBFDD +WORKDIR /workspace COPY $CONTAINERS_PATH/openbfdd.patch /tmp/openbfdd.patch -RUN git clone https://github.com/dyninc/OpenBFDD.git \ - /workspace/OpenBFDD \ - && \ - git apply /tmp/openbfdd.patch \ - && \ - ./autogen.sh \ - && \ - ./configure --enable-silent-rules \ - && \ - make \ - && \ - make install - -WORKDIR /workspace - COPY $CONTAINERS_PATH/py-requirements.txt /tmp/py-requirements.txt -# Update and install pip dependencies -RUN python3 -m pip install --upgrade pip \ - && \ - python3 -m pip install wheel \ - && \ - python3 -m pip install -r /tmp/py-requirements.txt +COPY $CONTAINERS_PATH/prepare.sh /tmp/prepare.sh + +RUN /tmp/prepare.sh CMD ["/usr/sbin/init"] diff --git a/utilities/containers/prepare.sh b/utilities/containers/prepare.sh new file mode 100755 index 000000000..3aac94590 --- /dev/null +++ b/utilities/containers/prepare.sh @@ -0,0 +1,36 @@ +#!/bin/bash -xe + +function compile_sparse() +{ + git clone git://git.kernel.org/pub/scm/devel/sparse/sparse.git \ + /workspace/sparse + + pushd sparse + make -j4 PREFIX=/usr HAVE_LLVM= HAVE_SQLITE= install + popd +} + +function compile_openbfdd() +{ + git clone https://github.com/dyninc/OpenBFDD.git \ + /workspace/OpenBFDD + + pushd OpenBFDD + git apply /tmp/openbfdd.patch + ./autogen.sh + ./configure --enable-silent-rules + make + make install + popd +} + +function install_python_dep() +{ + python3 -m pip install --upgrade pip + python3 -m pip install wheel + python3 -m pip install -r /tmp/py-requirements.txt +} + +compile_sparse +compile_openbfdd +install_python_dep diff --git a/utilities/containers/ubuntu/Dockerfile b/utilities/containers/ubuntu/Dockerfile index ce7ce16c6..82fa0d715 100755 --- a/utilities/containers/ubuntu/Dockerfile +++ b/utilities/containers/ubuntu/Dockerfile @@ -45,48 +45,21 @@ RUN apt update -y \ && \ apt clean -# Compile sparse from source -WORKDIR /workspace/sparse - -RUN git clone git://git.kernel.org/pub/scm/devel/sparse/sparse.git \ - /workspace/sparse \ - && \ - make -j4 PREFIX=/usr HAVE_LLVM= HAVE_SQLITE= install - -# Compile OpenBFDD from source -WORKDIR /workspace/OpenBFDD +WORKDIR /workspace COPY $CONTAINERS_PATH/openbfdd.patch /tmp/openbfdd.patch -RUN git clone https://github.com/dyninc/OpenBFDD.git \ - /workspace/OpenBFDD \ - && \ - git apply /tmp/openbfdd.patch \ - && \ - ./autogen.sh \ - && \ - ./configure --enable-silent-rules \ - && \ - make \ - && \ - make install - -WORKDIR /workspace - COPY $CONTAINERS_PATH/py-requirements.txt /tmp/py-requirements.txt +COPY $CONTAINERS_PATH/prepare.sh /tmp/prepare.sh + # Ubuntu 24.04 marks the Python installation as externally managed, allow pip # to install the packages despite that. ENV PIP_BREAK_SYSTEM_PACKAGES 1 -# Update and install pip dependencies -RUN python3 -m pip install --upgrade pip \ - && \ - python3 -m pip install wheel \ - && \ - python3 -m pip install -r /tmp/py-requirements.txt - # The Python Babel fails to detect timezone when it is set to UTC only. ENV TZ Etc/UTC +RUN /tmp/prepare.sh + CMD ["/sbin/init"] From patchwork Thu Jun 20 12:57:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ales Musil X-Patchwork-Id: 1950144 X-Patchwork-Delegate: dceara@redhat.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=PqR/KsGs; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4W4gX45DzGz20X8 for ; Thu, 20 Jun 2024 22:58:12 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 2F73B6163E; Thu, 20 Jun 2024 12:58:11 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id iQD6_wAAw-vv; Thu, 20 Jun 2024 12:58:09 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.9.56; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 7E0CA61627 Authentication-Results: smtp3.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=PqR/KsGs Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 7E0CA61627; Thu, 20 Jun 2024 12:58:09 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 4FDADC0012; Thu, 20 Jun 2024 12:58:09 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 03FA7C0011 for ; Thu, 20 Jun 2024 12:58:08 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 8333E61629 for ; Thu, 20 Jun 2024 12:57:15 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id ikxMczpLErYL for ; Thu, 20 Jun 2024 12:57:13 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=170.10.129.124; helo=us-smtp-delivery-124.mimecast.com; envelope-from=amusil@redhat.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp3.osuosl.org 55F39615FB Authentication-Results: smtp3.osuosl.org; dmarc=pass (p=none dis=none) header.from=redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 55F39615FB Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id 55F39615FB for ; Thu, 20 Jun 2024 12:57:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1718888232; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=bm4jNKlgxXMsvYSW6kxldEr8ASESTm+3GDQ27uCI0g8=; b=PqR/KsGswCj8HHTIa3a49VdOZs5uy4XPtb2hfCGKnSLnvN72JjAZvhw9JOWEgLfeprglu4 8Y95WDPmiu1qZ3HgPcDUAmwcWrSvydaoib/9/P93OKtnMv8ZqY5zk1/ti9orNQZXrpbWlI CRbk5bTcgPSHPmNttdExpjIS9nbmc9I= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-557-CCaYG2kGPn2JzpXlLI05fA-1; Thu, 20 Jun 2024 08:57:11 -0400 X-MC-Unique: CCaYG2kGPn2JzpXlLI05fA-1 Received: from mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.40]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 4E8BC1955DE9 for ; Thu, 20 Jun 2024 12:57:10 +0000 (UTC) Received: from amusil.redhat.com (unknown [10.45.226.145]) by mx-prod-int-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id D1B2319560AF; Thu, 20 Jun 2024 12:57:07 +0000 (UTC) From: Ales Musil To: dev@openvswitch.org Date: Thu, 20 Jun 2024 14:57:04 +0200 Message-ID: <20240620125704.729797-2-amusil@redhat.com> In-Reply-To: <20240620125704.729797-1-amusil@redhat.com> References: <20240620125704.729797-1-amusil@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.40 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH ovn 2/2] ci: Move DPDK build into container. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: dceara@redhat.com Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The DPDK was built as extra step in the CI, however it is useful to have it inside the container prepared. This also helps with reproduction of failures with DPDK by having the exact version inside the container already. Also bump the Ubuntu version for container builds to 24.04 to get Podman 4. This is required to avoid the tar permissions error: tar: dpdk-23.11/.ci: Cannot change mode to rwxrwxr-x: Operation not permitted Signed-off-by: Ales Musil Acked-by: Eelco Chaudron --- .ci/ci.sh | 6 -- .ci/dpdk-build.sh | 62 ------------------ .ci/dpdk-prepare.sh | 11 ---- .ci/linux-build.sh | 7 +- .github/workflows/containers.yml | 2 +- .github/workflows/test.yml | 81 +----------------------- Makefile.am | 2 - utilities/containers/fedora/Dockerfile | 1 + utilities/containers/prepare.sh | 56 ++++++++++++++++ utilities/containers/py-requirements.txt | 1 + utilities/containers/ubuntu/Dockerfile | 1 + 11 files changed, 66 insertions(+), 164 deletions(-) delete mode 100755 .ci/dpdk-build.sh delete mode 100755 .ci/dpdk-prepare.sh diff --git a/.ci/ci.sh b/.ci/ci.sh index 6beeace84..f543967dc 100755 --- a/.ci/ci.sh +++ b/.ci/ci.sh @@ -16,7 +16,6 @@ OVN_PATH=${OVN_PATH:-$PWD} OVS_PATH=${OVS_PATH:-$OVN_PATH/ovs} -DPDK_PATH=${DPDK_PATH:-$OVN_PATH/dpdk-dir} CONTAINER_CMD=${CONTAINER_CMD:-podman} CONTAINER_WORKSPACE="/workspace" CONTAINER_WORKDIR="/workspace/ovn-tmp" @@ -163,17 +162,12 @@ if [ "$ARCH" = "aarch64" ] && ! check_clang_version_ge "16.0.0"; then ASAN_OPTIONS="detect_leaks=0" fi -if [ -z "$DPDK" ]; then - mkdir -p "$DPDK_PATH" -fi - CONTAINER_ID="$($CONTAINER_CMD run --privileged -d \ --pids-limit=-1 \ --env ASAN_OPTIONS=$ASAN_OPTIONS \ -v /lib/modules/$(uname -r):/lib/modules/$(uname -r):ro \ -v $OVN_PATH:$CONTAINER_WORKSPACE/ovn:Z \ -v $OVS_PATH:$CONTAINER_WORKSPACE/ovs:Z \ - -v $DPDK_PATH:$CONTAINER_WORKSPACE/dpdk-dir:Z \ $IMAGE_NAME)" trap remove_container EXIT diff --git a/.ci/dpdk-build.sh b/.ci/dpdk-build.sh deleted file mode 100755 index 0c13c98c9..000000000 --- a/.ci/dpdk-build.sh +++ /dev/null @@ -1,62 +0,0 @@ -#!/bin/bash - -set -o errexit -set -x - -function build_dpdk() -{ - local DPDK_VER=$1 - local DPDK_OPTS="" - local DPDK_INSTALL_DIR="$(pwd)/dpdk-dir" - local VERSION_FILE="$DPDK_INSTALL_DIR/cached-version" - - rm -rf dpdk-src - rm -rf $DPDK_INSTALL_DIR - - if [ "${DPDK_VER##refs/*/}" != "${DPDK_VER}" ]; then - git clone --single-branch $DPDK_GIT dpdk-src -b "${DPDK_VER##refs/*/}" - pushd dpdk-src - git log -1 --oneline - else - wget https://fast.dpdk.org/rel/dpdk-$1.tar.xz - tar xvf dpdk-$1.tar.xz > /dev/null - DIR_NAME=$(tar -tf dpdk-$1.tar.xz | head -1 | cut -f1 -d"/") - mv ${DIR_NAME} dpdk-src - pushd dpdk-src - fi - - # Switching to 'default' machine to make the dpdk cache usable on - # different CPUs. We can't be sure that all CI machines are exactly same. - DPDK_OPTS="$DPDK_OPTS -Dmachine=default" - - # Disable building DPDK unit tests. Not needed for OVS build or tests. - DPDK_OPTS="$DPDK_OPTS -Dtests=false" - - # Disable DPDK developer mode, this results in less build checks and less - # meson verbose outputs. - DPDK_OPTS="$DPDK_OPTS -Ddeveloper_mode=disabled" - - # OVS compilation and the "ovn-system-dpdk" unit tests (run in the CI) - # only depend on virtio/tap drivers. - # We can disable all remaining drivers to save compilation time. - DPDK_OPTS="$DPDK_OPTS -Denable_drivers=net/null,net/tap,net/virtio" - # OVS depends on the vhost library (and its dependencies). - # net/tap depends on the gso library. - DPDK_OPTS="$DPDK_OPTS -Denable_libs=cryptodev,dmadev,gso,vhost" - - # Install DPDK using prefix. - DPDK_OPTS="$DPDK_OPTS --prefix=$DPDK_INSTALL_DIR" - - meson $DPDK_OPTS build - ninja -C build - ninja -C build install - popd - - # Remove examples sources. - rm -rf $DPDK_INSTALL_DIR/share/dpdk/examples - - echo "Installed DPDK in $DPDK_INSTALL_DIR" - echo "${DPDK_VER}" > ${VERSION_FILE} -} - -build_dpdk $DPDK_VER diff --git a/.ci/dpdk-prepare.sh b/.ci/dpdk-prepare.sh deleted file mode 100755 index 5543da90a..000000000 --- a/.ci/dpdk-prepare.sh +++ /dev/null @@ -1,11 +0,0 @@ -#!/bin/bash - -set -ev - -# Installing wheel separately because it may be needed to build some -# of the packages during dependency backtracking and pip >= 22.0 will -# stop backtracking on build failures: -# https://github.com/pypa/pip/issues/10655 -pip3 install --disable-pip-version-check --user wheel -pip3 install --disable-pip-version-check --user pyelftools -pip3 install --user 'meson==0.53.2' diff --git a/.ci/linux-build.sh b/.ci/linux-build.sh index d4c57d32c..75a9480f9 100755 --- a/.ci/linux-build.sh +++ b/.ci/linux-build.sh @@ -15,10 +15,10 @@ function install_dpdk() { local DPDK_INSTALL_DIR="$(pwd)/dpdk-dir" local VERSION_FILE="${DPDK_INSTALL_DIR}/cached-version" - local DPDK_LIB=${DPDK_INSTALL_DIR}/lib/x86_64-linux-gnu + local DPDK_PC=$(find $DPDK_INSTALL_DIR -type f -name libdpdk-libs.pc) # Export the following path for pkg-config to find the .pc file. - export PKG_CONFIG_PATH=$DPDK_LIB/pkgconfig/:$PKG_CONFIG_PATH + export PKG_CONFIG_PATH="$(dirname $DPDK_PC):$PKG_CONFIG_PATH" if [ ! -f "${VERSION_FILE}" ]; then echo "Could not find DPDK in $DPDK_INSTALL_DIR" @@ -26,8 +26,7 @@ function install_dpdk() fi # As we build inside a container we need to update the prefix. - sed -i -E "s|^prefix=.*|prefix=${DPDK_INSTALL_DIR}|" \ - "$DPDK_LIB/pkgconfig/libdpdk-libs.pc" + sed -i -E "s|^prefix=.*|prefix=${DPDK_INSTALL_DIR}|" $DPDK_PC # Update the library paths. sudo ldconfig diff --git a/.github/workflows/containers.yml b/.github/workflows/containers.yml index 87e28d645..4cce255a8 100644 --- a/.github/workflows/containers.yml +++ b/.github/workflows/containers.yml @@ -15,7 +15,7 @@ env: jobs: container: - runs-on: ubuntu-22.04 + runs-on: ubuntu-24.04 strategy: matrix: distro: [ fedora, ubuntu ] diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index efe2dac25..befa0bfac 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -12,74 +12,6 @@ concurrency: cancel-in-progress: true jobs: - build-dpdk: - env: - dependencies: gcc libnuma-dev ninja-build - CC: gcc - DPDK_GIT: https://dpdk.org/git/dpdk - DPDK_VER: 23.11 - name: dpdk gcc - outputs: - dpdk_key: ${{ steps.gen_dpdk_key.outputs.key }} - runs-on: ubuntu-20.04 - timeout-minutes: 30 - - steps: - - name: checkout - uses: actions/checkout@v4 - - - name: update PATH - run: | - echo "$HOME/bin" >> $GITHUB_PATH - echo "$HOME/.local/bin" >> $GITHUB_PATH - - - name: create ci signature file for the dpdk cache key - # This will collect most of DPDK related lines, so hash will be different - # if something changed in a way we're building DPDK including DPDK_VER. - # This also allows us to use cache from any branch as long as version - # and a way we're building DPDK stays the same. - run: | - grep -irE 'RTE_|DPDK|meson|ninja' .ci/dpdk-* > dpdk-ci-signature - grep -rwE 'DPDK_GIT|DPDK_VER' .github/ >> dpdk-ci-signature - if [ "${DPDK_VER##refs/*/}" != "${DPDK_VER}" ]; then - git ls-remote --heads $DPDK_GIT $DPDK_VER >> dpdk-ci-signature - fi - cat dpdk-ci-signature - - - name: generate ci DPDK key - id: gen_dpdk_key - env: - ci_key: ${{ hashFiles('dpdk-ci-signature') }} - run: echo 'key=dpdk-${{ env.ci_key }}' >> $GITHUB_OUTPUT - - - name: cache - id: dpdk_cache - uses: actions/cache@v4 - with: - path: dpdk-dir - key: ${{ steps.gen_dpdk_key.outputs.key }} - - - name: set up python - if: steps.dpdk_cache.outputs.cache-hit != 'true' - uses: actions/setup-python@v5 - with: - python-version: '3.9' - - - name: update APT cache - if: steps.dpdk_cache.outputs.cache-hit != 'true' - run: sudo apt update || true - - name: install common dependencies - if: steps.dpdk_cache.outputs.cache-hit != 'true' - run: sudo apt install -y ${{ env.dependencies }} - - - name: prepare - if: steps.dpdk_cache.outputs.cache-hit != 'true' - run: ./.ci/dpdk-prepare.sh - - - name: build - if: steps.dpdk_cache.outputs.cache-hit != 'true' - run: ./.ci/dpdk-build.sh - prepare-container: # This job has the following matrix, x: Job trigger, y: Branch # (scheduled jobs run only on main): @@ -93,7 +25,7 @@ jobs: env: DEPENDENCIES: podman name: Prepare container - runs-on: ubuntu-22.04 + runs-on: ubuntu-24.04 steps: - uses: actions/checkout@v4 @@ -124,7 +56,7 @@ jobs: run: podman pull ghcr.io/ovn-org/ovn-tests:${{ env.IMAGE_DISTRO }} - name: Export image - run: podman save -o /tmp/image.tar ovn-org/ovn-tests + run: podman save -o /tmp/image.tar ovn-org/ovn-tests:${{ env.IMAGE_DISTRO }} - name: Cache image id: image_cache @@ -134,7 +66,7 @@ jobs: key: ${{ github.sha }}/${{ github.event_name }} build-linux: - needs: [build-dpdk, prepare-container] + needs: [prepare-container] env: ARCH: ${{ matrix.cfg.arch }} CC: ${{ matrix.cfg.compiler }} @@ -204,13 +136,6 @@ jobs: sort -V | tail -1) working-directory: ovs - - name: cache dpdk - if: matrix.cfg.dpdk != '' - uses: actions/cache@v4 - with: - path: dpdk-dir - key: ${{ needs.build-dpdk.outputs.dpdk_key }} - - name: image cache id: image_cache uses: actions/cache@v4 diff --git a/Makefile.am b/Makefile.am index bfc9565e8..6b0f1913a 100644 --- a/Makefile.am +++ b/Makefile.am @@ -87,8 +87,6 @@ EXTRA_DIST = \ NOTICE \ .cirrus.yml \ .ci/ci.sh \ - .ci/dpdk-build.sh \ - .ci/dpdk-prepare.sh \ .ci/linux-build.sh \ .ci/linux-util.sh \ .ci/osx-build.sh \ diff --git a/utilities/containers/fedora/Dockerfile b/utilities/containers/fedora/Dockerfile index 9e17a6b20..078180cff 100755 --- a/utilities/containers/fedora/Dockerfile +++ b/utilities/containers/fedora/Dockerfile @@ -27,6 +27,7 @@ RUN dnf -y update \ libcap-ng-devel \ libtool \ net-tools \ + ninja-build \ nmap-ncat \ numactl-devel \ openssl \ diff --git a/utilities/containers/prepare.sh b/utilities/containers/prepare.sh index 3aac94590..8e35b4354 100755 --- a/utilities/containers/prepare.sh +++ b/utilities/containers/prepare.sh @@ -1,5 +1,8 @@ #!/bin/bash -xe +DPDK_GIT=https://dpdk.org/git/dpdk +DPDK_VER=23.11 + function compile_sparse() { git clone git://git.kernel.org/pub/scm/devel/sparse/sparse.git \ @@ -31,6 +34,59 @@ function install_python_dep() python3 -m pip install -r /tmp/py-requirements.txt } +function build_dpdk() +{ + local DPDK_OPTS="" + local DPDK_INSTALL_DIR="$(pwd)/dpdk-dir" + local VERSION_FILE="$DPDK_INSTALL_DIR/cached-version" + + if [ "${DPDK_VER##refs/*/}" != "${DPDK_VER}" ]; then + git clone --single-branch $DPDK_GIT dpdk-src \ + -b "${DPDK_VER##refs/*/}" + pushd dpdk-src + git log -1 --oneline + else + curl -O https://fast.dpdk.org/rel/dpdk-$DPDK_VER.tar.xz + tar --no-same-owner -xvf dpdk-$DPDK_VER.tar.xz > /dev/null + DIR_NAME=$(tar -tf dpdk-$DPDK_VER.tar.xz | head -1 | cut -f1 -d"/") + mv ${DIR_NAME} dpdk-src + pushd dpdk-src + fi + + # Switching to 'default' machine to make the dpdk cache usable on + # different CPUs. We can't be sure that all CI machines are exactly same. + DPDK_OPTS="$DPDK_OPTS -Dmachine=default" + + # Disable building DPDK unit tests. Not needed for OVS build or tests. + DPDK_OPTS="$DPDK_OPTS -Dtests=false" + + # Disable DPDK developer mode, this results in less build checks and less + # meson verbose outputs. + DPDK_OPTS="$DPDK_OPTS -Ddeveloper_mode=disabled" + + # OVS compilation and the "ovn-system-dpdk" unit tests (run in the CI) + # only depend on virtio/tap drivers. + # We can disable all remaining drivers to save compilation time. + DPDK_OPTS="$DPDK_OPTS -Denable_drivers=net/null,net/tap,net/virtio" + # OVS depends on the vhost library (and its dependencies). + # net/tap depends on the gso library. + DPDK_OPTS="$DPDK_OPTS -Denable_libs=cryptodev,dmadev,gso,vhost" + + # Install DPDK using prefix. + DPDK_OPTS="$DPDK_OPTS --prefix=$DPDK_INSTALL_DIR" + + meson $DPDK_OPTS build + ninja -C build + ninja -C build install + popd + + # Remove examples sources. + rm -rf $DPDK_INSTALL_DIR/share/dpdk/examples + + echo "${DPDK_VER}" > ${VERSION_FILE} +} + compile_sparse compile_openbfdd install_python_dep +build_dpdk diff --git a/utilities/containers/py-requirements.txt b/utilities/containers/py-requirements.txt index a8e8f17da..7b12b4ab5 100644 --- a/utilities/containers/py-requirements.txt +++ b/utilities/containers/py-requirements.txt @@ -1,4 +1,5 @@ flake8>=6.1.0 +meson==0.53.2 scapy sphinx setuptools diff --git a/utilities/containers/ubuntu/Dockerfile b/utilities/containers/ubuntu/Dockerfile index 82fa0d715..7cf075122 100755 --- a/utilities/containers/ubuntu/Dockerfile +++ b/utilities/containers/ubuntu/Dockerfile @@ -33,6 +33,7 @@ RUN apt update -y \ llvm-dev \ ncat \ net-tools \ + ninja-build \ python3-dev \ python3-pip \ selinux-policy-dev \