From patchwork Mon Sep 4 13:48:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Haller X-Patchwork-Id: 1829595 X-Patchwork-Delegate: pablo@netfilter.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=K6bRn1Yp; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=2620:137:e000::1:20; helo=out1.vger.email; envelope-from=netfilter-devel-owner@vger.kernel.org; receiver=patchwork.ozlabs.org) Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by legolas.ozlabs.org (Postfix) with ESMTP id 4RfVSm1Wynz1ynq for ; Mon, 4 Sep 2023 23:52:40 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238671AbjIDNwj (ORCPT ); Mon, 4 Sep 2023 09:52:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59036 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236363AbjIDNwi (ORCPT ); Mon, 4 Sep 2023 09:52:38 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 761CECCC for ; Mon, 4 Sep 2023 06:51:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1693835508; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dyJyzrJXTP//klorTIxVGxp4e2fPM814O5Kd1nVIANk=; b=K6bRn1Ypxq4NQfs/paJbjSGzUws/l6XbZn01rYNZR08Do4n5RNXFBhvs9jvF8YmVJ2OAib QjArmdI2T16PI/9NhLL28Jq+V5Og0gkdzWZvg9QnZt51G3KwwlDE/g7MR/rpy6Wqx9BZnV gawDe0ePgbR/RsBSOzEqJSzj2DCfdGE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-553-kMb_ec3UMr6WyyTlp12fgw-1; Mon, 04 Sep 2023 09:51:47 -0400 X-MC-Unique: kMb_ec3UMr6WyyTlp12fgw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id ECB42939EC1 for ; Mon, 4 Sep 2023 13:51:46 +0000 (UTC) Received: from localhost.localdomain (unknown [10.39.192.26]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 6D5521121314; Mon, 4 Sep 2023 13:51:46 +0000 (UTC) From: Thomas Haller To: NetFilter Cc: Thomas Haller Subject: [PATCH nft v3 01/11] tests/shell: rework command line parsing in "run-tests.sh" Date: Mon, 4 Sep 2023 15:48:03 +0200 Message-ID: <20230904135135.1568180-2-thaller@redhat.com> In-Reply-To: <20230904135135.1568180-1-thaller@redhat.com> References: <20230904135135.1568180-1-thaller@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: netfilter-devel@vger.kernel.org Parse the arguments in a loop, so that their order does not matter. Also, soon more command line arguments will be added, and this way of parsing seems more maintainable and flexible. Currently this is still after the is-root check and after unshare. That will be addressed later. Signed-off-by: Thomas Haller --- tests/shell/run-tests.sh | 95 +++++++++++++++++++++++++++------------- 1 file changed, 65 insertions(+), 30 deletions(-) diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh index b66ef4fa4d1f..ae8c6d934dcf 100755 --- a/tests/shell/run-tests.sh +++ b/tests/shell/run-tests.sh @@ -1,10 +1,5 @@ #!/bin/bash -# Configuration -TESTDIR="./$(dirname $0)/testcases" -SRC_NFT="$(dirname $0)/../../src/nft" -DIFF=$(which diff) - msg_error() { echo "E: $1 ..." >&2 exit 1 @@ -18,6 +13,29 @@ msg_info() { echo "I: $1" } +usage() { + echo " $0 [OPTIONS]" + echo + echo "OPTIONS:" + echo " -h|--help : print usage" + echo " -v : sets VERBOSE=y" + echo " -g : sets DUMPGEN=y" + echo " -V : sets VALGRIND=y" + echo " -K : sets KMEMLEAK=y" + echo + echo "ENVIRONMENT VARIABLES:" + echo " NFT= : Path to nft executable" + echo " VERBOSE=*|y : Enable verbose output" + echo " DUMPGEN=*|y : Regenerate dump files" + echo " VALGRIND=*|y : Run \$NFT in valgrind" + echo " KMEMLEAK=*|y : Check for kernel memleaks" +} + +# Configuration +TESTDIR="./$(dirname $0)/testcases" +SRC_NFT="$(dirname $0)/../../src/nft" +DIFF=$(which diff) + if [ "$(id -u)" != "0" ] ; then msg_error "this requires root!" fi @@ -31,6 +49,48 @@ if [ "${1}" != "run" ]; then fi shift +VERBOSE="$VERBOSE" +DUMPGEN="$DUMPGEN" +VALGRIND="$VALGRIND" +KMEMLEAK="$KMEMLEAK" + +TESTS=() + +while [ $# -gt 0 ] ; do + A="$1" + shift + case "$A" in + -v) + VERBOSE=y + ;; + -g) + DUMPGEN=y + ;; + -V) + VALGRIND=y + ;; + -K) + KMEMLEAK=y + ;; + -h|--help) + usage + exit 0 + ;; + --) + TESTS+=( "$@" ) + shift $# + ;; + *) + # Any unrecognized option is treated as a test name, and also + # enable verbose tests. + TESTS+=( "$A" ) + VERBOSE=y + ;; + esac +done + +SINGLE="${TESTS[*]}" + [ -z "$NFT" ] && NFT=$SRC_NFT ${NFT} > /dev/null 2>&1 ret=$? @@ -59,31 +119,6 @@ if [ ! -x "$DIFF" ] ; then DIFF=true fi -if [ "$1" == "-v" ] ; then - VERBOSE=y - shift -fi - -if [ "$1" == "-g" ] ; then - DUMPGEN=y - shift -fi - -if [ "$1" == "-V" ] ; then - VALGRIND=y - shift -fi - -if [ "$1" == "-K" ]; then - KMEMLEAK=y - shift -fi - -for arg in "$@"; do - SINGLE+=" $arg" - VERBOSE=y -done - kernel_cleanup() { $NFT flush ruleset $MODPROBE -raq \ From patchwork Mon Sep 4 13:48:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Haller X-Patchwork-Id: 1829596 X-Patchwork-Delegate: pablo@netfilter.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=GiQPdLW/; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=2620:137:e000::1:20; helo=out1.vger.email; envelope-from=netfilter-devel-owner@vger.kernel.org; receiver=patchwork.ozlabs.org) Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by legolas.ozlabs.org (Postfix) with ESMTP id 4RfVSm3c7Xz26jL for ; Mon, 4 Sep 2023 23:52:40 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232829AbjIDNwj (ORCPT ); Mon, 4 Sep 2023 09:52:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59042 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237621AbjIDNwi (ORCPT ); Mon, 4 Sep 2023 09:52:38 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 02468CD7 for ; Mon, 4 Sep 2023 06:51:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1693835509; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QbrQ3xK4+NxjhGJJdlN5TZjbMVId3IEsUYuyXJX1/b8=; b=GiQPdLW//3E9MzteLL49ICU2q9wauAbLP2sB/Vtyl5NcLuj6QXdQw77iE2XdKPP8XuwsxX XNBXZF2HrLuxTT9oMPgDPzgbB2Po5V0Wc02gHhYqpoNTxaaXEesFIRbomfMfeVm4+NzkZu 9p820YLW0WFjlUt/6X6iZnHRXg5qTO0= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-444-6JykIXGbMSyd7PkCVAoBBQ-1; Mon, 04 Sep 2023 09:51:48 -0400 X-MC-Unique: 6JykIXGbMSyd7PkCVAoBBQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B422929DD98A for ; Mon, 4 Sep 2023 13:51:47 +0000 (UTC) Received: from localhost.localdomain (unknown [10.39.192.26]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 34E3C1121314; Mon, 4 Sep 2023 13:51:47 +0000 (UTC) From: Thomas Haller To: NetFilter Cc: Thomas Haller Subject: [PATCH nft v3 02/11] tests/shell: rework finding tests and add "--list-tests" option Date: Mon, 4 Sep 2023 15:48:04 +0200 Message-ID: <20230904135135.1568180-3-thaller@redhat.com> In-Reply-To: <20230904135135.1568180-1-thaller@redhat.com> References: <20230904135135.1568180-1-thaller@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: netfilter-devel@vger.kernel.org Cleanup finding the test files. Also add a "--list-tests" option to see which tests are found and would run. Also get rid of the FIND="$(which find)" detection. Which system doesn't have a working find? Also, we can just fail when we try to use find, and don't need a check first. This is still after "unshare", which will be addressed next. Signed-off-by: Thomas Haller --- tests/shell/run-tests.sh | 53 ++++++++++++++++++++-------------------- 1 file changed, 26 insertions(+), 27 deletions(-) diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh index ae8c6d934dcf..184dd3f38be5 100755 --- a/tests/shell/run-tests.sh +++ b/tests/shell/run-tests.sh @@ -17,11 +17,12 @@ usage() { echo " $0 [OPTIONS]" echo echo "OPTIONS:" - echo " -h|--help : print usage" - echo " -v : sets VERBOSE=y" - echo " -g : sets DUMPGEN=y" - echo " -V : sets VALGRIND=y" - echo " -K : sets KMEMLEAK=y" + echo " -h|--help : print usage" + echo " -L|--list-tests : list test names and quit" + echo " -v : sets VERBOSE=y" + echo " -g : sets DUMPGEN=y" + echo " -V : sets VALGRIND=y" + echo " -K : sets KMEMLEAK=y" echo echo "ENVIRONMENT VARIABLES:" echo " NFT= : Path to nft executable" @@ -32,8 +33,8 @@ usage() { } # Configuration -TESTDIR="./$(dirname $0)/testcases" -SRC_NFT="$(dirname $0)/../../src/nft" +BASEDIR="$(dirname "$0")" +SRC_NFT="$BASEDIR/../../src/nft" DIFF=$(which diff) if [ "$(id -u)" != "0" ] ; then @@ -53,6 +54,7 @@ VERBOSE="$VERBOSE" DUMPGEN="$DUMPGEN" VALGRIND="$VALGRIND" KMEMLEAK="$KMEMLEAK" +DO_LIST_TESTS= TESTS=() @@ -76,6 +78,9 @@ while [ $# -gt 0 ] ; do usage exit 0 ;; + -L|--list-tests) + DO_LIST_TESTS=y + ;; --) TESTS+=( "$@" ) shift $# @@ -89,7 +94,19 @@ while [ $# -gt 0 ] ; do esac done -SINGLE="${TESTS[*]}" +find_tests() { + find "$1" -type f -executable | sort +} + +if [ "${#TESTS[@]}" -eq 0 ] ; then + TESTS=( $(find_tests "$BASEDIR/testcases/") ) + test "${#TESTS[@]}" -gt 0 || msg_error "Could not find tests" +fi + +if [ "$DO_LIST_TESTS" = y ] ; then + printf '%s\n' "${TESTS[@]}" + exit 0 +fi [ -z "$NFT" ] && NFT=$SRC_NFT ${NFT} > /dev/null 2>&1 @@ -100,15 +117,6 @@ else msg_info "using nft command: ${NFT}" fi -if [ ! -d "$TESTDIR" ] ; then - msg_error "missing testdir $TESTDIR" -fi - -FIND="$(which find)" -if [ ! -x "$FIND" ] ; then - msg_error "no find binary found" -fi - MODPROBE="$(which modprobe)" if [ ! -x "$MODPROBE" ] ; then msg_error "no modprobe binary found" @@ -143,14 +151,6 @@ kernel_cleanup() { nft_xfrm } -find_tests() { - if [ ! -z "$SINGLE" ] ; then - echo $SINGLE - return - fi - ${FIND} ${TESTDIR} -type f -executable | sort -} - printscript() { # (cmd, tmpd) cat < X-Patchwork-Id: 1829597 X-Patchwork-Delegate: pablo@netfilter.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=QpI2WSaB; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=2620:137:e000::1:20; helo=out1.vger.email; envelope-from=netfilter-devel-owner@vger.kernel.org; receiver=patchwork.ozlabs.org) Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by legolas.ozlabs.org (Postfix) with ESMTP id 4RfVSm5hR5z1yhH for ; Mon, 4 Sep 2023 23:52:40 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234430AbjIDNwj (ORCPT ); Mon, 4 Sep 2023 09:52:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59056 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237273AbjIDNwi (ORCPT ); Mon, 4 Sep 2023 09:52:38 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D82ADCDB for ; Mon, 4 Sep 2023 06:51:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1693835510; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mCaplx2b12gPOHy5dzAq/vtfxMJVXEqhklR3v9oyYAA=; b=QpI2WSaB81YyezqZR7ZvE6lwOEPnbC5kVIgdju6oG6KK4SrT05NmoXDmmuQAFDrXT2ovaY cMx1u/opKOHUULt6SDshDbC+3HqJyPYAqrF1eoznZ6nw7UTjg0vIdDoD7aFri+iv2k1ffc aeA2FvooGFMnxEqPx7VYqKXn24sBG1s= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-275-oetfPCniMwqUqiultya67w-1; Mon, 04 Sep 2023 09:51:48 -0400 X-MC-Unique: oetfPCniMwqUqiultya67w-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7C4603C19365 for ; Mon, 4 Sep 2023 13:51:48 +0000 (UTC) Received: from localhost.localdomain (unknown [10.39.192.26]) by smtp.corp.redhat.com (Postfix) with ESMTPS id F0D381121314; Mon, 4 Sep 2023 13:51:47 +0000 (UTC) From: Thomas Haller To: NetFilter Cc: Thomas Haller Subject: [PATCH nft v3 03/11] tests/shell: check test names before start and support directories Date: Mon, 4 Sep 2023 15:48:05 +0200 Message-ID: <20230904135135.1568180-4-thaller@redhat.com> In-Reply-To: <20230904135135.1568180-1-thaller@redhat.com> References: <20230904135135.1568180-1-thaller@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: netfilter-devel@vger.kernel.org Check for valid test names early. That's useful because we treat any unrecognized options as test names. We should detect a mistake early. While at it, also support specifying directory names instead of executable files. Signed-off-by: Thomas Haller --- tests/shell/run-tests.sh | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh index 184dd3f38be5..0a2598f10bed 100755 --- a/tests/shell/run-tests.sh +++ b/tests/shell/run-tests.sh @@ -103,6 +103,18 @@ if [ "${#TESTS[@]}" -eq 0 ] ; then test "${#TESTS[@]}" -gt 0 || msg_error "Could not find tests" fi +TESTSOLD=( "${TESTS[@]}" ) +TESTS=() +for t in "${TESTSOLD[@]}" ; do + if [ -f "$t" -a -x "$t" ] ; then + TESTS+=( "$t" ) + elif [ -d "$t" ] ; then + TESTS+=( $(find_tests "$t") ) + else + msg_error "Unknown test \"$t\"" + fi +done + if [ "$DO_LIST_TESTS" = y ] ; then printf '%s\n' "${TESTS[@]}" exit 0 From patchwork Mon Sep 4 13:48:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Haller X-Patchwork-Id: 1829594 X-Patchwork-Delegate: pablo@netfilter.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=idRgbYK1; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=2620:137:e000::1:20; helo=out1.vger.email; envelope-from=netfilter-devel-owner@vger.kernel.org; receiver=patchwork.ozlabs.org) Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by legolas.ozlabs.org (Postfix) with ESMTP id 4RfVSl68GXz1ynf for ; Mon, 4 Sep 2023 23:52:39 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232137AbjIDNwi (ORCPT ); Mon, 4 Sep 2023 09:52:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59058 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232829AbjIDNwi (ORCPT ); Mon, 4 Sep 2023 09:52:38 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 98966CDE for ; Mon, 4 Sep 2023 06:51:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1693835510; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4VA5K+sjWux1TfpCHpYkqMRAjfeHnEzoVXnDyLJvMn4=; b=idRgbYK1PRwt2ui5Ae0HsgbSMaFS6uqBOPoQluwnlNuKsqwr8dt0rotSe7QUC7fk5NKUvO 6Ja03wwCzJp0lbdfarI5p2Xwh+/VBXFGQsLqm0gbkEN4wwPn55X+MZrwAgpui+uoKU5jfJ md2UY2vgvpF85efOFvTAqcdwrTV5HMs= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-601-7_NUwGSNNfeoCjJg7tXxkw-1; Mon, 04 Sep 2023 09:51:49 -0400 X-MC-Unique: 7_NUwGSNNfeoCjJg7tXxkw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 44A0729DD985 for ; Mon, 4 Sep 2023 13:51:49 +0000 (UTC) Received: from localhost.localdomain (unknown [10.39.192.26]) by smtp.corp.redhat.com (Postfix) with ESMTPS id B8ECB1121314; Mon, 4 Sep 2023 13:51:48 +0000 (UTC) From: Thomas Haller To: NetFilter Cc: Thomas Haller Subject: [PATCH nft v3 04/11] tests/shell: export NFT_TEST_BASEDIR and NFT_TEST_TMPDIR for tests Date: Mon, 4 Sep 2023 15:48:06 +0200 Message-ID: <20230904135135.1568180-5-thaller@redhat.com> In-Reply-To: <20230904135135.1568180-1-thaller@redhat.com> References: <20230904135135.1568180-1-thaller@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: netfilter-devel@vger.kernel.org Let the test wrapper prepare and export two environment variables for the test: - "$NFT_TEST_BASEDIR" is just the top directory where the test scripts lie. - "$NFT_TEST_TMPDIR" is a `mktemp` directory created by "run-tests.sh" and removed at the end. Tests may use that to leave data there. This directory will be used for various things, like the "nft" wrapper in valgrind mode, the results of the tests and possibly as cache for feature detection. The "$NFT_TEST_TMPDIR" was already used before with the "VALGRIND=y" mode. It's only renamed and got an extended purpose. Also drop the unnecessary first detection of "$DIFF" and the "$SRC_NFT" variable. Also, note that the mktemp creates the temporary directory under /tmp. Which is commonly a tempfs. The user can override that by exporting TMPDIR. Signed-off-by: Thomas Haller --- tests/shell/run-tests.sh | 43 +++++++++++++++++++++++++--------------- 1 file changed, 27 insertions(+), 16 deletions(-) diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh index 0a2598f10bed..22a34f1f7898 100755 --- a/tests/shell/run-tests.sh +++ b/tests/shell/run-tests.sh @@ -32,10 +32,10 @@ usage() { echo " KMEMLEAK=*|y : Check for kernel memleaks" } -# Configuration -BASEDIR="$(dirname "$0")" -SRC_NFT="$BASEDIR/../../src/nft" -DIFF=$(which diff) +NFT_TEST_BASEDIR="$(dirname "$0")" + +# Export the base directory. It may be used by tests. +export NFT_TEST_BASEDIR if [ "$(id -u)" != "0" ] ; then msg_error "this requires root!" @@ -99,7 +99,7 @@ find_tests() { } if [ "${#TESTS[@]}" -eq 0 ] ; then - TESTS=( $(find_tests "$BASEDIR/testcases/") ) + TESTS=( $(find_tests "$NFT_TEST_BASEDIR/testcases/") ) test "${#TESTS[@]}" -gt 0 || msg_error "Could not find tests" fi @@ -120,7 +120,7 @@ if [ "$DO_LIST_TESTS" = y ] ; then exit 0 fi -[ -z "$NFT" ] && NFT=$SRC_NFT +[ -z "$NFT" ] && NFT="$NFT_TEST_BASEDIR/../../src/nft" ${NFT} > /dev/null 2>&1 ret=$? if [ ${ret} -eq 126 ] || [ ${ret} -eq 127 ]; then @@ -139,6 +139,23 @@ if [ ! -x "$DIFF" ] ; then DIFF=true fi +cleanup_on_exit() { + test -z "$NFT_TEST_TMPDIR" || rm -rf "$NFT_TEST_TMPDIR" +} +trap cleanup_on_exit EXIT + +_TMPDIR="${TMPDIR:-/tmp}" + +NFT_TEST_TMPDIR="$(mktemp --tmpdir="$_TMPDIR" -d "nft-test.$(date '+%Y%m%d-%H%M%S.%3N').XXXXXX")" +chmod 755 "$NFT_TEST_TMPDIR" + +ln -snf "$NFT_TEST_TMPDIR" "$_TMPDIR/nft-test.latest" + +# export the tmp directory for tests. They may use it, but create +# distinct files! It will be deleted on EXIT. +export NFT_TEST_TMPDIR + + kernel_cleanup() { $NFT flush ruleset $MODPROBE -raq \ @@ -193,16 +210,10 @@ EOF } if [ "$VALGRIND" == "y" ]; then - tmpd=$(mktemp -d) - chmod 755 $tmpd - - msg_info "writing valgrind logs to $tmpd" - - printscript "$NFT" "$tmpd" >${tmpd}/nft - trap "rm ${tmpd}/nft" EXIT - chmod a+x ${tmpd}/nft - - NFT="${tmpd}/nft" + msg_info "writing valgrind logs to $NFT_TEST_TMPDIR" + printscript "$NFT" "$NFT_TEST_TMPDIR" > "$NFT_TEST_TMPDIR/nft" + chmod a+x "$NFT_TEST_TMPDIR/nft" + NFT="$NFT_TEST_TMPDIR/nft" fi echo "" From patchwork Mon Sep 4 13:48:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Haller X-Patchwork-Id: 1829602 X-Patchwork-Delegate: pablo@netfilter.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=dbpsTHt0; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=2620:137:e000::1:20; helo=out1.vger.email; envelope-from=netfilter-devel-owner@vger.kernel.org; receiver=patchwork.ozlabs.org) Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by legolas.ozlabs.org (Postfix) with ESMTP id 4RfVTb74PZz1ynf for ; Mon, 4 Sep 2023 23:53:23 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239130AbjIDNxZ (ORCPT ); Mon, 4 Sep 2023 09:53:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41100 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237887AbjIDNxY (ORCPT ); Mon, 4 Sep 2023 09:53:24 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 86223CE0 for ; Mon, 4 Sep 2023 06:51:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1693835511; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SC94s1nGvKm+XcvSSdeodAD0889Jgt9wNRI6WAUlIMw=; b=dbpsTHt0XsNgWDz6cnej6E434NH1Ec4xkF0TQHS+Xcjx13xlRztKJPMB7NCDREQFizQy3Q kUsHk5t8Ks3K1b2A2p7H86nuP4i41XEU+iNsNEA9ilRorAI77Y9dEVwTC9ZN1wgEv6/POW ARcYh6/FGkf27q4p7HqkQ36TbyODlnM= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-424-f9dqCm1QO0qR1eB-CuTnGA-1; Mon, 04 Sep 2023 09:51:50 -0400 X-MC-Unique: f9dqCm1QO0qR1eB-CuTnGA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2AE4229DD986 for ; Mon, 4 Sep 2023 13:51:50 +0000 (UTC) Received: from localhost.localdomain (unknown [10.39.192.26]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 810241121318; Mon, 4 Sep 2023 13:51:49 +0000 (UTC) From: Thomas Haller To: NetFilter Cc: Thomas Haller Subject: [PATCH nft v3 05/11] tests/shell: run each test in separate namespace and allow rootless Date: Mon, 4 Sep 2023 15:48:07 +0200 Message-ID: <20230904135135.1568180-6-thaller@redhat.com> In-Reply-To: <20230904135135.1568180-1-thaller@redhat.com> References: <20230904135135.1568180-1-thaller@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: netfilter-devel@vger.kernel.org Don't unshare the entire shell script. Instead, call unshare each test separately. That means, all tests use now a different sandbox and will also allow (with further changes) to run them in parallel. Also, allow to run rootless/unprivileged. The script first tries to run a separate PID+USER+NET namespace. If that fails, it downgrades to USER+NET. If that fails, it downgrades to a separate NET namespace. If unshare still fails, the script fails entirely. That differs from before, where the script would proceed without sandboxing. The script will now always require that unsharing works, unless the user opts-out. If the user cannot unshare, they have two options. Either the NFT_TEST_NO_UNSHARE=y variable (-U/--no-unshare), which disables unshare done by the script. This allows the user to run the script as before. Alternatively, the user can set NFT_TEST_UNSHARE_CMD for the command to unshare, including setting it to empty for no unshare. If we are able to create a separate USER namespace, then this mode allows to run the test as rootless/unprivileged. We no longer require [ `id -u` = 0 ]. Some tests may not work as rootless. For example, the socket buffers is limited by /proc/sys/net/core/{wmem_max,rmem_max} which real-root can override, but rootless tests cannot. Such tests should check for [ "$NFT_TEST_HAVE_REALROOT" != y ] and skip gracefully. Usually, the user doesn't need to tell the script whether they have real-root. The script will autodetect it via [ `id -u` = 0 ]. But that won't work when run inside a rootless container already. In that case, the user would want to tell the script that there is no real-root. They can do so via the -R/--without-root option or NFT_TEST_HAVE_REALROOT=n. If tests wish, the can know whether they run inside "unshare" environment by checking for [ "$NFT_TEST_IS_UNSHARED" = y ]. When setting NFT_TEST_UNSHARE_CMD to override the unshare command, users may want to also set NFT_TEST_IS_UNSHARED= and NFT_TEST_HAVE_REALROOT= correctly. As we run each test in a separate unshare environment, we need a wrapper "tests/shell/helpers/test-wrapper.sh" around the test, which executes inside the tested environment. Also, each test gets its own temp directory prepared in NFT_TEST_TESTTMPDIR. This is also the place, where test artifacts and results will be collected. Signed-off-by: Thomas Haller --- tests/shell/helpers/test-wrapper.sh | 21 ++++++ tests/shell/run-tests.sh | 109 ++++++++++++++++++++++------ 2 files changed, 107 insertions(+), 23 deletions(-) create mode 100755 tests/shell/helpers/test-wrapper.sh diff --git a/tests/shell/helpers/test-wrapper.sh b/tests/shell/helpers/test-wrapper.sh new file mode 100755 index 000000000000..99546f060e26 --- /dev/null +++ b/tests/shell/helpers/test-wrapper.sh @@ -0,0 +1,21 @@ +#!/bin/bash -e + +# This wrapper wraps the invocation of the test. It is called by run-tests.sh, +# and already in the unshared namespace. +# +# For some printf debugging, you can also patch this file. + +TEST="$1" + +rc_test=0 +"$TEST" |& tee "$NFT_TEST_TESTTMPDIR/testout.log" || rc_test=$? + +if [ "$rc_test" -eq 0 ] ; then + echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc_test-ok" +else + echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc_test-failed" +fi + +$NFT list ruleset > "$NFT_TEST_TESTTMPDIR/ruleset-after" + +exit "$rc_test" diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh index 22a34f1f7898..3d0ef6fa8499 100755 --- a/tests/shell/run-tests.sh +++ b/tests/shell/run-tests.sh @@ -17,12 +17,14 @@ usage() { echo " $0 [OPTIONS]" echo echo "OPTIONS:" - echo " -h|--help : print usage" - echo " -L|--list-tests : list test names and quit" - echo " -v : sets VERBOSE=y" - echo " -g : sets DUMPGEN=y" - echo " -V : sets VALGRIND=y" - echo " -K : sets KMEMLEAK=y" + echo " -h|--help : print usage" + echo " -L|--list-tests : list test names and quit" + echo " -v : sets VERBOSE=y" + echo " -g : sets DUMPGEN=y" + echo " -V : sets VALGRIND=y" + echo " -K : sets KMEMLEAK=y" + echo " -R|--without-realroot : sets NFT_TEST_HAVE_REALROOT=n" + echo " -U|--no-unshare : sets NFT_TEST_NO_UNSHARE=y" echo echo "ENVIRONMENT VARIABLES:" echo " NFT= : Path to nft executable" @@ -30,6 +32,24 @@ usage() { echo " DUMPGEN=*|y : Regenerate dump files" echo " VALGRIND=*|y : Run \$NFT in valgrind" echo " KMEMLEAK=*|y : Check for kernel memleaks" + echo " NFT_TEST_HAVE_REALROOT=*|y : To indicate whether the test has real root permissions." + echo " Usually, you don't need this and it gets autodetected." + echo " You might want to set it, if you know better than the" + echo " \`id -u\` check, whether the user is root in the main namespace." + echo " Note that without real root, certain tests may not work," + echo " e.g. due to limited /proc/sys/net/core/{wmem_max,rmem_max}." + echo " Checks that cannot pass in such environment should check for" + echo " [ \"\$NFT_TEST_HAVE_REALROOT\" != y ] and skip gracefully." + echo " NFT_TEST_NO_UNSHARE=*|y : Usually, each test will run in a separate user namespace." + echo " That allows to run rootless." + echo " If creating a user namespace fails, fallback to only unshare the" + echo " network namespace. This requires root. If that still fails, error out." + echo " Setting NFT_TEST_NO_UNSHARE=y disables any unshare and the test runs" + echo " in the callers namespace. This is ignored when NFT_TEST_UNSHARE_CMD is set." + echo " NFT_TEST_UNSHARE_CMD=cmd : when set, NFT_TEST_NO_UNSHARE is ignored and the script" + echo " will not try to unshare. Instead, it uses this command to unshare." + echo " Set to empty to not unshare. You may want to export NFT_TEST_IS_UNSHARED=" + echo " and NFT_TEST_HAVE_REALROOT= accordingly." } NFT_TEST_BASEDIR="$(dirname "$0")" @@ -37,23 +57,12 @@ NFT_TEST_BASEDIR="$(dirname "$0")" # Export the base directory. It may be used by tests. export NFT_TEST_BASEDIR -if [ "$(id -u)" != "0" ] ; then - msg_error "this requires root!" -fi - -if [ "${1}" != "run" ]; then - if unshare -f -n true; then - unshare -n "${0}" run $@ - exit $? - fi - msg_warn "cannot run in own namespace, connectivity might break" -fi -shift - VERBOSE="$VERBOSE" DUMPGEN="$DUMPGEN" VALGRIND="$VALGRIND" KMEMLEAK="$KMEMLEAK" +NFT_TEST_HAVE_REALROOT="$NFT_TEST_HAVE_REALROOT" +NFT_TEST_NO_UNSHARE="$NFT_TEST_NO_UNSHARE" DO_LIST_TESTS= TESTS=() @@ -81,6 +90,12 @@ while [ $# -gt 0 ] ; do -L|--list-tests) DO_LIST_TESTS=y ;; + -R|--without-realroot) + NFT_TEST_HAVE_REALROOT=n + ;; + -U|--no-unshare) + NFT_TEST_NO_UNSHARE=y + ;; --) TESTS+=( "$@" ) shift $# @@ -120,6 +135,42 @@ if [ "$DO_LIST_TESTS" = y ] ; then exit 0 fi +if [ "$NFT_TEST_HAVE_REALROOT" = "" ] ; then + # The caller didn't set NFT_TEST_HAVE_REALROOT and didn't specify + # -R/--without-root option. Autodetect it based on `id -u`. + export NFT_TEST_HAVE_REALROOT="$(test "$(id -u)" = "0" && echo y || echo n)" +fi + +if [ -n "${NFT_TEST_UNSHARE_CMD+x}" ] ; then + # User overrides the unshare command. We ignore NFT_TEST_NO_UNSHARE. + # It may be set to empty, to not unshare. The user should export NFT_TEST_IS_UNSHARED= + # accordingly. + NFT_TEST_UNSHARE_CMD="${NFT_TEST_UNSHARE_CMD}" + NFT_TEST_IS_UNSHARED="${NFT_TEST_IS_UNSHARED}" +else + NFT_TEST_UNSHARE_CMD="" + NFT_TEST_IS_UNSHARED="n" + if [ "$NFT_TEST_NO_UNSHARE" != y ] ; then + # We unshare both if we NFT_TEST_HAVE_REALROOT and the rootless/unpriv + # case. Without real root, some tests may fail. Tests that don't work + # without real root should check for [ "$NFT_TEST_HAVE_REALROOT" != y ] + # and skip gracefully. + NFT_TEST_UNSHARE_CMD="unshare -f -p --mount-proc -U --map-root-user -n" + if ! $NFT_TEST_UNSHARE_CMD true &>/dev/null ; then + NFT_TEST_UNSHARE_CMD="unshare -f -U --map-root-user -n" + if ! $NFT_TEST_UNSHARE_CMD true &>/dev/null ; then + NFT_TEST_UNSHARE_CMD="unshare -f -n" + if ! $NFT_TEST_UNSHARE_CMD true &>/dev/null ; then + msg_error "Unshare does not work. Run as root with -U/--no-unshare/NFT_TEST_NO_UNSHARE=y or set NFT_TEST_UNSHARE_CMD" + fi + fi + fi + NFT_TEST_IS_UNSHARED=y + fi +fi +# If tests wish, they can know whether they are unshared via this variable. +export NFT_TEST_IS_UNSHARED + [ -z "$NFT" ] && NFT="$NFT_TEST_BASEDIR/../../src/nft" ${NFT} > /dev/null 2>&1 ret=$? @@ -157,7 +208,9 @@ export NFT_TEST_TMPDIR kernel_cleanup() { - $NFT flush ruleset + if [ "$NFT_TEST_IS_UNSHARED" != y ] ; then + $NFT flush ruleset + fi $MODPROBE -raq \ nft_reject_ipv4 nft_reject_bridge nft_reject_ipv6 nft_reject \ nft_redir_ipv4 nft_redir_ipv6 nft_redir \ @@ -275,18 +328,23 @@ for testfile in "${TESTS[@]}" ; do read taint < /proc/sys/kernel/tainted kernel_cleanup + # We also create and export a test-specific temporary directory. + NFT_TEST_TESTTMPDIR="$(mktemp -p "$NFT_TEST_TMPDIR" -d "test-${testfile//\//-}.XXXXXX")" + export NFT_TEST_TESTTMPDIR + msg_info "[EXECUTING] $testfile" - test_output=$(NFT="$NFT" DIFF=$DIFF ${testfile} 2>&1) + test_output="$(NFT="$NFT" DIFF=$DIFF $NFT_TEST_UNSHARE_CMD "$NFT_TEST_BASEDIR/helpers/test-wrapper.sh" "$testfile" 2>&1)" rc_got=$? echo -en "\033[1A\033[K" # clean the [EXECUTING] foobar line if [ "$rc_got" -eq 0 ] ; then + # FIXME: this should move inside test-wrapper.sh. # check nft dump only for positive tests dumppath="$(dirname ${testfile})/dumps" dumpfile="${dumppath}/$(basename ${testfile}).nft" rc_spec=0 if [ "$rc_got" -eq 0 ] && [ -f ${dumpfile} ]; then - test_output=$(${DIFF} -u ${dumpfile} <($NFT list ruleset) 2>&1) + test_output=$(${DIFF} -u ${dumpfile} <(cat "$NFT_TEST_TESTTMPDIR/ruleset-after") 2>&1) rc_spec=$? fi @@ -297,7 +355,7 @@ for testfile in "${TESTS[@]}" ; do if [ "$DUMPGEN" == "y" ] && [ "$rc_got" == 0 ] && [ ! -f "${dumpfile}" ]; then mkdir -p "${dumppath}" - $NFT list ruleset > "${dumpfile}" + cat "$NFT_TEST_TESTTMPDIR/ruleset-after" > "${dumpfile}" fi else ((failed++)) @@ -335,4 +393,9 @@ check_kmemleak_force msg_info "results: [OK] $ok [FAILED] $failed [TOTAL] $((ok+failed))" kernel_cleanup + +if [ "$failed" -gt 0 -a "$NFT_TEST_HAVE_REALROOT" != y ] ; then + msg_info "test was not running as real root" +fi + [ "$failed" -eq 0 ] From patchwork Mon Sep 4 13:48:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Haller X-Patchwork-Id: 1829598 X-Patchwork-Delegate: pablo@netfilter.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=2620:137:e000::1:20; helo=out1.vger.email; envelope-from=netfilter-devel-owner@vger.kernel.org; receiver=patchwork.ozlabs.org) Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by legolas.ozlabs.org (Postfix) with ESMTP id 4RfVTR5DZwz1yhH for ; Mon, 4 Sep 2023 23:53:15 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238262AbjIDNxR (ORCPT ); Mon, 4 Sep 2023 09:53:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41092 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229796AbjIDNxR (ORCPT ); Mon, 4 Sep 2023 09:53:17 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 68C1BCE2 for ; Mon, 4 Sep 2023 06:51:53 -0700 (PDT) Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-266-e1V1GpuTMRmtUKJfWjPTFQ-1; Mon, 04 Sep 2023 09:51:51 -0400 X-MC-Unique: e1V1GpuTMRmtUKJfWjPTFQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E718980602D for ; Mon, 4 Sep 2023 13:51:50 +0000 (UTC) Received: from localhost.localdomain (unknown [10.39.192.26]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 66E8F1121314; Mon, 4 Sep 2023 13:51:50 +0000 (UTC) From: Thomas Haller To: NetFilter Cc: Thomas Haller Subject: [PATCH nft v3 06/11] tests/shell: interpret an exit code of 77 from scripts as "skipped" Date: Mon, 4 Sep 2023 15:48:08 +0200 Message-ID: <20230904135135.1568180-7-thaller@redhat.com> In-Reply-To: <20230904135135.1568180-1-thaller@redhat.com> References: <20230904135135.1568180-1-thaller@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: netfilter-devel@vger.kernel.org Allow scripts to indicate that a test could not run by exiting 77. "77" is chosen as exit code from automake's testsuites ([1]). Compare to git-bisect which chooses 125 to indicate skipped. [1] https://www.gnu.org/software/automake/manual/html_node/Scripts_002dbased-Testsuites.html Signed-off-by: Thomas Haller --- tests/shell/helpers/test-wrapper.sh | 2 ++ tests/shell/run-tests.sh | 11 ++++++++++- 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/tests/shell/helpers/test-wrapper.sh b/tests/shell/helpers/test-wrapper.sh index 99546f060e26..8f0dc685e1fe 100755 --- a/tests/shell/helpers/test-wrapper.sh +++ b/tests/shell/helpers/test-wrapper.sh @@ -12,6 +12,8 @@ rc_test=0 if [ "$rc_test" -eq 0 ] ; then echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc_test-ok" +elif [ "$rc_test" -eq 77 ] ; then + echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc_test-skipped" else echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc_test-failed" fi diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh index 3d0ef6fa8499..f3b58e1e200b 100755 --- a/tests/shell/run-tests.sh +++ b/tests/shell/run-tests.sh @@ -271,6 +271,7 @@ fi echo "" ok=0 +skipped=0 failed=0 taint=0 @@ -366,6 +367,14 @@ for testfile in "${TESTS[@]}" ; do msg_warn "[DUMP FAIL] $testfile" fi fi + elif [ "$rc_got" -eq 77 ] ; then + ((skipped++)) + if [ "$VERBOSE" == "y" ] ; then + msg_warn "[SKIPPED] $testfile" + [ ! -z "$test_output" ] && echo "$test_output" + else + msg_warn "[SKIPPED] $testfile" + fi else ((failed++)) if [ "$VERBOSE" == "y" ] ; then @@ -390,7 +399,7 @@ echo "" kmemleak_found=0 check_kmemleak_force -msg_info "results: [OK] $ok [FAILED] $failed [TOTAL] $((ok+failed))" +msg_info "results: [OK] $ok [SKIPPED] $skipped [FAILED] $failed [TOTAL] $((ok+skipped+failed))" kernel_cleanup From patchwork Mon Sep 4 13:48:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Haller X-Patchwork-Id: 1829601 X-Patchwork-Delegate: pablo@netfilter.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=dcmh4+8e; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=2620:137:e000::1:20; helo=out1.vger.email; envelope-from=netfilter-devel-owner@vger.kernel.org; receiver=patchwork.ozlabs.org) Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by legolas.ozlabs.org (Postfix) with ESMTP id 4RfVTb4zC8z1yhH for ; Mon, 4 Sep 2023 23:53:23 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239093AbjIDNxY (ORCPT ); Mon, 4 Sep 2023 09:53:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41112 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238895AbjIDNxY (ORCPT ); Mon, 4 Sep 2023 09:53:24 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E0382CE5 for ; Mon, 4 Sep 2023 06:51:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1693835513; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TRp+9geDN7Gd6Q0zMTSUf0dW/Xt1QaWQfF4/fhcgCLg=; b=dcmh4+8e7m1hSWWsUrMhvqvHwFxeUS0mlgdunwiDk2uFm+ipPW0VTSSjcwWemFIXK+Sror RZ+bgkNr1ex4dzLFflpID3BqOxVGaRmNDJOPgWJrUGw4TF2VEo3D8jPym7rXe2c0VNnBEZ l5xW2hwy5E+K5Pu4581l45LHJRyANQA= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-621-NG_W9QtaNHy16dEh61ozrQ-1; Mon, 04 Sep 2023 09:51:52 -0400 X-MC-Unique: NG_W9QtaNHy16dEh61ozrQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AF23A939EC2 for ; Mon, 4 Sep 2023 13:51:51 +0000 (UTC) Received: from localhost.localdomain (unknown [10.39.192.26]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 2EFAE1121314; Mon, 4 Sep 2023 13:51:51 +0000 (UTC) From: Thomas Haller To: NetFilter Cc: Thomas Haller Subject: [PATCH nft v3 07/11] tests/shell: support --keep-logs option (NFT_TEST_KEEP_LOGS=y) to preserve test output Date: Mon, 4 Sep 2023 15:48:09 +0200 Message-ID: <20230904135135.1568180-8-thaller@redhat.com> In-Reply-To: <20230904135135.1568180-1-thaller@redhat.com> References: <20230904135135.1568180-1-thaller@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: netfilter-devel@vger.kernel.org The test output is now all collected in the temporary directory. On success, that directory is deleted. Add an option to always preserve that directory. Signed-off-by: Thomas Haller --- tests/shell/run-tests.sh | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh index f3b58e1e200b..8da0cc3d5702 100755 --- a/tests/shell/run-tests.sh +++ b/tests/shell/run-tests.sh @@ -25,6 +25,7 @@ usage() { echo " -K : sets KMEMLEAK=y" echo " -R|--without-realroot : sets NFT_TEST_HAVE_REALROOT=n" echo " -U|--no-unshare : sets NFT_TEST_NO_UNSHARE=y" + echo " -k|--keep-logs : sets NFT_TEST_KEEP_LOGS=y" echo echo "ENVIRONMENT VARIABLES:" echo " NFT= : Path to nft executable" @@ -50,6 +51,7 @@ usage() { echo " will not try to unshare. Instead, it uses this command to unshare." echo " Set to empty to not unshare. You may want to export NFT_TEST_IS_UNSHARED=" echo " and NFT_TEST_HAVE_REALROOT= accordingly." + echo " NFT_TEST_KEEP_LOGS=*|y: Keep the temp directory. On success, it will be deleted by default." } NFT_TEST_BASEDIR="$(dirname "$0")" @@ -61,6 +63,7 @@ VERBOSE="$VERBOSE" DUMPGEN="$DUMPGEN" VALGRIND="$VALGRIND" KMEMLEAK="$KMEMLEAK" +NFT_TEST_KEEP_LOGS="$NFT_TEST_KEEP_LOGS" NFT_TEST_HAVE_REALROOT="$NFT_TEST_HAVE_REALROOT" NFT_TEST_NO_UNSHARE="$NFT_TEST_NO_UNSHARE" DO_LIST_TESTS= @@ -87,6 +90,9 @@ while [ $# -gt 0 ] ; do usage exit 0 ;; + -k|--keep-logs) + NFT_TEST_KEEP_LOGS=y + ;; -L|--list-tests) DO_LIST_TESTS=y ;; @@ -202,8 +208,8 @@ chmod 755 "$NFT_TEST_TMPDIR" ln -snf "$NFT_TEST_TMPDIR" "$_TMPDIR/nft-test.latest" -# export the tmp directory for tests. They may use it, but create -# distinct files! It will be deleted on EXIT. +# export the tmp directory for tests. They may use it, but create distinct +# files! On success, it will be deleted on EXIT. See also "--keep-logs" export NFT_TEST_TMPDIR @@ -407,4 +413,12 @@ if [ "$failed" -gt 0 -a "$NFT_TEST_HAVE_REALROOT" != y ] ; then msg_info "test was not running as real root" fi +if [ "$failed" -gt 0 -o "$NFT_TEST_KEEP_LOGS" = y ] ; then + ln -snf "$NFT_TEST_TMPDIR" "$_TMPDIR/nftables-test.latest" + msg_info "check the temp directory \"$NFT_TEST_TMPDIR\" (\"$_TMPDIR/nftables-test.latest\")" + msg_info " ls -lad /tmp/nftables-test.latest/*/*" + msg_info " grep -R ^ /tmp/nftables-test.latest/" + NFT_TEST_TMPDIR= +fi + [ "$failed" -eq 0 ] From patchwork Mon Sep 4 13:48:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Haller X-Patchwork-Id: 1829600 X-Patchwork-Delegate: pablo@netfilter.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=d99zbktl; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=2620:137:e000::1:20; helo=out1.vger.email; envelope-from=netfilter-devel-owner@vger.kernel.org; receiver=patchwork.ozlabs.org) Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by legolas.ozlabs.org (Postfix) with ESMTP id 4RfVTS4J0Hz1yhH for ; Mon, 4 Sep 2023 23:53:16 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237273AbjIDNxS (ORCPT ); Mon, 4 Sep 2023 09:53:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41118 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237621AbjIDNxR (ORCPT ); Mon, 4 Sep 2023 09:53:17 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0F4E0CF6 for ; Mon, 4 Sep 2023 06:51:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1693835514; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fSiQhYgUH0xGuXT9jFizlEcbh7nfbgNNMnLZSbBSiog=; b=d99zbktlo+5tf7YEb8csrhUS3ywTZa7jqhiyXJfJ0H9nH/xDjsriAs9qtypVpMcZG10GuG fv2tt86QegyC5wF+WFYTN0u8t5N3L67lA+UFZjKXzdUpnEyo0oBpzLNu5rEeJcDtUkwV8x 1IEL9fk0XQBLr1RVKxFxkufTcWrV9z8= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-467-aMA4N6QrM16ootYmLsDU0g-1; Mon, 04 Sep 2023 09:51:52 -0400 X-MC-Unique: aMA4N6QrM16ootYmLsDU0g-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 766C11C0CCA2 for ; Mon, 4 Sep 2023 13:51:52 +0000 (UTC) Received: from localhost.localdomain (unknown [10.39.192.26]) by smtp.corp.redhat.com (Postfix) with ESMTPS id EADAA1121314; Mon, 4 Sep 2023 13:51:51 +0000 (UTC) From: Thomas Haller To: NetFilter Cc: Thomas Haller Subject: [PATCH nft v3 08/11] tests/shell: move the dump diff handling inside "test-wrapper.sh" Date: Mon, 4 Sep 2023 15:48:10 +0200 Message-ID: <20230904135135.1568180-9-thaller@redhat.com> In-Reply-To: <20230904135135.1568180-1-thaller@redhat.com> References: <20230904135135.1568180-1-thaller@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: netfilter-devel@vger.kernel.org This fits there better. At this point, we are still inside the unshared namespace and right after the test. The test-wrapper.sh should compare (and generate) the dumps. Also change behavior for DUMPGEN=y. - Previously it would only rewrite the dump if the dumpfile didn't exist yet. Now instead, always rewrite the file with DUMPGEN=y. The mode of operation is anyway, that the developer afterwards checks `git diff|status` to pick up the changes. There should be no changes to existing files (as existing tests are supposed to pass). So a diff there either means something went wrong (and we should see it) or it just means the dumps correctly should be regenerated. - also, only generate the file if the "dumps/" directory exists. This allows to write tests that don't have a dump file and don't get it automatically generated. The test wrapper will return a special error code 124 to indicate that the test passed, but the dumps file differed. Signed-off-by: Thomas Haller --- tests/shell/helpers/test-wrapper.sh | 56 +++++++++++++++++++++++++---- tests/shell/run-tests.sh | 45 +++++++++-------------- 2 files changed, 66 insertions(+), 35 deletions(-) diff --git a/tests/shell/helpers/test-wrapper.sh b/tests/shell/helpers/test-wrapper.sh index 8f0dc685e1fe..f9d759d0bb03 100755 --- a/tests/shell/helpers/test-wrapper.sh +++ b/tests/shell/helpers/test-wrapper.sh @@ -6,18 +6,60 @@ # For some printf debugging, you can also patch this file. TEST="$1" +TESTBASE="$(basename "$TEST")" +TESTDIR="$(dirname "$TEST")" rc_test=0 "$TEST" |& tee "$NFT_TEST_TESTTMPDIR/testout.log" || rc_test=$? -if [ "$rc_test" -eq 0 ] ; then - echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc_test-ok" +$NFT list ruleset > "$NFT_TEST_TESTTMPDIR/ruleset-after" + +DUMPPATH="$TESTDIR/dumps" +DUMPFILE="$DUMPPATH/$TESTBASE.nft" + +dump_written= +rc_dump= + +# The caller can request a re-geneating of the dumps, by setting +# DUMPGEN=y. +# +# This only will happen if the command completed with success. +# +# It also will only happen for tests, that have a "$DUMPPATH" directory. There +# might be tests, that don't want to have dumps created. The existence of the +# directory controls that. +if [ "$rc_test" -eq 0 -a "$DUMPGEN" = y -a -d "$DUMPPATH" ] ; then + dump_written=y + cat "$NFT_TEST_TESTTMPDIR/ruleset-after" > "$DUMPFILE" +fi + +if [ "$rc_test" -ne 77 -a -f "$DUMPFILE" ] ; then + rc_dump=0 + if [ "$dump_written" != y ] ; then + $DIFF -u "$DUMPFILE" "$NFT_TEST_TESTTMPDIR/ruleset-after" &> "$NFT_TEST_TESTTMPDIR/ruleset-diff" || rc_dump=$? + if [ "$rc_dump" -eq 0 ] ; then + rm -f "$NFT_TEST_TESTTMPDIR/ruleset-diff" + fi + fi +fi + +rc_exit="$rc_test" +if [ -n "$rc_dump" ] && [ "$rc_dump" -ne 0 ] ; then + echo "$DUMPFILE" > "$NFT_TEST_TESTTMPDIR/rc-failed-dump" + echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc-failed" + if [ "$rc_exit" -eq 0 ] ; then + # Special exit code to indicate dump diff. + rc_exit=124 + fi +elif [ "$rc_test" -eq 0 ] ; then + echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc-ok" elif [ "$rc_test" -eq 77 ] ; then - echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc_test-skipped" + echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc-skipped" else - echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc_test-failed" + echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc-failed" + if [ "$rc_test" -eq 124 ] ; then + rc_exit=125 + fi fi -$NFT list ruleset > "$NFT_TEST_TESTTMPDIR/ruleset-after" - -exit "$rc_test" +exit "$rc_exit" diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh index 8da0cc3d5702..a35337750ab7 100755 --- a/tests/shell/run-tests.sh +++ b/tests/shell/run-tests.sh @@ -30,7 +30,9 @@ usage() { echo "ENVIRONMENT VARIABLES:" echo " NFT= : Path to nft executable" echo " VERBOSE=*|y : Enable verbose output" - echo " DUMPGEN=*|y : Regenerate dump files" + echo " DUMPGEN=*|y : Regenerate dump files. Dump files are only recreated if the" + echo " test completes successfully and the \"dumps\" directory for the" + echo " test exits." echo " VALGRIND=*|y : Run \$NFT in valgrind" echo " KMEMLEAK=*|y : Check for kernel memleaks" echo " NFT_TEST_HAVE_REALROOT=*|y : To indicate whether the test has real root permissions." @@ -340,38 +342,25 @@ for testfile in "${TESTS[@]}" ; do export NFT_TEST_TESTTMPDIR msg_info "[EXECUTING] $testfile" - test_output="$(NFT="$NFT" DIFF=$DIFF $NFT_TEST_UNSHARE_CMD "$NFT_TEST_BASEDIR/helpers/test-wrapper.sh" "$testfile" 2>&1)" + test_output="$(NFT="$NFT" DIFF=$DIFF DUMPGEN="$DUMPGEN" $NFT_TEST_UNSHARE_CMD "$NFT_TEST_BASEDIR/helpers/test-wrapper.sh" "$testfile" 2>&1)" rc_got=$? echo -en "\033[1A\033[K" # clean the [EXECUTING] foobar line - if [ "$rc_got" -eq 0 ] ; then - # FIXME: this should move inside test-wrapper.sh. - # check nft dump only for positive tests - dumppath="$(dirname ${testfile})/dumps" - dumpfile="${dumppath}/$(basename ${testfile}).nft" - rc_spec=0 - if [ "$rc_got" -eq 0 ] && [ -f ${dumpfile} ]; then - test_output=$(${DIFF} -u ${dumpfile} <(cat "$NFT_TEST_TESTTMPDIR/ruleset-after") 2>&1) - rc_spec=$? - fi - - if [ "$rc_spec" -eq 0 ]; then - msg_info "[OK] $testfile" - [ "$VERBOSE" == "y" ] && [ ! -z "$test_output" ] && echo "$test_output" - ((ok++)) + if [ -s "$NFT_TEST_TESTTMPDIR/ruleset-diff" ] ; then + test_output="$test_output$(cat "$NFT_TEST_TESTTMPDIR/ruleset-diff")" + fi - if [ "$DUMPGEN" == "y" ] && [ "$rc_got" == 0 ] && [ ! -f "${dumpfile}" ]; then - mkdir -p "${dumppath}" - cat "$NFT_TEST_TESTTMPDIR/ruleset-after" > "${dumpfile}" - fi + if [ "$rc_got" -eq 0 ] ; then + ((ok++)) + msg_info "[OK] $testfile" + [ "$VERBOSE" == "y" ] && [ ! -z "$test_output" ] && echo "$test_output" + elif [ "$rc_got" -eq 124 ] ; then + ((failed++)) + if [ "$VERBOSE" == "y" ] ; then + msg_warn "[DUMP FAIL] $testfile: dump diff detected" + [ ! -z "$test_output" ] && echo "$test_output" else - ((failed++)) - if [ "$VERBOSE" == "y" ] ; then - msg_warn "[DUMP FAIL] $testfile: dump diff detected" - [ ! -z "$test_output" ] && echo "$test_output" - else - msg_warn "[DUMP FAIL] $testfile" - fi + msg_warn "[DUMP FAIL] $testfile" fi elif [ "$rc_got" -eq 77 ] ; then ((skipped++)) From patchwork Mon Sep 4 13:48:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Haller X-Patchwork-Id: 1829599 X-Patchwork-Delegate: pablo@netfilter.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=HlJLIZuN; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=2620:137:e000::1:20; helo=out1.vger.email; envelope-from=netfilter-devel-owner@vger.kernel.org; receiver=patchwork.ozlabs.org) Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by legolas.ozlabs.org (Postfix) with ESMTP id 4RfVTS0Cr0z1ynf for ; Mon, 4 Sep 2023 23:53:16 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229796AbjIDNxR (ORCPT ); Mon, 4 Sep 2023 09:53:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41122 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237273AbjIDNxR (ORCPT ); Mon, 4 Sep 2023 09:53:17 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B202CFB for ; Mon, 4 Sep 2023 06:51:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1693835514; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rZUcIJFh1dMQOGX4m5SqMIY22nIxuccBuELlRjbnH6g=; b=HlJLIZuNHTMASCy6KMAqoDsq/LuU7LRM81b/9sVQT/7fX15h05fvmpIFqCnwrcOoBbSY3/ f7XXjs/LnyUgsDlUqS/NO1LHVHKwh4OsX0Fj+HWpfDbmhZCjW3t5/OWuHZk86NsPGxZAcb CEbqKSPiKy4HTEuDJXQp2XhstJsFEuI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-205-Sc1ggCUoOLq6jszGjca0Fg-1; Mon, 04 Sep 2023 09:51:53 -0400 X-MC-Unique: Sc1ggCUoOLq6jszGjca0Fg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3E412939EC5 for ; Mon, 4 Sep 2023 13:51:53 +0000 (UTC) Received: from localhost.localdomain (unknown [10.39.192.26]) by smtp.corp.redhat.com (Postfix) with ESMTPS id B33661121314; Mon, 4 Sep 2023 13:51:52 +0000 (UTC) From: Thomas Haller To: NetFilter Cc: Thomas Haller Subject: [PATCH nft v3 09/11] tests/shell: rework printing of test results Date: Mon, 4 Sep 2023 15:48:11 +0200 Message-ID: <20230904135135.1568180-10-thaller@redhat.com> In-Reply-To: <20230904135135.1568180-1-thaller@redhat.com> References: <20230904135135.1568180-1-thaller@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: netfilter-devel@vger.kernel.org - "test-wrapper.sh" no longer will print the test output to its stdout. Instead, it only writes the testout.log file. - rework the loop "run-tests.sh" for printing the test results. It no longer captures the output of the test, as the wrapper is expected to be silent. Instead, they get the output from the result directory. The benefit is, that there is no duplication in what we print and the captured data in the result directory. The verbose mode is only for convenience, to safe looking at the test data. It's not essential otherwise. - also move the evaluation of the test result (and printing of the information) to a separate function. Later we want to run tests in parallel, so the steps need to be clearly separated. Signed-off-by: Thomas Haller --- tests/shell/helpers/test-wrapper.sh | 2 +- tests/shell/run-tests.sh | 116 +++++++++++++++++++--------- 2 files changed, 81 insertions(+), 37 deletions(-) diff --git a/tests/shell/helpers/test-wrapper.sh b/tests/shell/helpers/test-wrapper.sh index f9d759d0bb03..89ac9ff4ee15 100755 --- a/tests/shell/helpers/test-wrapper.sh +++ b/tests/shell/helpers/test-wrapper.sh @@ -10,7 +10,7 @@ TESTBASE="$(basename "$TEST")" TESTDIR="$(dirname "$TEST")" rc_test=0 -"$TEST" |& tee "$NFT_TEST_TESTTMPDIR/testout.log" || rc_test=$? +"$TEST" &> "$NFT_TEST_TESTTMPDIR/testout.log" || rc_test=$? $NFT list ruleset > "$NFT_TEST_TESTTMPDIR/ruleset-after" diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh index a35337750ab7..4f162b4514cc 100755 --- a/tests/shell/run-tests.sh +++ b/tests/shell/run-tests.sh @@ -1,16 +1,31 @@ #!/bin/bash +_msg() { + local level="$1" + shift + local msg + + msg="$level: $*" + if [ "$level" = E -o "$level" = W ] ; then + printf '%s\n' "$msg" >&2 + else + printf '%s\n' "$msg" + fi + if [ "$level" = E ] ; then + exit 1 + fi +} + msg_error() { - echo "E: $1 ..." >&2 - exit 1 + _msg E "$@" } msg_warn() { - echo "W: $1" >&2 + _msg W "$@" } msg_info() { - echo "I: $1" + _msg I "$@" } usage() { @@ -333,52 +348,81 @@ check_kmemleak() check_taint -for testfile in "${TESTS[@]}" ; do - read taint < /proc/sys/kernel/tainted - kernel_cleanup +print_test_header() { + local msglevel="$1" + local testfile="$2" + local status="$3" + local suffix="$4" + local text - # We also create and export a test-specific temporary directory. - NFT_TEST_TESTTMPDIR="$(mktemp -p "$NFT_TEST_TMPDIR" -d "test-${testfile//\//-}.XXXXXX")" - export NFT_TEST_TESTTMPDIR + if [ -n "$suffix" ] ; then + suffix=": $suffix" + fi + text="[$status]" + text="$(printf '%-12s' "$text")" + _msg "$msglevel" "$text $testfile$suffix" +} - msg_info "[EXECUTING] $testfile" - test_output="$(NFT="$NFT" DIFF=$DIFF DUMPGEN="$DUMPGEN" $NFT_TEST_UNSHARE_CMD "$NFT_TEST_BASEDIR/helpers/test-wrapper.sh" "$testfile" 2>&1)" - rc_got=$? - echo -en "\033[1A\033[K" # clean the [EXECUTING] foobar line +print_test_result() { + local NFT_TEST_TESTTMPDIR="$1" + local testfile="$2" + local rc_got="$3" + shift 3 - if [ -s "$NFT_TEST_TESTTMPDIR/ruleset-diff" ] ; then - test_output="$test_output$(cat "$NFT_TEST_TESTTMPDIR/ruleset-diff")" - fi + local result_msg_level="I" + local result_msg_status="OK" + local result_msg_suffix="" + local result_msg_files=( "$NFT_TEST_TESTTMPDIR/testout.log" "$NFT_TEST_TESTTMPDIR/ruleset-diff" ) if [ "$rc_got" -eq 0 ] ; then ((ok++)) - msg_info "[OK] $testfile" - [ "$VERBOSE" == "y" ] && [ ! -z "$test_output" ] && echo "$test_output" elif [ "$rc_got" -eq 124 ] ; then ((failed++)) - if [ "$VERBOSE" == "y" ] ; then - msg_warn "[DUMP FAIL] $testfile: dump diff detected" - [ ! -z "$test_output" ] && echo "$test_output" - else - msg_warn "[DUMP FAIL] $testfile" - fi + result_msg_level="W" + result_msg_status="DUMP FAIL" elif [ "$rc_got" -eq 77 ] ; then ((skipped++)) - if [ "$VERBOSE" == "y" ] ; then - msg_warn "[SKIPPED] $testfile" - [ ! -z "$test_output" ] && echo "$test_output" - else - msg_warn "[SKIPPED] $testfile" - fi + result_msg_level="I" + result_msg_status="SKIPPED" else ((failed++)) - if [ "$VERBOSE" == "y" ] ; then - msg_warn "[FAILED] $testfile: got $rc_got" - [ ! -z "$test_output" ] && echo "$test_output" - else - msg_warn "[FAILED] $testfile" + result_msg_level="W" + result_msg_status="FAILED" + result_msg_suffix="got $rc_got" + result_msg_files=( "$NFT_TEST_TESTTMPDIR/testout.log" ) + fi + + print_test_header "$result_msg_level" "$testfile" "$result_msg_status" "$result_msg_suffix" + + if [ "$VERBOSE" = "y" ] ; then + local f + + for f in "${result_msg_files[@]}"; do + if [ -s "$f" ] ; then + cat "$f" + fi + done + + if [ "$rc_got" -ne 0 ] ; then + msg_info "check \"$NFT_TEST_TESTTMPDIR\"" fi fi +} + +for testfile in "${TESTS[@]}" ; do + read taint < /proc/sys/kernel/tainted + kernel_cleanup + + # We also create and export a test-specific temporary directory. + NFT_TEST_TESTTMPDIR="$(mktemp -p "$NFT_TEST_TMPDIR" -d "test-${testfile//\//-}.XXXXXX")" + export NFT_TEST_TESTTMPDIR + + print_test_header I "$testfile" "EXECUTING" "" + NFT="$NFT" DIFF="$DIFF" DUMPGEN="$DUMPGEN" $NFT_TEST_UNSHARE_CMD "$NFT_TEST_BASEDIR/helpers/test-wrapper.sh" "$testfile" + rc_got=$? + echo -en "\033[1A\033[K" # clean the [EXECUTING] foobar line + + print_test_result "$NFT_TEST_TESTTMPDIR" "$testfile" "$rc_got" check_taint check_kmemleak From patchwork Mon Sep 4 13:48:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Haller X-Patchwork-Id: 1829603 X-Patchwork-Delegate: pablo@netfilter.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=EvA713du; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=2620:137:e000::1:20; helo=out1.vger.email; envelope-from=netfilter-devel-owner@vger.kernel.org; receiver=patchwork.ozlabs.org) Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by legolas.ozlabs.org (Postfix) with ESMTP id 4RfVTc2tnmz1ynq for ; Mon, 4 Sep 2023 23:53:24 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237887AbjIDNxZ (ORCPT ); Mon, 4 Sep 2023 09:53:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41138 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238749AbjIDNxY (ORCPT ); Mon, 4 Sep 2023 09:53:24 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 30B8DCFD for ; Mon, 4 Sep 2023 06:51:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1693835515; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2MOfwjXc8d3G3mcM8lpfpgm/7JrxN4VNhQ2asA8sZuM=; b=EvA713duOZxsUEwcMhsIGOTtYfayAOMWgAFpGQnY9l8MY05VuLlBLfHdVlE5qwX2ZGbzRd PL/LCEsQ+vxU0jDt/M+KJbQsbXv8zapvzN6TsReri3awsZqiDiusSrcsxYg+ikOP+yB5t2 cC3VjxtHUdVevL0I88eBtGdnApHohuc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-261-NPqwMjFuNwq9iSR-HczKXA-1; Mon, 04 Sep 2023 09:51:54 -0400 X-MC-Unique: NPqwMjFuNwq9iSR-HczKXA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 058DC939EC3 for ; Mon, 4 Sep 2023 13:51:54 +0000 (UTC) Received: from localhost.localdomain (unknown [10.39.192.26]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 7AD3D1121314; Mon, 4 Sep 2023 13:51:53 +0000 (UTC) From: Thomas Haller To: NetFilter Cc: Thomas Haller Subject: [PATCH nft v3 10/11] tests/shell: move taint check to "test-wrapper.sh" Date: Mon, 4 Sep 2023 15:48:12 +0200 Message-ID: <20230904135135.1568180-11-thaller@redhat.com> In-Reply-To: <20230904135135.1568180-1-thaller@redhat.com> References: <20230904135135.1568180-1-thaller@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: netfilter-devel@vger.kernel.org We will run tests in parallel. That means, we have multiple tests data and results in fly. That becomes simpler, if we move more result data to the test-wrapper and out of "run-tests.sh". Signed-off-by: Thomas Haller --- tests/shell/helpers/test-wrapper.sh | 18 +++++++++++++++--- tests/shell/run-tests.sh | 16 ++++------------ 2 files changed, 19 insertions(+), 15 deletions(-) diff --git a/tests/shell/helpers/test-wrapper.sh b/tests/shell/helpers/test-wrapper.sh index 89ac9ff4ee15..34d849e6e17e 100755 --- a/tests/shell/helpers/test-wrapper.sh +++ b/tests/shell/helpers/test-wrapper.sh @@ -9,11 +9,15 @@ TEST="$1" TESTBASE="$(basename "$TEST")" TESTDIR="$(dirname "$TEST")" +read tainted_before < /proc/sys/kernel/tainted + rc_test=0 "$TEST" &> "$NFT_TEST_TESTTMPDIR/testout.log" || rc_test=$? $NFT list ruleset > "$NFT_TEST_TESTTMPDIR/ruleset-after" +read tainted_after < /proc/sys/kernel/tainted + DUMPPATH="$TESTDIR/dumps" DUMPFILE="$DUMPPATH/$TESTBASE.nft" @@ -43,6 +47,10 @@ if [ "$rc_test" -ne 77 -a -f "$DUMPFILE" ] ; then fi fi +if [ "$tainted_before" != "$tainted_after" ]; then + echo "$tainted_after" > "$NFT_TEST_TESTTMPDIR/rc-failed-tainted" +fi + rc_exit="$rc_test" if [ -n "$rc_dump" ] && [ "$rc_dump" -ne 0 ] ; then echo "$DUMPFILE" > "$NFT_TEST_TESTTMPDIR/rc-failed-dump" @@ -51,13 +59,17 @@ if [ -n "$rc_dump" ] && [ "$rc_dump" -ne 0 ] ; then # Special exit code to indicate dump diff. rc_exit=124 fi -elif [ "$rc_test" -eq 0 ] ; then - echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc-ok" elif [ "$rc_test" -eq 77 ] ; then echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc-skipped" +elif [ "$rc_test" -eq 0 -a "$tainted_before" = "$tainted_after" ] ; then + echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc-ok" else echo "$rc_test" > "$NFT_TEST_TESTTMPDIR/rc-failed" - if [ "$rc_test" -eq 124 ] ; then + if [ "$rc_test" -eq 0 -a "$tainted_before" != "$tainted_after" ] ; then + # Special exit code to indicate tainted. + rc_exit=123 + elif [ "$rc_test" -eq 124 -o "$rc_test" -eq 123 ] ; then + # These exit codes are reserved rc_exit=125 fi fi diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh index 4f162b4514cc..1451d913bd3a 100755 --- a/tests/shell/run-tests.sh +++ b/tests/shell/run-tests.sh @@ -296,15 +296,6 @@ echo "" ok=0 skipped=0 failed=0 -taint=0 - -check_taint() -{ - read taint_now < /proc/sys/kernel/tainted - if [ $taint -ne $taint_now ] ; then - msg_warn "[FAILED] kernel is tainted: $taint -> $taint_now" - fi -} kmem_runs=0 kmemleak_found=0 @@ -346,7 +337,10 @@ check_kmemleak() fi } -check_taint +read kernel_tainted < /proc/sys/kernel/tainted +if [ "$kernel_tainted" -ne 0 ] ; then + msg_warn "[FAILED] kernel is tainted" +fi print_test_header() { local msglevel="$1" @@ -410,7 +404,6 @@ print_test_result() { } for testfile in "${TESTS[@]}" ; do - read taint < /proc/sys/kernel/tainted kernel_cleanup # We also create and export a test-specific temporary directory. @@ -424,7 +417,6 @@ for testfile in "${TESTS[@]}" ; do print_test_result "$NFT_TEST_TESTTMPDIR" "$testfile" "$rc_got" - check_taint check_kmemleak done From patchwork Mon Sep 4 13:48:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Haller X-Patchwork-Id: 1829604 X-Patchwork-Delegate: pablo@netfilter.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=M/pklE/D; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=2620:137:e000::1:20; helo=out1.vger.email; envelope-from=netfilter-devel-owner@vger.kernel.org; receiver=patchwork.ozlabs.org) Received: from out1.vger.email (out1.vger.email [IPv6:2620:137:e000::1:20]) by legolas.ozlabs.org (Postfix) with ESMTP id 4RfVTc4zKFz1yhH for ; Mon, 4 Sep 2023 23:53:24 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238749AbjIDNx0 (ORCPT ); Mon, 4 Sep 2023 09:53:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41152 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237621AbjIDNxY (ORCPT ); Mon, 4 Sep 2023 09:53:24 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0DBE3CFE for ; Mon, 4 Sep 2023 06:51:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1693835516; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AoSUWY9JQq/67guYpyEf28gaFCM3e2OFBijIclCDJHo=; b=M/pklE/DcX3AaYv+sjCvppWbGhMlrpULv1OTxsBE86xKAZrxQbDS1Cl7qIZkh+272GBTQI cqkMztptmOvanh4fUM5vlthsIEHWZUzuutyLOt1UojSraRgz0BH4ev3s2lmRef4TPNWuwB dp7hI0M5epii9no6nCzrfb3qnO+5f+Q= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-157-Fg3oBn71Oe-h_2w4KfWUag-1; Mon, 04 Sep 2023 09:51:55 -0400 X-MC-Unique: Fg3oBn71Oe-h_2w4KfWUag-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C241B1C0CCA6 for ; Mon, 4 Sep 2023 13:51:54 +0000 (UTC) Received: from localhost.localdomain (unknown [10.39.192.26]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 427B31121318; Mon, 4 Sep 2023 13:51:54 +0000 (UTC) From: Thomas Haller To: NetFilter Cc: Thomas Haller Subject: [PATCH nft v3 11/11] tests/shell: support running tests in parallel Date: Mon, 4 Sep 2023 15:48:13 +0200 Message-ID: <20230904135135.1568180-12-thaller@redhat.com> In-Reply-To: <20230904135135.1568180-1-thaller@redhat.com> References: <20230904135135.1568180-1-thaller@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: netfilter-devel@vger.kernel.org Add option to enable running jobs in parallel. The purpose is to speed up the run time of the tests. The global cleanup (removal of kernel modules) interferes with parallel jobs (or even with, unrelated jobs on the system). By setting NFT_TEST_JOBS= to a positive number, that cleanup is skipped. Signed-off-by: Thomas Haller --- tests/shell/run-tests.sh | 83 +++++++++++++++++++++++++++++++++------- 1 file changed, 69 insertions(+), 14 deletions(-) diff --git a/tests/shell/run-tests.sh b/tests/shell/run-tests.sh index 1451d913bd3a..160524e889ed 100755 --- a/tests/shell/run-tests.sh +++ b/tests/shell/run-tests.sh @@ -41,6 +41,7 @@ usage() { echo " -R|--without-realroot : sets NFT_TEST_HAVE_REALROOT=n" echo " -U|--no-unshare : sets NFT_TEST_NO_UNSHARE=y" echo " -k|--keep-logs : sets NFT_TEST_KEEP_LOGS=y" + echo " -j|--jobs : sets NFT_TEST_JOBS=12" echo echo "ENVIRONMENT VARIABLES:" echo " NFT= : Path to nft executable" @@ -69,6 +70,10 @@ usage() { echo " Set to empty to not unshare. You may want to export NFT_TEST_IS_UNSHARED=" echo " and NFT_TEST_HAVE_REALROOT= accordingly." echo " NFT_TEST_KEEP_LOGS=*|y: Keep the temp directory. On success, it will be deleted by default." + echo " NFT_TEST_JOBS=: by default, run test sequentially. Set to an integer > 1 to" + echo " run jobs in parallel. Leaving this unset or at zero means to run jobs sequentially" + echo " and perform global cleanups between tests (remove kernel modules). Setting this" + echo " to a positive number (including \"1\") means to disable such global cleanups." } NFT_TEST_BASEDIR="$(dirname "$0")" @@ -83,6 +88,7 @@ KMEMLEAK="$KMEMLEAK" NFT_TEST_KEEP_LOGS="$NFT_TEST_KEEP_LOGS" NFT_TEST_HAVE_REALROOT="$NFT_TEST_HAVE_REALROOT" NFT_TEST_NO_UNSHARE="$NFT_TEST_NO_UNSHARE" +NFT_TEST_JOBS="$NFT_TEST_JOBS" DO_LIST_TESTS= TESTS=() @@ -119,6 +125,9 @@ while [ $# -gt 0 ] ; do -U|--no-unshare) NFT_TEST_NO_UNSHARE=y ;; + -j|--jobs) + NFT_TEST_JOBS=12 + ;; --) TESTS+=( "$@" ) shift $# @@ -132,6 +141,11 @@ while [ $# -gt 0 ] ; do esac done +# normalize the jobs number to be an integer. +case "$NFT_TEST_JOBS" in + ''|*[!0-9]*) NFT_TEST_JOBS=0 ;; +esac + find_tests() { find "$1" -type f -executable | sort } @@ -199,13 +213,15 @@ ${NFT} > /dev/null 2>&1 ret=$? if [ ${ret} -eq 126 ] || [ ${ret} -eq 127 ]; then msg_error "cannot execute nft command: ${NFT}" -else - msg_info "using nft command: ${NFT}" fi +msg_info "using nft command: ${NFT}" +msg_info "parallel job mode: NFT_TEST_JOBS=$NFT_TEST_JOBS" -MODPROBE="$(which modprobe)" -if [ ! -x "$MODPROBE" ] ; then - msg_error "no modprobe binary found" +if [ "$NFT_TEST_JOBS" -eq 0 ] ; then + MODPROBE="$(which modprobe)" + if [ ! -x "$MODPROBE" ] ; then + msg_error "no modprobe binary found" + fi fi DIFF="$(which diff)" @@ -231,6 +247,11 @@ export NFT_TEST_TMPDIR kernel_cleanup() { + if [ "$NFT_TEST_JOBS" -ne 0 ] ; then + # When we run jobs in parallel (even with only one "parallel" + # job via `NFT_TEST_JOBS=1`), we skip such global cleanups. + return + fi if [ "$NFT_TEST_IS_UNSHARED" != y ] ; then $NFT flush ruleset fi @@ -403,23 +424,57 @@ print_test_result() { fi } -for testfile in "${TESTS[@]}" ; do - kernel_cleanup +declare -A JOBS_TEMPDIR +declare -A JOBS_PIDLIST - # We also create and export a test-specific temporary directory. - NFT_TEST_TESTTMPDIR="$(mktemp -p "$NFT_TEST_TMPDIR" -d "test-${testfile//\//-}.XXXXXX")" - export NFT_TEST_TESTTMPDIR +job_start() { + local testfile="$1" - print_test_header I "$testfile" "EXECUTING" "" + if [ "$NFT_TEST_JOBS" -le 1 ] ; then + print_test_header I "$testfile" "EXECUTING" "" + fi + + NFT_TEST_TESTTMPDIR="${JOBS_TEMPDIR["$testfile"]}" \ NFT="$NFT" DIFF="$DIFF" DUMPGEN="$DUMPGEN" $NFT_TEST_UNSHARE_CMD "$NFT_TEST_BASEDIR/helpers/test-wrapper.sh" "$testfile" rc_got=$? - echo -en "\033[1A\033[K" # clean the [EXECUTING] foobar line - print_test_result "$NFT_TEST_TESTTMPDIR" "$testfile" "$rc_got" + if [ "$NFT_TEST_JOBS" -le 1 ] ; then + echo -en "\033[1A\033[K" # clean the [EXECUTING] foobar line + fi + + return "$rc_got" +} + +job_wait() +{ + local num_jobs="$1" + + while [ "$JOBS_N_RUNNING" -gt 0 -a "$JOBS_N_RUNNING" -ge "$num_jobs" ] ; do + wait -n -p JOBCOMPLETED + rc_got="$?" + testfile2="${JOBS_PIDLIST[$JOBCOMPLETED]}" + print_test_result "${JOBS_TEMPDIR["$testfile2"]}" "$testfile2" "$rc_got" + ((JOBS_N_RUNNING--)) + check_kmemleak + done +} + +JOBS_N_RUNNING=0 +for testfile in "${TESTS[@]}" ; do + kernel_cleanup + + job_wait "$NFT_TEST_JOBS" - check_kmemleak + NFT_TEST_TESTTMPDIR="$(mktemp -p "$NFT_TEST_TMPDIR" -d "test-${testfile//\//-}.XXXXXX")" + JOBS_TEMPDIR["$testfile"]="$NFT_TEST_TESTTMPDIR" + + job_start "$testfile" & + JOBS_PIDLIST[$!]="$testfile" + ((JOBS_N_RUNNING++)) done +job_wait 0 + echo "" # kmemleak may report suspected leaks