From patchwork Tue Aug 14 04:56:55 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 957353 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 41qL1K2FQqz9s8T for ; Tue, 14 Aug 2018 14:57:09 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=popple.id.au Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 41qL1K0Wd9zF0dS for ; Tue, 14 Aug 2018 14:57:09 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=popple.id.au X-Original-To: pdbg@lists.ozlabs.org Delivered-To: pdbg@lists.ozlabs.org Received: from ozlabs.org (bilbo.ozlabs.org [203.11.71.1]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 41qL1B42THzF0Qs for ; Tue, 14 Aug 2018 14:57:02 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=popple.id.au Received: from authenticated.ozlabs.org (localhost [127.0.0.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPSA id 41qL1B2XZNz9rxx; Tue, 14 Aug 2018 14:57:02 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=popple.id.au From: Alistair Popple To: pdbg@lists.ozlabs.org, amitay@ozlabs.org Date: Tue, 14 Aug 2018 14:56:55 +1000 Message-Id: <20180814045655.20519-1-alistair@popple.id.au> X-Mailer: git-send-email 2.11.0 Subject: [Pdbg] [RFC] Add hw tests X-BeenThere: pdbg@lists.ozlabs.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "mailing list for https://github.com/open-power/pdbg development" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: pdbg-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Pdbg" --- Amitay, I was wondering if you had any thoughts of how we could better integrate the below prototype of some HW testing? What is here currently works, but has a few issues: 1. It only does a basic check to see if the host is powered on and skips the rest of the tests if it isn't rather than attempt to get the host into the right state. 2. You have to manually copy pdbg to the BMC. It would be good if we could automate deployment of pdbg to the BMC (eg. rsync ) 3. It creates a new connection for each command which is slow, so it would be nice if we could avoid that. 4. Environment setup (such as BMC host/user/pass) is hardcoded in the test_hw_bmc.sh script. 5. Testing of stdout and stderr. Using "2>&1" doesn't give consistent results when there is output on both stdout and stderr (for example progress bars) so it would be nice if we could add a "required_stderr" check seperate from the "required_output" check. 6. bmc_driver.sh is basically a copy of the existing driver.sh so there is a lot of code duplication. I had to update the driver to use regexes for matching command output (which depends on bash) as different platforms (eg. P8 vs. P9) will return different outputs for specific values. In general I like the simple approach here (run command, match output), so I would like to extend what we already have. Thanks. - Alistair Makefile.am | 3 +- tests/bmc_driver.sh | 247 +++++++++++++++++++++++++++++++++++++++++++++++++++ tests/test_hw_bmc.sh | 59 ++++++++++++ 3 files changed, 308 insertions(+), 1 deletion(-) create mode 100644 tests/bmc_driver.sh create mode 100755 tests/test_hw_bmc.sh diff --git a/Makefile.am b/Makefile.am index 571660d..245e593 100644 --- a/Makefile.am +++ b/Makefile.am @@ -7,7 +7,8 @@ bin_PROGRAMS = pdbg check_PROGRAMS = optcmd_test PDBG_TESTS = \ - tests/test_selection.sh + tests/test_selection.sh \ + tests/test_hw_bmc.sh TESTS = optcmd_test $(PDBG_TESTS) diff --git a/tests/bmc_driver.sh b/tests/bmc_driver.sh new file mode 100644 index 0000000..f7566f2 --- /dev/null +++ b/tests/bmc_driver.sh @@ -0,0 +1,247 @@ +#!/bin/bash + +# +# Simple test driver for shell based testsuite +# +# Each testsuite is a shell script, which sources this driver file. +# Following functions can be used to define testsuite and tests. +# +# test_group +# +# Define a test suite. This function should be called before calling any +# other functions +# +# test_result <--|output> +# +# Define the exit code and the output for the test. If there is no output +# from the commands, then '--' can be specified to denote empty output. +# Multi-line output can be added using a here document. +# +# test_run [] +# +# Define the command to execute for the test. +# +# test_skip +# +# This can be called before test_run, to skip the test. This is useful to +# write tests which are dependent on the environment (e.g. architecture). +# +# +# Example: +# +# test_group "my group of tests" +# +# test_result 0 output1 +# test_run command1 arguments1 +# +# test_result 1 -- +# test_run command2 arguments2 +# +# test_result 0 output3 +# if [ $condition ] ; then +# test_skip +# fi +# test_run command3 arguments3 +# + +set -u + +TESTDIR=$(dirname "$0") +if [ "$TESTDIR" = "." ] ; then + TESTDIR=$(cd "$TESTDIR"; pwd) +fi +SRCDIR=$(dirname "$TESTDIR") +PATH="$SRCDIR":$PATH + +test_name=${TEST_NAME:-$0} +test_logfile=${TEST_LOG:-} +test_trsfile=${TEST_TRS:-} +test_color=${TEST_COLOR:-yes} + +red= grn= lgn= blu= mgn= std= +if [ $test_color = yes ] ; then + red='' # Red. + grn='' # Green. + lgn='' # Light green. + blu='' # Blue. + mgn='' # Magenta. + std='' # No color. +fi + +test_started=0 +test_skipped=0 +test_defined=0 + +count_total=0 +count_skipped=0 +count_failed=0 + +trap 'test_end' 0 + +test_group () +{ + test_name=${1:-$test_name} + test_started=1 + + echo "-- $test_name" +} + +test_end () +{ + trap 0 + if [ $count_total -eq 0 ] ; then + status=99 + elif [ $count_failed -gt 0 ] ; then + status=1 + elif [ $count_skipped -eq $count_total ] ; then + status=77 + else + status=0 + fi + + exit $status +} + +test_error () +{ + echo "$@" + exit 99 +} + +test_log () +{ + if [ -z "$test_logfile" ] ; then + echo "$@" + else + echo "$@" >> "$test_logfile" + fi +} + +test_trs () +{ + if [ -n "$test_trsfile" ] ; then + echo "$@" >> "$test_trsfile" + fi +} + +test_output () +{ + rc=${1:-1} + required_rc=${2:-0} + output_mismatch=${3:-0} + + if [ $required_rc -eq 0 ] ; then + expect_failure=no + else + expect_failure=yes + fi + + case $rc:$expect_failure:$output_mismatch in + 0:*:1) col=$red res=FAIL ;; + 0:yes:*) col=$red res=XPASS ;; + 0:*:*) col=$grn res=PASS ;; + 77:*:*) col=$blu res=SKIP ;; + *:*:1) col=$red res=FAIL ;; + *:yes:*) col=$lgn res=XFAIL ;; + *:*:*) col=$red res=FAIL ;; + esac + + if [ -n "$test_logfile" ] ; then + test_log "${res} ${test_cmd} (exit status: ${rc})" + fi + test_trs ":test-result: ${res}" + + if [ -n "$test_trsfile" ] ; then + indent=" " + else + indent="" + fi + + echo "${indent}${col}${res}${std}: ${test_cmd}" + + count_total=$(( count_total + 1 )) + if [ $res = "SKIP" ] ; then + count_skipped=$(( count_skipped + 1 )) + elif [ $res = "XPASS" -o $res = "FAIL" ] ; then + count_failed=$(( count_failed + 1 )) + fi +} + +#--------------------------------------------------------------------- +# Public functions +#--------------------------------------------------------------------- + +test_result () +{ + if [ $test_started -eq 0 ] ; then + test_error "ERROR: missing call to test_group" + fi + + required_rc="${1:-0}" + if [ $# -eq 2 ] ; then + if [ "$2" = "--" ] ; then + required_output="" + else + required_output="$2" + fi + else + if ! tty -s ; then + required_output=$(cat) + else + required_output="" + fi + fi + + test_defined=1 +} + +test_skip () +{ + if [ $test_started -eq 0 ] ; then + test_error "ERROR: missing call to test_group" + fi + + test_skipped=1 +} + +test_run () +{ + output_mismatch=0 + + if [ $test_started -eq 0 ] ; then + test_error "ERROR: missing call to test_group" + fi + + if [ $test_defined -eq 0 ] ; then + test_error "ERROR: missing call to test_result" + fi + + test_cmd="$@" + + if [ $test_skipped -eq 1 ] ; then + test_output 77 + test_skipped=0 + test_defined=0 + return + fi + + output=$(sshpass -p $BMC_PASS ssh $BMC_USER@$BMC_HOST LD_LIBRARY_PATH=$PDBG_PATH "$@") + rc=$? + + if [ $rc -ne $required_rc ] ; then + test_log "expected rc: $required_rc" + test_log "output rc: $rc" + fi + + if [[ !("$output" =~ $required_output) ]] ; then + test_log "expected:" + test_log "$required_output" + test_log "output:" + test_log "$output" + output_mismatch=1 + fi + + test_output $rc $required_rc $output_mismatch + test_skipped=0 + test_defined=0 +} diff --git a/tests/test_hw_bmc.sh b/tests/test_hw_bmc.sh new file mode 100755 index 0000000..127cba2 --- /dev/null +++ b/tests/test_hw_bmc.sh @@ -0,0 +1,59 @@ +#!/bin/bash + +PDBG_PATH=/tmp/pdbg +BMC_USER= +BMC_PASS= +BMC_HOST= + +. $(dirname "$0")/bmc_driver.sh + +test_group "BMC HW tests" + +hw_state=0 + +do_skip () +{ + if [ "$hw_state" -ne 1 ] ; then + test_skip + fi +} + +test_result 0 <&1 + +test_result 0 <&1 + +test_result 0 </dev/null + +test_result 0 </dev/null