diff mbox series

[3/3] contrib/regression/btest-gcc.sh: Optionally handle XPASS.

Message ID 20231123170926.D33BA20432@pchp3.se.axis.com
State New
Headers show
Series A few contrib/regression/btest-gcc.sh updates. | expand

Commit Message

Hans-Peter Nilsson Nov. 23, 2023, 5:09 p.m. UTC 
Somewhat trivial, still tested on several runs (for
cris-elf): two starting from the same state, with/without
--handle-xpass-as-fail; the one "without" showing no change
in state compared to an unpatched baseline (with the same
input-state), and the one with --handle-xpass-as-fail some
XPASSing tests I'd noticed now correctly showed up as
regressions.  In another, separate run, with the same input
state but one of those XPASSing tests removed from "passes"
in the input-state, it correctly showed up as a
(non-regression) new FAIL.

-- >8 --
Tests with keys that match both PASS, FAIL (or now
optionally XPASS), count as fail.  XPASSes were previously
ignored.  Handling them as FAIL seems the most useful
alternative, but not counting XPASSes may be deliberate.
It's also a matter of compatibility, so make it optional.

Attempts to use --handle-xpass-as-fail was previously
flagged as a usage error.  If you pass it now, on state with
previous mixed XPASS and PASS results but doesn't change in
this run, the XPASS is discovered as a (new) regression.
For new XPASSing tests, it's handled as a new FAIL.

	* btest-gcc.sh (--handle-xpass-as-fail): New option.
---
 contrib/regression/btest-gcc.sh | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)
diff mbox series

Patch

diff --git a/contrib/regression/btest-gcc.sh b/contrib/regression/btest-gcc.sh
index 3c031e93709b..684019f715f1 100755
--- a/contrib/regression/btest-gcc.sh
+++ b/contrib/regression/btest-gcc.sh
@@ -22,17 +22,22 @@ 
 
 add_passes_despite_regression=0
 dashj=''
+handle_xpass_as_fail=false
 
 # <options> can be
 # --add-passes-despite-regression:
 #  Add new "PASSes" despite there being some regressions.
 # -j<n>:
 #  Pass '-j<n>' to make.
+# --handle-xpass-as-fail:
+#  Count XPASS as a FAIL (default ignored).
 
 while : ; do
   case "$1" in
    --add-passes-despite-regression)
     add_passes_despite_regression=1;;
+   --handle-xpass-as-fail)
+    handle_xpass_as_fail=true;;
    -j*)
     dashj=$1;;
    -*) echo "Invalid option: $1"; exit 2;;
@@ -203,7 +208,11 @@  done
 # Work out what failed
 for LOG in $TESTLOGS ; do
   L=`basename $LOG`
-  awk '/^FAIL: / { print "'$L'",$2; }' $LOG || exit 1
+  if $handle_xpass_as_fail ; then
+   awk '/^(FAIL|XPASS): / { print "'$L'",$2; }' $LOG || exit 1
+  else
+   awk '/^FAIL: / { print "'$L'",$2; }' $LOG || exit 1
+  fi
 done | sort | uniq > $FAILED || exit 1
 comm -12 $FAILED $PASSES >> $REGRESS || exit 1
 NUMREGRESS=`wc -l < $REGRESS | tr -d ' '`