diff mbox series

RFC PATCH: contrib/test_summary mode for submitting testsuite results to bunsen

Message ID 20240913201848.GC25474@redhat.com
State New
Headers show
Series RFC PATCH: contrib/test_summary mode for submitting testsuite results to bunsen | expand

Commit Message

Frank Ch. Eigler Sept. 13, 2024, 8:18 p.m. UTC 
Hi -

The gcc-testresults mailing list is a well-established place to plop
snippets of testsuite results.  It's an okay way to archive and
distribute overall counts, but it's not machine readable, and it's way
incomplete (lacks .log content, a lot of metadata, barely meaningfully
searchable) for trying to understand why something failed.

A few years ago, our team @ RH built the bunsen system to serve as a
structured repository for dejagnu (and other) testsuite types. [1] An
instance running here [2] has been collecting data from the sourceware
buildbots and individuals (hi iains!) for years.  It convers all the
toolchain pieces and then some, including gcc.  It features machine
readable indexing, cross-referenced test case browsing, regression
analysis, old data aging, and many other capabilities.  (It even has a
piece that extracts data from the gcc-testresults@ mailing list
archives, but there's too little there to bother. [4])

This patch attempts to make it easy for gcc developers who use the
contrib/test_summary script today to opt in to contributing their
results to a bunsen server (defaulting to the sourceware one [2]).

The prerequites are:
- git
- the bunsen t-upload-git-push script [3], feel free to grab just that
  into your $PATH
- sourceware account with bunsendb commit access [ask on
  admin-requests@sourceware.org]
- pretty much nothing else - specifically, gcc commit access is not
  necessary!

Here's how it looks when it's run after a random small gcc build/test:
.../contrib/test_summary -h
  [...]
 -b: instead of emailing, push test logs into a bunsen git repo
 -bg REPO: specify the bunsen git repo to override default
 -bt TAG: specify the bunsen git commit tag to override default

.../contrib/test_summary -b | env PATH=$BUNSEN/INST/bin:$PATH sh -x
+ echo master
+ echo basepoints/gcc-15-3524-ga523c2ba5862
+ echo a523c2ba58621c3630a1cd890d6db82879f92c90
+ echo git://gcc.gnu.org/git/gcc.git
+ find . -name '*.log' -o -name '*.sum' -o -name '.bunsen.*'
+ t-upload-git-push ssh://sourceware.org/git/bunsendb.git/ fche/gcc/x86_64-20240913-1516
ec57fb8ee928e341d1f0d1b09c1d571fb590bd2b refs/tags/fche/gcc/x86_64-20240913-1516

And here's what that dataset looks like on bunsen a few minutes later:

https://builder.sourceware.org/testrun/ec57fb8ee928e341d1f0d1b09c1d571fb590bd2b

If this is of any interest, I'd be glad to hack on this script further
to make it acceptable.


[1] https://sourceware.org/bunsen/
[2] https://builder.sourceware.org/testruns/
[3] https://sourceware.org/git/?p=bunsen.git;a=blob;f=bin/t-upload-git-push
[4] https://sourceware.org/git/?p=bunsen.git;a=blob;f=bin/t-sourceware-mails-import

Comments

Eric Gallager Sept. 15, 2024, 12:19 a.m. UTC | #1 
On Fri, Sep 13, 2024 at 4:19 PM Frank Ch. Eigler <fche@redhat.com> wrote:
>
> Hi -
>
> The gcc-testresults mailing list is a well-established place to plop
> snippets of testsuite results.  It's an okay way to archive and
> distribute overall counts, but it's not machine readable, and it's way
> incomplete (lacks .log content, a lot of metadata, barely meaningfully
> searchable) for trying to understand why something failed.
>
> A few years ago, our team @ RH built the bunsen system to serve as a
> structured repository for dejagnu (and other) testsuite types. [1] An
> instance running here [2] has been collecting data from the sourceware
> buildbots and individuals (hi iains!) for years.  It convers all the
> toolchain pieces and then some, including gcc.  It features machine
> readable indexing, cross-referenced test case browsing, regression
> analysis, old data aging, and many other capabilities.  (It even has a
> piece that extracts data from the gcc-testresults@ mailing list
> archives, but there's too little there to bother. [4])
>
> This patch attempts to make it easy for gcc developers who use the
> contrib/test_summary script today to opt in to contributing their
> results to a bunsen server (defaulting to the sourceware one [2]).
>
> The prerequites are:
> - git
> - the bunsen t-upload-git-push script [3], feel free to grab just that
>   into your $PATH
> - sourceware account with bunsendb commit access [ask on
>   admin-requests@sourceware.org]
> - pretty much nothing else - specifically, gcc commit access is not
>   necessary!
>
> Here's how it looks when it's run after a random small gcc build/test:
> .../contrib/test_summary -h
>   [...]
>  -b: instead of emailing, push test logs into a bunsen git repo
>  -bg REPO: specify the bunsen git repo to override default
>  -bt TAG: specify the bunsen git commit tag to override default
>
> .../contrib/test_summary -b | env PATH=$BUNSEN/INST/bin:$PATH sh -x
> + echo master
> + echo basepoints/gcc-15-3524-ga523c2ba5862
> + echo a523c2ba58621c3630a1cd890d6db82879f92c90
> + echo git://gcc.gnu.org/git/gcc.git
> + find . -name '*.log' -o -name '*.sum' -o -name '.bunsen.*'
> + t-upload-git-push ssh://sourceware.org/git/bunsendb.git/ fche/gcc/x86_64-20240913-1516
> ec57fb8ee928e341d1f0d1b09c1d571fb590bd2b refs/tags/fche/gcc/x86_64-20240913-1516
>
> And here's what that dataset looks like on bunsen a few minutes later:
>
> https://builder.sourceware.org/testrun/ec57fb8ee928e341d1f0d1b09c1d571fb590bd2b
>
> If this is of any interest, I'd be glad to hack on this script further
> to make it acceptable.
>

I pretty much only use `contrib/test_summary` via the
`mail-report.log` target in the top-level Makefile; maybe add a
`bunsen` target to the top-level Makefile, too, to simplify invoking
it?

>
> [1] https://sourceware.org/bunsen/
> [2] https://builder.sourceware.org/testruns/
> [3] https://sourceware.org/git/?p=bunsen.git;a=blob;f=bin/t-upload-git-push
> [4] https://sourceware.org/git/?p=bunsen.git;a=blob;f=bin/t-sourceware-mails-import
>
>
> diff --git a/contrib/ChangeLog b/contrib/ChangeLog
> index 9b36caf02bb1..def8dd8a8a73 100644
> --- a/contrib/ChangeLog
> +++ b/contrib/ChangeLog
> @@ -1,3 +1,9 @@
> +2024-09-13  Frank Ch. Eigler  <fche@redhat.com>
> +
> +       * test_summary: Add -b (bunsen) mode to report all test results
> +       into a https://sourceware.org/bunsen/ system instead of emailing
> +       extracts.
> +
>  2024-08-01  Thomas Schwinge  <tschwinge@baylibre.com>
>
>         * gcc_update (files_and_dependencies): Update for
> diff --git a/contrib/test_summary b/contrib/test_summary
> index 5760b053ec27..867ada4d6b81 100755
> --- a/contrib/test_summary
> +++ b/contrib/test_summary
> @@ -39,6 +39,9 @@ if test x"$1" = "x-h"; then
>       should be selected from the log files.
>   -f: force reports to be mailed; if omitted, only reports that differ
>       from the sent.* version are sent.
> + -b: instead of emailing, push test logs into a bunsen git repo
> + -bg REPO: specify the bunsen git repo to override default
> + -bt TAG: specify the bunsen git commit tag to override default
>  _EOF
>    exit 0
>  fi
> @@ -57,6 +60,9 @@ fi
>  : ${filesuffix=}; export filesuffix
>  : ${move=true}; export move
>  : ${forcemail=false}; export forcemail
> +: ${bunsen=false};
> +: ${bunsengit=ssh://sourceware.org/git/bunsendb.git/};
> +: ${bunsentag=`whoami`/gcc/`uname -m`-`date +%Y%m%d-%H%M`};
>  while true; do
>      case "$1" in
>        -o) filesuffix=.sent; move=false; : ${mailto=nobody}; shift;;
> @@ -64,10 +70,30 @@ while true; do
>        -p) prepend_logs=${prepend_logs+"$prepend_logs "}"$2"; shift 2;;
>        -i) append_logs=${append_logs+"$append_logs "}"$2"; shift 2;;
>        -m) mailto=$2; forcemail=true; shift 2;;
> +      -b) bunsen=true; shift;;
> +      -bg) bunsengit=$2; shift 2;;
> +      -bt) bunsentag=$2; shift 2;;
>        -f) unset mailto; forcemail=true; shift;;
>        *) break;;
>      esac
>  done
> +if [ "x$bunsen" = "xtrue" ]; then
> +    gitsrcdir=`dirname "$0"` # this script, contrib/test_summary
> +    gitsrcdir=`dirname "$gitsrcdir"` # and the parent directory
> +    if [ -d "$gitsrcdir/.git" ]; then # is this a git-hosted source tree?
> +        # gather basic build metadata for sourceware-buildbot-style .bunsen data
> +        gitbranch=`cd "$gitsrcdir"; git rev-parse --abbrev-ref HEAD`
> +        echo "echo '$gitbranch' > .bunsen.source.gitbranch &&"
> +        gitdescribe=`cd "$gitsrcdir"; git describe`
> +        echo "echo '$gitdescribe' > .bunsen.source.gitdescribe &&"
> +        gitname=`cd "$gitsrcdir"; git rev-parse HEAD`
> +        echo "echo '$gitname' > .bunsen.source.gitname &&"
> +        gitremote=`cd "$gitsrcdir"; git config --get remote.origin.url`
> +        echo "echo '$gitremote' > .bunsen.source.gitrepo &&"
> +    fi
> +    echo "(find . -name '*.log' -o -name '*.sum' -o -name '.bunsen.*' | t-upload-git-push '$bunsengit' '$bunsentag')"
> +    exit 0
> +fi
>  : ${mailto="\" address \""}; export mailto
>  files=`find . -name \*.sum$filesuffix -print | sort`
>  anyfile=false anychange=$forcemail &&
>
Frank Ch. Eigler Sept. 17, 2024, 3:13 p.m. UTC | #2 
Hi -

> [...]
> I pretty much only use `contrib/test_summary` via the
> `mail-report.log` target in the top-level Makefile; maybe add a
> `bunsen` target to the top-level Makefile, too, to simplify invoking
> it?

Something like this: ?


diff --git a/Makefile.in b/Makefile.in
index 966d60454960..41651e7592d2 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -2852,6 +2852,14 @@ mail-report-with-warnings.log: warning.log
 	chmod +x $@
 	echo If you really want to send e-mail, run ./$@ now
 
+bunsen-report.log:
+	if test x'$(BOOT_CFLAGS)' != x''; then \
+	    BOOT_CFLAGS='$(BOOT_CFLAGS)'; export BOOT_CFLAGS; \
+	fi; \
+	$(srcdir)/contrib/test_summary -b >$@
+	chmod +x $@
+	echo If you really want to send to bunsen, run ./$@ now
+
 # Local Vim config
 
 $(srcdir)/.local.vimrc:
diff --git a/Makefile.tpl b/Makefile.tpl
index da38dca697ad..e2c3f6bf6ca5 100644
--- a/Makefile.tpl
+++ b/Makefile.tpl
@@ -1034,6 +1034,14 @@ mail-report-with-warnings.log: warning.log
 	chmod +x $@
 	echo If you really want to send e-mail, run ./$@ now
 
+bunsen-report.log:
+	if test x'$(BOOT_CFLAGS)' != x''; then \
+	    BOOT_CFLAGS='$(BOOT_CFLAGS)'; export BOOT_CFLAGS; \
+	fi; \
+	$(srcdir)/contrib/test_summary -b >$@
+	chmod +x $@
+	echo If you really want to send to bunsen, run ./$@ now
+
 # Local Vim config
 
 $(srcdir)/.local.vimrc:
Eric Gallager Sept. 17, 2024, 4:52 p.m. UTC | #3 
On Tue, Sep 17, 2024 at 11:13 AM Frank Ch. Eigler <fche@redhat.com> wrote:
>
> Hi -
>
> > [...]
> > I pretty much only use `contrib/test_summary` via the
> > `mail-report.log` target in the top-level Makefile; maybe add a
> > `bunsen` target to the top-level Makefile, too, to simplify invoking
> > it?
>
> Something like this: ?
>

Yeah, something like that... I had no idea it'd be so easy to just
copy and paste the existing rule...

>
> diff --git a/Makefile.in b/Makefile.in
> index 966d60454960..41651e7592d2 100644
> --- a/Makefile.in
> +++ b/Makefile.in
> @@ -2852,6 +2852,14 @@ mail-report-with-warnings.log: warning.log
>         chmod +x $@
>         echo If you really want to send e-mail, run ./$@ now
>
> +bunsen-report.log:
> +       if test x'$(BOOT_CFLAGS)' != x''; then \
> +           BOOT_CFLAGS='$(BOOT_CFLAGS)'; export BOOT_CFLAGS; \
> +       fi; \
> +       $(srcdir)/contrib/test_summary -b >$@
> +       chmod +x $@
> +       echo If you really want to send to bunsen, run ./$@ now
> +
>  # Local Vim config
>
>  $(srcdir)/.local.vimrc:
> diff --git a/Makefile.tpl b/Makefile.tpl
> index da38dca697ad..e2c3f6bf6ca5 100644
> --- a/Makefile.tpl
> +++ b/Makefile.tpl
> @@ -1034,6 +1034,14 @@ mail-report-with-warnings.log: warning.log
>         chmod +x $@
>         echo If you really want to send e-mail, run ./$@ now
>
> +bunsen-report.log:
> +       if test x'$(BOOT_CFLAGS)' != x''; then \
> +           BOOT_CFLAGS='$(BOOT_CFLAGS)'; export BOOT_CFLAGS; \
> +       fi; \
> +       $(srcdir)/contrib/test_summary -b >$@
> +       chmod +x $@
> +       echo If you really want to send to bunsen, run ./$@ now
> +
>  # Local Vim config
>
>  $(srcdir)/.local.vimrc:
>
diff mbox series

Patch

diff --git a/contrib/ChangeLog b/contrib/ChangeLog
index 9b36caf02bb1..def8dd8a8a73 100644
--- a/contrib/ChangeLog
+++ b/contrib/ChangeLog
@@ -1,3 +1,9 @@ 
+2024-09-13  Frank Ch. Eigler  <fche@redhat.com>
+
+	* test_summary: Add -b (bunsen) mode to report all test results
+	into a https://sourceware.org/bunsen/ system instead of emailing
+	extracts.
+
 2024-08-01  Thomas Schwinge  <tschwinge@baylibre.com>
 
 	* gcc_update (files_and_dependencies): Update for
diff --git a/contrib/test_summary b/contrib/test_summary
index 5760b053ec27..867ada4d6b81 100755
--- a/contrib/test_summary
+++ b/contrib/test_summary
@@ -39,6 +39,9 @@  if test x"$1" = "x-h"; then
      should be selected from the log files.
  -f: force reports to be mailed; if omitted, only reports that differ
      from the sent.* version are sent.
+ -b: instead of emailing, push test logs into a bunsen git repo
+ -bg REPO: specify the bunsen git repo to override default
+ -bt TAG: specify the bunsen git commit tag to override default
 _EOF
   exit 0
 fi
@@ -57,6 +60,9 @@  fi
 : ${filesuffix=}; export filesuffix
 : ${move=true}; export move
 : ${forcemail=false}; export forcemail
+: ${bunsen=false};
+: ${bunsengit=ssh://sourceware.org/git/bunsendb.git/};
+: ${bunsentag=`whoami`/gcc/`uname -m`-`date +%Y%m%d-%H%M`};
 while true; do
     case "$1" in 
       -o) filesuffix=.sent; move=false; : ${mailto=nobody}; shift;;
@@ -64,10 +70,30 @@  while true; do
       -p) prepend_logs=${prepend_logs+"$prepend_logs "}"$2"; shift 2;;
       -i) append_logs=${append_logs+"$append_logs "}"$2"; shift 2;;
       -m) mailto=$2; forcemail=true; shift 2;;
+      -b) bunsen=true; shift;;
+      -bg) bunsengit=$2; shift 2;;
+      -bt) bunsentag=$2; shift 2;;
       -f) unset mailto; forcemail=true; shift;;
       *) break;;
     esac
 done
+if [ "x$bunsen" = "xtrue" ]; then
+    gitsrcdir=`dirname "$0"` # this script, contrib/test_summary
+    gitsrcdir=`dirname "$gitsrcdir"` # and the parent directory
+    if [ -d "$gitsrcdir/.git" ]; then # is this a git-hosted source tree?
+        # gather basic build metadata for sourceware-buildbot-style .bunsen data
+        gitbranch=`cd "$gitsrcdir"; git rev-parse --abbrev-ref HEAD`
+        echo "echo '$gitbranch' > .bunsen.source.gitbranch &&"
+        gitdescribe=`cd "$gitsrcdir"; git describe`
+        echo "echo '$gitdescribe' > .bunsen.source.gitdescribe &&"
+        gitname=`cd "$gitsrcdir"; git rev-parse HEAD`
+        echo "echo '$gitname' > .bunsen.source.gitname &&"
+        gitremote=`cd "$gitsrcdir"; git config --get remote.origin.url`
+        echo "echo '$gitremote' > .bunsen.source.gitrepo &&"
+    fi
+    echo "(find . -name '*.log' -o -name '*.sum' -o -name '.bunsen.*' | t-upload-git-push '$bunsengit' '$bunsentag')"
+    exit 0
+fi
 : ${mailto="\" address \""}; export mailto
 files=`find . -name \*.sum$filesuffix -print | sort`
 anyfile=false anychange=$forcemail &&