From patchwork Fri Sep 13 20:18:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Frank Ch. Eigler" X-Patchwork-Id: 1985445 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=bdPVw2y4; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gcc.gnu.org (client-ip=8.43.85.97; helo=server2.sourceware.org; envelope-from=gcc-patches-bounces~incoming=patchwork.ozlabs.org@gcc.gnu.org; receiver=patchwork.ozlabs.org) Received: from server2.sourceware.org (server2.sourceware.org [8.43.85.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4X55Hy4Nz6z1y1y for ; Sat, 14 Sep 2024 06:19:25 +1000 (AEST) Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 65E57385842C for ; Fri, 13 Sep 2024 20:19:22 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by sourceware.org (Postfix) with ESMTP id 5FC9F3858D28 for ; Fri, 13 Sep 2024 20:18:56 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 5FC9F3858D28 Authentication-Results: sourceware.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=redhat.com ARC-Filter: OpenARC Filter v1.0.0 sourceware.org 5FC9F3858D28 Authentication-Results: server2.sourceware.org; arc=none smtp.remote-ip=170.10.129.124 ARC-Seal: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1726258739; cv=none; b=v8XXTd1eyfmR9uM0cCifAmVsaY1lX9SviFUlT+s+NuolHKV1UEWv1goLjt0T/+ZWxsT4ztLUNwtuj1oKSRIwP5uXUJJ05VxHPtKSVozt0owLBzouSyvXRxSu1lt2PvzJebtSfTVWKL5zdLUIxP2w2Fo3GQZWEndl9yhvYIu2/LM= ARC-Message-Signature: i=1; a=rsa-sha256; d=sourceware.org; s=key; t=1726258739; c=relaxed/simple; bh=Ef88F3htwi+b7oNMlZlxPkQXqRLz1RZK48NzK0R3GJE=; h=DKIM-Signature:Date:From:To:Subject:Message-ID:MIME-Version; b=SOHt5uwvA6+cxP0hAy0QxFAM/Y9VRZgi0n1awvleZQ9/dGMeoQCdOaRdQsF9bsB1cZVplOL+hWI9/JW0fzcbKTxjjGqyYThdJ1M++UTggjMl/27YVl8pdRlF6cUM4aIEiVK6RZAXNrmvVXoRLb4yMh1jz5WOr6oqurCtb1Aunr0= ARC-Authentication-Results: i=1; server2.sourceware.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1726258736; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type; bh=vOo9WMYH97/4Vk52raU6vNp82MwQIj9lYT4O/mu6ai4=; b=bdPVw2y4wwZxSlyzVyqiy/L1iLGD3WulhMN+YEy0+RR0geFJnmnCYYRfjq1vehHnOuiQAj +Z2BRAu0iOBO01bjGL5x5kF6CYDsI6KTxYUPUuEyy6WCLKZ68NpFfM+qbX3wORs6IHCJ1f 459vp8SSf3hC+KPmR+LpmAXJV9IVqE8= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-91-D_ap1iyaPMqCUKDF8BnByQ-1; Fri, 13 Sep 2024 16:18:52 -0400 X-MC-Unique: D_ap1iyaPMqCUKDF8BnByQ-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id AE8C61956048; Fri, 13 Sep 2024 20:18:51 +0000 (UTC) Received: from redhat.com (unknown [10.22.8.6]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 5125119560AB; Fri, 13 Sep 2024 20:18:50 +0000 (UTC) Received: from fche by redhat.com with local (Exim 4.94.2) (envelope-from ) id 1spCki-0008Na-FK; Fri, 13 Sep 2024 16:18:48 -0400 Date: Fri, 13 Sep 2024 16:18:48 -0400 From: "Frank Ch. Eigler" To: gcc-patches@gcc.gnu.org Cc: bunsen@sourceware.org Subject: RFC PATCH: contrib/test_summary mode for submitting testsuite results to bunsen Message-ID: <20240913201848.GC25474@redhat.com> MIME-Version: 1.0 User-Agent: Mutt/1.12.0 (2019-05-25) X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Disposition: inline X-Spam-Status: No, score=-12.7 required=5.0 tests=BAYES_00, DKIMWL_WL_HIGH, DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, DKIM_VALID_EF, GIT_PATCH_0, KAM_SHORT, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL, SPF_HELO_NONE, SPF_NONE, TXREP autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces~incoming=patchwork.ozlabs.org@gcc.gnu.org Hi - The gcc-testresults mailing list is a well-established place to plop snippets of testsuite results. It's an okay way to archive and distribute overall counts, but it's not machine readable, and it's way incomplete (lacks .log content, a lot of metadata, barely meaningfully searchable) for trying to understand why something failed. A few years ago, our team @ RH built the bunsen system to serve as a structured repository for dejagnu (and other) testsuite types. [1] An instance running here [2] has been collecting data from the sourceware buildbots and individuals (hi iains!) for years. It convers all the toolchain pieces and then some, including gcc. It features machine readable indexing, cross-referenced test case browsing, regression analysis, old data aging, and many other capabilities. (It even has a piece that extracts data from the gcc-testresults@ mailing list archives, but there's too little there to bother. [4]) This patch attempts to make it easy for gcc developers who use the contrib/test_summary script today to opt in to contributing their results to a bunsen server (defaulting to the sourceware one [2]). The prerequites are: - git - the bunsen t-upload-git-push script [3], feel free to grab just that into your $PATH - sourceware account with bunsendb commit access [ask on admin-requests@sourceware.org] - pretty much nothing else - specifically, gcc commit access is not necessary! Here's how it looks when it's run after a random small gcc build/test: .../contrib/test_summary -h [...] -b: instead of emailing, push test logs into a bunsen git repo -bg REPO: specify the bunsen git repo to override default -bt TAG: specify the bunsen git commit tag to override default .../contrib/test_summary -b | env PATH=$BUNSEN/INST/bin:$PATH sh -x + echo master + echo basepoints/gcc-15-3524-ga523c2ba5862 + echo a523c2ba58621c3630a1cd890d6db82879f92c90 + echo git://gcc.gnu.org/git/gcc.git + find . -name '*.log' -o -name '*.sum' -o -name '.bunsen.*' + t-upload-git-push ssh://sourceware.org/git/bunsendb.git/ fche/gcc/x86_64-20240913-1516 ec57fb8ee928e341d1f0d1b09c1d571fb590bd2b refs/tags/fche/gcc/x86_64-20240913-1516 And here's what that dataset looks like on bunsen a few minutes later: https://builder.sourceware.org/testrun/ec57fb8ee928e341d1f0d1b09c1d571fb590bd2b If this is of any interest, I'd be glad to hack on this script further to make it acceptable. [1] https://sourceware.org/bunsen/ [2] https://builder.sourceware.org/testruns/ [3] https://sourceware.org/git/?p=bunsen.git;a=blob;f=bin/t-upload-git-push [4] https://sourceware.org/git/?p=bunsen.git;a=blob;f=bin/t-sourceware-mails-import diff --git a/contrib/ChangeLog b/contrib/ChangeLog index 9b36caf02bb1..def8dd8a8a73 100644 --- a/contrib/ChangeLog +++ b/contrib/ChangeLog @@ -1,3 +1,9 @@ +2024-09-13 Frank Ch. Eigler + + * test_summary: Add -b (bunsen) mode to report all test results + into a https://sourceware.org/bunsen/ system instead of emailing + extracts. + 2024-08-01 Thomas Schwinge * gcc_update (files_and_dependencies): Update for diff --git a/contrib/test_summary b/contrib/test_summary index 5760b053ec27..867ada4d6b81 100755 --- a/contrib/test_summary +++ b/contrib/test_summary @@ -39,6 +39,9 @@ if test x"$1" = "x-h"; then should be selected from the log files. -f: force reports to be mailed; if omitted, only reports that differ from the sent.* version are sent. + -b: instead of emailing, push test logs into a bunsen git repo + -bg REPO: specify the bunsen git repo to override default + -bt TAG: specify the bunsen git commit tag to override default _EOF exit 0 fi @@ -57,6 +60,9 @@ fi : ${filesuffix=}; export filesuffix : ${move=true}; export move : ${forcemail=false}; export forcemail +: ${bunsen=false}; +: ${bunsengit=ssh://sourceware.org/git/bunsendb.git/}; +: ${bunsentag=`whoami`/gcc/`uname -m`-`date +%Y%m%d-%H%M`}; while true; do case "$1" in -o) filesuffix=.sent; move=false; : ${mailto=nobody}; shift;; @@ -64,10 +70,30 @@ while true; do -p) prepend_logs=${prepend_logs+"$prepend_logs "}"$2"; shift 2;; -i) append_logs=${append_logs+"$append_logs "}"$2"; shift 2;; -m) mailto=$2; forcemail=true; shift 2;; + -b) bunsen=true; shift;; + -bg) bunsengit=$2; shift 2;; + -bt) bunsentag=$2; shift 2;; -f) unset mailto; forcemail=true; shift;; *) break;; esac done +if [ "x$bunsen" = "xtrue" ]; then + gitsrcdir=`dirname "$0"` # this script, contrib/test_summary + gitsrcdir=`dirname "$gitsrcdir"` # and the parent directory + if [ -d "$gitsrcdir/.git" ]; then # is this a git-hosted source tree? + # gather basic build metadata for sourceware-buildbot-style .bunsen data + gitbranch=`cd "$gitsrcdir"; git rev-parse --abbrev-ref HEAD` + echo "echo '$gitbranch' > .bunsen.source.gitbranch &&" + gitdescribe=`cd "$gitsrcdir"; git describe` + echo "echo '$gitdescribe' > .bunsen.source.gitdescribe &&" + gitname=`cd "$gitsrcdir"; git rev-parse HEAD` + echo "echo '$gitname' > .bunsen.source.gitname &&" + gitremote=`cd "$gitsrcdir"; git config --get remote.origin.url` + echo "echo '$gitremote' > .bunsen.source.gitrepo &&" + fi + echo "(find . -name '*.log' -o -name '*.sum' -o -name '.bunsen.*' | t-upload-git-push '$bunsengit' '$bunsentag')" + exit 0 +fi : ${mailto="\" address \""}; export mailto files=`find . -name \*.sum$filesuffix -print | sort` anyfile=false anychange=$forcemail &&