From patchwork Sat Sep 28 13:03:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Aaron Conole X-Patchwork-Id: 1990486 X-Patchwork-Delegate: echaudro@redhat.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=ewBUvNkQ; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4XG6wF1lqyz1xt9 for ; Sat, 28 Sep 2024 23:03:41 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 9282E405AB; Sat, 28 Sep 2024 13:03:39 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id Mk-fLwEcuacD; Sat, 28 Sep 2024 13:03:38 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.9.56; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 065E6403A6 Authentication-Results: smtp4.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=ewBUvNkQ Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTPS id 065E6403A6; Sat, 28 Sep 2024 13:03:38 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id C6D5BC002B; Sat, 28 Sep 2024 13:03:37 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id C1C60C002A for ; Sat, 28 Sep 2024 13:03:36 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id B01E84018D for ; Sat, 28 Sep 2024 13:03:36 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id u5z5fwAh8EYL for ; Sat, 28 Sep 2024 13:03:35 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=170.10.129.124; helo=us-smtp-delivery-124.mimecast.com; envelope-from=aconole@redhat.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp2.osuosl.org 3846C4017B Authentication-Results: smtp2.osuosl.org; dmarc=pass (p=none dis=none) header.from=redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 3846C4017B Authentication-Results: smtp2.osuosl.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=ewBUvNkQ Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id 3846C4017B for ; Sat, 28 Sep 2024 13:03:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1727528613; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Let6BspCdhtaRL/ffvs1yknnDp/qDYZHlKgiqCvJO+c=; b=ewBUvNkQr/MpCf9l1EGOkBbB9hx7RmpKO/yjxb7b7FHW/HEiDHnMzshaq/WzTIRVL5IlBO RxeLAPTVAjnkw91X/OkMMMJ7WDedxkjuL7OzjxK5w+9TtCGLKrfqHU7sBkh0jswje6fDy8 jvGIdA5U5K0lq2S/l8bcvGtX+Wwrmno= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-527-d5nbdJJtN5iEX7fWwYF3oQ-1; Sat, 28 Sep 2024 09:03:31 -0400 X-MC-Unique: d5nbdJJtN5iEX7fWwYF3oQ-1 Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (unknown [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id D59E3196C438; Sat, 28 Sep 2024 13:03:30 +0000 (UTC) Received: from RHTRH0061144.redhat.com (unknown [10.22.16.189]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id C6C611955DCB; Sat, 28 Sep 2024 13:03:29 +0000 (UTC) From: Aaron Conole To: dev@openvswitch.org Date: Sat, 28 Sep 2024 09:03:28 -0400 Message-ID: <20240928130328.31999-1-aconole@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH] testing: Document the 0-day robot support. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Dumitru Ceara Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" The 0-day robot has been testing patches for 6 years, and we've had support for other labs to integrate for 3. However, this isn't well documented, and has made it difficult for others to know how they can contribute to testing. To that end, this patch introduces some documentation for the 0-day robot and how to integrate into the patch reporting process. Signed-off-by: Aaron Conole --- .../contributing/submitting-patches.rst | 3 +- Documentation/topics/testing.rst | 105 ++++++++++++++++++ 2 files changed, 107 insertions(+), 1 deletion(-) diff --git a/Documentation/internals/contributing/submitting-patches.rst b/Documentation/internals/contributing/submitting-patches.rst index 8a8bc11b0a..c01ac7bbdc 100644 --- a/Documentation/internals/contributing/submitting-patches.rst +++ b/Documentation/internals/contributing/submitting-patches.rst @@ -70,7 +70,8 @@ Testing is also important: If you are using GitHub, then you may utilize the GitHub Actions CI build systems. They will run some of the above tests automatically -when you push changes to your repository. +when you push changes to your repository. See the "Continuous Integration" +section in :doc:`/topics/testing` for details on continuous integration. Email Subject ------------- diff --git a/Documentation/topics/testing.rst b/Documentation/topics/testing.rst index dcf10a4db2..49ee7d7ffa 100644 --- a/Documentation/topics/testing.rst +++ b/Documentation/topics/testing.rst @@ -545,3 +545,108 @@ Once you are done with experimenting you can tear down setup with:: Sometimes deployment of Proof of Concept may fail, if, for example, VMs don't have network reachability to the Internet. + + +Continuous Integration +---------------------- + +The Open vSwitch project can make use of multiple public and hosted +CI services to help developers ensure their patches don't introduce +additional regressions. Each service requires different forms of +configuration, and for the supported services the configuration +file(s) to automatically build Open vSwitch with various build +configurations and run the testsuites is/are provided in the +repostiory. For example, the GitHub Actions builds will be performed +with gcc, clang, sparse, including DPDK, etc. with the -Werror +compiler flag included. Therefore, the build will fail if a new +warning gets introduced by a change. + +Each ci system uses a different method of enablement, but most are available +from the GitHub settings page. By default, Open vSwitch includes a GitHub +actions running configuration, and this will automatically email the repository +owner. + +Currently, Open vSwitch project enables the following public CI services along +with the appropriate configuration settings:: + + - AppVeyor: appveyor.yml + - Cirrus-CI: .cirrus.yml + - GitHub Actions: .github/workflows + +GitHub Actions is available without any additional configuration. + +Additionally, as each patch is posted to the mailing list, the public CI +machinery will run additional tests on the patches and report results. + +Public report / Private lab hybrid testing +------------------------------------------ + +The Open vSwitch project makes use of the ozlabs patchwork instance +to track patch status and management. This patch tracking tool +provides information to maintainers on the state of patches proposed +for Open vSwitch. The CI process for Open vSwitch makes use of the +checks feature of the ozlabs patchwork instance. These allow developers +and maintainers to see what tests have been run, and report pass / fail +criteria. + +In order to know that a patch or series is ready for testing, the +Open vSwitch project makes use of the "0-day Robot", which is a hosted +jenkins instance running the pw-ci_ scripts. These can monitor a +running patchwork instance for new patches and submit the patch details +to other build systems, like jenkins. + +Once a patch is tested, it would be good to report the results. To this +end, the Open vSwitch "0-day Robot" will accept emails sent to +ovs-build@openvswitch.org formatted in the correct way to be reflected +on this page. This allows any lab to contribute to the testing and +validation of patches. Note that the ovs-build list participation +requires subscribing the reporting email account to the list. + +To report a test status to a particular patch, send exactly one email to +the mailing formatted as such:: + + From: your email + To: ovs-build@openvswitch.org + Date: Mon, 28 Jun 2021 00:00:00 +0000 + Subject: |STATUS| pwPATCHID commit subject + + Test-Label: your-robot-or-test-name + Test-Status: STATUS + http://patchwork.ozlabs.org/api/patches/PATCHID/ + + _ONE LINE DESCRIPTION_ + + Addtional details + +In the above example, the STATUS is one of "success" "warning" "fail" depending +on the outcome of the test. This value is case-sensitive. PATCHID should be +the patch for which the test was executed, and ONE LINE DESCRIPTION should be a +simple one line result description, ex: "robot-test: success" + +It is strongly recommended that if the patch is reporting a failure case, one +of the patch authors is cc'd as well. Please try to keep the details of a +pass or fail to smallest amount that includes useful data. For example, if +a build log can be hosted in a publicly viewable way, then please include a +URL rather than the entire build log. If that isn't possible, try to parse +the test logs and include a small snippet with the relevent details. Always +also try to publish what the test is running - that can help developers to +recreate failures themselves. + +Example for success:: + + From: 0-Day Robot + To: ovs-build@openvswitch.org + Date: Fri, 25 Jun 2021 14:30:37 +0400 + Subject: |success| pw1497375 [ovs-dev] [PATCH v2] checkpatch: Ignore macro definitions of FOR_EACH + + Test-Label: github-robot + Test-Status: success + http://patchwork.ozlabs.org/api/patches/1497375/ + + _github build: passed_ + Build URL: https://github.com/ovsrobot/ovs/actions/runs/972188336 + +If possible, please also try to support the ``recheck-request`` feature that +the "0-day Robot" currently supports. + +.. _pw-ci: https://github.com/ovsrobot/pw-ci