From patchwork Mon Mar 2 14:50:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1247682 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=b7CW+/vr; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48WNP13Zf2z9sPK for ; Tue, 3 Mar 2020 01:50:45 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 54F7D20497; Mon, 2 Mar 2020 14:50:41 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Re41yhXVWBYj; Mon, 2 Mar 2020 14:50:39 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 18D362048C; Mon, 2 Mar 2020 14:50:39 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 892BF1BF39F for ; Mon, 2 Mar 2020 14:50:38 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 8590985692 for ; Mon, 2 Mar 2020 14:50:38 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YnBcrWhQ+VSq for ; Mon, 2 Mar 2020 14:50:37 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f44.google.com (mail-wr1-f44.google.com [209.85.221.44]) by whitealder.osuosl.org (Postfix) with ESMTPS id EE3558357B for ; Mon, 2 Mar 2020 14:50:36 +0000 (UTC) Received: by mail-wr1-f44.google.com with SMTP id z11so4193831wro.9 for ; Mon, 02 Mar 2020 06:50:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gRpLr6aq3bOUfLf2o87SNd25g3iaxS+vMPfkYbg4WS0=; b=b7CW+/vrKhIRcmXXq/drYP78g//QwZcff3stgcbGLNCMHIlDtP5jMXrM6rrHOHU+3Y YypUlhGiWYLljDoqhf1/Y522wsMx9UEFNMryCTTQpj/DHLHOLTCQHjLcgBKHiC+D6/nh JJqCUAbwEsyrcEETq5Ms2aWWfkdpuY8ldtq2OypUdeLqfXF6Tkt4N3GII/0hAe6eDb7M g+v6WNf5f1cb0n+WBz8AZZVZ6uPnvau3mKNPm/4n+NHER7hJfKuckMAuRt5ALx6RB84o /42SwNySQYEbnjUufgxGDzwvjLPCbZnnBlIsUgfh4iQhGCoGjVegEki4o9vA/TwIli88 jNVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gRpLr6aq3bOUfLf2o87SNd25g3iaxS+vMPfkYbg4WS0=; b=PqlbU5xlM89pOfRisIBg0LWQEJAifjcd4tFKzOguVW5gQcvl/HrIqFJmY/wBo/9J6u sh0GFx/G/sVYWUNkOOoVCSH3EY6qld1A9M+9FOd50d7JUOC3wLTUEirjNqbaPxUJTEKj cv9I4c7AIxhyyRLsuGi8oOR9t8B23smL7D1KkR6P3yjPCBLFPbXRry7KFnvRfKOq8G3T MOMG86sHCF6pwCJmIJz1yRCuq7B1rUOF0q1P4M+XTutficXoSNDTRiW/yxFsKhz8XmQ6 NdlhUI3hI+dnFQ2VMfv1rCJwVaTs8JsXHJ/ahh21lHI1zqbquelFECkQbM6vZwvw6qfJ 3+6A== X-Gm-Message-State: ANhLgQ3N0MCwqGe2Y+jNDS+A+EWgOeaBxRVpG86BIHVvbUPy7yKuUL7/ HnepWrjU/eDaWWxFw8n9K4FFON2g X-Google-Smtp-Source: ADFU+vt90dlKvf2PIKh23vUQq5t6cuH04FHk2SfkNkBi6eZqdmX+4SZAnL1J+RnU+SZzy7WIwVkz9w== X-Received: by 2002:a5d:66cc:: with SMTP id k12mr20702wrw.72.1583160634849; Mon, 02 Mar 2020 06:50:34 -0800 (PST) Received: from hthiery01.sab.local ([213.135.10.150]) by smtp.gmail.com with ESMTPSA id h17sm29648355wro.52.2020.03.02.06.50.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Mar 2020 06:50:34 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Mon, 2 Mar 2020 15:50:03 +0100 Message-Id: <20200302145013.9574-2-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200302145013.9574-1-heiko.thiery@gmail.com> References: <20200302145013.9574-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 01/12] support/scripts/pkg-stats: store latest version in a dict X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This patch changes the type of the latest_version variable to a dict. This is for better readability/usability of the data. With this the json output is more descriptive in later processing of the json output. Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 31 +++++++++++++++---------------- 1 file changed, 15 insertions(+), 16 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 7721d98459..727b203032 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -71,7 +71,7 @@ class Package: self.url_status = None self.url_worker = None self.cves = list() - self.latest_version = (RM_API_STATUS_ERROR, None, None) + self.latest_version = {'status': RM_API_STATUS_ERROR, 'version': None, 'id': None} def pkgvar(self): return self.name.upper().replace("-", "_") @@ -460,9 +460,8 @@ def check_package_latest_version(packages): """ Fills in the .latest_version field of all Package objects - This field has a special format: - (status, version, id) - with: + This field is a dict and has the following keys: + - status: one of RM_API_STATUS_ERROR, RM_API_STATUS_FOUND_BY_DISTRO, RM_API_STATUS_FOUND_BY_PATTERN, RM_API_STATUS_NOT_FOUND @@ -478,7 +477,7 @@ def check_package_latest_version(packages): worker_pool = Pool(processes=64) results = worker_pool.map(check_package_latest_version_worker, (pkg.name for pkg in packages)) for pkg, r in zip(packages, results): - pkg.latest_version = r + pkg.latest_version = dict(zip(['status', 'version', 'id'], r)) del http_pool @@ -516,13 +515,13 @@ def calculate_stats(packages): stats["hash"] += 1 else: stats["no-hash"] += 1 - if pkg.latest_version[0] == RM_API_STATUS_FOUND_BY_DISTRO: + if pkg.latest_version['status'] == RM_API_STATUS_FOUND_BY_DISTRO: stats["rmo-mapping"] += 1 else: stats["rmo-no-mapping"] += 1 - if not pkg.latest_version[1]: + if not pkg.latest_version['version']: stats["version-unknown"] += 1 - elif pkg.latest_version[1] == pkg.current_version: + elif pkg.latest_version['version'] == pkg.current_version: stats["version-uptodate"] += 1 else: stats["version-not-uptodate"] += 1 @@ -688,29 +687,29 @@ def dump_html_pkg(f, pkg): f.write(" %s\n" % current_version) # Latest version - if pkg.latest_version[0] == RM_API_STATUS_ERROR: + if pkg.latest_version['status'] == RM_API_STATUS_ERROR: td_class.append("version-error") - if pkg.latest_version[1] is None: + if pkg.latest_version['version'] is None: td_class.append("version-unknown") - elif pkg.latest_version[1] != pkg.current_version: + elif pkg.latest_version['version'] != pkg.current_version: td_class.append("version-needs-update") else: td_class.append("version-good") - if pkg.latest_version[0] == RM_API_STATUS_ERROR: + if pkg.latest_version['status'] == RM_API_STATUS_ERROR: latest_version_text = "Error" - elif pkg.latest_version[0] == RM_API_STATUS_NOT_FOUND: + elif pkg.latest_version['status'] == RM_API_STATUS_NOT_FOUND: latest_version_text = "Not found" else: - if pkg.latest_version[1] is None: + if pkg.latest_version['version'] is None: latest_version_text = "Found, but no version" else: latest_version_text = "%s" % \ - (pkg.latest_version[2], str(pkg.latest_version[1])) + (pkg.latest_version['id'], str(pkg.latest_version['version'])) latest_version_text += "
" - if pkg.latest_version[0] == RM_API_STATUS_FOUND_BY_DISTRO: + if pkg.latest_version['status'] == RM_API_STATUS_FOUND_BY_DISTRO: latest_version_text += "found by distro" else: latest_version_text += "found by guess" From patchwork Mon Mar 2 14:50:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1247694 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=kUR84dB9; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48WNPn5zF1z9sRN for ; Tue, 3 Mar 2020 01:51:25 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 9C55385B0A; Mon, 2 Mar 2020 14:51:23 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LzYLSgQJZ3Sm; Mon, 2 Mar 2020 14:51:22 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id A201285A74; Mon, 2 Mar 2020 14:51:20 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id E2B7F1BF39F for ; Mon, 2 Mar 2020 14:51:11 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 8BDCF8589C for ; Mon, 2 Mar 2020 14:51:11 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cNwj-A5I8lmC for ; Mon, 2 Mar 2020 14:51:06 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f68.google.com (mail-wm1-f68.google.com [209.85.128.68]) by fraxinus.osuosl.org (Postfix) with ESMTPS id BB99E85660 for ; Mon, 2 Mar 2020 14:50:37 +0000 (UTC) Received: by mail-wm1-f68.google.com with SMTP id p9so11534503wmc.2 for ; Mon, 02 Mar 2020 06:50:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=u3Zemj8dFbRO5aW65oOhJKFaOpWIfZZTFufonP6rFZo=; b=kUR84dB9o/bjlKd1KIhMsHJEv8w4TrAMjrAHDVN+tWH4QU9YSykmprhoCeun9fiO+e D7nMEBKkRm+1EMVhbxdYxLMWZHG9r6ffkNr/sxe1FfAf7mYgqLv3IOIP4mikPTGbnZ1i WDRk7E9ofof9gxzCAvTpWkvI+1nKm6poHxr+/Kx+cUHKfvGQGGpH2qTGPq64OaL3Esl0 dlCX/pMsIJO4dxjJ/hZ11g2lPNJhhY0ou5aDvZ3IVbl6R94vLRCDXz9gf2QKdYUBrsVw ZnPQGsZxsF1lP8e8G3Ny8xWjArRtptf7EcPdCD7BFNVosMpTpUcdbMzhnZ3s4cRfDjiL WBxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=u3Zemj8dFbRO5aW65oOhJKFaOpWIfZZTFufonP6rFZo=; b=YRAzB/c5HSq7juEl22TRCUAhEXtGYJcnQsAPcVCXqFNR+6f+lRmSBKmqXXmxozrebV FV4VMlq779NhMHqTBshiwOG+EC2nLL9sS1L1rplg10jVhMpyXbbBmnvSMylRPJVVTGUX s8laa+AWnMxEQTcYDOe2PiyboF86z+XuxT8LX/PwqJ3L0Se+eVm8i/vBymnYxlRaK9Ah N/n8DTJvxsYcAKxYcDiat8yfGQ4a17zRgeWWA8P5b/FVB8LfhrDENfbNGKsDE8D33VJU rDVxNGw6Q3MnJcLN8NPKDxUZC/2ciJamCpg3Y9fYN1VxVhpVmQNIpUWSl/2NlxATLvlC CiJg== X-Gm-Message-State: APjAAAWjB97XsJr04OwzdoRForBU/zCnkQrS0ertCDUsc3J8V7OiXCXK mmSLa2oUvGUwU1rzzYfvmNoJPWBb X-Google-Smtp-Source: APXvYqzW2urb8SStu8JYcY2V/n5qqxO5FvoL4dprdZMbdWi8YHB+WoXGt8QPxGowqQFev1O7St5GLQ== X-Received: by 2002:a1c:688a:: with SMTP id d132mr19040479wmc.80.1583160635978; Mon, 02 Mar 2020 06:50:35 -0800 (PST) Received: from hthiery01.sab.local ([213.135.10.150]) by smtp.gmail.com with ESMTPSA id h17sm29648355wro.52.2020.03.02.06.50.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Mar 2020 06:50:35 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Mon, 2 Mar 2020 15:50:04 +0100 Message-Id: <20200302145013.9574-3-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200302145013.9574-1-heiko.thiery@gmail.com> References: <20200302145013.9574-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 02/12] support/scripts/pkg-stats: store patch files for the package X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Remove the patch_count attribute and use a class property instead. Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 727b203032..4116622c98 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -64,7 +64,7 @@ class Package: self.has_license = False self.has_license_files = False self.has_hash = False - self.patch_count = 0 + self.patch_files = [] self.warnings = 0 self.current_version = None self.url = None @@ -93,6 +93,10 @@ class Package: self.url_status = "Missing" fp.close() + @property + def patch_count(self): + return len(self.patch_files) + def set_infra(self): """ Fills in the .infras field @@ -131,10 +135,9 @@ class Package: """ Fills in the .patch_count field """ - self.patch_count = 0 pkgdir = os.path.dirname(self.path) for subdir, _, _ in os.walk(pkgdir): - self.patch_count += len(fnmatch.filter(os.listdir(subdir), '*.patch')) + self.patch_files = fnmatch.filter(os.listdir(subdir), '*.patch') def set_current_version(self): """ From patchwork Mon Mar 2 14:50:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1247683 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=nvf2ipZT; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48WNP31ZrFz9sRR for ; Tue, 3 Mar 2020 01:50:47 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id A30938609C; Mon, 2 Mar 2020 14:50:44 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NoGkgWVeUtdA; Mon, 2 Mar 2020 14:50:41 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 484478569B; Mon, 2 Mar 2020 14:50:41 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 426031BF39F for ; Mon, 2 Mar 2020 14:50:40 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 3F51B20496 for ; Mon, 2 Mar 2020 14:50:40 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RpJrnJ823Ieb for ; Mon, 2 Mar 2020 14:50:39 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f52.google.com (mail-wr1-f52.google.com [209.85.221.52]) by silver.osuosl.org (Postfix) with ESMTPS id CC2072000A for ; Mon, 2 Mar 2020 14:50:38 +0000 (UTC) Received: by mail-wr1-f52.google.com with SMTP id z11so4194008wro.9 for ; Mon, 02 Mar 2020 06:50:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tFhz+aOsoXWAAq6rDXjf89d8RmmpvJ14B4HZqzZeGSs=; b=nvf2ipZTj3kVyujfJvlbp08qYTatBsAA9eE7dC4n5WSbaq8lGCBtsLIZZdYf90lak7 usL7qc93CeJ6VZ6ZJmcuJ4kqLnkb0le9gmx1GWVo9+2AdhRPjronkAgox9jsEXCs7udY jREZtuHYWfEkJW8pZFGvoYyUxGwX7dFB2uAayWilUn9nsdHyK75iFGJrKGkbnoHaY5BX x0y3Nt30cy0Zz2ODEqQFKP5P1o//oN7/mAEkaP1qf/STRoDk1MLLuCCtUQeqmnH4inlh V4D7/Bj95joYbXFrX+gvvI+ph8NQ47R7+ppe3Y7R6ksDqpmC2pOL9Sia1hogs5HhLfsR SyZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tFhz+aOsoXWAAq6rDXjf89d8RmmpvJ14B4HZqzZeGSs=; b=pNyfOHyW6OcP8Mn8Ix9LjhbdUd4tHOMUWMznxoSyI3TjfLR+wDi1ycrhE3JRCdE856 SnvduTs8AuchKI3+eNxDI5vsRwmOGCjKHEn6DtKYYRa2NsMKt9ws+jNtE0Qz8f2qxsd4 2m6WFirdyGNL92ZfbSqv1pcZmq3BWG197qAJJ0qp1doRF8v3nvzCS2tqhGlXmCkEfpH2 baUvv6y68rANGbOdAqF9dl98EW/iWvsOM07gIBQT5pQgOG2ijPhVK7vib0JzAhGvBilj bWpIwEPHhUXV9JvXO7NS41f6d3VVFcYlIPMm4hqMmaf3CVoOve04HFlK+Eqo56734uYc 3g3g== X-Gm-Message-State: APjAAAVscfNaNfKBm08ElL1OuWPPIv6uK3yWun23qYfJ09/6i1B8bxrh CTbU1k1YTk9DhCk+gjTvSrYHDqVk X-Google-Smtp-Source: APXvYqwD7om265AuiAgDCVHG5Ssw2/zxtt1uoi+h5xJDBhwweRmBwWlrde4QDQ84z23fsidXWdEZoQ== X-Received: by 2002:adf:ecca:: with SMTP id s10mr23687926wro.255.1583160636810; Mon, 02 Mar 2020 06:50:36 -0800 (PST) Received: from hthiery01.sab.local ([213.135.10.150]) by smtp.gmail.com with ESMTPSA id h17sm29648355wro.52.2020.03.02.06.50.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Mar 2020 06:50:36 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Mon, 2 Mar 2020 15:50:05 +0100 Message-Id: <20200302145013.9574-4-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200302145013.9574-1-heiko.thiery@gmail.com> References: <20200302145013.9574-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 03/12] support/scripts/pkg-stats: set developers info X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Use the function 'parse_developers' function from getdeveloperlib that collect the information about the developers and the files they maintain. Then set the maintainer(s) to each package. Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 4116622c98..c40fe61d9e 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -30,10 +30,14 @@ import certifi import distutils.version import time import gzip +import sys from urllib3 import HTTPSConnectionPool from urllib3.exceptions import HTTPError from multiprocessing import Pool +sys.path.append('utils/') +from getdeveloperlib import parse_developers + NVD_START_YEAR = 2002 NVD_JSON_VERSION = "1.0" NVD_BASE_URL = "https://nvd.nist.gov/feeds/json/cve/" + NVD_JSON_VERSION @@ -171,6 +175,16 @@ class Package: """ return cve in self.all_ignored_cves.get(self.pkgvar(), []) + def set_developers(self, developers): + """ + Fills in the .developers field + """ + self.developers = [ + dev.name + for dev in developers + if dev.hasfile(self.path) + ] + def __eq__(self, other): return self.path == other.path @@ -891,6 +905,8 @@ def __main__(): 'HEAD']).splitlines()[0] print("Build package list ...") packages = get_pkglist(args.npackages, package_list) + print("Getting developers ...") + developers = parse_developers() print("Getting package make info ...") package_init_make_info() print("Getting package details ...") @@ -902,6 +918,7 @@ def __main__(): pkg.set_check_package_warnings() pkg.set_current_version() pkg.set_url() + pkg.set_developers(developers) print("Checking URL status") check_package_urls(packages) print("Getting latest versions ...") From patchwork Mon Mar 2 14:50:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1247693 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=dT1+9Hqc; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48WNPl2wSCz9sRR for ; Tue, 3 Mar 2020 01:51:23 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id B9BCA85ABE; Mon, 2 Mar 2020 14:51:21 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ImKQWZLsRDDU; Mon, 2 Mar 2020 14:51:19 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id AB12D8593F; Mon, 2 Mar 2020 14:51:19 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id A64A91BF39F for ; Mon, 2 Mar 2020 14:51:08 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id A31CC84E49 for ; Mon, 2 Mar 2020 14:51:08 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dbEi3_9If7DF for ; Mon, 2 Mar 2020 14:51:08 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f49.google.com (mail-wm1-f49.google.com [209.85.128.49]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 7D41385725 for ; Mon, 2 Mar 2020 14:50:39 +0000 (UTC) Received: by mail-wm1-f49.google.com with SMTP id e26so5346941wme.5 for ; Mon, 02 Mar 2020 06:50:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=uuArVBDxz9iADI5APulF8wilPzE8cA+kT6+NoXEHoLg=; b=dT1+9HqcZaV7Tb30tOYR/HC8nh+MV7GKSOp5JXfQ1ezpa9aUPdXq7j8a8dQVyEjWQM eXq/ygsIXCSvyOktWIU57Opc+KuEvYYCoqNmQPePburI96drs3o7yFpBmrl5ATF9S4fI p9hug6Wb6lPJ5NjKD6aif+OtDY1oA6gsbKbTsI19eoc5dsOdoQYEJJ6WGxlrl9zP6DI+ 1294wSFlmIbDr722BMwJrCfPOyEjbAB6eiywBr1ylMOin7cIi2FgVmZJ2zOKjsX8pkBT l4mzeudeCPrmHJZeniIUN9CU/t6MemHyuJpWnEfqRfmWC3KdlFN9wrVtLgpSG43Z6ZsC rpPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=uuArVBDxz9iADI5APulF8wilPzE8cA+kT6+NoXEHoLg=; b=XRN2uN3XoPD4C1WKFfoAoMLeLJ8i4pY3UiV7AFVDQsqo7FY/mJ/1h61vKygKPeAJVA xD9ZBXpsvGdeZ+yDFnSrnmYUL+cnBJTfbQw8IpIm029gzTH1zQzzd73OzKkteb7dRJD7 HE9IrUc5gSh4PfbdhPzEDyWqKdUbHU7wY/kFHSEwFYouoqQNfYlwIY73+cMvErqN5giw ueDJuXworfdfrhh36anTSWL4P9CEo9a5pwP9Xs+g1m2fVkLumpaDQnU97PlXwo5twy4X ImOekB9uFnUn6NQ4iaLWl3/rSR28ChOjaPXH9/dPh1qz6OJ5knF1kXuJLONyMNzfP0gW X9Kg== X-Gm-Message-State: APjAAAVW2URBNowwO1pcNma7TZHqrdv+qHutXMHvKPOg1XYyRu+6WrUB BIuovBVxUhNatcqipuMpf9vFuqLK X-Google-Smtp-Source: APXvYqwoTtWYamFBIAzml1Wa1qlTzEbuZYlnaWvaiUsyI81ZLgxMWSSOpUtICIuJ7bH142mx7oQroA== X-Received: by 2002:a7b:ce09:: with SMTP id m9mr20197774wmc.49.1583160637674; Mon, 02 Mar 2020 06:50:37 -0800 (PST) Received: from hthiery01.sab.local ([213.135.10.150]) by smtp.gmail.com with ESMTPSA id h17sm29648355wro.52.2020.03.02.06.50.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Mar 2020 06:50:37 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Mon, 2 Mar 2020 15:50:06 +0100 Message-Id: <20200302145013.9574-5-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200302145013.9574-1-heiko.thiery@gmail.com> References: <20200302145013.9574-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 04/12] support/scripts/pkg-stats: store licences of package X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index c40fe61d9e..b0ee4f3bac 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -56,7 +56,7 @@ http_pool = None class Package: - all_licenses = list() + all_licenses = dict() all_license_files = list() all_versions = dict() all_ignored_cves = dict() @@ -65,6 +65,7 @@ class Package: self.name = name self.path = path self.infras = None + self.license = None self.has_license = False self.has_license_files = False self.has_hash = False @@ -125,6 +126,7 @@ class Package: var = self.pkgvar() if var in self.all_licenses: self.has_license = True + self.license = self.all_licenses[var] if var in self.all_license_files: self.has_license_files = True @@ -387,7 +389,7 @@ def package_init_make_info(): if value == "unknown": continue pkgvar = pkgvar[:-8] - Package.all_licenses.append(pkgvar) + Package.all_licenses[pkgvar] = value elif pkgvar.endswith("_LICENSE_FILES"): if pkgvar.endswith("_MANIFEST_LICENSE_FILES"): From patchwork Mon Mar 2 14:50:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1247684 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=enLF0QIp; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48WNP839bgz9sPK for ; Tue, 3 Mar 2020 01:50:52 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 7A73B204B2; Mon, 2 Mar 2020 14:50:50 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ehudh-t5yyPj; Mon, 2 Mar 2020 14:50:45 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 46E7620494; Mon, 2 Mar 2020 14:50:45 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 1264C1BF39F for ; Mon, 2 Mar 2020 14:50:44 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 0F3EE8602E for ; Mon, 2 Mar 2020 14:50:44 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9Zy1mlqcn2vu for ; Mon, 2 Mar 2020 14:50:41 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f52.google.com (mail-wm1-f52.google.com [209.85.128.52]) by whitealder.osuosl.org (Postfix) with ESMTPS id DEB8A8357B for ; Mon, 2 Mar 2020 14:50:40 +0000 (UTC) Received: by mail-wm1-f52.google.com with SMTP id a132so1534380wme.1 for ; Mon, 02 Mar 2020 06:50:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ljn/FX73srmDEI99v6IpjynvoP4wHrSiEYOonCuBhZA=; b=enLF0QIpzhE99UM29pwtrMtV0WaPMgKoYoBwNLslxDM1WRwtt3FEivRBi5l1gEzmRH iaxvJ7wFVle3DOsXzxVDOvi2zolE9odjiES1p6ZG+Cgz4m5tp/UitARRpbbEoIYTFdH8 tAz89hsuT+AxPxLQAz9tW7I1PmoPVOMi2UdWar4D2NpseS1zNNfyeZwyB0TeYTfQcLtO cDSPFYiA51zS6mWeia9gr7SEJtoOe1YBdyyuydSCaZoptA6TmTzDS3MVbCwKugwFiSW1 NRf/2e6+594bfCBf/+rIhS+6xdNooZVIxdv7A22Ky1WTXdkx4IPR9PcEvNVPOMJ+W6kS 4oVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ljn/FX73srmDEI99v6IpjynvoP4wHrSiEYOonCuBhZA=; b=ARgO9IiQxzL/6wEHxCUaOby/ulPcxv/2RsggtmFPvpKFNA7K1R04SguEk9i6DgGsOg WVxoyg4UEX450MEbXJt8045KnkmMnZFyg1x5QsJbG7Ty8xcfnp3YBPkgkRpStRQIY34I DiZ/tf/BydebN1XXhJPjH/SVamIfkfWDztqqbak+ZF5Aw11uv4mcyd6DghqLO6tsMZZf i87Cr2pcO0WBuf+wr7QnnFB8aaMOSkSxluwntRBwAf3JvP4TKieh/hJdA1OJw40J8x6e 3IXzYSYSlU6KXaufe+c0j4qf/7PWnHeAUbX3pkIEkTuZ33M+rv50R5TZsIdKuAxuiQoN fEUw== X-Gm-Message-State: APjAAAUalAmis7fCEHOjhGIdn88pBBTqELb0MBQiBbC7imLT6hW0GL6U kkZzRymTveLkW/zTCe7so8jtcBa7 X-Google-Smtp-Source: APXvYqzd0qI8OhoqpfepPTtxfkLCW4sntFvo8SdkfwaYEynpfrk7YdFUVMX2gJXeiw+dzWK+iG1okg== X-Received: by 2002:a7b:c4d0:: with SMTP id g16mr18107590wmk.3.1583160638582; Mon, 02 Mar 2020 06:50:38 -0800 (PST) Received: from hthiery01.sab.local ([213.135.10.150]) by smtp.gmail.com with ESMTPSA id h17sm29648355wro.52.2020.03.02.06.50.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Mar 2020 06:50:37 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Mon, 2 Mar 2020 15:50:07 +0100 Message-Id: <20200302145013.9574-6-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200302145013.9574-1-heiko.thiery@gmail.com> References: <20200302145013.9574-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 05/12] support/scripts/pkg-stats: add package status X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Unify the status check information. The status is stored in a tuple. The first entry is the status that can be 'ok', 'warning' or 'error'. The second entry is a verbose message. The following checks are performed: - url: status of the URL check - license: status of the license presence check - license-files: status of the license file check - hash: status of the hash file presence check - patches: status of the patches count check - pkg-check: status of the check-package script result - developers: status if a package has developers in the DEVELOPERS file - version: status of the version check With that status information the following variables are replaced: has_license, has_license_files, has_hash, url_status Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 101 ++++++++++++++++++++++++++------------ 1 file changed, 69 insertions(+), 32 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index b0ee4f3bac..47f7507540 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -73,10 +73,10 @@ class Package: self.warnings = 0 self.current_version = None self.url = None - self.url_status = None self.url_worker = None self.cves = list() self.latest_version = {'status': RM_API_STATUS_ERROR, 'version': None, 'id': None} + self.status = {} def pkgvar(self): return self.name.upper().replace("-", "_") @@ -85,17 +85,17 @@ class Package: """ Fills in the .url field """ - self.url_status = "No Config.in" + self.status['url'] = ("warning", "no Config.in") for filename in os.listdir(os.path.dirname(self.path)): if fnmatch.fnmatch(filename, 'Config.*'): fp = open(os.path.join(os.path.dirname(self.path), filename), "r") for config_line in fp: if URL_RE.match(config_line): self.url = config_line.strip() - self.url_status = "Found" + self.status['url'] = ("ok", "found") fp.close() return - self.url_status = "Missing" + self.status['url'] = ("error", "missing") fp.close() @property @@ -121,30 +121,42 @@ class Package: def set_license(self): """ - Fills in the .has_license and .has_license_files fields + Fills in the .status['license'] and .status['license-files'] fields """ var = self.pkgvar() + self.status['license'] = ("error", "missing") + self.status['license-files'] = ("error", "missing") if var in self.all_licenses: - self.has_license = True self.license = self.all_licenses[var] + self.status['license'] = ("ok", "found") if var in self.all_license_files: - self.has_license_files = True + self.status['license-files'] = ("ok", "found") def set_hash_info(self): """ - Fills in the .has_hash field + Fills in the .status['hash'] field """ hashpath = self.path.replace(".mk", ".hash") - self.has_hash = os.path.exists(hashpath) + if os.path.exists(hashpath): + self.status['hash'] = ("ok", "found") + else: + self.status['hash'] = ("error", "missing") def set_patch_count(self): """ - Fills in the .patch_count field + Fills in the .patch_count, .patch_files and .status['patches'] fields """ pkgdir = os.path.dirname(self.path) for subdir, _, _ in os.walk(pkgdir): self.patch_files = fnmatch.filter(os.listdir(subdir), '*.patch') + if self.patch_count == 0: + self.status['patches'] = ("ok", "no patches") + elif self.patch_count < 5: + self.status['patches'] = ("warning", "some patches") + else: + self.status['patches'] = ("error", "lots of patches") + def set_current_version(self): """ Fills in the .current_version field @@ -155,10 +167,11 @@ class Package: def set_check_package_warnings(self): """ - Fills in the .warnings field + Fills in the .warnings and .status['pkg-check'] fields """ cmd = ["./utils/check-package"] pkgdir = os.path.dirname(self.path) + self.status['pkg-check'] = ("error", "Missing") for root, dirs, files in os.walk(pkgdir): for f in files: if f.endswith(".mk") or f.endswith(".hash") or f == "Config.in" or f == "Config.in.host": @@ -169,6 +182,10 @@ class Package: m = re.match("^([0-9]*) warnings generated", line) if m: self.warnings = int(m.group(1)) + if self.warnings == 0: + self.status['pkg-check'] = ("ok", "no warnings") + else: + self.status['pkg-check'] = ("error", "{} warnings".format(self.warnings)) return def is_cve_ignored(self, cve): @@ -179,7 +196,7 @@ class Package: def set_developers(self, developers): """ - Fills in the .developers field + Fills in the .developers and .status['developers'] field """ self.developers = [ dev.name @@ -187,6 +204,14 @@ class Package: if dev.hasfile(self.path) ] + if self.developers: + self.status['developers'] = ("ok", "{} developers".format(len(self.developers))) + else: + self.status['developers'] = ("warning", "no developers") + + def is_status_ok(self, name): + return self.status[name][0] == 'ok' + def __eq__(self, other): return self.path == other.path @@ -195,7 +220,7 @@ class Package: def __str__(self): return "%s (path='%s', license='%s', license_files='%s', hash='%s', patches=%d)" % \ - (self.name, self.path, self.has_license, self.has_license_files, self.has_hash, self.patch_count) + (self.name, self.path, self.is_status_ok('license'), self.is_status_ok('license-files'), self.status['hash'], self.patch_count) class CVE: @@ -409,23 +434,23 @@ def package_init_make_info(): def check_url_status_worker(url, url_status): - if url_status != "Missing" and url_status != "No Config.in": + if url_status[0] == 'ok': try: url_status_code = requests.head(url, timeout=30).status_code if url_status_code >= 400: - return "Invalid(%s)" % str(url_status_code) + return ("error", "invalid {}".format(url_status_code)) except requests.exceptions.RequestException: - return "Invalid(Err)" - return "Ok" + return ("error", "invalid (err)") + return ("ok", "valid") return url_status def check_package_urls(packages): Package.pool = Pool(processes=64) for pkg in packages: - pkg.url_worker = pkg.pool.apply_async(check_url_status_worker, (pkg.url, pkg.url_status)) + pkg.url_worker = pkg.pool.apply_async(check_url_status_worker, (pkg.url, pkg.status['url'])) for pkg in packages: - pkg.url_status = pkg.url_worker.get(timeout=3600) + pkg.status['url'] = pkg.url_worker.get(timeout=3600) def release_monitoring_get_latest_version_by_distro(pool, name): @@ -497,6 +522,18 @@ def check_package_latest_version(packages): results = worker_pool.map(check_package_latest_version_worker, (pkg.name for pkg in packages)) for pkg, r in zip(packages, results): pkg.latest_version = dict(zip(['status', 'version', 'id'], r)) + + if pkg.latest_version['status'] == RM_API_STATUS_ERROR: + pkg.status['version'] = ('warning', "Release Monitoring API error") + elif pkg.latest_version['status'] == RM_API_STATUS_NOT_FOUND: + pkg.status['version'] = ('warning', "Package not found on Release Monitoring") + + if pkg.latest_version['version'] is None: + pkg.status['version'] = ('warning', "No upstream version available on Release Monitoring") + elif pkg.latest_version['version'] != pkg.current_version: + pkg.status['version'] = ('error', "The newer version {} is available upstream".format(pkg.latest_version['version'])) + else: + pkg.status['version'] = ('ok', 'up-to-date') del http_pool @@ -522,15 +559,15 @@ def calculate_stats(packages): stats["infra-%s" % infra] += 1 else: stats["infra-unknown"] += 1 - if pkg.has_license: + if pkg.is_status_ok('license'): stats["license"] += 1 else: stats["no-license"] += 1 - if pkg.has_license_files: + if pkg.is_status_ok('license-files'): stats["license-files"] += 1 else: stats["no-license-files"] += 1 - if pkg.has_hash: + if pkg.is_status_ok('hash'): stats["hash"] += 1 else: stats["no-hash"] += 1 @@ -673,30 +710,30 @@ def dump_html_pkg(f, pkg): # License td_class = ["centered"] - if pkg.has_license: + if pkg.is_status_ok('license'): td_class.append("correct") else: td_class.append("wrong") f.write(" %s\n" % - (" ".join(td_class), boolean_str(pkg.has_license))) + (" ".join(td_class), boolean_str(pkg.is_status_ok('license')))) # License files td_class = ["centered"] - if pkg.has_license_files: + if pkg.is_status_ok('license-files'): td_class.append("correct") else: td_class.append("wrong") f.write(" %s\n" % - (" ".join(td_class), boolean_str(pkg.has_license_files))) + (" ".join(td_class), boolean_str(pkg.is_status_ok('license-files')))) # Hash td_class = ["centered"] - if pkg.has_hash: + if pkg.is_status_ok('hash'): td_class.append("correct") else: td_class.append("wrong") f.write(" %s\n" % - (" ".join(td_class), boolean_str(pkg.has_hash))) + (" ".join(td_class), boolean_str(pkg.is_status_ok('hash')))) # Current version if len(pkg.current_version) > 20: @@ -747,12 +784,12 @@ def dump_html_pkg(f, pkg): # URL status td_class = ["centered"] - url_str = pkg.url_status - if pkg.url_status == "Missing" or pkg.url_status == "No Config.in": + url_str = pkg.status['url'][1] + if pkg.status['url'][0] in ("error", "warning"): td_class.append("missing_url") - elif pkg.url_status.startswith("Invalid"): + if pkg.status['url'][0] == "error": td_class.append("invalid_url") - url_str = "%s" % (pkg.url, pkg.url_status) + url_str = "%s" % (pkg.url, pkg.status['url'][1]) else: td_class.append("good_url") url_str = "Link" % pkg.url From patchwork Mon Mar 2 14:50:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1247690 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=EREIaVKb; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48WNPZ4QxXz9sPK for ; Tue, 3 Mar 2020 01:51:14 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 04EBD859D5; Mon, 2 Mar 2020 14:51:11 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TqR0uDBqZRlr; Mon, 2 Mar 2020 14:51:10 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id E8083858D2; Mon, 2 Mar 2020 14:51:09 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 3DF461BF39F for ; Mon, 2 Mar 2020 14:51:08 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 389FA8589C for ; Mon, 2 Mar 2020 14:51:08 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id eETw9690xBoM for ; Mon, 2 Mar 2020 14:51:07 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 3D2B78587E for ; Mon, 2 Mar 2020 14:50:41 +0000 (UTC) Received: by mail-wm1-f43.google.com with SMTP id a25so87215wmm.0 for ; Mon, 02 Mar 2020 06:50:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=rkTAawy7/kWxgXZximMLfAI1cRIUjgf/w2vu+xvEgjs=; b=EREIaVKbSP5y0d5OD1b+BF3f1ACXbCtmWSoSDhrWbf1erFjWnxbjazdSJiAwdIH3Le avsLalTDSEnPaRbcIf4ZfG4/g7nOEWFOchZvc3IZiLnX+XtbKUzwNnZ9xSkP/tpEPmmZ ZejZ77MlWIVOjqdL7uWpejMRi+Ni22sDmY3e2Prg7pjlEPymZqNKrdn8TOlMfMm3UAAZ RlW8lEQmXvhnOSvjl/Nv6UtLEVE1suCgXp2S7IpgyYV9vF4eeyY7mzbcenzVY3U4nHz5 3U1RGgRh2ppifeAwje+9Jmsu5OvlIpYURJ/7AS8rsqph0H+ouTPopu2uiMGbbulC8xJw vC6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rkTAawy7/kWxgXZximMLfAI1cRIUjgf/w2vu+xvEgjs=; b=YoZ1b4tkIhdTzb+yBX06XV4P3lL5jiTYAJ9LkMIwtAj7p/iOA2v9H1nnx/zbQ3hVhg Y+YucCTvyPiCFD8v7SfLbWfECggyioD62qun+vOkLHyB/Q4PAfhdqx4g9tCXimqbimRM mTr0bLC7vmMqCQIn4b/ym7cFLOKkS5t/GQ8z5q9rCUsvGNign17BWgxmhOQdpvyPqUXg YAeagD0Pf19jQ3MGxV+5eSD8S0WEpPLqMqrgFjQpu+EWoTotqqyRi9JzezobEgFwulQI SfJfVa7r3QTBAedjC785yy0JEvMS4yVPDXr2cZPZPGbKvDCpIiw1QRUBiiWEK9thf2GG AsYw== X-Gm-Message-State: APjAAAWZ8MyrsuK2pG8BEzJ0e0clg3/XDochNnWAaiw1fa1w3Y4dW+1J GKpRtDc6TL1GisSYjBgxENClUaCj X-Google-Smtp-Source: APXvYqxfkefmWAdIP7PhBS6j62SIltCzk160GXFo8hF1rA4XZSQ4KQGlZWnJavVQ7KVjCfTd8fPW8Q== X-Received: by 2002:a7b:c957:: with SMTP id i23mr20509196wml.174.1583160639427; Mon, 02 Mar 2020 06:50:39 -0800 (PST) Received: from hthiery01.sab.local ([213.135.10.150]) by smtp.gmail.com with ESMTPSA id h17sm29648355wro.52.2020.03.02.06.50.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Mar 2020 06:50:38 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Mon, 2 Mar 2020 15:50:08 +0100 Message-Id: <20200302145013.9574-7-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200302145013.9574-1-heiko.thiery@gmail.com> References: <20200302145013.9574-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 06/12] support/scripts/pkg-stats: add package count to stats X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 1 + 1 file changed, 1 insertion(+) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 47f7507540..2847cb82f3 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -549,6 +549,7 @@ def check_package_cves(nvd_path, packages): def calculate_stats(packages): stats = defaultdict(int) + stats['packages'] = len(packages) for pkg in packages: # If packages have multiple infra, take the first one. For the # vast majority of packages, the target and host infra are the From patchwork Mon Mar 2 14:50:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1247689 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=mfTDqF8o; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48WNPW6t5nz9sPK for ; Tue, 3 Mar 2020 01:51:11 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 34EE184E49; Mon, 2 Mar 2020 14:51:09 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id era76kEL3uET; Mon, 2 Mar 2020 14:51:08 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id B9EE285593; Mon, 2 Mar 2020 14:51:08 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 5DC3C1BF39F for ; Mon, 2 Mar 2020 14:51:07 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 5A701858D2 for ; Mon, 2 Mar 2020 14:51:07 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Gvj01hGu3Dqz for ; Mon, 2 Mar 2020 14:51:06 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f46.google.com (mail-wm1-f46.google.com [209.85.128.46]) by fraxinus.osuosl.org (Postfix) with ESMTPS id A559E8589C for ; Mon, 2 Mar 2020 14:50:42 +0000 (UTC) Received: by mail-wm1-f46.google.com with SMTP id a141so10928310wme.2 for ; Mon, 02 Mar 2020 06:50:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=R4KEm9qeoAHmBvtoVu1vih8L1zzk+RYxy4miEirJujc=; b=mfTDqF8oZH7RtSXi3UGVc/yvNBPvDBP8cK5hqUZaoCstBzIkfbh19uBVgvijPyMprt 2Fycp2iwGi7SsArJFzLcXdLJNObi/jfA1nHucAIgrYmFML4k8jMnWygNOdbWiDRrrG70 l/H6ncY7sxgvWOVIzApsgQnrFUNTSzoKvMnosK+uFM1IjfM3mzxaYWXJUt887DSAfPqM dVJ6wUJSbDoClo1I04gcxfRRXTVibuzep1sb+r7mz14RMfPsabqDkjjEWkY7eUmESa5k LZsD3XIhimj3gfhBgZe1egM0yQHajMRl8qOsMrII0cIaVou1H7up8SUTqfd6GVmJNrGN TkDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=R4KEm9qeoAHmBvtoVu1vih8L1zzk+RYxy4miEirJujc=; b=A5z5Y644M0pry7i9gyWXzAQAq2CvMVAxrRIutSWZMRZXcbl+qM/bEBtUk1oF7s0tjq KjTq5JyBr3ZMZRxbm0gacY9C60fzOLz9/QPKkP7/sy8v3s8cVhDdNsC7i9c7cDBLla2n jv27Y998QoHFU+ssDnkfnLLx7LDaMwcFaa7dAjUVRVtUqhIN+pON9MqZEZtHeyBUBuBW BOdREt5MbBs/OeI/4Jn8ilwIodGbzceT6kF9O1dEPCr7dnVk4aYhhhWVyfsrRhwpItOj oVlfLjv2LLFxq2/StCPOW2ljxLfVFt02fgc7Bo4UKFckC8rN0txgzSpE9Zz53pF2SBwz xxRg== X-Gm-Message-State: APjAAAWWaszBBZNr5BMT9FiaOHr/BhtmBUvg3F5ijq0e2+Zbc1FQ3ExM 6NZuxCEQJjWh0Rh/2Xc5N6L321NA X-Google-Smtp-Source: APXvYqzcbLHevsO6YR7pD/RCrlSsRBhqzwRv5M28L4ZPGfASAx+fCKuxiF6K7nCFBlH7yqe10iHzLA== X-Received: by 2002:a1c:156:: with SMTP id 83mr20672335wmb.151.1583160640770; Mon, 02 Mar 2020 06:50:40 -0800 (PST) Received: from hthiery01.sab.local ([213.135.10.150]) by smtp.gmail.com with ESMTPSA id h17sm29648355wro.52.2020.03.02.06.50.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Mar 2020 06:50:39 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Mon, 2 Mar 2020 15:50:09 +0100 Message-Id: <20200302145013.9574-8-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200302145013.9574-1-heiko.thiery@gmail.com> References: <20200302145013.9574-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 07/12] support/scripts/pkg-stats: store pkg dir path X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This value can be used for later processing. In the buildroot-stats application this is used to create links pointing to the git repo of buildroot. Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 1 + 1 file changed, 1 insertion(+) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 2847cb82f3..1fd306a6d5 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -64,6 +64,7 @@ class Package: def __init__(self, name, path): self.name = name self.path = path + self.pkg_path = os.path.dirname(path) self.infras = None self.license = None self.has_license = False From patchwork Mon Mar 2 14:50:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1247691 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=DUf3+GuG; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48WNPb4fjdz9sRN for ; Tue, 3 Mar 2020 01:51:15 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 04FCD204B5; Mon, 2 Mar 2020 14:51:14 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id uCGadwvFCV4p; Mon, 2 Mar 2020 14:51:11 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 6EE0220496; Mon, 2 Mar 2020 14:51:11 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 593C61BF973 for ; Mon, 2 Mar 2020 14:51:08 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 55A0A8587E for ; Mon, 2 Mar 2020 14:51:08 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ppHdkEWh2Bxg for ; Mon, 2 Mar 2020 14:51:07 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f45.google.com (mail-wm1-f45.google.com [209.85.128.45]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 719D7858A6 for ; Mon, 2 Mar 2020 14:50:43 +0000 (UTC) Received: by mail-wm1-f45.google.com with SMTP id m3so10941309wmi.0 for ; Mon, 02 Mar 2020 06:50:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=HkCAuAaCvIIpJjfDDIqEbOcXD06rQTsWsiZYFEEDT3M=; b=DUf3+GuGeJ2DCYOyjbZ+mDSBjRIYNqHAV5LI8/WZw0JLlVpP/C3MD6vPzHf11NTLwd C2q/inGoG1H/a254t87CPJk1nVpO1o4W9UKCww9OlIBEdBHcH/NAdAB7Xq7P7Hy+E8ze mxeDqNxN9bb/6tQokXw6613x3hyfEZ3VntMaCj1lUcg3LcJ7Yu7x3ySV+YsqEC1G7+eD 1WjRBnxroXc+tufgFWmuQ+OseRBjLJXkHTNouv0ncDmgLK+S6R79gUrdXVwb6NjPhmew Nl7Us3OR2Wtr3kcMUoES4uVGHU4pXr8N15Ct4lqGUDnjdA4nNuAapKCOiDQ3I3MBHV3u cDkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HkCAuAaCvIIpJjfDDIqEbOcXD06rQTsWsiZYFEEDT3M=; b=Z4gco+y3h3P2U/OeD2dwnRTC69bVYpAncdYeOBidI5NtlXtt5+YODlZQ1dgLZo48Lc zMEgHA+7V78+ltCF72zxRwZ9Tr8iEI3Nvd6BeJrQlLCNHDCRP0yM732+fDjIjoiVxjdo 0xu4VA6QAYU8RaVJNxORaF3KaSZhIwEX1UkQyrutyy0O3Fp4ynYCeqVOx4USZ5BpWbFu Txh1pFisdDivo+e8kiEDvQVHFeDRXohYlObB7jDGVeCMN0QIpiJF+wje+CAThAvVKory 8lpcBXKomMLgkDQE4krt+m5qnfq0DrgjL55l/E4h5j6+M9QNix3bWIlMbCiN1zCszmWj od7A== X-Gm-Message-State: APjAAAVTNb4oVl9HmiqbdvyU06GCqT6fchjzn8ggxwrmoDKibAXellGp eXy1A9C4JoE1e75CvSjTufmLqhVG X-Google-Smtp-Source: APXvYqxhlCl3/3DJB248ZIhWVw19m4zRGQ3nKulwr0nLwoPHdP3ipOJ9Dc2qTPJnC8bmt35/zYl18w== X-Received: by 2002:a7b:cb97:: with SMTP id m23mr19212163wmi.37.1583160641649; Mon, 02 Mar 2020 06:50:41 -0800 (PST) Received: from hthiery01.sab.local ([213.135.10.150]) by smtp.gmail.com with ESMTPSA id h17sm29648355wro.52.2020.03.02.06.50.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Mar 2020 06:50:41 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Mon, 2 Mar 2020 15:50:10 +0100 Message-Id: <20200302145013.9574-9-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200302145013.9574-1-heiko.thiery@gmail.com> References: <20200302145013.9574-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 08/12] support/scripts/pkg-stats: add defconfig support X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Scan configs directory and create Defconfig objects. Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 43 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 41 insertions(+), 2 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 1fd306a6d5..432090c251 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -54,6 +54,34 @@ RM_API_STATUS_NOT_FOUND = 4 # because it's used by sub-processes. http_pool = None +class Defconfig: + def __init__(self, name, path): + self.name = name + self.path = path + self.developers = None + + def set_developers(self, developers): + """ + Fills in the .developers field + """ + self.developers = [ + developer.name + for developer in developers + if developer.hasfile(self.path) + ] + + +def get_defconfig_list(): + """ + Builds the list of Buildroot defconfigs, returning a list of Defconfig + objects. + """ + return [ + Defconfig(name[:-len('_defconfig')], os.path.join('configs', name)) + for name in os.listdir('configs') + if name.endswith('_defconfig') + ] + class Package: all_licenses = dict() @@ -886,7 +914,7 @@ def dump_html(packages, stats, date, commit, output): f.write(html_footer) -def dump_json(packages, stats, date, commit, output): +def dump_json(packages, defconfigs, stats, date, commit, output): # Format packages as a dictionnary instead of a list # Exclude local field that does not contains real date excluded_fields = ['url_worker', 'name'] @@ -897,6 +925,12 @@ def dump_json(packages, stats, date, commit, output): if k not in excluded_fields } for pkg in packages } + defconfigs = { + d.name: { + k: v + for k, v in d.__dict__.items() + } for d in defconfigs + } # Aggregate infrastructures into a single dict entry statistics = { k: v @@ -907,6 +941,7 @@ def dump_json(packages, stats, date, commit, output): # The actual structure to dump, add commit and date to it final = {'packages': pkgs, 'stats': statistics, + 'defconfigs': defconfigs, 'commit': commit, 'date': str(date)} @@ -948,6 +983,10 @@ def __main__(): packages = get_pkglist(args.npackages, package_list) print("Getting developers ...") developers = parse_developers() + print("Build defconfig list ...") + defconfigs = get_defconfig_list() + for d in defconfigs: + d.set_developers(developers) print("Getting package make info ...") package_init_make_info() print("Getting package details ...") @@ -974,7 +1013,7 @@ def __main__(): dump_html(packages, stats, date, commit, args.html) if args.json: print("Write JSON") - dump_json(packages, stats, date, commit, args.json) + dump_json(packages, defconfigs, stats, date, commit, args.json) __main__() From patchwork Mon Mar 2 14:50:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1247685 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=iT9wvGHI; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48WNPC4LWpz9sPK for ; Tue, 3 Mar 2020 01:50:55 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 5650C86174; Mon, 2 Mar 2020 14:50:51 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id g7vOoFINKvK7; Mon, 2 Mar 2020 14:50:47 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 7722185F9B; Mon, 2 Mar 2020 14:50:47 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 7C8DF1BF39F for ; Mon, 2 Mar 2020 14:50:46 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 7325F85F9B for ; Mon, 2 Mar 2020 14:50:46 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lVRTlbg4Oh6L for ; Mon, 2 Mar 2020 14:50:45 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f67.google.com (mail-wr1-f67.google.com [209.85.221.67]) by whitealder.osuosl.org (Postfix) with ESMTPS id 3950D8357B for ; Mon, 2 Mar 2020 14:50:44 +0000 (UTC) Received: by mail-wr1-f67.google.com with SMTP id z15so39329wrl.1 for ; Mon, 02 Mar 2020 06:50:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nKgjmAFM/r9ZpK3yJHqkR6dUo3I5ls2QWfO5NRTGJGU=; b=iT9wvGHIhOqtyxfHrKXKwOXPSMiG0GAhgZ9wxlmEuYgTGbsM9O03R1MqPXHbkhCRXu lkL2s5G2qKhOeHwJ9bZrryu1knt0scRDcd3hRHwczINLjdzstzJY1eQ2dwbl/c7bqCZD U3vNN4Qrqq4Sqs/0qjORQUS1OXMbirfnYnaGPfwlqxrGjR0j6RfEHDiwV3cFilCDistK mwSD450nsZKPZy54jfoFk94qIm8Jfnk3IHYFyn0K2A++K4PaQYX35GKsouBNEmR4KjwE upits2NTCZEMUVKHo/Y7eo6mm4AKy16lkfpJQuUfcW09qPRlUg0MxIinYb+F1pA8z+xe Gltw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nKgjmAFM/r9ZpK3yJHqkR6dUo3I5ls2QWfO5NRTGJGU=; b=Z7dMxDvYHsELqG/8c1WAFuO9uma0hBT/MqOF80wvqQB0QPrzkKFujiIGfo/DHM7RZ6 9WGSyqjQzAEGy2XtRxvygMzxD3A+fF1XuOTvQudM3GUuYiJ2M5kXss5GC/uCBd5g0yfj 2rmlWufYe7RAWD7LMNe+V7SXLr/zs4GpTqKaPjkD5KVvzLOvl9eC+POvGJOBMN3e0/RQ 0ldtTngFb8PBuYqYPyWW1jm8prfJcavT2BRn0a5auyJi/e79rqGhHhqDdzPOPW1jtGvw mwhuX0JSp/yEOHpMarWK9behYFqmwLhmeoO2DnCOmZxxoPDU1olkcrC/2akiCEqMKHE3 S8vQ== X-Gm-Message-State: ANhLgQ3AR76sFOSlAT/8Jlo1rMDSAiVDM9sj7EWd0FLfer0k9OuX92iS 5IVTxEvVWOhs5+efdYN1kVCOFqU7 X-Google-Smtp-Source: ADFU+vsll6xkLuFiPS3mhHZp6+TVT6FdFBw/i0l56JunHjUK6VKlaT1lrNbJCV5UdCVSDOuKRZYmrg== X-Received: by 2002:a5d:4b50:: with SMTP id w16mr4739wrs.230.1583160642543; Mon, 02 Mar 2020 06:50:42 -0800 (PST) Received: from hthiery01.sab.local ([213.135.10.150]) by smtp.gmail.com with ESMTPSA id h17sm29648355wro.52.2020.03.02.06.50.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Mar 2020 06:50:41 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Mon, 2 Mar 2020 15:50:11 +0100 Message-Id: <20200302145013.9574-10-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200302145013.9574-1-heiko.thiery@gmail.com> References: <20200302145013.9574-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 09/12] support/scripts/pkg-stats: add support for license hash check X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Store the names of license files and check if they are in the hash file. Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 432090c251..c9287b416e 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -85,7 +85,7 @@ def get_defconfig_list(): class Package: all_licenses = dict() - all_license_files = list() + all_license_files = dict() all_versions = dict() all_ignored_cves = dict() @@ -98,6 +98,7 @@ class Package: self.has_license = False self.has_license_files = False self.has_hash = False + self.license_files = None self.patch_files = [] self.warnings = 0 self.current_version = None @@ -159,6 +160,7 @@ class Package: self.license = self.all_licenses[var] self.status['license'] = ("ok", "found") if var in self.all_license_files: + self.license_files = self.all_license_files[var].split(' ') self.status['license-files'] = ("ok", "found") def set_hash_info(self): @@ -166,8 +168,18 @@ class Package: Fills in the .status['hash'] field """ hashpath = self.path.replace(".mk", ".hash") + self.status['hash-license'] = ("na", "no hash file") if os.path.exists(hashpath): self.status['hash'] = ("ok", "found") + self.status['hash-license'] = ("error", "no license in hash file") + # check if license files are in hash file + if self.license_files is not None: + self.status['hash-license'] = ("ok", "found") + with open(hashpath) as f: + content = f.read() + for license in self.license_files: + if content.find(license) == -1: + self.status['hash-license'] = ("error", "license missing in hash file") else: self.status['hash'] = ("error", "missing") @@ -449,7 +461,7 @@ def package_init_make_info(): if pkgvar.endswith("_MANIFEST_LICENSE_FILES"): continue pkgvar = pkgvar[:-14] - Package.all_license_files.append(pkgvar) + Package.all_license_files[pkgvar] = value elif pkgvar.endswith("_VERSION"): if pkgvar.endswith("_DL_VERSION"): From patchwork Mon Mar 2 14:50:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1247687 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=jzaxoqpE; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48WNPD6DTBz9sRR for ; Tue, 3 Mar 2020 01:50:56 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 23B9B8619A; Mon, 2 Mar 2020 14:50:55 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JnwsNnrQ3z3D; Mon, 2 Mar 2020 14:50:50 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id DF33A86117; Mon, 2 Mar 2020 14:50:49 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 0D40B1BF39F for ; Mon, 2 Mar 2020 14:50:48 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 0851D86096 for ; Mon, 2 Mar 2020 14:50:48 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 423ucolMI-X7 for ; Mon, 2 Mar 2020 14:50:45 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f68.google.com (mail-wr1-f68.google.com [209.85.221.68]) by whitealder.osuosl.org (Postfix) with ESMTPS id 33C82860A8 for ; Mon, 2 Mar 2020 14:50:45 +0000 (UTC) Received: by mail-wr1-f68.google.com with SMTP id q8so9728wrm.4 for ; Mon, 02 Mar 2020 06:50:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ik/E/X8zilHYacwrHfwx1Ixmy9paHB6Fr/9Pd7Wd/R4=; b=jzaxoqpEcc1G70g3nAdzOn33OE7eQcJtBmGXBkpsVhtUu/0onXp5nuUEO4JDKnB8Qk hzeChmqXHH6fc5gWHmGQqOMa+OT24Bc80XetniGPq2bwwXHOrdWyDmAOi5i7UbswTX7y q0bzbRkYc1byzy23Bwr6rvnGHDnDCmTGWQqsXCK1a7AP/LIc9nHmcS802znOf/6E8tlo LaSJPl5PLMCfsUEqyzvO1wiI/gadD40IKhXBJAGY3nuSAAwnD0NdACu607cOivOnugT2 oAJ6NVLzc81pzboq+vYO6YbOgeBYFtYwGxScoO2FJm30Ol1Y5Y8RdwXnh+B+k3hw+ogG Jx6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ik/E/X8zilHYacwrHfwx1Ixmy9paHB6Fr/9Pd7Wd/R4=; b=RMMXJWBrc4Fo2RDQRyIx/f6xBB0jK/J7NYeZsAN5DqBYOts5S9aOIXdwnU+rhRKk6P ygR7Trf4HgTOeWwufGnUCLTk5Z6PfzbU093nJj3HbRmRg9abCDm/+mmuw/zemeYugbn6 7AHfx/8Any56K3WqI91xdJ5GRIx0zjnRSlFU0J6mrzVC82oIoiLJz/hDr986x+r0PvAi oGm+UgladRBGSBJbDY20I22FShzLlGSfC2ieaLvcR9cbKwLu9lzQ0hpzoq/GFhB3XSAq WTZb+1cjHbkwZ3bCHDOv6g3kZ/6GlqeRCXhAe165PltOLGAXUC4riNI7XwWrQfHq8F+0 BLoA== X-Gm-Message-State: ANhLgQ0dg7aocyYPVZb+o+oTpdDgOhaU7HM6y3BaFg+bbj9DhORAVJG5 8BElYwQOxQtSz4ZqmbrpOepIfcE9 X-Google-Smtp-Source: ADFU+vtzYOF/wGgL7jpQ2p5jyGDao9b66h8QFEiyfUEaDcBxljwxa5OEYHDHw0IEx3IuBLqZMdIsIw== X-Received: by 2002:adf:ee85:: with SMTP id b5mr21847wro.34.1583160643374; Mon, 02 Mar 2020 06:50:43 -0800 (PST) Received: from hthiery01.sab.local ([213.135.10.150]) by smtp.gmail.com with ESMTPSA id h17sm29648355wro.52.2020.03.02.06.50.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Mar 2020 06:50:42 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Mon, 2 Mar 2020 15:50:12 +0100 Message-Id: <20200302145013.9574-11-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200302145013.9574-1-heiko.thiery@gmail.com> References: <20200302145013.9574-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 10/12] support/scripts/pkg-stats: set status to 'na' for virtual packages X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" If there is no infra set or infra is virtual the status is set to 'na'. This is done for the follwing checks: - license - license-files - hash - hash-license - patches - version Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index c9287b416e..4cf1a919f3 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -132,6 +132,15 @@ class Package: def patch_count(self): return len(self.patch_files) + @property + def has_valid_infra(self): + try: + if self.infras[0][1] == 'virtual': + return False + except IndexError: + return False + return True + def set_infra(self): """ Fills in the .infras field @@ -153,6 +162,11 @@ class Package: """ Fills in the .status['license'] and .status['license-files'] fields """ + if not self.has_valid_infra: + self.status['license'] = ("na", "no valid package infra") + self.status['license-files'] = ("na", "no valid package infra") + return + var = self.pkgvar() self.status['license'] = ("error", "missing") self.status['license-files'] = ("error", "missing") @@ -167,6 +181,11 @@ class Package: """ Fills in the .status['hash'] field """ + if not self.has_valid_infra: + self.status['hash'] = ("na", "no valid package infra") + self.status['hash-license'] = ("na", "no valid package infra") + return + hashpath = self.path.replace(".mk", ".hash") self.status['hash-license'] = ("na", "no hash file") if os.path.exists(hashpath): @@ -187,6 +206,10 @@ class Package: """ Fills in the .patch_count, .patch_files and .status['patches'] fields """ + if not self.has_valid_infra: + self.status['patches'] = ("na", "no valid package infra") + return + pkgdir = os.path.dirname(self.path) for subdir, _, _ in os.walk(pkgdir): self.patch_files = fnmatch.filter(os.listdir(subdir), '*.patch') @@ -564,6 +587,10 @@ def check_package_latest_version(packages): for pkg, r in zip(packages, results): pkg.latest_version = dict(zip(['status', 'version', 'id'], r)) + if not pkg.has_valid_infra: + pkg.status['version'] = ("na", "no valid package infra") + continue + if pkg.latest_version['status'] == RM_API_STATUS_ERROR: pkg.status['version'] = ('warning', "Release Monitoring API error") elif pkg.latest_version['status'] == RM_API_STATUS_NOT_FOUND: From patchwork Mon Mar 2 14:50:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1247688 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=nN2U/Mk6; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48WNPK2Pz4z9sPK for ; Tue, 3 Mar 2020 01:51:01 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 7180886138; Mon, 2 Mar 2020 14:50:59 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QGrs7SLAFiJz; Mon, 2 Mar 2020 14:50:56 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 8F90C86110; Mon, 2 Mar 2020 14:50:54 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 0F2B31BF973 for ; Mon, 2 Mar 2020 14:50:52 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id F40B6204A6 for ; Mon, 2 Mar 2020 14:50:51 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tQRF1LT42w8U for ; Mon, 2 Mar 2020 14:50:50 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f51.google.com (mail-wm1-f51.google.com [209.85.128.51]) by silver.osuosl.org (Postfix) with ESMTPS id 0A08E204A9 for ; Mon, 2 Mar 2020 14:50:47 +0000 (UTC) Received: by mail-wm1-f51.google.com with SMTP id 6so3055572wmi.5 for ; Mon, 02 Mar 2020 06:50:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=N85N//6s83vTtGmoFbzdZh3veLCiyTyiyARNubiFiGc=; b=nN2U/Mk6x4P2t9+PEkL7OzDrWDXTakxF10LKbQ/Zr6QvjKTzZ6pwQsdmMeC9PwAj9v +4q5V+2/HorsDou52b0MPCK2PkrLltgNQ7WshXKyvYrlma6DOTV9N0GX78WZ4MKuYkU1 +4ekCDvvoyuyu7oLFRnLR/qt1S2826+qjvsuBELS1teWV8toy22HvlLLHH9NqdYdVjLT cr6Qro8xyl/CxD5DDmcUzNVePw9hLwfJU6f8Qa9fRRRzJsJIPKjhFv50sAmH0KFH6gjM UEBTzxzNhMpK6yNdP/HTdvA1s9sdvgyO90hBIdlDAaXMcsPhWQCOVIXu7oVHdYU3ZzUc v/BA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=N85N//6s83vTtGmoFbzdZh3veLCiyTyiyARNubiFiGc=; b=Yj1BosWGfmjvUAjSUHg/9i6lw5eEIBKNmJLsB7bA23JoiQfMtqVcgUI5Jl7n9NPWFI ETcEkuzIMkF2XmsCZqWBKcrrCysCcjg3LcdyCGy+B0fuinTUcZKfqZy5MzGPrWG3AoUc Zw/iFs7tCyWvIHi0xLeH6uBCTxxuM1jlqPDsaFVeoTDaFZ/Clcjtgy6Qsn3jiCLIbpGJ kt/CWPRArmxeTDNA/8bttKBP8Z69wjyBNiqs4nDsSySnNY5tT4H2ofg5M66yBHNftGwJ QkvlLIlNUSmQI4gduizFYIkUBmvBINw1+uC7dHRqCmlxAo30/BSjzlEOQTtaqpVPJB7a K+rw== X-Gm-Message-State: APjAAAXXejfTzFRwwwqF2mW5REPSF7mZw5z+7W8QHtvziousnPj5NQ7u mu+QV1sC/vX3RdnCf71zKZ4lNnoY X-Google-Smtp-Source: APXvYqyfnJua8J339pujdzRoemwpJXy1lJDJHsgjQJf8cOn/qiAivwlJGUmAVlOPQLaRsfw3KPTkLA== X-Received: by 2002:a1c:a382:: with SMTP id m124mr19831699wme.90.1583160644181; Mon, 02 Mar 2020 06:50:44 -0800 (PST) Received: from hthiery01.sab.local ([213.135.10.150]) by smtp.gmail.com with ESMTPSA id h17sm29648355wro.52.2020.03.02.06.50.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Mar 2020 06:50:43 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Mon, 2 Mar 2020 15:50:13 +0100 Message-Id: <20200302145013.9574-12-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200302145013.9574-1-heiko.thiery@gmail.com> References: <20200302145013.9574-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 11/12] support/scripts/pkg-stats: add list of status checks to the json output X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 4cf1a919f3..9d058ca67d 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -88,6 +88,11 @@ class Package: all_license_files = dict() all_versions = dict() all_ignored_cves = dict() + # This is the list of all possible checks. Add new checks to this list so + # a tool that post-processeds the json output knows the checks before + # iterating over the packages. + status_checks = ['cve', 'developers', 'hash', 'hash-license', 'license', + 'license-files', 'patches', 'pkg-check', 'url', 'version'] def __init__(self, name, path): self.name = name @@ -981,6 +986,7 @@ def dump_json(packages, defconfigs, stats, date, commit, output): final = {'packages': pkgs, 'stats': statistics, 'defconfigs': defconfigs, + 'package_status_checks': Package.status_checks, 'commit': commit, 'date': str(date)} From patchwork Mon Mar 2 14:50:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1247686 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=WqHbNwpk; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48WNPC5xgkz9sRN for ; Tue, 3 Mar 2020 01:50:55 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 26240204A5; Mon, 2 Mar 2020 14:50:54 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BizAeCkig1eZ; Mon, 2 Mar 2020 14:50:53 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 257F3204A6; Mon, 2 Mar 2020 14:50:53 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id DA4A01BF39F for ; Mon, 2 Mar 2020 14:50:51 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id CCB43204A6 for ; Mon, 2 Mar 2020 14:50:51 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BxpK9RSRcOoP for ; Mon, 2 Mar 2020 14:50:50 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f68.google.com (mail-wr1-f68.google.com [209.85.221.68]) by silver.osuosl.org (Postfix) with ESMTPS id DDEB2204A7 for ; Mon, 2 Mar 2020 14:50:46 +0000 (UTC) Received: by mail-wr1-f68.google.com with SMTP id r7so29442wro.2 for ; Mon, 02 Mar 2020 06:50:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=qpG8vvJmrG1UHMhZthxQGVj2V1gCkKfYIaRY7vmO+Hw=; b=WqHbNwpkW9zCvQV3ksJZ5In8i0LgWSz5KW6E02LMkIkleVoHXBXVw9Kx4QcTvjxGyh Xrj0yMM2RY/eA8wGarejg+LjD7QBri0G6MUsoPcESCNrMtA424VC6MIT2i6nSLVP15fP Aw0VXkKb3KNC7ZHNZugO290SwKBbixp3Fs0mfqimzIh5ryEDn1k2kmlvuu8dfaz2rJXb Zv+SBoIFqOAUccb90k5zI4Ti40ulsEaAmaBXb7sGiEBNzgp2n9rM6DyY0oOe3UYcJy5B FKbWPR6mQ7mj1y3BKjRbaybqaAVTdlaFa/YS8EL1qMA75vDGCLId2KqOAJRpuVTKHauf GoNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qpG8vvJmrG1UHMhZthxQGVj2V1gCkKfYIaRY7vmO+Hw=; b=CfpD26Hqh99lwNHKvJ2IqQ+SxDIUkccXia+uVrPA2ht0Sb3wBhnxpe6HMp2IF0hyPh 3WLUT9YCt/pTRlvLoe9Kd02Id7goZQpTV+DJqws68fSc9vEnqlvlah6U3XFzdYpxl+HZ 4xLwQUaN8Gl0YimKn1FZudmpLf80GEsLYHGl06leGrOjFtHNO8O1X0dWN550PPEQmnKj HcZr3Qv3gKNqMDYilPFyBdWjR/qt7QlYKz/HcoFbJhK+0wWR58XXlr+d3dQ8gQM1TX2V 1ojg1U86chtHXrdH41SU6MnGvEkJ69PrkQqjyj1o8/gkF+SAcPzDZr+TtSfTwaqbYsU1 BCKg== X-Gm-Message-State: ANhLgQ01z2MR+BC1deLMizbpKbfIwBbMAmiJh69CiSTXdVM3oSyXntL2 dQbxiUQOaNdXI5+KiDn9e/TpnO77 X-Google-Smtp-Source: ADFU+vusQuDJrF0uKpEPBJYeZA60aOkMDpdgoBiA8WVms3WP8jkSTNZP0OKPrlqL4brFkKoGOTziUA== X-Received: by 2002:adf:ea87:: with SMTP id s7mr24030wrm.12.1583160645180; Mon, 02 Mar 2020 06:50:45 -0800 (PST) Received: from hthiery01.sab.local ([213.135.10.150]) by smtp.gmail.com with ESMTPSA id h17sm29648355wro.52.2020.03.02.06.50.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Mar 2020 06:50:44 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Mon, 2 Mar 2020 15:50:14 +0100 Message-Id: <20200302145013.9574-13-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200302145013.9574-1-heiko.thiery@gmail.com> References: <20200302145013.9574-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v4 12/12] support/scripts/pkg-stats: add status for cve check X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 9d058ca67d..b0ae681c22 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -619,6 +619,12 @@ def check_package_cves(nvd_path, packages): if pkg_name in packages and cve.affects(packages[pkg_name]): packages[pkg_name].cves.append(cve.identifier) + for pkg_name, pkg in packages.items(): + if len(pkg.cves) > 0: + pkg.status['cve'] = ('error', 'affected by cve') + else: + pkg.status['cve'] = ('ok', 'no cve found') + def calculate_stats(packages): stats = defaultdict(int)