From patchwork Fri Jan 3 15:18:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1217354 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="aFkTn6Kw"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 47q7pz0QpJz9sPn for ; Sat, 4 Jan 2020 02:19:06 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 5431C849BE; Fri, 3 Jan 2020 15:19:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RCCMQ620BM6T; Fri, 3 Jan 2020 15:19:01 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 1269F859FC; Fri, 3 Jan 2020 15:19:01 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 37DB71BF599 for ; Fri, 3 Jan 2020 15:19:00 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 33B7A20445 for ; Fri, 3 Jan 2020 15:19:00 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OwqG2J7C2jWF for ; Fri, 3 Jan 2020 15:18:59 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) by silver.osuosl.org (Postfix) with ESMTPS id D9E202036D for ; Fri, 3 Jan 2020 15:18:58 +0000 (UTC) Received: by mail-wm1-f48.google.com with SMTP id d73so8686763wmd.1 for ; Fri, 03 Jan 2020 07:18:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fzAmMYi6EkRW0NV1fQDcCyGSzR8btDUr9CMvGw1uNL8=; b=aFkTn6KwsNWByk+SwW+5Mkw24OgNp2NQZkzN4x+NHqjNQDZ77f2XIur6sy9Uy5CiJw R5WCpk23Vf2I2cxgAExxCuvrchQjZ+HqyRcJFK3BNjlgKKnoSwHrJK1DX74mXU6Wnce4 F7pyU5EDm9TPxBHCQY3qYoomMK6m48OgIEauxMgjzk3tCA3m1G75wAby4CJ+cFnx55hS IU/xmtAKnJRRpDsH8vL84JLo9JrR1PKqlJyTJyTgGc0v/uw2X4JTp6RG3XQfRY3FL4bU DjH3bDTt/yo5LNR9bW1J6PvcL+/WALUJ/cEZLcGmgEzbx649ubUy2oyURoUyC6ARca0D /JUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fzAmMYi6EkRW0NV1fQDcCyGSzR8btDUr9CMvGw1uNL8=; b=Fx+1OVzlsCX01tHWjNi0jBrUAICfQpDQ+/T4Lpo83P1R8A4UjdMOa/5cKm0Xl1hFtF W5tfvcNmIVi+ACNVWucNO4nVNy+ie57crr/5aZHhZVsLOnodZv6JeFiLc5x4LskoDYxW p4866fT2vm/F3R69k1wOJnh89arOX5UZZP4C7C2R2+zvLb1scjdLmxiHW1iZ6S44C/Qp ceRY30VOTUFI5tCohtMnX5bZfHFCvvZ0HBb6Ed1SOEe062n4udH/YWfaoJtfnYffwLGH 4tleHk55PDwwJszlbPOyiKqdBQndxU5JqSltLqsIArAVRxhDSyDt/7HnVKvofkkyZ6Kx J02Q== X-Gm-Message-State: APjAAAUfvVmw2k8IKJ6hHJBf1vmOqRlZAbLxE9pOeD6OnMQRaoZEW++S wPVZk7MYGvjFPzls8rhQxl4luzLm X-Google-Smtp-Source: APXvYqzDOCOx2UngfV30XW05ubjajs7Ylgfk0A/PNkAXkmnDMQ+KuNb4kek9Z1QC0DY2Iifr4YkxXA== X-Received: by 2002:a1c:6406:: with SMTP id y6mr20340840wmb.144.1578064737102; Fri, 03 Jan 2020 07:18:57 -0800 (PST) Received: from hthiery01.sab.local ([213.135.10.150]) by smtp.gmail.com with ESMTPSA id u16sm12460565wmj.41.2020.01.03.07.18.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Jan 2020 07:18:56 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Fri, 3 Jan 2020 16:18:38 +0100 Message-Id: <20200103151849.10956-2-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200103151849.10956-1-heiko.thiery@gmail.com> References: <20200103151849.10956-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 01/11] support/scripts/pkg-stats: store latest version in a hash X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This patch changes the name and the variable from latest_verion of type list to rm_status of type hash. This is for better readability/usability of the data. With this the json output is more descriptive. Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 35 ++++++++++++++++++----------------- 1 file changed, 18 insertions(+), 17 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index e477828f7b..8c64993aaf 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -61,7 +61,7 @@ class Package: self.url = None self.url_status = None self.url_worker = None - self.latest_version = (RM_API_STATUS_ERROR, None, None) + self.rm_version = {'status': RM_API_STATUS_ERROR, 'version': None, 'id': None} def pkgvar(self): return self.name.upper().replace("-", "_") @@ -331,11 +331,10 @@ def check_package_latest_version_worker(name): def check_package_latest_version(packages): """ - Fills in the .latest_version field of all Package objects + Fills in the .rm_version field of all Package objects + + This field is a dict and has the following keys: - This field has a special format: - (status, version, id) - with: - status: one of RM_API_STATUS_ERROR, RM_API_STATUS_FOUND_BY_DISTRO, RM_API_STATUS_FOUND_BY_PATTERN, RM_API_STATUS_NOT_FOUND @@ -351,7 +350,9 @@ def check_package_latest_version(packages): worker_pool = Pool(processes=64) results = worker_pool.map(check_package_latest_version_worker, (pkg.name for pkg in packages)) for pkg, r in zip(packages, results): - pkg.latest_version = r + pkg.rm_version['status'] = r[0] + pkg.rm_version['version'] = r[1] + pkg.rm_version['id'] = r[2] del http_pool @@ -379,13 +380,13 @@ def calculate_stats(packages): stats["hash"] += 1 else: stats["no-hash"] += 1 - if pkg.latest_version[0] == RM_API_STATUS_FOUND_BY_DISTRO: + if pkg.rm_version['status'] == RM_API_STATUS_FOUND_BY_DISTRO: stats["rmo-mapping"] += 1 else: stats["rmo-no-mapping"] += 1 - if not pkg.latest_version[1]: + if not pkg.rm_version['version']: stats["version-unknown"] += 1 - elif pkg.latest_version[1] == pkg.current_version: + elif pkg.rm_version['version'] == pkg.current_version: stats["version-uptodate"] += 1 else: stats["version-not-uptodate"] += 1 @@ -548,29 +549,29 @@ def dump_html_pkg(f, pkg): f.write(" %s\n" % current_version) # Latest version - if pkg.latest_version[0] == RM_API_STATUS_ERROR: + if pkg.rm_version['status'] == RM_API_STATUS_ERROR: td_class.append("version-error") - if pkg.latest_version[1] is None: + if pkg.rm_version['version'] is None: td_class.append("version-unknown") - elif pkg.latest_version[1] != pkg.current_version: + elif pkg.rm_version['version'] != pkg.current_version: td_class.append("version-needs-update") else: td_class.append("version-good") - if pkg.latest_version[0] == RM_API_STATUS_ERROR: + if pkg.rm_version['status'] == RM_API_STATUS_ERROR: latest_version_text = "Error" - elif pkg.latest_version[0] == RM_API_STATUS_NOT_FOUND: + elif pkg.rm_version['status'] == RM_API_STATUS_NOT_FOUND: latest_version_text = "Not found" else: - if pkg.latest_version[1] is None: + if pkg.rm_version['version'] is None: latest_version_text = "Found, but no version" else: latest_version_text = "%s" % \ - (pkg.latest_version[2], str(pkg.latest_version[1])) + (pkg.rm_version['id'], str(pkg.rm_version['version'])) latest_version_text += "
" - if pkg.latest_version[0] == RM_API_STATUS_FOUND_BY_DISTRO: + if pkg.rm_version['status'] == RM_API_STATUS_FOUND_BY_DISTRO: latest_version_text += "found by distro" else: latest_version_text += "found by guess" From patchwork Fri Jan 3 15:18:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1217355 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="AuveAEkE"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 47q7q50gXvz9sR0 for ; Sat, 4 Jan 2020 02:19:12 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id C058685F83; Fri, 3 Jan 2020 15:19:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KaIqVcNl5zDv; Fri, 3 Jan 2020 15:19:04 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 8B17A85F60; Fri, 3 Jan 2020 15:19:04 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 797141BF599 for ; Fri, 3 Jan 2020 15:19:01 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 7674286970 for ; Fri, 3 Jan 2020 15:19:01 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PO1r3M2w5l7S for ; Fri, 3 Jan 2020 15:19:00 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f68.google.com (mail-wm1-f68.google.com [209.85.128.68]) by whitealder.osuosl.org (Postfix) with ESMTPS id CF7EC836F1 for ; Fri, 3 Jan 2020 15:18:59 +0000 (UTC) Received: by mail-wm1-f68.google.com with SMTP id p9so8689315wmc.2 for ; Fri, 03 Jan 2020 07:18:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Jl4DbhcXMR88UyaonTLuwNDvY1eCJwiG9aY4cm5EN8w=; b=AuveAEkE/wzYQ2NGYpmIN+8CjzXZSVLxLD5O6xcNmeD5Ur7b9BaVl1scDQaQwzqa1B 7Rn/d0c/iP7DdAYth9kTwGvAASXKoltCMB1GHK2lTbWOSHYKrx06eB3Xrd5PO5hBDisx UHJJnnu1YFvS/AWX01dQnyj3h2FkZL4fjKswGN7LuVy38cV6cexOVu+uV5AZ2qCFtOuo 82vpZQhSpDJOrKrqONpVk7ft1OxA9Xw/S7zXi06dZDgmlrpWOtV5R4uMoGwoFHTXH3QY HdmF35tFD5vRMIPEmALjstWmrYpHnH0SkdLR2ehnZ4UQP74hftVfudxxUjHy63/Hyt3a X2jw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Jl4DbhcXMR88UyaonTLuwNDvY1eCJwiG9aY4cm5EN8w=; b=UUoyK60D1tWuFDl8b8Jpx7tpn4WIkCtPY3msOgH6Th/RqODgi91T3u/RY3iQylYsij nECiW2LGmhi6Vo2NUTiF6ce1MJfI8Irf0Rlng7TQhDqUqE0yyXJthz5OV0ZZ84SZHtky ZGqyVnV1Zi3Q49hh4ubfm36OYhLLKKBHM6ImmSGfun3kyMn7wF9EiXl/MROHjuEkSVsP m0nMW0wYPxNyvOXifvAjQUQvjvgsMcd/ylPp832BOU6IAUmMxRVkO+WmsnRMGVJsdiL6 0dLbs9Zlu/atf9DOXbLN4OQmZlUXriaPTIm4dYFdkcnh0BT9MSqkrcwaTasjbjTNbUC0 pD4Q== X-Gm-Message-State: APjAAAWgd3yx5pZgjsQH1HHzmHIzK8GHxT5/V2oDrnOM9O2OLoY7xeIy nueaENYjD03JpSIVDuB7507czQ// X-Google-Smtp-Source: APXvYqwiKgLAZ3dB3EPxCIh+JqPgYiABAJlVP2AK3RM23SD/cF4YuC+C2oYK6BQBlc0s+uJJtmQnPA== X-Received: by 2002:a7b:c183:: with SMTP id y3mr20577746wmi.0.1578064738085; Fri, 03 Jan 2020 07:18:58 -0800 (PST) Received: from hthiery01.sab.local ([213.135.10.150]) by smtp.gmail.com with ESMTPSA id u16sm12460565wmj.41.2020.01.03.07.18.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Jan 2020 07:18:57 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Fri, 3 Jan 2020 16:18:39 +0100 Message-Id: <20200103151849.10956-3-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200103151849.10956-1-heiko.thiery@gmail.com> References: <20200103151849.10956-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 02/11] support/scripts/pkg-stats: store pkg dir path X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Use this value where it make sense. Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 8c64993aaf..9cfbcf1acc 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -51,6 +51,7 @@ class Package: def __init__(self, name, path): self.name = name self.path = path + self.pkg_path = os.path.dirname(path) self.infras = None self.has_license = False self.has_license_files = False @@ -71,9 +72,9 @@ class Package: Fills in the .url field """ self.url_status = "No Config.in" - for filename in os.listdir(os.path.dirname(self.path)): + for filename in os.listdir(self.pkg_path): if fnmatch.fnmatch(filename, 'Config.*'): - fp = open(os.path.join(os.path.dirname(self.path), filename), "r") + fp = open(os.path.join(self.pkg_path, filename), "r") for config_line in fp: if URL_RE.match(config_line): self.url = config_line.strip() @@ -88,7 +89,7 @@ class Package: Fills in the .infras field """ self.infras = list() - with open(self.path, 'r') as f: + with open(os.path.join(self.pkg_path, self.name + '.mk'), 'r') as f: lines = f.readlines() for l in lines: match = INFRA_RE.match(l) @@ -139,8 +140,7 @@ class Package: Fills in the .warnings field """ cmd = ["./utils/check-package"] - pkgdir = os.path.dirname(self.path) - for root, dirs, files in os.walk(pkgdir): + for root, dirs, files in os.walk(self.pkg_path): for f in files: if f.endswith(".mk") or f.endswith(".hash") or f == "Config.in" or f == "Config.in.host": cmd.append(os.path.join(root, f)) @@ -732,6 +732,8 @@ def __main__(): 'HEAD']).splitlines()[0] print("Build package list ...") packages = get_pkglist(args.npackages, package_list) + print("Getting developers...") + developers = parse_developers() print("Getting package make info ...") package_init_make_info() print("Getting package details ...") From patchwork Fri Jan 3 15:18:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1217356 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="FaDOFUSB"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 47q7q54Jj0z9sRW for ; Sat, 4 Jan 2020 02:19:13 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 5003C21FA8; Fri, 3 Jan 2020 15:19:10 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bISa-pF2p5Gt; Fri, 3 Jan 2020 15:19:06 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 10CB020445; Fri, 3 Jan 2020 15:19:06 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 5815E1BF599 for ; Fri, 3 Jan 2020 15:19:02 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 54AD620445 for ; Fri, 3 Jan 2020 15:19:02 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NAosLsx2hSS2 for ; Fri, 3 Jan 2020 15:19:01 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f42.google.com (mail-wr1-f42.google.com [209.85.221.42]) by silver.osuosl.org (Postfix) with ESMTPS id 53C772036D for ; Fri, 3 Jan 2020 15:19:01 +0000 (UTC) Received: by mail-wr1-f42.google.com with SMTP id y17so42781911wrh.5 for ; Fri, 03 Jan 2020 07:19:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nD75AeRFSX+JC7ffpzS/BXqPWtPEY325AXgAd/t3N34=; b=FaDOFUSBDRbAIDjgOprPAB2QaLSz2Lmby5678ydKoan4T4wm+f51ZtWsLez1DWCJc5 xY77ggjybkssiaUs6XJnMpxOMN1t5tI/UtG8ElEKl1tHf6bd/H5F0INueZgAAS+DEjQX fD4hpK9+BI+GBAn9zcqbsoNtxA7dzJlNW6A4feVrilIZ2xsYEl/WScoogViba5FdZdkH J5vlkhfHdPGZod5s5osNpkx0xyY8HcB7hQ7112LhdOcf64js37e+6ADYboAlXp9DYS6V O17rCZPKhxaGHnHTGlC9e/b5aaz3HPKNbfYRQ2JSpOWYd2kynmCa4cYcNcbnZAdhDX5w pGIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nD75AeRFSX+JC7ffpzS/BXqPWtPEY325AXgAd/t3N34=; b=WLCBARQWlXQSMQjXk82gnbXLpUgKhi1kWy2xldNsPDV9KxaMhMdfYl/PHCis4fkFHv qGl1M1wTorCAaOt+wsiTyP52np8/GmndaDbQ3qpL6F4BqoELaq5vX+J/dosgjdPWWOhX ONWdgqFEiMLk2KjW/h6aIqsxwCsGIPAUAx+toB/ClYmTXiwoJ1axKrCSVSRWNYZTIh66 Sj/YRHKX4nOtnVZ0Jg4mzHJ1c0fQ5SB4H5fcStOmcWZd5TNRlPxO818NKxVtBXNTaLby K7ZoHmKRxUqKPi93uk40VRspPXvsUbNu/jYAs5O3NGqRh1dUNur+Rekg+X7s3hgi0q4a 5ReQ== X-Gm-Message-State: APjAAAWZn4u2jX8Xs2ahx/KlPPwhH5optSQTt6Lj5JyxH4OWCkea6VUQ NYzSD33M/VPFx7Hjinih/j54qJ5e X-Google-Smtp-Source: APXvYqzJep/zfDZsCAaABFpXEb3dpBo/e/iqWigjw3wjjbl5QcDcuPa9fO6APF8a88KhRkTxNZ0WhQ== X-Received: by 2002:a5d:44cd:: with SMTP id z13mr88900272wrr.104.1578064739538; Fri, 03 Jan 2020 07:18:59 -0800 (PST) Received: from hthiery01.sab.local ([213.135.10.150]) by smtp.gmail.com with ESMTPSA id u16sm12460565wmj.41.2020.01.03.07.18.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Jan 2020 07:18:58 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Fri, 3 Jan 2020 16:18:40 +0100 Message-Id: <20200103151849.10956-4-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200103151849.10956-1-heiko.thiery@gmail.com> References: <20200103151849.10956-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 03/11] support/scripts/pkg-stats: store patch info in a hash X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This patch changes the variable to store the patch count to a hash. This variable holds the patch count of the packages as well a list of the patch file names. Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 9cfbcf1acc..92fc01d655 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -56,7 +56,7 @@ class Package: self.has_license = False self.has_license_files = False self.has_hash = False - self.patch_count = 0 + self.patches = {'count':0, 'files': None} self.warnings = 0 self.current_version = None self.url = None @@ -115,17 +115,16 @@ class Package: """ Fills in the .has_hash field """ - hashpath = self.path.replace(".mk", ".hash") + hashpath = os.path.join(self.pkg_path, self.name + '.hash') self.has_hash = os.path.exists(hashpath) def set_patch_count(self): """ Fills in the .patch_count field """ - self.patch_count = 0 - pkgdir = os.path.dirname(self.path) - for subdir, _, _ in os.walk(pkgdir): - self.patch_count += len(fnmatch.filter(os.listdir(subdir), '*.patch')) + for subdir, _, _ in os.walk(self.pkg_path): + self.patches['files'] = fnmatch.filter(os.listdir(subdir), '*.patch') + self.patches['count'] = len(self.patches['files']) def set_current_version(self): """ @@ -160,7 +159,7 @@ class Package: def __str__(self): return "%s (path='%s', license='%s', license_files='%s', hash='%s', patches=%d)" % \ - (self.name, self.path, self.has_license, self.has_license_files, self.has_hash, self.patch_count) + (self.name, self.path, self.has_license, self.has_license_files, self.has_hash, self.patches['count']) def get_pkglist(npackages, package_list): @@ -390,7 +389,7 @@ def calculate_stats(packages): stats["version-uptodate"] += 1 else: stats["version-not-uptodate"] += 1 - stats["patches"] += pkg.patch_count + stats["patches"] += pkg.patches['count'] return stats @@ -495,14 +494,14 @@ def dump_html_pkg(f, pkg): # Patch count td_class = ["centered"] - if pkg.patch_count == 0: + if pkg.patches['count'] == 0: td_class.append("nopatches") - elif pkg.patch_count < 5: + elif pkg.patches['count'] < 5: td_class.append("somepatches") else: td_class.append("lotsofpatches") f.write(" %s\n" % - (" ".join(td_class), str(pkg.patch_count))) + (" ".join(td_class), str(pkg.patches['count']))) # Infrastructure infra = infra_str(pkg.infras) From patchwork Fri Jan 3 15:18:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1217358 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="FoOdqjqQ"; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 47q7q70cGBz9sRm for ; Sat, 4 Jan 2020 02:19:14 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id D23BA879F0; Fri, 3 Jan 2020 15:19:09 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id abXxO94Xna6B; Fri, 3 Jan 2020 15:19:07 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 179B987A39; Fri, 3 Jan 2020 15:19:07 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 785901BF9BA for ; Fri, 3 Jan 2020 15:19:03 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 74C1D20445 for ; Fri, 3 Jan 2020 15:19:03 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TcrACIrN1koB for ; Fri, 3 Jan 2020 15:19:03 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f50.google.com (mail-wr1-f50.google.com [209.85.221.50]) by silver.osuosl.org (Postfix) with ESMTPS id B275B2036D for ; Fri, 3 Jan 2020 15:19:02 +0000 (UTC) Received: by mail-wr1-f50.google.com with SMTP id g17so42734270wro.2 for ; Fri, 03 Jan 2020 07:19:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=G3XAvVxomTxDpxWp6TM2AzCT6J4QNq1Oj2PPonQsr68=; b=FoOdqjqQ9bHZ2+PtcYnASwVVFSbOkbQD2d5y2kuW0EFb7gnet9EIM13KS8xMjgQ9F9 IwG5ZMukZGgUsD+ujYesqMHY2pTA/6aR8XJH6pyBEfvCPpfneMsFUnct+sVQZvqkg05V hkyoM0NqlObn5u00OyjNbPg4s+UcpLAtONOf93PzldPjOZ13hsqfR+uVHttwo6ugIvcF lHkR1x2Q67XeVgcuIFo39JKEO0OVe4suUISmtp3PwATnJvsm/web/geDvAtE9w9YCOXs w01XjiwlJmMzw2KkR1h68XAWBjW0bzOWbh8j34KUjXmi3VMgucH9XYzd6IrVEJ2ZSh1o qSZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=G3XAvVxomTxDpxWp6TM2AzCT6J4QNq1Oj2PPonQsr68=; b=tKd79CCcNbzQvnmHAbIm1/n6FUPCSAdz68mkQkN/vQnntYzcTgGmtfapJvdnc1bKsT zbFySWxabO9J/uK6I3FsW5O3o3cvXGgV0KEekoPafULOIZ8zzldDjuHMeEH+3JO6tXFB a7CquYeX91LL6rkNeBI0BEaozTMyTJU9A4DbZgNp3JlamNkfeXEdPW92kiMZ4oiw7S7l z6lPercqH3hfje/qVXuOWAiZvaSgs08MgxcoBASworzZ6De01xR9R5OpkkCvy4NIurij DuqR2KGEjVgiSii3XctS9lsGI9k15kU78GGuMgP/tPKQ+mWoFTUowdy1CYzAfHoW4Sqr CNjQ== X-Gm-Message-State: APjAAAVwGuxKvtnlNneRHqfEj66nmA8KwKiYrlk8PdmCgM2edL5j/0zg gSKDDPhPTNrX0bb8a5fLDAD3QaLV X-Google-Smtp-Source: APXvYqxnXWsR+FfS6fJMYBFF9DX8EenburDSIObLy9/3m5oHoHnIX4Vn+SeUudaPf2Q3gtL5PmXbGQ== X-Received: by 2002:adf:d4ca:: with SMTP id w10mr83476860wrk.53.1578064741054; Fri, 03 Jan 2020 07:19:01 -0800 (PST) Received: from hthiery01.sab.local ([213.135.10.150]) by smtp.gmail.com with ESMTPSA id u16sm12460565wmj.41.2020.01.03.07.18.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Jan 2020 07:19:00 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Fri, 3 Jan 2020 16:18:41 +0100 Message-Id: <20200103151849.10956-5-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200103151849.10956-1-heiko.thiery@gmail.com> References: <20200103151849.10956-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 04/11] support/scripts/pkg-stats: do not exclued pkg name in json output X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 92fc01d655..4021aacceb 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -676,7 +676,7 @@ def dump_html(packages, stats, date, commit, output): def dump_json(packages, stats, date, commit, output): # Format packages as a dictionnary instead of a list # Exclude local field that does not contains real date - excluded_fields = ['url_worker', 'name'] + excluded_fields = ['url_worker'] pkgs = { pkg.name: { k: v From patchwork Fri Jan 3 15:18:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1217361 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="n+NvmUbn"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 47q7qB742nz9sR0 for ; Sat, 4 Jan 2020 02:19:18 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id BA3BE21577; Fri, 3 Jan 2020 15:19:16 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dGdZNii1k8cn; Fri, 3 Jan 2020 15:19:09 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 18CC721561; Fri, 3 Jan 2020 15:19:08 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 401E31BF599 for ; Fri, 3 Jan 2020 15:19:05 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 39E7F84821 for ; Fri, 3 Jan 2020 15:19:05 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HKkW4zxxgpFO for ; Fri, 3 Jan 2020 15:19:04 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f52.google.com (mail-wr1-f52.google.com [209.85.221.52]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 2922B85F4B for ; Fri, 3 Jan 2020 15:19:04 +0000 (UTC) Received: by mail-wr1-f52.google.com with SMTP id y11so42740650wrt.6 for ; Fri, 03 Jan 2020 07:19:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=14x5hogCwl8p5pV9FXBgFT/DFZ/NylxuYq9U+Gp7lQ8=; b=n+NvmUbn+xtUDv9QodJupuRlldI35goqXhzBtZHEIoKUQtovF8yy1oSe0OBvPvHKmo MjOZpFPZTc9cUEH3CbZgdRmqzDWP5VZTAVDThxANp32XYzRREb+JrZ8cwhh8suy4fijx nr9llRzXd5RRI2urlIFnR65P70xXgmdZ4fBqCsz+MY7iaMAHXMZ7p9jaN4wf4f9X7648 2ToOwJl6DqzHrPbxsRc2yTYoLND3boRYSOGeZr+/RDlx0JUkSyw0kKS0sy2+tlApKpH4 j7UHMtTDtPQJS40aB9QYZ3hp/UwICDaAS+sctQMpqO7EVSFyl6l4w2gkOd1h1h0s15BS trXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=14x5hogCwl8p5pV9FXBgFT/DFZ/NylxuYq9U+Gp7lQ8=; b=iyQTlWsLQM/ejkuejAW8E3aadoMszIi9HqE4i0J/wQzHio7YDfN5tS9iJD3WFqF7Qk s56vmN69hV1cQ9Aiy0oZszu3B7/DPwVajqbYFdWg0gm38w5DapgT2140z3zwpXYpql/H ky/18h1pG3hfduCu+XpujTkUnAC8uBr3mOcdgTozskPDaRqu2nrswSo9czuEoCIfi3s6 AJsP8YkpShJi3ecVdathUq8Vut+yuIHkaN6wcdU1E+4AjywWZKfhwhSIme7wp03VG270 OCoSlEgbWHaCBGkCoSwkmaXOUitR4+etbrSpVJEBsxcJkLmQVTOsmnrY5EMwlE3L3Mxv wDzw== X-Gm-Message-State: APjAAAVhOQpPU7NQLwUzA5AnGMPYgoEHfatMDJXK2q6zlvnLweCAVzUH deg/K9rZF0hITNq+rlJ736MKxku9 X-Google-Smtp-Source: APXvYqwvqS69mzQLweYQeMuh/xE/3w8OIc290kC96wFt3I+fsCEAtFwYS2UPUgtVqNb2K+vzs2ak7A== X-Received: by 2002:a5d:51cc:: with SMTP id n12mr90618518wrv.177.1578064742344; Fri, 03 Jan 2020 07:19:02 -0800 (PST) Received: from hthiery01.sab.local ([213.135.10.150]) by smtp.gmail.com with ESMTPSA id u16sm12460565wmj.41.2020.01.03.07.19.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Jan 2020 07:19:01 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Fri, 3 Jan 2020 16:18:42 +0100 Message-Id: <20200103151849.10956-6-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200103151849.10956-1-heiko.thiery@gmail.com> References: <20200103151849.10956-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 05/11] support/scripts/pkg-stats: parse and set developers info X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" This patch collects the developers information and stores developers to the Pakckage instance. Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 59 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 4021aacceb..d520da6807 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -20,6 +20,7 @@ import argparse import datetime import fnmatch import os +import glob from collections import defaultdict import re import subprocess @@ -42,6 +43,50 @@ RM_API_STATUS_NOT_FOUND = 4 # because it's used by sub-processes. http_pool = None +class Developer: + def __init__(self, name, files): + self.name = name + self.files = files + +def parse_developers(basepath=None): + """Parse the DEVELOPERS file and return a list of Developer objects.""" + developers = [] + linen = 0 + if basepath is None: + basepath = os.getcwd() + with open(os.path.join(basepath, "DEVELOPERS"), "r") as f: + files = [] + name = None + for line in f: + line = line.strip() + if line.startswith("#"): + continue + elif line.startswith("N:"): + if name is not None or len(files) != 0: + print("Syntax error in DEVELOPERS file, line %d" % linen) + name = line[2:].strip() + elif line.startswith("F:"): + fname = line[2:].strip() + #dev_files = glob.glob(os.path.join(basepath, fname)) + dev_files = glob.glob(fname) + if len(dev_files) == 0: + print("WARNING: '%s' doesn't match any file" % fname) + files += dev_files + elif line == "": + if not name: + continue + developers.append(Developer(name, files)) + files = [] + name = None + else: + print("Syntax error in DEVELOPERS file, line %d: '%s'" % (linen, line)) + + return None + linen += 1 + # handle last developer + if name is not None: + developers.append(Developer(name, files)) + return developers class Package: all_licenses = list() @@ -56,6 +101,7 @@ class Package: self.has_license = False self.has_license_files = False self.has_hash = False + self.developers = None self.patches = {'count':0, 'files': None} self.warnings = 0 self.current_version = None @@ -151,6 +197,16 @@ class Package: self.warnings = int(m.group(1)) return + def set_developers(self, developers): + """ + Fills in the .developers field + """ + self.developers = list() + for dev in developers: + for f in dev.files: + if self.pkg_path[2:] == f[:-1]: + self.developers.append((dev.name)) + def __eq__(self, other): return self.path == other.path @@ -729,6 +785,8 @@ def __main__(): date = datetime.datetime.utcnow() commit = subprocess.check_output(['git', 'rev-parse', 'HEAD']).splitlines()[0] + print("Getting developers...") + developers = parse_developers() print("Build package list ...") packages = get_pkglist(args.npackages, package_list) print("Getting developers...") @@ -744,6 +802,7 @@ def __main__(): pkg.set_check_package_warnings() pkg.set_current_version() pkg.set_url() + pkg.set_developers(developers) print("Checking URL status") check_package_urls(packages) print("Getting latest versions ...") From patchwork Fri Jan 3 15:18:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1217359 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="ddiF/Fmd"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 47q7q70c4Gz9sR0 for ; Sat, 4 Jan 2020 02:19:14 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id E7C30869C9; Fri, 3 Jan 2020 15:19:11 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id P6ai7rnHExMa; Fri, 3 Jan 2020 15:19:09 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 8D13E8698E; Fri, 3 Jan 2020 15:19:09 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 4405D1BF599 for ; Fri, 3 Jan 2020 15:19:06 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 414D486970 for ; Fri, 3 Jan 2020 15:19:06 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Jnium0BWCE3q for ; Fri, 3 Jan 2020 15:19:05 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f48.google.com (mail-wr1-f48.google.com [209.85.221.48]) by whitealder.osuosl.org (Postfix) with ESMTPS id 89F90836F1 for ; Fri, 3 Jan 2020 15:19:05 +0000 (UTC) Received: by mail-wr1-f48.google.com with SMTP id y11so42740726wrt.6 for ; Fri, 03 Jan 2020 07:19:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Cebh0OfcJ+f/NG0OIE4A01HjwIN9vGCgNsfA9NOmECA=; b=ddiF/FmdaiJKo35ODdpa0Jxf/b8jMW15D9RMcIsd+vSMLBgWkqZKAgapjLlTg/iYZi GrFym4yjkMaoxzYglhKYwmfF4PjDlYVgbvUHojAQcj/232SMOFozsgujrmhvHSH5xvB1 odvISEyMxZ6Pw6E+IzcBUxNg5ncmVYRft0KdyeHY5Q4D9hEFyG4ABwPH0X1do9fdOQk5 72Ek/NvmH3AhuGi8pXxvZvN9O2PFkgylmqF03W+/NFORCgQgyMHQ8DWPkWzzxPWfnenM d+LKCy8l7hndXSpIA+FCTi9JpIYPB71GkE8Q++tox5pjMSH5VbN1/J39n3bh9T1osqrg MsmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Cebh0OfcJ+f/NG0OIE4A01HjwIN9vGCgNsfA9NOmECA=; b=UKIDjaA/l2M9xAg/FbFX9Eef5uoR0UIFeCpDG16tzw/H+CNXv/MHvwgmaIAXBflt0M gY491JwfU8Qm/Oyvir8+zUH95+27+Ojhx5CO05bYHSc0MvtbePXhMXjEbTK5Dy8n/+rK 2X/ZP2t6vPUBCPIigHeGcNbcfoPCckKSrz03TxFWkUSX1uX/DEpYjU9pLLRiX9FwVEif t+0H6scokqv+5sE+jnIOOqHn4GZNkYf1Ou1pz9JGzEDk79T4Y3e2xmgu8oF/jP9OJDds ieKtZ8iEDDVolpZhihjRQyb4M53u1CFogEZfGlj8t7RbvERMdqqdsa7QwfkLaP54t2BO kS/A== X-Gm-Message-State: APjAAAW70Kgc7nEEPOGm0qsySas0/YZJMtw3eeCINoY82h5jTNHNie8C U28YXhwXxdn69kHAEk8FG2S4xhUy X-Google-Smtp-Source: APXvYqyWwGGjByf+XaYyJsQh88Rr40EEPZAKHyaSUE/5MNgFLNNkPkRsfvvZHXnKP7o68caQfPkK8Q== X-Received: by 2002:a5d:494b:: with SMTP id r11mr37864016wrs.184.1578064743743; Fri, 03 Jan 2020 07:19:03 -0800 (PST) Received: from hthiery01.sab.local ([213.135.10.150]) by smtp.gmail.com with ESMTPSA id u16sm12460565wmj.41.2020.01.03.07.19.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Jan 2020 07:19:03 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Fri, 3 Jan 2020 16:18:43 +0100 Message-Id: <20200103151849.10956-7-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200103151849.10956-1-heiko.thiery@gmail.com> References: <20200103151849.10956-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 06/11] support/scripts/pkg-stats: store licences of package X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index d520da6807..a5e87a7167 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -89,7 +89,7 @@ def parse_developers(basepath=None): return developers class Package: - all_licenses = list() + all_licenses = dict() all_license_files = list() all_versions = dict() @@ -98,6 +98,7 @@ class Package: self.path = path self.pkg_path = os.path.dirname(path) self.infras = None + self.license = None self.has_license = False self.has_license_files = False self.has_hash = False @@ -154,6 +155,7 @@ class Package: var = self.pkgvar() if var in self.all_licenses: self.has_license = True + self.license = self.all_licenses[var] if var in self.all_license_files: self.has_license_files = True @@ -302,7 +304,7 @@ def package_init_make_info(): if value == "unknown": continue pkgvar = pkgvar[:-8] - Package.all_licenses.append(pkgvar) + Package.all_licenses[pkgvar] = value elif pkgvar.endswith("_LICENSE_FILES"): if pkgvar.endswith("_MANIFEST_LICENSE_FILES"): @@ -785,8 +787,6 @@ def __main__(): date = datetime.datetime.utcnow() commit = subprocess.check_output(['git', 'rev-parse', 'HEAD']).splitlines()[0] - print("Getting developers...") - developers = parse_developers() print("Build package list ...") packages = get_pkglist(args.npackages, package_list) print("Getting developers...") From patchwork Fri Jan 3 15:18:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1217360 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="fl/VfNBT"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 47q7q75HvQz9sRW for ; Sat, 4 Jan 2020 02:19:15 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 947D185F88; Fri, 3 Jan 2020 15:19:12 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Oadbkzw_hlFo; Fri, 3 Jan 2020 15:19:10 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 8FC2985DEC; Fri, 3 Jan 2020 15:19:10 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id C3B181BF599 for ; Fri, 3 Jan 2020 15:19:07 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id BF46187D41 for ; Fri, 3 Jan 2020 15:19:07 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6zQTapTK4PIA for ; Fri, 3 Jan 2020 15:19:06 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) by hemlock.osuosl.org (Postfix) with ESMTPS id 980EC87816 for ; Fri, 3 Jan 2020 15:19:06 +0000 (UTC) Received: by mail-wm1-f42.google.com with SMTP id f129so8898525wmf.2 for ; Fri, 03 Jan 2020 07:19:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BmeD61psyM0zQB0iOOy5S42Hlq5n32gREMsy9HE3JZk=; b=fl/VfNBTM7HmepIxeFdH27lwyEICuRU5BaUuL2X3lVjlggGJ+usPkasT/cBTEar8kJ VzdiKGlOdlsatvgu3gbwJFz5a6xPzDgLZaLuOX/BrbMpihslqKP8NER4MGq2c2tduoSY 9eOakOVhsOftnkgJWfLLEBzMaASHKCCYTSPq9v9qotrJXPd00vv1Hmbq8GjZvuiBx3Li 2UaigIrigNHGhwMAmNYL+3PAtzC1kTHPZjPI8JuP6YkcqWPv0qRIM4ZPhQ3CgEjA2FMF anGUPkT9xbFKLhDVXEYtv951c+dn0alY0DR5qrA6cS0L3RDZDignGvyvzEwXHOTKafLa 1DSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BmeD61psyM0zQB0iOOy5S42Hlq5n32gREMsy9HE3JZk=; b=ohltnD/gA5SDAmafae6fbXTo3k1Qczoq09zQ3ZhzUzOS3eW4Jutv78AJ4s/77ecuNP nRtTI2Bnozf7KW7LGuu8GYn7ZEduNJAcM4Md0bKt0qnJvEHWqNSc48pyWGauOYmqSdF/ Czps6xEA0nKxbERDXZ3zrz6lL3mhPp6l6O06EYG71erqVWTc4z8w8nAQEsqPUteMYs+p NYiZLhM//EN+9UIZGm3naenjsOvAV/6uhxDCXstXcyimtKnkNzlxxZjoH7VKjM8r6/W5 0JkRaj77FAQaU9lOZZcz7J5y0NqALj/JacYUnahcAjqGp9ehh+KxZkc57O76ASodBiDb 4Jcg== X-Gm-Message-State: APjAAAUVeSPa0ZXoDFtm21x+CKiDq5bm75RtrnXm11nnL0a/7d80jHGa VlKO43P+++wKGZgZGFIA8yErvOds X-Google-Smtp-Source: APXvYqwVNIlD7IJplRZiddN9P8zGwoFStu/s1uey48+3c4PSFwPJq/qkpk7psWp0pptpDBSIoxHx8g== X-Received: by 2002:a1c:7c11:: with SMTP id x17mr19757205wmc.168.1578064744901; Fri, 03 Jan 2020 07:19:04 -0800 (PST) Received: from hthiery01.sab.local ([213.135.10.150]) by smtp.gmail.com with ESMTPSA id u16sm12460565wmj.41.2020.01.03.07.19.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Jan 2020 07:19:04 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Fri, 3 Jan 2020 16:18:44 +0100 Message-Id: <20200103151849.10956-8-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200103151849.10956-1-heiko.thiery@gmail.com> References: <20200103151849.10956-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 07/11] support/scripts/pkg-stats: store dependencies of package X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index a5e87a7167..324700adbf 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -92,6 +92,7 @@ class Package: all_licenses = dict() all_license_files = list() all_versions = dict() + all_dependencies = dict() def __init__(self, name, path): self.name = name @@ -106,6 +107,7 @@ class Package: self.patches = {'count':0, 'files': None} self.warnings = 0 self.current_version = None + self.dependencies = None self.url = None self.url_status = None self.url_worker = None @@ -182,6 +184,14 @@ class Package: if var in self.all_versions: self.current_version = self.all_versions[var] + def set_dependencies(self): + """ + Fills in the .dependencies field + """ + var = self.pkgvar() + if var in self.all_dependencies: + self.dependencies = self.all_dependencies[var] + def set_check_package_warnings(self): """ Fills in the .warnings field @@ -284,7 +294,7 @@ def get_pkglist(npackages, package_list): def package_init_make_info(): # Fetch all variables at once variables = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y", "-s", "printvars", - "VARS=%_LICENSE %_LICENSE_FILES %_VERSION"]) + "VARS=%_LICENSE %_LICENSE_FILES %_VERSION %_DEPENDENCIES"]) variable_list = variables.splitlines() # We process first the host package VERSION, and then the target @@ -318,6 +328,11 @@ def package_init_make_info(): pkgvar = pkgvar[:-8] Package.all_versions[pkgvar] = value + elif pkgvar.endswith("_FINAL_DEPENDENCIES"): + value = [v for v in value.split(' ') if not v.startswith('host-')] + pkgvar = pkgvar[:-19] + Package.all_dependencies[pkgvar] = value + def check_url_status_worker(url, url_status): if url_status != "Missing" and url_status != "No Config.in": @@ -801,6 +816,7 @@ def __main__(): pkg.set_patch_count() pkg.set_check_package_warnings() pkg.set_current_version() + pkg.set_dependencies() pkg.set_url() pkg.set_developers(developers) print("Checking URL status") From patchwork Fri Jan 3 15:18:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1217364 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="DHLNiGOv"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 47q7qG134Qz9sRW for ; Sat, 4 Jan 2020 02:19:22 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id A440485DD1; Fri, 3 Jan 2020 15:19:20 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vrFOU9i8JmhT; Fri, 3 Jan 2020 15:19:19 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id C954985313; Fri, 3 Jan 2020 15:19:19 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 62CD91BF599 for ; Fri, 3 Jan 2020 15:19:18 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 301912201C for ; Fri, 3 Jan 2020 15:19:18 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hr0WqQ1Psm7b for ; Fri, 3 Jan 2020 15:19:09 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) by silver.osuosl.org (Postfix) with ESMTPS id E9E592155C for ; Fri, 3 Jan 2020 15:19:07 +0000 (UTC) Received: by mail-wm1-f48.google.com with SMTP id d73so8687203wmd.1 for ; Fri, 03 Jan 2020 07:19:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WhlQEBR2dcPwBpEkRlGl4txbVHv8vszr3wlYLoPRZ90=; b=DHLNiGOvBHYvdvltZrpdYnu5v+aCVHtQkiXVEogkQLLmjPedCMeO1LJct6Er0iz0im ctvY56+VqkGQ4bwU8RCUR5JSFSg70tfGJJK9+ettw+YdgcZO6RgZGOUuXTR1SDNdlmH7 ninkomtoak6S22OrLHziCB5RDDOh8cGNPQEDciup1gVhvpk/KxAqLDhaKtQCe52y3qA7 JDydA4d8WZf2o+7PfP9/yEtIr/Wy/Q39VTHOVXHo6W2xjPq1YobQcTlbVRGqhQn+GCpq NfJI43DP3OBeU0IotAljSJa+gNWL+mFYrKkpOCScVqdinwOMZuucJi30PaJFJxSvsYB0 VzOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WhlQEBR2dcPwBpEkRlGl4txbVHv8vszr3wlYLoPRZ90=; b=Mx5Xl/4hH1UolcRYmxLLSbR1jksz4EHuiVrc8gzUcU8sZ4jfYTlaUlqMQsiaoQVaS3 bVTleLRf5ynuNqC9hCzSw8EM0njmfy34Y5Ko8cADt4WzSGGdwezeN5O25i3f3OL2HgSP jmg1ewumFPV05k2T9bhXdCflbVELAma5Bm/i1sp7qdWFftJw3jCgG3BN4ctVRJ/lsJv3 kZ7tbhtNo5JSTDsBnaxKgxPD2Cgcfz3PSfKO8b2I+AbLWOXlEhMuKwlFGXqNF6Qa4J1U kJ1J9L+arVn+T9lJ1YomFR90rbxqruUjZyeaXuNacruPNyH5AOiDC9KVJJFeWWO32/Qz asJg== X-Gm-Message-State: APjAAAXVpAKdAF8nfIr42ofRA0XAyjlk0+EclwVqowdVV/XvVDygtU9S wBYiv1ko/uk0NP39k49+jRLMntlw X-Google-Smtp-Source: APXvYqxPc6yVb8tENur0bpKm0lm/MKwiUuj6WF14cbHmxFHFgPmcvj0uCkCbyIShStzcAeJWSEn69w== X-Received: by 2002:a05:600c:290f:: with SMTP id i15mr20650786wmd.115.1578064746192; Fri, 03 Jan 2020 07:19:06 -0800 (PST) Received: from hthiery01.sab.local ([213.135.10.150]) by smtp.gmail.com with ESMTPSA id u16sm12460565wmj.41.2020.01.03.07.19.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Jan 2020 07:19:05 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Fri, 3 Jan 2020 16:18:45 +0100 Message-Id: <20200103151849.10956-9-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200103151849.10956-1-heiko.thiery@gmail.com> References: <20200103151849.10956-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 08/11] support/scripts/pkg-stats: add generic package status field X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Each check stores the status as a tuple: (status, message). The status value should be one of: 'ok', 'warning', 'error' and the message holds more verbose information. Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 42 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 324700adbf..b0c2db2b93 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -112,6 +112,7 @@ class Package: self.url_status = None self.url_worker = None self.rm_version = {'status': RM_API_STATUS_ERROR, 'version': None, 'id': None} + self.status = {} def pkgvar(self): return self.name.upper().replace("-", "_") @@ -121,6 +122,7 @@ class Package: Fills in the .url field """ self.url_status = "No Config.in" + self.status['url'] = ("error", "no Config.in") for filename in os.listdir(self.pkg_path): if fnmatch.fnmatch(filename, 'Config.*'): fp = open(os.path.join(self.pkg_path, filename), "r") @@ -128,9 +130,11 @@ class Package: if URL_RE.match(config_line): self.url = config_line.strip() self.url_status = "Found" + self.status['url'] = ("ok", "found") fp.close() return self.url_status = "Missing" + self.status['url'] = ("error", "missing") fp.close() def set_infra(self): @@ -155,11 +159,16 @@ class Package: Fills in the .has_license and .has_license_files fields """ var = self.pkgvar() + self.status['license'] = ("error", "missing") + self.status['license-files'] = ("error", "missing") if var in self.all_licenses: self.has_license = True self.license = self.all_licenses[var] + self.status['license'] = ("ok", "found") + if var in self.all_license_files: self.has_license_files = True + self.status['license-files'] = ("ok", "found") def set_hash_info(self): """ @@ -167,6 +176,10 @@ class Package: """ hashpath = os.path.join(self.pkg_path, self.name + '.hash') self.has_hash = os.path.exists(hashpath) + if self.has_hash: + self.status['hash'] = ("ok", "found") + else: + self.status['hash'] = ("error", "missing") def set_patch_count(self): """ @@ -176,6 +189,13 @@ class Package: self.patches['files'] = fnmatch.filter(os.listdir(subdir), '*.patch') self.patches['count'] = len(self.patches['files']) + if self.patches['count'] == 0: + self.status['patches'] = ("ok", "no patches") + elif self.patches['count'] < 5: + self.status['patches'] = ("warning", "some patches") + else: + self.status['patches'] = ("error", "lots of patches") + def set_current_version(self): """ Fills in the .current_version field @@ -197,6 +217,7 @@ class Package: Fills in the .warnings field """ cmd = ["./utils/check-package"] + self.status['pkg-check'] = ("error", "missing") for root, dirs, files in os.walk(self.pkg_path): for f in files: if f.endswith(".mk") or f.endswith(".hash") or f == "Config.in" or f == "Config.in.host": @@ -207,6 +228,10 @@ class Package: m = re.match("^([0-9]*) warnings generated", line) if m: self.warnings = int(m.group(1)) + if self.warnings == 0: + self.status['pkg-check'] = ("ok", "no warnings") + else: + self.status['pkg-check'] = ("error", "{} warnings".format(self.warnings)) return def set_developers(self, developers): @@ -214,11 +239,15 @@ class Package: Fills in the .developers field """ self.developers = list() + self.status['developers'] = ("warning", "no developers") for dev in developers: for f in dev.files: if self.pkg_path[2:] == f[:-1]: self.developers.append((dev.name)) + if self.developers: + self.status['developers'] = ("ok", "{} developers".format(len(self.developers))) + def __eq__(self, other): return self.path == other.path @@ -425,6 +454,19 @@ def check_package_latest_version(packages): pkg.rm_version['status'] = r[0] pkg.rm_version['version'] = r[1] pkg.rm_version['id'] = r[2] + pkg.rm_version['version'] == pkg.current_version + + if pkg.rm_version['status'] == RM_API_STATUS_ERROR: + pkg.status['version'] = ('warning', 'rm api error') + elif pkg.rm_version['status'] == RM_API_STATUS_NOT_FOUND: + pkg.status['version'] = ('warning', 'rm package not found') + + if pkg.rm_version['version'] is None: + pkg.status['version'] = ('warning', 'no upstream version available') + elif pkg.rm_version['version'] != pkg.current_version: + pkg.status['version'] = ('error', 'package needs update') + else: + pkg.status['version'] = ('ok', 'up-to-date') del http_pool From patchwork Fri Jan 3 15:18:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1217363 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="VTCjh4cB"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 47q7qG0FSJz9sR0 for ; Sat, 4 Jan 2020 02:19:22 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id B8DDD2201C; Fri, 3 Jan 2020 15:19:18 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Fw39m3uebpPP; Fri, 3 Jan 2020 15:19:15 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id A30E32156C; Fri, 3 Jan 2020 15:19:11 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 1154A1BF599 for ; Fri, 3 Jan 2020 15:19:10 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 073BC869C4 for ; Fri, 3 Jan 2020 15:19:10 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XGj2BpiVbofp for ; Fri, 3 Jan 2020 15:19:09 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f42.google.com (mail-wr1-f42.google.com [209.85.221.42]) by whitealder.osuosl.org (Postfix) with ESMTPS id 0C98F86970 for ; Fri, 3 Jan 2020 15:19:09 +0000 (UTC) Received: by mail-wr1-f42.google.com with SMTP id y17so42782300wrh.5 for ; Fri, 03 Jan 2020 07:19:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=LExQxcuijj7MHUVtMn+xG9vyZvsJ33kzBdQFUxPcwkw=; b=VTCjh4cBkyjcO/XFtuTrjG8I3CIiIoad0ETmWotTIRHZPH12ybnlQrClXV9ljVJ5IS lRVpT56CG8ymTcwMAYE2njdt55xU3qVq9KemREUx+VUFBbF5BJq5tnJfjJOOjQOQXh1j 6UoBhgJjPnZrrrVsU52xVlpjvqiq8XxY7kH3k1+9ShOOaRMZU+HmWoThaum32t3mixqu ol5xU2Hj07UZ+n8d5+fkwNAEOeSfUJvRdjV1aGUFhNkp58WyH0C/CGDcwN2i2e82sRDs d315iBgFbiIzuVp7fCLpXG4oIrwAQ6CQyaRL5e1/twOspQQDzkL6SLqXSCyxCkmxBUfL DGfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=LExQxcuijj7MHUVtMn+xG9vyZvsJ33kzBdQFUxPcwkw=; b=iS9eZchjAl8qxqyX3krhudmBxAHh4LSkz8pzZRydDuwN7sWZvRbO8O+4b3+B/EqWVl q/7TL7TxMBeVRxlIAe22HN27vqeNW5Y4PEo0uCl2sMELSAycsjGzg8bh23/uKAGlJxAm ZjlkMqH7eSlcGwcdb1vK60wX5ugmv3QmrSYClWqEN7aCEYT2IFhp84X1YNiLK29Gmthn F/A/7hM7ckxufJtGNNVMrbJQXD+Q5rxb/MgP912m3o6ksoPLIuuMaA2VuoNGc/L+CmA/ JNpjorhIBVtHiv1/nW4kiPCLqEzQmobnpGGYVRfAQAI1qYYjy3TklDyV86tAbiW6+8f3 t8Gg== X-Gm-Message-State: APjAAAX+4Dd7FCRvNQK3rLYmKFYFsRun2i8pyK/jdHz6njFXx58cbHgZ UTm1+tMJtC4kbDN81Quw4Zv5eY5y X-Google-Smtp-Source: APXvYqx0Cy5hSPxetLTYFNkQjk0ts/9ZFlj/nMlJPEskc9Bnrpv5lQKsvf1vygkj+TJzTaXMQ5QklA== X-Received: by 2002:a5d:55d1:: with SMTP id i17mr86430159wrw.165.1578064747243; Fri, 03 Jan 2020 07:19:07 -0800 (PST) Received: from hthiery01.sab.local ([213.135.10.150]) by smtp.gmail.com with ESMTPSA id u16sm12460565wmj.41.2020.01.03.07.19.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Jan 2020 07:19:06 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Fri, 3 Jan 2020 16:18:46 +0100 Message-Id: <20200103151849.10956-10-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200103151849.10956-1-heiko.thiery@gmail.com> References: <20200103151849.10956-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 09/11] support/scripts/pkg-stats; use url status from dict for check X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index b0c2db2b93..dd3c6566a0 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -364,23 +364,23 @@ def package_init_make_info(): def check_url_status_worker(url, url_status): - if url_status != "Missing" and url_status != "No Config.in": + if url_status[0] == "ok": try: url_status_code = requests.head(url, timeout=30).status_code if url_status_code >= 400: - return "Invalid(%s)" % str(url_status_code) + return ('error', 'invalid({})'.format(url_status_code)) except requests.exceptions.RequestException: - return "Invalid(Err)" - return "Ok" + return ('error', 'invalid(Err)') + return ("ok", "Valid") return url_status def check_package_urls(packages): Package.pool = Pool(processes=64) for pkg in packages: - pkg.url_worker = pkg.pool.apply_async(check_url_status_worker, (pkg.url, pkg.url_status)) + pkg.url_worker = pkg.pool.apply_async(check_url_status_worker, (pkg.url, pkg.status['url'])) for pkg in packages: - pkg.url_status = pkg.url_worker.get(timeout=3600) + pkg.status['url'] = pkg.url_worker.get(timeout=3600) def release_monitoring_get_latest_version_by_distro(pool, name): From patchwork Fri Jan 3 15:18:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1217365 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="DAkVD/OJ"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 47q7qN5cbRz9sPn for ; Sat, 4 Jan 2020 02:19:28 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id D707D86B0E; Fri, 3 Jan 2020 15:19:26 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id SG6BfDMhWQ6S; Fri, 3 Jan 2020 15:19:19 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 8F04A86A02; Fri, 3 Jan 2020 15:19:17 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 69F5C1BF9BA for ; Fri, 3 Jan 2020 15:19:16 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 6297122008 for ; Fri, 3 Jan 2020 15:19:16 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5UxW-PkB4LU4 for ; Fri, 3 Jan 2020 15:19:11 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr1-f43.google.com (mail-wr1-f43.google.com [209.85.221.43]) by silver.osuosl.org (Postfix) with ESMTPS id A223A21577 for ; Fri, 3 Jan 2020 15:19:10 +0000 (UTC) Received: by mail-wr1-f43.google.com with SMTP id y17so42782372wrh.5 for ; Fri, 03 Jan 2020 07:19:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WaMD+Uzrbug7ssNCBMI07KPWpTF6T++4TU5CG4FAtg8=; b=DAkVD/OJCdMbMxxaUqrr8Ahjm/k8dlRz/oAdCKzDHEBOqPdXz7NjMpolcJj5ZXRzmD /eKQjsod+nld66geU+ttqWvxdVQVX9KHHgrqPsJq5+0q5z9XJHYRYGuHkoDCCfl+/Yj1 dNMTugtl2muo3lIBisXwTvV/27+AldWrOVauKiQTViiadwVs92z7Fun32hoj3uQMr5aY uq8GJ6KCdM1on5OkBxhHpumRY/vbFDMg6+BrJrodAzUxVCvDh+yy81ukn4wkm3ThrFD5 zVdd5oaiu+1NKBVWbmTToiY2Gv96R0yRFELhUiDtkJIT6i1rEUg/OQl7QyP1OtNC1u/o BrNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WaMD+Uzrbug7ssNCBMI07KPWpTF6T++4TU5CG4FAtg8=; b=NDYitTtBjHeFZgniI4eVgqGKog9yxikg18v7tIcY7MA2j9E0KTzSwWHspmcVLB61Gj 16N4q2frRrBDHcdgZ5Y4f5z1j364k/R5y+sQdat79fs136YLrocncycfISgDreY9iXMI UWqcdf/cCniwOjoyneyhcLFJXDdHEC2T1toq5lsvxRdWd1Px0495yy4kWLOvWeafGyc3 PwTb7D7kxqIdCkVL8PWlrHZcKiR/Ys4/rdHvTRkUkKAzkvjcmqxe5bvoh0vTVCjD7FzL bxB0PkWVSUjHtXrP4Abg9g6kodueE1whaWeQBDIA0/dAqM107o5iU1tmr3OweZC3sNZ4 /rlw== X-Gm-Message-State: APjAAAVJOBJsbsDTOHx7jDApETgUjeP6vUrzN9wVf3nt7chmsVqxDyik D16rMLC1bZA1S3qKQA4bD7DNPM5t X-Google-Smtp-Source: APXvYqzoqbLeeE9jowfAcXQ1Y6B/f9UQJCkeP+L3RCA1M4ebacwzfpPAiZKDhCqFdjY9+USe4YJUtg== X-Received: by 2002:a5d:534d:: with SMTP id t13mr12472120wrv.77.1578064748976; Fri, 03 Jan 2020 07:19:08 -0800 (PST) Received: from hthiery01.sab.local ([213.135.10.150]) by smtp.gmail.com with ESMTPSA id u16sm12460565wmj.41.2020.01.03.07.19.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Jan 2020 07:19:08 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Fri, 3 Jan 2020 16:18:47 +0100 Message-Id: <20200103151849.10956-11-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200103151849.10956-1-heiko.thiery@gmail.com> References: <20200103151849.10956-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 10/11] support/scripts/pkg-stats: add package count to stats X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index dd3c6566a0..afd9cacafb 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -457,21 +457,22 @@ def check_package_latest_version(packages): pkg.rm_version['version'] == pkg.current_version if pkg.rm_version['status'] == RM_API_STATUS_ERROR: - pkg.status['version'] = ('warning', 'rm api error') + pkg.status['version'] = ('warning', 'RM API error') elif pkg.rm_version['status'] == RM_API_STATUS_NOT_FOUND: - pkg.status['version'] = ('warning', 'rm package not found') + pkg.status['version'] = ('warning', 'RM package not found') if pkg.rm_version['version'] is None: pkg.status['version'] = ('warning', 'no upstream version available') elif pkg.rm_version['version'] != pkg.current_version: - pkg.status['version'] = ('error', 'package needs update') + pkg.status['version'] = ('error', 'Package needs update') else: - pkg.status['version'] = ('ok', 'up-to-date') + pkg.status['version'] = ('ok', 'update-to-date') del http_pool def calculate_stats(packages): stats = defaultdict(int) + stats["packages"] = len(packages) for pkg in packages: # If packages have multiple infra, take the first one. For the # vast majority of packages, the target and host infra are the @@ -743,6 +744,8 @@ def dump_html_all_pkgs(f, packages): def dump_html_stats(f, stats): f.write("\n") f.write("\n") + f.write(" \n" % + stats["packages"]) infras = [infra[6:] for infra in stats.keys() if infra.startswith("infra-")] for infra in infras: f.write(" \n" % From patchwork Fri Jan 3 15:18:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Heiko Thiery X-Patchwork-Id: 1217362 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Mzii6Caa"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 47q7qB73gRz9sPn for ; Sat, 4 Jan 2020 02:19:18 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 530C7869F2; Fri, 3 Jan 2020 15:19:17 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ofKRvbxjC6T3; Fri, 3 Jan 2020 15:19:15 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id B1AA686A02; Fri, 3 Jan 2020 15:19:14 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 03CCF1BF9BA for ; Fri, 3 Jan 2020 15:19:13 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id F219E85F7F for ; Fri, 3 Jan 2020 15:19:12 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7N3nbC4OOW9i for ; Fri, 3 Jan 2020 15:19:12 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f49.google.com (mail-wm1-f49.google.com [209.85.128.49]) by fraxinus.osuosl.org (Postfix) with ESMTPS id BF9BD85D90 for ; Fri, 3 Jan 2020 15:19:11 +0000 (UTC) Received: by mail-wm1-f49.google.com with SMTP id a5so8696310wmb.0 for ; Fri, 03 Jan 2020 07:19:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=08RSQInEfkrmrvUlDzsbs3AG1RQvhrRIa79Ub9VaUl4=; b=Mzii6CaaQcqTF20AAIwJrWAI3sbUg/GLfqoLZi6HCUwgUTpoYFtpNdPop9VonjAr6b /3ncpDOR9rDe7VgE/NkKMgAI/aZcLFbW1wnuSwpEUP43b5ArcC+YDyulxEtQM07y1Tl9 1P1mcCILIJBrEFCMZdfcyWk2PFGdIfmng0ZeVF56P8DZNcpw7SbhgeODqsabB81ozkqW rMQnKzm8ia82aHmtC2HwxfAlWmt+XpD+bXPTMMv4ovKVXU66RIp2iD5uPM0TGrRn23MV DJ2BNjWRtHWuIOzk6oTTQF15ez1Ed6EV5ddgyFnyil1TtfViFPlkqstxCZDMVr/9i18Y WP8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=08RSQInEfkrmrvUlDzsbs3AG1RQvhrRIa79Ub9VaUl4=; b=AS2OKe6b1OL5E/DwQQ0WRsbPWoRB54CVUrFT+t0YeEzlm3m2T8j/U6JE6fs2WKL8Du bbAidfZG3SmsSkxYlGb3Gi107lWp7ZQ9BTP7N8yQPAVihdd2ad7BxdS5VY/KJl+M78GE inVRz0T9OJlbcpqwFZdvFSIaVebKzs2/SERQkdYB9XP2CzCrj5ENer2W0HruH7yZpQHI tNQsZnTKsvz6dnfiF8l0nLxnhYpfOyd/i/c8xbIQ0UINeFRzbj5afxjHJt4veiRyNLWd uprQ4gEtlSofBrfThWEBnicn9JSw+c/vZrMoPsyJlNz4l+mWRgLwYjtU4eASWQcx7Cgm 68vg== X-Gm-Message-State: APjAAAXu2qXJoUpEPoDi9GWLDu3rU8iKLJYBz50vaAbIJfa9HyROm86T lF5Jjs/bJOyNCdZVQ8iGe6y3/sTt X-Google-Smtp-Source: APXvYqx4aKTaIL6QPB6Xio64R8asuW1oiJ9D3ImfOdyA3e44il7ztR5duoBG2HFfG/8l7R4mnzvRoQ== X-Received: by 2002:a05:600c:230d:: with SMTP id 13mr19796108wmo.12.1578064749983; Fri, 03 Jan 2020 07:19:09 -0800 (PST) Received: from hthiery01.sab.local ([213.135.10.150]) by smtp.gmail.com with ESMTPSA id u16sm12460565wmj.41.2020.01.03.07.19.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 03 Jan 2020 07:19:09 -0800 (PST) From: Heiko Thiery To: buildroot@buildroot.org Date: Fri, 3 Jan 2020 16:18:48 +0100 Message-Id: <20200103151849.10956-12-heiko.thiery@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200103151849.10956-1-heiko.thiery@gmail.com> References: <20200103151849.10956-1-heiko.thiery@gmail.com> MIME-Version: 1.0 Subject: [Buildroot] [PATCH 11/11] support/scripts/pkg-stats: create and store defconfig information X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Heiko Thiery Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Collect information about developers and store with defconfig name. Signed-off-by: Heiko Thiery --- support/scripts/pkg-stats | 48 +++++++++++++++++++++++++++++++++++---- 1 file changed, 44 insertions(+), 4 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index afd9cacafb..a85aefadf4 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -88,6 +88,34 @@ def parse_developers(basepath=None): developers.append(Developer(name, files)) return developers +class Defconfig: + def __init__(self, name, path): + self.name = name + self.path = path + self.developers = None + + def set_developers(self, developers): + """ + Fills in the .developers field + """ + self.developers = list() + for dev in developers: + for f in dev.files: + if self.path == f: + self.developers.append(dev.name) + +def get_defconfig_list(): + """ + Builds the list of Buildroot defconfigs, returning a list of Defconfig + objects. + """ + defconfigs = list() + files = [f for f in os.listdir('configs')] + for name in files: + d = Defconfig(name[:-10], os.path.join('configs', name)) + defconfigs.append(d) + return defconfigs + class Package: all_licenses = dict() all_license_files = list() @@ -791,7 +819,7 @@ def dump_html(packages, stats, date, commit, output): f.write(html_footer) -def dump_json(packages, stats, date, commit, output): +def dump_json(packages, defconfigs, stats, date, commit, output): # Format packages as a dictionnary instead of a list # Exclude local field that does not contains real date excluded_fields = ['url_worker'] @@ -802,6 +830,13 @@ def dump_json(packages, stats, date, commit, output): if k not in excluded_fields } for pkg in packages } + + defconfigs = { + d.name: { + k: v + for k, v in d.__dict__.items() + } for d in defconfigs + } # Aggregate infrastructures into a single dict entry statistics = { k: v @@ -811,6 +846,7 @@ def dump_json(packages, stats, date, commit, output): statistics['infra'] = {k[6:]: v for k, v in stats.items() if k.startswith('infra-')} # The actual structure to dump, add commit and date to it final = {'packages': pkgs, + 'defconfigs': defconfigs, 'stats': statistics, 'commit': commit, 'date': str(date)} @@ -847,10 +883,14 @@ def __main__(): date = datetime.datetime.utcnow() commit = subprocess.check_output(['git', 'rev-parse', 'HEAD']).splitlines()[0] - print("Build package list ...") - packages = get_pkglist(args.npackages, package_list) print("Getting developers...") developers = parse_developers() + print("Build defconfig list ...") + defconfigs = get_defconfig_list() + for d in defconfigs: + d.set_developers(developers) + print("Build package list ...") + packages = get_pkglist(args.npackages, package_list) print("Getting package make info ...") package_init_make_info() print("Getting package details ...") @@ -875,7 +915,7 @@ def __main__(): dump_html(packages, stats, date, commit, args.html) if args.json: print("Write JSON") - dump_json(packages, stats, date, commit, args.json) + dump_json(packages, defconfigs, stats, date, commit, args.json) __main__()
Total number of packages%s
Packages using the %s infrastructure%s