From patchwork Sat Feb 8 21:57:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Titouan Christophe X-Patchwork-Id: 1235389 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.133; helo=hemlock.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=railnova.eu Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=railnova-eu.20150623.gappssmtp.com header.i=@railnova-eu.20150623.gappssmtp.com header.a=rsa-sha256 header.s=20150623 header.b=wIBI5pxb; dkim-atps=neutral Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48FQzL12LJz9sPJ for ; Sun, 9 Feb 2020 08:58:38 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 4CD7F8806D; Sat, 8 Feb 2020 21:58:36 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JHpKlMNJTUWx; Sat, 8 Feb 2020 21:58:35 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 0EAD588072; Sat, 8 Feb 2020 21:58:35 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 8414B1BF357 for ; Sat, 8 Feb 2020 21:58:34 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 7F5D688072 for ; Sat, 8 Feb 2020 21:58:34 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GwUlDz3xtF04 for ; Sat, 8 Feb 2020 21:58:32 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f65.google.com (mail-wm1-f65.google.com [209.85.128.65]) by hemlock.osuosl.org (Postfix) with ESMTPS id 01AEB8806D for ; Sat, 8 Feb 2020 21:58:31 +0000 (UTC) Received: by mail-wm1-f65.google.com with SMTP id t23so6021475wmi.1 for ; Sat, 08 Feb 2020 13:58:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=railnova-eu.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=IT116wj6+/T5qtgYmtF0I5SZ6LhglAxhZauL4vEqc3s=; b=wIBI5pxba5g6OXCB/bKARHJ0h6C9V38UnpKnvzowRcGNuv81er7/xzHds42HAEJO5v 3aGeDNSEGu73bbyZ5rQWRu1NHKI+QcyeZaxlqSwrKG+CSWrSIp1K0CJSOME3xU/LnZWv dSR35ueEfPUlwfNtOiSi8yqauqcAJ0puwaXzvXya4brZh37LjOeaawlHTXsBJriB2fwh eKe6HgQqx/PyjYpWQaMMH6GCpeBnVvRkZfT6NN/XFvSE2gMdQ5va/CYpsGm6zjYZR+Hz h029ywFL5FZLuQ2PRyVEEJeP/njjisf5TwNsz0xfTHQ1KZUS0syokicXVCSLDi//8qnL Bb8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=IT116wj6+/T5qtgYmtF0I5SZ6LhglAxhZauL4vEqc3s=; b=LUiETfWsHYUton0Bx2q7AeH+qVNplMr2ZX7ZM0em/lUiSEQJx82uoNKj9XJovKXhyA vPDVAvuaxQ20Bx3ku2/1y1DG/v7dHWoaBdzLJ9ESrM8DkRKNKxiZN1HPSnXjREU9bM+Q sOxu4WGz2GM5EJukgaxSZUutmlwDm6xaRLhRcjUUwA/U8DOPNcp5rCVJ/4VKz9uhdpAF 4Tli9M/YxpvgYKbP17EYXNv3goF6mLCgfoGMYlw8SIE4iafemnimgb8JqCQO7ylKf/tp fK30ySW4Pbx+T1zZStAEcymNLbFFFSpmXo5u29HPUsvYv6U+WPKhn2g+YeyGj8QOFzew e9OQ== X-Gm-Message-State: APjAAAVk30mwf4JWwf0WyE5uGa85BywNdzYfSIZTEQc84wGFU03U0Dek PFQWn3g8wbpC4kJuUbPaWOwA2JlwAjIs0Q== X-Google-Smtp-Source: APXvYqyM0m3PcyN58/msJThfo92WoBIi3fymI8ZLu4yCJge/AaniYbJlzAqxAgdwh8iv4G0AXSEo/w== X-Received: by 2002:a7b:c753:: with SMTP id w19mr5998938wmk.34.1581199109798; Sat, 08 Feb 2020 13:58:29 -0800 (PST) Received: from localhost.localdomain ([2a02:a03f:63cf:a300:5cd0:a0a7:9509:1ef9]) by smtp.gmail.com with ESMTPSA id s8sm9178844wrt.57.2020.02.08.13.58.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 08 Feb 2020 13:58:29 -0800 (PST) From: Titouan Christophe To: buildroot@buildroot.org Date: Sat, 8 Feb 2020 22:57:51 +0100 Message-Id: <20200208215752.13628-2-titouan.christophe@railnova.eu> X-Mailer: git-send-email 2.24.1 In-Reply-To: <20200208215752.13628-1-titouan.christophe@railnova.eu> References: <20200208215752.13628-1-titouan.christophe@railnova.eu> MIME-Version: 1.0 Subject: [Buildroot] [PATCH v2 1/2] support/scripts/pkg-stats: add support for CVE reporting X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Korsgaard , Matt Weber , Thomas Petazzoni , Titouan Christophe , Thomas De Schampheleire Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: Thomas Petazzoni This commit extends the pkg-stats script to grab information about the CVEs affecting the Buildroot packages. To do so, it downloads the NVD database from https://nvd.nist.gov/vuln/data-feeds in JSON format, and processes the JSON file to determine which of our packages is affected by which CVE. The information is then displayed in both the HTML output and the JSON output of pkg-stats. To use this feature, you have to pass the new --nvd-path option, pointing to a writable directory where pkg-stats will store the NVD database. If the local database is less than 24 hours old, it will not re-download it. If it is more than 24 hours old, it will re-download only the files that have really been updated by upstream NVD. Packages can use the newly introduced _IGNORE_CVES variable to tell pkg-stats that some CVEs should be ignored: it can be because a patch we have is fixing the CVE, or because the CVE doesn't apply in our case. >From an implementation point of view: - A new class CVE implement most of the required functionalities: - Downloading the yearly NVD files - Reading and extracting relevant data from these files - Matching Packages against a CVE - Support for the format "1.0" of the NVD feeds, currently much more easier to process than the version "1.1", as the latter only provides CPE IDs. Both feed versions seem to provide the same data anyway. - The statistics are extended with the total number of CVEs, and the total number of packages that have at least one CVE pending. - The HTML output is extended with these new details. There are no changes to the code generating the JSON output because the existing code is smart enough to automatically expose the new information. This development is a collective effort with Titouan Christophe and Thomas De Schampheleire . Signed-off-by: Thomas Petazzoni Signed-off-by: Titouan Christophe --- Changes v1 -> v2 (Titouan): * Don't extract database files from gzip to json in downloader * Refactor CVEs traversal and matching in the CVE class * Simplify the NVD files downloader * Index the packages by name in a dict for faster CVE matching * Fix small typos and python idioms --- support/scripts/pkg-stats | 149 +++++++++++++++++++++++++++++++++++++- 1 file changed, 148 insertions(+), 1 deletion(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index e477828f7b..2784a43d05 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -26,10 +26,17 @@ import subprocess import requests # URL checking import json import certifi +import distutils.version +import time +import gzip from urllib3 import HTTPSConnectionPool from urllib3.exceptions import HTTPError from multiprocessing import Pool +NVD_START_YEAR = 2002 +NVD_JSON_VERSION = "1.0" +NVD_BASE_URL = "https://nvd.nist.gov/feeds/json/cve/" + NVD_JSON_VERSION + INFRA_RE = re.compile(r"\$\(eval \$\(([a-z-]*)-package\)\)") URL_RE = re.compile(r"\s*https?://\S*\s*$") @@ -47,6 +54,7 @@ class Package: all_licenses = list() all_license_files = list() all_versions = dict() + all_ignored_cves = dict() def __init__(self, name, path): self.name = name @@ -61,6 +69,7 @@ class Package: self.url = None self.url_status = None self.url_worker = None + self.cves = list() self.latest_version = (RM_API_STATUS_ERROR, None, None) def pkgvar(self): @@ -152,6 +161,12 @@ class Package: self.warnings = int(m.group(1)) return + def is_cve_ignored(self, cve): + """ + Tells if the CVE is ignored by the package + """ + return cve in self.all_ignored_cves.get(self.pkgvar(), []) + def __eq__(self, other): return self.path == other.path @@ -163,6 +178,103 @@ class Package: (self.name, self.path, self.has_license, self.has_license_files, self.has_hash, self.patch_count) +class CVE: + """An accessor class for CVE Items in NVD files""" + def __init__(self, nvd_cve): + """Initialize a CVE from its NVD JSON representation""" + self.nvd_cve = nvd_cve + + @staticmethod + def download_nvd_year(nvd_path, year): + metaf = "nvdcve-%s-%s.meta" % (NVD_JSON_VERSION, year) + path_metaf = os.path.join(nvd_path, metaf) + jsonf_gz = "nvdcve-%s-%s.json.gz" % (NVD_JSON_VERSION, year) + path_jsonf_gz = os.path.join(nvd_path, jsonf_gz) + + # If the database file is less than a day old, we assume the NVD data + # locally available is recent enough. + if os.path.exists(path_jsonf_gz) and os.stat(path_jsonf_gz).st_mtime >= time.time() - 86400: + return path_jsonf_gz + + # If not, we download the meta file + url = "%s/%s" % (NVD_BASE_URL, metaf) + print("Getting %s" % url) + page_meta = requests.get(url) + page_meta.raise_for_status() + if os.path.exists(path_metaf): + # If the meta file already existed, we compare the existing + # one with the data newly downloaded. If they are different, + # we need to re-download the database. + meta_known = open(path_metaf, "r").read() + if page_meta.text == meta_known: + return path_jsonf_gz + + # Grab the compressed JSON NVD, and write files to disk + url = "%s/%s" % (NVD_BASE_URL, jsonf_gz) + print("Getting %s" % url) + page_data = requests.get(url) + page_data.raise_for_status() + open(path_jsonf_gz, "wb").write(page_data.content) + open(path_metaf, "w").write(page_meta.text) + return path_jsonf_gz + + @classmethod + def read_nvd_dir(cls, nvd_dir): + """ + Iterate over all the CVEs contained in NIST Vulnerability Database + feeds since NVD_START_YEAR. If the files are missing or outdated in + nvd_dir, a fresh copy will be downloaded, and kept in .json.gz + """ + for year in range(NVD_START_YEAR, datetime.datetime.now().year + 1): + filename = CVE.download_nvd_year(nvd_dir, year) + content = json.load(gzip.GzipFile(filename)) + for cve in content["CVE_Items"]: + yield cls(cve['cve']) + + def each_product(self): + """Iterate over each product section of this cve""" + for vendor in self.nvd_cve['affects']['vendor']['vendor_data']: + for product in vendor['product']['product_data']: + yield product + + @property + def identifier(self): + """The CVE unique identifier""" + return self.nvd_cve['CVE_data_meta']['ID'] + + @property + def pkg_names(self): + """The set of package names referred by this CVE definition""" + return set(p['product_name'] for p in self.each_product()) + + def affects(self, br_pkg): + """ + True if the Buildroot Package object passed as argument is affected + by this CVE. + """ + for product in self.each_product(): + if product['product_name'] != br_pkg.name: + continue + + for v in product['version']['version_data']: + if v["version_affected"] == "=": + if br_pkg.current_version == v["version_value"]: + return True + elif v["version_affected"] == "<=": + pkg_version = distutils.version.LooseVersion(br_pkg.current_version) + if not hasattr(pkg_version, "version"): + print("Cannot parse package '%s' version '%s'" % (br_pkg.name, br_pkg.current_version)) + continue + cve_affected_version = distutils.version.LooseVersion(v["version_value"]) + if not hasattr(cve_affected_version, "version"): + print("Cannot parse CVE affected version '%s'" % v["version_value"]) + continue + return pkg_version <= cve_affected_version + else: + print("version_affected: %s" % v['version_affected']) + return False + + def get_pkglist(npackages, package_list): """ Builds the list of Buildroot packages, returning a list of Package @@ -227,7 +339,7 @@ def get_pkglist(npackages, package_list): def package_init_make_info(): # Fetch all variables at once variables = subprocess.check_output(["make", "BR2_HAVE_DOT_CONFIG=y", "-s", "printvars", - "VARS=%_LICENSE %_LICENSE_FILES %_VERSION"]) + "VARS=%_LICENSE %_LICENSE_FILES %_VERSION %_IGNORE_CVES"]) variable_list = variables.splitlines() # We process first the host package VERSION, and then the target @@ -261,6 +373,10 @@ def package_init_make_info(): pkgvar = pkgvar[:-8] Package.all_versions[pkgvar] = value + elif pkgvar.endswith("_IGNORE_CVES"): + pkgvar = pkgvar[:-12] + Package.all_ignored_cves[pkgvar] = value.split(" ") + def check_url_status_worker(url, url_status): if url_status != "Missing" and url_status != "No Config.in": @@ -355,6 +471,13 @@ def check_package_latest_version(packages): del http_pool +def check_package_cves(nvd_path, packages): + for cve in CVE.read_nvd_dir(nvd_path): + for pkg_name in cve.pkg_names: + if pkg_name in packages and cve.affects(packages[pkg_name]): + packages[pkg_name].cves.append(cve.identifier) + + def calculate_stats(packages): stats = defaultdict(int) for pkg in packages: @@ -390,6 +513,9 @@ def calculate_stats(packages): else: stats["version-not-uptodate"] += 1 stats["patches"] += pkg.patch_count + stats["total-cves"] += len(pkg.cves) + if len(pkg.cves) != 0: + stats["pkg-cves"] += 1 return stats @@ -601,6 +727,17 @@ def dump_html_pkg(f, pkg): f.write(" %s\n" % (" ".join(td_class), url_str)) + # CVEs + td_class = ["centered"] + if len(pkg.cves) == 0: + td_class.append("correct") + else: + td_class.append("wrong") + f.write(" \n" % " ".join(td_class)) + for cve in pkg.cves: + f.write(" %s
\n" % (cve, cve)) + f.write(" \n") + f.write(" \n") @@ -618,6 +755,7 @@ def dump_html_all_pkgs(f, packages): Latest version Warnings Upstream URL +CVEs """) for pkg in sorted(packages): @@ -656,6 +794,10 @@ def dump_html_stats(f, stats): stats["version-not-uptodate"]) f.write("Packages with no known upstream version%s\n" % stats["version-unknown"]) + f.write("Packages affected by CVEs%s\n" % + stats["pkg-cves"]) + f.write("Total number of CVEs affecting all packages%s\n" % + stats["total-cves"]) f.write("\n") @@ -714,6 +856,8 @@ def parse_args(): help='Number of packages') packages.add_argument('-p', dest='packages', action='store', help='List of packages (comma separated)') + parser.add_argument('--nvd-path', dest='nvd_path', + help='Path to the local NVD database') args = parser.parse_args() if not args.html and not args.json: parser.error('at least one of --html or --json (or both) is required') @@ -746,6 +890,9 @@ def __main__(): check_package_urls(packages) print("Getting latest versions ...") check_package_latest_version(packages) + if args.nvd_path: + print("Checking packages CVEs") + check_package_cves(args.nvd_path, {p.name: p for p in packages}) print("Calculate stats") stats = calculate_stats(packages) if args.html: