From patchwork Tue Jun 16 17:03:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matt Weber X-Patchwork-Id: 1310596 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=rockwellcollins.com Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 49mZKv6JqTz9sRW for ; Wed, 17 Jun 2020 03:04:03 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id BC2C72667F; Tue, 16 Jun 2020 17:04:01 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QMct9dAKqx3A; Tue, 16 Jun 2020 17:03:54 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 6AC822667D; Tue, 16 Jun 2020 17:03:53 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 658D61BF5DC for ; Tue, 16 Jun 2020 17:03:47 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 62F808954E for ; Tue, 16 Jun 2020 17:03:47 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vslNG-dIRRgp for ; Tue, 16 Jun 2020 17:03:46 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from secvs05.rockwellcollins.com (secvs05.rockwellcollins.com [205.175.225.131]) by hemlock.osuosl.org (Postfix) with ESMTPS id 339AA89556 for ; Tue, 16 Jun 2020 17:03:46 +0000 (UTC) IronPort-SDR: Eicm6w8a12W2dj45JAAuIZ1GGq9lzRMZImuDmT6+6sMzP4nHOaLr7p6EOmwnIUoQNaPkm0INqA 6ZP+fYbZiRNnZqWqUPoehSsuk2usFpRBK6lopviBMHmZDZYoAC4bPNQM+u3Eg78Dh13h0H8AXw 1zUft5h1bt/DERJk6PJKrKj6mVIzOkOAQZJAKIdCQrIMTb5JEum5NyblImRtgKaQg+C8mMxVhx ylzbwNGIhwd0Zh2eXNHrQV2/4eIPdHhdvbOtIpV7xmzRJcoMGPSAvYU3s5+stzLaatanfRrixM n/4= Received: from ofwgwc03.rockwellcollins.com (HELO dtulimr02.rockwellcollins.com) ([205.175.225.12]) by secvs05.rockwellcollins.com with ESMTP; 16 Jun 2020 12:03:45 -0500 X-Received: from biscuits.rockwellcollins.com (biscuits.rockwellcollins.lab [10.148.119.137]) by dtulimr02.rockwellcollins.com (Postfix) with ESMTP id C123C20076; Tue, 16 Jun 2020 12:03:44 -0500 (CDT) From: Matt Weber To: buildroot@buildroot.org Date: Tue, 16 Jun 2020 12:03:39 -0500 Message-Id: <20200616170341.45098-8-matthew.weber@rockwellcollins.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200616170341.45098-1-matthew.weber@rockwellcollins.com> References: <20200616170341.45098-1-matthew.weber@rockwellcollins.com> Subject: [Buildroot] [RFC v9 08/10] support/scripts/cpe-report: new script X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Matt Weber MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" The script supports looking up all the CPEs provided in a make cpe-info csv file export from a target Buildroot build. It checks the current version and suggests a CPE needs update or possibly an initial submission is required to NIST. Adds option to allow alternate locations for the dictionary URL and caching of a processed dictionary to speed up execution. Outputs a cpe/ folder with propsed xml generated from the dictionary contents to propose updated versions to NIST. For missing CPE matches, a cpe-report-missing.txt is created by the script that can be used later to manually create proposed new NIST dictionary entries. Ref: NIST has a group email (cpe_dictionary@nist.gov) used to recieve these version update and new entry xml files. They do process the XML and provide feedback. In some cases they will propose back something different where the vendor or version is slightly different. Limitations - Currently any use of non-number version identifiers isn't supported by NIST as they use ranges to determine impact of a CVE - Any Linux version from a non-upstream is also not supported without manually adjusting the information as the custom kernel will more then likely not match the upstream version used in the dictionary Signed-off-by: Matt Weber --- Changes v8 - Updated to just output missing and needs version update - Optional processed dictionary caching support - Optional dictionary URL - Creation of a missing status file (cpe-report-missing.txt) - Adjusted index used in CSV for removal of CVE patched item v5 -> v7 - No change v5 [Ricardo - Updated v4 comments about general flake formatting cleanup - Incorporated parts of patch 1/2 suggestions for optimizations [Ricardo/Arnout - Collectly, decided to move cpe report analysis to this script and use a seperate module cpedb class [Arnout - Rename cpe_dict to instead be cpedb v1 -> v4 - Patch did not exist and was part of pkg-stats file --- support/scripts/cpe-report | 70 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) create mode 100755 support/scripts/cpe-report diff --git a/support/scripts/cpe-report b/support/scripts/cpe-report new file mode 100755 index 0000000000..7242a372b2 --- /dev/null +++ b/support/scripts/cpe-report @@ -0,0 +1,70 @@ +#!/usr/bin/env python + +import argparse +import sys +import csv +from cpedb import CPEDB + +CPE_XML_URL = "https://static.nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz" + + +def get_target_cpe_report(cpe_report_file, cpedb): + report_cpe_exact_match = "" + report_cpe_needing_update = "" + report_cpe_needing_update_list = "" + report_cpe_missing = "" + + print("CPE: Checking for matches...") + try: + with open(cpe_report_file) as cpe_file: + cpe_list = csv.reader(cpe_file) + next(cpe_list) # make cpe-info has a one line header + for cpe in cpe_list: + result = cpedb.find(cpe[0]) + if not result: + result = cpedb.find_partial(cpedb.get_cpe_no_version(cpe[0])) + if not result: + report_cpe_missing += cpe[0] + "," + cpe[1] + "," + cpe[3] + "\n" + else: + latest_version = cpedb.find_partial_latest_version(cpedb.get_cpe_no_version(cpe[0])) + report_cpe_needing_update += cpe[0] + ", Latest Version Guess from Dict[" + latest_version + "]\n" + report_cpe_needing_update_list += cpe[0] + "\n" + else: + report_cpe_exact_match += cpe[0] + "\n" + except (OSError, IOError) as e: + print("CPE: report csv file (%s): %s" % (e.errno, e.strerror)) + sys.exit(1) + + print("CPE: Found but may REQUIRE an UPDATE:\n" + report_cpe_needing_update) + print("CPE: Not found:\n" + report_cpe_missing) + + fp = open('cpe-report-missing.txt', 'w+') + fp.write(report_cpe_missing) + fp.close() + + for cpe in report_cpe_needing_update_list.splitlines(): + cpedb.update(cpe) + print("XML Generation Complete of NIST update files, see ./cpe/*") + + +def parse_args(): + parser = argparse.ArgumentParser() + parser.add_argument('-c', dest='cpe_report', action='store', required=True, + help='CPE Report generated by make cpe-info (csv format)') + parser.add_argument('-u', dest='url', action='store', required=False, + help='(optional)URL to the NIST dict (official-cpe-dictionary_v2.3.xml.gz)') + return parser.parse_args() + + +def __main__(): + args = parse_args() + cpedb = CPEDB() + url = CPE_XML_URL + if args.url: + url = args.url + cpedb.get_xml_dict(url) + print("Performing Target CPE Report Analysis...") + get_target_cpe_report(args.cpe_report, cpedb) + + +__main__()