From patchwork Wed Nov 28 19:36:28 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Higgins X-Patchwork-Id: 1004798 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:e::133; helo=bombadil.infradead.org; envelope-from=linux-um-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="TI1rq1kL"; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.b="r5egsVPR"; dkim-atps=neutral Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:e::133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 434rYF4R7Qz9s3l for ; Thu, 29 Nov 2018 06:38:25 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=vhs2J1l5uLB/lE+V2Ti47QbzsR6rD7h+7lYcb/Hmrf0=; b=TI1rq1kLPeKNxu IbsUcIfd/JwpHcxpAaGIwgZMNa8pFpL64bGyvXhHNEWjDBC5cFC/NAYSjiMOG3ccOs9rUfv7EBT08 ZgHb22sABAm0C/Q06taCltCKzPxM8W0aOFKj/LrSLEDs+gMNBoZwh1DvRUrPuxuQVogOMjzBEU5yv 0zybW6xQnwNB8njt+cc7vBivKv+68OfBB/8044s3o4zgPVK28SD78+CHN/VymsA3NDFCU2fobZrR/ ewDgiWa02k8NWm+hlxTPJtnl+gbpfbdMzGLUsSfTDehk14T8FdOL9tKw4xrWty53/grOOjz1FB6+I rk3KRpriH4KcPZxchFfA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gS5f6-0003gQ-JN; Wed, 28 Nov 2018 19:38:16 +0000 Received: from mail-qt1-x84a.google.com ([2607:f8b0:4864:20::84a]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1gS5f3-0003dm-0f for linux-um@lists.infradead.org; Wed, 28 Nov 2018 19:38:14 +0000 Received: by mail-qt1-x84a.google.com with SMTP id n45so25100347qta.5 for ; Wed, 28 Nov 2018 11:38:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=1xbhMNvec4B5xSL1oHl4qV1ywjoofQ6muA5OEf5mErs=; b=r5egsVPRtZM1HiSYPImpPHgaH/CRRTATj35KzJJyDHka55x2wrsE33cx+AjID9zNBs wDl7b+lgf5EXTzSL6dm3PZL1l2IHSLReffNuF9QESIRxzvyJ8YdjxFKR+9FpUx54h6QM p0zzWxOZQ4nGxJqiHkY6IQhw3/9mCvKkz8XPMhFCrQJetPWFAWoa4qL/phXTSdJ93o64 oYLhFcm/fQJNxeK3gh06IQxgJFWl3iyOZubW9VDx3tzfF+RZIac3DLldmaqpaQvgmOPI F44sZgceb68ovu2sBcpMLosRACXuUwStlhbR9a4u05NGltWhikJwTHtMC3pvlZfbvwtJ +4cQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=1xbhMNvec4B5xSL1oHl4qV1ywjoofQ6muA5OEf5mErs=; b=UKfIJBjSJHGh5r7zNPfcQpfRrSr9Ji60FG8MtSqbkEbklBEuOyH8OZj0BLHl+uCFYh F9Bzh6eQI5wvczJS7mbwyKD6hWMXSqyQmROTUcrPhQdN/E/9n/73qEh07QVRU6NSe899 JVCEK0hGsLkNKW9WjaLanfSGL66S44ZA717+WtQtLl+5V0aXVctG7BxlOcSIIHXtCt4s KEki9MJJ4qAtoDLLzmzyrg2Q1TFpRp/gf+AVDJ/bDc535muQg+hALziIlTBkp+OQgbIy J/IMDweORMn2TWXg6nCtusk1RKDKA3umZt717PEPFLDEADo1Xf/pcSN5SUbgBSw30BvF OLvg== X-Gm-Message-State: AA+aEWb54QYzxwV5Nc90G/upiCkw+4lWUgqpt4WaqWLQtZFUlS6KU2Mg 3veAYCBG/GNrcVot5IEbuxjMUNg13Pmz86tvM1NNXw== X-Google-Smtp-Source: AFSGD/Vp+eB7mYlveEADB2/q7q2KIhbKFmASYVVYOI9B5oqatosjGn+i5nND9oFLxE3mJ+hoTvDPmtlDSUMyjBueDGCklw== X-Received: by 2002:a37:67d7:: with SMTP id b206mr4522605qkc.17.1543433891235; Wed, 28 Nov 2018 11:38:11 -0800 (PST) Date: Wed, 28 Nov 2018 11:36:28 -0800 In-Reply-To: <20181128193636.254378-1-brendanhiggins@google.com> Message-Id: <20181128193636.254378-12-brendanhiggins@google.com> Mime-Version: 1.0 References: <20181128193636.254378-1-brendanhiggins@google.com> X-Mailer: git-send-email 2.20.0.rc0.387.gc7a69e6b6c-goog Subject: [RFC v3 11/19] kunit: add Python libraries for handing KUnit config and kernel From: Brendan Higgins To: gregkh@linuxfoundation.org, keescook@google.com, mcgrof@kernel.org, shuah@kernel.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20181128_113813_055942_7A753005 X-CRM114-Status: GOOD ( 13.49 ) X-Spam-Score: -7.7 (-------) X-Spam-Report: SpamAssassin version 3.4.2 on bombadil.infradead.org summary: Content analysis details: (-7.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:84a listed in] [list.dnswl.org] -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -0.0 SPF_PASS SPF: sender matches SPF record 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -0.0 DKIMWL_WL_MED DKIMwl.org - Whitelisted Medium sender X-BeenThere: linux-um@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: brakmo@fb.com, jdike@addtoit.com, dri-devel@lists.freedesktop.org, linux-kselftest@vger.kernel.org, frowand.list@gmail.com, robh@kernel.org, linux-nvdimm@lists.01.org, richard@nod.at, knut.omang@oracle.com, kieran.bingham@ideasonboard.com, Felix Guo , joel@jms.id.au, khilman@baylibre.com, Brendan Higgins , Tim.Bird@sony.com, linux-um@lists.infradead.org, rostedt@goodmis.org, julia.lawall@lip6.fr, dan.j.williams@intel.com, kunit-dev@googlegroups.com, linux-kernel@vger.kernel.org, daniel@ffwll.ch, mpe@ellerman.id.au, joe@perches.com Sender: "linux-um" Errors-To: linux-um-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org The ultimate goal is to create minimal isolated test binaries; in the meantime we are using UML to provide the infrastructure to run tests, so define an abstract way to configure and run tests that allow us to change the context in which tests are built without affecting the user. This also makes pretty and dynamic error reporting, and a lot of other nice features easier. kunit_config.py: - parse .config and Kconfig files. kunit_kernel.py: provides helper functions to: - configure the kernel using kunitconfig. - build the kernel with the appropriate configuration. - provide function to invoke the kernel and stream the output back. Signed-off-by: Felix Guo Signed-off-by: Brendan Higgins --- tools/testing/kunit/.gitignore | 3 + tools/testing/kunit/kunit_config.py | 60 +++++++++++++ tools/testing/kunit/kunit_kernel.py | 126 ++++++++++++++++++++++++++++ 3 files changed, 189 insertions(+) create mode 100644 tools/testing/kunit/.gitignore create mode 100644 tools/testing/kunit/kunit_config.py create mode 100644 tools/testing/kunit/kunit_kernel.py diff --git a/tools/testing/kunit/.gitignore b/tools/testing/kunit/.gitignore new file mode 100644 index 0000000000000..c791ff59a37a9 --- /dev/null +++ b/tools/testing/kunit/.gitignore @@ -0,0 +1,3 @@ +# Byte-compiled / optimized / DLL files +__pycache__/ +*.py[cod] \ No newline at end of file diff --git a/tools/testing/kunit/kunit_config.py b/tools/testing/kunit/kunit_config.py new file mode 100644 index 0000000000000..183bd5e758762 --- /dev/null +++ b/tools/testing/kunit/kunit_config.py @@ -0,0 +1,60 @@ +# SPDX-License-Identifier: GPL-2.0 + +import collections +import re + +CONFIG_IS_NOT_SET_PATTERN = r'^# CONFIG_\w+ is not set$' +CONFIG_PATTERN = r'^CONFIG_\w+=\S+$' + +KconfigEntryBase = collections.namedtuple('KconfigEntry', ['raw_entry']) + + +class KconfigEntry(KconfigEntryBase): + + def __str__(self) -> str: + return self.raw_entry + + +class KconfigParseError(Exception): + """Error parsing Kconfig defconfig or .config.""" + + +class Kconfig(object): + """Represents defconfig or .config specified using the Kconfig language.""" + + def __init__(self): + self._entries = [] + + def entries(self): + return set(self._entries) + + def add_entry(self, entry: KconfigEntry) -> None: + self._entries.append(entry) + + def is_subset_of(self, other: "Kconfig") -> bool: + return self.entries().issubset(other.entries()) + + def write_to_file(self, path: str) -> None: + with open(path, 'w') as f: + for entry in self.entries(): + f.write(str(entry) + '\n') + + def parse_from_string(self, blob: str) -> None: + """Parses a string containing KconfigEntrys and populates this Kconfig.""" + self._entries = [] + is_not_set_matcher = re.compile(CONFIG_IS_NOT_SET_PATTERN) + config_matcher = re.compile(CONFIG_PATTERN) + for line in blob.split('\n'): + line = line.strip() + if not line: + continue + elif config_matcher.match(line) or is_not_set_matcher.match(line): + self._entries.append(KconfigEntry(line)) + elif line[0] == '#': + continue + else: + raise KconfigParseError('Failed to parse: ' + line) + + def read_from_file(self, path: str) -> None: + with open(path, 'r') as f: + self.parse_from_string(f.read()) diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py new file mode 100644 index 0000000000000..bba7ea7ca1869 --- /dev/null +++ b/tools/testing/kunit/kunit_kernel.py @@ -0,0 +1,126 @@ +# SPDX-License-Identifier: GPL-2.0 + +import logging +import subprocess +import os + +import kunit_config + +KCONFIG_PATH = '.config' + +class ConfigError(Exception): + """Represents an error trying to configure the Linux kernel.""" + + +class BuildError(Exception): + """Represents an error trying to build the Linux kernel.""" + + +class LinuxSourceTreeOperations(object): + """An abstraction over command line operations performed on a source tree.""" + + def make_mrproper(self): + try: + subprocess.check_output(['make', 'mrproper']) + except OSError as e: + raise ConfigError('Could not call make command: ' + e) + except subprocess.CalledProcessError as e: + raise ConfigError(e.output) + + def make_olddefconfig(self): + try: + subprocess.check_output(['make', 'ARCH=um', 'olddefconfig']) + except OSError as e: + raise ConfigError('Could not call make command: ' + e) + except subprocess.CalledProcessError as e: + raise ConfigError(e.output) + + def make(self, jobs): + try: + subprocess.check_output([ + 'make', + 'ARCH=um', + '--jobs=' + str(jobs)]) + except OSError as e: + raise BuildError('Could not call execute make: ' + e) + except subprocess.CalledProcessError as e: + raise BuildError(e.output) + + def linux_bin(self, params, timeout): + """Runs the Linux UML binary. Must be named 'linux'.""" + process = subprocess.Popen( + ['./linux'] + params, + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE) + process.wait(timeout=timeout) + return process + + +class LinuxSourceTree(object): + """Represents a Linux kernel source tree with KUnit tests.""" + + def __init__(self): + self._kconfig = kunit_config.Kconfig() + self._kconfig.read_from_file('kunitconfig') + self._ops = LinuxSourceTreeOperations() + + def clean(self): + try: + self._ops.make_mrproper() + except ConfigError as e: + logging.error(e) + return False + return True + + def build_config(self): + self._kconfig.write_to_file(KCONFIG_PATH) + try: + self._ops.make_olddefconfig() + except ConfigError as e: + logging.error(e) + return False + validated_kconfig = kunit_config.Kconfig() + validated_kconfig.read_from_file(KCONFIG_PATH) + if not self._kconfig.is_subset_of(validated_kconfig): + logging.error('Provided Kconfig is not contained in validated .config!') + return False + return True + + def build_reconfig(self): + """Creates a new .config if it is not a subset of the kunitconfig.""" + if os.path.exists(KCONFIG_PATH): + existing_kconfig = kunit_config.Kconfig() + existing_kconfig.read_from_file(KCONFIG_PATH) + if not self._kconfig.is_subset_of(existing_kconfig): + print('Regenerating .config ...') + os.remove(KCONFIG_PATH) + return self.build_config() + else: + return True + else: + print('Generating .config ...') + return self.build_config() + + def build_um_kernel(self, jobs): + try: + self._ops.make_olddefconfig() + self._ops.make(jobs) + except (ConfigError, BuildError) as e: + logging.error(e) + return False + used_kconfig = kunit_config.Kconfig() + used_kconfig.read_from_file(KCONFIG_PATH) + if not self._kconfig.is_subset_of(used_kconfig): + logging.error('Provided Kconfig is not contained in final config!') + return False + return True + + def run_kernel(self, args=[]): + timeout = None + args.extend(['mem=256M']) + process = self._ops.linux_bin(args, timeout) + with open('test.log', 'w') as f: + for line in process.stdout: + f.write(line.rstrip().decode('ascii') + '\n') + yield line.rstrip().decode('ascii')