From patchwork Tue May 16 06:58:19 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chris Packham X-Patchwork-Id: 762804 X-Patchwork-Delegate: trini@ti.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.denx.de (dione.denx.de [81.169.180.215]) by ozlabs.org (Postfix) with ESMTP id 3wRpFR1dwkz9s7k for ; Tue, 16 May 2017 16:58:34 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="HjksNKra"; dkim-atps=neutral Received: by lists.denx.de (Postfix, from userid 105) id EFA1CC2222E; Tue, 16 May 2017 06:58:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on lists.denx.de X-Spam-Level: X-Spam-Status: No, score=0.0 required=5.0 tests=FREEMAIL_FROM, RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2, T_DKIM_INVALID autolearn=unavailable autolearn_force=no version=3.4.0 Received: from lists.denx.de (localhost [IPv6:::1]) by lists.denx.de (Postfix) with ESMTP id 0DD19C21D66; Tue, 16 May 2017 06:58:24 +0000 (UTC) Received: by lists.denx.de (Postfix, from userid 105) id 8AFBAC21D66; Tue, 16 May 2017 06:58:22 +0000 (UTC) Received: from mail-io0-f175.google.com (mail-io0-f175.google.com [209.85.223.175]) by lists.denx.de (Postfix) with ESMTPS id AB480C21CA4 for ; Tue, 16 May 2017 06:58:21 +0000 (UTC) Received: by mail-io0-f175.google.com with SMTP id p24so87783440ioi.0 for ; Mon, 15 May 2017 23:58:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=mjbakXmIgw8QJxvm9om1t9d/f2/L8xLY4FPFNEJjhTk=; b=HjksNKra24yLJcubZ+TLHovMTuAxXNuVzYPG9LDddqkxeCu5gkOWs2K29IN/RfZq/r yoktCWhZl02B3J1+aoxEOOfUaz2flwCGLaHoYj3DhpjvMClLCnZKqAnOMR8E1PSzbVa+ Bqmie+8vnNJoJ936zAx8Yi6sRMY7sr/kurvN3C3aGUEP3/3zi4584jS6+D4V5/+doD1W Kfb2Z/qvP7sid5RJGImRjDTsZh+uSQc+ELzXFVBJMENpfZNoYbEjceRkJjlRpOoPE4Wd 34dW3m0csDFsqDzIhBWbiST54achy4o0RBR1Te1lZO6WS4ujhRHtXuU2WDcdHLT8Lx5m 4EHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=mjbakXmIgw8QJxvm9om1t9d/f2/L8xLY4FPFNEJjhTk=; b=NewDeTJQ3NOECGEa9fqP6hpZj+0xHYn/zbbhvpTHz3D7P8iSG+Xx70xOe7Nqol4fGa j3CK1eV/q9CqIugOrAgm2/djhl0szQO+aDnW614xU7tuO/DbfxbJZ7TLusaZCZeDc+tH CPN5EgaQJnyDefwC4X2mJ4Tiy/HctgplPTNJxZr1b/hbOZXvAkVDaVMO0k+1NRuWdZWH FQ7xpHPxNXe/FZcGAR46wDP0kjTvyL5fbkK5hEeo7OjPyXykoZdk4VoqU0akLSqsGpEE SKCZrBqtbkfym/8NKh02MNF1ST0Ys2WwqbAbRb9rsSQjzGgKOdx/9FAfT95B6CwAXQmf vcCQ== X-Gm-Message-State: AODbwcDSpwcez/5pd3Xfchq05fRPBIkajE85LcYQuRhu5eNgtCo+BZea kK4N09Q9xVHc2ZnecI+SW5uX8SHTbQ== X-Received: by 10.107.130.25 with SMTP id e25mr8843726iod.31.1494917900192; Mon, 15 May 2017 23:58:20 -0700 (PDT) MIME-Version: 1.0 Received: by 10.79.169.2 with HTTP; Mon, 15 May 2017 23:58:19 -0700 (PDT) Received: by 10.79.169.2 with HTTP; Mon, 15 May 2017 23:58:19 -0700 (PDT) In-Reply-To: References: <20170515114736.17521-1-sjg@chromium.org> <20170515114736.17521-6-sjg@chromium.org> From: Chris Packham Date: Tue, 16 May 2017 18:58:19 +1200 Message-ID: To: Simon Glass X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: Joe Hershberger , Tom Rini , u-boot Subject: Re: [U-Boot] [PATCH 5/6] moveconfig: Support building a simple config database X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.18 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" On 15/05/2017 11:47 PM, "Simon Glass" wrote: Add a -b option which scans all the defconfigs and builds a database of all the CONFIG options used by each. This is useful for querying later. At present this only works with the separate -b option, which does not move any configs. It would be possible to adjust the script to build the database automatically when moving configs, but this might not be useful as the database does not change that often. Signed-off-by: Simon Glass --- tools/moveconfig.py | 81 ++++++++++++++++++++++++++++++ ++++++++++++++++++----- 1 file changed, 74 insertions(+), 7 deletions(-) Arguments: @@ -957,6 +989,7 @@ class Slot: make_cmd: command name of GNU Make. reference_src_dir: Determine the true starting config state from this source tree. + db_queue: output queue to write config info for the database """ self.options = options self.progress = progress @@ -964,6 +997,7 @@ class Slot: self.devnull = devnull self.make_cmd = (make_cmd, 'O=' + self.build_dir) self.reference_src_dir = reference_src_dir + self.db_queue = db_queue self.parser = KconfigParser(configs, options, self.build_dir) self.state = STATE_IDLE self.failed_boards = set() @@ -1039,6 +1073,8 @@ class Slot: if self.current_src_dir: self.current_src_dir = None self.do_defconfig() + elif self.options.build_db: + self.do_build_db() else: self.do_savedefconfig() elif self.state == STATE_SAVEDEFCONFIG: @@ -1088,6 +1124,17 @@ class Slot: cwd=self.current_src_dir) self.state = STATE_AUTOCONF + def do_build_db(self): + """Add the board to the database""" + configs = {} + with open(os.path.join(self.build_dir, AUTO_CONF_PATH)) as fd: + for line in fd.readlines(): + if line.startswith('CONFIG'): + config, value = line.split('=', 1) + configs[config] = value.rstrip() + self.db_queue.put([self.defconfig, configs]) + self.finish(True) + def do_savedefconfig(self): """Update the .config and run 'make savedefconfig'.""" @@ -1170,7 +1217,7 @@ class Slots: """Controller of the array of subprocess slots.""" - def __init__(self, configs, options, progress, reference_src_dir): + def __init__(self, configs, options, progress, reference_src_dir, db_queue): """Create a new slots controller. Arguments: @@ -1179,6 +1226,7 @@ class Slots: progress: A progress indicator. reference_src_dir: Determine the true starting config state from this source tree. + db_queue: output queue to write config info for the database """ self.options = options self.slots = [] @@ -1186,7 +1234,7 @@ class Slots: make_cmd = get_make_cmd() for i in range(options.jobs): self.slots.append(Slot(configs, options, progress, devnull, - make_cmd, reference_src_dir)) + make_cmd, reference_src_dir, db_queue)) def add(self, defconfig): """Add a new subprocess if a vacant slot is found. @@ -1298,7 +1346,7 @@ class ReferenceSource: return self.src_dir -def move_config(configs, options): +def move_config(configs, options, db_queue): """Move config options to defconfig files. Arguments: @@ -1308,6 +1356,8 @@ def move_config(configs, options): if len(configs) == 0: if options.force_sync: print 'No CONFIG is specified. You are probably syncing defconfigs.', + elif options.build_db: + print 'Building %s database' % CONFIG_DATABASE else: print 'Neither CONFIG nor --force-sync is specified. Nothing will happen.', else: @@ -1326,7 +1376,7 @@ def move_config(configs, options): defconfigs = get_all_defconfigs() progress = Progress(len(defconfigs)) - slots = Slots(configs, options, progress, reference_src_dir) + slots = Slots(configs, options, progress, reference_src_dir, db_queue) # Main loop to process defconfig files: # Add a new subprocess into a vacant slot. @@ -1353,6 +1403,8 @@ def main(): parser = optparse.OptionParser() # Add options here + parser.add_option('-b', '--build-db', action='store_true', default=False, + help='build a CONFIG database') parser.add_option('-c', '--color', action='store_true', default=False, help='display the log in color') parser.add_option('-C', '--commit', action='store_true', default=False, @@ -1385,7 +1437,7 @@ def main(): (options, configs) = parser.parse_args() - if len(configs) == 0 and not options.force_sync: + if len(configs) == 0 and not any((options.force_sync, options.build_db)): parser.print_usage() sys.exit(1) @@ -1395,10 +1447,17 @@ def main(): check_top_directory() + config_db = {} + db_queue = Queue.Queue() + t = DatabaseThread(config_db, db_queue) + t.setDaemon(True) + t.start() + if not options.cleanup_headers_only: check_clean_directory() update_cross_compile(options.color) - move_config(configs, options) + move_config(configs, options, db_queue) + db_queue.join() if configs: cleanup_headers(configs, options) @@ -1418,5 +1477,13 @@ def main(): msg += '\n\nRsync all defconfig files using moveconfig.py' subprocess.call(['git', 'commit', '-s', '-m', msg]) + if options.build_db: + with open(CONFIG_DATABASE, 'w') as fd: + for defconfig, configs in config_db.iteritems(): + print >>fd, '%s' % defconfig Not sure how much we care but the python folks would say fd.write("%s\n" % defconfig) is more natural and will work for python 2 and 3. + for config in sorted(configs.keys()): + print >>fd, ' %s=%s' % (config, configs[config]) + print >>fd + if __name__ == '__main__': main() -- 2.13.0.rc2.291.g57267f2277-goog diff --git a/tools/moveconfig.py b/tools/moveconfig.py index ba40c33521..ed576f4b83 100755 --- a/tools/moveconfig.py +++ b/tools/moveconfig.py @@ -199,11 +199,13 @@ import glob import multiprocessing import optparse import os +import Queue import re import shutil import subprocess import sys import tempfile +import threading import time SHOW_GNU_MAKE = 'scripts/show-gnu-make' @@ -260,6 +262,7 @@ COLOR_LIGHT_CYAN = '1;36' COLOR_WHITE = '1;37' AUTO_CONF_PATH = 'include/config/auto.conf' +CONFIG_DATABASE = 'moveconfig.db' ### helper functions ### @@ -937,6 +940,34 @@ class KconfigParser: return log + +class DatabaseThread(threading.Thread): + """This thread processes results from Slot threads. + + It collects the data in the master config directary. There is only one + result thread, and this helps to serialise the build output. + """ + def __init__(self, config_db, db_queue): + """Set up a new result thread + + Args: + builder: Builder which will be sent each result + """ + threading.Thread.__init__(self) + self.config_db = config_db + self.db_queue= db_queue + + def run(self): + """Called to start up the result thread. + + We collect the next result job and pass it on to the build. + """ + while True: + defconfig, configs = self.db_queue.get() + self.config_db[defconfig] = configs + self.db_queue.task_done() + + class Slot: """A slot to store a subprocess. @@ -946,7 +977,8 @@ class Slot: for faster processing. """ - def __init__(self, configs, options, progress, devnull, make_cmd, reference_src_dir): + def __init__(self, configs, options, progress, devnull, make_cmd, + reference_src_dir, db_queue): """Create a new process slot.