From patchwork Fri Aug 19 14:25:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 1668177 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=weidmueller.onmicrosoft.com header.i=@weidmueller.onmicrosoft.com header.a=rsa-sha256 header.s=selector1-weidmueller-onmicrosoft-com header.b=Aunu4njN; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4M8PKJ5FvQz1yg7 for ; Sat, 20 Aug 2022 00:29:40 +1000 (AEST) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 462AA84C0D; Fri, 19 Aug 2022 16:27:03 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=fail (p=none dis=none) header.from=weidmueller.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=weidmueller.onmicrosoft.com header.i=@weidmueller.onmicrosoft.com header.b="Aunu4njN"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id DC06384B67; Fri, 19 Aug 2022 16:26:22 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-1.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,SPF_HELO_PASS,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no version=3.4.2 Received: from EUR01-DB5-obe.outbound.protection.outlook.com (mail-eopbgr150088.outbound.protection.outlook.com [40.107.15.88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 08B3484B92 for ; Fri, 19 Aug 2022 16:26:04 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=weidmueller.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=stefan.herbrechtsmeier-oss@weidmueller.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=oH0hrHiCqLgIVB4bDoqsoyRr+3ofCDuAYhkGnVQrPJuuGM/ispa1lnvIzFHE0ZHEj0dwn+rOf13KH/abHoufZ7b07NcF/u/Mfl2sEyhqFK8DDcAkqCynlhALz78vrTqBA/Hw8KH11DyAI0qg3cw9k4M0EwugJG05g8TE/oYsx5KXbJv25JPXQaKyV/RthpnAEwGrDC4EuJB3ar4GQCelDlS/riKJjLRh4dfSWhHEY3kBZwtAL1tQrNMnFAklfUTWbUKCo1pBWhrYweB52/u62LEWPRxyA4IrzXDkmdAQQ6CNTDzX/psBPnEOeybKrY4wJlck6ttpSeW2W704VwAnCg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=iuaSTspMhbYdOaeYdvoRjCQ/MaYYulblApgdCdfIAck=; b=kCBcyEfXUkfptvPNyDENgm6yO+yqSLBq7stNs+cVhGJNsQ96zV59czJ/00zYKGY4a8Bbk+DiKmPS+LEs5yu396L8uBIyByJ1ITmqPlQ0b1uvBmn7D+0yZBV7mFQ0xprFh08HxNcyXuRP/2HS/xfz2EIR0DUgGzvr25WpkwGVe7tZV5L3FpGJtbTtvhJBkjEmhjx1aNlLXVA5dzE4U8oryCJvQjmZ0jP/XbER327eENVHm5o2q0jMOTTBjlzW9Bjjw5MBAK/FKWrUF2/qDc6QhMn0qSeTFbsQEDNgN1QEE5+l5p/QqDDnEDXXBozOyyjTA+cKvA1JonF+4bSjaVs6Fw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.onmicrosoft.com; s=selector1-weidmueller-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iuaSTspMhbYdOaeYdvoRjCQ/MaYYulblApgdCdfIAck=; b=Aunu4njNUuW8SE6D0usILVo5Xp2xP1G9lz2SRQa11AtrUlr0E5hKrtNQQNwnjW39NAUeh6o+6uTzX2piAYnGev5J0QASeEggE5r/rfRYlAHSVWe9J/6DPoDQRB6RCJcfgBTATpFkxwpIY7MU2ebEpFp/b36UMcDjSbz4Sk/E2bc= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by PR3PR08MB5739.eurprd08.prod.outlook.com (2603:10a6:102:8e::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5546.18; Fri, 19 Aug 2022 14:26:03 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::f95b:2e64:9ee6:4d44]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::f95b:2e64:9ee6:4d44%3]) with mapi id 15.20.5546.018; Fri, 19 Aug 2022 14:26:03 +0000 From: Stefan Herbrechtsmeier To: u-boot@lists.denx.de Cc: Stefan Herbrechtsmeier , Alper Nebi Yasak , Simon Glass Subject: [PATCH v6 13/21] binman: Move compression bintool management into entry class Date: Fri, 19 Aug 2022 16:25:30 +0200 Message-Id: <20220819142538.24847-14-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220819142538.24847-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20220819142538.24847-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: AS4PR10CA0015.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:5dc::11) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: dd00b973-5ad6-4bbf-040e-08da81eebebb X-MS-TrafficTypeDiagnostic: PR3PR08MB5739:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: zx+I/1mLSWYgz4eiMtYFzhNHAQ/95JkXd//Zc8uY6kZAdNstkwzN/tvoXYDzyu2WUnHBa9QopVxCQBv85l1y1XVIgPh1ugJ6ebq7zSxXfJhr9f7YJ1aEVyL7gUsHxA8/5PMPFDR/p9OPoCvwvshvys9Qhrdkfq91uYL6WLVoBRJPKRbNce1a2deMXaLS5P+CQw5C54y2p0oJ4U1x96pqDMOiuv5hwSpxOzjsm5ufUT/5llgmtntWjpUyviDekZg0nObU9epqiBjmKRAS3i7hsW28GiOQ8ogxcsAqFNSFg8pSSxt8NO16jmDphpVBmgAg5imZe8qtO0q7pYcdtFIigm2H79cybGlGSSheAgQaGiwOfRcJGutyVxOeNtvk/WlclsRzWXL6VfFbDfU+u1lFTXZCOSRvbv0Wct/LH9j4oZJL/cQDN05rHZg1pz+sfb0gAl7OjZ/KvymPwsx8Z7XdWdgiTM0hyBgRDHmFfUCgZXchqNMMWlD1ASnuN0eSFsu/ADu88jyi+W86AVLXA01lbdqlqFMm5vw3yUkXFZDsLzOa52T5rt/CnmUcPqnJQ3TeChPZVU0XDVcqMRtTtn5gLCneh2hwWGrNxOYJcDd+ikQImVXQskSpiofbG7xqyAwWEieBcYlMfYFIwQkBsMd9cjV/erUS8KaQ7nPG5zUiJKamu0HSqb30qeWkSFuvgQErjm5KU5RyT0sFKGg5B56sCfePm0JS//zQ6aAdlwPVRRbpqAClvMsSeHhcKdu13b+hjeAHct758j3Z7JbVnusc2g== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PAXPR08MB6969.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230016)(4636009)(396003)(366004)(376002)(136003)(346002)(39860400002)(4326008)(8676002)(66476007)(66946007)(66556008)(54906003)(316002)(6916009)(36756003)(5660300002)(8936002)(2906002)(478600001)(6486002)(41300700001)(6506007)(6666004)(6512007)(26005)(52116002)(2616005)(86362001)(1076003)(38350700002)(38100700002)(83380400001)(186003); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: AP0MqeJgQ0FqFnseBxb5Ak3OhMlMdeWniuWz88CiOqdsTNa7W20rOj6pvXcRwx+jiwS4IsBE+F4G8s9tGlOsSVAyQvffx08u+PBabsLIJsE4/yTKOIXHJ475XWwmcgc8tAPBNeELexO5jqLpLhtaQ/DxAfXE0VJ8TzgIySvaCGhxhqAZHTOvLL/ACaIGQoriauvPy/yIu3MCxA9/85OmuIipqqh5v1X73ELlQxhTRhi9hD4ErQ0HCCUfgUJ38VDg7R0perTkR4GKmwj9NKIY21/fQUXDAPUEqhGPDpUHh4pH3NmFzb6dw21Vu0mbtgb4cAqjneH4tynJzMIFKEP8srsXZ72/vp1zvJQMAom2azHRkvsgpFYBnHf0oYXbnVFrt4lFFa/80W3ZIxLThIKUwaw9zWELu9tWmqfvJ2oSkvI2F3C6ngA/5oM1E2tX+DVgvqs6ojX7LVidp4N052i6IMsbkiHq3jQjP+dQPffTc803yTLafLbcgq7wTBayBNlysf1yz3U6JURs8hTAKa9a4ojArKfd3IbEv3O55x+0JZEfR5K0qkzFttqcDs8ATfvRFuYCJfgukclgsFR96xQFrlEu67jIUFzejqnUNl8h27TE393VVudVm/DS/w+p1zMzi//itQ/4w+K6rcas7knjEDjZO4P9EpKdd197lfjLMBzRgP1Acf56nvWYlM4BptqgikRzelKdUBBy1vqU6NXYS+Ye0KLSPV6PSgMKdeG39qCFRg1rve/amb6SR/IQIFf70OA14gv0dQM3w37CzmuDC9wDD4uWuHMG7TWxeK6ob/UAWY6T7hxnMgRG87ruMQWaVvOKi14YJ70RYIuseA2Iw9giw7xW//e/GrtqsyucbOH1Iq8Q3EHrCXK/Diy4iAVyH/vX1VnQHRzV9fDpZMyThu64DxmCNFhDI5FDdubA0NCvEGSN/DN7luWb3DZwS1b+Nnq1Nn+UJcCYLWeh0UxVcZl7kYOkM2YfiyVXwMVp8JHmE7zeu1KPe/cbq+Z5smBaajA4PTaWfpqTNFbp9Vz2AVMuDVCxOMDvDGUgIpKn6GOMLYG/zFAaX4TWAbTXSOhWNV1ONPcXsExubRyJekBKS2RBne0NvvB35+nDHIy/Slm0QczZ/m2QIPPvOINy2RzWNsudOz0lJRZaNp3ZFU/sRsuqc76FfL+CpXptfE7cg77Pq2tZLKurivcNZ+Gf9IW+Tutp8KiawBvvOAfJrtrrSQ6f7RJKvpH/DKE405GV9f8a5mcLjT/2dxh4a6z+xdXudb41pURLOUogGfnXMDmtT1RmVgAAuejwN/1WuOajV5ZMzkX7zAIhUDmyW+vKZ4dlqQKQ4dC4h6IsVUcgKecratOYVZ3c1vcL9aRbHuT+lF8oLgKBFMqg/WDzu12qt0GYyo78YOMEZ2cfOkJ1+tOuyR/68of0xvycXRFzqDS0qE/G2Q86RZfM3nX4AxWHVhfamc+gcONGJ6qLoeJW80NQn52+kBWSGdP87v1Jj9Ocbp9W2hMmSrWA0vrxH0q1sm/YsD69+ozQfnX3LVlY/XsolhDErrvkmZnyDi11DHkZWxLcWY4uRTLFs2isGnFIvsTOfY8/UJdAfwnpRLtoYJsyOA== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: dd00b973-5ad6-4bbf-040e-08da81eebebb X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Aug 2022 14:26:02.9410 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: qr5VrFaOAo1cXfnad4lvV31yOk+QnqOc6Uzl2vV0gDJDWo5uD5snWi2pAV+v5/7GWSxT7n1iYupdsH55H1yBHQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PR3PR08MB5739 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.6 at phobos.denx.de X-Virus-Status: Clean From: Stefan Herbrechtsmeier Move management of the bintool to compress and decompress data into the entry class and add the bintool to the list of required bintools. Signed-off-by: Stefan Herbrechtsmeier Reviewed-by: Simon Glass Reviewed-by: Simon Glass --- Changes in v6: - Set uncomp_size after decompress in DecompressData function Changes in v5: - Add commit to move compression bintool management into entry class tools/binman/comp_util.py | 69 ------------------- tools/binman/entry.py | 23 +++++-- tools/binman/ftest.py | 15 ++-- .../binman/test/237_compress_dtb_invalid.dts | 16 +++++ 4 files changed, 41 insertions(+), 82 deletions(-) delete mode 100644 tools/binman/comp_util.py create mode 100644 tools/binman/test/237_compress_dtb_invalid.dts diff --git a/tools/binman/comp_util.py b/tools/binman/comp_util.py deleted file mode 100644 index 269bbf7975..0000000000 --- a/tools/binman/comp_util.py +++ /dev/null @@ -1,69 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0+ -# Copyright 2022 Google LLC -# -"""Utilities to compress and decompress data""" - -import tempfile - -from binman import bintool -from patman import tools - -LZ4 = bintool.Bintool.create('lz4') -HAVE_LZ4 = LZ4.is_present() - -LZMA_ALONE = bintool.Bintool.create('lzma_alone') -HAVE_LZMA_ALONE = LZMA_ALONE.is_present() - - -def compress(indata, algo): - """Compress some data using a given algorithm - - Note that for lzma this uses an old version of the algorithm, not that - provided by xz. - - This requires 'lz4' and 'lzma_alone' tools. It also requires an output - directory to be previously set up, by calling PrepareOutputDir(). - - Args: - indata (bytes): Input data to compress - algo (str): Algorithm to use ('none', 'lz4' or 'lzma') - - Returns: - bytes: Compressed data - """ - if algo == 'none': - return indata - if algo == 'lz4': - data = LZ4.compress(indata) - # cbfstool uses a very old version of lzma - elif algo == 'lzma': - data = LZMA_ALONE.compress(indata) - else: - raise ValueError("Unknown algorithm '%s'" % algo) - return data - -def decompress(indata, algo): - """Decompress some data using a given algorithm - - Note that for lzma this uses an old version of the algorithm, not that - provided by xz. - - This requires 'lz4' and 'lzma_alone' tools. It also requires an output - directory to be previously set up, by calling PrepareOutputDir(). - - Args: - indata (bytes): Input data to decompress - algo (str): Algorithm to use ('none', 'lz4' or 'lzma') - - Returns: - (bytes) Compressed data - """ - if algo == 'none': - return indata - if algo == 'lz4': - data = LZ4.decompress(indata) - elif algo == 'lzma': - data = LZMA_ALONE.decompress(indata) - else: - raise ValueError("Unknown algorithm '%s'" % algo) - return data diff --git a/tools/binman/entry.py b/tools/binman/entry.py index 398d5caa39..5fa826aa71 100644 --- a/tools/binman/entry.py +++ b/tools/binman/entry.py @@ -11,7 +11,6 @@ import pathlib import sys from binman import bintool -from binman import comp_util from dtoc import fdt_util from patman import tools from patman.tools import to_hex, to_hex_size @@ -82,6 +81,7 @@ class Entry(object): missing_bintools: List of missing bintools for this entry update_hash: True if this entry's "hash" subnode should be updated with a hash of the entry contents + comp_bintool: Bintools used for compress and decompress data """ def __init__(self, section, etype, node, name_prefix=''): # Put this here to allow entry-docs and help to work without libfdt @@ -116,6 +116,7 @@ class Entry(object): self.bintools = {} self.missing_bintools = [] self.update_hash = True + self.comp_bintool = None @staticmethod def FindEntryClass(etype, expanded): @@ -1083,7 +1084,9 @@ features to produce new behaviours. self.uncomp_data = indata if self.compress != 'none': self.uncomp_size = len(indata) - data = comp_util.compress(indata, self.compress) + data = self.comp_bintool.compress(indata) + else: + data = indata return data def DecompressData(self, indata): @@ -1095,9 +1098,11 @@ features to produce new behaviours. Returns: Decompressed data """ - data = comp_util.decompress(indata, self.compress) if self.compress != 'none': + data = self.comp_bintool.decompress(indata) self.uncomp_size = len(data) + else: + data = indata self.uncomp_data = data return data @@ -1138,8 +1143,18 @@ features to produce new behaviours. Args: btools (dict of Bintool): + + Raise: + ValueError if compression algorithm is not supported """ - pass + algo = self.compress + if algo != 'none': + algos = ['lz4', 'lzma'] + if algo not in algos: + raise ValueError("Unknown algorithm '%s'" % algo) + names = {'lzma': 'lzma_alone'} + name = names.get(self.compress, self.compress) + self.comp_bintool = self.AddBintool(btools, name) @classmethod def AddBintool(self, tools, name): diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py index 7ab9289a88..939b265d7c 100644 --- a/tools/binman/ftest.py +++ b/tools/binman/ftest.py @@ -5228,15 +5228,6 @@ fdt fdtmap Extract the devicetree blob from the fdtmap self._DoBinman(*args) self.assertIn('failed to fetch with all methods', stdout.getvalue()) - def testInvalidCompress(self): - with self.assertRaises(ValueError) as e: - comp_util.compress(b'', 'invalid') - self.assertIn("Unknown algorithm 'invalid'", str(e.exception)) - - with self.assertRaises(ValueError) as e: - comp_util.decompress(b'1234', 'invalid') - self.assertIn("Unknown algorithm 'invalid'", str(e.exception)) - def testBintoolDocs(self): """Test for creation of bintool documentation""" with test_util.capture_sys_output() as (stdout, stderr): @@ -5773,6 +5764,12 @@ fdt fdtmap Extract the devicetree blob from the fdtmap orig2 = self._decompress(comp_data) self.assertEqual(orig, orig2) + def testInvalidCompress(self): + """Test that invalid compress algorithm is detected""" + with self.assertRaises(ValueError) as e: + self._DoTestFile('237_compress_dtb_invalid.dts') + self.assertIn("Unknown algorithm 'invalid'", str(e.exception)) + if __name__ == "__main__": unittest.main() diff --git a/tools/binman/test/237_compress_dtb_invalid.dts b/tools/binman/test/237_compress_dtb_invalid.dts new file mode 100644 index 0000000000..228139060b --- /dev/null +++ b/tools/binman/test/237_compress_dtb_invalid.dts @@ -0,0 +1,16 @@ +// SPDX-License-Identifier: GPL-2.0+ + +/dts-v1/; + +/ { + #address-cells = <1>; + #size-cells = <1>; + + binman { + u-boot { + }; + u-boot-dtb { + compress = "invalid"; + }; + }; +};