From patchwork Fri Aug 19 14:00:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 1668140 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=85.214.62.61; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=weidmueller.onmicrosoft.com header.i=@weidmueller.onmicrosoft.com header.a=rsa-sha256 header.s=selector1-weidmueller-onmicrosoft-com header.b=rgJNy1hx; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4M8Nn63Bfyz1ygd for ; Sat, 20 Aug 2022 00:05:14 +1000 (AEST) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id D4E3284BCF; Fri, 19 Aug 2022 16:02:21 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=fail (p=none dis=none) header.from=weidmueller.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=weidmueller.onmicrosoft.com header.i=@weidmueller.onmicrosoft.com header.b="rgJNy1hx"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id D6C0B84B89; Fri, 19 Aug 2022 16:01:23 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.2 Received: from EUR01-DB5-obe.outbound.protection.outlook.com (mail-eopbgr150080.outbound.protection.outlook.com [40.107.15.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id CBA4784B76 for ; Fri, 19 Aug 2022 16:01:05 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=weidmueller.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=stefan.herbrechtsmeier-oss@weidmueller.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kqt/xqjyxLpd1HQ6EFp3Hjj5zmDg+JnP4JpixqZnS2JElzK2uyuLSgSTsgys4cS0iNHY7xBx/ecvVppz90mihwZJGI9hcGVS3AW9ghRe2s9yrvnqkBaFFZF3zrXJLkOACeEP/CL1UAXRum/fle5wtlR10o/uGmVVohOd1R4yyJmPaavg0o6gCGGgDqSOfxVV2SHKwPVVC0LnK2sRNaXFDbhcqdYTzUjam/alosUAFyTDeDRs6bx6elTVbZ2zlcDnrHQqZG3cHVWAOmLDYRN8dDXbhv9LLUS47SnRvupurlFok0krXGxYB6IpuX9jv3Yr8Iw53FOtn8Gln/RW11C+ew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jNwM05vniF+gDtlbr0b7/yPr1sU3mR4p1uY9sfWW65Q=; b=egw/GH3JWoKe51lCRvOJg2jat85tnYDPFKWbkrIM87Q+ffgLtaHi8hkOsL1RJZWdaRl51oDuXOVRy7iVgPNX6pPE8bZkgR73GktFnffD4kwKqIeg9eC5LsA8ztfKRzKBGnVtgaIcEivl36KUQr0Hn+C8WMCFuXXj+PK/C8+DPg0HCZXB5nvKOfbHzVJDPD8wevOKDGuF5oHmMX64rYVGLujHbFF9sy7c8YSS5B2o996wFNvDDVniP85/MJ3Z2Bw7aiHPj6Ff+EZeEavGp/k1jNoF++7YJhLDtfUVTSLbTUNlJ4OjcEgaPq4eNL3p6ge/Ia1zpBAlRCoB3hPWSZQyqA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.onmicrosoft.com; s=selector1-weidmueller-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jNwM05vniF+gDtlbr0b7/yPr1sU3mR4p1uY9sfWW65Q=; b=rgJNy1hxvkT+zp9NwdgJcXqMKBlxXNXv+E9+MFtdwp4NsNpUt7L4uoQbKqooXmQ5o7fy3QCScbmolFbQQd1tLhZWmyQwYv6sWsrv5a/GPMBAFoZFax/TBOa/Iu8tGOBt1pGKhI65QYwmAB5IOUrBkNNfupDTR5+n5WbiXgWKH48= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by DB9PR08MB6427.eurprd08.prod.outlook.com (2603:10a6:10:25a::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5546.18; Fri, 19 Aug 2022 14:01:04 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::f95b:2e64:9ee6:4d44]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::f95b:2e64:9ee6:4d44%3]) with mapi id 15.20.5546.018; Fri, 19 Aug 2022 14:01:04 +0000 From: Stefan Herbrechtsmeier To: u-boot@lists.denx.de Cc: Stefan Herbrechtsmeier , Alper Nebi Yasak , Simon Glass Subject: [PATCH v5 21/21] binman: Add zstd bintool Date: Fri, 19 Aug 2022 16:00:35 +0200 Message-Id: <20220819140035.32760-22-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220819140035.32760-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20220819140035.32760-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: AS4P251CA0011.EURP251.PROD.OUTLOOK.COM (2603:10a6:20b:5d2::7) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 76fb3b67-c004-4384-9c5d-08da81eb4161 X-MS-TrafficTypeDiagnostic: DB9PR08MB6427:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: e9F0sI6l4iUCqKkPBEOaA/zRV407ev8IIgsvjcB7e4nGNopKMcZ5s7t8osmYxKeZ3mhErBw0BQ+ccxwvBgZU0vngZkWZBGtAHvMn+MJxpUQfjlsixhptPxa7rKjnWSYZdHMSTikBB9vl7ABsQlRkeYd8+QcYLGARDoEMQpMLxqIPFPZ2yx25HKqoZTptIY0UvsgwsXDy9ILD/0BFRos/WVMrIgrrP6yNb6cbsWPfTF3sNPLHF2IX24v9ofPaO6GAED/QBt/ys9VDQPwh0ool7BJHkicJnMb3WlU74l8N9Bp8AhGFBZlFSp4s8koTGo+5H42gGwQfazw4Nq7DDpPb3m5w0daFZ5aL69jpyWVqEVfYk+2c1ohx21B4CazYblqaXYdfQRfq1K0xZqMKo8nHhI/gcMyN9uzJNyFlx5EZGQc7mv96MOHStBh4Dm3hTPU2O3Y3Tqwy1/oqoiVJljQoz8a1p96RhESzGOIO32ZW1aymlvA5dBXB+HTRBJVp77CkUfmR1OnCr0j53UAtbUC3B9ecQaWi7an8woi/Mt+L+0rWopRYG7enbKkdgunAo1SOc377qChtn3GsNvHjNdgb7xQCDOeIpymqsv2AJsyIqdesYczlUGn/qJxt2H7i0cBC07bLJIP2GF2PrhxokQvcZpN5EzzaQs0u1jaNi1FyWkdzkCNBeFzCZjsIsF4mgV23GkqyhI6vQxubSN0G+w4+LPsDLksLPyP+JwUY8Ws076LJG9TgZAm4TrAXoB4KcZKL27Q65aaSkUMfHk7+zn/n2w== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PAXPR08MB6969.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230016)(4636009)(376002)(366004)(396003)(136003)(346002)(39860400002)(83380400001)(478600001)(38100700002)(38350700002)(54906003)(6916009)(41300700001)(6486002)(316002)(8936002)(5660300002)(1076003)(2616005)(186003)(52116002)(2906002)(6506007)(6666004)(6512007)(26005)(8676002)(4326008)(66476007)(66556008)(86362001)(66946007)(36756003); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?RtjcTTiCN30Tb5KtshXT6hrdrVKP?= =?utf-8?q?ZXfI9muWDO0IWkH0x9qJJiRR7zgJI2gvjhFp0eTx6A/Jooh7eNaFRk4zwVLsmJaOQ?= =?utf-8?q?O93BBBu6WZl4UDmd8674PcVZpH4Ip6XXvxo71cqPNIEhRhn4G8RLrOaF4oKPj9UFd?= =?utf-8?q?hL9QDZ4liqMHHViDkkZvUwD9kwrYfUe6SZ9vTNGxavIRENZEI/7qsEAnq4rKG1qMc?= =?utf-8?q?xGoIKaYvtjo4n+d4lPUFU85RGm7aD6Be2I0y/fLUf3mdmhTP2n8pJm/ZHAXkKFfJl?= =?utf-8?q?DjjUGKocgY4d8AGvL+/gF7ZYacIi+7SxbvSXPCZngaPs1wUOOwppMI8wP4UTblfYG?= =?utf-8?q?4fhA7M4LBL5ANa581jO+6m2aCNMBtEWI7WBLDkNxc6GEQJ7S4RJJ48cccA21UK6vL?= =?utf-8?q?HILIRH7/35/hxA7Kx2BncztFIcAw0xNrMnGEFgl565eoq1nYtFkimDzV89ByGBNeN?= =?utf-8?q?XiOMBLdn41ljj0EQIgtC9GAzJmBwL67lBa+5g0p9aztWJtfM3oEaO2fRZH1uf0KX9?= =?utf-8?q?RL5wmdKYcLSxKTm926TedOxQ7Ov/9QzdTK33mmiKCx5ONTI1TU/YHk/QepE+2VIIZ?= =?utf-8?q?C7/N0IKCbU6mWlkywnEY0ja2kPS9kSFJ4jgBBjAg13zPhvBNWCVhKEC857/R2xL37?= =?utf-8?q?gz5DuFy69iJ8WKMwCoNU+1Y5t91tG6wNv40rvD7NvWenO8OrxWnUaCVOBGT/QKOWh?= =?utf-8?q?FWuSe8rrZ/BPSdTJtz7hsUBKPU+SfGmDeI+C5l4eYnEtEcTaNY0nRKv1Iy0fZ8l1T?= =?utf-8?q?29dAF+jgn6IeVbTXp0E61xNM3U0zjVMUD4ZXXoURnk2Ns4I04l33rDqreJpDB9wBA?= =?utf-8?q?KXry2PugaYWoR0kBgYvwUUb2ZHka8Qdm3M5FdvBM7IqMzmOLUKUwpNWG7jvReKWgq?= =?utf-8?q?WNa6tdBchx4UU+wuXxF0FNCF16b1DoIm50z8Ccwo8pMydaZhZG1u1MkX9UTtpaXt3?= =?utf-8?q?8n3Q4HRI46vY6vzGRZAKADJr/A7RipdW4EzBZmqm+OYi4mxriwUuEERCRXGj2E7Vn?= =?utf-8?q?3LWNsfjd0muH48uSg0JmG7Zi0cO+gtbqgEWenOj+tPHjlcBmO3zSDAfvkodVzBGTo?= =?utf-8?q?4DPV2nKzu/t7K/Jyz0dA1HjJgbTbYqFFUb21XJaFX6aIDimtiBHxhVwFZYVuLopqC?= =?utf-8?q?/YgyVVQl2Ug9SpltBUXpWsDKcv/7Rg+7s7xwtLuO3kMqUNfFIkV27ib+QtwGpsffT?= =?utf-8?q?yeO5+3B3W6wUi3vw8GMd4VaeLvBrOtgSjZzkoH1vektZXrpXEocbKiKfFJhpGkZQM?= =?utf-8?q?wbkgEqSIbJwrqLILwip6vj8qaKzvQXfNNRyhtAZYRw/jReV7ox3L/HkcChVRGk+v7?= =?utf-8?q?wXYaXiRdDHkhO/Dk9uxUs+JYdSmr8PmkZOfB329dOK0dYhiuGhmxpD6IwB2Deqm4W?= =?utf-8?q?zoaOHhjkwBCnDcIux3GhiHlTjyR2jR8g7Hs4GEDtVAmZhUjr/vgnYUt+DQtEJfUy3?= =?utf-8?q?a4j1QJJCX3GXr6stz4dnSG4jrpPo4sEg0scGMv3AgSqfgbgILIX8PKdxNfudQjgRk?= =?utf-8?q?GZmV5WUgOXBUz5H+BcIKfC56Y4hocuHJ2Q=3D=3D?= X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: 76fb3b67-c004-4384-9c5d-08da81eb4161 X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Aug 2022 14:01:04.1497 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: qitALPlw7rIsv4briPz0e2FrwuuRDdHIiF4J999tXnIWLpWCH7uCPSj4NX38HZbag2tZjyNCEzFYDSToKH8MFA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB9PR08MB6427 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.6 at phobos.denx.de X-Virus-Status: Clean From: Stefan Herbrechtsmeier Add zstd bintool to binman to support on-the-fly compression. Signed-off-by: Stefan Herbrechtsmeier --- Changes in v5: - Test that zstd compress of device-tree files failed - Add algorithm name to entry.py - Add tool name to ftest.py Changes in v2: - Add commit to add zstd bintool tools/binman/btool/zstd.py | 30 +++++++++++++++++++++ tools/binman/entry.py | 2 +- tools/binman/etype/blob_dtb.py | 4 +++ tools/binman/ftest.py | 12 +++++++-- tools/binman/test/238_compress_dtb_zstd.dts | 16 +++++++++++ 5 files changed, 61 insertions(+), 3 deletions(-) create mode 100644 tools/binman/btool/zstd.py create mode 100644 tools/binman/test/238_compress_dtb_zstd.dts diff --git a/tools/binman/btool/zstd.py b/tools/binman/btool/zstd.py new file mode 100644 index 0000000000..299bd37126 --- /dev/null +++ b/tools/binman/btool/zstd.py @@ -0,0 +1,30 @@ +# SPDX-License-Identifier: GPL-2.0+ +# Copyright (C) 2022 Weidmüller Interface GmbH & Co. KG +# Stefan Herbrechtsmeier +# +"""Bintool implementation for zstd + +zstd allows compression and decompression of files. + +Documentation is available via:: + + man zstd +""" + +from binman import bintool + +# pylint: disable=C0103 +class Bintoolzstd(bintool.BintoolPacker): + """Compression/decompression using the zstd algorithm + + This bintool supports running `zstd` to compress and decompress data, as + used by binman. + + It is also possible to fetch the tool, which uses `apt` to install it. + + Documentation is available via:: + + man zstd + """ + def __init__(self, name): + super().__init__(name) diff --git a/tools/binman/entry.py b/tools/binman/entry.py index 0d4990cf5d..8e4a5e301a 100644 --- a/tools/binman/entry.py +++ b/tools/binman/entry.py @@ -1157,7 +1157,7 @@ features to produce new behaviours. """ algo = self.compress if algo != 'none': - algos = ['bzip2', 'gzip', 'lz4', 'lzma', 'lzo', 'xz'] + algos = ['bzip2', 'gzip', 'lz4', 'lzma', 'lzo', 'xz', 'zstd'] if algo not in algos: raise ValueError("Unknown algorithm '%s'" % algo) names = {'lzma': 'lzma_alone', 'lzo': 'lzop'} diff --git a/tools/binman/etype/blob_dtb.py b/tools/binman/etype/blob_dtb.py index 5a6a454748..6a3fbc4775 100644 --- a/tools/binman/etype/blob_dtb.py +++ b/tools/binman/etype/blob_dtb.py @@ -47,6 +47,10 @@ class Entry_blob_dtb(Entry_blob): def ProcessContents(self): """Re-read the DTB contents so that we get any calculated properties""" _, indata = state.GetFdtContents(self.GetFdtEtype()) + + if self.compress == 'zstd' and self.prepend != 'length': + self.Raise('The zstd compression requires a length header') + data = self.CompressData(indata) return self.ProcessContentsUpdate(data) diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py index b337e2168c..134301365e 100644 --- a/tools/binman/ftest.py +++ b/tools/binman/ftest.py @@ -107,7 +107,7 @@ BASE_DTB_PROPS = ['offset', 'size', 'image-pos'] REPACK_DTB_PROPS = ['orig-offset', 'orig-size'] # Supported compression bintools -COMP_BINTOOLS = ['bzip2', 'gzip', 'lz4', 'lzma_alone', 'lzop', 'xz'] +COMP_BINTOOLS = ['bzip2', 'gzip', 'lz4', 'lzma_alone', 'lzop', 'xz', 'zstd'] class TestFunctional(unittest.TestCase): """Functional tests for binman @@ -5797,7 +5797,8 @@ fdt fdtmap Extract the devicetree blob from the fdtmap def testCompUtilPadding(self): """Test padding of compression algorithms""" - for bintool in self.comp_bintools.values(): + # Skip zstd because it doesn't support padding + for bintool in [v for k,v in self.comp_bintools.items() if k != 'zstd']: self._CheckBintool(bintool) data = bintool.compress(COMPRESS_DATA) self.assertNotEqual(COMPRESS_DATA, data) @@ -5805,6 +5806,13 @@ fdt fdtmap Extract the devicetree blob from the fdtmap orig = bintool.decompress(data) self.assertEquals(COMPRESS_DATA, orig) + def testCompressDtbZstd(self): + """Test that zstd compress of device-tree files failed""" + with self.assertRaises(ValueError) as e: + self._DoTestFile('238_compress_dtb_zstd.dts') + self.assertIn("Node '/binman/u-boot-dtb': The zstd compression " + "requires a length header", str(e.exception)) + if __name__ == "__main__": unittest.main() diff --git a/tools/binman/test/238_compress_dtb_zstd.dts b/tools/binman/test/238_compress_dtb_zstd.dts new file mode 100644 index 0000000000..90cf85d1e2 --- /dev/null +++ b/tools/binman/test/238_compress_dtb_zstd.dts @@ -0,0 +1,16 @@ +// SPDX-License-Identifier: GPL-2.0+ + +/dts-v1/; + +/ { + #address-cells = <1>; + #size-cells = <1>; + + binman { + u-boot { + }; + u-boot-dtb { + compress = "zstd"; + }; + }; +};