From patchwork Mon Aug 8 10:51:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 1664529 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=weidmueller.onmicrosoft.com header.i=@weidmueller.onmicrosoft.com header.a=rsa-sha256 header.s=selector1-weidmueller-onmicrosoft-com header.b=cYqmYPxR; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=85.214.62.61; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4M1Y3D1G1Mz9s2R for ; Mon, 8 Aug 2022 20:53:44 +1000 (AEST) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id C191C84898; Mon, 8 Aug 2022 12:52:20 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=fail (p=none dis=none) header.from=weidmueller.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=weidmueller.onmicrosoft.com header.i=@weidmueller.onmicrosoft.com header.b="cYqmYPxR"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id DB61F84193; Mon, 8 Aug 2022 12:52:01 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.2 Received: from EUR05-VI1-obe.outbound.protection.outlook.com (mail-vi1eur05on2076.outbound.protection.outlook.com [40.107.21.76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id C7C78844C5 for ; Mon, 8 Aug 2022 12:51:55 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=weidmueller.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=stefan.herbrechtsmeier-oss@weidmueller.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=H53j1oIXmPiq6u372yC+lXGuoRVQmgNlAzwkpjsSpclUmZBlIraNp6uUlfQ3PPiKQYd+NgRv3gF/KzKlRCiA6vEY1SXwOXXrbnWcQZWSjnJseYC/NMtyCeOsYUL6JD0gvSpL9wfDRhNevJUgrCYmkdMLwX69/Wa9v8RlQF0mPAB3zjrqqwu13BLB0hyUL+xbrTOIBeGUp9STCK2wIaRi7Pesb6NCKjcGqKiaDK3uc09oaLzfBxTpn/LR2VgyKwZequvcYb91UuCXqGRGy4cHeM7RUJwJaIYz+zSxfaJ8kSL6z8SGJhbPrpd85I1eWMbJGtVgDoiftlFOF+M5r27cqg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zM5xy+ftsDIvqqmoXkPO00t6o0bU/3tucDqHBAwElEg=; b=hO8aT9rTpw4o7WqRBfr7J1vnMR4HxSfcPJFkiZbfOtl7nDRbzMkKGQrhvNOQIFDPHZ6GGLFozpOW2Loxog9bgnSNkTcRnUclubagnXlLjR7gzkv1nE3hqoIcU8IjUxM2e2hMcpR+G97CzdSGi3wDhEA9VXWh0AiHhZjKjP9AYR5rhG0sHr3+KFsfLEJVr5pO0wSHbwKvh/M72fUrKEagMDoCOQ8vT8PZlikhpeGDmCT7iJq4ySBCvuHrDQaRRf6u/aah3IKQ8tBVXbRW2zi8zIagyoh6xIb+mi2Phc7bueOGSFL2RZNBCxlTtO+AaqKRbngnp4f5CokPRS6Q9S6zMg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.onmicrosoft.com; s=selector1-weidmueller-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zM5xy+ftsDIvqqmoXkPO00t6o0bU/3tucDqHBAwElEg=; b=cYqmYPxREkcmbguBB97DTA/fjp/BmpP8gvAvvLeKxFJ4fWYCBQ2PW2sBGAUs7VUwN2wK+hghExeN9YiOiYbLGtaDzTu6b1tL1EVmQmHfuzrHO8THAaCNaz1lfSpd48upjIhVIp2NUJhuLt4UBptTlcvFMkoPz1e3RaCtwPUZF1Y= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by HE1PR0801MB1897.eurprd08.prod.outlook.com (2603:10a6:3:4a::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5504.16; Mon, 8 Aug 2022 10:51:53 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::f95b:2e64:9ee6:4d44]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::f95b:2e64:9ee6:4d44%3]) with mapi id 15.20.5504.020; Mon, 8 Aug 2022 10:51:53 +0000 From: Stefan Herbrechtsmeier To: u-boot@lists.denx.de Cc: Stefan Herbrechtsmeier , Alper Nebi Yasak , Simon Glass Subject: [PATCH v2 10/10] binman: Add zstd bintool Date: Mon, 8 Aug 2022 12:51:25 +0200 Message-Id: <20220808105125.21356-10-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220808105125.21356-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20220808105125.21356-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0058.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:49::19) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ef5b1bf7-76d2-4106-dfc8-08da792c0148 X-MS-TrafficTypeDiagnostic: HE1PR0801MB1897:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: xCGITSVEtNe2A8Iz/eFUFmWDq8ppMc4OZvtftkcqwpOYiGT9ih01W5yw+B6vsL11rPQwipgjgh56sJ435PYvgQ9WKY277EwNuyPtuJvMyU5K2P00vmSOjqR/D2NrrU7/ZRAqprRqs/zgv61sdXPODTSliFk3RMcrvEBXVqK0ke9vAm7PS7navenSiHHnW4fx//yK1GAH97btML66hcvui1Mz0DgO3OZm+hsNlceKJJIcXvyqVax7PqWiQmjaADuXUFssk+z9A/o4KL4fS+XdWaQg3YzLmoCeJh+FqzKTrzZknHmB/Vn9C4S7kO+GdrXvsnf5oR1YXr9qbaNUYFQRSQdsSSuwoL3NSIEKXwouZdg6hKSyaWBnKGn87UR7HqqikWuH7mnyvR4nIm3N5Fa4ASnYlc2NwuugtBbFiZmldF2F6sPPx4cugyyNf790vJgzgaHnKWSMuu4nA0MaU4l9rbfBFeMhihsFFwADakvI73t/nixNZSSqLwauPsYJ8Ff0qvbcMNKfmDRM2ZsGymYsGJp1kMCmIDmybbpRpOk6IgFSk+igH7OTI3LJu8R9Vc/YrjAFBas+VHGyXGwG8wyTGsZVZ8LbHynCnZjLVLHROL0UgM9NVarpN8I5SyQOKN/nNIMBtjb4z2XLhEijMW6P/aPsBz299EHtlTwVKaj9CpjzfwAVYLrA6hvC5d2q8tB0HF/ct/xSVGrD3YqkgM4OYvodRjTny1UTxZ8Pxivm4uvUiCnB37Q2ZIaSzK/iwukzGW2Pd6IwKTVYPVSKpWIFYu6FeqJaiUP8zPsimDaeJIY= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PAXPR08MB6969.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230016)(4636009)(346002)(376002)(136003)(366004)(39860400002)(396003)(26005)(6512007)(1076003)(36756003)(2616005)(186003)(4326008)(6916009)(83380400001)(86362001)(316002)(41300700001)(6506007)(52116002)(2906002)(6666004)(478600001)(66946007)(6486002)(66556008)(38100700002)(66476007)(38350700002)(5660300002)(54906003)(8676002)(8936002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?Bc+CPc4YvFft2Dkud2MkRqeh5VpC?= =?utf-8?q?zI3Oa1XEQ6pPjmHmNO1HVEeRCBxV4sv7SWlMLZ9e03DfQIGzZj33lmUlIMeXshmJf?= =?utf-8?q?MMT9lscuyBTVaws7+azqHrlF4/UgoHeGU/17vLh0mfSeziCod2t8Dp/0B5K3vDtRn?= =?utf-8?q?gC8MQeCoSeTYdokkY+jY8McdkS54pvX6GpgjHTEzE6lwIz/QT4vuJbpn4QNEMYP47?= =?utf-8?q?j5idgRiy9SpvjO9tdXFoEbC7h3UErUY4Ch04pB2VPypTmMsDx/oCZESZmQsBRa2ri?= =?utf-8?q?MaLKN8l1M4HxRi9IJi1Fge7o7UsJwNZpFI0+rn2kzG0xxxl7yiyf2Q5a4pypS58LH?= =?utf-8?q?ItPCEuFwq53Boy9M86HiO3UE1runQyiROJ/4WXmm6xVEbISsXpan3Pe2mUPvKbW5k?= =?utf-8?q?BMy9pM5O0mC2judfyA2qXHg69sHBQsDNID+gKdq9Osztw/Kkin3XD9mc6kD0A6fMZ?= =?utf-8?q?XsGMVMzqpmLHXtXn6EcSknql1YUbe7Vrpmt1QCvPntvTBtih8LtUFNAy15fZ2Iy40?= =?utf-8?q?3g70Ng3GY724o5OaOPLwI0N+Fk7gh7PF3qJG/XEm+gyCyzUz+49TBTEvX+Yi72gj2?= =?utf-8?q?gjKkjIU7lX3r9N91gc+33o4LxBysUbwdSMKpBYEB252itKeFKK28Io7NDysnPx6TX?= =?utf-8?q?Ws5+YPw+ub0s9ZRzXSxP4xi1fQOWxAtCkjFsJRDUTIrKMxKW9O2UEQbV6Ikk+l6AW?= =?utf-8?q?WNBOj+/OM5doD+Z+FT7wTB1Fakq8enJC7uEcFXqrOECR4NA/XPZDPGZAf9vjnH0h/?= =?utf-8?q?YZUvAJ+0heCPAgR9URBXJFCGqudaOCmJiIWl0IsWOz1JEOnj0tAI0/TgaqZjLeLTu?= =?utf-8?q?+BCbc/ibiRFEm/q4sDNTinIVHUBEcqeVRW4D/5/6vAFC6loTKlQ9u8FeJg22INlPT?= =?utf-8?q?8NvAePbfQoDxI5Ywp3yLUTcEs9Xf/fZOX7VK+qXeU+XVpcQ7lkADJYwNw3ogHyJni?= =?utf-8?q?SsJSwA98jbHPg36BP9qvu2AMmhw1zBtTrEqoo90wOuOLIC6+seFy5kTSOjwOOlwGd?= =?utf-8?q?7h6q+BRPGnZYK5lGrOLr2TdpoCcOxOntHU2CzlpzsSUMnJNQfG5F3ekK4DY+EJWET?= =?utf-8?q?PrXV8H0PAIyWt7ho6eQGzLkSKUs+3xiZL3m8rw0KwRNBhJS14miPWVZ3IvLBYd6kw?= =?utf-8?q?nmOq1DNyUu0y8uTy3vZ0D+JfKtDeTaPDt2Sm7FGaUh9BLp69vqH0+XLe59Rd0xZC/?= =?utf-8?q?2T7g4sXsaExB4tpKj1p4Mi84Q5oCM7hn/Ti+SoPajoxNSUd04SKVaxB4Pmy0mvkhi?= =?utf-8?q?Ry2SR/pscbjUx/QwhBiByxDoiIakbBF2CTgCTxOz7F3n/Gw818MEITOdktqXybFep?= =?utf-8?q?ZxzXygTcZMEhdB5VOpcGOGIIt/S5JObFg5qYTzKzdoNBHRRalnVSOnxck6akjTMem?= =?utf-8?q?2vlB7nDQiJ5vvq2k5sTs/dnaiU+XK5q8b5t9Q9GGbLwK4ujfUswhfIh25TvAZpcwS?= =?utf-8?q?xuBQ8RutMKQcvjBqtppgabMaQqfXB+X4jyrRkhgAOiq4zzjgVoa9NxJXPOrFNh2/+?= =?utf-8?q?oTJNg3fTLNUqaLe7MCwuBKwi771GiLrbsA=3D=3D?= X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: ef5b1bf7-76d2-4106-dfc8-08da792c0148 X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 08 Aug 2022 10:51:53.4358 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: xBu9K78sCm6m4WMdt9MYOhGWtF+5o9kBRzIO0iTlBOqDquMout0diUfCm+M/UPUsXfG0P4RPtEM/min/zT2nWw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR0801MB1897 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.6 at phobos.denx.de X-Virus-Status: Clean From: Stefan Herbrechtsmeier Add zstd bintool to binman to support on-the-fly compression. Signed-off-by: Stefan Herbrechtsmeier Reviewed-by: Simon Glass --- Changes in v2: - Added tools/binman/btool/zstd.py | 30 ++++++++++++++++++++++++++++++ tools/binman/comp_util.py | 18 +++++++++--------- tools/binman/etype/blob_dtb.py | 4 ++++ tools/binman/ftest.py | 3 ++- 4 files changed, 45 insertions(+), 10 deletions(-) create mode 100644 tools/binman/btool/zstd.py diff --git a/tools/binman/btool/zstd.py b/tools/binman/btool/zstd.py new file mode 100644 index 0000000000..299bd37126 --- /dev/null +++ b/tools/binman/btool/zstd.py @@ -0,0 +1,30 @@ +# SPDX-License-Identifier: GPL-2.0+ +# Copyright (C) 2022 Weidmüller Interface GmbH & Co. KG +# Stefan Herbrechtsmeier +# +"""Bintool implementation for zstd + +zstd allows compression and decompression of files. + +Documentation is available via:: + + man zstd +""" + +from binman import bintool + +# pylint: disable=C0103 +class Bintoolzstd(bintool.BintoolPacker): + """Compression/decompression using the zstd algorithm + + This bintool supports running `zstd` to compress and decompress data, as + used by binman. + + It is also possible to fetch the tool, which uses `apt` to install it. + + Documentation is available via:: + + man zstd + """ + def __init__(self, name): + super().__init__(name) diff --git a/tools/binman/comp_util.py b/tools/binman/comp_util.py index 35835450e2..0f15fae600 100644 --- a/tools/binman/comp_util.py +++ b/tools/binman/comp_util.py @@ -11,7 +11,7 @@ from binman import bintool from patman import tools # Supported compressions -COMPRESSIONS = ['bzip2', 'gzip', 'lz4', 'lzma', 'lzo', 'xz'] +COMPRESSIONS = ['bzip2', 'gzip', 'lz4', 'lzma', 'lzo', 'xz', 'zstd'] bintools = {} @@ -34,14 +34,14 @@ def compress(indata, algo): Note that for lzma this uses an old version of the algorithm, not that provided by xz. - This requires 'bzip2', 'gzip', 'lz4', 'lzma_alone' 'lzop' and 'xz' tools. - It also requires an output directory to be previously set up, by calling - PrepareOutputDir(). + This requires 'bzip2', 'gzip', 'lz4', 'lzma_alone' 'lzop', 'xz' and 'zstd' + tools. It also requires an output directory to be previously set up, by + calling PrepareOutputDir(). Args: indata (bytes): Input data to compress algo (str): Algorithm to use ('none', 'bzip2', 'gzip', 'lz4', 'lzma', - 'lzo' or 'xz') + 'lzo', 'xz' or 'zstd') Returns: bytes: Compressed data @@ -62,14 +62,14 @@ def decompress(indata, algo): Note that for lzma this uses an old version of the algorithm, not that provided by xz. - This requires 'bzip2', 'gzip', 'lz4', 'lzma_alone', 'lzop' and 'xz' tools. - It also requires an output directory to be previously set up, by calling - PrepareOutputDir(). + This requires 'bzip2', 'gzip', 'lz4', 'lzma_alone', 'lzop', 'xz' and 'zstd' + tools. It also requires an output directory to be previously set up, by + calling PrepareOutputDir(). Args: indata (bytes): Input data to decompress algo (str): Algorithm to use ('none', 'bzip2', 'gzip', 'lz4', 'lzma', - 'lzo' or 'xz') + 'lzo', 'xz' or 'zstd') Returns: (bytes) Compressed data diff --git a/tools/binman/etype/blob_dtb.py b/tools/binman/etype/blob_dtb.py index 652b8abd8f..8d0b88d5b0 100644 --- a/tools/binman/etype/blob_dtb.py +++ b/tools/binman/etype/blob_dtb.py @@ -45,6 +45,10 @@ class Entry_blob_dtb(Entry_blob): def ProcessContents(self): """Re-read the DTB contents so that we get any calculated properties""" _, indata = state.GetFdtContents(self.GetFdtEtype()) + + if self.compress == 'zstd' and self.prepend != 'length': + self.Raise('The zstd compression requires a length header') + data = self.CompressData(indata) if self.prepend == 'length': hdr = struct.pack('