From patchwork Tue Aug 16 06:41:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 1666696 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=weidmueller.onmicrosoft.com header.i=@weidmueller.onmicrosoft.com header.a=rsa-sha256 header.s=selector1-weidmueller-onmicrosoft-com header.b=QPK2iQZz; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4M6M8h64t4z1yfd for ; Tue, 16 Aug 2022 16:45:08 +1000 (AEST) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 961D384A0F; Tue, 16 Aug 2022 08:43:14 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=fail (p=none dis=none) header.from=weidmueller.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=weidmueller.onmicrosoft.com header.i=@weidmueller.onmicrosoft.com header.b="QPK2iQZz"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 07D4D848C4; Tue, 16 Aug 2022 08:42:40 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.2 Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-eopbgr60046.outbound.protection.outlook.com [40.107.6.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 4896F8437C for ; Tue, 16 Aug 2022 08:42:30 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=weidmueller.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=stefan.herbrechtsmeier-oss@weidmueller.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=iJ8cFFGxGO30dpPdqUJeOsrDL5sCCp++YM6NKEnYIvY/ODiPamTbA8uV9ybwGh2WCP7RMaaCvUd53wFXDwEC5ED8XL9cobkskIg7lTQ+vaQ+/HPs+jeZhNqb3dxJk2qPmfp8U7bEinPsBtdNAK0/lQF/uhzBv/oRfLIaD6lGvpUzmANoACgUTu1dJQOSMHS42KDD0YVKdF2QwSNJdrkSccxsoI2c9CQikMUVSJvyWfN+FwTAhaIfcg1C2LPmFbqUmSgr5zKMiIFel1VQcWNzjTO23jOdoIrH3s4CFS8zcMhwywHjFExYwdtFnBHuADCPHf7PAL1Y65G/1zA72X4xfA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=BXK15Q2QksGyAbySRrBo8NEtWAYTTN3HjwQNCS5Jvvo=; b=W33AG14Y6fcf0hjMJwH94iIe9rK/yuwsgjFxrglhDe++5lHyYouQ3+aSfdQBm7+/PLmJuz0aiH+roPO0y06uGMmYqHDeQLlk7+p6xclqO6VEqS+38+CWQ881apv2tmD7pfPj1SNS11MPNsEdTDpal1jyM63d3zKbW5bjtAnjgewSjUIYkhlK1WXTluY4AqtCUlWJYtKHiqW/AT+VZmCvt90KFQr7K4duCrfuADpP3hS89YyxgcT07iNjGVVGE/agHksKuHc2NZPheD3Xoa6IOep9QL7zdAfKduieeE8bLAyLqwty/s5aewQ3qFHi0Ct1ZPAqMAreL8jfXdXnpbxO6A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.onmicrosoft.com; s=selector1-weidmueller-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BXK15Q2QksGyAbySRrBo8NEtWAYTTN3HjwQNCS5Jvvo=; b=QPK2iQZz+4RvkM2JveA9tmWzcYxc7DURVjz3WOA3xO0ZpJZqerxf5vVkIiPmTvvKvbDhGUnFAK1SCEx2NVGAuwa0uqGif7P4LtzGct/E+SeX8eO2pIUtY9iQOeUgEu1RZG42oXZoJfNNX1dMyterhhcTp7Xkb+4qSs7gNh/wkew= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by DU0PR08MB8662.eurprd08.prod.outlook.com (2603:10a6:10:402::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5525.11; Tue, 16 Aug 2022 06:42:29 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::f95b:2e64:9ee6:4d44]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::f95b:2e64:9ee6:4d44%3]) with mapi id 15.20.5525.011; Tue, 16 Aug 2022 06:42:29 +0000 From: Stefan Herbrechtsmeier To: u-boot@lists.denx.de Cc: Stefan Herbrechtsmeier , Alper Nebi Yasak , Simon Glass Subject: [PATCH v3 12/13] binman: Add zstd bintool Date: Tue, 16 Aug 2022 08:41:50 +0200 Message-Id: <20220816064151.9848-12-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220816064151.9848-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20220816064151.9848-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: AS9PR06CA0744.eurprd06.prod.outlook.com (2603:10a6:20b:487::19) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 419ddcda-da69-4c66-1dd9-08da7f527d43 X-MS-TrafficTypeDiagnostic: DU0PR08MB8662:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: HJ90hnRdoVmAGN8ovQTfZ7ovHJDdFLYmXy+tvQ7Yt/CYbkcyBuhLnFHSGf4ZeR/PkpmpQZBjNi7SxMHycz26m07ocfQomNlJx/CGaJ8RrKVkr9Lr5MU9SukOMuh6blmCiZh04suz/DWsc2PWRM1q7NSleCqj7261qSivcCI8caeeE7P3H3LQWTjbbrXC1T5lMJdyBFp1e7YsVWggZpPjPK3GhPCmjaBdYCMvzioTzH4e8DTunRg8OwdNza7R01TSUR/6l7KEOj77Tp0xdQJL5r31bdhl0zrqOmIyH7ss5n3nOZr1KNsFeWDYA/zPZIaBjqzqMIP3r3Ajsfy/wLJlf8JyfMeBd7MX+xiXkqgpF3MHK72H1RFcLvP/9wIPWGE1QG+cOTu3qnUC0JXldF7iHXTrtS6IQGlXeCzkkMm2mD83nhT7VKNuzQAGE49+LrlQuztUguazedyZvOf1qARVMkjF92UpENXjQyo1/jRaIC1bQrhYpAy6VoTS0LtK5XpgnvbnH8WMsD9fI19lh1934qkVmZCXepAEoFiVebkTSyAp1uZfFIzVTU5FFHBT2Nu2+aydlnSWILkMk+vy2U+lnW2oljNYgU3IhUVTePbP6WbpRae4iUFISXh4zDUUs/tDL5/8HXghIebSgjFm3NtMUprVvjVNnSH1d351hUmDZ1yoEJhQ9DRMMazwZ2H9VAhZka0AlBe95QjctUx/akQ9t39q0UpUGbMYBfLtxNd3X0bTZfW3GF4+n0dQ7pfHmSFsRcQH4FYBPtPDxjDOOuwewGSSuoPtmbVQO6NRADOXm/IdIWd575jwEMC41mHeRwKA X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PAXPR08MB6969.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230016)(4636009)(346002)(376002)(39860400002)(136003)(396003)(366004)(316002)(6666004)(41300700001)(52116002)(6506007)(6916009)(54906003)(86362001)(478600001)(6486002)(38100700002)(38350700002)(83380400001)(2616005)(186003)(1076003)(26005)(6512007)(66476007)(66556008)(4326008)(8676002)(66946007)(36756003)(2906002)(8936002)(5660300002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?FRUB8dB+1Z8fZ5kcjOxjAf1OBKcG?= =?utf-8?q?XjXs+aSPkIOOKm2UbcbiJlWVZdfel3Xf9zsAcAa7tiM50MZamng7I+kyfcl6uRqte?= =?utf-8?q?b4idk3KUyrnPLNgOV6ln+EnbBk0FXAz/F+W1qQEeAo/3Ch3VfSR0U2/gO9nMkiAZS?= =?utf-8?q?pWi8YFJV/2yoM2p0TAr5AWha7h7ge4s6M4/e1Hrz2R8qVba2V2rAiFlZFIr0fVOE9?= =?utf-8?q?ECrdNzZNvbQdxgNTnIkRC1C4tQDNcObEFOl20nCSK7KiBnjhSorL6Vp7UXBwBV/kl?= =?utf-8?q?dKTY3Xbo5vrlzi78kEGRnejfAAQybZx6CBiaZIYJ9lJVN2fmhW3BFG+QddxG6sRrN?= =?utf-8?q?/IOYzFC5qPPKkNH0mUUiSTQjs1figbUNnpI6dAJ2uQY3tMChrmMX200oJPNVVIph8?= =?utf-8?q?Tn+xFnauTuPIU90+CjPezXxvQgz9GGAxKGPRU/DOLBilTDHlLW6blc9IHXR5O/i4L?= =?utf-8?q?Isdl4CyEYOhHTc3+XQNFiEuOqC2Kz7kkopWnNOWbUevjUvpmM3KE9N63PDbfj+eBO?= =?utf-8?q?pJj3wnhauQVjnnMahWCm5SNXPrCG98xzPM4foFl4pvRFH8opeJCOklNcWyt9F+aPM?= =?utf-8?q?GxcYD1jr4KBWBem1wQvQJ0CY/ZHltAfrfW++S0aUF3g6fE4zNt8fyiiX5IS/KKxGR?= =?utf-8?q?KMbW6yOsGh3zOruq8dIyy+Y4gBaBOE7CLn4KQaWfxj469LNhXM8U4w61oCH8TBJeK?= =?utf-8?q?RixQ49qj0YRBQ+MEo9gw2X8aX8mdoGcMYfdLGDYkShk8pA5TPApCeM89Oxkw6NeNY?= =?utf-8?q?r7llgP+5WHsfbxPc2xlfGZR1EbDL4HQlfRQqopO7CWT/3g2WRpPpt+1lnT2FKHQaB?= =?utf-8?q?n0fJcLRfRZEV5qV3Ivq/QcoHJs+zWMqPgHXUF+w9IUn59ODyByKDGJYCv1p/8CIvX?= =?utf-8?q?iAUyS7z57fXFMojf9YgUsEoKUkeEMzcOW1PhgrE37gxFGHxtPCy6V37QBpZfDOAnu?= =?utf-8?q?95dOzI8Zct7ofJin2q9lRdD8MqH5O6VhmLcwnZTjNz8unEg8UspLNo5icgGhdGL0Z?= =?utf-8?q?IKzss1zS1m3fvY3GLRSJAwKsMRz6BNE80rPuWGkfoVWjDy3v+lP3D0MkxvsVC/eAV?= =?utf-8?q?TuIRpxVw6ngpk5rCWql0DhxWzNIeS/0LmGKVUUM3Lm+q5fqxEqpSkJGeefnrpnXho?= =?utf-8?q?hhOd2oXoj+6rUh/phcHSQXhA/lvR4LQRP30RdeeFs5yoXAROfZjPNwsjeyaWEj4Aj?= =?utf-8?q?PwZYV6r9z0zangFaSjcJsLRlyX3hUtR3sEJTA1/wig8rQQKhTTX+wwNsUKv4wDfEN?= =?utf-8?q?2/I4BhZIWMYZ8OIXl5OMaEZrYaAJWqRu+C8oyt5dL2z3IsmsDXP/43JeMn7ohnwid?= =?utf-8?q?aWblvw1GFSruz7D/5qbJYoP06bN82dYzTYXZdWOYVE4JTNiZ4EEo/QSndWdeo0a2m?= =?utf-8?q?Br/fEMtwwplcZVtjQfLlec0hOx5d0fmVXi0rHFg/MB2DP0ydpNI3n9jrBq+J09KGm?= =?utf-8?q?SRClGTumlwBj5MefOr9TzyTRONClRLHNCO7UaQSgK3Pgc70/5HqE6S7Nn6XreJn1j?= =?utf-8?q?iUqjtkerxPBDKpsK3Qua/AVWawyvFRf+Kg=3D=3D?= X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: 419ddcda-da69-4c66-1dd9-08da7f527d43 X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Aug 2022 06:42:29.3661 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: QvH5COZZ2YSC2PS0HBxaELDjYlpUBAm47+6uCXsnQmKyaRTLt4IJXoab+O80kgt8G9i4NEc7XXGQIuIWx92dag== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DU0PR08MB8662 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.6 at phobos.denx.de X-Virus-Status: Clean From: Stefan Herbrechtsmeier Add zstd bintool to binman to support on-the-fly compression. Signed-off-by: Stefan Herbrechtsmeier --- Changes in v3: - Rebase Changes in v2: - Added tools/binman/btool/zstd.py | 30 ++++++++++++++++++++++++++++++ tools/binman/comp_util.py | 4 +++- tools/binman/etype/blob_dtb.py | 4 ++++ tools/binman/ftest.py | 3 ++- 4 files changed, 39 insertions(+), 2 deletions(-) create mode 100644 tools/binman/btool/zstd.py diff --git a/tools/binman/btool/zstd.py b/tools/binman/btool/zstd.py new file mode 100644 index 0000000000..299bd37126 --- /dev/null +++ b/tools/binman/btool/zstd.py @@ -0,0 +1,30 @@ +# SPDX-License-Identifier: GPL-2.0+ +# Copyright (C) 2022 Weidmüller Interface GmbH & Co. KG +# Stefan Herbrechtsmeier +# +"""Bintool implementation for zstd + +zstd allows compression and decompression of files. + +Documentation is available via:: + + man zstd +""" + +from binman import bintool + +# pylint: disable=C0103 +class Bintoolzstd(bintool.BintoolPacker): + """Compression/decompression using the zstd algorithm + + This bintool supports running `zstd` to compress and decompress data, as + used by binman. + + It is also possible to fetch the tool, which uses `apt` to install it. + + Documentation is available via:: + + man zstd + """ + def __init__(self, name): + super().__init__(name) diff --git a/tools/binman/comp_util.py b/tools/binman/comp_util.py index 8c0fe5078f..6b4ab646e0 100644 --- a/tools/binman/comp_util.py +++ b/tools/binman/comp_util.py @@ -13,6 +13,7 @@ This supports the following compression algorithm: lzma lzo xz + zstd Note that for lzma this uses an old version of the algorithm, not that provided by xz. @@ -24,6 +25,7 @@ This requires the following tools: lzma_alone lzop xz + zstd It also requires an output directory to be previously set up, by calling PrepareOutputDir(). @@ -35,7 +37,7 @@ from binman import bintool from patman import tools # Supported compression algorithms -ALGORITHMS = ['bzip2', 'gzip', 'lz4', 'lzma', 'lzo', 'xz'] +ALGORITHMS = ['bzip2', 'gzip', 'lz4', 'lzma', 'lzo', 'xz', 'zstd'] bintools = {} diff --git a/tools/binman/etype/blob_dtb.py b/tools/binman/etype/blob_dtb.py index 4fd2ecda83..fab79e43cc 100644 --- a/tools/binman/etype/blob_dtb.py +++ b/tools/binman/etype/blob_dtb.py @@ -48,6 +48,10 @@ class Entry_blob_dtb(Entry_blob): def ProcessContents(self): """Re-read the DTB contents so that we get any calculated properties""" _, indata = state.GetFdtContents(self.GetFdtEtype()) + + if self.compress == 'zstd' and self.prepend != 'length': + self.Raise('The zstd compression requires a length header') + data = self.CompressData(indata) if self.prepend == 'length': hdr = struct.pack('