From patchwork Tue Aug 16 08:42:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 1666740 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=weidmueller.onmicrosoft.com header.i=@weidmueller.onmicrosoft.com header.a=rsa-sha256 header.s=selector1-weidmueller-onmicrosoft-com header.b=h92Qptw6; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4M6Pps0F6Tz1yft for ; Tue, 16 Aug 2022 18:44:53 +1000 (AEST) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id D157A84A22; Tue, 16 Aug 2022 10:43:17 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=fail (p=none dis=none) header.from=weidmueller.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=weidmueller.onmicrosoft.com header.i=@weidmueller.onmicrosoft.com header.b="h92Qptw6"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 94228848C4; Tue, 16 Aug 2022 10:42:47 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.2 Received: from EUR02-HE1-obe.outbound.protection.outlook.com (mail-eopbgr10053.outbound.protection.outlook.com [40.107.1.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 6487284979 for ; Tue, 16 Aug 2022 10:42:40 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=weidmueller.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=stefan.herbrechtsmeier-oss@weidmueller.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fmrPdJcx/i4gFQ9r0uFoUaFZrsaDbCKYz5me57avLhKQRiYx2jm0LJB+rSAcCsRuXprgLrcXpAcfRYjsFKBeUrienGnJTBhaqJT5TLLKRVpfS7AYqpi/TMUHoJj0zwUx2l3pSY20gCaVxNBfOttebbXywVPsdh7uD8aUUOJ2ULCi84ZTAE1YkDKl+dmz0vH4YXJlF2F5ke768cLJWEQzTm2Gik6QZGDO6nt9g/HiSe6cF9dausrDeAF27YpvY/WIgJIWITWbCkQR0YqolxOZvldWvY9+tn8gZoIVWzi785G0D+7fqij4OMFgbz/k0UE5w2y22YeH2dBJoaSrdcWCgw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=J5WZ7aLKdPIDbhQvPLnO4URTYhxsSvCuDTwn3IWJUe0=; b=bTWrPzkKGIipA4ioOaW7Xvp1NNdlgSFWCgWDBIcdYmbnWfpq9tALFWxuL15Ex1e2aNGNeVWH9VHNKsY3RD1GtmvBPRMqGoZHIwHw/NRqR9nXcCBQ4AUab+9IfW7w/4z9GTzQXAB4jdKrTq/qgZ7yBvYGatksi86p8fhW6ouPPjI8kp6P2hoF93FCwBKVuBSm46A6L9xYThTz5WYcAyiyGlE6QUnYFkV9W1h1el28w6/ghgNZ5ILbZGxPzjOMveOylVq/m+KVkyAR7vrxFl1NudlR7VTJqJAfsQX3hLxulmth+HUmLSUyGdcKxU9elhah/dxtb1LnpHr13lYIwEjbRg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.onmicrosoft.com; s=selector1-weidmueller-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=J5WZ7aLKdPIDbhQvPLnO4URTYhxsSvCuDTwn3IWJUe0=; b=h92Qptw60t5WYtpbmBcrpT5fy6GBcancM1ca2NOQsrnIgZcBKJUm1Hw3F7vEOs/nTbzzFeXCJA95NlJsHpHtqTPgXQn69wuUHeqECoCIvRNvn4gmqx7hC/DVi/zJMSrdPB+NNeSkhGaOIhlBnedJJwl28kSLap2d966zQHCgNlc= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by AM6PR08MB3096.eurprd08.prod.outlook.com (2603:10a6:209:43::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5504.17; Tue, 16 Aug 2022 08:42:39 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::f95b:2e64:9ee6:4d44]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::f95b:2e64:9ee6:4d44%3]) with mapi id 15.20.5525.011; Tue, 16 Aug 2022 08:42:39 +0000 From: Stefan Herbrechtsmeier To: u-boot@lists.denx.de Cc: Stefan Herbrechtsmeier , Alper Nebi Yasak , Simon Glass Subject: [PATCH v4 12/13] binman: Add zstd bintool Date: Tue, 16 Aug 2022 10:42:09 +0200 Message-Id: <20220816084210.14972-12-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220816084210.14972-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20220816084210.14972-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0139.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:96::14) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: d483a843-08c3-49e6-2efe-08da7f634671 X-MS-TrafficTypeDiagnostic: AM6PR08MB3096:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: eIEX4bd4KAnqo7ekXd/MrLylL6jxVn9d0gfUlJzQ72mbft8wJgeamo7ETeSRqEkrrBJmo8URiBes5h23I6Mfvr+VGbsVwhN/czsk/Q7KV6ubuJA1IMSCUYCbi3DYMg02JLoMtfou+O5gSsrTmkDKW5G87aAMC8XJEHTUWpXsyonHKPcLjGLQNTfOUjjvrA/EvfsWxAkFdIM5scWa8R5taryutk5azBBAk5fB+eKc9RJbMz+7JVMGGQRRDeQsq1WulU0e6KzNVeOz6FUusAYcO+pJIdRG/lt50ZHUvtIkplFESIWZ7TH+RNS39qLlUlECJkVNKqe6fjuIjSfnWeaZO4gZpx+JNvriqzbKWK0HYSH+YmBS9kdrOeYOhey91kizwCxt1hmk3fcwZHsHOY4hPAgOlbI3bxP8Cznb1J+dmjZE96XzQ9/xGljjmmKLc2XElRyIawh6cVAITSXiaBkNpOYm2hXpt2Am+HWKySKhV3NVFWe/42dMGA+N+XU+I53NLKF/hVhqt2he3PXPH34Pt3vuwtn1vyfRA1l/enMNvw1DEn4I06nbs9v49yswCJb8Ja+VYxBGyfi3qiOyjHx7Gdw264iLG1mZSkRm3/II4OZE9DXDRI2qOR2lFiuLKBiUj2GIV9owllKpd1aJYvirDW+S3O9Ytgx04Yy8R0pXRjgQKLuRxoyMdPh2qp5QHy3OCza0PU5kGbx7UiUNWp3//MkgCVe/RYnwlP8eb9pxmz4InXfjwsgJroh3dnEYOW9wGZGf6whgCxJfQkQ2cPJYTo9PAO7UW+EcSn4WDaFfukg= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PAXPR08MB6969.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230016)(4636009)(136003)(396003)(376002)(366004)(346002)(39860400002)(6666004)(66556008)(66476007)(38100700002)(38350700002)(478600001)(8676002)(8936002)(66946007)(6486002)(5660300002)(54906003)(4326008)(6506007)(316002)(186003)(26005)(6512007)(2616005)(36756003)(1076003)(83380400001)(86362001)(6916009)(2906002)(52116002)(41300700001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?n/M0NFxGD3FISMJzaHxKokiaODyZ?= =?utf-8?q?etEq4acgqlPVHbgmoQRmgMCt1Eafs26GLsUhniGIPW/7bIIedd3ACWnQ5i2M70H9z?= =?utf-8?q?LPCF6cROfRoPAbb5scZKFkUsw5l2tUtVu/8kw7khfL1Av/dYAMI9n4Z70/JV7FV75?= =?utf-8?q?ekdPN1ReGrBvoSvjJSx8xlxVvuoYyrkl9jdfPT3K2LHv06NHlglt8xvaUeOZ64Uck?= =?utf-8?q?g4/N0esD+bAUm2Jt/v3T8lJ+Z01qEcaspjHYY9PsblmPhSGqvSw355iHgSeM/ErA9?= =?utf-8?q?SPqz9Ysi0vEbNRMKfCF4hn2NS4oj3gyONDZL+ADhQLn31BrSi6w4aDFJJtuwkSalL?= =?utf-8?q?0aXxv29HyKSMVOMePnliiduZuBDfBAX1k4/O+L7xKoTb5YhgHOxjZYysVgrWVKZGp?= =?utf-8?q?pFaKshVz83DLtKaNsVN+ffUrODgpiqasEnWAMMj5MDdd7sbhYdTnN3p0CZJsAIPqf?= =?utf-8?q?XWVCtvmFzXc+jv/0e51dt5tLg2kAiyujRhMcNos/yMIT2eM611m8uL0ujwcHxL5Ig?= =?utf-8?q?99p91hnt0p8mPCci92fknbJbwVvgHzcbuXoKN8zPq+lgi0G9LFLAotUQv8l2Bggwo?= =?utf-8?q?SZ70+UwzMZne/EEXRevIB7+M6OEeYPq5MK1B2Pe/dboeyrcAdh5xrYGP3tvgLQjUQ?= =?utf-8?q?6Q//EGWstF7Bw5KodS3m6HHX5Hiw1YQ6nnd0hldUWDzahO5gf4wkJwcrlTctNoPpc?= =?utf-8?q?XTjYBcC4QVTMewG0YDwAGYEvC5u0KbkFXzFnIrVjeKB76j3DcxGeUiPMAb0gmSEox?= =?utf-8?q?ZBITk1yggBfWv97JiQ9KyzPtBwoe+KfkAmXEyjS/hRhf5WpCV1FvNQO+tF4HBqA+j?= =?utf-8?q?yt/6q6Ux7LF0R3RHDbEC7Al3NgdVCs5+JLpvha8MEji9SuDxWOviR3Ihky0fnq4g7?= =?utf-8?q?ON7yecyXs2W4ynIVs+r8v2A4qvhZQVItCi1Wddr5KrMA9ujQt+jMOJp3XUq7Ws/Tq?= =?utf-8?q?2mQIh3XIp9ujqdayUFwy8w2brFbI4WqmXvsLWVBUoGzwPp8DeNUAKNLW5i9iGkSma?= =?utf-8?q?nprVsxRlqCfHJ7R4tqgimgsNLybhlXuF0ENiGZoEKy5YBgfk1x8uiwnsR4tr/CoNd?= =?utf-8?q?A5YvdS+mfR/wQ/ZSKQcA7F2uvjzfAVknV3eaoOC2LzPiZqMn+BFaQQaZv+R4MHmdI?= =?utf-8?q?F3V924PZKTJPBAOXvGwu5v/Ug+NKK/hDTUeuADbYNXiCOW/atHTWUbkV0MLCZL//V?= =?utf-8?q?Hcg0RgM3eaxrq71mPKqRyE/vbMW+z9eVYEOze2R8H2eqFMfdFGkCZtbH+LNUPeZe9?= =?utf-8?q?s+jG7/d5cGiaGt93igyJmXNciAAECgl99CUfW13cm5WGVykIay/ZAJHsTTyMDjR0u?= =?utf-8?q?5IXY4JJWrlp1b8H8vVqF/a0mCdGBQfAjYV3FINYiUGMHpQ7Mkd3JoU3PJeKEwVK7q?= =?utf-8?q?bP8kbiQr3XYwVTozFfBgiNQkr1DSGC55VBOxawVG55uSdqsF90ukTcb3ixK093i2/?= =?utf-8?q?Zd/n3xONGwcwoyj+mwXRYpi1iNwyOy4HacbjTepb08cut09ZpA0XKMK/NOyG0CYJR?= =?utf-8?q?NRiMOTi+fOsUGQ58CI5iqGo+gcP+6k32KA=3D=3D?= X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: d483a843-08c3-49e6-2efe-08da7f634671 X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Aug 2022 08:42:39.0578 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: jg0yTFtpRnDBubJBujimi1AxragqDIaKZKXwRwm1BGnsmQeM5jZ0JV7QqjKnB609YpIlPTS5+I0hN0glrVTLVA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR08MB3096 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.6 at phobos.denx.de X-Virus-Status: Clean From: Stefan Herbrechtsmeier Add zstd bintool to binman to support on-the-fly compression. Signed-off-by: Stefan Herbrechtsmeier Reviewed-by: Simon Glass --- (no changes since v3) Changes in v3: - Rebase Changes in v2: - Added tools/binman/btool/zstd.py | 30 ++++++++++++++++++++++++++++++ tools/binman/comp_util.py | 4 +++- tools/binman/etype/blob_dtb.py | 4 ++++ tools/binman/ftest.py | 3 ++- 4 files changed, 39 insertions(+), 2 deletions(-) create mode 100644 tools/binman/btool/zstd.py diff --git a/tools/binman/btool/zstd.py b/tools/binman/btool/zstd.py new file mode 100644 index 0000000000..299bd37126 --- /dev/null +++ b/tools/binman/btool/zstd.py @@ -0,0 +1,30 @@ +# SPDX-License-Identifier: GPL-2.0+ +# Copyright (C) 2022 Weidmüller Interface GmbH & Co. KG +# Stefan Herbrechtsmeier +# +"""Bintool implementation for zstd + +zstd allows compression and decompression of files. + +Documentation is available via:: + + man zstd +""" + +from binman import bintool + +# pylint: disable=C0103 +class Bintoolzstd(bintool.BintoolPacker): + """Compression/decompression using the zstd algorithm + + This bintool supports running `zstd` to compress and decompress data, as + used by binman. + + It is also possible to fetch the tool, which uses `apt` to install it. + + Documentation is available via:: + + man zstd + """ + def __init__(self, name): + super().__init__(name) diff --git a/tools/binman/comp_util.py b/tools/binman/comp_util.py index 8c0fe5078f..6b4ab646e0 100644 --- a/tools/binman/comp_util.py +++ b/tools/binman/comp_util.py @@ -13,6 +13,7 @@ This supports the following compression algorithm: lzma lzo xz + zstd Note that for lzma this uses an old version of the algorithm, not that provided by xz. @@ -24,6 +25,7 @@ This requires the following tools: lzma_alone lzop xz + zstd It also requires an output directory to be previously set up, by calling PrepareOutputDir(). @@ -35,7 +37,7 @@ from binman import bintool from patman import tools # Supported compression algorithms -ALGORITHMS = ['bzip2', 'gzip', 'lz4', 'lzma', 'lzo', 'xz'] +ALGORITHMS = ['bzip2', 'gzip', 'lz4', 'lzma', 'lzo', 'xz', 'zstd'] bintools = {} diff --git a/tools/binman/etype/blob_dtb.py b/tools/binman/etype/blob_dtb.py index 4fd2ecda83..fab79e43cc 100644 --- a/tools/binman/etype/blob_dtb.py +++ b/tools/binman/etype/blob_dtb.py @@ -48,6 +48,10 @@ class Entry_blob_dtb(Entry_blob): def ProcessContents(self): """Re-read the DTB contents so that we get any calculated properties""" _, indata = state.GetFdtContents(self.GetFdtEtype()) + + if self.compress == 'zstd' and self.prepend != 'length': + self.Raise('The zstd compression requires a length header') + data = self.CompressData(indata) if self.prepend == 'length': hdr = struct.pack('