From patchwork Fri Aug 19 14:25:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 1668163 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=weidmueller.onmicrosoft.com header.i=@weidmueller.onmicrosoft.com header.a=rsa-sha256 header.s=selector1-weidmueller-onmicrosoft-com header.b=FtuIJJn0; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4M8PGV5M03z1yg7 for ; Sat, 20 Aug 2022 00:27:14 +1000 (AEST) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 18F0284B67; Fri, 19 Aug 2022 16:26:24 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=fail (p=none dis=none) header.from=weidmueller.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=weidmueller.onmicrosoft.com header.i=@weidmueller.onmicrosoft.com header.b="FtuIJJn0"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id E3CFE84B6B; Fri, 19 Aug 2022 16:26:07 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.2 Received: from EUR01-HE1-obe.outbound.protection.outlook.com (mail-eopbgr130079.outbound.protection.outlook.com [40.107.13.79]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 812A184A3E for ; Fri, 19 Aug 2022 16:25:58 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=weidmueller.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=stefan.herbrechtsmeier-oss@weidmueller.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WSodFyUhiPLiLRjHW5TTRDr7CfJMvT5OS42tw30wovHBoteIhYP3HBsFHsVSeOGLEkKUVyaxv4U+8TkfUJ1lolt3fIk4Xawk48rPlf1sfqy9gA0el3hsg8KDIMzav3i8oI6SMNJflp8PLGZwTUf27HNUzyZUgRLG/5X6cX1YGvUNmaZDUim8mb93rY05ex+ceMy0+YP2xAksizRQQ9S5BNE3+/M7IOb2hERkZX4eexwn3U3avLI+INdwFo57wmp21SUfH4C9ETxm1jBnaOi75VhvspJAjYm1kbzIAE/C7s7+WZESPiL7KHavfvz55FVRt2pUTXrUpKr6aZxJxZYyOg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=THN6MxRaklmhzY8BS8ZlW8U6cCIP8SUdkOgGMQF+DMc=; b=RXQB1EYtonbX/jT2Hphczx62TPc5wAezCkCQJdZp1wal+XnwU5ExrVXaokMd64kl+H3qIAKBBRP5XvT6Sola13TyIEk6gjviT7C86Agx0eTBjcTVa1GlqB1BKtdnBVgPE/7l0142pKJv7+0AYuRUaGPAxcadeniJhhyxYeFdbRbC1TZQBROfC0DUhHE/fSa7a67HlLicj30KaUuD92oSEFTtvM5ruoWBrzkXR37Z2ElwaimlJBUjtFGexcODLSROzFOCOBV055MmTtgSelmRftSxggwnV1rAdXykCajsvGOaz4PBuO2PwNB3Y6dSCV8d8/BfM8KOE7Y8uOcH+BPsyg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.onmicrosoft.com; s=selector1-weidmueller-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=THN6MxRaklmhzY8BS8ZlW8U6cCIP8SUdkOgGMQF+DMc=; b=FtuIJJn0BNwDw3smtFuJPIwz9iBFgRLW5Mdnt+04JIk86DTC3HGIcvy36dA3d9HqsexjeQgyOxS1sD3l0LixfxScwEavCDMl6mv0mxyDeZ/MqHupAZGJ/8xtvxHMMAXbTI8w7Bu3P7Q6+9Su3ApEUrx/oXryGpjIAN45hfwf0o0= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by AS8PR08MB6855.eurprd08.prod.outlook.com (2603:10a6:20b:398::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5504.15; Fri, 19 Aug 2022 14:25:57 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::f95b:2e64:9ee6:4d44]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::f95b:2e64:9ee6:4d44%3]) with mapi id 15.20.5546.018; Fri, 19 Aug 2022 14:25:57 +0000 From: Stefan Herbrechtsmeier To: u-boot@lists.denx.de Cc: Stefan Herbrechtsmeier , Alper Nebi Yasak , Heiko Thiery , Simon Glass Subject: [PATCH v6 00/21] binman: Rework compression support Date: Fri, 19 Aug 2022 16:25:17 +0200 Message-Id: <20220819142538.24847-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.30.2 X-ClientProxiedBy: AS4PR10CA0015.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:20b:5dc::11) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 04ffb8fc-51c4-4378-79d2-08da81eebb43 X-MS-TrafficTypeDiagnostic: AS8PR08MB6855:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: NNoGPcTxeBYahnG3iRW850Mv3IWs09UW13uCCBWThRZYHGVbNwaAKduQsQyemwNmtyctkOMQfNefIF8RjsSmXjj3ET8nDs4CAk3z14t52tdBsZKMyXmjyijQ5gOPwBo8gjezHBoU6irzU0xgukhYmaJ4QgWlH0c80PndsbdaaMxL0dI6Jh7KstQVAdAYlwBAzouBnes7ngIjtZ6dorKHu0JQ9SQYceV4r/o3KRS6Q8iDWL5fTR1m2NTvURUciINuJEKVnt1vhhvrF+/xkb4/edaNm2UJnJlShRbLhGuJJaljAgIuLiRg470seuZ48jm8ay4SEeS1J2zHKzd+q7Wbw79ZAInUzBxX0vGrJ8oNg7aV9FaKEdUQ+qLn2C2FhRt6yZtQp3mXUGga6GlCwpJhHVCpL6G/EUuyGkhSWzkrWRkCHkuspvkpfFLEO2JzZo01JvGYwlCiatJBSqD/a5zfADOZcqgRkMgpcS0V2euKMOTfCAIGNBNTFY6ztsjyIlOdJzhQgTRRE98+ptnOyMR+jLdCY8Lpma3JgTJtdUB9PKiUfoio4hGbzet9br86wph4c8+9JGravQqUPxxujQnuv6TiydfAJ1wYKYODHaAAx78SoylhZbCBa/k7C4bJmDqXp9LpjAyn01pmxqtlDcPXGdEGDdqJDGzUJsHqmDlfkTI2KSzlCgrmDbhJK6GdDQ0cgInjaU/8O+qJmQuRxU0Hv0+7VshdoQ7/yeUXzEkmZtoMP6wftbepHC0XOdCTb50n X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PAXPR08MB6969.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230016)(4636009)(366004)(396003)(376002)(39860400002)(136003)(346002)(6512007)(6506007)(52116002)(2616005)(2906002)(8936002)(5660300002)(6666004)(478600001)(41300700001)(26005)(186003)(38350700002)(1076003)(38100700002)(54906003)(86362001)(83380400001)(6916009)(36756003)(4326008)(8676002)(66946007)(66476007)(66556008)(6486002)(316002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: i2uk79+G6wz4kg0kTJM97EDDR7ABREMUtDEVWMP8A9D5IeLV2rOCQCkIBdrdrWben1dzD5HHslnebPzpHU4X5H/jdRafDB9ZbhhTTwBFjlVAfgadBkEX2mtWG7Ml4Bl0hbev6f66DoocvOEP2xbH7RsUYG8uIB37K8zy43H6uiOKqYAZvecHUsJWztIgv1PqodpiIRfFnm3ire8uqiudODHrtZBEe5363sz1LK6wIGub0CZFFv1HdHmQma0f70U8qL/qdCAqkq1qrnuR+DDd1bK2MjFiZRJAexqyXLhwRxqZ267RsZn1pA1pqaO7PmAUoKLCLaRRUt05fu9TVExGTP3T/oDdeq1dSwui5mvJwbUv3+QQxrvfV7dBGHpVIMCHDlWG37Gto1sWUeUThsElKl12HyfsemG9FUyp9fqCQgIjcOQ6GdOB3/SBlnaSA3r99ZaeoYNOcJRCSlpXFkt6hGB2F+q8/W3gHYm9opAIa4afUBNsfP9akQWmzowUlTER0kXLWmKb/ASOPzqDBZOtsiXLXBCPESI8dJwhHLKhDtHfbLOf9kXNP/cPryuCkf6xtfzvY+VwCKMHWh6nMuQKXZwTtB1vELtwxUIjZzRFZo3MJSg63K2EGeILODSSxUC7QgIrUcgmwE/a6LzroJAnAC8Gi8xzcsvnYYqlNyY5dhZrj4fSwhb8FA//8+q12Vly0bfj+7lUUSHyx0mxDdJbb87AxIvZWIj9sJf+iT6nX9LAlSN1qlC8NN2A0iOCru39IASgm6gQNR2OPp9vNuffzM2h5cV2dl++P2I3Wi2NboEHQTWxC2hNikhIHEthU8t/YeXln700kcT/BthDcSgCSKTP6ldgXY1JxskW7BmPo7ySNKYepa+9y8rRQgnlQmBApU43R/N9gTokCREUsB0JpiKJkPjE6F9N9dEecMTzYm3jTizPBC8gbcUY+z0zIvFKsN26PevID803nGfRYP4vjfrnY5cw+rU0LDxH3VOta+RqBmlAFvqckk2zzbFP7GAYmwogzPPd89IsPIWjwpb/lqNZwDMvx6nhabmA9cNFyu0V9AUhfb1acbvVoW0bF5noz0wbsnrGQvFMdyrJCF+G4WDKrSszXfDACYxcD0g1XMXaIUf4YSMq77nLyhmPCMZOnkxHl5k0y43qIlMTmbf4sN48rITKRIKGN2ideL0ujmfIjMJ077qLtRthcjxrbQrAvo6P6zSpiQaXBJMVFU0mQjIhOVihz7SLAYPDB0Ic2vVWRl/tnNXRF9ZWvDwKhquRixxCp4vJfaxfZ02KWnfDyN0z/Af5vtsUxFzISa5pfBLthM0QU/Boi0rLwvqZW4rIx5wFSNPkdkTvn+YUB7GhBeosfYODvC4M5KBpjdujzvC0rUlo/er2vQ074HhvOvJxPFzolDm8sXUsTniMzeYVRPwy16cdYYWpv+sf+AtoFCvx41dNhlCPzfUQvEPLpGKtNuxfOrq1P1i79LAF/OuPd91VDKkbe4gZ/F/1QUq/lsxONFWZUdygmrFpPSC2vtFBKfwmIsmVcNBotl5YAM8Hc07C297T3uGf8rnYBUMaG8qk/9/W/X0EtXNkTpAkmeDoIfuUOK26wNSD61QgYeUoew== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: 04ffb8fc-51c4-4378-79d2-08da81eebb43 X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Aug 2022 14:25:57.1257 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: R4CXSO6MFc5BZdFbGqh/p95liYHxN5PQZS1OmAT5WM3T2vTVQNLOzKtOVKhPK+lF0zNY02U/ysQwtua1URlkrw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS8PR08MB6855 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.6 at phobos.denx.de X-Virus-Status: Clean From: Stefan Herbrechtsmeier Rework the compression support of binman. Add a length header attribute to dtb entry, manage the compression bintool in the entry class and add support for several compression tools. Changes in v6: - Set uncomp_size after decompress in DecompressData function - Set data with tools.get_bytes(0, 1024) instead returning Changes in v5: - Add commit to avoid duplicates in bintool lists - Add commit to forward AddBintools calls - Add commit to forward AddBintools calls to base class - Add commit to collect bintools before usage - Add commit to check only section data in multi section test - Add commit to add DecompressData function to entry class - Fix decompress - Disable header in testInvalidCompress function - Remove header value in testInvalidCompress - Add commit to move compression bintools creation into test setup - Add commit to select compression bintools in cbfs_util class - Add commit to move compression bintool management into entry class - Use record_missing_bintool function - Use tools.get_bytes function - Add support to DecompressData function - Reuse 85_compress_section.dts file - Remove 236_compress_dtb_missing_bintool.dts file - Use self.comp_bintools dict - Use _CheckBintool(bintool) function - Remove self.fetch_package not None check in fetch function - Add algorithm name to entry.py - Add tool name to ftest.py - Add algorithm name to entry.py - Add tool name to ftest.py - Add algorithm name to entry.py - Add tool name to entry.py and ftest.py - Add algorithm name to entry.py - Add tool name to ftest.py - Test that zstd compress of device-tree files failed - Add algorithm name to entry.py - Add tool name to ftest.py Changes in v4: - Add missing 236_compress_dtb_missing_bintool.dts file Changes in v3: - Move comp_util.py changes (drop with_header) into separate commits. - Add commit to disable compressed data header - Add commit to remove obsolete compressed data header handling - Add commit to support missing compression tools - Use 'tools.get_bytes(0, 64)' instead of 'bytes([0]) * 64' - Check if tool is present - Rename tests - Document class properties Changes in v2: - Add commit to skip elf tests - Reworked to make the feature optional. - Add commit to add compression tests - Add commit to add BintoolPacker class to bintool - Add commit to add bzip2 bintool - Add commit to add gzip bintool - Add commit to add lzop bintool - Add commit to add xz bintool - Add commit to add zstd bintool Stefan Herbrechtsmeier (21): binman: Skip elf tests if python elftools is not available binman: Avoid duplicates in bintool lists binman: Forward AddBintools calls to sub entries in cbfs_util binman: Forward AddBintools calls to base class binman: Collect bintools before usage binman: Check only section data in multi section test binman: Add DecompressData function to entry class binman: Add length header attribute to dtb entry binman: Disable compressed data header binman: Remove obsolete compressed data header handling binman: Move compression bintools creation into test setup binman: Select compression bintools in cbfs_util class binman: Move compression bintool management into entry class binman: Support missing compression tools binman: Add compression tests binman: Add BintoolPacker class to bintool binman: Add bzip2 bintool binman: Add gzip bintool binman: Add lzop bintool binman: Add xz bintool binman: Add zstd bintool tools/binman/bintool.py | 106 ++++++++++++ tools/binman/btool/bzip2.py | 30 ++++ tools/binman/btool/gzip.py | 31 ++++ tools/binman/btool/lzop.py | 30 ++++ tools/binman/btool/xz.py | 31 ++++ tools/binman/btool/zstd.py | 30 ++++ tools/binman/cbfs_util.py | 20 +-- tools/binman/cbfs_util_test.py | 4 +- tools/binman/comp_util.py | 76 --------- tools/binman/control.py | 5 +- tools/binman/elf_test.py | 14 ++ tools/binman/entries.rst | 3 + tools/binman/entry.py | 54 ++++++- tools/binman/entry_test.py | 9 ++ tools/binman/etype/blob_dtb.py | 31 ++++ tools/binman/etype/cbfs.py | 5 + tools/binman/etype/gbb.py | 1 + tools/binman/etype/intel_ifwi.py | 1 + tools/binman/etype/mkimage.py | 1 + tools/binman/etype/section.py | 4 +- tools/binman/etype/vblock.py | 1 + tools/binman/ftest.py | 152 +++++++++++++++--- .../test/235_compress_dtb_prepend_invalid.dts | 17 ++ .../test/236_compress_dtb_prepend_length.dts | 19 +++ .../binman/test/237_compress_dtb_invalid.dts | 16 ++ tools/binman/test/238_compress_dtb_zstd.dts | 16 ++ 26 files changed, 590 insertions(+), 117 deletions(-) create mode 100644 tools/binman/btool/bzip2.py create mode 100644 tools/binman/btool/gzip.py create mode 100644 tools/binman/btool/lzop.py create mode 100644 tools/binman/btool/xz.py create mode 100644 tools/binman/btool/zstd.py delete mode 100644 tools/binman/comp_util.py create mode 100644 tools/binman/test/235_compress_dtb_prepend_invalid.dts create mode 100644 tools/binman/test/236_compress_dtb_prepend_length.dts create mode 100644 tools/binman/test/237_compress_dtb_invalid.dts create mode 100644 tools/binman/test/238_compress_dtb_zstd.dts