From patchwork Fri Aug 19 14:00:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 1668114 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=weidmueller.onmicrosoft.com header.i=@weidmueller.onmicrosoft.com header.a=rsa-sha256 header.s=selector1-weidmueller-onmicrosoft-com header.b=mxu84Dk7; dkim-atps=neutral Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4M8Nhc4KwSz1ygd for ; Sat, 20 Aug 2022 00:01:20 +1000 (AEST) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 52E4E84B78; Fri, 19 Aug 2022 16:01:06 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=fail (p=none dis=none) header.from=weidmueller.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=weidmueller.onmicrosoft.com header.i=@weidmueller.onmicrosoft.com header.b="mxu84Dk7"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id CF0CB84B5C; Fri, 19 Aug 2022 16:00:59 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.2 Received: from EUR04-HE1-obe.outbound.protection.outlook.com (mail-eopbgr70071.outbound.protection.outlook.com [40.107.7.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id B016884981 for ; Fri, 19 Aug 2022 16:00:56 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=weidmueller.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=stefan.herbrechtsmeier-oss@weidmueller.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kch8CHRbQ9epRgT9lzgs8YLSOk2b+AHBl0HrOj2jfo/M9Eyg06WPoJD5YCv6hGvwlyNkCpc08eqHgun8rSrqq3vLPxQBYvoCHpYgzXTAY+ilqmB85zQGWNE0yrwLEtXGkr/+esCwvRBVSLakXNMBAEnnG3A57Lzjh7CsWCRM10TQwaxOgf77nsWMvQpifRGHlZrrB6bPNGil8sW/+uCxskulJ5XddlmvUD8vvpPi3Q/5o5rrmkYfX+4wy6Xx+6ggckV5aFLntCxN1vzEn3/iXfgrlfcceB+5vQU3+rPSjaaESgiDsKESwkPTiq3UNofU9U1EO++pm0X6cO8ZvoSAtA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Gk7k2Yl01i6wC+OkP5mSGuNKJc5or8P4Ga0w8NiDIws=; b=d34U22+pOZlRBkdVaGC1lAo66fxc79Bk5o+hc1GQq4gv8hMCiARd6tYeIaU8fvYFpewnXtKm5J3pYlEObjL9muTGsylrqbwNmNAB9c3epcP+imU8a34dph6ocIXhQBYu/6NxIodeJ0v1fUt38f+YKo/lwhvSH2/y223OQej1Cl1cED5xp1OJ/I2GXZVYTt8ujGUqj5o5/KxmPDcLXEsC5uVLgPzfGC3Cg9xRHmYTrgXNv6qJWfdncSRJAbgYklt4tI10Bm+x3ApLhc9rLhQ2Xtl1yZW05kBsGvPy/TB3R9GPI/PpUWRzqiSO8gmXiBweRCUV+FnN5COQqugKB4rvoA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.onmicrosoft.com; s=selector1-weidmueller-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Gk7k2Yl01i6wC+OkP5mSGuNKJc5or8P4Ga0w8NiDIws=; b=mxu84Dk7hqRy0yHBL10ok+OZ9GWwDTs/O7jSBO9t73M3T7fJSAksOqKlWoo2bq2k10esh4+5vuBheUv2pLKJql7a70zZDkY3UMOmeDMgcEp82pxH6NavdZ4Csq+SxcSmO25jrUkpW1rVcVg9b/XBvVovMK9agIY7Rz51cSH51n0= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by VI1PR0802MB2302.eurprd08.prod.outlook.com (2603:10a6:800:9e::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5546.16; Fri, 19 Aug 2022 14:00:53 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::f95b:2e64:9ee6:4d44]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::f95b:2e64:9ee6:4d44%3]) with mapi id 15.20.5546.018; Fri, 19 Aug 2022 14:00:53 +0000 From: Stefan Herbrechtsmeier To: u-boot@lists.denx.de Cc: Stefan Herbrechtsmeier , Alper Nebi Yasak , Heiko Thiery , Simon Glass Subject: [PATCH v5 00/21] binman: Rework compression support Date: Fri, 19 Aug 2022 16:00:14 +0200 Message-Id: <20220819140035.32760-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.30.2 X-ClientProxiedBy: AS4P251CA0011.EURP251.PROD.OUTLOOK.COM (2603:10a6:20b:5d2::7) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: defd21f7-e506-4e39-818a-08da81eb3afd X-MS-TrafficTypeDiagnostic: VI1PR0802MB2302:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: MCq4ja7QnJ2eL90gV3IQrld30cIsLclllixIXVhxacttImebed5mV5mOTPZ293Z258Kgm0Di79B9ozFJ/BK7rJ/RJmDt70xdasXIJoCagFETnoyodt1CG1oiB0tzz4JKlvG/AIHjfB7+CLPLbwc4bWPI6ZT7SFHHVMkeW28sfZdCfi09qYyEwjwkRhOkzANN+f/ZVOg35qNZpXk8EUtE+d6/5mZoy7nbEGjRL37Uwmd+EeOsKtqQBbwX8R50PnmvwCp/QQ04e9hC/3ysm3wlS7tD4mLdvkZMabbsDcYQBLIU8dq9rg5T52Kd4gLxztiTJOpTVDICm8PTYzOHEGFS8pfIOFq5DNHjT8mIW8T6i93ssKgFOQ2fQ8a/ao4/PX+dkrNz59QDScTNNyY+Bg0EhmuIpmM0tUPltw15vZpzM3zQ7dPGyuMkRYqPtR/i8UxkRtm9c9UbddJZEkqnynNu80nM82IjJb9FIbN2qvtUzmjhNdwWWcl5G6BcX5y7n6wVVWG29KvbXW/2FfMOn0G7eEp7hd4XxBkwqBlQ7Oejf6pkMr52/WWMdy7odOk3jGjh9GqSnG4nREiotOfrGlAwiti9ByU+zk7/cRuXGQZTZbGqOnicDkAbkGMppd/quxxtgIncQZ9Q2qjdjWNkJwVasY17TRnLToU0NOUToTWryixwFkGlqn6yS5vC8awK7+0DG7TYNdBUKtmxE81ljcer5JxSK9b7WimU5RCBryNn6BDf9e8CaUp1xcKM0ZO7tvRo X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PAXPR08MB6969.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230016)(4636009)(346002)(39860400002)(376002)(396003)(136003)(366004)(6486002)(6666004)(2616005)(478600001)(316002)(2906002)(54906003)(6916009)(36756003)(41300700001)(86362001)(1076003)(186003)(38350700002)(83380400001)(38100700002)(26005)(52116002)(6512007)(6506007)(4326008)(5660300002)(66946007)(66476007)(8676002)(8936002)(66556008); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: YnKZoq3SV5dHnWaK1CLPZNryisrGvuk54UjS7ihkzdBkSMG/UqQObsI6OXXlOn7Fg/dE6ObUGjLXrBiFJlHhQzzFn9FXiVW35Fy4uSVZYK1BzI/rR4iceIe6fkb6WK7vsMXNOEKuhrgvMxoyLsMG23uYu55qOiE8jMWX+B8opcherjmvivormNB+FiHG8kvvp3RrW90zBCONFSFlrhAix5L1mBwd8UZebw+O+gkQmeSyx4cT5Nsy/JfOWs/UQ7/GqMLzz/cbdm/eB9cOCfu/nraxRgH9NbTRj3r2eJc6NPPfxGrE6RWVSrO1qW/fkbPPwbCWo2f0RTqaGkie7qwJO1OK0jqAAYpn16X9zorqa3AlHXAulhDlJkdlqK2FC4t9EGw4XuVgf2YeIdVpaoOdHrNnLijRdE3LzAs/y66AeYyslciymz0yigLlX9Yd0Nr1BNRJ1s9zGo4D0WMt4AKzv1a/uAuvfXv9nrWEX5TC95oE6RuAHr+PU//8zr032Akia7PGpAZen3d+uzSu9dJXYuiiwHC2I4ms5K+rTYXoEhzQgofxH1RKAhKS+joUmOLh25q7bGOBo5Rv0IxoPD1iNdNCW2boXmNwe8C88CgvzEc3PS1tvGUQJ+G1CYVzJ1FDZrDOJq6//oCKO1y2F1/CeJZINv0J85rYGGXebAXOkm+sceMIwyM2eiogOIwOwGMzn0QWhqMZ/eKgfLvJuj1juWJS/xKBQn3J92gR5I5S/hmlOk60uqHghL8SSHAGiMGFwAqK8LWBxAa8dX7lwdlXEN1bdoQkQWlTQazyHBxw6SoJZR0bwHhqSZMDNj9DAbOL+DrEgeH+zqtZ/eQgfxpriqj4iObnSsOW6iCQBFV9zqcVel8AHNDvYL705oC47crUSF4sQyilMkRJ6CutSdFUgi1IIx40J6PVZmI99GvST/jB5VoaI3phlC1EaXu3GmIFfsDd5Yp0jPQbo1pSQThCCk6LeinO93ow8C+10oxdxExOoMlFO8dZKf6+j5B6gt8y80v3n6bcUdXTihH89Z9JZ3Ei6EcfrWYigmu74TADD5u7JyB4l7WiZ3eujznyF7rITIbw1bdXY6q/6KgyEfPxtWXETPRDmQsICV21h07HvCJ+hlx9YevdGepvM1R+PdYTyUhaMWTxLHz8+xnhjWbRratSPCeN9apw3qaYecpaORVdMUU7yxCgBQ2o1F6ZVW/YIHkVIO3goQc0nxAqHOlCw7qWJklCB8jOFsfVjPnivY+sMRtG10KggDRp/tJQC/ERbR4RHP13CWDmeYvfrvd0t+k6Ys4pHyZHC9La0oVVh9k+O7isCn2/QGncjnVMReqYPjwk281U0/+CQAXvE33dCiMM0/FJdIgapKkNWYlB1RpXQrczsuUgaf2oyV4tGmgcTAqOvyuE0ZTVN20R9VVtzZ5XvwMd4pB1jifMMcYmW/JYCdSsp5bbKYEmvOqLGuDFOEjQMpazlMyOKWY42O7RJX1UJ6mxYZ1lHhtLYNYikXFMV3wTd96n4EXmvlhNB70AlWuviwlxB4TKkYCfTURVObcgR09Q5HI7El6sweRrtqea+BP69EyZrkgNq2rQHtC9Dh+AXPztvigrVf8fn04X9g== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: defd21f7-e506-4e39-818a-08da81eb3afd X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Aug 2022 14:00:53.4429 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 6Thrr7XgZtJhq2SqFGpU4jAC6MaNB8YNQNKMWZdMO3VxnyqKhzJZndrCaM32CmDOmkt7RNDEJe1vkI2TUPImFQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0802MB2302 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.6 at phobos.denx.de X-Virus-Status: Clean From: Stefan Herbrechtsmeier Rework the compression support of binman. Add a length header attribute to dtb entry, manage the compression bintool in the entry class and add support for several compression tools. Changes in v5: - Add commit to avoid duplicates in bintool lists - Add commit to forward AddBintools calls - Add commit to forward AddBintools calls to base class - Add commit to collect bintools before usage - Add commit to check only section data in multi section test - Add commit to add DecompressData function to entry class - Fix decompress - Disable header in testInvalidCompress function - Remove header value in testInvalidCompress - Add commit to move compression bintools creation into test setup - Add commit to select compression bintools in cbfs_util class - Add commit to move compression bintool management into entry class - Use record_missing_bintool function - Use tools.get_bytes function - Add support to DecompressData function - Reuse 85_compress_section.dts file - Remove 236_compress_dtb_missing_bintool.dts file - Use self.comp_bintools dict - Use _CheckBintool(bintool) function - Remove self.fetch_package not None check in fetch function - Add algorithm name to entry.py - Add tool name to ftest.py - Add algorithm name to entry.py - Add tool name to ftest.py - Add algorithm name to entry.py - Add tool name to entry.py and ftest.py - Add algorithm name to entry.py - Add tool name to ftest.py - Test that zstd compress of device-tree files failed - Add algorithm name to entry.py - Add tool name to ftest.py Changes in v4: - Add missing 236_compress_dtb_missing_bintool.dts file Changes in v3: - Move comp_util.py changes (drop with_header) into separate commits. - Add commit to disable compressed data header - Add commit to remove obsolete compressed data header handling - Add commit to support missing compression tools - Use 'tools.get_bytes(0, 64)' instead of 'bytes([0]) * 64' - Check if tool is present - Rename tests - Document class properties Changes in v2: - Add commit to skip elf tests - Reworked to make the feature optional. - Add commit to add compression tests - Add commit to add BintoolPacker class to bintool - Add commit to add bzip2 bintool - Add commit to add gzip bintool - Add commit to add lzop bintool - Add commit to add xz bintool - Add commit to add zstd bintool Stefan Herbrechtsmeier (21): binman: Skip elf tests if python elftools is not available binman: Avoid duplicates in bintool lists binman: Forward AddBintools calls to sub entries in cbfs_util binman: Forward AddBintools calls to base class binman: Collect bintools before usage binman: Check only section data in multi section test binman: Add DecompressData function to entry class binman: Add length header attribute to dtb entry binman: Disable compressed data header binman: Remove obsolete compressed data header handling binman: Move compression bintools creation into test setup binman: Select compression bintools in cbfs_util class binman: Move compression bintool management into entry class binman: Support missing compression tools binman: Add compression tests binman: Add BintoolPacker class to bintool binman: Add bzip2 bintool binman: Add gzip bintool binman: Add lzop bintool binman: Add xz bintool binman: Add zstd bintool tools/binman/bintool.py | 106 ++++++++++++ tools/binman/btool/bzip2.py | 30 ++++ tools/binman/btool/gzip.py | 31 ++++ tools/binman/btool/lzop.py | 30 ++++ tools/binman/btool/xz.py | 31 ++++ tools/binman/btool/zstd.py | 30 ++++ tools/binman/cbfs_util.py | 20 +-- tools/binman/cbfs_util_test.py | 4 +- tools/binman/comp_util.py | 76 --------- tools/binman/control.py | 5 +- tools/binman/elf_test.py | 14 ++ tools/binman/entries.rst | 3 + tools/binman/entry.py | 54 ++++++- tools/binman/entry_test.py | 9 ++ tools/binman/etype/blob_dtb.py | 31 ++++ tools/binman/etype/cbfs.py | 5 + tools/binman/etype/gbb.py | 1 + tools/binman/etype/intel_ifwi.py | 1 + tools/binman/etype/mkimage.py | 1 + tools/binman/etype/section.py | 4 +- tools/binman/etype/vblock.py | 1 + tools/binman/ftest.py | 152 +++++++++++++++--- .../test/235_compress_dtb_prepend_invalid.dts | 17 ++ .../test/236_compress_dtb_prepend_length.dts | 19 +++ .../binman/test/237_compress_dtb_invalid.dts | 16 ++ tools/binman/test/238_compress_dtb_zstd.dts | 16 ++ 26 files changed, 590 insertions(+), 117 deletions(-) create mode 100644 tools/binman/btool/bzip2.py create mode 100644 tools/binman/btool/gzip.py create mode 100644 tools/binman/btool/lzop.py create mode 100644 tools/binman/btool/xz.py create mode 100644 tools/binman/btool/zstd.py delete mode 100644 tools/binman/comp_util.py create mode 100644 tools/binman/test/235_compress_dtb_prepend_invalid.dts create mode 100644 tools/binman/test/236_compress_dtb_prepend_length.dts create mode 100644 tools/binman/test/237_compress_dtb_invalid.dts create mode 100644 tools/binman/test/238_compress_dtb_zstd.dts