From patchwork Tue Aug 2 12:27:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 1663000 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=weidmueller.onmicrosoft.com header.i=@weidmueller.onmicrosoft.com header.a=rsa-sha256 header.s=selector1-weidmueller-onmicrosoft-com header.b=BieCKK1T; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LxvS6418hz9s5W for ; Tue, 2 Aug 2022 22:29:10 +1000 (AEST) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 7FD1B8455E; Tue, 2 Aug 2022 14:29:07 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=fail (p=none dis=none) header.from=weidmueller.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=weidmueller.onmicrosoft.com header.i=@weidmueller.onmicrosoft.com header.b="BieCKK1T"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 7831084547; Tue, 2 Aug 2022 14:29:05 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS autolearn=ham autolearn_force=no version=3.4.2 Received: from EUR04-HE1-obe.outbound.protection.outlook.com (mail-eopbgr70054.outbound.protection.outlook.com [40.107.7.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 6186F8455E for ; Tue, 2 Aug 2022 14:28:59 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=weidmueller.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=stefan.herbrechtsmeier-oss@weidmueller.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dmwohUOa+tGe+8HDs0lu+jaSS1CnckMqW3taUk966AbF7uUzTkTL3AORUjbHECyXuxDMffy/agpqQ+ydYvjxPdMEvLqmIkGBfv2fQq+JAxxyB7VpD2qoNjpOKlJvOniSUyDlGLrl30/oR61E+IR3nPREU8OQ8PIBGmPdCJpOL3dhh2D69+5eVfV9/tVnuouQ6978AkDgoL+dbtBA38lnvxt+hZjRR5sDQK7Mm5kGQElMQwdlOpeO0BjBsjzXerwL9Z02TulBbLRnNQeNIaa9lPtRAR58RdfQl9CZXmYJGOCxEyLlAa+BeWV2bI9+UDU/Ij4aMXU2YdHGyreLkerFMw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jTqF0Lbe3Su0iCojIgleI4rCxhziSri1dJk0XNKAakY=; b=nYXsKufSsMhTBMMM8m3Wc2UgnjoRbAfekzd9mGkj0jsGiKoi6wcqKEfPfhCP2Jmmlc/rkx4teg4lTJVkDDFKsX9JqKtMXufaiscAvXxtJny95dIUWfgjNVTadZH3+FFUyGbeyk//XpWnP6BWFXhtqpBn9+cvRuvhx0TU5Gsb/4mDPkwoCEUuJM3/9QhICLbEhu2lIMlu7zNt+IHmRmrGwhF3RaRx32YOwh9mnpSkNy/Z/cIwChKStlvhxzSV6ZH6J5wa68zyJWHELSg/cJFtjoUcmnFhcx2UrRhbiWuYDOaOBWZTBXbGI0zSNjghnE0el4N50fxhMYJeskdLvADbrw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.onmicrosoft.com; s=selector1-weidmueller-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jTqF0Lbe3Su0iCojIgleI4rCxhziSri1dJk0XNKAakY=; b=BieCKK1TI1V/Y7rp2ivP4IYLvAEYwD0YV/z5XNd74GtTRGbOQIMgRduL4LGo1GnRczqvmLL1T+EBUnQPyDbu0NolbA2kIf8pzwS+3GEWwFhkYoZtoM/ypNybzWTUARHpOKjMjTFHkr1/iPcDVkokLw2q3c9EjvsxzJQONozHqSY= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by VI1PR08MB5470.eurprd08.prod.outlook.com (2603:10a6:803:136::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5482.12; Tue, 2 Aug 2022 12:28:57 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::49a6:e5ff:7f94:8787]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::49a6:e5ff:7f94:8787%7]) with mapi id 15.20.5482.016; Tue, 2 Aug 2022 12:28:57 +0000 From: Stefan Herbrechtsmeier To: u-boot@lists.denx.de Cc: Stefan Herbrechtsmeier , Alper Nebi Yasak , Simon Glass Subject: [PATCH 1/3] binman: Use low level compression commands in tests Date: Tue, 2 Aug 2022 14:27:56 +0200 Message-Id: <20220802122758.20155-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.30.2 X-ClientProxiedBy: FR0P281CA0073.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:1e::13) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 9f262d55-7bfc-45d7-baa4-08da74829202 X-MS-TrafficTypeDiagnostic: VI1PR08MB5470:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: nBKPXlGPQgjZOJ2V6hVReezLffj1mIdswr98wc6kfVCEA9tp26NIi7EGsHGgv5rdnuoCQ0SXlYqVaYbwhifypJRnKFhZ/8TjEIywLu97MfBCZr67UHkmVvcTV9ZCtMBUdu+808S+A6YKOigAr8WoHnMKdMjCh+MXM8X7wYdlNU/hwDJgVWQ26dLeWpPEIt4kwdgp6VNF02aU1hHdhlwnX73iEbuF2Aea/rvxiAzbr/kfejl+GfBUXchjGffdlc0D22a2cW6rVYKXiZj0u7EdM3nJAHjxhNXWlMY/B3m2GM0D1ZUvSiJtWypRUbseFfrVDZFHxk3xHgQLSrKLR09JAjt7omyS/+9iq1uCsFSoEX+VOkS1m/zR24GSXx0LPdm7cbjFY2ZXbyk8pzLdrlj/NxL3QPz0mv8sEPK0MTVcA8sqWFAJ+eTT2WtEATeFPChz+xb6UztEpW+Y+n0xSP26d9IDjwYlWA5W9oJoTRkLWOWqRRZCu98B6RvEH8zi0xDnpoAl9T1ZnlcuQalFMWb0V98xNlJESOj4DL1dQY0LFzATrGGsQLfWiZy8MGDuS2Eh5SU4L2VpntDbcy6fyQk4sGRfC7MWufmbba9hbK1Wszt5O2ZpuZ8N6G8OcIo4xyOnBm8jI2hDxLhwHMl3Qxc5ExmM4xS6FfyhpwUiik3LdBOf5AZHQJ1mgs3sfc6SDzhE7e5KX/8rfoQVWULjEbOW/0n3Ay9YAE6NyhQII8kNAvr/d6JYJ2Mq+Uk2ppIrRqGiNiAGu5wt475giLEJUYDSH4HdXmMKJT+QJjda7swdShw= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PAXPR08MB6969.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230016)(4636009)(376002)(366004)(346002)(39860400002)(136003)(396003)(186003)(1076003)(38100700002)(83380400001)(38350700002)(2616005)(4326008)(6916009)(66476007)(66556008)(66946007)(36756003)(8676002)(54906003)(316002)(8936002)(26005)(6512007)(478600001)(6486002)(6506007)(86362001)(52116002)(2906002)(41300700001)(5660300002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: fawqG5ITOeVxFYfmqdjqyYkoAJvWMMsf0uH125iEwpPDXCiJ7fMFWU30Fet6VaAtXft2mQEsHCdLb7DEvZFJnJtzPZ01hZyyNYmAW7cUSsp7IbmmLjbd1oXbAfjRDdUQJy8pOnsuzgrTHUASkXtp+7vGLg1HFbPERGpbgBy4a+7JzpsEfj9fijl9rYFfFskTq9dupyhRpaHl3XvpA1rJnYpfT8mAIKTsRjtB++0S3oRH1JQztEZiGV7VpaYQnx3NMNSIdtv9O1rhNjCJkDckNiqDh0hqdlenAHv5nTAB2L1QwDwAYj/ibTftSKrw7gFShFzlGuZ8zdxSe91NRYZNdcUq+uPIz4Vten98Izs52qEe/i5+RZjt5DJPVLe0aOBvwco4C3MTrmrdwulYwcMMbZn6ZytxpHZgpXogur6iFP1lajQpoQl2mOQMsxS4kMJVfuW0gT1x3elXFRA8aTUskkovFvMz9mDw99r7uF+Sx5ZxY2ntbWGhX9sQX8EX+TY4V3Xi3UCOhTOvaTk7Z3hxCooBUnDzc8dNMQg45pdaGLK/lGj8l2zOOiZZMt5PYYfvrNSna3SwOLoYjFAcZ2Rr/gMN8UMxSnDkqJBTAHxSKhbSyNJyWt2h+35yMydaw2kjd6DEx/b5B/vunL+2N15ziMlDAnLdq97H/g+OG5SiQacSQidFMkoycEg88jvZRZ6L8a9LJ0ez/+sZIhRRdotssT63GW6juS87v8qmyo8B0oG+5pvKcmeEqwnM60Qm3PWSDieZu7HY2eGCaFAvcchxraxShzAvI2nEEMfa/tdJmKuDzCD09VjpsFmGF6CPOpkmUGbbtnk5W5GzpWuNPZX2cAMfPJzRPNNIVn2SwxtIrXprpxU+IQXEK7ArEOVjo+dDtIT9QvpznYtPgpvaRJoi3Qa8KEKPnI66Gw3y09Rre2o6jWWK0+RFItXVD05c4Fs+BHK7W5kjgGNN6SX8ScumhSCIPSKZh60JiK2uIU35o9b/aBPh5W+r0lBRngsqxTbyx9JLP7tBm8uaQo+IYUDzrXtUwLt5Im53N37SYx9PZNn8p1wVzxDXkA2piTeVA8H5eoSrLuojnfDbK5IJMf6wxOABR0/GmYSFkFcLPJ+XIZzV6hKxBu7LJuL+5LrFT1Kkm/mlbqnzX6EHi9NQmZoLB3UrhP3FQDT+AfQVvxw6Xkt8dgqe4AJB4Nw2LJ7vRTdi84WH61vsUOp5/W3ykpeqoItS7hKVUJl7xkozVVu1w4GoNQ+Qqw+P6SpZ4DIMUNeSqGwZIdAgJXrNmGTky2lYNKk9rQAYxqaWjPLk1CQo1GAsnfgstYp3BEq3ejHqBNmJt19/VcIpRGkTF9iFcvPT9iltnP/f+mspN8y2fszb5EHYAXcSNmVU7gsZ+9fJDO1eOBqmv/UTkBUWEpGGieGPIOGZ51Qq7Q5J5xPz11gU9EjcXdMbTJ3w5ZYnthyD/y1Xdnl0ECQR/Cx8HEgWsN4QiLiCwy8P0kvtJwgfQg5VPsWQR4rnoO4PIxROVf68uJraV1wr26pTLVWw+X1wqGq2Y3dYkSdO6fZNLC84D+4GCf+mKm76Umqm5lUMHtY2CzN2rfXlmDIAKheqJ0JoWGyHSg== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9f262d55-7bfc-45d7-baa4-08da74829202 X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Aug 2022 12:28:57.2600 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: TJu8cExhnKCMJS71AaQa10uIxRfkAnEb2BkAWv2wTY7l1jxKaZ96seSr1gIJob+5v0OsXS+RLMneBczTzIMXgw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR08MB5470 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.6 at phobos.denx.de X-Virus-Status: Clean From: Stefan Herbrechtsmeier Use the low level compression commands in the tests to detect failures in the utilities to compress and decompress data. The decompression in U-Boot expects plain compression data without any header. Signed-off-by: Stefan Herbrechtsmeier --- This commit breaks the binman tests because the tests wrongly expects a header in front of the data. tools/binman/ftest.py | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py index fa1f421c05..b1f564ed7d 100644 --- a/tools/binman/ftest.py +++ b/tools/binman/ftest.py @@ -107,6 +107,8 @@ BASE_DTB_PROPS = ['offset', 'size', 'image-pos'] # Extra properties expected to be in the device tree when allow-repack is used REPACK_DTB_PROPS = ['orig-offset', 'orig-size'] +# Tools +LZ4 = bintool.Bintool.create('lz4') class TestFunctional(unittest.TestCase): """Functional tests for binman @@ -1967,7 +1969,7 @@ class TestFunctional(unittest.TestCase): self._ResetDtbs() def _decompress(self, data): - return comp_util.decompress(data, 'lz4') + return LZ4.decompress(data) def testCompress(self): """Test compression of blobs""" @@ -2856,7 +2858,8 @@ class TestFunctional(unittest.TestCase): def testExtractCbfsRaw(self): """Test extracting CBFS compressed data without decompressing it""" data = self._RunExtractCmd('section/cbfs/u-boot-dtb', decomp=False) - dtb = comp_util.decompress(data, 'lzma', with_header=False) + lzma_alone = bintool.Bintool.create('lzma_alone') + dtb = lzma_alone.decompress(data) self.assertEqual(EXTRACT_DTB_SIZE, len(dtb)) def testExtractBadEntry(self): @@ -4412,6 +4415,9 @@ class TestFunctional(unittest.TestCase): } self.assertEqual(expected, props) + def _compress(self, data): + return LZ4.compress(data) + def testCompressExtra(self): """Test compression of a section with no fixed size""" self._CheckLz4() @@ -4427,15 +4433,17 @@ class TestFunctional(unittest.TestCase): rest = base[len(U_BOOT_DATA):] # Check compressed data - section1 = self._decompress(rest) - expect1 = comp_util.compress(COMPRESS_DATA + U_BOOT_DATA, 'lz4') - self.assertEquals(expect1, rest[:len(expect1)]) + expect1 = self._compress(COMPRESS_DATA + U_BOOT_DATA) + data1 = rest[:len(expect1)] + self.assertEquals(expect1, data1) + section1 = self._decompress(data1) self.assertEquals(COMPRESS_DATA + U_BOOT_DATA, section1) rest1 = rest[len(expect1):] - section2 = self._decompress(rest1) - expect2 = comp_util.compress(COMPRESS_DATA + COMPRESS_DATA, 'lz4') - self.assertEquals(expect2, rest1[:len(expect2)]) + expect2 = self._compress(COMPRESS_DATA + COMPRESS_DATA) + data2 = rest1[:len(expect2)] + self.assertEquals(expect2, data2) + section2 = self._decompress(data2) self.assertEquals(COMPRESS_DATA + COMPRESS_DATA, section2) rest2 = rest1[len(expect2):] From patchwork Tue Aug 2 12:27:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Herbrechtsmeier X-Patchwork-Id: 1663001 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=weidmueller.onmicrosoft.com header.i=@weidmueller.onmicrosoft.com header.a=rsa-sha256 header.s=selector1-weidmueller-onmicrosoft-com header.b=q7Bl0tX5; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=2a01:238:438b:c500:173d:9f52:ddab:ee01; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Received: from phobos.denx.de (phobos.denx.de [IPv6:2a01:238:438b:c500:173d:9f52:ddab:ee01]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LxvSN5S7kz9s5W for ; Tue, 2 Aug 2022 22:29:24 +1000 (AEST) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id CC93F8456D; Tue, 2 Aug 2022 14:29:10 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=fail (p=none dis=none) header.from=weidmueller.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=weidmueller.onmicrosoft.com header.i=@weidmueller.onmicrosoft.com header.b="q7Bl0tX5"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 1368784547; Tue, 2 Aug 2022 14:29:07 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS autolearn=ham autolearn_force=no version=3.4.2 Received: from EUR04-HE1-obe.outbound.protection.outlook.com (mail-eopbgr70074.outbound.protection.outlook.com [40.107.7.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id CE26684560 for ; Tue, 2 Aug 2022 14:29:00 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=weidmueller.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=stefan.herbrechtsmeier-oss@weidmueller.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hjSoM2EHEU+Al5M8lbZBZMBL304PiUkYdjmMoEk25oqyyfI2KE0UqtUmIoViv8o2X06rHR1Y8E7f8VeIv1W/7zezmy/Z65wq4CZXCS9EjDb7iKrosA/eM++PWudR2PnYiiQPes4tYYgEaTecdl4MAJXgyYKNaX2Nd9rhEJmAHSC0v78KLU0hobZ71+5SwaO24zG52RPfVPS+V3RByCOahTKTcOPOT+lOruD9Zw2CTCqbkgtNAzd99EB2HE+RhN43H9jOsFajZcJnPZRudWbK3fgG8f2KEiM0ROILryB6iNnPW32/yB5RuXGpX1NsL1ZTkcDaVnD74ga68bR2m/Ae1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9ypOC/sbpXpVEE70gqLzA8MhU3sMjBZMar0zOl7yyRo=; b=fFZW9cyVW5FYR+FQJfevbSN/RbSPE9w6bk9Cj7X99if0tN+UnzAJE1NrCRya43IgwHeKJm3jT4s3W5TRsMtkMc7VfC051XU5qskMz9eiIiVzyGMN0qMAURHqkcU2Ca4i1kV/LZleL8Lu+eUIr5jIe1jW6M0I0TdSvydzsiEfydBqwlIr5Dq4YdFZU+HVUc/jt0FDUKMrlCnbnpZEw2RFuSW81NyWrRGAwRXRz4bI+kofwZVatLYl5XgxQON2N15ipEyEnl+4lTN4r23DuW9iM/trtXc+rJLS/RnpKpuTcDWZrCipwRZTj7iy5sRMA4b3DurRHoCKo4cLTUcRo43qaA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.onmicrosoft.com; s=selector1-weidmueller-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9ypOC/sbpXpVEE70gqLzA8MhU3sMjBZMar0zOl7yyRo=; b=q7Bl0tX5sbstDEpcZupKrUctzQGPj51i/b7wbB5ftoozWAkfHgHcC2f3e1vkyQmcjMAEVJ12Yk2XI6SN2+qkLMowho9XGPJhWXdLB0+/cLNUf4JgEXgx3WmHLQEO8gOpkd0l81Mg6amnmuriMMRzrvNwFK/cj5YGZNxf5vVIXNU= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by VI1PR08MB4447.eurprd08.prod.outlook.com (2603:10a6:803:f3::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5482.12; Tue, 2 Aug 2022 12:28:57 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::49a6:e5ff:7f94:8787]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::49a6:e5ff:7f94:8787%7]) with mapi id 15.20.5482.016; Tue, 2 Aug 2022 12:28:57 +0000 From: Stefan Herbrechtsmeier To: u-boot@lists.denx.de Cc: Stefan Herbrechtsmeier , Alper Nebi Yasak , Simon Glass Subject: [PATCH 2/3] binman: Remove header from compressed data Date: Tue, 2 Aug 2022 14:27:57 +0200 Message-Id: <20220802122758.20155-2-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220802122758.20155-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20220802122758.20155-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0073.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:1e::13) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ad632ff0-13cd-4868-8401-08da7482925d X-MS-TrafficTypeDiagnostic: VI1PR08MB4447:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: RMyZ5uLTtzwVgGPqBq3iglhDK1RyoevwpUSVIYtXyi1i0kC/FVOxkG7IZqerXZDbwdX6Y84TPC+gxThHHQ0VRtNKn2r5XQ2eI+EBxvwTTtvuzM1hKpjTUpbn5buCyL6P09rrc3HYuIimQG1xJN5mqio37Arit6TGj5D3JD+6qIXCCF/XD2pstYnJcmq0PDoQUylxyY7Spb14az7HhUGWXJbLjG9ab0kiGqv5+rOQ3WYapuk/7SvSqxYAbw7715ezYQ0wwAraAxr4QZTU8hW6oJ3Nq5Z7pN3hjWRtRP5K/3I/uEV2YT3POrZuZ0NtGV5qbB+7Cx5S4VI0pFQq89e5R/Rgf/INLAOysQe9ILvE2jUhlYjjRV/QMfU4RPphDJqhfvtzImq6pygXIKVtDo7uUz4PezbHToN6hQBQmWM6y1fzCd8nhlQgBlvR63wL5Pc6Tf/il65l5PhG2olMj0/jOknDgYhSg3CcHTnSiHv30fWYmxzj9AU00NczgULY+hE8Ms+sTOb+yHn/vL3Hh+SNvWSaqbrEvv4NoJkscpww5MQSdyEZLzyogR4tzDgoF+0MxqZuFh10TzDPSstWxpiCzrMtEfb6gHzIhQr+TTTMMbORz5O87EmAkXYBanQBLhqIcIDpoCWmSn0s9lymLWJF0oZqmu8AdZgDq96qQQTriXE49u0NjB9Nv/7iW26YEUsNcPzwQvem3PLUyCZJ/XzOFlrQNDZDqNeEUHzkDgcMlSzzMVeXeRYJI2fefDZHsZuleh5If2yXfooCNDrOz3Iv9GSGvvr2KOWpoORkuRvhH7Y= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PAXPR08MB6969.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230016)(4636009)(376002)(39860400002)(346002)(366004)(136003)(396003)(1076003)(186003)(86362001)(2616005)(83380400001)(8936002)(5660300002)(8676002)(66556008)(66946007)(66476007)(4326008)(36756003)(2906002)(38100700002)(6486002)(41300700001)(478600001)(6916009)(316002)(6506007)(6512007)(26005)(52116002)(54906003)(38350700002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Ro7Oq3BbQnahjLfsh1qKzCDfOpQeL9ts4IvZdImBv0zt9YtuRHVR7yzHsx9gB2KhKU9EvG/CiLMBHdsM1Ix8S50HSLz+93ML8pyq1okwT4KXQf9FjnSenWgShU7mLEZB5b1md9X1dqYkxXkOthS5MUizgPwVUWbSaDvNUcEMdJZUB9wl9qjSHvXKgw3mfbJE6+R0PQHSGEngA0MkBDbsw6qae34/Dc7QcPPnRdYKOa2/ZuUVAtL6Ub9/gxfQ8ssF/5CejacgQQ5Z1YzpUIbIqqlYRHDsG2DUvmfEwSWSVD1xj2kCLA062B+cgDRLWKdeRky++TnZvK/EpWEnkw3boZWDfIqweWUuvjwJam15PCzm5gVsCRp1agYTF8Lq/posxnkMzjgvS5sLhvRlkWpyLbAUaRU+joi48VHRWAs83qDV6bifW9qSN5vhd9+9hvxUhzHP5CPUPDXdAcv81yd5WCit5NOAoDsN8qA6bqCzhyB6TzcWAdW39QKkO6cm1tkGIb1WhJ6uo5nAXGFhFmfwGSYDSzbJ3wdKhuv1a475sbyER653isDRW2mXdOAtXbKhTjy/FnSPTfTNbkwk3ucLXxYH9728uUXAFxWYY8uYzqDIR2MMSesysc0uQV+HwXrFGZAmqKW/4diG2OEaC/JGxeKh64IkcQV05Szr7vUKye9BCIB8kdMnJG04wZlV6hEjrwZCQ2fSxiVALHjXpjDYoCwEaqbb7ybPShShiulfhsAsm1intte5upFmlWV0uRYpgRzSdewnyUrrLAnxCC45ImK9PqNbBVITia6eIVYVb5FMGvLUwH3mpgaHQozOk/He+bhrc/uEZC7L81cdGva9hg2z77WRBTl45eQM2YqTN0zyO72KqcXCKRTG1yF+ekHkrF8fcCD5AirmzkkzPduGEqfzIo6ukBMx3R0MztkmYu5fiD8Dg5QxncLv7Kj6sVGbZwT89oKACnqUWzrRb48TLqHxnGJixsx2yj01NvodQSBckxNxbSzWkXJFu4wAZRcEzkIHMv0s8YCWSoUFCbUlw/SR42AJy61TNE1NPuSzS+pdGHMjGzodxTsqa7y88tVIReTs/NvNkFZm55rTwjokVjpTsYfx00BDz9PPfHG2y1ocRCiWg38FIjHqc52IRyKvDfnAIDacHSxcz6veN8gdgSAOe536+sCxR5PP3IMFW5ZUXdi7OT6fcHmd/Lp7QTFKSNF+Bv10443lZtNdpHufncLMNVffYY4LsYg/4ckOZ6bTZzFUqgIqtylQzmNUSlxSfONptk2mETlDu5ip7m7mF94WFAk9NSZ7wIHsP+URumoQZQN/i7Z+Uv9trnNOnteOik///xcowIAPOWiifdTfQtQVtPxmrrhusRU88afyN9hi05K6H1E+C+CmsmmwB8i20gsLo/P9bhALhlFul6WbTSPzir/G9tgrfdcs3T0qJI36CSaUIZcO4yhAZC5hZ+GynqScMSTUpL1U1j4NfRxGQ19u79NvindhZnM02tmNzRXtqYGw67pTFWwVJNGNbbS/n7FYZZkqzWdQTtR+NnZFKN1otF+2MVBnzfJPI0vHVHByR4syhsB/bVOYXD6HNcPKztcHnp9zCZKikRHyxK5N7w== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: ad632ff0-13cd-4868-8401-08da7482925d X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Aug 2022 12:28:57.7612 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: NJAYm0ODfYS+CjCowQXbi50CIFrOf/RfOzCFqBfMqeabqq+c11HELd18JbwiRUHnLDZMdbwNqLmyKMYWrtU/mg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR08MB4447 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.6 at phobos.denx.de X-Virus-Status: Clean From: Stefan Herbrechtsmeier Remove header from compressed data because this is uncommon, not supported by U-Boot and incompatible with external compressed artifacts. The header was introduced as part of commit eb0f4a4cb402 ("binman: Support replacing data in a cbfs") to allow device tree entries to be larger that the compressed contents. Regarding the commit "this is necessary to cope with a compressed device tree being updated in such a way that it shrinks after the entry size is already set (an obscure case)". This case need to be fixed without influence the compressed data by itself. Signed-off-by: Stefan Herbrechtsmeier --- tools/binman/cbfs_util.py | 8 ++++---- tools/binman/comp_util.py | 11 ++--------- tools/binman/entry.py | 2 +- 3 files changed, 7 insertions(+), 14 deletions(-) diff --git a/tools/binman/cbfs_util.py b/tools/binman/cbfs_util.py index 9cad03886f..a1836f4ad3 100644 --- a/tools/binman/cbfs_util.py +++ b/tools/binman/cbfs_util.py @@ -241,9 +241,9 @@ class CbfsFile(object): """Handle decompressing data if necessary""" indata = self.data if self.compress == COMPRESS_LZ4: - data = comp_util.decompress(indata, 'lz4', with_header=False) + data = comp_util.decompress(indata, 'lz4') elif self.compress == COMPRESS_LZMA: - data = comp_util.decompress(indata, 'lzma', with_header=False) + data = comp_util.decompress(indata, 'lzma') else: data = indata self.memlen = len(data) @@ -362,9 +362,9 @@ class CbfsFile(object): elif self.ftype == TYPE_RAW: orig_data = data if self.compress == COMPRESS_LZ4: - data = comp_util.compress(orig_data, 'lz4', with_header=False) + data = comp_util.compress(orig_data, 'lz4') elif self.compress == COMPRESS_LZMA: - data = comp_util.compress(orig_data, 'lzma', with_header=False) + data = comp_util.compress(orig_data, 'lzma') self.memlen = len(orig_data) self.data_len = len(data) attr = struct.pack(ATTR_COMPRESSION_FORMAT, diff --git a/tools/binman/comp_util.py b/tools/binman/comp_util.py index dc76adab35..269bbf7975 100644 --- a/tools/binman/comp_util.py +++ b/tools/binman/comp_util.py @@ -3,7 +3,6 @@ # """Utilities to compress and decompress data""" -import struct import tempfile from binman import bintool @@ -16,7 +15,7 @@ LZMA_ALONE = bintool.Bintool.create('lzma_alone') HAVE_LZMA_ALONE = LZMA_ALONE.is_present() -def compress(indata, algo, with_header=True): +def compress(indata, algo): """Compress some data using a given algorithm Note that for lzma this uses an old version of the algorithm, not that @@ -41,12 +40,9 @@ def compress(indata, algo, with_header=True): data = LZMA_ALONE.compress(indata) else: raise ValueError("Unknown algorithm '%s'" % algo) - if with_header: - hdr = struct.pack(' X-Patchwork-Id: 1663002 X-Patchwork-Delegate: sjg@chromium.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=weidmueller.onmicrosoft.com header.i=@weidmueller.onmicrosoft.com header.a=rsa-sha256 header.s=selector1-weidmueller-onmicrosoft-com header.b=XxMsomkQ; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.denx.de (client-ip=85.214.62.61; helo=phobos.denx.de; envelope-from=u-boot-bounces@lists.denx.de; receiver=) Received: from phobos.denx.de (phobos.denx.de [85.214.62.61]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4LxvSd3lxLz9s5W for ; Tue, 2 Aug 2022 22:29:37 +1000 (AEST) Received: from h2850616.stratoserver.net (localhost [IPv6:::1]) by phobos.denx.de (Postfix) with ESMTP id 8FFD18457C; Tue, 2 Aug 2022 14:29:13 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=fail (p=none dis=none) header.from=weidmueller.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=u-boot-bounces@lists.denx.de Authentication-Results: phobos.denx.de; dkim=pass (1024-bit key; unprotected) header.d=weidmueller.onmicrosoft.com header.i=@weidmueller.onmicrosoft.com header.b="XxMsomkQ"; dkim-atps=neutral Received: by phobos.denx.de (Postfix, from userid 109) id 3762F84565; Tue, 2 Aug 2022 14:29:08 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on phobos.denx.de X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS autolearn=ham autolearn_force=no version=3.4.2 Received: from EUR04-HE1-obe.outbound.protection.outlook.com (mail-eopbgr70074.outbound.protection.outlook.com [40.107.7.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by phobos.denx.de (Postfix) with ESMTPS id 0967484564 for ; Tue, 2 Aug 2022 14:29:01 +0200 (CEST) Authentication-Results: phobos.denx.de; dmarc=pass (p=none dis=none) header.from=weidmueller.com Authentication-Results: phobos.denx.de; spf=pass smtp.mailfrom=stefan.herbrechtsmeier-oss@weidmueller.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MRDJVvGfeklPWnhCRN/doP5qUSmrG+a/3LI0y6290cbF/PccGUCLxxk7/pBvlwEwKogtQfrdb2snIMDGpy/LziKkiB6Fw/jkRd7eKr7chFO6u/o5AmFO/b82TDe2KMlZQ852ySeaVhD+y9KRY6Hqf0AhxqHmo2/4OKCwEfBQtv6e0Rs9LsaISj4pveik4j1ZGkhZEnsd4zCHz+piK3JZX6S88synA9HPKG3/g+G0R6iScVBeZlT64XFlgjxAhI0HhXf067cEhO5gExKo6q5xutYAK/A2fqMnEk7W7FF0N/E6LlbnRuz6eg1BwmO0dl4uT+pIQGkacxUwuHc/gUsLCg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=VA/rHKw992W4RTrSQBakDqtftyyVWvTVLsj89qz30Co=; b=a7BxwliGbGVN/dvWWhvN8aC1aZZ1F91CKWI3K3/cWcLd+OqdtgyHaClbP/kYScATndCvmH6M7LbpCLRSV4Oz1porklyKZVax3UJZnV/lAsCPt+gim7TVxvIUlevG+8/3bVujUV6+c1w5f+UqPpyGHXAAzPkabdPOGWD+dA41vklIbBKJ9Cim1hbsp+35QyKPNHBNDfIPNuktRSHII1uDlreG+/5KWGD/Xp3UF6FofIJBNJ4gqtm/jlBX2lICUgOcejebgDrY+T+isgZTmnFVARQcUWCrsKAl4wbSv9qd66wguM85GcRI5vfuj7G9l6wWlSODCJKqEyCqYKrxVa+OrA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=weidmueller.com; dmarc=pass action=none header.from=weidmueller.com; dkim=pass header.d=weidmueller.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=weidmueller.onmicrosoft.com; s=selector1-weidmueller-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VA/rHKw992W4RTrSQBakDqtftyyVWvTVLsj89qz30Co=; b=XxMsomkQ19I9e/iu0yssr1zZ+/d+ZnZFCv6T4OpoaGfEWJBzovGkAnkJdqfMePYE24EiFWalLpTX0oIcN5RrUPFYo60nR/4iC5tqvSjhE9SjQ/edzBqNkgkPw56wqj6UZCvTAgqNLBHo5teiXQby1OosYJ6naTW8KG/zLkqueJw= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=weidmueller.com; Received: from PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) by VI1PR08MB4447.eurprd08.prod.outlook.com (2603:10a6:803:f3::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5482.12; Tue, 2 Aug 2022 12:28:58 +0000 Received: from PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::49a6:e5ff:7f94:8787]) by PAXPR08MB6969.eurprd08.prod.outlook.com ([fe80::49a6:e5ff:7f94:8787%7]) with mapi id 15.20.5482.016; Tue, 2 Aug 2022 12:28:58 +0000 From: Stefan Herbrechtsmeier To: u-boot@lists.denx.de Cc: Stefan Herbrechtsmeier , Alper Nebi Yasak , Simon Glass Subject: [PATCH 3/3] binman: Add gzip bintool Date: Tue, 2 Aug 2022 14:27:58 +0200 Message-Id: <20220802122758.20155-3-stefan.herbrechtsmeier-oss@weidmueller.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20220802122758.20155-1-stefan.herbrechtsmeier-oss@weidmueller.com> References: <20220802122758.20155-1-stefan.herbrechtsmeier-oss@weidmueller.com> X-ClientProxiedBy: FR0P281CA0073.DEUP281.PROD.OUTLOOK.COM (2603:10a6:d10:1e::13) To PAXPR08MB6969.eurprd08.prod.outlook.com (2603:10a6:102:1d8::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 20fd8e39-40a6-4e41-46aa-08da748292a0 X-MS-TrafficTypeDiagnostic: VI1PR08MB4447:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: bx4rLkVox/ho0qpXOtt0xScTBGbNmOvDXhABO21EqjMWPHfZ+sdUpkxHPVpFa97lMftsVlYJf8iw8IfXdUDAUxv7zksz2GcgttGsWS4vRwYqhzKDfuCKmgQ7AWXn4B+4Ovd4YNpugHI7+vmd2g4qgkqX2RkbEmRhrBjK25NAwvDxLDA8WG6Xo+Gsev4sscqzjAo/QXcVDvMK8qn+P/jKntk0f6OzDijRjLvKL93qCAJYG1FEXb6qxqgrEr52U4AYyTPeEwSqK5pOQTPfrqbYypkKfQfVDxeohqSclzz1CkBCmg2mivo5LlySNDBkBzgiIelBkfhxnQLl0X1oNqAfD1gy6w/OaEWj+ShWYN5e+1ZG1XXl/fKqX78PWR+KK9qC8Hc0dWr4f+w5NCmwbVR7nHZcIUaJUNwbbhsWMZkBR+Y5NBSpZ+UdfGYSMm8zYDNBAjIGHxwI4wxr/IgM9E1eQ2kuT/I0JCVbw73cdM5EqxGKwipPyNMcR/PZ8gQ7Sem2TKGRSmFjdqkvT70FimpUm+NPc3izJp3+L7HndN2QEEByzkMhO8o82IbKbm1XNw+KJ42mIjRaaX+qq28hztPPMf8FjPaXkqtuFiunyhXNdJQdTiiv7S+anp2CYqudLpwwNJxQh66IyYuWzgLz138cLvQDojd7SYNyEI3ZGVU3gkdBWC+MbjL+af1MqbNRdlXv7dAmH7eRAZC5w9+SC3KGafvl7WW0HaBZyQpOdyywQJsaqJ7Ogf06kGTOIaLKHNo5qu3DUK5T+UK+bngj2Qw19yjfLJ8gzt/g2p2By/fqxWc= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:PAXPR08MB6969.eurprd08.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230016)(4636009)(376002)(39860400002)(346002)(366004)(136003)(396003)(1076003)(186003)(86362001)(2616005)(83380400001)(8936002)(5660300002)(8676002)(66556008)(66946007)(66476007)(4326008)(36756003)(2906002)(38100700002)(6486002)(41300700001)(478600001)(6916009)(316002)(6506007)(6512007)(26005)(52116002)(54906003)(38350700002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: TX/+/U2o2jssVEGA3/D7IKKsuje0M0ifeFXkN37zNqhuRNED2h5C8v8CreSM+cPnHZFhGZSaSGD0L/KT4VPGb8on21lL0oXUynigmk7U7YjyDgww/jm5nxZKlwF6z+peSmcBse8jUohQHh9STfTWDH5fwvEpmmFNGVeFXGh+OmkKxhgQErmTHlKRlym+dFhhO0LLuE4ZAZ4jNU2+ZfYAT0ijQsOT4qpy6ub9ni+ceKSCPIYS3t86q0kcPVwDOCWs173HU4+RbqOYwEiZqVbDZWhiRfjOndrEaMzxobdw7uXjtb/MIT9oA9MR8B3JoME9E36ossviLvmKGBcAWx78bA1zardyIeu451EcsRT6G1FHmnhHxTO+6aLRnmAY/OG7S4wATpP4rI5RRYBqp6/r/Jts1UURfJwJiyI32Nt9T4VpvB+ZW6NqkP6xKehgEHCtdmOprHdfgIxqc0btM6RYFZy1s18u0iOnnZH6B6yPNXdWgV5/KjOfxC7ClufEEH9FVOok85p37DvgsvRnyGg4944UoesJ33t9Wqp539x5k25pXrCUTTqlMLjxU8pRbF0TuMuZEsZTeDyIM/EPoUgAo0W3sr92NZiboAkRej9pLn4AOjudbzRsbjGvoPGiWkfkYvZC2CqDs4Br/90WMXLUn7W3eSGe035dPGszaICk0Sd5Q5cWw8OSpruCK2NVUKlf/B9+bl00HwUNwuCFX8cijd9aB95HBecVHbzZIXMx/NLRHqAvHDuCRYjqqzS25esm0xMlMIJzLnw55HbNHiLAfEoOYYBtO+N7ou0d8YCdaEIcRARZ5Q/2eOwxuv1GcXQqairIxVhg2lc2HjUxw66qJoS1A9sk/V4DVRcI0nAXpA6qSbkuBDGrrld1uNPlhzfyfswS42sqfLVByW+Df8UOUi9ka2DIoZFLlki0xamxYGyhhRl0ghjNINMsbLbTpcQ5LiDhQ296fTnnBT4JFh0dwPyT+loBa1HKVr0Y5LS25fGT1gnJl0aot2xaI+8aTCU/PAA3f/xC/msXWSQKAmtqDcTX2uldM6I088NeB8eXh6Jav2TzNlCYKkLaCSD8he99iWVRwWxrKeBiVmEagQehft1IPRRiAG6ZZM9SIetYPuyb4ZPQxC5iD6m/w73vVRi7GFRWSukHQyYi1O+IvQmC635NWsYHCzi9CN/CNYIfXoPGRaa86QMTlkNoRoD4vrkECRDJVr4bz/922TtjrmNU2fKY4j7FANBK/sGajFiZhMyP5NexWNliMdnrnidiKIO6yHlklr6JnKixzh35hLopk08Fk4pUmQdcanRH9PNo8Zheb5PSwJfIYokPk874Kdo78U8PfNng1fg+qD80+LhF5OI9RomrRzVJPCjOcFmP4dmYL64r2QwKhdJgLTeuPjYNzdcB5Fj+4/yIB2NH2mA9LX/a8Vty1MvCObFd4n1iWmBhNNIE3kQWv8AEt3YFvb96wPuODZLURd1eBHtzr5leMpXu41ogC3CCJMbIubcfcNkIsGA/BlKJhTRM0nlmonK4LB2FxsGKht06WuxxhF/ufaEvuPI98sllhpwpGIE+S8BVLG+oh6GyQgOLJRgqRf6P7FLI2IfpRzBIPBZj29oNfg== X-OriginatorOrg: weidmueller.com X-MS-Exchange-CrossTenant-Network-Message-Id: 20fd8e39-40a6-4e41-46aa-08da748292a0 X-MS-Exchange-CrossTenant-AuthSource: PAXPR08MB6969.eurprd08.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Aug 2022 12:28:58.2143 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: e4289438-1c5f-4c95-a51a-ee553b8b18ec X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 04HJjRWblFw5muG8fiZPbqpWhunJyAvWcLIwQ0xNwt8d/6K5+xG5K+/la0DT/TXjz8A9LcqZ6MWe+vVJi/1wWw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR08MB4447 X-BeenThere: u-boot@lists.denx.de X-Mailman-Version: 2.1.39 Precedence: list List-Id: U-Boot discussion List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: u-boot-bounces@lists.denx.de Sender: "U-Boot" X-Virus-Scanned: clamav-milter 0.103.6 at phobos.denx.de X-Virus-Status: Clean From: Stefan Herbrechtsmeier Add gzip bintool to binman to support on-the-fly compression of Linux kernel images and FPGA bitstreams. The SPL basic fitImage implementation supports only gzip decompression. Signed-off-by: Stefan Herbrechtsmeier Reviewed-by: Simon Glass --- tools/binman/btool/gzip.py | 94 ++++++++++++++++++++++++++++++++++++++ tools/binman/comp_util.py | 17 +++++-- tools/binman/ftest.py | 18 ++++++++ 3 files changed, 124 insertions(+), 5 deletions(-) create mode 100644 tools/binman/btool/gzip.py diff --git a/tools/binman/btool/gzip.py b/tools/binman/btool/gzip.py new file mode 100644 index 0000000000..27fb654ed5 --- /dev/null +++ b/tools/binman/btool/gzip.py @@ -0,0 +1,94 @@ +# SPDX-License-Identifier: GPL-2.0+ +# Copyright 2022 Google LLC +# +"""Bintool implementation for gzip + +gzip allows compression and decompression of files. + +Documentation is available via:: + + man gzip +""" + +import re +import tempfile + +from binman import bintool +from patman import tools + +# pylint: disable=C0103 +class Bintoolgzip(bintool.Bintool): + """Compression/decompression using the gzip algorithm + + This bintool supports running `gzip` to compress and decompress data, as + used by binman. + + It is also possible to fetch the tool, which uses `apt` to install it. + + Documentation is available via:: + + man gzip + """ + def __init__(self, name): + super().__init__(name, 'gzip compression') + + def compress(self, indata): + """Compress data with gzip + + Args: + indata (bytes): Data to compress + + Returns: + bytes: Compressed data + """ + with tempfile.NamedTemporaryFile(prefix='comp.tmp', + dir=tools.get_output_dir()) as tmp: + tools.write_file(tmp.name, indata) + args = ['-c', tmp.name] + return self.run_cmd(*args, binary=True) + + def decompress(self, indata): + """Decompress data with gzip + + Args: + indata (bytes): Data to decompress + + Returns: + bytes: Decompressed data + """ + with tempfile.NamedTemporaryFile(prefix='decomp.tmp', + dir=tools.get_output_dir()) as inf: + tools.write_file(inf.name, indata) + args = ['-cd', inf.name] + return self.run_cmd(*args, binary=True) + + def fetch(self, method): + """Fetch handler for gzip + + This installs the gzip package using the apt utility. + + Args: + method (FETCH_...): Method to use + + Returns: + True if the file was fetched and now installed, None if a method + other than FETCH_BIN was requested + + Raises: + Valuerror: Fetching could not be completed + """ + if method != bintool.FETCH_BIN: + return None + return self.apt_install('gzip') + + def version(self): + """Version handler + + Returns: + str: Version number of gzip + """ + out = self.run_cmd('-V').strip() + if not out: + return super().version() + m_version = re.match(r'.*gzip ([0-9.]*).*', out) + return m_version.group(1) if m_version else out diff --git a/tools/binman/comp_util.py b/tools/binman/comp_util.py index 269bbf7975..1d30f1613c 100644 --- a/tools/binman/comp_util.py +++ b/tools/binman/comp_util.py @@ -8,6 +8,9 @@ import tempfile from binman import bintool from patman import tools +GZIP = bintool.Bintool.create('gzip') +HAVE_GZIP = GZIP.is_present() + LZ4 = bintool.Bintool.create('lz4') HAVE_LZ4 = LZ4.is_present() @@ -21,19 +24,21 @@ def compress(indata, algo): Note that for lzma this uses an old version of the algorithm, not that provided by xz. - This requires 'lz4' and 'lzma_alone' tools. It also requires an output + This requires 'gzip', 'lz4' and 'lzma_alone' tools. It also requires an output directory to be previously set up, by calling PrepareOutputDir(). Args: indata (bytes): Input data to compress - algo (str): Algorithm to use ('none', 'lz4' or 'lzma') + algo (str): Algorithm to use ('none', 'gzip', 'lz4' or 'lzma') Returns: bytes: Compressed data """ if algo == 'none': return indata - if algo == 'lz4': + if algo == 'gzip': + data = GZIP.compress(indata) + elif algo == 'lz4': data = LZ4.compress(indata) # cbfstool uses a very old version of lzma elif algo == 'lzma': @@ -48,12 +53,12 @@ def decompress(indata, algo): Note that for lzma this uses an old version of the algorithm, not that provided by xz. - This requires 'lz4' and 'lzma_alone' tools. It also requires an output + This requires 'gzip', 'lz4' and 'lzma_alone' tools. It also requires an output directory to be previously set up, by calling PrepareOutputDir(). Args: indata (bytes): Input data to decompress - algo (str): Algorithm to use ('none', 'lz4' or 'lzma') + algo (str): Algorithm to use ('none', 'gzip', 'lz4' or 'lzma') Returns: (bytes) Compressed data @@ -62,6 +67,8 @@ def decompress(indata, algo): return indata if algo == 'lz4': data = LZ4.decompress(indata) + elif algo == 'gzip': + data = GZIP.decompress(indata) elif algo == 'lzma': data = LZMA_ALONE.decompress(indata) else: diff --git a/tools/binman/ftest.py b/tools/binman/ftest.py index b1f564ed7d..5956ffa6e4 100644 --- a/tools/binman/ftest.py +++ b/tools/binman/ftest.py @@ -5226,6 +5226,24 @@ fdt fdtmap Extract the devicetree blob from the fdtmap comp_util.decompress(b'1234', 'invalid') self.assertIn("Unknown algorithm 'invalid'", str(e.exception)) + def testGzipCompress(self): + """Test gzip compress""" + if not comp_util.HAVE_GZIP: + self.skipTest('gzip not available') + data = comp_util.compress(COMPRESS_DATA, 'gzip') + gzip = bintool.Bintool.create('gzip') + orig = gzip.decompress(data) + self.assertEquals(COMPRESS_DATA, orig) + + def testGzipDecompress(self): + """Test gzip decompress""" + if not comp_util.HAVE_GZIP: + self.skipTest('gzip not available') + gzip = bintool.Bintool.create('gzip') + data = gzip.compress(COMPRESS_DATA) + orig = comp_util.decompress(data, 'gzip') + self.assertEquals(COMPRESS_DATA, orig) + def testBintoolDocs(self): """Test for creation of bintool documentation""" with test_util.capture_sys_output() as (stdout, stderr):