From patchwork Mon Jun 3 08:56:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dominique Martinet X-Patchwork-Id: 1942765 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=googlegroups.com header.i=@googlegroups.com header.a=rsa-sha256 header.s=20230601 header.b=qSqw8zsg; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=googlegroups.com (client-ip=2607:f8b0:4864:20::53c; helo=mail-pg1-x53c.google.com; envelope-from=swupdate+bncbcwivbv7sugrbmek62zamgqemze4dqy@googlegroups.com; receiver=patchwork.ozlabs.org) Received: from mail-pg1-x53c.google.com (mail-pg1-x53c.google.com [IPv6:2607:f8b0:4864:20::53c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Vt6yr1ysJz23tB for ; Mon, 3 Jun 2024 18:56:19 +1000 (AEST) Received: by mail-pg1-x53c.google.com with SMTP id 41be03b00d2f7-5e4df21f22dsf3557488a12.0 for ; Mon, 03 Jun 2024 01:56:19 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1717404977; cv=pass; d=google.com; s=arc-20160816; b=uZqPN4LbMk2jC6YidqIwphzlbEBuBOGwk42Kvf6p+AH9eK+DzvkwUq+A19rhCgot2Z ibq8hFuT9IO/eC9+Cfy0EE/k6b7YumvS3RfXhNw5oHAM6xe4TdUx9adUxLjdN0cTJivg MKb351WJDqm3c4tW5oEU+Zw1zR1bSlGkWe28lG8GFyTXhREGvwCyt3j2mp0DwQkQisga xKsIYHaFCe/35TFZSCSysqAQOtcKdTsg8iRXuuFU2YyMt9fYsPVwJ5DXp2Z1TMYKwJit +kBq8OfHQca30S/E6RosYLMXHUFHWTRlfR2KWMC5bOGPkCeNPyEwES0/WR2BaeYDwEs6 ofcA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:sender:dkim-signature; bh=yXNH0j4ORX1ZM7z/sQoq1CRuz21S3snLkNe6Ba9P3Vg=; fh=3syp7VCE447qPFpY9bTYKNlFEUOBVo2koJ8Jj/UQCZE=; b=tK2GeRhWSTp4Q6dK3sQpMaOWwE4nzv7/rzYnEehnNYeQjSAwvPek7HT26ZEjdypdPQ hsson2ZcNL0Q/NLQAAirToNzhWKezjDSdmQQzmQ1gS0xLawNTfISOy/Bk6TRA4WJiZeX X+kXsz3Gq6xUdpT3NszIUW/UlmLi6uVvdbrn6e11D1IBfz2A/jarr4W35ilRRSF7tQMe VgiDEKD62Ze1LDqFvc2zhWo8VyOrlswdKgnAvUVkLroSS0WZfZgqPEKs4DQJU+NQrIxl 62N2Q21lEpGnpp1EIjln7ZWVF1d4UBGHr8Ch8bpu1r2oLbTwUg5pB0FP8awbVKy1OpFJ ugxA==; darn=patchwork.ozlabs.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@atmark-techno.com header.s=gw2_bookworm header.b=HPJaxCLe; dkim=pass header.i=@atmark-techno.com header.s=google header.b=ZzMp+BPl; spf=pass (google.com: domain of dominique.martinet@atmark-techno.com designates 35.74.137.57 as permitted sender) smtp.mailfrom=dominique.martinet@atmark-techno.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=atmark-techno.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1717404977; x=1718009777; darn=patchwork.ozlabs.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:references:in-reply-to:message-id :date:subject:cc:to:from:sender:from:to:cc:subject:date:message-id :reply-to; bh=yXNH0j4ORX1ZM7z/sQoq1CRuz21S3snLkNe6Ba9P3Vg=; b=qSqw8zsgVyZ+4wlrSrkFN6aEttfs/55HAg2WpTmytHy+Gz/JQAJrgX8doHPHBLVQpr RY9NhU5AiFCUt554kNE9RUPPKyEKNyLHe6XBfAnvz7ns10ZzixMdrpIsde6yNSsYISMj UidxNYrbQrSt42kokv3gGWledsw7tr4xC6L5n31k72KY90Ot7DA2BLnTW2xATaN7Gdmt seZ8NGgfY5NkkWSmUT2tUD19suztoWkaMebp6JSlK3jOhoFfPVKwvztkysylX1RSXUXr rUea50WCbP4U0N5WtmICG7/pnvNOychJLdbO0zPWeB5iJWO7K38+QpO/5GwjKfJK8zoX xYNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717404977; x=1718009777; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :x-beenthere:x-gm-message-state:sender:from:to:cc:subject:date :message-id:reply-to; bh=yXNH0j4ORX1ZM7z/sQoq1CRuz21S3snLkNe6Ba9P3Vg=; b=kjCp2sBuWNoCVv8IEdCgZMwf2dco1QNzVFRtyh8Cx7ZmFo+xiVmSfweHp7855O8gQW ZMkh6dd7dvYA912kJLF47OEOwE/WqVhpbXA42xF0LuEjeiJfGuy4Ttzgkjf6XRKzXUYP j0ELoT2BnWl5CCBHRyozdCVEng0W5kRLcBqSQ5j5O25xwSeZRZheNGzJSBAjN9Tf6N2f ubuU5hUz2sxa2ReCqAZsGWd/6J9Cc0230/+9jM862E4ei18ON+deaBNBBd2e2jw8z6ua PZZX701FCegFSpdx5UOX1vfrYp2erMxp2WIh/xJs4h6yLp1GSbpAg2XA2U8zU9TQjkar Uiqg== Sender: swupdate@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCWS4iMOUKgynd6Sgsw6d4QEpHyLts+J2hLmWMDUNYy+A2DT2HRePLPcQxhlLeW8XM5xpaE712nEJE8M73RGbAmkq/ThoGOf/Q4q4GQ7NQ== X-Gm-Message-State: AOJu0YxByKsRAFHK3mMJOgpsDeSHXsgme0CeaWT789BgiaKWp1niZCLv 6r9SYlheCodS0LO4QATVkDbY1WeXeNKoh7HT0vnBrONuRJvgD920 X-Google-Smtp-Source: AGHT+IENtVjLaIlWlQQzuNBJuDP12w/K4sUlvQPBsMMq3dVZiROkZbkUtuaDOk6t++yQ0P2bjXbPOg== X-Received: by 2002:a17:903:32c6:b0:1f6:1a86:37ba with SMTP id d9443c01a7336-1f6359933b6mr109036545ad.2.1717404977062; Mon, 03 Jun 2024 01:56:17 -0700 (PDT) X-BeenThere: swupdate@googlegroups.com Received: by 2002:a17:903:2309:b0:1f6:f27:5506 with SMTP id d9443c01a7336-1f61747e580ls4093315ad.2.-pod-prod-00-us; Mon, 03 Jun 2024 01:56:15 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCXFLNS1j3IjodJdEWCLg/G1AHW9QIbVeuiWorXHia6ckt9Ca6oEhtqTlp65CkdirrqNeAVpkzSe5ZbDMVr2jGqnF86+9QV/drFK X-Received: by 2002:a17:902:e846:b0:1f2:fe12:b7be with SMTP id d9443c01a7336-1f61c1b7446mr167605175ad.32.1717404974919; Mon, 03 Jun 2024 01:56:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1717404974; cv=none; d=google.com; s=arc-20160816; b=oDP2bVlXwHHlktVaks+GLkRq/CbgMNZ/L7ZFxNw3vzpwdBh6VmLttO36Qtfy3X7cQn M90KLsm7ZCo0jlMkf+6ZsVMZOIhF33PfJh9PZokNLMCiu3kWAPAvOsKdtB7++YZgy7Bg ErNjXkO2Bwj+J7OQ5Z8hyP4owdtHxl9IpjEffayDzUXo+Afy3RtvUAfhtHJOpaf4BPih zohQOAuNRk04gV8eyxbKut4+58rfb7+uah6VBO/xbVXS2PAZhXZY0Mt3Rv1O4X7XR7Pq G2bpcjOHWn04DMBMm+cEqwvD+ifCFn3pqfYnTi1GjAsOR4/Mie8sDRsAnDq30eXQaT+I hT+A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:dkim-signature:dkim-signature; bh=qxzLKZ0mD5OgWM1wDLPIg6xEzDzOc28sI4dcdD4+Z28=; fh=TQEJJmfLa3DlMeSwidtgAUEu34qHDcoVqFsXZ7kK7Hc=; b=QPC0VbdKuUM2jGmOJi5UaBotia0rk6pv/f0nsDJJuj92cidYU2w32PSfbyH/14C0Tv JJLZm/g+BQkRayWZIM4YsiMSjsl8LtmoyHpJxp4vsk7LraUZ10yrTHQOP7EvbNqZSQnw HlcgYSvmeToaUHYogqUCZCN7TH5WsX4R8hpYRi5gTWtUsRKQUjmqLv8mrVPzvhGnEp5A klSc0I7CNQUklNgfljRt3ZS0rALIFNg2YW99JacZhrL5Hva8A8up1ZqsWMnUVUaZJWEF ex1QsDEzCNjIiMxsYHp9fGROQRL7a7hT/09TgUd55l72o9eHBKXhPZRALYH5prP11qqy C2XA==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@atmark-techno.com header.s=gw2_bookworm header.b=HPJaxCLe; dkim=pass header.i=@atmark-techno.com header.s=google header.b=ZzMp+BPl; spf=pass (google.com: domain of dominique.martinet@atmark-techno.com designates 35.74.137.57 as permitted sender) smtp.mailfrom=dominique.martinet@atmark-techno.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=atmark-techno.com Received: from gw2.atmark-techno.com (gw2.atmark-techno.com. [35.74.137.57]) by gmr-mx.google.com with ESMTPS id d9443c01a7336-1f632361d1fsi2905355ad.5.2024.06.03.01.56.14 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Jun 2024 01:56:14 -0700 (PDT) Received-SPF: pass (google.com: domain of dominique.martinet@atmark-techno.com designates 35.74.137.57 as permitted sender) client-ip=35.74.137.57; Received: from gw2.atmark-techno.com (localhost [127.0.0.1]) by gw2.atmark-techno.com (Postfix) with ESMTP id CDAB5283 for ; Mon, 3 Jun 2024 17:56:12 +0900 (JST) Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) by gw2.atmark-techno.com (Postfix) with ESMTPS id 99E9AA80 for ; Mon, 3 Jun 2024 17:56:10 +0900 (JST) Received: by mail-pf1-f200.google.com with SMTP id d2e1a72fcca58-7025fafc37eso950781b3a.0 for ; Mon, 03 Jun 2024 01:56:10 -0700 (PDT) X-Forwarded-Encrypted: i=1; AJvYcCUj8ah/la5lw8eiwLaGtmPFqx95x5Ycs9wHN+k0wwnB5aKK2oe/Hi8y2BIXtsQF80K5dL3xXF9+nPfnuPagw8cMM3YBYZ1uJIgr X-Received: by 2002:a05:6a21:199:b0:1af:8cc6:abc7 with SMTP id adf61e73a8af0-1b26f17ca5dmr12656841637.15.1717404969633; Mon, 03 Jun 2024 01:56:09 -0700 (PDT) X-Received: by 2002:a05:6a21:199:b0:1af:8cc6:abc7 with SMTP id adf61e73a8af0-1b26f17ca5dmr12656821637.15.1717404969231; Mon, 03 Jun 2024 01:56:09 -0700 (PDT) Received: from pc-0182.atmarktech (117.209.187.35.bc.googleusercontent.com. [35.187.209.117]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-6c35a6a3679sm4917270a12.81.2024.06.03.01.56.08 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 03 Jun 2024 01:56:08 -0700 (PDT) Received: from [::1] (helo=pc-0182.atmark.tech) by pc-0182.atmarktech with esmtp (Exim 4.96) (envelope-from ) id 1sE3U6-009riT-1O; Mon, 03 Jun 2024 17:56:06 +0900 From: Dominique Martinet To: stefano.babic@swupdate.org, swupdate@googlegroups.com Cc: Dominique Martinet Subject: [swupdate] [PATCH RFC 7/9] cpio_utils: implement chunked hash verification Date: Mon, 3 Jun 2024 17:56:00 +0900 Message-Id: <20240603085602.2351411-8-dominique.martinet@atmark-techno.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240603085602.2351411-1-dominique.martinet@atmark-techno.com> References: <20240603085602.2351411-1-dominique.martinet@atmark-techno.com> MIME-Version: 1.0 X-Original-Sender: dominique.martinet@atmark-techno.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@atmark-techno.com header.s=gw2_bookworm header.b=HPJaxCLe; dkim=pass header.i=@atmark-techno.com header.s=google header.b=ZzMp+BPl; spf=pass (google.com: domain of dominique.martinet@atmark-techno.com designates 35.74.137.57 as permitted sender) smtp.mailfrom=dominique.martinet@atmark-techno.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=atmark-techno.com Precedence: list Mailing-list: list swupdate@googlegroups.com; contact swupdate+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: swupdate@googlegroups.com X-Google-Group-Id: 605343134186 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , This is not used yet as nothing sets copyfile's chunk_hashes yet. Signed-off-by: Dominique Martinet --- Kconfig | 4 ++ core/cpio_utils.c | 130 ++++++++++++++++++++++++++++++++++++++++++++++ include/util.h | 2 + 3 files changed, 136 insertions(+) diff --git a/Kconfig b/Kconfig index 7139617c4515..f1dbe3049676 100644 --- a/Kconfig +++ b/Kconfig @@ -522,6 +522,10 @@ config PKCS11 key available in a file. This is implemented with wolfSSL independent from the SSL implementation and replaces the plain key method. +config CHUNKED_HASH + bool "support for chuked checksums input" + default y + comment "Compressors (zlib always on)" config GUNZIP diff --git a/core/cpio_utils.c b/core/cpio_utils.c index 382b22c36434..d296fe8d4525 100644 --- a/core/cpio_utils.c +++ b/core/cpio_utils.c @@ -19,6 +19,7 @@ #ifdef CONFIG_ZSTD #include #endif +#include #include "generated/autoconf.h" #include "cpiohdr.h" @@ -466,6 +467,79 @@ static int hash_compare(struct swupdate_digest *dgst, unsigned char *hash) return 0; } +#ifdef CONFIG_CHUNKED_HASH +struct ChunkedHashState { + PipelineStep upstream_step; + void *upstream_state; + + /* null-terminated array of checksums */ + unsigned char *chunked_hashes; + + /* mmaped buffer */ + void *chunk_data; + size_t chunk_size; + size_t valid; + + /* amount written */ + size_t offset; +}; + + +static int chunked_hash_step(void *_state, void *buffer, size_t size) +{ + struct ChunkedHashState *state = (struct ChunkedHashState *)_state; + int count = 0, n; + + while (count < size) { + if (state->offset == state->valid) { + /* input_step does full reads */ + if (state->valid && state->valid < state->chunk_size) + return count; + + if (!IsValidHash(state->chunked_hashes)) { + ERROR("No hash left for chunk -- file too large?"); + return -EFAULT; + } + + n = state->upstream_step(state->upstream_state, state->chunk_data, state->chunk_size); + if (n <= 0) { + return n; + } + + /* + * possible improvement: in theory we can re-init a given buffer after final(), + * but swupdate_HASH_* does not expose this API at this point. + * Realloc every time. + */ + void *dgst = swupdate_HASH_init(SHA_DEFAULT); + if (!dgst) + return -EFAULT; + + if (swupdate_HASH_update(dgst, state->chunk_data, n) < 0 || + hash_compare(dgst, state->chunked_hashes) < 0) { + swupdate_HASH_cleanup(dgst); + return -EFAULT; + } + swupdate_HASH_cleanup(dgst); + + state->chunked_hashes += SHA256_HASH_LENGTH; + state->valid = n; + state->offset = 0; + } + + n = state->valid - state->offset; + if (n > size) + n = size; + memcpy(buffer + count, state->chunk_data + state->offset, n); + count += n; + size -= n; + state->offset += n; + } + + return count; +} +#endif + int copyfile(struct swupdate_copy *args) { unsigned int percent, prevpercent = 0; @@ -492,6 +566,10 @@ int copyfile(struct swupdate_copy *args) .outlen = 0, .eof = false }; +#ifdef CONFIG_CHUNKED_HASH + struct ChunkedHashState chunked_hash_state = { 0 }; +#endif + #if defined(CONFIG_GUNZIP) || defined(CONFIG_ZSTD) struct DecompressState decompress_state = { .upstream_step = NULL, .upstream_state = NULL, @@ -619,6 +697,52 @@ int copyfile(struct swupdate_copy *args) step = &input_step; state = &input_state; +#ifdef CONFIG_CHUNKED_HASH + if (args->chunked_hashes) { + char tmpfilename[MAX_IMAGE_FNAME]; + int tmpfd; + + /* this requires a large-ish buffer for chunk size: use a temporary file mmaped */ + /* XXX make configurable */ + chunked_hash_state.chunk_size = 512*1024; + snprintf(tmpfilename, sizeof(tmpfilename), "%s/swtmp-datachunkXXXXXX", get_tmpdir()); + tmpfd = mkstemp(tmpfilename); + if (tmpfd < 0) { + ERROR("Could not open temporary fd, error %d", errno); + ret = -EFAULT; + goto copyfile_exit; + } + unlink(tmpfilename); + + if (ftruncate(tmpfd, chunked_hash_state.chunk_size) < 0) { + ERROR("Could not open temporary fd, error %d", errno); + ret = -EFAULT; + close(tmpfd); + goto copyfile_exit; + } + + chunked_hash_state.chunk_data = + mmap(NULL, chunked_hash_state.chunk_size, PROT_READ|PROT_WRITE, + MAP_SHARED, tmpfd, 0); + + if (chunked_hash_state.chunk_data == MAP_FAILED) { + ERROR("Could not mmap temporary data, error %d", errno); + ret = -EFAULT; + close(tmpfd); + goto copyfile_exit; + } + + /* close immediately, mmap stays valid until munmap */ + close(tmpfd); + chunked_hash_state.chunked_hashes = args->chunked_hashes; + + chunked_hash_state.upstream_step = step; + chunked_hash_state.upstream_state = state; + step = &chunked_hash_step; + state = &chunked_hash_state; + } +#endif + if (args->encrypted) { decrypt_state.upstream_step = step; decrypt_state.upstream_state = state; @@ -687,6 +811,12 @@ copyfile_exit: if (decrypt_state.dcrypt) { swupdate_DECRYPT_cleanup(decrypt_state.dcrypt); } +#ifdef CONFIG_CHUNKED_HASH + if (args->chunked_hashes) { + if (chunked_hash_state.chunk_data) + munmap(chunked_hash_state.chunk_data, chunked_hash_state.chunk_size); + } +#endif if (input_state.dgst) { swupdate_HASH_cleanup(input_state.dgst); } diff --git a/include/util.h b/include/util.h index 1703514269f1..13b5d0df6414 100644 --- a/include/util.h +++ b/include/util.h @@ -87,6 +87,8 @@ struct swupdate_copy { uint32_t *checksum; /* sw-description sha256 checksum */ unsigned char *hash; + /* juxtaposed sha256 checksums */ + unsigned char *chunked_hashes; /* encryption */ bool encrypted; const char *imgivt;