From patchwork Mon Jun 3 08:55:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dominique Martinet X-Patchwork-Id: 1942757 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=googlegroups.com header.i=@googlegroups.com header.a=rsa-sha256 header.s=20230601 header.b=GtliULbQ; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=googlegroups.com (client-ip=2607:f8b0:4864:20::63c; helo=mail-pl1-x63c.google.com; envelope-from=swupdate+bncbcwivbv7sugrblek62zamgqeolea5ci@googlegroups.com; receiver=patchwork.ozlabs.org) Received: from mail-pl1-x63c.google.com (mail-pl1-x63c.google.com [IPv6:2607:f8b0:4864:20::63c]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Vt6ym6dGtz20Pb for ; Mon, 3 Jun 2024 18:56:15 +1000 (AEST) Received: by mail-pl1-x63c.google.com with SMTP id d9443c01a7336-1f682e70799sf3129645ad.1 for ; Mon, 03 Jun 2024 01:56:15 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1717404973; cv=pass; d=google.com; s=arc-20160816; b=e/ZC2vhKBI5nh5NPAxU8vT1c/dhjvXhD5U8UQnnMCxpJer6lzAc7Ei2TxBlmGHBtUj aGnM/ugT0ACRcgABLcJidpP2Nuy6MzLaXMU+V3xZV90tau4/KQqcWupMEljyGEDV8XR8 xfeTC824Vj1HS3Rred5botR8HyFLEdi53rgW1n/oYPfzTw5MDeQiMj0+p0AdouB1roi9 V6nter6MtQJOMmZikpA5+LNbpKEeeGCyllnrUaQSK4DTBHqv4P2+N6xHuS5h6ln/9Ogj 88jV+VWkMs1CJAMZSO/3iRsERe3QK0v0sL7Ed3QUpi2w+XpzVFKED3mBaTrT2+cbvKYB edRA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:mime-version:message-id:date :subject:cc:to:from:sender:dkim-signature; bh=0NopvoWcKlR7CUdf0FH/d0hO8XWLaHLSal5NnG6Yqf0=; fh=nK1TOWJVZs+ikff5+AAX7AoGLCtebT8HykLLAgCodO8=; b=NtThFV0prfb59mwzqOEPdOADlIPOWwuGF951TU/naRbbSza4z/exWe/QsrteW70ivy mztizUIKmz+8slJUTuRX4jLUuFMYmbgiaTRyK6YngYJ83vu4TMQB6eCAsv6EokIt5YBF nRtigzUyNW7ADUu84jD7gaHB3DxyNjh6zgoGzZV+a8tRtQga845SXLV67HgPJTf0792L eVF9trsT3rmk8K76MKXwmPkkvAdnsqjxXYT3IQKjlFnzgvQedDvZvp2NtT+2APNL+HqS PfF3/NDyP15YAifkzeIg/sJiJGZ9VoCv24Xpy7jwsI/5oMNBbTTh6buDiiUvqAHu0ZAm GOFw==; darn=patchwork.ozlabs.org ARC-Authentication-Results: i=2; gmr-mx.google.com; dkim=pass header.i=@atmark-techno.com header.s=gw2_bookworm header.b=IuxO3pPH; dkim=pass header.i=@atmark-techno.com header.s=google header.b=jeEHomMm; spf=pass (google.com: domain of dominique.martinet@atmark-techno.com designates 35.74.137.57 as permitted sender) smtp.mailfrom=dominique.martinet@atmark-techno.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=atmark-techno.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlegroups.com; s=20230601; t=1717404973; x=1718009773; darn=patchwork.ozlabs.org; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:x-original-authentication-results :x-original-sender:mime-version:message-id:date:subject:cc:to:from :sender:from:to:cc:subject:date:message-id:reply-to; bh=0NopvoWcKlR7CUdf0FH/d0hO8XWLaHLSal5NnG6Yqf0=; b=GtliULbQPI5Omu5lbwvKqCla1S3uyQUUnouNpj6oAhp3VKuTZSBeXteiTSsHii/7Vl IdlTS2mYcMufWP22UIis10gzZUVnq0srgBdxOSuhC+rq+33xRpy/zRdy2heYbMd/FQe4 UHgkPomKyW7IyMW8IMkWn6xIs0DB6LmgUafRfYqpS9s3E81NcCxOrCjq7N4fhA7eNkhb a8AlnWtFVm2UjLQA/eo6Ecy7X0aJfeWDZ9Wsji8eW8lLngK6PjilWA0CFbIUbDl0Ptd/ VqqHKADNXOdcmG4D5V6X0dqmB+4kzdv6zt8RNMgR/i7aj2bjkPHSrpvtnvpnh3Lc9gOV KOHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717404973; x=1718009773; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :x-spam-checked-in-group:list-id:mailing-list:precedence :x-original-authentication-results:x-original-sender:mime-version :message-id:date:subject:cc:to:from:x-beenthere:x-gm-message-state :sender:from:to:cc:subject:date:message-id:reply-to; bh=0NopvoWcKlR7CUdf0FH/d0hO8XWLaHLSal5NnG6Yqf0=; b=ujOPbJQP+TrpMJkcUKT4xFcL9xCySl1q9ELxuF+Urzn9cU/VTa2dcoYcFthYQDMSK7 7izdy3giEqm9vWm0PSox6VQB7ABFAsx9WM496MMJMBQPmfIPX6/iZZE+R+aPIzk2wpk2 OVWzkFqn3hmQvKQV5f61LM+zlBFlJ0847rZH7oHrXjLfHCchM+wv+y7y+GrubxWDroJ0 1333SbEmYRF0cVFsaaO4R1s4uQlR25lJS/a1ZoN4Mtq+ieguEbE3PYyL6YgGlx41Un5J GsF1nes1/Kh9jEGZw3sTNy+vEjgDqWG/cTKxUwif7v1VZId32Ss98J8+RPhiH5XZ+jlE 2Abg== Sender: swupdate@googlegroups.com X-Forwarded-Encrypted: i=2; AJvYcCUz/8o7SyYaNQIsKPS+5a0MxvZlMJPWjZNSvQHp2gMC4FEplXlyXnlBo/FI36nMuu0mcChfbMsXeLWMXUzW2XuqL0lOnDxt2QUkvURxig== X-Gm-Message-State: AOJu0YwXJ+N8hD3UN1C0nvNRojb52shVg7JbEyerC2MJ9oUuU9tyrM7J rvkKHhO4NdfGe9sN/TFCyTLRDHC3ymoXjb0HL4rRk0XNYa48/B/i X-Google-Smtp-Source: AGHT+IER9baLcCqGxsHoJnbvbATFBsRzw8bE2RBYwP0zdQjYqSPIP3CsqU04+Jq6lzLp1xDzEtiQzQ== X-Received: by 2002:a17:903:2352:b0:1f6:fa0:da71 with SMTP id d9443c01a7336-1f636fefaa2mr74695265ad.18.1717404972961; Mon, 03 Jun 2024 01:56:12 -0700 (PDT) X-BeenThere: swupdate@googlegroups.com Received: by 2002:a17:903:32cf:b0:1f3:555:b70a with SMTP id d9443c01a7336-1f616f943e4ls7685065ad.0.-pod-prod-09-us; Mon, 03 Jun 2024 01:56:11 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCXNmzy4wYBRswD9os4AM91EkN5i8LU1PQl3Qf26JQ8qLqTOMylsHKEt5Es57r784G5wZTohKJnFzR419DOwMnUD29A7eBW6xcoJ X-Received: by 2002:a05:6a20:da95:b0:1b0:c39:f41d with SMTP id adf61e73a8af0-1b26f275926mr8383170637.50.1717404971397; Mon, 03 Jun 2024 01:56:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1717404971; cv=none; d=google.com; s=arc-20160816; b=BRdxPr1XNjP0kroiixVjiC/kGseXIKI8w+Q0/viRBmmXxk/eQb1eUCAywDgDwm8I4O pBweRIPVcH2CWPQRh1oZCCs1NWS+YZOb732XaXKdXbE0j/JCSQ2o8GCTUGyPccj6tcRP 9DjL7cuF273bFGdiLb9vf5fmNWT4i9ce2dAq8v6TJ4/+6lR61B+m/TKXMKkbMiIpo6rp VsRnzr80/E0pY5E7R+7HlK8z3mZmKGrNICtG7+xBz7SrLgDP1uY7udNIQmhhjDhNJA0B Vzj0PJkikj1ZI83ELasi8xD6TolQkkTA7X0By8HG8rgUjHskuqJlOcHzSmVKrSL0BWxU itkg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:dkim-signature:dkim-signature; bh=UOmIQrU4HnpI97kai0owdmxWJ5OWG9PCQR+xHdQNP8g=; fh=QS5bsT3SeuIU2QL1Ypi4EfNHgs6MUGXhEraFnHCTh+I=; b=0iTGVtKoq2nC5ErgHNRCzxFa9PkbQOfX1xR8Z+krOkquc92BgDwXPnoecJLQhXM3WT Xu1FEP2Ut7f7virqoLtugkf1rx51vn4aNFdfF4iMj27Sw0dy5N3bk/XEDkLa/VmD2dGy Pqg4D5S+twe/DErar7c2AhrMy6gWEx/664GQ4Hz0aWcZ0STHaXbjXOLnUn1zEK9be2ZY yPRJn6Rf7/Vv4qOh5IuiCr0rSzs7vzOZ9pE3ktRB0H0F2oh/PcexMUDUpfXOwZYSQ6iq HjqC1IkoOTxQdYUhhE2QQpc7c9dcKaIVkzgTWNNnOcXNxxvAEOpv2ISmicJO30BLRAYJ pi6A==; dara=google.com ARC-Authentication-Results: i=1; gmr-mx.google.com; dkim=pass header.i=@atmark-techno.com header.s=gw2_bookworm header.b=IuxO3pPH; dkim=pass header.i=@atmark-techno.com header.s=google header.b=jeEHomMm; spf=pass (google.com: domain of dominique.martinet@atmark-techno.com designates 35.74.137.57 as permitted sender) smtp.mailfrom=dominique.martinet@atmark-techno.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=atmark-techno.com Received: from gw2.atmark-techno.com (gw2.atmark-techno.com. [35.74.137.57]) by gmr-mx.google.com with ESMTPS id 98e67ed59e1d1-2c1b99e7533si319924a91.0.2024.06.03.01.56.11 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Jun 2024 01:56:11 -0700 (PDT) Received-SPF: pass (google.com: domain of dominique.martinet@atmark-techno.com designates 35.74.137.57 as permitted sender) client-ip=35.74.137.57; Received: from gw2.atmark-techno.com (localhost [127.0.0.1]) by gw2.atmark-techno.com (Postfix) with ESMTP id 6B92492F for ; Mon, 3 Jun 2024 17:56:09 +0900 (JST) Received: from mail-pl1-f198.google.com (mail-pl1-f198.google.com [209.85.214.198]) by gw2.atmark-techno.com (Postfix) with ESMTPS id 1F458283 for ; Mon, 3 Jun 2024 17:56:09 +0900 (JST) Received: by mail-pl1-f198.google.com with SMTP id d9443c01a7336-1f302bf0c29so25876365ad.0 for ; Mon, 03 Jun 2024 01:56:09 -0700 (PDT) X-Forwarded-Encrypted: i=1; AJvYcCVVMRnOwFGLtMwPSRWxYMUh4fp5uLmDMeAMupUqp020y9cdR+wiqqlQZU7QbmK22lXvfx7G8rvlXEytUHJKFvyqQHqG2qL6T1Mr X-Received: by 2002:a17:903:189:b0:1f6:7e5f:5a51 with SMTP id d9443c01a7336-1f67e5f626fmr17537235ad.36.1717404968078; Mon, 03 Jun 2024 01:56:08 -0700 (PDT) X-Received: by 2002:a17:903:189:b0:1f6:7e5f:5a51 with SMTP id d9443c01a7336-1f67e5f626fmr17537085ad.36.1717404967656; Mon, 03 Jun 2024 01:56:07 -0700 (PDT) Received: from pc-0182.atmarktech (35.112.198.104.bc.googleusercontent.com. [104.198.112.35]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1f63235ae5csm61267965ad.79.2024.06.03.01.56.07 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 03 Jun 2024 01:56:07 -0700 (PDT) Received: from [::1] (helo=pc-0182.atmark.tech) by pc-0182.atmarktech with esmtp (Exim 4.96) (envelope-from ) id 1sE3U6-009riT-03; Mon, 03 Jun 2024 17:56:06 +0900 From: Dominique Martinet To: stefano.babic@swupdate.org, swupdate@googlegroups.com Cc: Dominique Martinet Subject: [swupdate] [PATCH RFC 0/9] Proof of concept of chunked checksums Date: Mon, 3 Jun 2024 17:55:53 +0900 Message-Id: <20240603085602.2351411-1-dominique.martinet@atmark-techno.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 X-Original-Sender: dominique.martinet@atmark-techno.com X-Original-Authentication-Results: gmr-mx.google.com; dkim=pass header.i=@atmark-techno.com header.s=gw2_bookworm header.b=IuxO3pPH; dkim=pass header.i=@atmark-techno.com header.s=google header.b=jeEHomMm; spf=pass (google.com: domain of dominique.martinet@atmark-techno.com designates 35.74.137.57 as permitted sender) smtp.mailfrom=dominique.martinet@atmark-techno.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=atmark-techno.com Precedence: list Mailing-list: list swupdate@googlegroups.com; contact swupdate+owners@googlegroups.com List-ID: X-Spam-Checked-In-Group: swupdate@googlegroups.com X-Google-Group-Id: 605343134186 List-Post: , List-Help: , List-Archive: , List-Unsubscribe: , This is the poc I had promised in my mail last week: https://groups.google.com/g/swupdate/c/sNnb1SvHPxw/m/8qjMm7JYAAAJ There's quite a bit of cleanup first that would probably be good to take regardless -- Stefano, do you want me to resend the fixes / cleanup separately first? The core of the implementation itself is in patches 7 & 8; 7 (adding a step to cpio_utils) is as I had expected fairly straightforward, we'll probably want to make the chunk size configurable and perhaps change the chunk implementation if mmap causes issues but it wasn't too bad. 8 (config) is ugly because we're parsing it all upfront, I had suggested keeping a reference to the config in img and freeing it in free_img instead and that's still possible but ultimately keeping a handle to the config keeps the whole data as string in memory so it's better to re-alloc something more efficient and move it there; keeping a reference to the config handle also wouldn't work well with the lua external parsing so this is probably better this way. (I haven't implemented the external parser C-lua glue here, but will do before sending the final patch if we go that way -- I've never used it though so might need help testing) Dominique Martinet (9): parser: set_find_path: properly return to-free string array cpio_utils: move out hash comparison copyfile cleanup: use copyimage when we can copyfile refactor: replace the 13 arguments by a struct cpio_utils: cleanup state chaining code cpio_utils: fill_buffer: fix return value on EOF after short read cpio_utils: implement chunked hash verification config parsing: add handling for chunked_hashes cpio_utils: skip full sha256 if chunked hash is set Kconfig | 4 + core/cpio_utils.c | 383 ++++++++++++++++++---------- core/installer.c | 20 +- core/parsing_library.c | 55 +++- core/stream_interface.c | 24 +- corelib/lua_interface.c | 38 +-- corelib/parsing_library_libconfig.c | 2 +- corelib/parsing_library_libjson.c | 2 +- handlers/copy_handler.c | 20 +- handlers/delta_handler.c | 42 ++- handlers/rdiff_handler.c | 13 +- handlers/readback_handler.c | 20 +- include/parselib.h | 5 +- include/swupdate_image.h | 1 + include/util.h | 42 ++- parser/parser.c | 7 +- 16 files changed, 422 insertions(+), 256 deletions(-)