mbox series

[RFC,0/9] Proof of concept of chunked checksums

Message ID 20240603085602.2351411-1-dominique.martinet@atmark-techno.com
Headers show
Series Proof of concept of chunked checksums | expand

Message

Dominique Martinet June 3, 2024, 8:55 a.m. UTC 
This is the poc I had promised in my mail last week:
https://groups.google.com/g/swupdate/c/sNnb1SvHPxw/m/8qjMm7JYAAAJ

There's quite a bit of cleanup first that would probably be good to take
regardless -- Stefano, do you want me to resend the fixes / cleanup
separately first?

The core of the implementation itself is in patches 7 & 8;
7 (adding a step to cpio_utils) is as I had expected fairly
straightforward, we'll probably want to make the chunk size configurable
and perhaps change the chunk implementation if mmap causes issues but it
wasn't too bad.
8 (config) is ugly because we're parsing it all upfront, I had suggested
keeping a reference to the config in img and freeing it in free_img
instead and that's still possible but ultimately keeping a handle to the
config keeps the whole data as string in memory so it's better to
re-alloc something more efficient and move it there; keeping a reference
to the config handle also wouldn't work well with the lua external
parsing so this is probably better this way.
(I haven't implemented the external parser C-lua glue here, but will do
before sending the final patch if we go that way -- I've never used it
though so might need help testing)


Dominique Martinet (9):
  parser: set_find_path: properly return to-free string array
  cpio_utils: move out hash comparison
  copyfile cleanup: use copyimage when we can
  copyfile refactor: replace the 13 arguments by a struct
  cpio_utils: cleanup state chaining code
  cpio_utils: fill_buffer: fix return value on EOF after short read
  cpio_utils: implement chunked hash verification
  config parsing: add handling for chunked_hashes
  cpio_utils: skip full sha256 if chunked hash is set

 Kconfig                             |   4 +
 core/cpio_utils.c                   | 383 ++++++++++++++++++----------
 core/installer.c                    |  20 +-
 core/parsing_library.c              |  55 +++-
 core/stream_interface.c             |  24 +-
 corelib/lua_interface.c             |  38 +--
 corelib/parsing_library_libconfig.c |   2 +-
 corelib/parsing_library_libjson.c   |   2 +-
 handlers/copy_handler.c             |  20 +-
 handlers/delta_handler.c            |  42 ++-
 handlers/rdiff_handler.c            |  13 +-
 handlers/readback_handler.c         |  20 +-
 include/parselib.h                  |   5 +-
 include/swupdate_image.h            |   1 +
 include/util.h                      |  42 ++-
 parser/parser.c                     |   7 +-
 16 files changed, 422 insertions(+), 256 deletions(-)