| Message ID | 20241201211304.2199299-1-danwalkes@trellis-logic.com |
|---|---|
| State | New |
| Delegated to: | Stefano Babic |
| Headers | show |
| Series | [meta-swupdate] swupdate classes: fix hash mismatch on master | expand |
Sending this for review and comment. Khem I think as discussed at https://groups.google.com/g/swupdate/c/8K-9H7C9o5E/m/a9fqhIOjAAAJ the correct fix here is more complicated. These changes work for me on https://github.com/OE4T/tegra-demo-distro/tree/master/layers/meta-tegrademo/dynamic-layers/meta-swupdate but I could use some help testing the encryption/signing setup since I don't have a demo setup for this. Also if you prefer attempting to rollback and split S and UNPACKDIR for a simpler version which still modifies S let me know and I can try that instead. On Sunday, December 1, 2024 at 2:13:11 PM UTC-7 Dan Walkes wrote: > The fix discussed in [1] resolves build issues on styhead but > introduces an issue which causes hash mismatches. > > The first build succeeds to generate a working .swu, however > future builds fail with hash mismatch and a message like this: > > ``` > [ERROR] : SWUPDATE failed [0] ERROR cpio_utils.c : __swupdate_copy : 670 : > HASH mismatch : > 5bb39965bfcedd87fc02bd51fac6636b924aed606d0984dcebae6a3250b01af6 <--> > ff70a8dbc50a9ce2bad724554f5f14b8c933ad87fc544d054142755e8ef07f1e > [ERROR] : SWUPDATE failed [0] ERROR archive_handler.c : > install_archive_image : 344 : Error copying extracted file > ``` > > This happens because the modifications made to files in the > S directory only happen once. This means there's a single > conversion of `$swupdate_get_sha256(` to the file sha, and future > builds use the hash previously placed in the file in the S directory > which is no longer correct when the rootfs file hash changes. > > It would probably be possible to make a simpler change here which > splits the UNPACKDIR and S and continues to make runtime > edits in S. However, it seems the intent of S and/or UNPACKDIR > is to contain unmodified source as discussed in [1]. > > This change instead: > 1. Moves the sw-description file to WORKDIR for editing. > 2. Continues to use S as source location for sw-description > and other source files which are not edited. > 3. Makes all edits in WORKDIR. > 4. When preparing the cpio, prepares from WORKDIR instead of > S. using any existing files in WORKDIR if they exist. If they > do not exist, symlinks the relevant files in S instead. > 6. Adds the -L option to cpio to prevent adding symlinks to the > cpio file, and avoid the need to copy from S to WORKDIR when > preparing the cpio. > > 1: https://groups.google.com/g/swupdate/c/8K-9H7C9o5E/m/a9fqhIOjAAAJ > > Signed-off-by: Dan Walkes <danw...@trellis-logic.com> > --- > classes-recipe/swupdate-common.bbclass | 37 +++++++++++++++----------- > classes-recipe/swupdate-image.bbclass | 2 +- > 2 files changed, 23 insertions(+), 16 deletions(-) > > diff --git a/classes-recipe/swupdate-common.bbclass > b/classes-recipe/swupdate-common.bbclass > index 7b49561..085a7b5 100644 > --- a/classes-recipe/swupdate-common.bbclass > +++ b/classes-recipe/swupdate-common.bbclass > @@ -66,22 +66,22 @@ def swupdate_getdepends(d): > > return depstr > > -def swupdate_write_sha256(s): > +def swupdate_write_sha256(workdir): > import re > write_lines = [] > - with open(os.path.join(s, "sw-description"), 'r') as f: > + with open(os.path.join(workdir, "sw-description"), 'r') as f: > for line in f: > shastr = r"sha256.+=.+@(.+\")" > m = > re.match(r"^(?P<before_placeholder>.+)(sha256|version).+[=:].*(?P<quote>[\'\"])@(?P<filename>.*)(?P=quote)", > line) > if m: > filename = m.group('filename') > bb.warn("Syntax for sha256 changed, please use $swupdate_get_sha256(%s)" % > filename) > - hash = swupdate_get_sha256(None, s, filename) > + hash = swupdate_get_sha256(None, workdir, filename) > write_lines.append(line.replace("@%s" % (filename), hash)) > else: > write_lines.append(line) > > - with open(os.path.join(s, "sw-description"), 'w+') as f: > + with open(os.path.join(workdir, "sw-description"), 'w+') as f: > for line in write_lines: > f.write(line) > > @@ -97,7 +97,7 @@ def swupdate_exec_functions(d, s, write_lines): > write_lines[index] = line > > > -def swupdate_expand_bitbake_variables(d, s): > +def swupdate_expand_bitbake_variables(d, s, workdir): > write_lines = [] > > with open(os.path.join(s, "sw-description"), 'r') as f: > @@ -131,7 +131,7 @@ def swupdate_expand_bitbake_variables(d, s): > > swupdate_exec_functions(d, s, write_lines) > > - with open(os.path.join(s, "sw-description"), 'w+') as f: > + with open(os.path.join(workdir, "sw-description"), 'w+') as f: > for line in write_lines: > f.write(line) > > @@ -173,17 +173,18 @@ def prepare_sw_description(d): > import subprocess > > s = d.getVar('S') > - swupdate_expand_bitbake_variables(d, s) > + workdir = d.getVar('WORKDIR') > + swupdate_expand_bitbake_variables(d, s, workdir) > > - swupdate_write_sha256(s) > + swupdate_write_sha256(workdir) > > encrypt = d.getVar('SWUPDATE_ENCRYPT_SWDESC') > if encrypt: > bb.note("Encryption of sw-description") > - shutil.copyfile(os.path.join(s, 'sw-description'), os.path.join(s, > 'sw-description.plain')) > + shutil.copyfile(os.path.join(workdir, 'sw-description'), > os.path.join(workdir, 'sw-description.plain')) > key,iv = swupdate_extract_keys(d.getVar('SWUPDATE_AES_FILE')) > - iv = swupdate_get_IV(d, s, 'sw-description') > - swupdate_encrypt_file(os.path.join(s, 'sw-description.plain'), > os.path.join(s, 'sw-description'), key, iv) > + iv = swupdate_get_IV(d, workdir, 'sw-description') > + swupdate_encrypt_file(os.path.join(workdir, 'sw-description.plain'), > os.path.join(workdir, 'sw-description'), key, iv) > > signing = d.getVar('SWUPDATE_SIGNING') > if signing == "1": > @@ -191,8 +192,8 @@ def prepare_sw_description(d): > signing = "RSA" > if signing: > > - sw_desc_sig = os.path.join(s, 'sw-description.sig') > - sw_desc = os.path.join(s, 'sw-description.plain' if encrypt else > 'sw-description') > + sw_desc_sig = os.path.join(workdir, 'sw-description.sig') > + sw_desc = os.path.join(workdir, 'sw-description.plain' if encrypt else > 'sw-description') > > if signing == "CUSTOM": > signcmd = [] > @@ -233,6 +234,7 @@ def swupdate_add_src_uri(d, list_for_cpio): > import shutil > > s = d.getVar('S') > + workdir = d.getVar('WORKDIR') > exclude = (d.getVar("SWUPDATE_SRC_URI_EXCLUDE") or "").split() > > fetch = bb.fetch2.Fetch([], d) > @@ -310,9 +312,14 @@ def swupdate_add_artifacts(d, list_for_cpio): > > def swupdate_create_cpio(d, swudeploydir, list_for_cpio): > s = d.getVar('S') > - os.chdir(s) > + workdir = d.getVar('WORKDIR') > + os.chdir(workdir) > + for file in list_for_cpio: > + if not os.path.exists(file): > + os.symlink(os.path.join(s, file), file) > + > updateimage = d.getVar('IMAGE_NAME') + '.swu' > - line = 'for i in ' + ' '.join(list_for_cpio) + '; do echo $i;done | cpio > -ov -H crc --reproducible > ' + os.path.join(swudeploydir, updateimage) > + line = 'for i in ' + ' '.join(list_for_cpio) + '; do echo $i;done | cpio > -ov -H crc --reproducible -L > ' + os.path.join(swudeploydir, updateimage) > os.system(line) > os.chdir(swudeploydir) > updateimage_link = d.getVar('IMAGE_LINK_NAME') > diff --git a/classes-recipe/swupdate-image.bbclass > b/classes-recipe/swupdate-image.bbclass > index 1cd3eeb..bf8c5cc 100644 > --- a/classes-recipe/swupdate-image.bbclass > +++ b/classes-recipe/swupdate-image.bbclass > @@ -40,7 +40,7 @@ python do_swupdate_copy_swdescription() { > > import shutil > > - workdir = d.getVar('S') > + workdir = d.getVar('WORKDIR') > filespath = d.getVar('FILESPATH') > sw_desc_path = bb.utils.which(filespath, "sw-description") > shutil.copyfile(sw_desc_path, os.path.join(workdir, "sw-description")) > -- > 2.34.1 > >
diff --git a/classes-recipe/swupdate-common.bbclass b/classes-recipe/swupdate-common.bbclass index 7b49561..085a7b5 100644 --- a/classes-recipe/swupdate-common.bbclass +++ b/classes-recipe/swupdate-common.bbclass @@ -66,22 +66,22 @@ def swupdate_getdepends(d): return depstr -def swupdate_write_sha256(s): +def swupdate_write_sha256(workdir): import re write_lines = [] - with open(os.path.join(s, "sw-description"), 'r') as f: + with open(os.path.join(workdir, "sw-description"), 'r') as f: for line in f: shastr = r"sha256.+=.+@(.+\")" m = re.match(r"^(?P<before_placeholder>.+)(sha256|version).+[=:].*(?P<quote>[\'\"])@(?P<filename>.*)(?P=quote)", line) if m: filename = m.group('filename') bb.warn("Syntax for sha256 changed, please use $swupdate_get_sha256(%s)" % filename) - hash = swupdate_get_sha256(None, s, filename) + hash = swupdate_get_sha256(None, workdir, filename) write_lines.append(line.replace("@%s" % (filename), hash)) else: write_lines.append(line) - with open(os.path.join(s, "sw-description"), 'w+') as f: + with open(os.path.join(workdir, "sw-description"), 'w+') as f: for line in write_lines: f.write(line) @@ -97,7 +97,7 @@ def swupdate_exec_functions(d, s, write_lines): write_lines[index] = line -def swupdate_expand_bitbake_variables(d, s): +def swupdate_expand_bitbake_variables(d, s, workdir): write_lines = [] with open(os.path.join(s, "sw-description"), 'r') as f: @@ -131,7 +131,7 @@ def swupdate_expand_bitbake_variables(d, s): swupdate_exec_functions(d, s, write_lines) - with open(os.path.join(s, "sw-description"), 'w+') as f: + with open(os.path.join(workdir, "sw-description"), 'w+') as f: for line in write_lines: f.write(line) @@ -173,17 +173,18 @@ def prepare_sw_description(d): import subprocess s = d.getVar('S') - swupdate_expand_bitbake_variables(d, s) + workdir = d.getVar('WORKDIR') + swupdate_expand_bitbake_variables(d, s, workdir) - swupdate_write_sha256(s) + swupdate_write_sha256(workdir) encrypt = d.getVar('SWUPDATE_ENCRYPT_SWDESC') if encrypt: bb.note("Encryption of sw-description") - shutil.copyfile(os.path.join(s, 'sw-description'), os.path.join(s, 'sw-description.plain')) + shutil.copyfile(os.path.join(workdir, 'sw-description'), os.path.join(workdir, 'sw-description.plain')) key,iv = swupdate_extract_keys(d.getVar('SWUPDATE_AES_FILE')) - iv = swupdate_get_IV(d, s, 'sw-description') - swupdate_encrypt_file(os.path.join(s, 'sw-description.plain'), os.path.join(s, 'sw-description'), key, iv) + iv = swupdate_get_IV(d, workdir, 'sw-description') + swupdate_encrypt_file(os.path.join(workdir, 'sw-description.plain'), os.path.join(workdir, 'sw-description'), key, iv) signing = d.getVar('SWUPDATE_SIGNING') if signing == "1": @@ -191,8 +192,8 @@ def prepare_sw_description(d): signing = "RSA" if signing: - sw_desc_sig = os.path.join(s, 'sw-description.sig') - sw_desc = os.path.join(s, 'sw-description.plain' if encrypt else 'sw-description') + sw_desc_sig = os.path.join(workdir, 'sw-description.sig') + sw_desc = os.path.join(workdir, 'sw-description.plain' if encrypt else 'sw-description') if signing == "CUSTOM": signcmd = [] @@ -233,6 +234,7 @@ def swupdate_add_src_uri(d, list_for_cpio): import shutil s = d.getVar('S') + workdir = d.getVar('WORKDIR') exclude = (d.getVar("SWUPDATE_SRC_URI_EXCLUDE") or "").split() fetch = bb.fetch2.Fetch([], d) @@ -310,9 +312,14 @@ def swupdate_add_artifacts(d, list_for_cpio): def swupdate_create_cpio(d, swudeploydir, list_for_cpio): s = d.getVar('S') - os.chdir(s) + workdir = d.getVar('WORKDIR') + os.chdir(workdir) + for file in list_for_cpio: + if not os.path.exists(file): + os.symlink(os.path.join(s, file), file) + updateimage = d.getVar('IMAGE_NAME') + '.swu' - line = 'for i in ' + ' '.join(list_for_cpio) + '; do echo $i;done | cpio -ov -H crc --reproducible > ' + os.path.join(swudeploydir, updateimage) + line = 'for i in ' + ' '.join(list_for_cpio) + '; do echo $i;done | cpio -ov -H crc --reproducible -L > ' + os.path.join(swudeploydir, updateimage) os.system(line) os.chdir(swudeploydir) updateimage_link = d.getVar('IMAGE_LINK_NAME') diff --git a/classes-recipe/swupdate-image.bbclass b/classes-recipe/swupdate-image.bbclass index 1cd3eeb..bf8c5cc 100644 --- a/classes-recipe/swupdate-image.bbclass +++ b/classes-recipe/swupdate-image.bbclass @@ -40,7 +40,7 @@ python do_swupdate_copy_swdescription() { import shutil - workdir = d.getVar('S') + workdir = d.getVar('WORKDIR') filespath = d.getVar('FILESPATH') sw_desc_path = bb.utils.which(filespath, "sw-description") shutil.copyfile(sw_desc_path, os.path.join(workdir, "sw-description"))
The fix discussed in [1] resolves build issues on styhead but introduces an issue which causes hash mismatches. The first build succeeds to generate a working .swu, however future builds fail with hash mismatch and a message like this: ``` [ERROR] : SWUPDATE failed [0] ERROR cpio_utils.c : __swupdate_copy : 670 : HASH mismatch : 5bb39965bfcedd87fc02bd51fac6636b924aed606d0984dcebae6a3250b01af6 <--> ff70a8dbc50a9ce2bad724554f5f14b8c933ad87fc544d054142755e8ef07f1e [ERROR] : SWUPDATE failed [0] ERROR archive_handler.c : install_archive_image : 344 : Error copying extracted file ``` This happens because the modifications made to files in the S directory only happen once. This means there's a single conversion of `$swupdate_get_sha256(` to the file sha, and future builds use the hash previously placed in the file in the S directory which is no longer correct when the rootfs file hash changes. It would probably be possible to make a simpler change here which splits the UNPACKDIR and S and continues to make runtime edits in S. However, it seems the intent of S and/or UNPACKDIR is to contain unmodified source as discussed in [1]. This change instead: 1. Moves the sw-description file to WORKDIR for editing. 2. Continues to use S as source location for sw-description and other source files which are not edited. 3. Makes all edits in WORKDIR. 4. When preparing the cpio, prepares from WORKDIR instead of S. using any existing files in WORKDIR if they exist. If they do not exist, symlinks the relevant files in S instead. 6. Adds the -L option to cpio to prevent adding symlinks to the cpio file, and avoid the need to copy from S to WORKDIR when preparing the cpio. 1: https://groups.google.com/g/swupdate/c/8K-9H7C9o5E/m/a9fqhIOjAAAJ Signed-off-by: Dan Walkes <danwalkes@trellis-logic.com> --- classes-recipe/swupdate-common.bbclass | 37 +++++++++++++++----------- classes-recipe/swupdate-image.bbclass | 2 +- 2 files changed, 23 insertions(+), 16 deletions(-)