diff mbox series

[meta-swupdate,2/2] Add support for automatic decrypted size in sw-description

Message ID 20211222140401.780-2-sam.van.den.berge@gmail.com
State Changes Requested
Headers show
Series None | expand

Commit Message

Sam Van Den Berge Dec. 22, 2021, 2:04 p.m. UTC
The same principle from automatic sha256 insertion is applied for
automatic decrypted-size insertion:

images: (
	{
		filename = "image.ubifs";
		volume = "root_r";
		sha256 = "@image.ubifs";
		encrypted = true;
		properties: {
		    decrypted-size = "@image.ubifs";
		}
	},
);

This is usefull especially for *encrypted* UBI images which need the
"decrypted-size" property when the image is installed.

Signed-off-by: Sam Van Den Berge <sam.van.den.berge@gmail.com>
---
 classes/swupdate-common.bbclass | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

Comments

Sam Van Den Berge Jan. 7, 2022, 2:03 p.m. UTC | #1
Hello,

Would it be possible to get some feedback on those 2 patches?

Thanks in advance.

Kr,
Sam.
Stefano Babic Jan. 10, 2022, 4:27 p.m. UTC | #2
Hi Sam,

On 07.01.22 09:03, Sam Van Den Berge wrote:
> Hello,
> 
> Would it be possible to get some feedback on those 2 patches?
> 

I am not currently in office, I will check your changes in next days.

Best regards,
Stefano Babic

> Thanks in advance.
> 
> Kr,
> Sam.
>
Stefano Babic Jan. 17, 2022, 3:30 p.m. UTC | #3
Hi Sam,

On 22.12.21 09:04, Sam Van Den Berge wrote:
> The same principle from automatic sha256 insertion is applied for
> automatic decrypted-size insertion:
> 
> images: (
> 	{
> 		filename = "image.ubifs";
> 		volume = "root_r";
> 		sha256 = "@image.ubifs";
> 		encrypted = true;
> 		properties: {
> 		    decrypted-size = "@image.ubifs";
> 		}
> 	},
> );
> 
> This is usefull especially for *encrypted* UBI images which need the
> "decrypted-size" property when the image is installed.
> 

Same is for "decompressed-size", when a ubifs is not encrypted but 
compressed. Anyway, it looks to me that adding a specific regex for each 
attribute ("decrypted-size" vs "decompressed-size") does not scale well.

I would split the issue : one topic is to add a function (that is 
something like swupdate_decrypted_size()) that can derive a value from 
an artifact. The other topic and my concern is how to bind the function 
to the attribute in sw-description. sha256 was the first function added 
to automate the generation of sw-description from a template. After 
that, full variable substitution was added.

What I would like to see but I never investigated is to add a general 
way to bind results from function to attributes, like an extended 
variable substitution. This is already supported in OE, just see for 
example "${@bb.utils..".

If we have the following mechanismus:

	decrypted-size = "${@os.path.getsize(@@IMAGE@@))";

we do not need any further rule to regexp, and it scales much more 
better. Can you check if this can be done ?

> Signed-off-by: Sam Van Den Berge <sam.van.den.berge@gmail.com>
> ---
>   classes/swupdate-common.bbclass | 25 +++++++++++++++++++++++++
>   1 file changed, 25 insertions(+)
> 
> diff --git a/classes/swupdate-common.bbclass b/classes/swupdate-common.bbclass
> index e6e5a5f..79f81df 100644
> --- a/classes/swupdate-common.bbclass
> +++ b/classes/swupdate-common.bbclass
> @@ -91,6 +91,29 @@ def swupdate_write_sha256(s):
>           for line in write_lines:
>               f.write(line)
>   
> +def swupdate_write_decrypted_size(s, src):
> +    import re
> +    write_lines = []
> +
> +    decrypted_size = os.path.getsize(src)
> +
> +    with open(os.path.join(s, "sw-description"), 'r') as f:
> +       for line in f:
> +          m = re.match(r"^\s*(decrypted-size).+[=:].*[\'\"]@(?P<filename>.*)[\'\"]", line)
> +          if m:
> +              filename = m.group('filename')
> +              if os.path.basename(src) == filename:
> +                  bb.note("decrypted-size of %s (%s) is %s" % (filename, src, decrypted_size))
> +                  write_lines.append(line.replace("@%s" % (filename), str(decrypted_size)))
> +              else:
> +                  write_lines.append(line)
> +          else:
> +              write_lines.append(line)
> +
> +    with open(os.path.join(s, "sw-description"), 'w+') as f:
> +        for line in write_lines:
> +            f.write(line)
> +
>   def swupdate_expand_bitbake_variables(d, s):
>       write_lines = []
>   
> @@ -313,6 +336,7 @@ def swupdate_add_src_uri(d, list_for_cpio):
>                   if not key or not iv:
>                       bb.fatal("Encryption required, but no key found")
>                   swupdate_encrypt_file(local, dst, key, iv)
> +                swupdate_write_decrypted_size(s, local)
>               else:
>                   shutil.copyfile(local, dst)
>               list_for_cpio.append(filename)
> @@ -329,6 +353,7 @@ def add_image_to_swu(d, deploydir, imagename, s, encrypt, list_for_cpio):
>           key,iv = swupdate_extract_keys(d.getVar('SWUPDATE_AES_FILE', True))
>           bb.note("Encryption requested for %s" %(imagename))
>           swupdate_encrypt_file(src, dst, key, iv)
> +        swupdate_write_decrypted_size(s, src)
>       else:
>           shutil.copyfile(src, dst)
>       list_for_cpio.append(target_imagename)
> 

Best regards,
Stefano Babic
Sam Van Den Berge Jan. 25, 2022, 12:17 p.m. UTC | #4
On Mon, Jan 17, 2022 at 04:30:42PM +0100, Stefano Babic wrote:
> Hi Sam,
> 
> On 22.12.21 09:04, Sam Van Den Berge wrote:
> > The same principle from automatic sha256 insertion is applied for
> > automatic decrypted-size insertion:
> > 
> > images: (
> > 	{
> > 		filename = "image.ubifs";
> > 		volume = "root_r";
> > 		sha256 = "@image.ubifs";
> > 		encrypted = true;
> > 		properties: {
> > 		    decrypted-size = "@image.ubifs";
> > 		}
> > 	},
> > );
> > 
> > This is usefull especially for *encrypted* UBI images which need the
> > "decrypted-size" property when the image is installed.
> > 
> 
> Same is for "decompressed-size", when a ubifs is not encrypted but
> compressed. Anyway, it looks to me that adding a specific regex for each
> attribute ("decrypted-size" vs "decompressed-size") does not scale well.
> 
> I would split the issue : one topic is to add a function (that is something
> like swupdate_decrypted_size()) that can derive a value from an artifact.
> The other topic and my concern is how to bind the function to the attribute
> in sw-description. sha256 was the first function added to automate the
> generation of sw-description from a template. After that, full variable
> substitution was added.
> 
> What I would like to see but I never investigated is to add a general way to
> bind results from function to attributes, like an extended variable
> substitution. This is already supported in OE, just see for example
> "${@bb.utils..".
> 
> If we have the following mechanismus:
> 
> 	decrypted-size = "${@os.path.getsize(@@IMAGE@@))";
> 
> we do not need any further rule to regexp, and it scales much more better.
> Can you check if this can be done ?

Sorry for the late answer. If I understand correctly we would have an
sw-description file which contains lines like

 	decrypted-size = "${@os.path.getsize(@@IMAGE@@))";

These lines then contain function calls to methods we can implement ourselves.
At the time when an image is created, the meta-swupdate layer would read the
sw-description file and fill in the necessary parts by calling the appropriate
functions.

After doing some (quick) investigation I came to the conclusion that it won't
be that easy. At first F-strings seemed like a good candidate.  It supports
variable substitution and calling  functions. However F-strings is merely a
syntax thing. I didn't find any easy way to convert a regular string into an
F-string. The latter would be needed because we would have to read the
'sw-description' file and then convert it into an F-string.

Another alternative would be Template strings[1]. However this only supports
variable substitution and not calling functions.

Same goes for the Format String Syntax [2]. This one however supports accessing
an object's arguments. Theoretically we could use this to call a function. E.g.
"{image.sha256sum}".format(image) where image is a class and sha256sum is a
function of that class decorated with the @property decorator. We would have to
implement such a function for every 'automatic' attribute we want
(decrypted-size, decompressed-size, sha256sum, ...) This all feels a bit
cumbersome.

Another possible solution is to use regex + eval. The regex part we are doing
at the moment and we could extend this with a bit of eval() code. However this
also doesn't seem like a good plan to me.

Finally there exist templating engines which do support what you suggest (e.g.
jinja2) however that is an additional dependency and also that doesn't look
like a good idea to me.

I'm not aware of any other solution but on the other hand I'm also not a python
expert so it could verry well be that I missed something.

Kr,
Sam.

[1] https://docs.python.org/3/library/string.html#template-strings
[2] https://docs.python.org/3/library/string.html#format-string-syntax
[3] https://docs.python.org/3/library/functions.html#property

> > Signed-off-by: Sam Van Den Berge <sam.van.den.berge@gmail.com>
> > ---
> >   classes/swupdate-common.bbclass | 25 +++++++++++++++++++++++++
> >   1 file changed, 25 insertions(+)
> > 
> > diff --git a/classes/swupdate-common.bbclass b/classes/swupdate-common.bbclass
> > index e6e5a5f..79f81df 100644
> > --- a/classes/swupdate-common.bbclass
> > +++ b/classes/swupdate-common.bbclass
> > @@ -91,6 +91,29 @@ def swupdate_write_sha256(s):
> >           for line in write_lines:
> >               f.write(line)
> > +def swupdate_write_decrypted_size(s, src):
> > +    import re
> > +    write_lines = []
> > +
> > +    decrypted_size = os.path.getsize(src)
> > +
> > +    with open(os.path.join(s, "sw-description"), 'r') as f:
> > +       for line in f:
> > +          m = re.match(r"^\s*(decrypted-size).+[=:].*[\'\"]@(?P<filename>.*)[\'\"]", line)
> > +          if m:
> > +              filename = m.group('filename')
> > +              if os.path.basename(src) == filename:
> > +                  bb.note("decrypted-size of %s (%s) is %s" % (filename, src, decrypted_size))
> > +                  write_lines.append(line.replace("@%s" % (filename), str(decrypted_size)))
> > +              else:
> > +                  write_lines.append(line)
> > +          else:
> > +              write_lines.append(line)
> > +
> > +    with open(os.path.join(s, "sw-description"), 'w+') as f:
> > +        for line in write_lines:
> > +            f.write(line)
> > +
> >   def swupdate_expand_bitbake_variables(d, s):
> >       write_lines = []
> > @@ -313,6 +336,7 @@ def swupdate_add_src_uri(d, list_for_cpio):
> >                   if not key or not iv:
> >                       bb.fatal("Encryption required, but no key found")
> >                   swupdate_encrypt_file(local, dst, key, iv)
> > +                swupdate_write_decrypted_size(s, local)
> >               else:
> >                   shutil.copyfile(local, dst)
> >               list_for_cpio.append(filename)
> > @@ -329,6 +353,7 @@ def add_image_to_swu(d, deploydir, imagename, s, encrypt, list_for_cpio):
> >           key,iv = swupdate_extract_keys(d.getVar('SWUPDATE_AES_FILE', True))
> >           bb.note("Encryption requested for %s" %(imagename))
> >           swupdate_encrypt_file(src, dst, key, iv)
> > +        swupdate_write_decrypted_size(s, src)
> >       else:
> >           shutil.copyfile(src, dst)
> >       list_for_cpio.append(target_imagename)
> > 
> 
> Best regards,
> Stefano Babic
> 
> -- 
> =====================================================================
> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: +49-8142-66989-53 Fax: +49-8142-66989-80 Email: sbabic@denx.de
> =====================================================================
diff mbox series

Patch

diff --git a/classes/swupdate-common.bbclass b/classes/swupdate-common.bbclass
index e6e5a5f..79f81df 100644
--- a/classes/swupdate-common.bbclass
+++ b/classes/swupdate-common.bbclass
@@ -91,6 +91,29 @@  def swupdate_write_sha256(s):
         for line in write_lines:
             f.write(line)
 
+def swupdate_write_decrypted_size(s, src):
+    import re
+    write_lines = []
+
+    decrypted_size = os.path.getsize(src)
+
+    with open(os.path.join(s, "sw-description"), 'r') as f:
+       for line in f:
+          m = re.match(r"^\s*(decrypted-size).+[=:].*[\'\"]@(?P<filename>.*)[\'\"]", line)
+          if m:
+              filename = m.group('filename')
+              if os.path.basename(src) == filename:
+                  bb.note("decrypted-size of %s (%s) is %s" % (filename, src, decrypted_size))
+                  write_lines.append(line.replace("@%s" % (filename), str(decrypted_size)))
+              else:
+                  write_lines.append(line)
+          else:
+              write_lines.append(line)
+
+    with open(os.path.join(s, "sw-description"), 'w+') as f:
+        for line in write_lines:
+            f.write(line)
+
 def swupdate_expand_bitbake_variables(d, s):
     write_lines = []
 
@@ -313,6 +336,7 @@  def swupdate_add_src_uri(d, list_for_cpio):
                 if not key or not iv:
                     bb.fatal("Encryption required, but no key found")
                 swupdate_encrypt_file(local, dst, key, iv)
+                swupdate_write_decrypted_size(s, local)
             else:
                 shutil.copyfile(local, dst)
             list_for_cpio.append(filename)
@@ -329,6 +353,7 @@  def add_image_to_swu(d, deploydir, imagename, s, encrypt, list_for_cpio):
         key,iv = swupdate_extract_keys(d.getVar('SWUPDATE_AES_FILE', True))
         bb.note("Encryption requested for %s" %(imagename))
         swupdate_encrypt_file(src, dst, key, iv)
+        swupdate_write_decrypted_size(s, src)
     else:
         shutil.copyfile(src, dst)
     list_for_cpio.append(target_imagename)