[2/3] binman: Remove header from compressed data

From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

Remove header from compressed data because this is uncommon, not
supported by U-Boot and incompatible with external compressed artifacts.

The header was introduced as part of commit eb0f4a4cb402 ("binman:
Support replacing data in a cbfs") to allow device tree entries to be
larger that the compressed contents. Regarding the commit "this is
necessary to cope with a compressed device tree being updated in such a
way that it shrinks after the entry size is already set (an obscure
case)". This case need to be fixed without influence the compressed data
by itself.

Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>

---

 tools/binman/cbfs_util.py |  8 ++++----
 tools/binman/comp_util.py | 11 ++---------
 tools/binman/entry.py     |  2 +-
 3 files changed, 7 insertions(+), 14 deletions(-)

Hi Stefan,

On Tue, 2 Aug 2022 at 06:29, Stefan Herbrechtsmeier
<stefan.herbrechtsmeier-oss@weidmueller.com> wrote:
>
> From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
>
> Remove header from compressed data because this is uncommon, not
> supported by U-Boot and incompatible with external compressed artifacts.
>
> The header was introduced as part of commit eb0f4a4cb402 ("binman:
> Support replacing data in a cbfs") to allow device tree entries to be
> larger that the compressed contents. Regarding the commit "this is
> necessary to cope with a compressed device tree being updated in such a
> way that it shrinks after the entry size is already set (an obscure
> case)". This case need to be fixed without influence the compressed data
> by itself.

I was not able to find a way around this due to the chicken-and egg
problem. Compressed data has an unpredictable size and adding an extra
uncompressed byte may increase or decrease the compressed size.

So my solution was to add the header.

It is optional though, so can we perhaps have a property in the
description to enable it?

Regards,
Simon


>
> Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
>
> ---
>
>  tools/binman/cbfs_util.py |  8 ++++----
>  tools/binman/comp_util.py | 11 ++---------
>  tools/binman/entry.py     |  2 +-
>  3 files changed, 7 insertions(+), 14 deletions(-)
>
> diff --git a/tools/binman/cbfs_util.py b/tools/binman/cbfs_util.py
> index 9cad03886f..a1836f4ad3 100644
> --- a/tools/binman/cbfs_util.py
> +++ b/tools/binman/cbfs_util.py
> @@ -241,9 +241,9 @@ class CbfsFile(object):
>          """Handle decompressing data if necessary"""
>          indata = self.data
>          if self.compress == COMPRESS_LZ4:
> -            data = comp_util.decompress(indata, 'lz4', with_header=False)
> +            data = comp_util.decompress(indata, 'lz4')
>          elif self.compress == COMPRESS_LZMA:
> -            data = comp_util.decompress(indata, 'lzma', with_header=False)
> +            data = comp_util.decompress(indata, 'lzma')
>          else:
>              data = indata
>          self.memlen = len(data)
> @@ -362,9 +362,9 @@ class CbfsFile(object):
>          elif self.ftype == TYPE_RAW:
>              orig_data = data
>              if self.compress == COMPRESS_LZ4:
> -                data = comp_util.compress(orig_data, 'lz4', with_header=False)
> +                data = comp_util.compress(orig_data, 'lz4')
>              elif self.compress == COMPRESS_LZMA:
> -                data = comp_util.compress(orig_data, 'lzma', with_header=False)
> +                data = comp_util.compress(orig_data, 'lzma')
>              self.memlen = len(orig_data)
>              self.data_len = len(data)
>              attr = struct.pack(ATTR_COMPRESSION_FORMAT,
> diff --git a/tools/binman/comp_util.py b/tools/binman/comp_util.py
> index dc76adab35..269bbf7975 100644
> --- a/tools/binman/comp_util.py
> +++ b/tools/binman/comp_util.py
> @@ -3,7 +3,6 @@
>  #
>  """Utilities to compress and decompress data"""
>
> -import struct
>  import tempfile
>
>  from binman import bintool
> @@ -16,7 +15,7 @@ LZMA_ALONE = bintool.Bintool.create('lzma_alone')
>  HAVE_LZMA_ALONE = LZMA_ALONE.is_present()
>
>
> -def compress(indata, algo, with_header=True):
> +def compress(indata, algo):
>      """Compress some data using a given algorithm
>
>      Note that for lzma this uses an old version of the algorithm, not that
> @@ -41,12 +40,9 @@ def compress(indata, algo, with_header=True):
>          data = LZMA_ALONE.compress(indata)
>      else:
>          raise ValueError("Unknown algorithm '%s'" % algo)
> -    if with_header:
> -        hdr = struct.pack('<I', len(data))
> -        data = hdr + data
>      return data
>
> -def decompress(indata, algo, with_header=True):
> +def decompress(indata, algo):
>      """Decompress some data using a given algorithm
>
>      Note that for lzma this uses an old version of the algorithm, not that
> @@ -64,9 +60,6 @@ def decompress(indata, algo, with_header=True):
>      """
>      if algo == 'none':
>          return indata
> -    if with_header:
> -        data_len = struct.unpack('<I', indata[:4])[0]
> -        indata = indata[4:4 + data_len]
>      if algo == 'lz4':
>          data = LZ4.decompress(indata)
>      elif algo == 'lzma':
> diff --git a/tools/binman/entry.py b/tools/binman/entry.py
> index a07a588864..8cbfd43af9 100644
> --- a/tools/binman/entry.py
> +++ b/tools/binman/entry.py
> @@ -1069,7 +1069,7 @@ features to produce new behaviours.
>              indata: Data to compress
>
>          Returns:
> -            Compressed data (first word is the compressed size)
> +            Compressed data
>          """
>          self.uncomp_data = indata
>          if self.compress != 'none':
> --
> 2.30.2
>

Hi Simon,

Am 02.08.2022 um 14:41 schrieb Simon Glass:
> Hi Stefan,
> 
> On Tue, 2 Aug 2022 at 06:29, Stefan Herbrechtsmeier
> <stefan.herbrechtsmeier-oss@weidmueller.com> wrote:
>>
>> From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
>>
>> Remove header from compressed data because this is uncommon, not
>> supported by U-Boot and incompatible with external compressed artifacts.
>>
>> The header was introduced as part of commit eb0f4a4cb402 ("binman:
>> Support replacing data in a cbfs") to allow device tree entries to be
>> larger that the compressed contents. Regarding the commit "this is
>> necessary to cope with a compressed device tree being updated in such a
>> way that it shrinks after the entry size is already set (an obscure
>> case)". This case need to be fixed without influence the compressed data
>> by itself.
> 
> I was not able to find a way around this due to the chicken-and egg
> problem. Compressed data has an unpredictable size and adding an extra
> uncompressed byte may increase or decrease the compressed size.

Is it possible to use the `pad-after` attribute to record the unused 
space. In this case it is possible to calculate the size of the 
compressed data.

Do you have a test for this use case?

> So my solution was to add the header.

Is the header used outside of binman? I don't spot it in the decompress 
fitImage implementation.


> It is optional though, so can we perhaps have a property in the
> description to enable it?

Is this header needed and supported outside of binman?

At the moment the header is incompatible and not well documented. It 
took me some time to find out why my gzip compression via binman doesn't 
work as expected because I assume a compatibility between binman 
compress and fitImage decompress.

Regards
   Stefan

Hi Stefan,

On Tue, 2 Aug 2022 at 07:45, Stefan Herbrechtsmeier
<stefan.herbrechtsmeier-oss@weidmueller.com> wrote:
>
> Hi Simon,
>
> Am 02.08.2022 um 14:41 schrieb Simon Glass:
> > Hi Stefan,
> >
> > On Tue, 2 Aug 2022 at 06:29, Stefan Herbrechtsmeier
> > <stefan.herbrechtsmeier-oss@weidmueller.com> wrote:
> >>
> >> From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
> >>
> >> Remove header from compressed data because this is uncommon, not
> >> supported by U-Boot and incompatible with external compressed artifacts.
> >>
> >> The header was introduced as part of commit eb0f4a4cb402 ("binman:
> >> Support replacing data in a cbfs") to allow device tree entries to be
> >> larger that the compressed contents. Regarding the commit "this is
> >> necessary to cope with a compressed device tree being updated in such a
> >> way that it shrinks after the entry size is already set (an obscure
> >> case)". This case need to be fixed without influence the compressed data
> >> by itself.
> >
> > I was not able to find a way around this due to the chicken-and egg
> > problem. Compressed data has an unpredictable size and adding an extra
> > uncompressed byte may increase or decrease the compressed size.
>
> Is it possible to use the `pad-after` attribute to record the unused
> space. In this case it is possible to calculate the size of the
> compressed data.

Well if you update that attribute it can change the size of the DTB
which is the chicken-and-egg problem again. As far as I know, if
people set the size of the region to something a bit larger than
needed, the problem is avoided. But the image generation does need to
be deterministic.

>
> Do you have a test for this use case?

There are various ones, e.g. testCompressDtb()

>
> > So my solution was to add the header.
>
> Is the header used outside of binman? I don't spot it in the decompress
> fitImage implementation.

It is used in the Chromium OS verified boot implementation, but not elsewhere.

>
>
> > It is optional though, so can we perhaps have a property in the
> > description to enable it?
>
> Is this header needed and supported outside of binman?
>
> At the moment the header is incompatible and not well documented. It
> took me some time to find out why my gzip compression via binman doesn't
> work as expected because I assume a compatibility between binman
> compress and fitImage decompress.

Yes I understand that, however you can pass a parameter to not include
the size value.

It would also be possible to add a property to the image (top-level
section) that indicates whether this field is present, such property
to apply to the whole image. We could have it default to off, if you
like.

Regards,
Simon

Hi Simon,

Am 03.08.2022 um 20:14 schrieb Simon Glass:
> Hi Stefan,
> 
> On Tue, 2 Aug 2022 at 07:45, Stefan Herbrechtsmeier
> <stefan.herbrechtsmeier-oss@weidmueller.com> wrote:
>>
>> Hi Simon,
>>
>> Am 02.08.2022 um 14:41 schrieb Simon Glass:
>>> Hi Stefan,
>>>
>>> On Tue, 2 Aug 2022 at 06:29, Stefan Herbrechtsmeier
>>> <stefan.herbrechtsmeier-oss@weidmueller.com> wrote:
>>>>
>>>> From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
>>>>
>>>> Remove header from compressed data because this is uncommon, not
>>>> supported by U-Boot and incompatible with external compressed artifacts.
>>>>
>>>> The header was introduced as part of commit eb0f4a4cb402 ("binman:
>>>> Support replacing data in a cbfs") to allow device tree entries to be
>>>> larger that the compressed contents. Regarding the commit "this is
>>>> necessary to cope with a compressed device tree being updated in such a
>>>> way that it shrinks after the entry size is already set (an obscure
>>>> case)". This case need to be fixed without influence the compressed data
>>>> by itself.
>>>
>>> I was not able to find a way around this due to the chicken-and egg
>>> problem. Compressed data has an unpredictable size and adding an extra
>>> uncompressed byte may increase or decrease the compressed size.
>>
>> Is it possible to use the `pad-after` attribute to record the unused
>> space. In this case it is possible to calculate the size of the
>> compressed data.
> 
> Well if you update that attribute it can change the size of the DTB
> which is the chicken-and-egg problem again. As far as I know, if
> people set the size of the region to something a bit larger than
> needed, the problem is avoided. But the image generation does need to
> be deterministic.

Does this means the size is only needed for the creation of the fitImage 
and not for decompression in u-boot?

> 
>>
>> Do you have a test for this use case?
> 
> There are various ones, e.g. testCompressDtb()

Thanks. Now I understand the problem and why you call it a 
chicken-and-egg problem. It wasn't clear to me that the attributes are 
inside the DTB.

But I wonder how the decompression of the DTB works if the fitImage 
implementation doesn't support the header.

>>> So my solution was to add the header.
>>
>> Is the header used outside of binman? I don't spot it in the decompress
>> fitImage implementation.
> 
> It is used in the Chromium OS verified boot implementation, but not elsewhere.
> 
>>
>>
>>> It is optional though, so can we perhaps have a property in the
>>> description to enable it?
>>
>> Is this header needed and supported outside of binman?
>>
>> At the moment the header is incompatible and not well documented. It
>> took me some time to find out why my gzip compression via binman doesn't
>> work as expected because I assume a compatibility between binman
>> compress and fitImage decompress.
> 
> Yes I understand that, however you can pass a parameter to not include
> the size value.

Do we need the header outside of the DTB? Otherwise we could handle this 
special or we could add a special compression type.

> It would also be possible to add a property to the image (top-level
> section) that indicates whether this field is present, such property
> to apply to the whole image. We could have it default to off, if you
> like.

Do we really need the header outside of the DTB entry?

Regards
   Stefan

Hi Stefan,

On Thu, 4 Aug 2022 at 01:50, Stefan Herbrechtsmeier
<stefan.herbrechtsmeier-oss@weidmueller.com> wrote:
>
> Hi Simon,
>
> Am 03.08.2022 um 20:14 schrieb Simon Glass:
> > Hi Stefan,
> >
> > On Tue, 2 Aug 2022 at 07:45, Stefan Herbrechtsmeier
> > <stefan.herbrechtsmeier-oss@weidmueller.com> wrote:
> >>
> >> Hi Simon,
> >>
> >> Am 02.08.2022 um 14:41 schrieb Simon Glass:
> >>> Hi Stefan,
> >>>
> >>> On Tue, 2 Aug 2022 at 06:29, Stefan Herbrechtsmeier
> >>> <stefan.herbrechtsmeier-oss@weidmueller.com> wrote:
> >>>>
> >>>> From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
> >>>>
> >>>> Remove header from compressed data because this is uncommon, not
> >>>> supported by U-Boot and incompatible with external compressed artifacts.
> >>>>
> >>>> The header was introduced as part of commit eb0f4a4cb402 ("binman:
> >>>> Support replacing data in a cbfs") to allow device tree entries to be
> >>>> larger that the compressed contents. Regarding the commit "this is
> >>>> necessary to cope with a compressed device tree being updated in such a
> >>>> way that it shrinks after the entry size is already set (an obscure
> >>>> case)". This case need to be fixed without influence the compressed data
> >>>> by itself.
> >>>
> >>> I was not able to find a way around this due to the chicken-and egg
> >>> problem. Compressed data has an unpredictable size and adding an extra
> >>> uncompressed byte may increase or decrease the compressed size.
> >>
> >> Is it possible to use the `pad-after` attribute to record the unused
> >> space. In this case it is possible to calculate the size of the
> >> compressed data.
> >
> > Well if you update that attribute it can change the size of the DTB
> > which is the chicken-and-egg problem again. As far as I know, if
> > people set the size of the region to something a bit larger than
> > needed, the problem is avoided. But the image generation does need to
> > be deterministic.
>
> Does this means the size is only needed for the creation of the fitImage
> and not for decompression in u-boot?

Possibly, but of course we cannot do that. As you say, U-Boot mainline
does not expect or support the header, at present.

>
> >
> >>
> >> Do you have a test for this use case?
> >
> > There are various ones, e.g. testCompressDtb()
>
> Thanks. Now I understand the problem and why you call it a
> chicken-and-egg problem. It wasn't clear to me that the attributes are
> inside the DTB.

OK good.

>
> But I wonder how the decompression of the DTB works if the fitImage
> implementation doesn't support the header.

It doesn't. Something needs to change here for compression to work.

>
> >>> So my solution was to add the header.
> >>
> >> Is the header used outside of binman? I don't spot it in the decompress
> >> fitImage implementation.
> >
> > It is used in the Chromium OS verified boot implementation, but not elsewhere.
> >
> >>
> >>
> >>> It is optional though, so can we perhaps have a property in the
> >>> description to enable it?
> >>
> >> Is this header needed and supported outside of binman?
> >>
> >> At the moment the header is incompatible and not well documented. It
> >> took me some time to find out why my gzip compression via binman doesn't
> >> work as expected because I assume a compatibility between binman
> >> compress and fitImage decompress.
> >
> > Yes I understand that, however you can pass a parameter to not include
> > the size value.
>
> Do we need the header outside of the DTB? Otherwise we could handle this
> special or we could add a special compression type.
>
> > It would also be possible to add a property to the image (top-level
> > section) that indicates whether this field is present, such property
> > to apply to the whole image. We could have it default to off, if you
> > like.
>
> Do we really need the header outside of the DTB entry?

That's an interesting question. It is possible that we only need it if
DTB is present and is compressed. It should be possible to check this
by adjusting the tests and checking for failures.

But I am not sure it is a good idea, since it is wildly inconsistent.
I do prefer things to be deterministic - i.e.  you specify what you
want and you get it. If binman starts adopting obscure conventions it
could be confusing.

Regards,
Simon

Hi Simon,

Am 04.08.2022 um 15:57 schrieb Simon Glass:
> Hi Stefan,
> 
> On Thu, 4 Aug 2022 at 01:50, Stefan Herbrechtsmeier
> <stefan.herbrechtsmeier-oss@weidmueller.com> wrote:
>>
>> Hi Simon,
>>
>> Am 03.08.2022 um 20:14 schrieb Simon Glass:
>>> Hi Stefan,
>>>
>>> On Tue, 2 Aug 2022 at 07:45, Stefan Herbrechtsmeier
>>> <stefan.herbrechtsmeier-oss@weidmueller.com> wrote:
>>>>
>>>> Hi Simon,
>>>>
>>>> Am 02.08.2022 um 14:41 schrieb Simon Glass:
>>>>> Hi Stefan,
>>>>>
>>>>> On Tue, 2 Aug 2022 at 06:29, Stefan Herbrechtsmeier
>>>>> <stefan.herbrechtsmeier-oss@weidmueller.com> wrote:
>>>>>>
>>>>>> From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
>>>>>>
>>>>>> Remove header from compressed data because this is uncommon, not
>>>>>> supported by U-Boot and incompatible with external compressed artifacts.
>>>>>>
>>>>>> The header was introduced as part of commit eb0f4a4cb402 ("binman:
>>>>>> Support replacing data in a cbfs") to allow device tree entries to be
>>>>>> larger that the compressed contents. Regarding the commit "this is
>>>>>> necessary to cope with a compressed device tree being updated in such a
>>>>>> way that it shrinks after the entry size is already set (an obscure
>>>>>> case)". This case need to be fixed without influence the compressed data
>>>>>> by itself.
>>>>>
>>>>> I was not able to find a way around this due to the chicken-and egg
>>>>> problem. Compressed data has an unpredictable size and adding an extra
>>>>> uncompressed byte may increase or decrease the compressed size.
>>>>
>>>> Is it possible to use the `pad-after` attribute to record the unused
>>>> space. In this case it is possible to calculate the size of the
>>>> compressed data.
>>>
>>> Well if you update that attribute it can change the size of the DTB
>>> which is the chicken-and-egg problem again. As far as I know, if
>>> people set the size of the region to something a bit larger than
>>> needed, the problem is avoided. But the image generation does need to
>>> be deterministic.
>>
>> Does this means the size is only needed for the creation of the fitImage
>> and not for decompression in u-boot?
> 
> Possibly, but of course we cannot do that. As you say, U-Boot mainline
> does not expect or support the header, at present.
> 
>>
>>>
>>>>
>>>> Do you have a test for this use case?
>>>
>>> There are various ones, e.g. testCompressDtb()
>>
>> Thanks. Now I understand the problem and why you call it a
>> chicken-and-egg problem. It wasn't clear to me that the attributes are
>> inside the DTB.
> 
> OK good.
> 
>>
>> But I wonder how the decompression of the DTB works if the fitImage
>> implementation doesn't support the header.
> 
> It doesn't. Something needs to change here for compression to work.
> 
>>
>>>>> So my solution was to add the header.
>>>>
>>>> Is the header used outside of binman? I don't spot it in the decompress
>>>> fitImage implementation.
>>>
>>> It is used in the Chromium OS verified boot implementation, but not elsewhere.
>>>
>>>>
>>>>
>>>>> It is optional though, so can we perhaps have a property in the
>>>>> description to enable it?
>>>>
>>>> Is this header needed and supported outside of binman?
>>>>
>>>> At the moment the header is incompatible and not well documented. It
>>>> took me some time to find out why my gzip compression via binman doesn't
>>>> work as expected because I assume a compatibility between binman
>>>> compress and fitImage decompress.
>>>
>>> Yes I understand that, however you can pass a parameter to not include
>>> the size value.
>>
>> Do we need the header outside of the DTB? Otherwise we could handle this
>> special or we could add a special compression type.
>>
>>> It would also be possible to add a property to the image (top-level
>>> section) that indicates whether this field is present, such property
>>> to apply to the whole image. We could have it default to off, if you
>>> like.
>>
>> Do we really need the header outside of the DTB entry?
> 
> That's an interesting question. It is possible that we only need it if
> DTB is present and is compressed. It should be possible to check this
> by adjusting the tests and checking for failures.
> 
> But I am not sure it is a good idea, since it is wildly inconsistent.
> I do prefer things to be deterministic - i.e.  you specify what you
> want and you get it. If binman starts adopting obscure conventions it
> could be confusing.

I add tests for gzip, lz4 and lzma_alone and all support padding at the 
end and don't need the header. Even the testCompressDtb works without 
the header. Furthermore I add bzip2, lzop, xz and zstd support to 
bintool and only zstd doesn't support padding. Do we really need the 
header or could we add an error if DTB is used together with zstd?

Should I commit bzip2, lzop, xz and zstd support to bintool?

Regards
   Stefan

Hi Stefan,

On Fri, 5 Aug 2022 at 01:51, Stefan Herbrechtsmeier
<stefan.herbrechtsmeier-oss@weidmueller.com> wrote:
>
> Hi Simon,
>
> Am 04.08.2022 um 15:57 schrieb Simon Glass:
> > Hi Stefan,
> >
> > On Thu, 4 Aug 2022 at 01:50, Stefan Herbrechtsmeier
> > <stefan.herbrechtsmeier-oss@weidmueller.com> wrote:
> >>
> >> Hi Simon,
> >>
> >> Am 03.08.2022 um 20:14 schrieb Simon Glass:
> >>> Hi Stefan,
> >>>
> >>> On Tue, 2 Aug 2022 at 07:45, Stefan Herbrechtsmeier
> >>> <stefan.herbrechtsmeier-oss@weidmueller.com> wrote:
> >>>>
> >>>> Hi Simon,
> >>>>
> >>>> Am 02.08.2022 um 14:41 schrieb Simon Glass:
> >>>>> Hi Stefan,
> >>>>>
> >>>>> On Tue, 2 Aug 2022 at 06:29, Stefan Herbrechtsmeier
> >>>>> <stefan.herbrechtsmeier-oss@weidmueller.com> wrote:
> >>>>>>
> >>>>>> From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
> >>>>>>
> >>>>>> Remove header from compressed data because this is uncommon, not
> >>>>>> supported by U-Boot and incompatible with external compressed artifacts.
> >>>>>>
> >>>>>> The header was introduced as part of commit eb0f4a4cb402 ("binman:
> >>>>>> Support replacing data in a cbfs") to allow device tree entries to be
> >>>>>> larger that the compressed contents. Regarding the commit "this is
> >>>>>> necessary to cope with a compressed device tree being updated in such a
> >>>>>> way that it shrinks after the entry size is already set (an obscure
> >>>>>> case)". This case need to be fixed without influence the compressed data
> >>>>>> by itself.
> >>>>>
> >>>>> I was not able to find a way around this due to the chicken-and egg
> >>>>> problem. Compressed data has an unpredictable size and adding an extra
> >>>>> uncompressed byte may increase or decrease the compressed size.
> >>>>
> >>>> Is it possible to use the `pad-after` attribute to record the unused
> >>>> space. In this case it is possible to calculate the size of the
> >>>> compressed data.
> >>>
> >>> Well if you update that attribute it can change the size of the DTB
> >>> which is the chicken-and-egg problem again. As far as I know, if
> >>> people set the size of the region to something a bit larger than
> >>> needed, the problem is avoided. But the image generation does need to
> >>> be deterministic.
> >>
> >> Does this means the size is only needed for the creation of the fitImage
> >> and not for decompression in u-boot?
> >
> > Possibly, but of course we cannot do that. As you say, U-Boot mainline
> > does not expect or support the header, at present.
> >
> >>
> >>>
> >>>>
> >>>> Do you have a test for this use case?
> >>>
> >>> There are various ones, e.g. testCompressDtb()
> >>
> >> Thanks. Now I understand the problem and why you call it a
> >> chicken-and-egg problem. It wasn't clear to me that the attributes are
> >> inside the DTB.
> >
> > OK good.
> >
> >>
> >> But I wonder how the decompression of the DTB works if the fitImage
> >> implementation doesn't support the header.
> >
> > It doesn't. Something needs to change here for compression to work.
> >
> >>
> >>>>> So my solution was to add the header.
> >>>>
> >>>> Is the header used outside of binman? I don't spot it in the decompress
> >>>> fitImage implementation.
> >>>
> >>> It is used in the Chromium OS verified boot implementation, but not elsewhere.
> >>>
> >>>>
> >>>>
> >>>>> It is optional though, so can we perhaps have a property in the
> >>>>> description to enable it?
> >>>>
> >>>> Is this header needed and supported outside of binman?
> >>>>
> >>>> At the moment the header is incompatible and not well documented. It
> >>>> took me some time to find out why my gzip compression via binman doesn't
> >>>> work as expected because I assume a compatibility between binman
> >>>> compress and fitImage decompress.
> >>>
> >>> Yes I understand that, however you can pass a parameter to not include
> >>> the size value.
> >>
> >> Do we need the header outside of the DTB? Otherwise we could handle this
> >> special or we could add a special compression type.
> >>
> >>> It would also be possible to add a property to the image (top-level
> >>> section) that indicates whether this field is present, such property
> >>> to apply to the whole image. We could have it default to off, if you
> >>> like.
> >>
> >> Do we really need the header outside of the DTB entry?
> >
> > That's an interesting question. It is possible that we only need it if
> > DTB is present and is compressed. It should be possible to check this
> > by adjusting the tests and checking for failures.
> >
> > But I am not sure it is a good idea, since it is wildly inconsistent.
> > I do prefer things to be deterministic - i.e.  you specify what you
> > want and you get it. If binman starts adopting obscure conventions it
> > could be confusing.
>
> I add tests for gzip, lz4 and lzma_alone and all support padding at the
> end and don't need the header. Even the testCompressDtb works without
> the header. Furthermore I add bzip2, lzop, xz and zstd support to
> bintool and only zstd doesn't support padding. Do we really need the
> header or could we add an error if DTB is used together with zstd?

OK great!

Yes, I tried pretty hard to avoid it, but could not make it work.
Would you like me to take a look at the situations that spark it? It
might be around replacing things, too.

I really tried to avoid banning things, since it is such a pain and
confusing for people. But since this is error (and a corner case) I
think it would be fine to require a particular property to enable the
advanced functionality.

>
> Should I commit bzip2, lzop, xz and zstd support to bintool?

Yes please.

Regards,
Simon

Hi Simon,

Am 05.08.2022 um 18:48 schrieb Simon Glass:
> Hi Stefan,
> 
> On Fri, 5 Aug 2022 at 01:51, Stefan Herbrechtsmeier
> <stefan.herbrechtsmeier-oss@weidmueller.com> wrote:
>>
>> Hi Simon,
>>
>> Am 04.08.2022 um 15:57 schrieb Simon Glass:
>>> Hi Stefan,
>>>
>>> On Thu, 4 Aug 2022 at 01:50, Stefan Herbrechtsmeier
>>> <stefan.herbrechtsmeier-oss@weidmueller.com> wrote:
>>>>
>>>> Hi Simon,
>>>>
>>>> Am 03.08.2022 um 20:14 schrieb Simon Glass:
>>>>> Hi Stefan,
>>>>>
>>>>> On Tue, 2 Aug 2022 at 07:45, Stefan Herbrechtsmeier
>>>>> <stefan.herbrechtsmeier-oss@weidmueller.com> wrote:
>>>>>>
>>>>>> Hi Simon,
>>>>>>
>>>>>> Am 02.08.2022 um 14:41 schrieb Simon Glass:
>>>>>>> Hi Stefan,
>>>>>>>
>>>>>>> On Tue, 2 Aug 2022 at 06:29, Stefan Herbrechtsmeier
>>>>>>> <stefan.herbrechtsmeier-oss@weidmueller.com> wrote:
>>>>>>>>
>>>>>>>> From: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
>>>>>>>>
>>>>>>>> Remove header from compressed data because this is uncommon, not
>>>>>>>> supported by U-Boot and incompatible with external compressed artifacts.
>>>>>>>>
>>>>>>>> The header was introduced as part of commit eb0f4a4cb402 ("binman:
>>>>>>>> Support replacing data in a cbfs") to allow device tree entries to be
>>>>>>>> larger that the compressed contents. Regarding the commit "this is
>>>>>>>> necessary to cope with a compressed device tree being updated in such a
>>>>>>>> way that it shrinks after the entry size is already set (an obscure
>>>>>>>> case)". This case need to be fixed without influence the compressed data
>>>>>>>> by itself.
>>>>>>>
>>>>>>> I was not able to find a way around this due to the chicken-and egg
>>>>>>> problem. Compressed data has an unpredictable size and adding an extra
>>>>>>> uncompressed byte may increase or decrease the compressed size.
>>>>>>
>>>>>> Is it possible to use the `pad-after` attribute to record the unused
>>>>>> space. In this case it is possible to calculate the size of the
>>>>>> compressed data.
>>>>>
>>>>> Well if you update that attribute it can change the size of the DTB
>>>>> which is the chicken-and-egg problem again. As far as I know, if
>>>>> people set the size of the region to something a bit larger than
>>>>> needed, the problem is avoided. But the image generation does need to
>>>>> be deterministic.
>>>>
>>>> Does this means the size is only needed for the creation of the fitImage
>>>> and not for decompression in u-boot?
>>>
>>> Possibly, but of course we cannot do that. As you say, U-Boot mainline
>>> does not expect or support the header, at present.
>>>
>>>>
>>>>>
>>>>>>
>>>>>> Do you have a test for this use case?
>>>>>
>>>>> There are various ones, e.g. testCompressDtb()
>>>>
>>>> Thanks. Now I understand the problem and why you call it a
>>>> chicken-and-egg problem. It wasn't clear to me that the attributes are
>>>> inside the DTB.
>>>
>>> OK good.
>>>
>>>>
>>>> But I wonder how the decompression of the DTB works if the fitImage
>>>> implementation doesn't support the header.
>>>
>>> It doesn't. Something needs to change here for compression to work.
>>>
>>>>
>>>>>>> So my solution was to add the header.
>>>>>>
>>>>>> Is the header used outside of binman? I don't spot it in the decompress
>>>>>> fitImage implementation.
>>>>>
>>>>> It is used in the Chromium OS verified boot implementation, but not elsewhere.
>>>>>
>>>>>>
>>>>>>
>>>>>>> It is optional though, so can we perhaps have a property in the
>>>>>>> description to enable it?
>>>>>>
>>>>>> Is this header needed and supported outside of binman?
>>>>>>
>>>>>> At the moment the header is incompatible and not well documented. It
>>>>>> took me some time to find out why my gzip compression via binman doesn't
>>>>>> work as expected because I assume a compatibility between binman
>>>>>> compress and fitImage decompress.
>>>>>
>>>>> Yes I understand that, however you can pass a parameter to not include
>>>>> the size value.
>>>>
>>>> Do we need the header outside of the DTB? Otherwise we could handle this
>>>> special or we could add a special compression type.
>>>>
>>>>> It would also be possible to add a property to the image (top-level
>>>>> section) that indicates whether this field is present, such property
>>>>> to apply to the whole image. We could have it default to off, if you
>>>>> like.
>>>>
>>>> Do we really need the header outside of the DTB entry?
>>>
>>> That's an interesting question. It is possible that we only need it if
>>> DTB is present and is compressed. It should be possible to check this
>>> by adjusting the tests and checking for failures.
>>>
>>> But I am not sure it is a good idea, since it is wildly inconsistent.
>>> I do prefer things to be deterministic - i.e.  you specify what you
>>> want and you get it. If binman starts adopting obscure conventions it
>>> could be confusing.
>>
>> I add tests for gzip, lz4 and lzma_alone and all support padding at the
>> end and don't need the header. Even the testCompressDtb works without
>> the header. Furthermore I add bzip2, lzop, xz and zstd support to
>> bintool and only zstd doesn't support padding. Do we really need the
>> header or could we add an error if DTB is used together with zstd?
> 
> OK great!
> 
> Yes, I tried pretty hard to avoid it, but could not make it work.
> Would you like me to take a look at the situations that spark it? It
> might be around replacing things, too.
> 
> I really tried to avoid banning things, since it is such a pain and
> confusing for people. But since this is error (and a corner case) I
> think it would be fine to require a particular property to enable the
> advanced functionality.

I have send a new version. Would be nice if you could check if you need 
the new attribute to append the length header.

>> Should I commit bzip2, lzop, xz and zstd support to bintool?

I have add the tools to the series. Maybe the test environment need an 
update to include all compression tools.

Regards
   Stefan