| Message ID | 20210211103807.9022-1-agust@denx.de |
|---|---|
| Headers | show |
| Series | Extend swupdate class to support swu image fstype | expand |
Hi Anatolji, On 11.02.21 11:38, Anatolij Gustschin wrote: > Building .swu images with meta-swupdate-boards does not work > when building them by devtool in installed eSDK: > > $ devtool build-image update-image > ... > ERROR: Specified recipe update-image is not an image recipe Well, in fact it is *NOT* an update recipe. It inherits swupdate, not image. The error is consistent. > > This series adds swu image fstype to swupdate class, so that it is > possible to generate .swu images via normal image recipes extended Ok, so you want for the very simple case (just rootfs in SWU) to generate quite automatically a SWU. But if we have more artifacts in SWU, I do not see many advantages for this, I still need to add the right DEPENDS and all required SWUPDATE_ flags. Is it just for lazy people ? > by appended options. With these patches applied, the swu image building > works in eSDK by "devtool build-image core-image-full-cmdline".> Patch 4 is an example how an image recipe can be extended to > enable .swu image generation (tested for 'wandboard' machine). > > Anatolij Gustschin (4): > swupdate_class: prepare to use SRC_URI in image recipes > swupdate_class: fix do_swuimage circular dependency > swupdate_class: enable building .swu images via swu image fstype > core-image: add example for .swu image generation > > classes/swupdate.bbclass | 44 ++++++++++++++++++- > .../images/core-image-full-cmdline.bbappend | 26 +++++++++++ > .../wandboard/sw-description | 43 ++++++++++++++++++ > 3 files changed, 111 insertions(+), 2 deletions(-) > create mode 100644 recipes-extended/images/core-image-full-cmdline.bbappend > create mode 100644 recipes-extended/images/core-image-full-cmdline/wandboard/sw-description > Regards, Stefano
Hi Stefano, On Thu, 11 Feb 2021 13:25:54 +0100 Stefano Babic sbabic@denx.de wrote: >Hi Anatolji, > >On 11.02.21 11:38, Anatolij Gustschin wrote: >> Building .swu images with meta-swupdate-boards does not work >> when building them by devtool in installed eSDK: >> >> $ devtool build-image update-image >> ... >> ERROR: Specified recipe update-image is not an image recipe > >Well, in fact it is *NOT* an update recipe. It inherits swupdate, not >image. The error is consistent. Yes, the error is expected here since this is not an image recipe. devtool in eSDK has two separate commands for building images and regular recipes: devtool build-image core-image-full-cmdline devtool build recipe-name Therefore building an .swu image does not work in eSDK environment. But a customer needs this and wants to build .swu via own image recipes using eSDK workflow. ... >> This series adds swu image fstype to swupdate class, so that it is >> possible to generate .swu images via normal image recipes extended > >Ok, so you want for the very simple case (just rootfs in SWU) to >generate quite automatically a SWU. But if we have more artifacts in >SWU, I do not see many advantages for this, I still need to add the >right DEPENDS and all required SWUPDATE_ flags. Is it just for lazy people ? This is mainly for a customer who wants to build .swu images in eSDK using different image recipes. This should be possible with mainline meta-swupdate. Thanks, Anatolij
Hi Anatolji, On 11.02.21 14:38, Anatolij Gustschin wrote: > Hi Stefano, > > On Thu, 11 Feb 2021 13:25:54 +0100 > Stefano Babic sbabic@denx.de wrote: > >> Hi Anatolji, >> >> On 11.02.21 11:38, Anatolij Gustschin wrote: >>> Building .swu images with meta-swupdate-boards does not work >>> when building them by devtool in installed eSDK: >>> >>> $ devtool build-image update-image >>> ... >>> ERROR: Specified recipe update-image is not an image recipe >> >> Well, in fact it is *NOT* an update recipe. It inherits swupdate, not >> image. The error is consistent. > > Yes, the error is expected here since this is not an image recipe. > devtool in eSDK has two separate commands for building images and > regular recipes: Ok > > devtool build-image core-image-full-cmdline clear > devtool build recipe-name but even devtool build update-image fails ? I have supposed this works. > > Therefore building an .swu image does not work in eSDK environment. > But a customer needs this and wants to build .swu via own image > recipes using eSDK workflow. > > ... >>> This series adds swu image fstype to swupdate class, so that it is >>> possible to generate .swu images via normal image recipes extended >> >> Ok, so you want for the very simple case (just rootfs in SWU) to >> generate quite automatically a SWU. But if we have more artifacts in >> SWU, I do not see many advantages for this, I still need to add the >> right DEPENDS and all required SWUPDATE_ flags. Is it just for lazy people ? > > This is mainly for a customer who wants to build .swu images in eSDK > using different image recipes. This should be possible with mainline > meta-swupdate. > > Thanks, > Anatolij >
Hi Stefano, On Thu, 11 Feb 2021 15:07:00 +0100 Stefano Babic sbabic@denx.de wrote: ... >> devtool build-image core-image-full-cmdline > >clear > >> devtool build recipe-name > >but even devtool build update-image fails ? I have supposed this works. Yes, it fails: $ devtool build update-image NOTE: Starting bitbake server... ERROR: No recipe named 'update-image' in your workspace When I prepare the recipe workspace by $ devtool modify update-image and try a build, I get: $ devtool build update-image NOTE: Starting bitbake server... NOTE: Reconnecting to bitbake server... NOTE: Retrying server connection (#1)... Loading cache: 100% |###############################################################################################################################################################################| Time: 0:00:00 Loaded 2539 entries from dependency cache. Parsing recipes: 100% |#############################################################################################################################################################################| Time: 0:00:00 Parsing of 1736 .bb files complete (1733 cached, 3 parsed). 2539 targets, 463 skipped, 0 masked, 0 errors. Loading cache: 100% |###############################################################################################################################################################################| Time: 0:00:01 Loaded 2538 entries from dependency cache. Parsing recipes: 100% |#############################################################################################################################################################################| Time: 0:00:00 Parsing of 1736 .bb files complete (1735 cached, 1 parsed). 2539 targets, 463 skipped, 0 masked, 0 errors. NOTE: Resolving any missing task queue dependencies ERROR: Task do_populate_sysroot does not exist for target update-image (/esdk/fslc-framebuffer_sdk/layers/poky/meta-swupdate-boards/recipes-extended/images/update-image.bb:do_populate_sysroot). Close matches: do_populate_lic ERROR: Command execution failed: 1 And if I comment out "deltask do_populate_sysroot" line in meta-swupdate/classes/swupdate.bbclass, the build command seems to complete, but there is no update-image*.swu in deploy directory: $ devtool build update-image NOTE: Starting bitbake server... NOTE: Reconnecting to bitbake server... NOTE: Retrying server connection (#1)... Loading cache: 100% |###############################################################################################################################################################################| Time: 0:00:00 Loaded 2538 entries from dependency cache. Parsing recipes: 100% |#############################################################################################################################################################################| Time: 0:00:00 Parsing of 1736 .bb files complete (1733 cached, 3 parsed). 2539 targets, 463 skipped, 0 masked, 0 errors. Loading cache: 100% |###############################################################################################################################################################################| Time: 0:00:01 Loaded 2538 entries from dependency cache. Parsing recipes: 100% |#############################################################################################################################################################################| Time: 0:00:00 Parsing of 1736 .bb files complete (1735 cached, 1 parsed). 2539 targets, 463 skipped, 0 masked, 0 errors. NOTE: Resolving any missing task queue dependencies Initialising tasks: 100% |##########################################################################################################################################################################| Time: 0:00:00 Sstate summary: Wanted 0 Found 0 Missed 0 Current 84 (0% match, 100% complete) NOTE: Executing Tasks NOTE: Tasks Summary: Attempted 388 tasks of which 388 didn't need to be rerun and all succeeded. $ ls tmp/deploy/images/wandboard/update-image* ls: cannot access 'tmp/deploy/images/wandboard/update-image*': No such file or directory Thanks, Anatolij
Hi Anatolij, Am 11.02.21 um 11:38 schrieb Anatolij Gustschin: > Building .swu images with meta-swupdate-boards does not work > when building them by devtool in installed eSDK: > > $ devtool build-image update-image > ... > ERROR: Specified recipe update-image is not an image recipe > > This series adds swu image fstype to swupdate class, so that it is > possible to generate .swu images via normal image recipes extended > by appended options. Is this a common approach? Is the content of the bbappend image or machine specific? The WIC image use dependency and configuration variables from the machine and thereby supports different images. Regards Stefan
Hi Stefano, Am 11.02.21 um 13:25 schrieb Stefano Babic: > Hi Anatolji, > > On 11.02.21 11:38, Anatolij Gustschin wrote: >> This series adds swu image fstype to swupdate class, so that it is >> possible to generate .swu images via normal image recipes extended > > Ok, so you want for the very simple case (just rootfs in SWU) to > generate quite automatically a SWU. But if we have more artifacts in > SWU, I do not see many advantages for this, I still need to add the > right DEPENDS and all required SWUPDATE_ flags. Is it just for lazy people ? I think an image type for swupdate is reasonable to support different images. The WIC image type supports different artifacts and could create file systems on the fly. At the moment it is possible to create an SD card with arbitrary deployed artifacts in different partitions with various filesystems for any image. Doing the same to create an swu update file is much more complicated. Regards Stefan
Hi Stefan, On 13.02.21 17:05, Stefan Herbrechtsmeier wrote: > Hi Stefano, > > Am 11.02.21 um 13:25 schrieb Stefano Babic: >> Hi Anatolji, >> >> On 11.02.21 11:38, Anatolij Gustschin wrote: >>> This series adds swu image fstype to swupdate class, so that it is >>> possible to generate .swu images via normal image recipes extended >> >> Ok, so you want for the very simple case (just rootfs in SWU) to >> generate quite automatically a SWU. But if we have more artifacts in >> SWU, I do not see many advantages for this, I still need to add the >> right DEPENDS and all required SWUPDATE_ flags. Is it just for lazy >> people ? > > I think an image type for swupdate is reasonable to support different > images. The WIC image type supports different artifacts and could create > file systems on the fly. At the moment it is possible to create an SD > card with arbitrary deployed artifacts in different partitions with > various filesystems for any image. Right. > Doing the same to create an swu > update file is much more complicated. Yes, I am just bothering how this can be done in the right way. There are sure a lot of projects where the SWU just contains the rootfs, and then a .swu is quite straightforward, but there are a raising number of projects (at least, this is my experience) where the SWU contains update for separate entities on the same device (i.e. microcontroller firmwares) or even to forward software to other devices. At the end, I come with a very specialized and project specific sw-description that fits that project, but it cannot be reused on other projects. I do not know how a general .swu image type can be factorized as Anatolji is trying to do. Regards, Stefano
Hi Stefano, Am 14.02.21 um 10:34 schrieb Stefano Babic: > Hi Stefan, > > On 13.02.21 17:05, Stefan Herbrechtsmeier wrote: >> Hi Stefano, >> >> Am 11.02.21 um 13:25 schrieb Stefano Babic: >>> Hi Anatolji, >>> >>> On 11.02.21 11:38, Anatolij Gustschin wrote: >>>> This series adds swu image fstype to swupdate class, so that it is >>>> possible to generate .swu images via normal image recipes extended >>> >>> Ok, so you want for the very simple case (just rootfs in SWU) to >>> generate quite automatically a SWU. But if we have more artifacts in >>> SWU, I do not see many advantages for this, I still need to add the >>> right DEPENDS and all required SWUPDATE_ flags. Is it just for lazy >>> people ? >> >> I think an image type for swupdate is reasonable to support different >> images. The WIC image type supports different artifacts and could >> create file systems on the fly. At the moment it is possible to create >> an SD card with arbitrary deployed artifacts in different partitions >> with various filesystems for any image. > > Right. > >> Doing the same to create an swu update file is much more complicated. > > Yes, I am just bothering how this can be done in the right way. There > are sure a lot of projects where the SWU just contains the rootfs, and > then a .swu is quite straightforward, but there are a raising number of > projects (at least, this is my experience) where the SWU contains update > for separate entities on the same device (i.e. microcontroller > firmwares) or even to forward software to other devices. At the end, I > come with a very specialized and project specific sw-description that > fits that project, but it cannot be reused on other projects. I do not > know how a general .swu image type can be factorized as Anatolji is > trying to do. But is this really a problem for an swu image class? The main difference is how you define you configuration. In most cases an image class use global configurations from the machine configuration and in case of a wic image a shared wks configuration file from a common directory. At the moment swupdate use a special recipe on top of an other main image recipe. I think that the main problems are the scatter information in the sw-description and the recipe. Other classes generate the configuration file on the fly (ex. kernel-fitimage) or use a separate configuration file (ex. WIC). Regards Stefan
Hi Stefan, On 14.02.21 16:11, Stefan Herbrechtsmeier wrote: > Hi Stefano, > > Am 14.02.21 um 10:34 schrieb Stefano Babic: >> Hi Stefan, >> >> On 13.02.21 17:05, Stefan Herbrechtsmeier wrote: >>> Hi Stefano, >>> >>> Am 11.02.21 um 13:25 schrieb Stefano Babic: >>>> Hi Anatolji, >>>> >>>> On 11.02.21 11:38, Anatolij Gustschin wrote: >>>>> This series adds swu image fstype to swupdate class, so that it is >>>>> possible to generate .swu images via normal image recipes extended >>>> >>>> Ok, so you want for the very simple case (just rootfs in SWU) to >>>> generate quite automatically a SWU. But if we have more artifacts in >>>> SWU, I do not see many advantages for this, I still need to add the >>>> right DEPENDS and all required SWUPDATE_ flags. Is it just for lazy >>>> people ? >>> >>> I think an image type for swupdate is reasonable to support different >>> images. The WIC image type supports different artifacts and could >>> create file systems on the fly. At the moment it is possible to >>> create an SD card with arbitrary deployed artifacts in different >>> partitions with various filesystems for any image. >> >> Right. >> >>> Doing the same to create an swu update file is much more complicated. >> >> Yes, I am just bothering how this can be done in the right way. There >> are sure a lot of projects where the SWU just contains the rootfs, and >> then a .swu is quite straightforward, but there are a raising number >> of projects (at least, this is my experience) where the SWU contains >> update for separate entities on the same device (i.e. microcontroller >> firmwares) or even to forward software to other devices. At the end, I >> come with a very specialized and project specific sw-description that >> fits that project, but it cannot be reused on other projects. I do not >> know how a general .swu image type can be factorized as Anatolji is >> trying to do. > > But is this really a problem for an swu image class? The main difference > is how you define you configuration. In most cases an image class use > global configurations from the machine configuration and in case of a > wic image a shared wks configuration file from a common directory. At > the moment swupdate use a special recipe on top of an other main image > recipe. A global (MACHINE or DISTRO) configuration covers just the simple cases, but there are many projects creating multiple SWUs. So a MACHINE configuration does not work in many cases. The difference with a WKS is that we need here a configuration for the device (sw-description), that is also interpreted by the device, and a configuration for the build system, that should not go on the device (that means, the recipe), and the build system can require some part from sw-description (or in any cases, it fills automatically some parts). I cannot say that SWUpdate use a recipe on top of others, it is a recipe that depends on one (simple rootfs) or many other recipes, and in some cases even from images generated for other OE machines. > > I think that the main problems are the scatter information in the > sw-description and the recipe. Yes, but they contain information for different actors, the device (sw-description) and the buildsystem (the recipe). Information in recipe are useless for the device, but the buildsystem can use some information from sw-description (and it extends it). > Other classes generate the configuration > file on the fly (ex. kernel-fitimage) well, let me say that I am not so fond of it. It creates a fitImage for most cases with a hard-coded and generated .its, that is kernel+DT+Ramdisk, but there are some other cases where additional images must be packed into a fitImage and the kernel class is not configurable for such things (i.e. fpga bitstreams, etc.). Anyway, in most cases a fitImage consists of kernel+DT, and then at least 90% of cases are covered - I do not see the same in SWUpdate. > or use a separate configuration > file (ex. WIC). Right, but wks is a configuration file for the buildsystem and the device does not need to bother with it. We have a use case here more similar to fitImage, but where there is quite no "standard" case like fitImage to allow the generation of a sw-description. Regards, Stefano
Hi Stefano, Am 14.02.21 um 18:15 schrieb Stefano Babic: > Hi Stefan, > > On 14.02.21 16:11, Stefan Herbrechtsmeier wrote: >> Hi Stefano, >> >> Am 14.02.21 um 10:34 schrieb Stefano Babic: >>> Hi Stefan, >>> >>> On 13.02.21 17:05, Stefan Herbrechtsmeier wrote: >>>> Hi Stefano, >>>> >>>> Am 11.02.21 um 13:25 schrieb Stefano Babic: >>>>> Hi Anatolji, >>>>> >>>>> On 11.02.21 11:38, Anatolij Gustschin wrote: >>>>>> This series adds swu image fstype to swupdate class, so that it is >>>>>> possible to generate .swu images via normal image recipes extended >>>>> >>>>> Ok, so you want for the very simple case (just rootfs in SWU) to >>>>> generate quite automatically a SWU. But if we have more artifacts in >>>>> SWU, I do not see many advantages for this, I still need to add the >>>>> right DEPENDS and all required SWUPDATE_ flags. Is it just for lazy >>>>> people ? >>>> >>>> I think an image type for swupdate is reasonable to support >>>> different images. The WIC image type supports different artifacts >>>> and could create file systems on the fly. At the moment it is >>>> possible to create an SD card with arbitrary deployed artifacts in >>>> different partitions with various filesystems for any image. >>> >>> Right. >>> >>>> Doing the same to create an swu update file is much more complicated. >>> >>> Yes, I am just bothering how this can be done in the right way. There >>> are sure a lot of projects where the SWU just contains the rootfs, >>> and then a .swu is quite straightforward, but there are a raising >>> number of projects (at least, this is my experience) where the SWU >>> contains update for separate entities on the same device (i.e. >>> microcontroller firmwares) or even to forward software to other >>> devices. At the end, I come with a very specialized and project >>> specific sw-description that fits that project, but it cannot be >>> reused on other projects. I do not know how a general .swu image type >>> can be factorized as Anatolji is trying to do. >> >> But is this really a problem for an swu image class? The main >> difference is how you define you configuration. In most cases an image >> class use global configurations from the machine configuration and in >> case of a wic image a shared wks configuration file from a common >> directory. At the moment swupdate use a special recipe on top of an >> other main image recipe. > > A global (MACHINE or DISTRO) configuration covers just the simple cases, > but there are many projects creating multiple SWUs. So a MACHINE > configuration does not work in many cases. But a global configuration could be override in recipes or you could share it between multiple recipes. It should work for the simple and the complex solutions. > The difference with a WKS is that we need here a configuration for the > device (sw-description), that is also interpreted by the device, and a > configuration for the build system, that should not go on the device > (that means, the recipe), and the build system can require some part > from sw-description (or in any cases, it fills automatically some > parts). I cannot say that SWUpdate use a recipe on top of others, it is > a recipe that depends on one (simple rootfs) or many other recipes, and > in some cases even from images generated for other OE machines. The WKS also need some information for the device but they are generated / extracted from the configuration and the WKS could also depends on other recipes and there artifacts. At the moment I need an image recipe and an extra vfat image type class to create a boot partition. Furthermore I need a update recipe per each image recipe. >> I think that the main problems are the scatter information in the >> sw-description and the recipe. > > Yes, but they contain information for different actors, the device > (sw-description) and the buildsystem (the recipe). Information in recipe > are useless for the device, but the buildsystem can use some information > from sw-description (and it extends it). > >> Other classes generate the configuration file on the fly (ex. >> kernel-fitimage) > > well, let me say that I am not so fond of it. It creates a fitImage for > most cases with a hard-coded and generated .its, that is > kernel+DT+Ramdisk, but there are some other cases where additional > images must be packed into a fitImage and the kernel class is not > configurable for such things (i.e. fpga bitstreams, etc.). Anyway, in > most cases a fitImage consists of kernel+DT, and then at least 90% of > cases are covered - I do not see the same in SWUpdate. You are right a sw-description generator will become very complex. >> or use a separate configuration file (ex. WIC). > > Right, but wks is a configuration file for the buildsystem and the > device does not need to bother with it. We have a use case here more > similar to fitImage, but where there is quite no "standard" case like > fitImage to allow the generation of a sw-description. The problem is not the missing standard case but the flexibility of our configuration. Maybe we could define a swu.in or sw-description.in configuration which extends the sw-description with build information? Something like a source entry for the image, file and script entries. We could reuse the filename to rename the files and the compressed and encrypted entry to prepare the files. Furthermore we could add a filesystem and name entry to an image to create a file system on the fly from IMAGE_BOOT_FILES. Regards Stefan
Hi Stefan, On Sat, 13 Feb 2021 16:39:03 +0100 Stefan Herbrechtsmeier stefan@herbrechtsmeier.net wrote: >Hi Anatolij, > >Am 11.02.21 um 11:38 schrieb Anatolij Gustschin: >> Building .swu images with meta-swupdate-boards does not work >> when building them by devtool in installed eSDK: >> >> $ devtool build-image update-image >> ... >> ERROR: Specified recipe update-image is not an image recipe >> >> This series adds swu image fstype to swupdate class, so that it is >> possible to generate .swu images via normal image recipes extended >> by appended options. > >Is this a common approach? Is the content of the bbappend image or >machine specific? The WIC image use dependency and configuration >variables from the machine and thereby supports different images. The content is image and machine specific. bbappend is just an example here to demonstrate the content for .swu generation. For custom images I expect that the swu specific options will be added to the usual image files directly. I think it is better to add bbappend in patch 4 to meta-swupdate-boards, so for only meta-swupdate users the core-image-full-cmdline image will not change. Will to this in v2 series. Thanks, Anatolij
Hi Anatolji, On 15.02.21 23:49, Anatolij Gustschin wrote: > Hi Stefan, > > On Sat, 13 Feb 2021 16:39:03 +0100 > Stefan Herbrechtsmeier stefan@herbrechtsmeier.net wrote: > >> Hi Anatolij, >> >> Am 11.02.21 um 11:38 schrieb Anatolij Gustschin: >>> Building .swu images with meta-swupdate-boards does not work >>> when building them by devtool in installed eSDK: >>> >>> $ devtool build-image update-image >>> ... >>> ERROR: Specified recipe update-image is not an image recipe >>> >>> This series adds swu image fstype to swupdate class, so that it is >>> possible to generate .swu images via normal image recipes extended >>> by appended options. >> >> Is this a common approach? Is the content of the bbappend image or >> machine specific? The WIC image use dependency and configuration >> variables from the machine and thereby supports different images. > > The content is image and machine specific. bbappend is just an > example here to demonstrate the content for .swu generation. > For custom images I expect that the swu specific options will be > added to the usual image files directly. Right, else this extension has no meaning. > I think it is better > to add bbappend in patch 4 to meta-swupdate-boards, so for only > meta-swupdate users the core-image-full-cmdline image will not > change. Will to this in v2 series. But I am expecting that boards and more important customers who do not use IMAGE_FSTYPES += "swu" wont be touched by this series, else we can have compatibility problems. Regards, Stefano
Hi Stefan, On 15.02.21 20:21, Stefan Herbrechtsmeier wrote: > Hi Stefano, > > Am 14.02.21 um 18:15 schrieb Stefano Babic: >> Hi Stefan, >> >> On 14.02.21 16:11, Stefan Herbrechtsmeier wrote: >>> Hi Stefano, >>> >>> Am 14.02.21 um 10:34 schrieb Stefano Babic: >>>> Hi Stefan, >>>> >>>> On 13.02.21 17:05, Stefan Herbrechtsmeier wrote: >>>>> Hi Stefano, >>>>> >>>>> Am 11.02.21 um 13:25 schrieb Stefano Babic: >>>>>> Hi Anatolji, >>>>>> >>>>>> On 11.02.21 11:38, Anatolij Gustschin wrote: >>>>>>> This series adds swu image fstype to swupdate class, so that it is >>>>>>> possible to generate .swu images via normal image recipes extended >>>>>> >>>>>> Ok, so you want for the very simple case (just rootfs in SWU) to >>>>>> generate quite automatically a SWU. But if we have more artifacts in >>>>>> SWU, I do not see many advantages for this, I still need to add the >>>>>> right DEPENDS and all required SWUPDATE_ flags. Is it just for >>>>>> lazy people ? >>>>> >>>>> I think an image type for swupdate is reasonable to support >>>>> different images. The WIC image type supports different artifacts >>>>> and could create file systems on the fly. At the moment it is >>>>> possible to create an SD card with arbitrary deployed artifacts in >>>>> different partitions with various filesystems for any image. >>>> >>>> Right. >>>> >>>>> Doing the same to create an swu update file is much more complicated. >>>> >>>> Yes, I am just bothering how this can be done in the right way. >>>> There are sure a lot of projects where the SWU just contains the >>>> rootfs, and then a .swu is quite straightforward, but there are a >>>> raising number of projects (at least, this is my experience) where >>>> the SWU contains update for separate entities on the same device >>>> (i.e. microcontroller firmwares) or even to forward software to >>>> other devices. At the end, I come with a very specialized and >>>> project specific sw-description that fits that project, but it >>>> cannot be reused on other projects. I do not know how a general .swu >>>> image type can be factorized as Anatolji is trying to do. >>> >>> But is this really a problem for an swu image class? The main >>> difference is how you define you configuration. In most cases an >>> image class use global configurations from the machine configuration >>> and in case of a wic image a shared wks configuration file from a >>> common directory. At the moment swupdate use a special recipe on top >>> of an other main image recipe. >> >> A global (MACHINE or DISTRO) configuration covers just the simple >> cases, but there are many projects creating multiple SWUs. So a >> MACHINE configuration does not work in many cases. > > But a global configuration could be override in recipes or you could > share it between multiple recipes. Then it is a .inc file that we can include in multiple recipes, but for its nature, a SWU configuration is related to an image, and not to a MACHINE / DISTRO. > It should work for the simple and the > complex solutions. .inc file, yes, but SWU variables in a MACHINE config is quite misleading, because this should mean it is machine specific, and it is really more image specific. > > >> The difference with a WKS is that we need here a configuration for the >> device (sw-description), that is also interpreted by the device, and a >> configuration for the build system, that should not go on the device >> (that means, the recipe), and the build system can require some part >> from sw-description (or in any cases, it fills automatically some >> parts). I cannot say that SWUpdate use a recipe on top of others, it is >> a recipe that depends on one (simple rootfs) or many other recipes, >> and in some cases even from images generated for other OE machines. > > The WKS also need some information for the device but they are generated > / extracted from the configuration and the WKS could also depends on > other recipes and there artifacts. But the device has nothing to do with it, while in case of SWU, sw-description is the description of the release and must understand it. > > At the moment I need an image recipe and an extra vfat image type class > to create a boot partition. Ok, this is to generate the .wic for the first setup of the device. I guess this cannot be avoided. > Furthermore I need a update recipe per each > image recipe. What I do is to have a .inc file shared by all update recipes, so that the update recipes are very simple. But yes, an update recipe per image is necessary, even if quite empty. > >>> I think that the main problems are the scatter information in the >>> sw-description and the recipe. >> >> Yes, but they contain information for different actors, the device >> (sw-description) and the buildsystem (the recipe). Information in >> recipe are useless for the device, but the buildsystem can use some >> information from sw-description (and it extends it). >> >>> Other classes generate the configuration file on the fly (ex. >>> kernel-fitimage) >> >> well, let me say that I am not so fond of it. It creates a fitImage >> for most cases with a hard-coded and generated .its, that is >> kernel+DT+Ramdisk, but there are some other cases where additional >> images must be packed into a fitImage and the kernel class is not >> configurable for such things (i.e. fpga bitstreams, etc.). Anyway, in >> most cases a fitImage consists of kernel+DT, and then at least 90% of >> cases are covered - I do not see the same in SWUpdate. > > You are right a sw-description generator will become very complex. > IMHO sw-description is code instead of simple configuration, and it is written by developers / integrators, because requirements and use case strongly differ. For example, a customer of mine wants to group some artifacts (U-Boot, firmware, a blob) together and update all of them if one of the artifacts belonging to the group needs to be updated due to version mismatch. Some requirements can be satisfied using hooks in sw-description and embedded Lua code. An automatic generation of sw-description is then impossible but very simple cases (aka update of just rootfs). >>> or use a separate configuration file (ex. WIC). >> >> Right, but wks is a configuration file for the buildsystem and the >> device does not need to bother with it. We have a use case here more >> similar to fitImage, but where there is quite no "standard" case like >> fitImage to allow the generation of a sw-description. > > The problem is not the missing standard case but the flexibility of our > configuration. > > Maybe we could define a swu.in or sw-description.in configuration which > extends the sw-description with build information? I do not know how, information must be removed then because they should not reach the device. And there is not just OE, but also Buildroot and Debian. I am also careful to change sw-description: it is ok to add new features / attributes, but they should not break update for devices already in field if they do not use the new features. > Something like a > source entry for the image, file and script entries. With just a hook, it does not work, I think. And handler must be specified (what about to update a UBIFS instead of eMMC ?), at the end it is a sw-description written in another format. > We could reuse the > filename to rename the files and the compressed and encrypted entry to > prepare the files. Furthermore we could add a filesystem and name entry > to an image to create a file system on the fly from IMAGE_BOOT_FILES. > Regards, Stefano
Hi Stefano, Am 16.02.21 um 09:02 schrieb Stefano Babic: > Hi Stefan, > > On 15.02.21 20:21, Stefan Herbrechtsmeier wrote: >> Hi Stefano, >> >> Am 14.02.21 um 18:15 schrieb Stefano Babic: >>> Hi Stefan, >>> >>> On 14.02.21 16:11, Stefan Herbrechtsmeier wrote: >>>> Hi Stefano, >>>> >>>> Am 14.02.21 um 10:34 schrieb Stefano Babic: >>>>> Hi Stefan, >>>>> >>>>> On 13.02.21 17:05, Stefan Herbrechtsmeier wrote: >>>>>> Hi Stefano, >>>>>> >>>>>> Am 11.02.21 um 13:25 schrieb Stefano Babic: >>>>>>> Hi Anatolji, >>>>>>> >>>>>>> On 11.02.21 11:38, Anatolij Gustschin wrote: >>>>>>>> This series adds swu image fstype to swupdate class, so that it is >>>>>>>> possible to generate .swu images via normal image recipes extended >>>>>>> >>>>>>> Ok, so you want for the very simple case (just rootfs in SWU) to >>>>>>> generate quite automatically a SWU. But if we have more artifacts in >>>>>>> SWU, I do not see many advantages for this, I still need to add the >>>>>>> right DEPENDS and all required SWUPDATE_ flags. Is it just for >>>>>>> lazy people ? >>>>>> >>>>>> I think an image type for swupdate is reasonable to support >>>>>> different images. The WIC image type supports different artifacts >>>>>> and could create file systems on the fly. At the moment it is >>>>>> possible to create an SD card with arbitrary deployed artifacts in >>>>>> different partitions with various filesystems for any image. >>>>> >>>>> Right. >>>>> >>>>>> Doing the same to create an swu update file is much more complicated. >>>>> >>>>> Yes, I am just bothering how this can be done in the right way. >>>>> There are sure a lot of projects where the SWU just contains the >>>>> rootfs, and then a .swu is quite straightforward, but there are a >>>>> raising number of projects (at least, this is my experience) where >>>>> the SWU contains update for separate entities on the same device >>>>> (i.e. microcontroller firmwares) or even to forward software to >>>>> other devices. At the end, I come with a very specialized and >>>>> project specific sw-description that fits that project, but it >>>>> cannot be reused on other projects. I do not know how a general >>>>> .swu image type can be factorized as Anatolji is trying to do. >>>> >>>> But is this really a problem for an swu image class? The main >>>> difference is how you define you configuration. In most cases an >>>> image class use global configurations from the machine configuration >>>> and in case of a wic image a shared wks configuration file from a >>>> common directory. At the moment swupdate use a special recipe on top >>>> of an other main image recipe. >>> >>> A global (MACHINE or DISTRO) configuration covers just the simple >>> cases, but there are many projects creating multiple SWUs. So a >>> MACHINE configuration does not work in many cases. >> >> But a global configuration could be override in recipes or you could >> share it between multiple recipes. > > Then it is a .inc file that we can include in multiple recipes, but for > its nature, a SWU configuration is related to an image, and not to a > MACHINE / DISTRO. Does newer versions of OE support files relative to an include? Older versions doesn't set the THISDIR for a inc file. Why is a swu related to an image? What is the difference between a sd image and an update in terms of it content? Both contain a rootfs and some additional images like bootloader or firmware. Why is an update related to an image? Does the content of the rootfs really matter? >> It should work for the simple and the complex solutions. > > .inc file, yes, but SWU variables in a MACHINE config is quite > misleading, because this should mean it is machine specific, and it is > really more image specific. An .inc file is hard to override. A configuration file could be reused between images and it is possible to provide a default configuration for a machine or distro. >>> The difference with a WKS is that we need here a configuration for >>> the device (sw-description), that is also interpreted by the device, >>> and a configuration for the build system, that should not go on the >>> device (that means, the recipe), and the build system can require >>> some part from sw-description (or in any cases, it fills >>> automatically some parts). I cannot say that SWUpdate use a recipe on >>> top of others, it is >>> a recipe that depends on one (simple rootfs) or many other recipes, >>> and in some cases even from images generated for other OE machines. >> >> The WKS also need some information for the device but they are >> generated / extracted from the configuration and the WKS could also >> depends on other recipes and there artifacts. > > But the device has nothing to do with it, while in case of SWU, > sw-description is the description of the release and must understand it. What do you mean? >> At the moment I need an image recipe and an extra vfat image type >> class to create a boot partition. > > Ok, this is to generate the .wic for the first setup of the device. I > guess this cannot be avoided. WIC avoid this problem and generates the file system on the fly. Furthermore the update of a full filesystem instead of its content have advantages and this leads to images for every partition. >> Furthermore I need a update recipe per each image recipe. > > What I do is to have a .inc file shared by all update recipes, so that > the update recipes are very simple. But yes, an update recipe per image > is necessary, even if quite empty. But you could avoid this with a swu image class and a configuration file. >>>> I think that the main problems are the scatter information in the >>>> sw-description and the recipe. >>> >>> Yes, but they contain information for different actors, the device >>> (sw-description) and the buildsystem (the recipe). Information in >>> recipe are useless for the device, but the buildsystem can use some >>> information from sw-description (and it extends it). >>> >>>> Other classes generate the configuration file on the fly (ex. >>>> kernel-fitimage) >>> >>> well, let me say that I am not so fond of it. It creates a fitImage >>> for most cases with a hard-coded and generated .its, that is >>> kernel+DT+Ramdisk, but there are some other cases where additional >>> images must be packed into a fitImage and the kernel class is not >>> configurable for such things (i.e. fpga bitstreams, etc.). Anyway, in >>> most cases a fitImage consists of kernel+DT, and then at least 90% of >>> cases are covered - I do not see the same in SWUpdate. >> >> You are right a sw-description generator will become very complex. >> > > IMHO sw-description is code instead of simple configuration, and it is > written by developers / integrators, because requirements and use case > strongly differ. For example, a customer of mine wants to group some > artifacts (U-Boot, firmware, a blob) together and update all of them if > one of the artifacts belonging to the group needs to be updated due to > version mismatch. Some requirements can be satisfied using hooks in > sw-description and embedded Lua code. An automatic generation of > sw-description is then impossible but very simple cases (aka update of > just rootfs). So we should keep the sw-description. The question is how we could define the build steps. >>>> or use a separate configuration file (ex. WIC). >>> >>> Right, but wks is a configuration file for the buildsystem and the >>> device does not need to bother with it. We have a use case here more >>> similar to fitImage, but where there is quite no "standard" case like >>> fitImage to allow the generation of a sw-description. >> >> The problem is not the missing standard case but the flexibility of >> our configuration. >> >> Maybe we could define a swu.in or sw-description.in configuration >> which extends the sw-description with build information? > > I do not know how, information must be removed then because they should > not reach the device. And there is not just OE, but also Buildroot and > Debian. So you prefare a common solution like an swu build tool? > I am also careful to change sw-description: it is ok to add new features > / attributes, but they should not break update for devices already in > field if they do not use the new features. The build information shouldn't influence the update and should be removed from the shipped sw-description. >> Something like a source entry for the image, file and script entries. > > With just a hook, it does not work, I think. And handler must be > specified (what about to update a UBIFS instead of eMMC ?), at the end > it is a sw-description written in another format. I don't speak about an other file format in front of sw-description. >> We could reuse the filename to rename the files and the compressed and >> encrypted entry to prepare the files. Furthermore we could add a >> filesystem and name entry to an image to create a file system on the >> fly from IMAGE_BOOT_FILES. >> My idea is to add build information into the sw-description. I have rethink the solution and would use specific comments as decorations / annotation inside the sw-description. This makes it easy to extract and remove the build information from the sw-description. Each decorations include a name for the file and an optional compression and encryption. We could support raw files from the deploy directory, on the fly filesystems via IMAGE_BOOT_FILES and the rootfs: //! boot.bin.gz --source=rawcopy --sourceparams="file=boot-@MACHINE@.bin" --compression=zlib //! boot.vfat.gz --source=bootimg --fstype=vfat --compression=zlib //! rootfs.ext4.gz --source=rootfs --fstype=ext4 --compression=zlib Regards Stefan
Hi Stefan, On 16.02.21 21:34, Stefan Herbrechtsmeier wrote: > Hi Stefano, > > Am 16.02.21 um 09:02 schrieb Stefano Babic: >> Hi Stefan, >> >> On 15.02.21 20:21, Stefan Herbrechtsmeier wrote: >>> Hi Stefano, >>> >>> Am 14.02.21 um 18:15 schrieb Stefano Babic: >>>> Hi Stefan, >>>> >>>> On 14.02.21 16:11, Stefan Herbrechtsmeier wrote: >>>>> Hi Stefano, >>>>> >>>>> Am 14.02.21 um 10:34 schrieb Stefano Babic: >>>>>> Hi Stefan, >>>>>> >>>>>> On 13.02.21 17:05, Stefan Herbrechtsmeier wrote: >>>>>>> Hi Stefano, >>>>>>> >>>>>>> Am 11.02.21 um 13:25 schrieb Stefano Babic: >>>>>>>> Hi Anatolji, >>>>>>>> >>>>>>>> On 11.02.21 11:38, Anatolij Gustschin wrote: >>>>>>>>> This series adds swu image fstype to swupdate class, so that it is >>>>>>>>> possible to generate .swu images via normal image recipes extended >>>>>>>> >>>>>>>> Ok, so you want for the very simple case (just rootfs in SWU) to >>>>>>>> generate quite automatically a SWU. But if we have more >>>>>>>> artifacts in >>>>>>>> SWU, I do not see many advantages for this, I still need to add the >>>>>>>> right DEPENDS and all required SWUPDATE_ flags. Is it just for >>>>>>>> lazy people ? >>>>>>> >>>>>>> I think an image type for swupdate is reasonable to support >>>>>>> different images. The WIC image type supports different artifacts >>>>>>> and could create file systems on the fly. At the moment it is >>>>>>> possible to create an SD card with arbitrary deployed artifacts >>>>>>> in different partitions with various filesystems for any image. >>>>>> >>>>>> Right. >>>>>> >>>>>>> Doing the same to create an swu update file is much more >>>>>>> complicated. >>>>>> >>>>>> Yes, I am just bothering how this can be done in the right way. >>>>>> There are sure a lot of projects where the SWU just contains the >>>>>> rootfs, and then a .swu is quite straightforward, but there are a >>>>>> raising number of projects (at least, this is my experience) where >>>>>> the SWU contains update for separate entities on the same device >>>>>> (i.e. microcontroller firmwares) or even to forward software to >>>>>> other devices. At the end, I come with a very specialized and >>>>>> project specific sw-description that fits that project, but it >>>>>> cannot be reused on other projects. I do not know how a general >>>>>> .swu image type can be factorized as Anatolji is trying to do. >>>>> >>>>> But is this really a problem for an swu image class? The main >>>>> difference is how you define you configuration. In most cases an >>>>> image class use global configurations from the machine >>>>> configuration and in case of a wic image a shared wks configuration >>>>> file from a common directory. At the moment swupdate use a special >>>>> recipe on top of an other main image recipe. >>>> >>>> A global (MACHINE or DISTRO) configuration covers just the simple >>>> cases, but there are many projects creating multiple SWUs. So a >>>> MACHINE configuration does not work in many cases. >>> >>> But a global configuration could be override in recipes or you could >>> share it between multiple recipes. >> >> Then it is a .inc file that we can include in multiple recipes, but >> for its nature, a SWU configuration is related to an image, and not to >> a MACHINE / DISTRO. > > Does newer versions of OE support files relative to an include? Older > versions doesn't set the THISDIR for a inc file. I never need it - .inc files are searched across the layers, so I can put a .inc somewhere and then add a require even in another layer with the path from the TOPDIR of the layer with the .inc, like "require recipes-bsp/u-boot/u-boot.inc" to make an example. > > Why is a swu related to an image? In fact, it is not...the relation is introduced now by this patchset. If we introduce a .swu in IMAGE_FSTYPES, *it* is an image. This is also why I am saying that adding support for FSTYPES just simplifies the very simple use case, that is update=rootfs. In most my use casesm it is not. A SWU is a package with multiple artifacts and with instructions how the new software should be installed on the device. Not always it is derived from an image recipe, I have also normal packages (that could be installed as part of rootfs by bitbake) that generate a SWU, too, because it is required to deliver in this way. Or even no OE package at all, the SWU contains configuration / calibration / data with instructions (in sw-description) so that SWUpdate knows what it has to do. Completely unrelated. But if the topic of the series is "swu image fstype", this binds a swu with an image recipe. > What is the difference between a sd > image and an update in terms of it content? Both contain a rootfs and > some additional images like bootloader or firmware. Well, differences are in the usage. With SD, the device is passive: you go in front of the device, you turn the device off, you insert the SD. Device boots or not, but there is no instructions / code to update. In OTA and in SWUpdate, the content (the SWU), apart of the artifacts that we can consider like the SD, has also instructions and code to perform a smooth update. This can help to fix hardware problems (change the behavior before update), provide a migration with repartitioning, whatever. And the behavior can be detected at runtime, so that buildsystem cannot provide everything. > > Why is an update related to an image? IMHO it is not, but adding swu fstype binds it. > Does the content of the rootfs > really matter? No > >>> It should work for the simple and the complex solutions. >> >> .inc file, yes, but SWU variables in a MACHINE config is quite >> misleading, because this should mean it is machine specific, and it is >> really more image specific. > > An .inc file is hard to override. Mmmhhh...I do not think so - a .inc is just a common part of a recipe, and like all resipes, it is overridden by later set (after the require statement) or by a .bbappend. > A configuration file could be reused > between images and it is possible to provide a default configuration for > a machine or distro. ok, let's say it is just another place to set SWUPDATE_ variables (or to set the behavior of the class). > > >>>> The difference with a WKS is that we need here a configuration for >>>> the device (sw-description), that is also interpreted by the device, >>>> and a configuration for the build system, that should not go on the >>>> device (that means, the recipe), and the build system can require >>>> some part from sw-description (or in any cases, it fills >>>> automatically some parts). I cannot say that SWUpdate use a recipe >>>> on top of others, it is >>>> a recipe that depends on one (simple rootfs) or many other recipes, >>>> and in some cases even from images generated for other OE machines. >>> >>> The WKS also need some information for the device but they are >>> generated / extracted from the configuration and the WKS could also >>> depends on other recipes and there artifacts. >> >> But the device has nothing to do with it, while in case of SWU, >> sw-description is the description of the release and must understand it. > > What do you mean? See above - WKS is 100% completed by the buildsystem, behavior of an update via SWU is determined by the device at runtime. WKS just need to pack all stuff together, you do not need to inform the device what to do with the packed data. > >>> At the moment I need an image recipe and an extra vfat image type >>> class to create a boot partition. >> >> Ok, this is to generate the .wic for the first setup of the device. I >> guess this cannot be avoided. > > WIC avoid this problem and generates the file system on the fly. I miss what you want to say here or what you find as problem. Is it the generation of filesystem, maybe empty ? > > Furthermore the update of a full filesystem instead of its content have > advantages and this leads to images for every partition. Also I do not get, the "images" section in sw-description provides exactly what you mean, and this feature was from the early beginning in SWUpdate. SWUpdate does not know the contents. > >>> Furthermore I need a update recipe per each image recipe. >> >> What I do is to have a .inc file shared by all update recipes, so that >> the update recipes are very simple. But yes, an update recipe per >> image is necessary, even if quite empty. > > But you could avoid this with a swu image class and a configuration file. So I need an image class, sw-description and a configuration file. Currently we have the image class to create the SWU, sw-description and a recipe for the SWU. Apart that configuration is put into a .conf instead of a recipe (and yes, I factorize this with .inc currently), I am missing the real difference and advantages.. > >>>>> I think that the main problems are the scatter information in the >>>>> sw-description and the recipe. >>>> >>>> Yes, but they contain information for different actors, the device >>>> (sw-description) and the buildsystem (the recipe). Information in >>>> recipe are useless for the device, but the buildsystem can use some >>>> information from sw-description (and it extends it). >>>> >>>>> Other classes generate the configuration file on the fly (ex. >>>>> kernel-fitimage) >>>> >>>> well, let me say that I am not so fond of it. It creates a fitImage >>>> for most cases with a hard-coded and generated .its, that is >>>> kernel+DT+Ramdisk, but there are some other cases where additional >>>> images must be packed into a fitImage and the kernel class is not >>>> configurable for such things (i.e. fpga bitstreams, etc.). Anyway, >>>> in most cases a fitImage consists of kernel+DT, and then at least >>>> 90% of cases are covered - I do not see the same in SWUpdate. >>> >>> You are right a sw-description generator will become very complex. >>> >> >> IMHO sw-description is code instead of simple configuration, and it is >> written by developers / integrators, because requirements and use case >> strongly differ. For example, a customer of mine wants to group some >> artifacts (U-Boot, firmware, a blob) together and update all of them >> if one of the artifacts belonging to the group needs to be updated due >> to version mismatch. Some requirements can be satisfied using hooks in >> sw-description and embedded Lua code. An automatic generation of >> sw-description is then impossible but very simple cases (aka update of >> just rootfs). > > So we should keep the sw-description. I do not see any way to generate it but in the very simple cases (that I have not since a long time, anyway). > The question is how we could > define the build steps. Ok > > >>>>> or use a separate configuration file (ex. WIC). >>>> >>>> Right, but wks is a configuration file for the buildsystem and the >>>> device does not need to bother with it. We have a use case here more >>>> similar to fitImage, but where there is quite no "standard" case >>>> like fitImage to allow the generation of a sw-description. >>> >>> The problem is not the missing standard case but the flexibility of >>> our configuration. >>> >>> Maybe we could define a swu.in or sw-description.in configuration >>> which extends the sw-description with build information? >> >> I do not know how, information must be removed then because they >> should not reach the device. And there is not just OE, but also >> Buildroot and Debian. > > So you prefare a common solution like an swu build tool? This is an option, but it is much more interesting for projects based on Debian. But as sw-description is part of SWUpdate, and the project is unaware of the buildsystem, is quite confusing if sw-description contains parts that are very specific to a buildsystem. > >> I am also careful to change sw-description: it is ok to add new >> features / attributes, but they should not break update for devices >> already in field if they do not use the new features. > > The build information shouldn't influence the update and should be > removed from the shipped sw-description. But if I retrieve the info from sw-description, I process it and then I remove from sw-description again, what is the main difference as letting this build information outside sw-description ? > >>> Something like a source entry for the image, file and script entries. >> >> With just a hook, it does not work, I think. And handler must be >> specified (what about to update a UBIFS instead of eMMC ?), at the end >> it is a sw-description written in another format. > > I don't speak about an other file format in front of sw-description. Ok > >>> We could reuse the filename to rename the files and the compressed >>> and encrypted entry to prepare the files. Furthermore we could add a >>> filesystem and name entry to an image to create a file system on the >>> fly from IMAGE_BOOT_FILES. >>> > > My idea is to add build information into the sw-description. I have > rethink the solution and would use specific comments as decorations / > annotation inside the sw-description. This makes it easy to extract and > remove the build information from the sw-description. Each decorations > include a name for the file and an optional compression and encryption. > We could support raw files from the deploy directory, on the fly > filesystems via IMAGE_BOOT_FILES and the rootfs: > > //! boot.bin.gz --source=rawcopy > --sourceparams="file=boot-@MACHINE@.bin" --compression=zlib > //! boot.vfat.gz --source=bootimg --fstype=vfat --compression=zlib > //! rootfs.ext4.gz --source=rootfs --fstype=ext4 --compression=zlib > Which is your main goal ? So it is more important to drop the recipe for the SWU or to drop the configuration via variables and use WKS like instructions ? Or both ? Best regards, Stefano
Hi Stefano, Am 17.02.21 um 10:02 schrieb Stefano Babic: > Hi Stefan, > > On 16.02.21 21:34, Stefan Herbrechtsmeier wrote: >> Hi Stefano, >> >> Am 16.02.21 um 09:02 schrieb Stefano Babic: >>> Hi Stefan, >>> >>> On 15.02.21 20:21, Stefan Herbrechtsmeier wrote: >>>> Hi Stefano, >>>> >>>> Am 14.02.21 um 18:15 schrieb Stefano Babic: >>>>> Hi Stefan, >>>>> >>>>> On 14.02.21 16:11, Stefan Herbrechtsmeier wrote: >>>>>> Hi Stefano, >>>>>> >>>>>> Am 14.02.21 um 10:34 schrieb Stefano Babic: >>>>>>> Hi Stefan, >>>>>>> >>>>>>> On 13.02.21 17:05, Stefan Herbrechtsmeier wrote: >>>>>>>> Hi Stefano, >>>>>>>> >>>>>>>> Am 11.02.21 um 13:25 schrieb Stefano Babic: >>>>>>>>> Hi Anatolji, >>>>>>>>> >>>>>>>>> On 11.02.21 11:38, Anatolij Gustschin wrote: >>>>>>>>>> This series adds swu image fstype to swupdate class, so that it is >>>>>>>>>> possible to generate .swu images via normal image recipes extended >>>>>>>>> >>>>>>>>> Ok, so you want for the very simple case (just rootfs in SWU) to >>>>>>>>> generate quite automatically a SWU. But if we have more >>>>>>>>> artifacts in >>>>>>>>> SWU, I do not see many advantages for this, I still need to add the >>>>>>>>> right DEPENDS and all required SWUPDATE_ flags. Is it just for >>>>>>>>> lazy people ? >>>>>>>> >>>>>>>> I think an image type for swupdate is reasonable to support >>>>>>>> different images. The WIC image type supports different artifacts >>>>>>>> and could create file systems on the fly. At the moment it is >>>>>>>> possible to create an SD card with arbitrary deployed artifacts >>>>>>>> in different partitions with various filesystems for any image. >>>>>>> >>>>>>> Right. >>>>>>> >>>>>>>> Doing the same to create an swu update file is much more >>>>>>>> complicated. >>>>>>> >>>>>>> Yes, I am just bothering how this can be done in the right way. >>>>>>> There are sure a lot of projects where the SWU just contains the >>>>>>> rootfs, and then a .swu is quite straightforward, but there are a >>>>>>> raising number of projects (at least, this is my experience) where >>>>>>> the SWU contains update for separate entities on the same device >>>>>>> (i.e. microcontroller firmwares) or even to forward software to >>>>>>> other devices. At the end, I come with a very specialized and >>>>>>> project specific sw-description that fits that project, but it >>>>>>> cannot be reused on other projects. I do not know how a general >>>>>>> .swu image type can be factorized as Anatolji is trying to do. >>>>>> >>>>>> But is this really a problem for an swu image class? The main >>>>>> difference is how you define you configuration. In most cases an >>>>>> image class use global configurations from the machine >>>>>> configuration and in case of a wic image a shared wks configuration >>>>>> file from a common directory. At the moment swupdate use a special >>>>>> recipe on top of an other main image recipe. >>>>> >>>>> A global (MACHINE or DISTRO) configuration covers just the simple >>>>> cases, but there are many projects creating multiple SWUs. So a >>>>> MACHINE configuration does not work in many cases. >>>> >>>> But a global configuration could be override in recipes or you could >>>> share it between multiple recipes. >>> >>> Then it is a .inc file that we can include in multiple recipes, but >>> for its nature, a SWU configuration is related to an image, and not to >>> a MACHINE / DISTRO. >> >> Does newer versions of OE support files relative to an include? Older >> versions doesn't set the THISDIR for a inc file. > > I never need it - .inc files are searched across the layers, so I can > put a .inc somewhere and then add a require even in another layer with > the path from the TOPDIR of the layer with the .inc, like "require > recipes-bsp/u-boot/u-boot.inc" to make an example. How do you share a sw-description file between different layers? >> Why is a swu related to an image? > > In fact, it is not...the relation is introduced now by this patchset. If > we introduce a .swu in IMAGE_FSTYPES, *it* is an image. This is also why > I am saying that adding support for FSTYPES just simplifies the very > simple use case, that is update=rootfs. In most my use casesm it is not. But isn't it a valid use case? In my case many images consists of a rootfs, boot loader, firmware and recovery images. The updates differ only in the rootfs image. > A SWU is a package with multiple artifacts and with instructions how the > new software should be installed on the device. Not always it is derived > from an image recipe, I have also normal packages (that could be > installed as part of rootfs by bitbake) that generate a SWU, too, > because it is required to deliver in this way. Why do you have to combine both in one? The same would be possible with a recipe which installs or deploys the file and a generalized baremetal-image. > Or even no OE package at > all, the SWU contains configuration / calibration / data with > instructions (in sw-description) so that SWUpdate knows what it has to > do. Completely unrelated. Does it matter if it is handled by an empty package or an empty image? > But if the topic of the series is "swu image fstype", this binds a swu > with an image recipe. Only if the topic imply to remove the old way. >> What is the difference between a sd >> image and an update in terms of it content? Both contain a rootfs and >> some additional images like bootloader or firmware. > > Well, differences are in the usage. With SD, the device is passive: you > go in front of the device, you turn the device off, you insert the SD. > Device boots or not, but there is no instructions / code to update. In > OTA and in SWUpdate, the content (the SWU), apart of the artifacts that > we can consider like the SD, has also instructions and code to perform a > smooth update. This can help to fix hardware problems (change the > behavior before update), provide a migration with repartitioning, > whatever. And the behavior can be detected at runtime, so that > buildsystem cannot provide everything. In both cases the build systems take some artifacts and extend it with additional information. In case of the wks it doesn't matter which image the user build. Why does it matters for an swu? >> Why is an update related to an image? > > IMHO it is not, but adding swu fstype binds it. Why? Does it really matter which image (rootfs) is packed inside the swu? The image class only specify how an image should be packed. >> Does the content of the rootfs >> really matter? > > No But why we need a recipe per image. >>>> It should work for the simple and the complex solutions. >>> >>> .inc file, yes, but SWU variables in a MACHINE config is quite >>> misleading, because this should mean it is machine specific, and it is >>> really more image specific. >> >> An .inc file is hard to override. > > Mmmhhh...I do not think so - a .inc is just a common part of a recipe, > and like all resipes, it is overridden by later set (after the require > statement) or by a .bbappend. The main difference is that an .inc file is immediately parsed. The WKS_FILE variable could be overridden by an recipe. >> A configuration file could be reused >> between images and it is possible to provide a default configuration for >> a machine or distro. > > ok, let's say it is just another place to set SWUPDATE_ variables (or to > set the behavior of the class). It depends on the implementation. Maybe it is independent of the class and a configuration for an swu build tool. >>>>> The difference with a WKS is that we need here a configuration for >>>>> the device (sw-description), that is also interpreted by the device, >>>>> and a configuration for the build system, that should not go on the >>>>> device (that means, the recipe), and the build system can require >>>>> some part from sw-description (or in any cases, it fills >>>>> automatically some parts). I cannot say that SWUpdate use a recipe >>>>> on top of others, it is >>>>> a recipe that depends on one (simple rootfs) or many other recipes, >>>>> and in some cases even from images generated for other OE machines. >>>> >>>> The WKS also need some information for the device but they are >>>> generated / extracted from the configuration and the WKS could also >>>> depends on other recipes and there artifacts. >>> >>> But the device has nothing to do with it, while in case of SWU, >>> sw-description is the description of the release and must understand it. >> >> What do you mean? > > See above - WKS is 100% completed by the buildsystem, behavior of an > update via SWU is determined by the device at runtime. WKS just need to > pack all stuff together, you do not need to inform the device what to do > with the packed data. > >> >>>> At the moment I need an image recipe and an extra vfat image type >>>> class to create a boot partition. >>> >>> Ok, this is to generate the .wic for the first setup of the device. I >>> guess this cannot be avoided. >> >> WIC avoid this problem and generates the file system on the fly. > > I miss what you want to say here or what you find as problem. Is it the > generation of filesystem, maybe empty ? The problem is that WIC could generate filesystems on the fly. You doesn't need multiple recipe to create different filesystems. >> Furthermore the update of a full filesystem instead of its content have >> advantages and this leads to images for every partition. > > Also I do not get, the "images" section in sw-description provides > exactly what you mean, and this feature was from the early beginning in > SWUpdate. SWUpdate does not know the contents. > >> >>>> Furthermore I need a update recipe per each image recipe. >>> >>> What I do is to have a .inc file shared by all update recipes, so that >>> the update recipes are very simple. But yes, an update recipe per >>> image is necessary, even if quite empty. >> >> But you could avoid this with a swu image class and a configuration file. > > So I need an image class, sw-description and a configuration file. > Currently we have the image class to create the SWU, sw-description and > a recipe for the SWU. Apart that configuration is put into a .conf > instead of a recipe (and yes, I factorize this with .inc currently), I > am missing the real difference and advantages.. An image class could create an swu for arbitrary images. The machine could define the dependencies and content of the swu beside the rootfs. An image could override the dependencies and content. The image class could reuse exists variables to create filesystems. >>>>>> I think that the main problems are the scatter information in the >>>>>> sw-description and the recipe. >>>>> >>>>> Yes, but they contain information for different actors, the device >>>>> (sw-description) and the buildsystem (the recipe). Information in >>>>> recipe are useless for the device, but the buildsystem can use some >>>>> information from sw-description (and it extends it). >>>>> >>>>>> Other classes generate the configuration file on the fly (ex. >>>>>> kernel-fitimage) >>>>> >>>>> well, let me say that I am not so fond of it. It creates a fitImage >>>>> for most cases with a hard-coded and generated .its, that is >>>>> kernel+DT+Ramdisk, but there are some other cases where additional >>>>> images must be packed into a fitImage and the kernel class is not >>>>> configurable for such things (i.e. fpga bitstreams, etc.). Anyway, >>>>> in most cases a fitImage consists of kernel+DT, and then at least >>>>> 90% of cases are covered - I do not see the same in SWUpdate. >>>> >>>> You are right a sw-description generator will become very complex. >>>> >>> >>> IMHO sw-description is code instead of simple configuration, and it is >>> written by developers / integrators, because requirements and use case >>> strongly differ. For example, a customer of mine wants to group some >>> artifacts (U-Boot, firmware, a blob) together and update all of them >>> if one of the artifacts belonging to the group needs to be updated due >>> to version mismatch. Some requirements can be satisfied using hooks in >>> sw-description and embedded Lua code. An automatic generation of >>> sw-description is then impossible but very simple cases (aka update of >>> just rootfs). >> >> So we should keep the sw-description. > > I do not see any way to generate it but in the very simple cases (that I > have not since a long time, anyway). But we could do semi generation via variable expansion. This allows the image class to define the artifact name, compression type, encryption and filesystem type. >> The question is how we could >> define the build steps. > > Ok > >> >> >>>>>> or use a separate configuration file (ex. WIC). >>>>> >>>>> Right, but wks is a configuration file for the buildsystem and the >>>>> device does not need to bother with it. We have a use case here more >>>>> similar to fitImage, but where there is quite no "standard" case >>>>> like fitImage to allow the generation of a sw-description. >>>> >>>> The problem is not the missing standard case but the flexibility of >>>> our configuration. >>>> >>>> Maybe we could define a swu.in or sw-description.in configuration >>>> which extends the sw-description with build information? >>> >>> I do not know how, information must be removed then because they >>> should not reach the device. And there is not just OE, but also >>> Buildroot and Debian. >> >> So you prefare a common solution like an swu build tool? > > This is an option, but it is much more interesting for projects based on > Debian. But as sw-description is part of SWUpdate, and the project is > unaware of the buildsystem, is quite confusing if sw-description > contains parts that are very specific to a buildsystem. Okay. >>> I am also careful to change sw-description: it is ok to add new >>> features / attributes, but they should not break update for devices >>> already in field if they do not use the new features. >> >> The build information shouldn't influence the update and should be >> removed from the shipped sw-description. > > But if I retrieve the info from sw-description, I process it and then I > remove from sw-description again, what is the main difference as letting > this build information outside sw-description ? What is the advantage of two files? Does it make sense to use the same sw-description for different build instructions if it excluded the source name of the artifacts. >>>> Something like a source entry for the image, file and script entries. >>> >>> With just a hook, it does not work, I think. And handler must be >>> specified (what about to update a UBIFS instead of eMMC ?), at the end >>> it is a sw-description written in another format. >> >> I don't speak about an other file format in front of sw-description. > > Ok > >> >>>> We could reuse the filename to rename the files and the compressed >>>> and encrypted entry to prepare the files. Furthermore we could add a >>>> filesystem and name entry to an image to create a file system on the >>>> fly from IMAGE_BOOT_FILES. >>>> >> >> My idea is to add build information into the sw-description. I have >> rethink the solution and would use specific comments as decorations / >> annotation inside the sw-description. This makes it easy to extract and >> remove the build information from the sw-description. Each decorations >> include a name for the file and an optional compression and encryption. >> We could support raw files from the deploy directory, on the fly >> filesystems via IMAGE_BOOT_FILES and the rootfs: >> >> //! boot.bin.gz --source=rawcopy >> --sourceparams="file=boot-@MACHINE@.bin" --compression=zlib >> //! boot.vfat.gz --source=bootimg --fstype=vfat --compression=zlib >> //! rootfs.ext4.gz --source=rootfs --fstype=ext4 --compression=zlib >> > > Which is your main goal ? So it is more important to drop the recipe for > the SWU or to drop the configuration via variables and use WKS like > instructions ? Or both ? The main goal is to drop the duplicated recipe for the swu and to reuse the configurations of WIC. I would like to build a wks and swu image of an arbitrary image. Regards Stefan
Hi Stefan, On 28.02.21 21:06, Stefan Herbrechtsmeier wrote: > Hi Stefano, > > Am 17.02.21 um 10:02 schrieb Stefano Babic: >> Hi Stefan, >> >> On 16.02.21 21:34, Stefan Herbrechtsmeier wrote: >>> Hi Stefano, >>> >>> Am 16.02.21 um 09:02 schrieb Stefano Babic: >>>> Hi Stefan, >>>> >>>> On 15.02.21 20:21, Stefan Herbrechtsmeier wrote: >>>>> Hi Stefano, >>>>> >>>>> Am 14.02.21 um 18:15 schrieb Stefano Babic: >>>>>> Hi Stefan, >>>>>> >>>>>> On 14.02.21 16:11, Stefan Herbrechtsmeier wrote: >>>>>>> Hi Stefano, >>>>>>> >>>>>>> Am 14.02.21 um 10:34 schrieb Stefano Babic: >>>>>>>> Hi Stefan, >>>>>>>> >>>>>>>> On 13.02.21 17:05, Stefan Herbrechtsmeier wrote: >>>>>>>>> Hi Stefano, >>>>>>>>> >>>>>>>>> Am 11.02.21 um 13:25 schrieb Stefano Babic: >>>>>>>>>> Hi Anatolji, >>>>>>>>>> >>>>>>>>>> On 11.02.21 11:38, Anatolij Gustschin wrote: >>>>>>>>>>> This series adds swu image fstype to swupdate class, so that >>>>>>>>>>> it is >>>>>>>>>>> possible to generate .swu images via normal image recipes >>>>>>>>>>> extended >>>>>>>>>> >>>>>>>>>> Ok, so you want for the very simple case (just rootfs in SWU) to >>>>>>>>>> generate quite automatically a SWU. But if we have more >>>>>>>>>> artifacts in >>>>>>>>>> SWU, I do not see many advantages for this, I still need to >>>>>>>>>> add the >>>>>>>>>> right DEPENDS and all required SWUPDATE_ flags. Is it just for >>>>>>>>>> lazy people ? >>>>>>>>> >>>>>>>>> I think an image type for swupdate is reasonable to support >>>>>>>>> different images. The WIC image type supports different artifacts >>>>>>>>> and could create file systems on the fly. At the moment it is >>>>>>>>> possible to create an SD card with arbitrary deployed artifacts >>>>>>>>> in different partitions with various filesystems for any image. >>>>>>>> >>>>>>>> Right. >>>>>>>> >>>>>>>>> Doing the same to create an swu update file is much more >>>>>>>>> complicated. >>>>>>>> >>>>>>>> Yes, I am just bothering how this can be done in the right way. >>>>>>>> There are sure a lot of projects where the SWU just contains the >>>>>>>> rootfs, and then a .swu is quite straightforward, but there are a >>>>>>>> raising number of projects (at least, this is my experience) where >>>>>>>> the SWU contains update for separate entities on the same device >>>>>>>> (i.e. microcontroller firmwares) or even to forward software to >>>>>>>> other devices. At the end, I come with a very specialized and >>>>>>>> project specific sw-description that fits that project, but it >>>>>>>> cannot be reused on other projects. I do not know how a general >>>>>>>> .swu image type can be factorized as Anatolji is trying to do. >>>>>>> >>>>>>> But is this really a problem for an swu image class? The main >>>>>>> difference is how you define you configuration. In most cases an >>>>>>> image class use global configurations from the machine >>>>>>> configuration and in case of a wic image a shared wks configuration >>>>>>> file from a common directory. At the moment swupdate use a special >>>>>>> recipe on top of an other main image recipe. >>>>>> >>>>>> A global (MACHINE or DISTRO) configuration covers just the simple >>>>>> cases, but there are many projects creating multiple SWUs. So a >>>>>> MACHINE configuration does not work in many cases. >>>>> >>>>> But a global configuration could be override in recipes or you could >>>>> share it between multiple recipes. >>>> >>>> Then it is a .inc file that we can include in multiple recipes, but >>>> for its nature, a SWU configuration is related to an image, and not to >>>> a MACHINE / DISTRO. >>> >>> Does newer versions of OE support files relative to an include? Older >>> versions doesn't set the THISDIR for a inc file. >> >> I never need it - .inc files are searched across the layers, so I can >> put a .inc somewhere and then add a require even in another layer with >> the path from the TOPDIR of the layer with the .inc, like "require >> recipes-bsp/u-boot/u-boot.inc" to make an example. > > How do you share a sw-description file between different layers? I just put it into a repo and then I fetch it from recipe. > > >>> Why is a swu related to an image? >> >> In fact, it is not...the relation is introduced now by this patchset. If >> we introduce a .swu in IMAGE_FSTYPES, *it* is an image. This is also why >> I am saying that adding support for FSTYPES just simplifies the very >> simple use case, that is update=rootfs. In most my use casesm it is not. > > But isn't it a valid use case? Of course it is. But simplifying this use case should not discard flexibility and cause other use cases, even if they are rare, impossible to be implemented. > In my case many images consists of a > rootfs, boot loader, firmware and recovery images. The updates differ > only in the rootfs image. Ok - what I do in this case is to have a sw-description with dynamic rootfs and yes, one recipe for each rootfs image to set this variable. Let's see what is going on. > > >> A SWU is a package with multiple artifacts and with instructions how the >> new software should be installed on the device. Not always it is derived >> from an image recipe, I have also normal packages (that could be >> installed as part of rootfs by bitbake) that generate a SWU, too, >> because it is required to deliver in this way. > > Why do you have to combine both in one? The same would be possible with > a recipe which installs or deploys the file and a generalized > baremetal-image. No. This is for SWUpdate, that is this package *must* be a SWU because SWUpdate have to accept it and process it. One use case is a license file with keys, and sw-description contains instructions how to process it to allow to install further SWUs. This license file can be part of a rootfs or not, and it is required in any case for migration. But I have several other use case where what is delivered does not belong to the classic schema (rootfs and just software to be installed). > > >> Or even no OE package at >> all, the SWU contains configuration / calibration / data with >> instructions (in sw-description) so that SWUpdate knows what it has to >> do. Completely unrelated. > > Does it matter if it is handled by an empty package or an empty image? > > >> But if the topic of the series is "swu image fstype", this binds a swu >> with an image recipe. > > Only if the topic imply to remove the old way. Right, but this is what I discovered with Anatolji's patches, because they break some use cases of mine. > > >>> What is the difference between a sd >>> image and an update in terms of it content? Both contain a rootfs and >>> some additional images like bootloader or firmware. >> >> Well, differences are in the usage. With SD, the device is passive: you >> go in front of the device, you turn the device off, you insert the SD. >> Device boots or not, but there is no instructions / code to update. In >> OTA and in SWUpdate, the content (the SWU), apart of the artifacts that >> we can consider like the SD, has also instructions and code to perform a >> smooth update. This can help to fix hardware problems (change the >> behavior before update), provide a migration with repartitioning, >> whatever. And the behavior can be detected at runtime, so that >> buildsystem cannot provide everything. > > In both cases the build systems take some artifacts and extend it with > additional information. In case of the wks it doesn't matter which image > the user build. Why does it matters for an swu? > > >>> Why is an update related to an image? >> >> IMHO it is not, but adding swu fstype binds it. > > Why? Because in OE a fstype is strict bound to an image (I need a recipe image to use it), while a SWU is *not* bound to an image, it just collects a list of artifacts. If we want to add a fstype, that is a class to for .swu to be used with IMAGE_FSTYPES, we bind with an image. But as you also say, the SWU is unrelated to an image or a rootfs. > > Does it really matter which image (rootfs) is packed inside the swu? It does not matter : in fact, it is currently unaware. The recipe just gets a list of artifacts, they can be a rootfs or not, it does not matter at all. But this is the current state, what this series is willing to change and then bind to a rootfs. > > The image class only specify how an image should be packed. Right, but we need an image for it, and there are simply *no images* in some use cases. > > >>> Does the content of the rootfs >>> really matter? >> >> No > > But why we need a recipe per image. Exactly because it is unrelated, and we need to describe which components are part of a SWU. > > >>>>> It should work for the simple and the complex solutions. >>>> >>>> .inc file, yes, but SWU variables in a MACHINE config is quite >>>> misleading, because this should mean it is machine specific, and it is >>>> really more image specific. >>> >>> An .inc file is hard to override. >> >> Mmmhhh...I do not think so - a .inc is just a common part of a recipe, >> and like all resipes, it is overridden by later set (after the require >> statement) or by a .bbappend. > > The main difference is that an .inc file is immediately parsed. The > WKS_FILE variable could be overridden by an recipe. Right - but then we need a WKS file for each image. To maintain, it is not quite different as having a recipe per image for each SWU. I am quite losing which are the goals here or what we want to address. I try to sumarize, maybe I misunderstand something: - we need currently a recipe per image in case of different rootfs, and adding a .swu fstype lets to avoid the recipe. However, sw-description cannot be generated as I said previously, it can be factorized among images. - there are some cases not covered with current generation, related to filesystem generation. I am not sure I have understood this. SWUpdate delivers artifacts. If a filesystem must be created, the device can generally do this itself, first eample at first boot after update. Or this can be executed inside the update process as part of a preinstall part. Why this should be done when SWU is generated, is something I am missing: if the filesystem contains files, well, then it is an "image" and can be built in OE, and then added to the list of artifacts. If the filesystem is empty, if it is generated at build time it must be delivered, too, increasing the SWU size (but an empty filesystem can be well compressed), and it looks to me more suitable that the device does it. - is there another goal ? > > >>> A configuration file could be reused >>> between images and it is possible to provide a default configuration for >>> a machine or distro. >> >> ok, let's say it is just another place to set SWUPDATE_ variables (or to >> set the behavior of the class). > > It depends on the implementation. Maybe it is independent of the class > and a configuration for an swu build tool. Ok > > >>>>>> The difference with a WKS is that we need here a configuration for >>>>>> the device (sw-description), that is also interpreted by the device, >>>>>> and a configuration for the build system, that should not go on the >>>>>> device (that means, the recipe), and the build system can require >>>>>> some part from sw-description (or in any cases, it fills >>>>>> automatically some parts). I cannot say that SWUpdate use a recipe >>>>>> on top of others, it is >>>>>> a recipe that depends on one (simple rootfs) or many other recipes, >>>>>> and in some cases even from images generated for other OE machines. >>>>> >>>>> The WKS also need some information for the device but they are >>>>> generated / extracted from the configuration and the WKS could also >>>>> depends on other recipes and there artifacts. >>>> >>>> But the device has nothing to do with it, while in case of SWU, >>>> sw-description is the description of the release and must understand >>>> it. >>> >>> What do you mean? >> >> See above - WKS is 100% completed by the buildsystem, behavior of an >> update via SWU is determined by the device at runtime. WKS just need to >> pack all stuff together, you do not need to inform the device what to do >> with the packed data. >> >>> >>>>> At the moment I need an image recipe and an extra vfat image type >>>>> class to create a boot partition. >>>> >>>> Ok, this is to generate the .wic for the first setup of the device. I >>>> guess this cannot be avoided. >>> >>> WIC avoid this problem and generates the file system on the fly. >> >> I miss what you want to say here or what you find as problem. Is it the >> generation of filesystem, maybe empty ? > > The problem is that WIC could generate filesystems on the fly. You > doesn't need multiple recipe to create different filesystems. So is this the issue you want to address ? Anyway, the WIC is also quite limited. It works fine for the rootfs, and I can use the rawcopy plugin with --source rootfs. For some parts (see IMAGE_BOOTFILES), we can add also some files to a generated filesystem (EFI is the clear example). But I cannot combine partitions from different images, like a rootfs and a partition just containing the application files, that is generated with a different image recipe. > > >>> Furthermore the update of a full filesystem instead of its content have >>> advantages and this leads to images for every partition. >> >> Also I do not get, the "images" section in sw-description provides >> exactly what you mean, and this feature was from the early beginning in >> SWUpdate. SWUpdate does not know the contents. >> >>> >>>>> Furthermore I need a update recipe per each image recipe. >>>> >>>> What I do is to have a .inc file shared by all update recipes, so that >>>> the update recipes are very simple. But yes, an update recipe per >>>> image is necessary, even if quite empty. >>> >>> But you could avoid this with a swu image class and a configuration >>> file. >> >> So I need an image class, sw-description and a configuration file. >> Currently we have the image class to create the SWU, sw-description and >> a recipe for the SWU. Apart that configuration is put into a .conf >> instead of a recipe (and yes, I factorize this with .inc currently), I >> am missing the real difference and advantages.. > > An image class could create an swu for arbitrary images. The machine > could define the dependencies and content of the swu beside the rootfs. > An image could override the dependencies and content. The image class > could reuse exists variables to create filesystems. Ok, I am quite sure now that the big issue on your use case is that filesystems on empty partitions are not generated when SWU is built. > > >>>>>>> I think that the main problems are the scatter information in the >>>>>>> sw-description and the recipe. >>>>>> >>>>>> Yes, but they contain information for different actors, the device >>>>>> (sw-description) and the buildsystem (the recipe). Information in >>>>>> recipe are useless for the device, but the buildsystem can use some >>>>>> information from sw-description (and it extends it). >>>>>> >>>>>>> Other classes generate the configuration file on the fly (ex. >>>>>>> kernel-fitimage) >>>>>> >>>>>> well, let me say that I am not so fond of it. It creates a fitImage >>>>>> for most cases with a hard-coded and generated .its, that is >>>>>> kernel+DT+Ramdisk, but there are some other cases where additional >>>>>> images must be packed into a fitImage and the kernel class is not >>>>>> configurable for such things (i.e. fpga bitstreams, etc.). Anyway, >>>>>> in most cases a fitImage consists of kernel+DT, and then at least >>>>>> 90% of cases are covered - I do not see the same in SWUpdate. >>>>> >>>>> You are right a sw-description generator will become very complex. >>>>> >>>> >>>> IMHO sw-description is code instead of simple configuration, and it is >>>> written by developers / integrators, because requirements and use case >>>> strongly differ. For example, a customer of mine wants to group some >>>> artifacts (U-Boot, firmware, a blob) together and update all of them >>>> if one of the artifacts belonging to the group needs to be updated due >>>> to version mismatch. Some requirements can be satisfied using hooks in >>>> sw-description and embedded Lua code. An automatic generation of >>>> sw-description is then impossible but very simple cases (aka update of >>>> just rootfs). >>> >>> So we should keep the sw-description. >> >> I do not see any way to generate it but in the very simple cases (that I >> have not since a long time, anyway). > > But we could do semi generation via variable expansion. This allows the > image class to define the artifact name, compression type, encryption > and filesystem type. > > >>> The question is how we could >>> define the build steps. >> >> Ok >> >>> >>> >>>>>>> or use a separate configuration file (ex. WIC). >>>>>> >>>>>> Right, but wks is a configuration file for the buildsystem and the >>>>>> device does not need to bother with it. We have a use case here more >>>>>> similar to fitImage, but where there is quite no "standard" case >>>>>> like fitImage to allow the generation of a sw-description. >>>>> >>>>> The problem is not the missing standard case but the flexibility of >>>>> our configuration. >>>>> >>>>> Maybe we could define a swu.in or sw-description.in configuration >>>>> which extends the sw-description with build information? >>>> >>>> I do not know how, information must be removed then because they >>>> should not reach the device. And there is not just OE, but also >>>> Buildroot and Debian. >>> >>> So you prefare a common solution like an swu build tool? >> >> This is an option, but it is much more interesting for projects based on >> Debian. But as sw-description is part of SWUpdate, and the project is >> unaware of the buildsystem, is quite confusing if sw-description >> contains parts that are very specific to a buildsystem. > > Okay. > >>>> I am also careful to change sw-description: it is ok to add new >>>> features / attributes, but they should not break update for devices >>>> already in field if they do not use the new features. >>> >>> The build information shouldn't influence the update and should be >>> removed from the shipped sw-description. >> >> But if I retrieve the info from sw-description, I process it and then I >> remove from sw-description again, what is the main difference as letting >> this build information outside sw-description ? > > What is the advantage of two files? They are unrelated and no dependencies are created. > Does it make sense to use the same > sw-description for different build instructions if it excluded the > source name of the artifacts. This is what I often do with variable substitution (for rootfs), having just one sw-description, but multiple recipes for the different use cases. I am trying to understand if the proposed is just an alternative implementation or there is really some added values. > > >>>>> Something like a source entry for the image, file and script entries. >>>> >>>> With just a hook, it does not work, I think. And handler must be >>>> specified (what about to update a UBIFS instead of eMMC ?), at the end >>>> it is a sw-description written in another format. >>> >>> I don't speak about an other file format in front of sw-description. >> >> Ok >> >>> >>>>> We could reuse the filename to rename the files and the compressed >>>>> and encrypted entry to prepare the files. Furthermore we could add a >>>>> filesystem and name entry to an image to create a file system on the >>>>> fly from IMAGE_BOOT_FILES. >>>>> >>> >>> My idea is to add build information into the sw-description. I have >>> rethink the solution and would use specific comments as decorations / >>> annotation inside the sw-description. This makes it easy to extract and >>> remove the build information from the sw-description. Each decorations >>> include a name for the file and an optional compression and encryption. >>> We could support raw files from the deploy directory, on the fly >>> filesystems via IMAGE_BOOT_FILES and the rootfs: >>> >>> //! boot.bin.gz --source=rawcopy >>> --sourceparams="file=boot-@MACHINE@.bin" --compression=zlib >>> //! boot.vfat.gz --source=bootimg --fstype=vfat --compression=zlib >>> //! rootfs.ext4.gz --source=rootfs --fstype=ext4 --compression=zlib >>> >> >> Which is your main goal ? So it is more important to drop the recipe for >> the SWU or to drop the configuration via variables and use WKS like >> instructions ? Or both ? > > The main goal is to drop the duplicated recipe for the swu Ok, this is clear. > and to reuse > the configurations of WIC. You mean same configuration ? So are you thinking to a WIC plugin that generates a SWU ? > I would like to build a wks and swu image of > an arbitrary image. Regards, Stefano
Hi Stefano, apologize the long delay. Am 01.03.21 um 10:02 schrieb Stefano Babic: > Hi Stefan, > > On 28.02.21 21:06, Stefan Herbrechtsmeier wrote: >> Hi Stefano, >> >> Am 17.02.21 um 10:02 schrieb Stefano Babic: >>> Hi Stefan, >>> >>> On 16.02.21 21:34, Stefan Herbrechtsmeier wrote: >>>> Hi Stefano, >>>> >>>> Am 16.02.21 um 09:02 schrieb Stefano Babic: >>>>> Hi Stefan, >>>>> >>>>> On 15.02.21 20:21, Stefan Herbrechtsmeier wrote: >>>>>> Hi Stefano, >>>>>> >>>>>> Am 14.02.21 um 18:15 schrieb Stefano Babic: >>>>>>> Hi Stefan, >>>>>>> >>>>>>> On 14.02.21 16:11, Stefan Herbrechtsmeier wrote: >>>>>>>> Hi Stefano, >>>>>>>> >>>>>>>> Am 14.02.21 um 10:34 schrieb Stefano Babic: >>>>>>>>> Hi Stefan, >>>>>>>>> >>>>>>>>> On 13.02.21 17:05, Stefan Herbrechtsmeier wrote: >>>>>>>>>> Hi Stefano, >>>>>>>>>> >>>>>>>>>> Am 11.02.21 um 13:25 schrieb Stefano Babic: >>>>>>>>>>> Hi Anatolji, >>>>>>>>>>> >>>>>>>>>>> On 11.02.21 11:38, Anatolij Gustschin wrote: [snip] For me the rootfs is only a specific image. An image consists of different deploy artifacts and / or packages. I distinguish between normal recipes which produce packages and deploy binary artifacts on one side and images which create special deploy artifacts only. I think the normal system update swu should be independent of the concrete rootfs image. At the moment I could create a wic image for an arbitrary rootfs image and I don't understand why this shouldn't be possible for system update swu. In any case you could override the common configuration and use image specific configurations for specific images. > I am quite losing which are the goals here or what we want to address. I > try to sumarize, maybe I misunderstand something: > > - we need currently a recipe per image in case of different rootfs, and > adding a .swu fstype lets to avoid the recipe. However, sw-description > cannot be generated as I said previously, it can be factorized among > images. A swu fstype allows a generic default sw-description per distro or machine and support an arbitrary rootfs. > - there are some cases not covered with current generation, related to > filesystem generation. I am not sure I have understood this. SWUpdate > delivers artifacts. If a filesystem must be created, the device can > generally do this itself, first eample at first boot after update. Or > this can be executed inside the update process as part of a preinstall > part. Why this should be done when SWU is generated, is something I am > missing: if the filesystem contains files, well, then it is an "image" > and can be built in OE, and then added to the list of artifacts. Yes, but you need a recipe per image. The WIC image can do this without a special recipe per image / partition for arbitrary deploy artifacts. An common example is the boot partition or a partition with a filesystem and the recovery FIT image. > If the > filesystem is empty, if it is generated at build time it must be > delivered, too, increasing the SWU size (but an empty filesystem can be > well compressed), and it looks to me more suitable that the device does it. > > - is there another goal ? No. [snip] >> The problem is that WIC could generate filesystems on the fly. You >> doesn't need multiple recipe to create different filesystems. > > So is this the issue you want to address ? Anyway, the WIC is also quite > limited. It works fine for the rootfs, and I can use the rawcopy plugin > with --source rootfs. For some parts (see IMAGE_BOOTFILES), we can add > also some files to a generated filesystem (EFI is the clear example). > But I cannot combine partitions from different images, like a rootfs and > a partition just containing the application files, that is generated > with a different image recipe. This should be possible with the rawcopy plugin or you could split the rootfs on the fly with --exclude-path and --change-directory. And again you could avoid an additional recipe if you use the later option. [snip] > Ok, I am quite sure now that the big issue on your use case is that > filesystems on empty partitions are not generated when SWU is built. No, my issue is the needed extra recipes. WIC could create images with renamed deployed artifacts on the fly and I could build arbitrary images. [snip] >> Does it make sense to use the same sw-description for different build >> instructions if it excluded the source name of the artifacts. > > This is what I often do with variable substitution (for rootfs), having > just one sw-description, but multiple recipes for the different use > cases. I am trying to understand if the proposed is just an alternative > implementation or there is really some added values. The added value is no needed extra recipes and that it works out of the box. [snip] >>> Which is your main goal ? So it is more important to drop the recipe for >>> the SWU or to drop the configuration via variables and use WKS like >>> instructions ? Or both ? >> >> The main goal is to drop the duplicated recipe for the swu > > Ok, this is clear. > >> and to reuse the configurations of WIC. > > You mean same configuration ? So are you thinking to a WIC plugin that > generates a SWU ? This is one possible solution. An other solution could be an extra boot-image recipe. In any case we should reuse matching WIC variables. Regards Stefan
Hi Stefan, On 05.04.21 15:36, Stefan Herbrechtsmeier wrote: > Hi Stefano, > > apologize the long delay. > No problem - it looks like that Eastern's holiday give us some time to check again older topics ;-) > Am 01.03.21 um 10:02 schrieb Stefano Babic: >> Hi Stefan, >> >> On 28.02.21 21:06, Stefan Herbrechtsmeier wrote: >>> Hi Stefano, >>> >>> Am 17.02.21 um 10:02 schrieb Stefano Babic: >>>> Hi Stefan, >>>> >>>> On 16.02.21 21:34, Stefan Herbrechtsmeier wrote: >>>>> Hi Stefano, >>>>> >>>>> Am 16.02.21 um 09:02 schrieb Stefano Babic: >>>>>> Hi Stefan, >>>>>> >>>>>> On 15.02.21 20:21, Stefan Herbrechtsmeier wrote: >>>>>>> Hi Stefano, >>>>>>> >>>>>>> Am 14.02.21 um 18:15 schrieb Stefano Babic: >>>>>>>> Hi Stefan, >>>>>>>> >>>>>>>> On 14.02.21 16:11, Stefan Herbrechtsmeier wrote: >>>>>>>>> Hi Stefano, >>>>>>>>> >>>>>>>>> Am 14.02.21 um 10:34 schrieb Stefano Babic: >>>>>>>>>> Hi Stefan, >>>>>>>>>> >>>>>>>>>> On 13.02.21 17:05, Stefan Herbrechtsmeier wrote: >>>>>>>>>>> Hi Stefano, >>>>>>>>>>> >>>>>>>>>>> Am 11.02.21 um 13:25 schrieb Stefano Babic: >>>>>>>>>>>> Hi Anatolji, >>>>>>>>>>>> >>>>>>>>>>>> On 11.02.21 11:38, Anatolij Gustschin wrote: > > [snip] > > For me the rootfs is only a specific image. An image consists of > different deploy artifacts and / or packages. Right, I see the same0 > I distinguish between > normal recipes which produce packages and deploy binary artifacts on one > side and images which create special deploy artifacts only. > > I think the normal system update swu should be independent of the > concrete rootfs image. IMHO it is independent now. The SWU does not know if one of the artifact is an image. We can have a SWU with any kind of artifacts, even without a rootfs. > At the moment I could create a wic image for an > arbitrary rootfs image and I don't understand why this shouldn't be > possible for system update swu. Well, right, but a WIC depends on an image that is a rootfs, while SWU not. In fact, in a wic I set a --source rootfs if I want to put the filesystem in one of the partition. I cannot (at the moment) add a different image, like for example another filesystem. The only exception is the generation of EFI images via plugin. > In any case you could override the > common configuration and use image specific configurations for specific > images. That's right, but there are always a 1:1 relationship with a rootfs / image (or how we name it). But I do not see how to build a single WIC where each partitition is generated with a different recipe, and I put each image in a different partition. Just as simple case: two images with OS and application in separate partitions. On the other side, I do not see how generated a wic from SWU can help. A SWU is understood just from SWUpdate, it is not a filesystem (then putting in a partition makes less sense). > >> I am quite losing which are the goals here or what we want to address. >> I try to sumarize, maybe I misunderstand something: >> >> - we need currently a recipe per image in case of different rootfs, >> and adding a .swu fstype lets to avoid the recipe. However, >> sw-description cannot be generated as I said previously, it can be >> factorized among images. > > A swu fstype allows a generic default sw-description per distro or > machine and support an arbitrary rootfs. Yes, but this contradicts your first statement, that an image is just a set of artifacts to be deplowed. A SWU fstype means a rootfs. > > >> - there are some cases not covered with current generation, related to >> filesystem generation. I am not sure I have understood this. SWUpdate >> delivers artifacts. If a filesystem must be created, the device can >> generally do this itself, first eample at first boot after update. Or >> this can be executed inside the update process as part of a preinstall >> part. Why this should be done when SWU is generated, is something I am >> missing: if the filesystem contains files, well, then it is an "image" >> and can be built in OE, and then added to the list of artifacts. > > Yes, but you need a recipe per image. That's right - but I am not understanding why this is a problem when I can get as benefit much flexibility. > The WIC image can do this without > a special recipe per image / partition for arbitrary deploy artifacts. How ? It is not clear to me, maybe I haven't solved this. Let's say I need to create several filesystem for a device, and then I put all together with a WIC. But the WIC is specified for each filesystem, that is the deployable of each recipe having a "inherit image". How can I get a WIC to deploy all images at once ? Is it possible ? (I am not aware of it). > An common example is the boot partition or a partition with a filesystem > and the recovery FIT image. IMHO they are not so good examples - the boot (EFI) partition cannot be any kind of generated image, that means I cannot create an image as usual, set the filesystem to VFAT and put it as first partition via WIC. WIC generates the partition following a schema, and this matches for most cases (but not in all of them). > > >> If the filesystem is empty, if it is generated at build time it must >> be delivered, too, increasing the SWU size (but an empty filesystem >> can be well compressed), and it looks to me more suitable that the >> device does it. >> >> - is there another goal ? > > No. > > > [snip] > >>> The problem is that WIC could generate filesystems on the fly. You >>> doesn't need multiple recipe to create different filesystems. >> >> So is this the issue you want to address ? Anyway, the WIC is also >> quite limited. It works fine for the rootfs, and I can use the rawcopy >> plugin with --source rootfs. For some parts (see IMAGE_BOOTFILES), we >> can add also some files to a generated filesystem (EFI is the clear >> example). >> But I cannot combine partitions from different images, like a rootfs >> and a partition just containing the application files, that is >> generated with a different image recipe. > > This should be possible with the rawcopy plugin or you could split the > rootfs on the fly with --exclude-path and --change-directory. And again > you could avoid an additional recipe if you use the later option. Ok, so at the end the discussion is to avoid to have a second recipe. This is fine with me if it does not break any use case. This was not with Anatolji's patches, they are breaking some use cases of mine. > > > [snip] > >> Ok, I am quite sure now that the big issue on your use case is that >> filesystems on empty partitions are not generated when SWU is built. > > No, my issue is the needed extra recipes. Ok, understood. > WIC could create images with > renamed deployed artifacts on the fly and I could build arbitrary images. I know they can build artifacts on the fly, my intention is to have a SWU that is not aware of all about what is deployed. Anyway, I get your goal and if we find a suiztable way where the two ways can coexist, it is fine. > > > [snip] > >>> Does it make sense to use the same sw-description for different build >>> instructions if it excluded the source name of the artifacts. >> >> This is what I often do with variable substitution (for rootfs), >> having just one sw-description, but multiple recipes for the different >> use cases. I am trying to understand if the proposed is just an >> alternative implementation or there is really some added values. > > The added value is no needed extra recipes and that it works out of the > box. Ok, got it. > > > [snip] > >>>> Which is your main goal ? So it is more important to drop the recipe >>>> for >>>> the SWU or to drop the configuration via variables and use WKS like >>>> instructions ? Or both ? >>> >>> The main goal is to drop the duplicated recipe for the swu >> >> Ok, this is clear. >> >>> and to reuse the configurations of WIC. >> >> You mean same configuration ? So are you thinking to a WIC plugin that >> generates a SWU ? > > This is one possible solution. An other solution could be an extra > boot-image recipe. In any case we should reuse matching WIC variables. > > Regards > Stefan Regards, Stefano
Hi Stefano, Am 05.04.21 um 16:20 schrieb Stefano Babic: > Hi Stefan, > > On 05.04.21 15:36, Stefan Herbrechtsmeier wrote: >> Hi Stefano, >> >> apologize the long delay. >> > > No problem - it looks like that Eastern's holiday give us some time to > check again older topics ;-) > >> Am 01.03.21 um 10:02 schrieb Stefano Babic: >>> Hi Stefan, >>> >>> On 28.02.21 21:06, Stefan Herbrechtsmeier wrote: >>>> Hi Stefano, >>>> >>>> Am 17.02.21 um 10:02 schrieb Stefano Babic: >>>>> Hi Stefan, >>>>> >>>>> On 16.02.21 21:34, Stefan Herbrechtsmeier wrote: >>>>>> Hi Stefano, >>>>>> >>>>>> Am 16.02.21 um 09:02 schrieb Stefano Babic: >>>>>>> Hi Stefan, >>>>>>> >>>>>>> On 15.02.21 20:21, Stefan Herbrechtsmeier wrote: >>>>>>>> Hi Stefano, >>>>>>>> >>>>>>>> Am 14.02.21 um 18:15 schrieb Stefano Babic: >>>>>>>>> Hi Stefan, >>>>>>>>> >>>>>>>>> On 14.02.21 16:11, Stefan Herbrechtsmeier wrote: >>>>>>>>>> Hi Stefano, >>>>>>>>>> >>>>>>>>>> Am 14.02.21 um 10:34 schrieb Stefano Babic: >>>>>>>>>>> Hi Stefan, >>>>>>>>>>> >>>>>>>>>>> On 13.02.21 17:05, Stefan Herbrechtsmeier wrote: >>>>>>>>>>>> Hi Stefano, >>>>>>>>>>>> >>>>>>>>>>>> Am 11.02.21 um 13:25 schrieb Stefano Babic: >>>>>>>>>>>>> Hi Anatolji, >>>>>>>>>>>>> >>>>>>>>>>>>> On 11.02.21 11:38, Anatolij Gustschin wrote: >> >> [snip] >> >> For me the rootfs is only a specific image. An image consists of >> different deploy artifacts and / or packages. > > Right, I see the same0 > >> I distinguish between normal recipes which produce packages and deploy >> binary artifacts on one side and images which create special deploy >> artifacts only. >> >> I think the normal system update swu should be independent of the >> concrete rootfs image. > > IMHO it is independent now. The SWU does not know if one of the artifact > is an image. We can have a SWU with any kind of artifacts, even without > a rootfs. > >> At the moment I could create a wic image for an arbitrary rootfs image >> and I don't understand why this shouldn't be possible for system >> update swu. > > Well, right, but a WIC depends on an image that is a rootfs, while SWU > not. In fact, in a wic I set a --source rootfs if I want to put the > filesystem in one of the partition. I cannot (at the moment) add a > different image, like for example another filesystem. The only exception > is the generation of EFI images via plugin. Is the --source rootfs mandatory? Couldn't you create an empty rootfs and use rawcopy to reuse other image artifacts? >> In any case you could override the common configuration and use image >> specific configurations for specific images. > > That's right, but there are always a 1:1 relationship with a rootfs / > image (or how we name it). But I do not see how to build a single WIC > where each partitition is generated with a different recipe, and I put > each image in a different partition. Just as simple case: two images > with OS and application in separate partitions. This is possible. You could use rawcopy if you already have a recipe for the application image or you could install the application in a specific directory and use --exclude-path and --change-directory to split the rootfs into two partitions. > On the other side, I do not see how generated a wic from SWU can help. A > SWU is understood just from SWUpdate, it is not a filesystem (then > putting in a partition makes less sense). What do you mean? >>> I am quite losing which are the goals here or what we want to >>> address. I try to sumarize, maybe I misunderstand something: >>> >>> - we need currently a recipe per image in case of different rootfs, >>> and adding a .swu fstype lets to avoid the recipe. However, >>> sw-description cannot be generated as I said previously, it can be >>> factorized among images. >> >> A swu fstype allows a generic default sw-description per distro or >> machine and support an arbitrary rootfs. > > Yes, but this contradicts your first statement, that an image is just a > set of artifacts to be deplowed. A SWU fstype means a rootfs. You are right. It must be deployed artifacts and packages. >>> - there are some cases not covered with current generation, related >>> to filesystem generation. I am not sure I have understood this. >>> SWUpdate delivers artifacts. If a filesystem must be created, the >>> device can generally do this itself, first eample at first boot after >>> update. Or this can be executed inside the update process as part of >>> a preinstall part. Why this should be done when SWU is generated, is >>> something I am missing: if the filesystem contains files, well, then >>> it is an "image" and can be built in OE, and then added to the list >>> of artifacts. >> >> Yes, but you need a recipe per image. > > That's right - but I am not understanding why this is a problem when I > can get as benefit much flexibility. Because everybody have to pay the price even if he doesn't need the flexibility. The complexity should scale with the needed flexibility. > >> The WIC image can do this without a special recipe per image / >> partition for arbitrary deploy artifacts. > > How ? It is not clear to me, maybe I haven't solved this. Let's say I > need to create several filesystem for a device, and then I put all > together with a WIC. But the WIC is specified for each filesystem, that > is the deployable of each recipe having a "inherit image". How can I get > a WIC to deploy all images at once ? Is it possible ? (I am not aware of > it). What is the content of the filesystem? You could use rawcopy for filesystem images (created by other image recipes), bootimg-partition to create a filesystem from deployed images and split the rootfs into different partitions. It is possible to create partitions for boot loader, kernel, recovery, system and data without any additional recipe. >> An common example is the boot partition or a partition with a >> filesystem and the recovery FIT image. This is possible. We use a dual copy for boot loader, firmware, recovery, kernel and system. We create a wic image without any additional recipe. > IMHO they are not so good examples - the boot (EFI) partition cannot be > any kind of generated image, that means I cannot create an image as > usual, set the filesystem to VFAT and put it as first partition via WIC. Again this should be possible with rawcopy and an fstype class for vfat. But why do you need a special recipe for this image? bootimg-partition could create a vfat partition for you and copy and rename images from deploy into the filesystem. > WIC generates the partition following a schema, and this matches for > most cases (but not in all of them). What case isn't supported by wic? >>> If the filesystem is empty, if it is generated at build time it must >>> be delivered, too, increasing the SWU size (but an empty filesystem >>> can be well compressed), and it looks to me more suitable that the >>> device does it. >>> >>> - is there another goal ? >> >> No. >> >> >> [snip] >> >>>> The problem is that WIC could generate filesystems on the fly. You >>>> doesn't need multiple recipe to create different filesystems. >>> >>> So is this the issue you want to address ? Anyway, the WIC is also >>> quite limited. It works fine for the rootfs, and I can use the >>> rawcopy plugin with --source rootfs. For some parts (see >>> IMAGE_BOOTFILES), we can add also some files to a generated >>> filesystem (EFI is the clear example). >>> But I cannot combine partitions from different images, like a rootfs >>> and a partition just containing the application files, that is >>> generated with a different image recipe. >> >> This should be possible with the rawcopy plugin or you could split the >> rootfs on the fly with --exclude-path and --change-directory. And >> again you could avoid an additional recipe if you use the later option. > > Ok, so at the end the discussion is to avoid to have a second recipe. > This is fine with me if it does not break any use case. This was not > with Anatolji's patches, they are breaking some use cases of mine. Okay >> [snip] >> >>> Ok, I am quite sure now that the big issue on your use case is that >>> filesystems on empty partitions are not generated when SWU is built. >> >> No, my issue is the needed extra recipes. > > Ok, understood. > >> WIC could create images with renamed deployed artifacts on the fly and >> I could build arbitrary images. > > I know they can build artifacts on the fly, my intention is to have a > SWU that is not aware of all about what is deployed. What do you mean by this? Do you mean the license install example? > Anyway, I get your goal and if we find a suiztable way where the two > ways can coexist, it is fine. Okay. This should be possible as the two could be different classes. Regards Stefan
Hi Stefan, On 05.04.21 17:21, Stefan Herbrechtsmeier wrote: > Hi Stefano, > > Am 05.04.21 um 16:20 schrieb Stefano Babic: >> Hi Stefan, >> >> On 05.04.21 15:36, Stefan Herbrechtsmeier wrote: >>> Hi Stefano, >>> >>> apologize the long delay. >>> >> >> No problem - it looks like that Eastern's holiday give us some time to >> check again older topics ;-) >> >>> Am 01.03.21 um 10:02 schrieb Stefano Babic: >>>> Hi Stefan, >>>> >>>> On 28.02.21 21:06, Stefan Herbrechtsmeier wrote: >>>>> Hi Stefano, >>>>> >>>>> Am 17.02.21 um 10:02 schrieb Stefano Babic: >>>>>> Hi Stefan, >>>>>> >>>>>> On 16.02.21 21:34, Stefan Herbrechtsmeier wrote: >>>>>>> Hi Stefano, >>>>>>> >>>>>>> Am 16.02.21 um 09:02 schrieb Stefano Babic: >>>>>>>> Hi Stefan, >>>>>>>> >>>>>>>> On 15.02.21 20:21, Stefan Herbrechtsmeier wrote: >>>>>>>>> Hi Stefano, >>>>>>>>> >>>>>>>>> Am 14.02.21 um 18:15 schrieb Stefano Babic: >>>>>>>>>> Hi Stefan, >>>>>>>>>> >>>>>>>>>> On 14.02.21 16:11, Stefan Herbrechtsmeier wrote: >>>>>>>>>>> Hi Stefano, >>>>>>>>>>> >>>>>>>>>>> Am 14.02.21 um 10:34 schrieb Stefano Babic: >>>>>>>>>>>> Hi Stefan, >>>>>>>>>>>> >>>>>>>>>>>> On 13.02.21 17:05, Stefan Herbrechtsmeier wrote: >>>>>>>>>>>>> Hi Stefano, >>>>>>>>>>>>> >>>>>>>>>>>>> Am 11.02.21 um 13:25 schrieb Stefano Babic: >>>>>>>>>>>>>> Hi Anatolji, >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 11.02.21 11:38, Anatolij Gustschin wrote: >>> >>> [snip] >>> >>> For me the rootfs is only a specific image. An image consists of >>> different deploy artifacts and / or packages. >> >> Right, I see the same0 >> >>> I distinguish between normal recipes which produce packages and >>> deploy binary artifacts on one side and images which create special >>> deploy artifacts only. >>> >>> I think the normal system update swu should be independent of the >>> concrete rootfs image. >> >> IMHO it is independent now. The SWU does not know if one of the >> artifact is an image. We can have a SWU with any kind of artifacts, >> even without a rootfs. >> >>> At the moment I could create a wic image for an arbitrary rootfs >>> image and I don't understand why this shouldn't be possible for >>> system update swu. >> >> Well, right, but a WIC depends on an image that is a rootfs, while SWU >> not. In fact, in a wic I set a --source rootfs if I want to put the >> filesystem in one of the partition. I cannot (at the moment) add a >> different image, like for example another filesystem. The only >> exception is the generation of EFI images via plugin. > > Is the --source rootfs mandatory? Couldn't you create an empty rootfs > and use rawcopy to reuse other image artifacts? No, it is not - yes, an empty fs + rawcopy will work, too. > > >>> In any case you could override the common configuration and use image >>> specific configurations for specific images. >> >> That's right, but there are always a 1:1 relationship with a rootfs / >> image (or how we name it). But I do not see how to build a single WIC >> where each partitition is generated with a different recipe, and I put >> each image in a different partition. Just as simple case: two images >> with OS and application in separate partitions. > > This is possible. You could use rawcopy if you already have a recipe for > the application image or you could install the application in a specific > directory and use --exclude-path and --change-directory to split the > rootfs into two partitions. Ok, nice > > >> On the other side, I do not see how generated a wic from SWU can help. >> A SWU is understood just from SWUpdate, it is not a filesystem (then >> putting in a partition makes less sense). > > What do you mean? I guess I have mixed up, ignore this sentence... > > >>>> I am quite losing which are the goals here or what we want to >>>> address. I try to sumarize, maybe I misunderstand something: >>>> >>>> - we need currently a recipe per image in case of different rootfs, >>>> and adding a .swu fstype lets to avoid the recipe. However, >>>> sw-description cannot be generated as I said previously, it can be >>>> factorized among images. >>> >>> A swu fstype allows a generic default sw-description per distro or >>> machine and support an arbitrary rootfs. >> >> Yes, but this contradicts your first statement, that an image is just >> a set of artifacts to be deplowed. A SWU fstype means a rootfs. > > You are right. It must be deployed artifacts and packages. Ok > > >>>> - there are some cases not covered with current generation, related >>>> to filesystem generation. I am not sure I have understood this. >>>> SWUpdate delivers artifacts. If a filesystem must be created, the >>>> device can generally do this itself, first eample at first boot >>>> after update. Or this can be executed inside the update process as >>>> part of a preinstall part. Why this should be done when SWU is >>>> generated, is something I am missing: if the filesystem contains >>>> files, well, then it is an "image" and can be built in OE, and then >>>> added to the list of artifacts. >>> >>> Yes, but you need a recipe per image. >> >> That's right - but I am not understanding why this is a problem when I >> can get as benefit much flexibility. > > Because everybody have to pay the price even if he doesn't need the > flexibility. The complexity should scale with the needed flexibility. That's right - we need a way to have both without one of them that forbids the other one. > >> >>> The WIC image can do this without a special recipe per image / >>> partition for arbitrary deploy artifacts. >> >> How ? It is not clear to me, maybe I haven't solved this. Let's say I >> need to create several filesystem for a device, and then I put all >> together with a WIC. But the WIC is specified for each filesystem, >> that is the deployable of each recipe having a "inherit image". How >> can I get a WIC to deploy all images at once ? Is it possible ? (I am >> not aware of it). > > What is the content of the filesystem? You could use rawcopy for > filesystem images (created by other image recipes), mmmhh...yes, --source rawcopy --sourceparams=... should be enough for that. > bootimg-partition to > create a filesystem from deployed images and split the rootfs into > different partitions. It is possible to create partitions for boot > loader, kernel, recovery, system and data without any additional recipe. > > >>> An common example is the boot partition or a partition with a >>> filesystem and the recovery FIT image. > > This is possible. We use a dual copy for boot loader, firmware, > recovery, kernel and system. We create a wic image without any > additional recipe. Ok > > >> IMHO they are not so good examples - the boot (EFI) partition cannot >> be any kind of generated image, that means I cannot create an image as >> usual, set the filesystem to VFAT and put it as first partition via WIC. > > Again this should be possible with rawcopy and an fstype class for vfat. > But why do you need a special recipe for this image? No, I wanted just to find examples. > bootimg-partition > could create a vfat partition for you and copy and rename images from > deploy into the filesystem. Yes, that is fine. > > >> WIC generates the partition following a schema, and this matches for >> most cases (but not in all of them). > > What case isn't supported by wic? Well, the most frequent use case for me to use wic is to deploy a first image or if the device boots from a SD device. In all cases, where the boot device is different from the main storage (SPI vs eMMC or NAND, for example), it is useless. > > >>>> If the filesystem is empty, if it is generated at build time it must >>>> be delivered, too, increasing the SWU size (but an empty filesystem >>>> can be well compressed), and it looks to me more suitable that the >>>> device does it. >>>> >>>> - is there another goal ? >>> >>> No. >>> >>> >>> [snip] >>> >>>>> The problem is that WIC could generate filesystems on the fly. You >>>>> doesn't need multiple recipe to create different filesystems. >>>> >>>> So is this the issue you want to address ? Anyway, the WIC is also >>>> quite limited. It works fine for the rootfs, and I can use the >>>> rawcopy plugin with --source rootfs. For some parts (see >>>> IMAGE_BOOTFILES), we can add also some files to a generated >>>> filesystem (EFI is the clear example). >>>> But I cannot combine partitions from different images, like a rootfs >>>> and a partition just containing the application files, that is >>>> generated with a different image recipe. >>> >>> This should be possible with the rawcopy plugin or you could split >>> the rootfs on the fly with --exclude-path and --change-directory. And >>> again you could avoid an additional recipe if you use the later option. >> >> Ok, so at the end the discussion is to avoid to have a second recipe. >> This is fine with me if it does not break any use case. This was not >> with Anatolji's patches, they are breaking some use cases of mine. > > Okay > >>> [snip] >>> >>>> Ok, I am quite sure now that the big issue on your use case is that >>>> filesystems on empty partitions are not generated when SWU is built. >>> >>> No, my issue is the needed extra recipes. >> >> Ok, understood. >> >>> WIC could create images with renamed deployed artifacts on the fly >>> and I could build arbitrary images. >> >> I know they can build artifacts on the fly, my intention is to have a >> SWU that is not aware of all about what is deployed. > > What do you mean by this? Do you mean the license install example? > > >> Anyway, I get your goal and if we find a suiztable way where the two >> ways can coexist, it is fine. > > Okay. This should be possible as the two could be different classes. +1 We can maybe factorize some part (common functions), but without dependencies that block one usage or the other one. Best regards, Stefano