[v3,2/6] Add documentation for merged defconfigs

Signed-off-by: Sam Bobroff <sam.bobroff@au1.ibm.com>
---
 docs/manual/adding-board-support.txt |  3 ++
 docs/manual/appendix.txt             |  1 +
 docs/manual/merged-defconfigs.txt    | 60 ++++++++++++++++++++++++++++++++++++
 3 files changed, 64 insertions(+)
 create mode 100644 docs/manual/merged-defconfigs.txt

Hi Sam,

Le 29/06/2016 à 07:06, Sam Bobroff a écrit :
> Signed-off-by: Sam Bobroff <sam.bobroff@au1.ibm.com>
> ---
>  docs/manual/adding-board-support.txt |  3 ++
>  docs/manual/appendix.txt             |  1 +
>  docs/manual/merged-defconfigs.txt    | 60 ++++++++++++++++++++++++++++++++++++
>  3 files changed, 64 insertions(+)
>  create mode 100644 docs/manual/merged-defconfigs.txt
> 
> diff --git a/docs/manual/adding-board-support.txt b/docs/manual/adding-board-support.txt
> index f6d74ae..086a67e 100644
> --- a/docs/manual/adding-board-support.txt
> +++ b/docs/manual/adding-board-support.txt
> @@ -24,6 +24,9 @@ savedefconfig+. This will generate a minimal +defconfig+ file at the
>  root of the Buildroot source tree. Move this file into the +configs/+
>  directory, and rename it +<boardname>_defconfig+.
>  
> +It is also possible to create configurations by specifying differences from
> +an existing configuration. See xref:merged-defconfigs[].
> +
>  It is recommended to use as much as possible upstream versions of the
>  Linux kernel and bootloaders, and to use as much as possible default
>  kernel and bootloader configurations. If they are incorrect for your
> diff --git a/docs/manual/appendix.txt b/docs/manual/appendix.txt
> index 87a20bd..9abc37e 100644
> --- a/docs/manual/appendix.txt
> +++ b/docs/manual/appendix.txt
> @@ -3,6 +3,7 @@
>  
>  include::makedev-syntax.txt[]
>  include::makeusers-syntax.txt[]
> +include::merged-defconfigs.txt[]
>  
>  
>  // Automatically generated lists:
> diff --git a/docs/manual/merged-defconfigs.txt b/docs/manual/merged-defconfigs.txt
> new file mode 100644
> index 0000000..e1f50fd
> --- /dev/null
> +++ b/docs/manual/merged-defconfigs.txt
> @@ -0,0 +1,60 @@
> +// -*- mode:doc -*- ;
> +// vim: set syntax=asciidoc:
> +
> +[[merged-defconfigs]]
> +== Merged configurations
> +
> +Buildroot supports the use of the Linux kernel's +merge_config.sh+ script to
> +create configurations by merging a set of partial configurations (or overlays)
> +on top of a base configuration. This may be useful when there are several
> +configurations that are specializations of a single generic configuration.
> +
> +Files in the +configs/+ directory with the extension
> ++.mk+ will be read by the top level Makefile and the Make function
> ++merge_config+ can be used within them to define new merged configurations.
> +
> +The merge_config function takes three parameters: the first is the name of the
> +new configuration (which should end with +_defconfig+), the second is the name
> +of the base configuration file and the third is a space separated list of the
> +partial configuration files to merge from the +configs/+ directory. All files
> +should be specified relative to the +configs/+ directory and the name of the
> +Make fragment file should match the name of the configuration it creates.

This series has been discussed with other developers and we concluded that this
feature is not the responsibility of Buildroot to generate a defconfig from
partial configuration files.

Instead we recommend to use merge_config.sh from outside of Buildroot to
generate the merged defconfig before calling make. Maybe we should add some
documentation in the manual to explain how to use it.

Best regards,
Romain

> +
> +The new configuration can be selected as if it were a normal configuration, i.e. with
> ++make foo_defconfig+.
> +
> +Partial configurations have the same format as full configuration
> +files except that they contain only a subset of values. They can be created by
> +starting with a normal configuration file and deleting lines as appropriate.
> +
> +External configuration files from the +$(BR2_EXTERNAL)/configs+ directory are fully
> +supported, and merge configurations may freely refer to files from either
> +location.
> +
> +Example:
> +
> +If configs/foo_defconfig contains:
> +-----
> +BR2_TARGET_GENERIC_HOSTNAME="buildroot"
> +-----
> +
> +And configs/bar_defconfig.mk contains:
> +-----
> +$(eval $(call merge_config,bar_defconfig,foo_defconfig,b.config))
> +-----
> +
> +And configs/b.config contains:
> +-----
> +BR2_TARGET_GENERIC_ISSUE="Welcome to BAR"
> +-----
> +
> +After running the command:
> +-----
> +make bar_defconfig
> +-----
> +
> +The resulting .config file would contain:
> +-----
> +BR2_TARGET_GENERIC_HOSTNAME="buildroot"
> +BR2_TARGET_GENERIC_ISSUE="Welcome to BAR"
> +-----
>

On Fri, Jul 01, 2016 at 03:18:58PM +0200, Romain Naour wrote:
> Hi Sam,
> 
[snip]
> 
> This series has been discussed with other developers and we concluded that this
> feature is not the responsibility of Buildroot to generate a defconfig from
> partial configuration files.

Fair enough.

> Instead we recommend to use merge_config.sh from outside of Buildroot to
> generate the merged defconfig before calling make. Maybe we should add some
> documentation in the manual to explain how to use it.

That seems fine, it might even be possible to add something to
$(BR2_EXTERNAL)/external.mk to hook it directly into the Makefiles.

Could you elaborate a bit on what you mean by using merge_config.sh outside of
buildroot? Would the fragment files still be checked into buildroot somewhere
(but just not hooked into the build system) or would they be maintained outside
it?

> Best regards,
> Romain

Cheers,
Sam.

Hello Sam,

On Tue, 5 Jul 2016 16:06:25 +1000, Sam Bobroff wrote:

> > Instead we recommend to use merge_config.sh from outside of Buildroot to
> > generate the merged defconfig before calling make. Maybe we should add some
> > documentation in the manual to explain how to use it.  
> 
> That seems fine, it might even be possible to add something to
> $(BR2_EXTERNAL)/external.mk to hook it directly into the Makefiles.
> 
> Could you elaborate a bit on what you mean by using merge_config.sh outside of
> buildroot? Would the fragment files still be checked into buildroot somewhere
> (but just not hooked into the build system) or would they be maintained outside
> it?

The idea is that Buildroot just takes as input a configuration file.
How this configuration file is provided/generated is not Buildroot's
responsibility.

For example, you could have in your BR2_EXTERNAL tree a bunch of
defconfig fragments, and then a "genconfig" shell script to generate the
final configuration you want to build using the defconfig fragments.
This "genconfig" script relies on merge_config.sh to actually merge the
relevant fragments together.

Thomas

On Fri, Jul 01, 2016 at 03:18:58PM +0200, Romain Naour wrote:
> Hi Sam,
> 
> This series has been discussed with other developers and we concluded that this
> feature is not the responsibility of Buildroot to generate a defconfig from
> partial configuration files.
> 
> Instead we recommend to use merge_config.sh from outside of Buildroot to
> generate the merged defconfig before calling make. Maybe we should add some
> documentation in the manual to explain how to use it.
> 
> Best regards,
> Romain
> 

Earlier we had two independent patch sets that we were asked to merge.

   1) Merged defconfigs (now this patch set).
   2) Subdirs for defconfigs.

If #1 has been rejected as undesireable, should we now prepare an
indepenent patch set for #2 or is that also not wanted now?

There is currently no way for us to manage this behavior in a
BR2_EXTERNAL layer alone because there is no Makefile from the
BR2_EXTERNAL path sourced in the defconfig case.  Is there any issue
with a patch that will source a file like $(BR2_EXTERNAL)/external-defconfig.mk?

I realize we could have a new script, but we already have users that are
use to 'make <system>_defconfig && make' so we would like to be able to
preserve that behavior even though we are moving to merged defconfig
fragments.

On Tue, 2016-07-05 at 14:32 -0500, Patrick Williams wrote:
> On Fri, Jul 01, 2016 at 03:18:58PM +0200, Romain Naour wrote:
> > Hi Sam,
> > 
> > This series has been discussed with other developers and we concluded that this
> > feature is not the responsibility of Buildroot to generate a defconfig from
> > partial configuration files.
> > 
> > Instead we recommend to use merge_config.sh from outside of Buildroot to
> > generate the merged defconfig before calling make. Maybe we should add some
> > documentation in the manual to explain how to use it.
> > 
> > Best regards,
> > Romain
> > 
> 
> Earlier we had two independent patch sets that we were asked to merge.
> 
>    1) Merged defconfigs (now this patch set).
>    2) Subdirs for defconfigs.
> 
> If #1 has been rejected as undesireable, should we now prepare an
> indepenent patch set for #2 or is that also not wanted now?
> 
> There is currently no way for us to manage this behavior in a
> BR2_EXTERNAL layer alone because there is no Makefile from the
> BR2_EXTERNAL path sourced in the defconfig case.  Is there any issue
> with a patch that will source a file like $(BR2_EXTERNAL)/external-defconfig.mk?

Actually there might be - Sam was telling me yesterday that the main
Makefile includes $(BR2_EXTERNAL)/external.mk, which is something we may
use to get mergeconfigs in op-build. That might work for nested configs
as well.

> 
> I realize we could have a new script, but we already have users that are
> use to 'make <system>_defconfig && make' so we would like to be able to
> preserve that behavior even though we are moving to merged defconfig
> fragments.
>

On Wed, Jul 06, 2016 at 08:13:43AM +1000, Samuel Mendoza-Jonas wrote:
> On Tue, 2016-07-05 at 14:32 -0500, Patrick Williams wrote:
> > On Fri, Jul 01, 2016 at 03:18:58PM +0200, Romain Naour wrote:
> > > Hi Sam,
> > > 
> > > This series has been discussed with other developers and we concluded that this
> > > feature is not the responsibility of Buildroot to generate a defconfig from
> > > partial configuration files.
> > > 
> > > Instead we recommend to use merge_config.sh from outside of Buildroot to
> > > generate the merged defconfig before calling make. Maybe we should add some
> > > documentation in the manual to explain how to use it.
> > > 
> > > Best regards,
> > > Romain
> > > 
> > 
> > Earlier we had two independent patch sets that we were asked to merge.
> > 
> >    1) Merged defconfigs (now this patch set).
> >    2) Subdirs for defconfigs.
> > 
> > If #1 has been rejected as undesireable, should we now prepare an
> > indepenent patch set for #2 or is that also not wanted now?
> > 
> > There is currently no way for us to manage this behavior in a
> > BR2_EXTERNAL layer alone because there is no Makefile from the
> > BR2_EXTERNAL path sourced in the defconfig case.  Is there any issue
> > with a patch that will source a file like $(BR2_EXTERNAL)/external-defconfig.mk?
> 
> Actually there might be - Sam was telling me yesterday that the main
> Makefile includes $(BR2_EXTERNAL)/external.mk, which is something we may
> use to get mergeconfigs in op-build. That might work for nested configs
> as well.
> 

I've already been down this route.  The 'include
$(BR2_EXTERNAL)/external.mk' is inside a big 'ifeq
($(BR2_HAVE_DOT_CONFIG),y)' on line 336.  That effectively means if you
don't already have a .config (which comes about from running a
'defconfig' recipe) then you don't get a source to external.mk.  That is
why I'm asking about a new 'external-defconfig.mk'.

> > 
> > I realize we could have a new script, but we already have users that are
> > use to 'make <system>_defconfig && make' so we would like to be able to
> > preserve that behavior even though we are moving to merged defconfig
> > fragments.
> > 
>

On Tue, Jul 05, 2016 at 08:16:44AM +0200, Thomas Petazzoni wrote:
> Hello Sam,
> 
> On Tue, 5 Jul 2016 16:06:25 +1000, Sam Bobroff wrote:
> 
> > > Instead we recommend to use merge_config.sh from outside of Buildroot to
> > > generate the merged defconfig before calling make. Maybe we should add some
> > > documentation in the manual to explain how to use it.  
> > 
> > That seems fine, it might even be possible to add something to
> > $(BR2_EXTERNAL)/external.mk to hook it directly into the Makefiles.
> > 
> > Could you elaborate a bit on what you mean by using merge_config.sh outside of
> > buildroot? Would the fragment files still be checked into buildroot somewhere
> > (but just not hooked into the build system) or would they be maintained outside
> > it?
> 
> The idea is that Buildroot just takes as input a configuration file.
> How this configuration file is provided/generated is not Buildroot's
> responsibility.
> 
> For example, you could have in your BR2_EXTERNAL tree a bunch of
> defconfig fragments, and then a "genconfig" shell script to generate the
> final configuration you want to build using the defconfig fragments.
> This "genconfig" script relies on merge_config.sh to actually merge the
> relevant fragments together.
> 

Ah right, yes I'm sure something like that will work for work that stays
external to buildroot, but the case I'm concerned about is where BR2_EXTERNAL
isn't being used. We want to get our defconfigs out there for everyone to use
and contribute to :-) It looks like existing defconfigs could make use of this
too: from a bit of diff'ing in the configs directory there are already configs
that share a lot of values and would be easier to maintain using merging.

So how would you suggest handling merged configs that should be contributed
back to the community? Would you expect that we maintain our own merge-config
system and fragments and then contribute the full configs back to buildroot?

That seems like the simplest way to do it but I can't see a good way to handle
patches to them: users who have downloaded buildroot would have the defconfigs
but not the input fragments that were used to create them, so it would be
difficult to handle changes they need to make.

Would an alternative approach be to include the fragments and genconfig system
in buildroot but make it a separate component, so it wouldn't add complexity to
the build system? (e.g. you "make configs" to update your defconfigs.)

Cheers,
Sam.

Hi Patrick, Samuel, All,

Le 06/07/2016 à 00:41, Patrick Williams a écrit :
> On Wed, Jul 06, 2016 at 08:13:43AM +1000, Samuel Mendoza-Jonas wrote:
>> On Tue, 2016-07-05 at 14:32 -0500, Patrick Williams wrote:
>>> On Fri, Jul 01, 2016 at 03:18:58PM +0200, Romain Naour wrote:
>>>> Hi Sam,
>>>>
>>>> This series has been discussed with other developers and we concluded that this
>>>> feature is not the responsibility of Buildroot to generate a defconfig from
>>>> partial configuration files.
>>>>
>>>> Instead we recommend to use merge_config.sh from outside of Buildroot to
>>>> generate the merged defconfig before calling make. Maybe we should add some
>>>> documentation in the manual to explain how to use it.
>>>>
>>>> Best regards,
>>>> Romain
>>>>
>>>
>>> Earlier we had two independent patch sets that we were asked to merge.
>>>
>>>    1) Merged defconfigs (now this patch set).
>>>    2) Subdirs for defconfigs.
>>>
>>> If #1 has been rejected as undesireable, should we now prepare an
>>> indepenent patch set for #2 or is that also not wanted now?
>>>
>>> There is currently no way for us to manage this behavior in a
>>> BR2_EXTERNAL layer alone because there is no Makefile from the
>>> BR2_EXTERNAL path sourced in the defconfig case.  Is there any issue
>>> with a patch that will source a file like $(BR2_EXTERNAL)/external-defconfig.mk?
>>
>> Actually there might be - Sam was telling me yesterday that the main
>> Makefile includes $(BR2_EXTERNAL)/external.mk, which is something we may
>> use to get mergeconfigs in op-build. That might work for nested configs
>> as well.
>>
> 
> I've already been down this route.  The 'include
> $(BR2_EXTERNAL)/external.mk' is inside a big 'ifeq
> ($(BR2_HAVE_DOT_CONFIG),y)' on line 336.  That effectively means if you
> don't already have a .config (which comes about from running a
> 'defconfig' recipe) then you don't get a source to external.mk.  That is
> why I'm asking about a new 'external-defconfig.mk'.

Patrick, you're right. It's currently no way to include an external makefile
when BR2_HAVE_DOT_CONFIG is not set. Neither from local.mk or external.mk.

Indeed, adding $(BR2_EXTERNAL)/external-defconfig.mk should work in this case.

I tried with this patch [1] and used Samuel's patch [2] applied to
external-defconfig.mk instead of the Buildroot Makefile.

Lets see what other Buidroot developers think about this, especially Yann which
is currently working on multi-BR2_EXTERNAL.

About the nested configs directory, I don't see any particular problem...

Best regards,
Romain


[1]
https://github.com/RomainNaour/buildroot/commit/396744b032567b0d7fcc6e3d56c37143efde368e
[2] http://patchwork.ozlabs.org/patch/641838/

> 
>>>
>>> I realize we could have a new script, but we already have users that are
>>> use to 'make <system>_defconfig && make' so we would like to be able to
>>> preserve that behavior even though we are moving to merged defconfig
>>> fragments.
>>>
>>
> 
> 
> 
> _______________________________________________
> buildroot mailing list
> buildroot@busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot
>

Romain, Samuel, Patrick, All,

On 2016-07-06 21:08 +0200, Romain Naour spake thusly:
> Le 06/07/2016 à 00:41, Patrick Williams a écrit :
> > On Wed, Jul 06, 2016 at 08:13:43AM +1000, Samuel Mendoza-Jonas wrote:
> >> On Tue, 2016-07-05 at 14:32 -0500, Patrick Williams wrote:
> >>> On Fri, Jul 01, 2016 at 03:18:58PM +0200, Romain Naour wrote:
> >>>> Hi Sam,
> >>>>
> >>>> This series has been discussed with other developers and we concluded that this
> >>>> feature is not the responsibility of Buildroot to generate a defconfig from
> >>>> partial configuration files.
> >>>>
> >>>> Instead we recommend to use merge_config.sh from outside of Buildroot to
> >>>> generate the merged defconfig before calling make. Maybe we should add some
> >>>> documentation in the manual to explain how to use it.
> >>>>
> >>>> Best regards,
> >>>> Romain
> >>>>
> >>>
> >>> Earlier we had two independent patch sets that we were asked to merge.
> >>>
> >>>    1) Merged defconfigs (now this patch set).
> >>>    2) Subdirs for defconfigs.
> >>>
> >>> If #1 has been rejected as undesireable, should we now prepare an
> >>> indepenent patch set for #2 or is that also not wanted now?
> >>>
> >>> There is currently no way for us to manage this behavior in a
> >>> BR2_EXTERNAL layer alone because there is no Makefile from the
> >>> BR2_EXTERNAL path sourced in the defconfig case.  Is there any issue
> >>> with a patch that will source a file like $(BR2_EXTERNAL)/external-defconfig.mk?

I can understand the frustration. I am sorry that we came to that
conclusion and that you feel disapointed.

We did reject this because, as Romain explained, we believe it is much
more flexible to handle such a case outside of Buildroot, as Romain
explained.

> >> Actually there might be - Sam was telling me yesterday that the main
> >> Makefile includes $(BR2_EXTERNAL)/external.mk, which is something we may
> >> use to get mergeconfigs in op-build. That might work for nested configs
> >> as well.
> >>
> > 
> > I've already been down this route.  The 'include
> > $(BR2_EXTERNAL)/external.mk' is inside a big 'ifeq
> > ($(BR2_HAVE_DOT_CONFIG),y)' on line 336.  That effectively means if you
> > don't already have a .config (which comes about from running a
> > 'defconfig' recipe) then you don't get a source to external.mk.  That is
> > why I'm asking about a new 'external-defconfig.mk'.
> 
> Patrick, you're right. It's currently no way to include an external makefile
> when BR2_HAVE_DOT_CONFIG is not set. Neither from local.mk or external.mk.
> 
> Indeed, adding $(BR2_EXTERNAL)/external-defconfig.mk should work in this case.
> 
> I tried with this patch [1] and used Samuel's patch [2] applied to
> external-defconfig.mk instead of the Buildroot Makefile.
[--SNIP--]
> [1] https://github.com/RomainNaour/buildroot/commit/396744b032567b0d7fcc6e3d56c37143efde368e

No, we already rejected a similar proposal (by me!) to source custom
help from br2-external:
    http://lists.busybox.net/pipermail/buildroot/2016-March/thread.html#156705
    http://lists.busybox.net/pipermail/buildroot/2016-April/thread.html#158710

The correct solution would be that we source external.mk unconditionally.
The patch is actually trivial (I had it locally some time ago, when
working on the aforementioned custom help).

For this, we should ensure it is safe to do so, but in my (very little)
epxerimentation, it seemed OK to do so. However, we must be really
carefull in doing so.

Regards,
Yann E. MORIN.