[OpenWrt-Devel] build: allow openwrt.git packages to be replaced by feeds

Currently, replacing a package available in openwrt.git requires
modifications in openwrt.git, or requires duplicating the package in a
feed but with a different name, which causes all kind of problems
related to dependencies (all packages selecting it would have to be
modified accordingly to select the new package).

With this change, if a package with the same name is present both in
feeds/ and package/ folders, the one in feeds/ will override the one
in package/, both in the menuconfig and during the build.

This mechanism is particularly useful for vendor tree, or in general for
application which needs to replace one particular package which exists
within openwrt.git by a custom/newer version.

Signed-off-by: Mathieu Olivari <mathieu@qca.qualcomm.com>
---
 include/scan.awk |   17 +++++++++++++++++
 include/scan.mk  |    2 +-
 scripts/feeds    |    9 ++++++---
 3 files changed, 24 insertions(+), 4 deletions(-)
 create mode 100644 include/scan.awk

On 16/01/2015 22:45, Mathieu Olivari wrote:
> Currently, replacing a package available in openwrt.git requires 
> modifications in openwrt.git, or requires duplicating the package
> in a feed but with a different name, which causes all kind of
> problems related to dependencies (all packages selecting it would
> have to be modified accordingly to select the new package).
> 
> With this change, if a package with the same name is present both
> in feeds/ and package/ folders, the one in feeds/ will override the
> one in package/, both in the menuconfig and during the build.
> 
> This mechanism is particularly useful for vendor tree, or in
> general for application which needs to replace one particular
> package which exists within openwrt.git by a custom/newer version.
> 

i think this is a bad solution. this is like overriding files in /sbin
with ones in ~/sbin. it will cause lots of side effects and bogus bug
reports. how about you simply upstream your magic feature and have it
added to packages.git ? your patch seems to work around qca not
upstreaming its work properly.


> Signed-off-by: Mathieu Olivari <mathieu@qca.qualcomm.com> --- 
> include/scan.awk |   17 +++++++++++++++++ include/scan.mk  |    2
> +- scripts/feeds    |    9 ++++++--- 3 files changed, 24
> insertions(+), 4 deletions(-) create mode 100644 include/scan.awk
> 
> diff --git a/include/scan.awk b/include/scan.awk new file mode
> 100644 index 0000000..39b2977 --- /dev/null +++ b/include/scan.awk 
> @@ -0,0 +1,17 @@ +BEGIN { FS="/" } +$1 ~ /^feeds/ { FEEDS[$NF]=$0
> } +$1 !~ /^feeds/ { PKGS[$NF]=$0 } +END { +	# Filter-out OpenWrt
> packages which have a feeds equivalent +	for (pkg in PKGS) +		if
> (pkg in FEEDS) +			delete PKGS[pkg] +	n = asort(PKGS) +	for (i=1; i
> <= n; i++) { +		print PKGS[i] +	} +	n = asort(FEEDS) +	for (i=1; i
> <= n; i++){ +		print FEEDS[i] +	} +} diff --git a/include/scan.mk
> b/include/scan.mk index 0998333..138707d 100644 ---
> a/include/scan.mk +++ b/include/scan.mk @@ -43,7 +43,7 @@ endef
> 
> $(FILELIST): rm -f $(TMP_DIR)/info/.files-$(SCAN_TARGET)-* -	$(call
> FIND_L, $(SCAN_DIR)) $(SCAN_EXTRA) -mindepth 1 $(if
> $(SCAN_DEPTH),-maxdepth $(SCAN_DEPTH)) -name Makefile | xargs grep
> -HE 'call (Build/DefaultTargets|Build(Package|Target)|.+Package)' |
> sed -e 's#^$(SCAN_DIR)/##' -e 's#/Makefile:.*##' | uniq > $@ +
> $(call FIND_L, $(SCAN_DIR)) $(SCAN_EXTRA) -mindepth 1 $(if
> $(SCAN_DEPTH),-maxdepth $(SCAN_DEPTH)) -name Makefile | xargs grep
> -HE 'call (Build/DefaultTargets|Build(Package|Target)|.+Package)' |
> sed -e 's#^$(SCAN_DIR)/##' -e 's#/Makefile:.*##' | uniq | awk -f
> include/scan.awk > $@
> 
> $(TMP_DIR)/info/.files-$(SCAN_TARGET).mk: $(FILELIST) ( \ diff
> --git a/scripts/feeds b/scripts/feeds index 31ad544..01d9041
> 100755 --- a/scripts/feeds +++ b/scripts/feeds @@ -384,12 +384,15
> @@ sub install_package {
> 
> # previously installed packages set the runtime package # newly
> installed packages set the source package -	$installed{$src} and
> return 0; +	$installed{$src} and !is_core_package($src) and return
> 0;
> 
> # check previously installed packages -	$installed{$name} and
> return 0; +	$installed{$name} and !is_core_package($name) and
> return 0; $installed{$src} = 1; -	warn "Installing package
> '$src'\n"; + +	is_core_package($src) +		and warn "Overriding
> package '$src'\n" +		or warn "Installing package '$src'\n";
> 
> $install_method{$type} or do { warn "Unknown installation method:
> '$type'\n";
>

For packages like mac80211, a lot of contributions go straight upstream to the kernel actually. It's understandable that openwrt.git maintains its own stabilization branch using quilt on this package.
But it also makes sense for some users to generate OpenWrt firmware with a more recent version of backports, typically from the wireless-testing tip. To test a certain class of drivers as an example.

Maintaining its own mac80211 or hostapd package (to pull it from their respective tip) is currently very painful. It requires maintaining a separate patch-set, sometimes requires small customizations to scripts; each of them conflicting every time a new openwrt version gets pulled & rebased.

Not being able to replace core packages while still pulling/rebasing openwrt.git easily is one of the reasons that makes OpenWrt contributions difficult.
The whole reason behind this patch is to facilitate OpenWrt contributions, not prevent them.

We can talk about the implementation; but I think leaving the opportunity to replace core packages when necessary would be useful to a lot of people.

Thanks,
Mathieu