Message ID | 20221213130718.1727795-1-vicamo.yang@canonical.com |
---|---|
State | New |
Headers | show |
Series | [SRU,PULL,linux-firmware,Lunar] UBUNTU: [Packaging] scripts: add back list-lts-update-files | expand |
Applied to linux-firmware lunar branch. Note that your submission email also contains the patch content inline. Please don't do that. I also just realized that the patch author email and SOB are different (gmail vs. canonical). I accept it this time but only because I missed it in your jammy submission. In the future, please make sure the SOB and author name/email are identical for patches that you author yourself. ...Juerg > BugLink: https://bugs.launchpad.net/bugs/1999406 > > [Impact] > > When migrating to a new hwe kernel or introduced a oem kernel of a > newer version, the firmware blobs might not match the expectation of > the new kernel. For example, a hwe-5.15 on Focal may include new > drivers that relies on firmware blobs that are only available in Jammy. > > [Fix] > > A previously obsoleted (and removed in Kinetic and Lunar) script > `list-lts-update-files` is used to enumerate firmware blobs to be > backported. It's retored and revised with additional support to > pick the right blobs especially in regard to ath and iwlwifi drivers. > > On Focal, the commits in need are enumerated by: > > $ debian/scripts/list-lts-update-files focal jammy \ > ../jammy/debian.master/abi/fwinfo | \ > xargs git log --graph --oneline focal..jammy -- > * 36f2ea9f7 ath11k: WCN6855 hw2.0: add WLAN.HSP.1.1-03125-QCAHSPSWPL_V1_V2_SILICONZ_LITE-3 > * 05b5dc001 ath11k: WCN6855 hw2.0: add board-2.bin and regdb.bin > * 7a3005059 brcm: Add 43455 based AP6255 NVRAM for the ACEPC T8 Mini PC > * ad185afa1 cypress: update firmware for cyw4373 sdio > * 77d3eb8ef cypress: update firmware for cyw43570 pcie > * 92e9acdbd cypress: update firmware for cyw4356 sdio > * 5f88084be cypress: update firmware for cyw4354 sdio > * f97e31677 cypress: update firmware for cyw43455 sdio > * 3df9ea0b9 cypress: update firmware for cyw43430 sdio > * 6150015cf cypress: update firmware for cyw43012 sdio > * 2548d065b brcm: Add nvram for the Chuwi Hi8 (CWI509) tablet > * e45c137e7 brcm: Add nvram for the Predia Basic tablet > * d52886242 brcm: Add NVRAM for Vamrs 96boards Rock960 > * b503c9660 Merge branch 'ath10k-20201023' of git://git.kernel.org/pub/scm/linux/kernel/git/kvalo/linux-firmware into main > |\ > | * 34cb5fce2 ath11k: IPQ8074 hw2.0: add to WLAN.HK.2.1.0.1-01238-QCAHKSWPL_SILICONZ-2 > | * c0a8efd24 ath11k: IPQ8074 hw2.0: add board-2.bin > | * ac7f5e93f ath11k: IPQ6018 hw1.0: add to WLAN.HK.2.1.0.1-01238-QCAHKSWPL_SILICONZ-2 > | * 2594e510a ath11k: IPQ6018 hw1.0: add board-2.bin > * 04f71fe56 cypress: add Cypress firmware and clm_blob files > > Commit 05b5dc001 and 36f2ea9f7 include updates to other existing files, > so they are dropped from this SRU. An additional commit that modifies > only WHENCE, commit 0b558e8a7, found when resolving conflicts, were also > added. > > On Jammy, > > $ debian/scripts/list-lts-update-files jammy kinetic \ > ../kinetic/debian.master/abi/fwinfo | \ > xargs git log --graph --oneline jammy..kinetic -- > * 35f8de521 UBUNTU: Add pre-compiled echoaudio firmware > * c954892f6 Add initial AzureWave AW-CM256SM NVRAM file > * 86f0c5642 ath10k: WCN3990 hw1.0: add board-2.bin > * c3624ebd6 Merge branch 'ath10k-20220423' of git://git.kernel.org/pub/scm/linux/kernel/git/kvalo/linux-firmware into main > |\ > | * 1962cbab3 ath10k: QCA99X0 hw2.0: add board-2.bin > | * 0d5e9f7e0 ath11k: WCN6750 hw1.0: add to WLAN.MSL.1.0.1-00887-QCAMSLSWPLZ-1 > | * a50132f7f ath11k: WCN6750 hw1.0: add board-2.bin > | * 97f8b7563 ath11k: QCN9074 hw1.0: add to WLAN.HK.2.5.0.1-01208-QCAHKSWPL_SILICONZ-1 > | * f56505fe2 ath11k: QCN9074 hw1.0: add board-2.bin > * | dfa3c4c30 qcom: add firmware files for Adreno a420 & related generations > * | 4a43f1a84 qcom: add firmware files for Adreno a330 > * | ac21ab5d1 Mellanox: Add lc_ini_bundle for xx.2010.1006 > |/ > * 4ffcf980a brcm: rename Rock960 NVRAM to AP6356S and link devices to it > > Commit 4ffcf980a involves rename existing files in WHENCE, so ignored. > > While Jammy has 3 additional oem kernels, they were also tested: > > $ debian/scripts/list-lts-update-files jammy kinetic \ > ../oem-5.17/debian.oem/abi/fwinfo | \ > xargs git log --graph --oneline jammy..kinetic -- > (no new commits) > > $ debian/scripts/list-lts-update-files jammy lunar \ > ../oem-6.0/debian.oem/abi/fwinfo | \ > xargs git log --graph --oneline jammy..lunar -- > * 06dbfbc74 iwlwifi: add new FWs from core69-81 release > > $ debian/scripts/list-lts-update-files jammy lunar \ > ../oem-6.1/debian.oem/abi/fwinfo | \ > xargs git log --graph --oneline jammy..lunar -- > * 51fff4e69 i915: Add versionless HuC files for current platforms > > On Lunar, while it's not an LTS series and there will not be any hwe/oem > kernel backported, the only change proposed is the script itself. > > [Test Case] > > To ensure no exiting bits being overwritten, the changes are reviewed > manually. > > [Where problems could occur] > > These firmware blobs are only referenced in the new hwe/oem kernels, and > shall not have side effect. On Focal and Jammy, while we have rolled out > oem and hwe kernels a long time ago, these changes will enable the > corresponding hardware that were meant to be enabled along with the > oem/hwe kernel migration at the time. > > [Other Info] > > While there is no hwe/oem kernel planned for Kinetic, and unlike the > devel series Lunar, adding this script to it is quite meaningless, so > it's not nominated for fix here. > > ---------------------------------------------------------------- > The following changes since commit e3ff59307071a7867b3436f48bf69a7156abd2f1: > > UBUNTU: Ubuntu-20221212.git0707b2f2-0ubuntu1 (2022-12-12 08:51:34 +0100) > > are available in the Git repository at: > > https://git.launchpad.net/~vicamo/ubuntu/+source/linux-firmware bug-1999406/update-fw-for-hwe-kernels/lunar > > for you to fetch changes up to 834d95c984cd90dba9c6e9d497732fa08dd9a920: > > UBUNTU: [Packaging] scripts: add back list-lts-update-files (2022-12-13 20:05:46 +0800) > > ---------------------------------------------------------------- > You-Sheng Yang (vicamo) (1): > UBUNTU: [Packaging] scripts: add back list-lts-update-files > > debian/scripts/list-lts-update-files | 248 +++++++++++++++++++++++++++++++++++ > 1 file changed, 248 insertions(+) > create mode 100755 debian/scripts/list-lts-update-files > > diff --git a/debian/scripts/list-lts-update-files b/debian/scripts/list-lts-update-files > new file mode 100755 > index 000000000..46fa3dfff > --- /dev/null > +++ b/debian/scripts/list-lts-update-files > @@ -0,0 +1,248 @@ > +#!/bin/bash > +# > +# Used to get a list of firmware files needed to update a LTS > +# linux-firmware package for a backport kernel. For example to > +# get a list of files for trusty to support the linux-lts-xenial > +# kernel run: > +# > +# debian/scripts/list-lts-update-files [--dump-filtered] trusty xenial /path/to/xenial/fwinfo > +# > +# This will generate a list of files that satisfy all of these > +# conditions: > +# > +# 1) In the xenial branch of linux-firmware > +# 2) Not in the trusty branch of linux-firmware > +# 3) In the specified fwinfo file > +# > +# This can be used as a starting point for finding upstream > +# commits needed to add the missing firmware files. > +# > +# The fwinfo file can be omitted, in which case the output is a list > +# of firmware files in @newbranch but not in @oldbranch. > + > +#set -ex > + > +dump_filtered= > +if [ $# -ge 1 ]; then > + case "$1" in > + --dump-filtered) > + dump_filtered=1; shift;; > + *) ;; > + esac > +fi > + > +if [ $# -lt 2 ]; then > + echo "Usage $0 [--dump-filtered] <oldbranch> <newbranch> [<fwinfo>]" > + exit 1 > +fi > + > +OLDBRANCH="$1" > +NEWBRANCH="$2" > +FWINFO="$3" > + > +TMPDIR=$(mktemp -d) > + > +function cleanup { > + rm -r "$TMPDIR" > +} > +trap cleanup EXIT > + > +function normalize_link { > + # handle relative paths: a/b/c/../../d/e -> a/d/e > + local s="$1" > + s="${s#./}" > + while true; do > + case "$s" in > + */../*) > + s=$(echo "$s" | sed 's,[^/]\+/../,,') > + ;; > + *) > + break ;; > + esac > + done > + echo "$s" > +} > + > +function grep_links { > + grep ^WHENCE "$1" | sed "s,^,$2:," | xargs git show | grep ^Link: > +} > + > +if ! git ls-tree -r --name-only "$OLDBRANCH" >"$TMPDIR/files.tmp" > +then > + exit 1 > +fi > +cat "$TMPDIR/files.tmp" | sort >"$TMPDIR/oldbranch-files" > + > +if ! git ls-tree -r --name-only "$NEWBRANCH" >"$TMPDIR/files.tmp" > +then > + exit 1 > +fi > +cat "$TMPDIR/files.tmp" | sort >"$TMPDIR/newbranch-files" > + > +# Get files in @newbranch not in @oldbranch > +comm -13 "$TMPDIR/oldbranch-files" "$TMPDIR/newbranch-files" >"$TMPDIR/new-files" > + > +if [ "$3" == "" ]; then > + cat "$TMPDIR/new-files" > + exit 0 > +fi > + > +if ! cat $FWINFO | head -n1 | grep "firmware: " > /dev/null > +then > + echo "Invliad fwinfo file $FWINFO" > + exit 1 > +fi > + > +declare -A oldbranch_files > +while read of; do > + oldbranch_files["$of"]="old" > +done < "$TMPDIR/oldbranch-files" > + > +while read dc1 ol dc2 of; do > + if [ -z "${oldbranch_files[$ol]:-}" ]; then > + oldbranch_files["$ol"]="link" > + fi > +done < <(grep_links "$TMPDIR/oldbranch-files" "$OLDBRANCH") > + > +declare -A new_files > +while read nf; do > + if [ -n "${oldbranch_files["$nf"]:-}" ]; then > + # new file but matches a previous link > + new_files["$nf"]="duplicated" > + elif [ -n "$(git log "$OLDBRANCH" -1 -- "$nf")" ]; then > + new_files["$nf"]="deleted" > + else > + new_files["$nf"]="unused" > + fi > +done < "$TMPDIR/new-files" > + > +declare -A newlink_files > +while read dc1 nl dc2 nf; do > + > + nf="$(normalize_link "$(dirname "$nl")/$nf")" > + newlink_files["$nl"]="$nf" > +done < <(grep_links "$TMPDIR/newbranch-files" "$NEWBRANCH") > + > +function check_duplicated { > + if [ -n "${oldbranch_files["$1"]:-}" ]; then > + return 0 > + fi > + > + if [ -n "${newlink_files["$1"]:-}" ]; then > + if check_duplicated "${newlink_files["$1"]}"; then > + return 0 > + fi > + fi > + > + return 1 > +} > + > +function mark_if_unused { > + if [ -n "${new_files["$1"]:-}" ]; then > + if [ "${new_files["$1"]}" = "unused" ]; then > + new_files["$1"]="$2" > + fi > + > + return 0 > + fi > + > + if [ -n "${newlink_files["$1"]:-}" ]; then > + if mark_if_unused "${newlink_files["$1"]}" "$2"; then > + return 0 > + fi > + fi > + > + return 1 > +} > + > +while read fw_pattern; do > + fws=() > + > + case "$fw_pattern" in > + *\**) > + # pattern with wildcard > + for nf in "${!new_files[@]}"; do > + case "$nf" in > + $fw_pattern) > + fws+=("$nf");; > + *) > + ;; > + esac > + done > + ;; > + *) > + fws+=("$fw_pattern") > + ;; > + esac > + > + for fw in "${fws[@]}"; do > + case "$fw" in > + iwlwifi-*.ucode) > + # check if new_files contains one with lower-equal FW API > + prefix="${fw%-*}" > + max_fw_api="${fw##*-}" > + max_fw_api="${max_fw_api%.ucode}" > + found= > + for api in $(seq $max_fw_api -1 1); do > + nf="$prefix-$api.ucode" > + > + if [ -n "$found" ]; then > + mark_if_unused "$nf" "iwlwifi fwapi skipped" || true > + continue > + fi > + > + if check_duplicated "$nf"; then > + mark_if_unused "$nf" "duplicated" || true > + found=1 > + elif mark_if_unused "$nf" "new"; then > + found=1 > + fi > + done > + ;; > + *) > + if check_duplicated "$fw"; then > + mark_if_unused "$fw" "duplicated" || true > + else > + mark_if_unused "$fw" "new" || true > + fi > + ;; > + esac > + done > +done < <(cat "$FWINFO" | sed 's/firmware: \+//' | sort) > + > +declare -A ath_hidden_fw > +for nf in "${!new_files[@]}"; do > + [ "${new_files[$nf]}" = "unused" ] || continue > + > + case "$nf" in > + ath*k/*) > + model=${nf#*/} > + model=${model%%/*} > + if [ -z "${ath_hidden_fw[$model]}" ]; then > + if ! grep -q "/$model/" "$FWINFO"; then > + ath_hidden_fw["$model"]=1 > + fi > + fi > + > + # ath*k build firmware path at runtime, so set as new if not > + # previously marked duplicated. > + if [ -n "${ath_hidden_fw[$model]:-}" ]; then > + if check_duplicated "$nf"; then > + mark_if_unused "$nf" "duplicated" || true > + else > + mark_if_unused "$nf" "new" || true > + fi > + fi > + ;; > + esac > +done > + > +for nf in "${!new_files[@]}"; do > + [ "${new_files[$nf]}" = "new" ] && echo "$nf" > +done | sort > + > +if [ -n "$dump_filtered" ]; then > + for nf in "${!new_files[@]}"; do > + [ "${new_files[$nf]}" = "new" ] || echo "# $nf: ${new_files[$nf]}" > + done | sort > +fi >
diff --git a/debian/scripts/list-lts-update-files b/debian/scripts/list-lts-update-files new file mode 100755 index 000000000..46fa3dfff --- /dev/null +++ b/debian/scripts/list-lts-update-files @@ -0,0 +1,248 @@ +#!/bin/bash +# +# Used to get a list of firmware files needed to update a LTS +# linux-firmware package for a backport kernel. For example to +# get a list of files for trusty to support the linux-lts-xenial +# kernel run: +# +# debian/scripts/list-lts-update-files [--dump-filtered] trusty xenial /path/to/xenial/fwinfo +# +# This will generate a list of files that satisfy all of these +# conditions: +# +# 1) In the xenial branch of linux-firmware +# 2) Not in the trusty branch of linux-firmware +# 3) In the specified fwinfo file +# +# This can be used as a starting point for finding upstream +# commits needed to add the missing firmware files. +# +# The fwinfo file can be omitted, in which case the output is a list +# of firmware files in @newbranch but not in @oldbranch. + +#set -ex + +dump_filtered= +if [ $# -ge 1 ]; then + case "$1" in + --dump-filtered) + dump_filtered=1; shift;; + *) ;; + esac +fi + +if [ $# -lt 2 ]; then + echo "Usage $0 [--dump-filtered] <oldbranch> <newbranch> [<fwinfo>]" + exit 1 +fi + +OLDBRANCH="$1" +NEWBRANCH="$2" +FWINFO="$3" + +TMPDIR=$(mktemp -d) + +function cleanup { + rm -r "$TMPDIR" +} +trap cleanup EXIT + +function normalize_link { + # handle relative paths: a/b/c/../../d/e -> a/d/e + local s="$1" + s="${s#./}" + while true; do + case "$s" in + */../*) + s=$(echo "$s" | sed 's,[^/]\+/../,,') + ;; + *) + break ;; + esac + done + echo "$s" +} + +function grep_links { + grep ^WHENCE "$1" | sed "s,^,$2:," | xargs git show | grep ^Link: +} + +if ! git ls-tree -r --name-only "$OLDBRANCH" >"$TMPDIR/files.tmp" +then + exit 1 +fi +cat "$TMPDIR/files.tmp" | sort >"$TMPDIR/oldbranch-files" + +if ! git ls-tree -r --name-only "$NEWBRANCH" >"$TMPDIR/files.tmp" +then + exit 1 +fi +cat "$TMPDIR/files.tmp" | sort >"$TMPDIR/newbranch-files" + +# Get files in @newbranch not in @oldbranch +comm -13 "$TMPDIR/oldbranch-files" "$TMPDIR/newbranch-files" >"$TMPDIR/new-files" + +if [ "$3" == "" ]; then + cat "$TMPDIR/new-files" + exit 0 +fi + +if ! cat $FWINFO | head -n1 | grep "firmware: " > /dev/null +then + echo "Invliad fwinfo file $FWINFO" + exit 1 +fi + +declare -A oldbranch_files +while read of; do + oldbranch_files["$of"]="old" +done < "$TMPDIR/oldbranch-files" + +while read dc1 ol dc2 of; do + if [ -z "${oldbranch_files[$ol]:-}" ]; then + oldbranch_files["$ol"]="link" + fi +done < <(grep_links "$TMPDIR/oldbranch-files" "$OLDBRANCH") + +declare -A new_files +while read nf; do + if [ -n "${oldbranch_files["$nf"]:-}" ]; then + # new file but matches a previous link + new_files["$nf"]="duplicated" + elif [ -n "$(git log "$OLDBRANCH" -1 -- "$nf")" ]; then + new_files["$nf"]="deleted" + else + new_files["$nf"]="unused" + fi +done < "$TMPDIR/new-files" + +declare -A newlink_files +while read dc1 nl dc2 nf; do + + nf="$(normalize_link "$(dirname "$nl")/$nf")" + newlink_files["$nl"]="$nf" +done < <(grep_links "$TMPDIR/newbranch-files" "$NEWBRANCH") + +function check_duplicated { + if [ -n "${oldbranch_files["$1"]:-}" ]; then + return 0 + fi + + if [ -n "${newlink_files["$1"]:-}" ]; then + if check_duplicated "${newlink_files["$1"]}"; then + return 0 + fi + fi + + return 1 +} + +function mark_if_unused { + if [ -n "${new_files["$1"]:-}" ]; then + if [ "${new_files["$1"]}" = "unused" ]; then + new_files["$1"]="$2" + fi + + return 0 + fi + + if [ -n "${newlink_files["$1"]:-}" ]; then + if mark_if_unused "${newlink_files["$1"]}" "$2"; then + return 0 + fi + fi + + return 1 +} + +while read fw_pattern; do + fws=() + + case "$fw_pattern" in + *\**) + # pattern with wildcard + for nf in "${!new_files[@]}"; do + case "$nf" in + $fw_pattern) + fws+=("$nf");; + *) + ;; + esac + done + ;; + *) + fws+=("$fw_pattern") + ;; + esac + + for fw in "${fws[@]}"; do + case "$fw" in + iwlwifi-*.ucode) + # check if new_files contains one with lower-equal FW API + prefix="${fw%-*}" + max_fw_api="${fw##*-}" + max_fw_api="${max_fw_api%.ucode}" + found= + for api in $(seq $max_fw_api -1 1); do + nf="$prefix-$api.ucode" + + if [ -n "$found" ]; then + mark_if_unused "$nf" "iwlwifi fwapi skipped" || true + continue + fi + + if check_duplicated "$nf"; then + mark_if_unused "$nf" "duplicated" || true + found=1 + elif mark_if_unused "$nf" "new"; then + found=1 + fi + done + ;; + *) + if check_duplicated "$fw"; then + mark_if_unused "$fw" "duplicated" || true + else + mark_if_unused "$fw" "new" || true + fi + ;; + esac + done +done < <(cat "$FWINFO" | sed 's/firmware: \+//' | sort) + +declare -A ath_hidden_fw +for nf in "${!new_files[@]}"; do + [ "${new_files[$nf]}" = "unused" ] || continue + + case "$nf" in + ath*k/*) + model=${nf#*/} + model=${model%%/*} + if [ -z "${ath_hidden_fw[$model]}" ]; then + if ! grep -q "/$model/" "$FWINFO"; then + ath_hidden_fw["$model"]=1 + fi + fi + + # ath*k build firmware path at runtime, so set as new if not + # previously marked duplicated. + if [ -n "${ath_hidden_fw[$model]:-}" ]; then + if check_duplicated "$nf"; then + mark_if_unused "$nf" "duplicated" || true + else + mark_if_unused "$nf" "new" || true + fi + fi + ;; + esac +done + +for nf in "${!new_files[@]}"; do + [ "${new_files[$nf]}" = "new" ] && echo "$nf" +done | sort + +if [ -n "$dump_filtered" ]; then + for nf in "${!new_files[@]}"; do + [ "${new_files[$nf]}" = "new" ] || echo "# $nf: ${new_files[$nf]}" + done | sort +fi