From patchwork Mon Sep 14 23:48:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sam Voss X-Patchwork-Id: 1364002 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=rockwellcollins.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=rockwellcollins.com header.i=@rockwellcollins.com header.a=rsa-sha256 header.s=hrcrc2020 header.b=kxpe5o4v; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4Br33f60cLz9sTp for ; Tue, 15 Sep 2020 09:49:01 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 14F5885DF7; Mon, 14 Sep 2020 23:48:59 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TZ7Bji_QkEXD; Mon, 14 Sep 2020 23:48:58 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 1CDF385778; Mon, 14 Sep 2020 23:48:58 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id EB8261BF348 for ; Mon, 14 Sep 2020 23:48:55 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id CE1D120419 for ; Mon, 14 Sep 2020 23:48:55 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZXIpo3hkcxvL for ; Mon, 14 Sep 2020 23:48:54 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from ch3vs04.rockwellcollins.com (ch3vs04.rockwellcollins.com [205.175.226.52]) by silver.osuosl.org (Postfix) with ESMTPS id 13E4E20384 for ; Mon, 14 Sep 2020 23:48:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rockwellcollins.com; s=hrcrc2020; h=from:to:cc:subject:date:message-id; bh=/yAM8glOwcfEz8Y14jEU8fV/ScPVWw+26AyfFbI5o/M=; b=kxpe5o4vaxCCuH9O2pY6Ner8CT1LwGhxKQ3hZcT6P44bk58ZKjBk/LBi DRgLKUB7c4VB5Cj7trQgTWf9upR+mN+RyUocXr8cEoOlPTTDTXPd1UD4Y Yi3/d3wWkP0xLoe7BjSonKa4rQuWtvFiDzUSrX8hDohE/+UG1s9OOObgf 5bFEQDvDzUuO0JbdnS6++/VvtxLtV/Kaq00tWvDt1n5yAh+6IlAts1iEQ 0xvzM5Un6sDUOYgLeu7RHWqdpHOlE8KRdIx6uL0GBMRjRbV/M/A+gh5vu MFl8xbCbebZTL6+Hg9i/frD1KzcWZJWjnivLwsxvLs7+eK5wd8NlT5b8W g==; IronPort-SDR: iPe/1e2aiomV5QUpIypu5lDAW1TSxahgPXFa02UQ5A/Mi5KUJyygDGhMBxKjdAwU1rUXXesHoU AOHNMDbUqLK4WWWMMnLfPzXb/fzAW/xUtk9HCmQDCj7UZBCV4x2r+9qQ5LziQXFMlsCRIcMjZF 7usqcKfNETCT6TcCusbjKLdKfLbmhqjI1Vsfzi0/yjwNyedWRlE0LXt2ywpq8ziEJmI+VHA8Z5 8Q4IBPBRYZv6YqZKcFsiaLlgMjFQ68lACW2j89CHHFSfBu5UMlrs3Zuwwh/FkKD5097d8ii8L8 9TI= Received: from ofwch3n02.rockwellcollins.com (HELO ciulimr02.rockwellcollins.com) ([205.175.226.14]) by ch3vs04.rockwellcollins.com with ESMTP; 14 Sep 2020 18:48:52 -0500 X-Received: from eggs.rockwellcollins.com (eggs.rockwellcollins.lab [10.148.117.18]) by ciulimr02.rockwellcollins.com (Postfix) with ESMTP id 8CC7720052; Mon, 14 Sep 2020 18:48:52 -0500 (CDT) From: Sam Voss To: buildroot@buildroot.org, christian@paral.in, patrick.havelange@essensium.com Date: Mon, 14 Sep 2020 18:48:42 -0500 Message-Id: <20200914234842.9043-1-sam.voss@rockwellcollins.com> X-Mailer: git-send-email 2.17.1 Subject: [Buildroot] [RFC] support/cargo: split main package from vendor code X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Sam Voss MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Allow top-level packages to be split from the main code and vendor code so they will be archived separately. In this commit, it focuses on a main app compared to the open-source pieces pulled through crates.io required building for a rust package. In general, anything that will have a "backend2" (possibly Python/pip, Node/NPM, Rust/Cargo and so on) would be able to leverage this system. Instead of a single -.tar.gz we end up with that file plus --vendor.tar.gz which must be extracted into the main source as a post-extract step. This is a hacky initial implementation, and will need more cleanup. Signed-off-by: Sam Voss --- support/download/cargo | 26 +++++++++++++++++-- support/download/dl-wrapper | 25 ++++++++++++++++++ .../scripts/find-vendor-licenses | 7 ++++- 3 files changed, 55 insertions(+), 3 deletions(-) rename package/ripgrep/find_vendor_licenses.sh => support/scripts/find-vendor-licenses (64%) diff --git a/support/download/cargo b/support/download/cargo index 0ce94cf16b..6162c908d4 100755 --- a/support/download/cargo +++ b/support/download/cargo @@ -29,13 +29,19 @@ if [ $# -le 2 ] ; then exit 1 fi; +buildroot_dir=$(pwd) tmpd="$(mktemp -d -p "$2")" cd "${tmpd}" tar xf "${1}" cd ./*/ echo "Running cargo vendor." -CARGO_HOME="${3}"/cargo_home cargo vendor -q --locked VENDOR +CARGO_HOME="${3}"/cargo_home cargo vendor -q --locked ../VENDOR +pushd ../VENDOR +mkdir licenses +${buildroot_dir}/support/scripts/find-vendor-licenses . | xargs -I{} cp --parents {} -t licenses/ +popd + # Create the local .cargo/config with vendor info mkdir -p .cargo/ cat <.cargo/config @@ -56,10 +62,26 @@ date="2020-02-06 01:02:03" # Create GNU-format tarballs, since that's the format of the tarballs on # sources.buildroot.org and used in the *.hash files -echo "Creating final archive." +echo "Creating updated source archive, which will use the captured vendor info." tar cf new.tar --null --verbatim-files-from --numeric-owner --format=gnu \ --owner=0 --group=0 --mtime="${date}" -T files.list.sorted gzip -6 -n new.tar.gz mv "${1}" "${1}".old mv new.tar.gz "${1}" rm "${1}".old + +# Cache the vendor portion + +# Generate the archive, sort with the C locale so that it is reproducible. +find "$(basename "$OLDPWD/../VENDOR")" -not -type d -print0 >vendor-files.list +LC_ALL=C sort -z vendor-files.list.sorted +# let's use a fixed hardcoded date to be reproducible +date="2020-02-06 01:02:03" + +# Create GNU-format tarballs, since that's the format of the tarballs on +# sources.buildroot.org and used in the *.hash files +echo "Creating vendor info archive." +tar cf new.tar --null --verbatim-files-from --numeric-owner --format=gnu \ + --owner=0 --group=0 --mtime="${date}" -T vendor-files.list.sorted +gzip -6 -n vendor.tar.gz +mv vendor.tar.gz ${1}-vendor diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper index 5e52b3e60f..70dfa6f231 100755 --- a/support/download/dl-wrapper +++ b/support/download/dl-wrapper @@ -115,6 +115,9 @@ main() { tmpd="$(mktemp -d "${BUILD_DIR}/.${output##*/}.XXXXXX")" tmpf="${tmpd}/output" + # Some backends may also pull in recursive-vendor code (such as cargo) + tmpf_vendor="${tmpf}-vendor" + # Helpers expect to run in a directory that is *really* trashable, so # they are free to create whatever files and/or sub-dirs they might need. # Doing the 'cd' here rather than in all backends is easier. @@ -176,6 +179,7 @@ main() { # tmp_output is in the same directory as the final output, so we can # later move it atomically. tmp_output="$(mktemp "${output}.XXXXXX")" + tmp_vendor_output="$(mktemp "${output/.tar.gz/-vendor.tar.gz}.XXXXXX")" # 'mktemp' creates files with 'go=-rwx', so the files are not accessible # to users other than the one doing the download (and root, of course). @@ -192,6 +196,13 @@ main() { new_mode=$(printf "%04o" $((0${new_mode} & ~0$(umask)))) chmod ${new_mode} "${tmp_output}" + # same logic for vendor... + if [ ! -z "$backend2" ]; then + [ -x "${tmpf_vendor}" ] && new_mode=755 || new_mode=644 + new_mode=$(printf "%04o" $((0${new_mode} & ~0$(umask)))) + chmod ${new_mode} "${tmp_vendor_output}" + fi + # We must *not* unlink tmp_output, otherwise there is a small window # during which another download process may create the same tmp_output # name (very, very unlikely; but not impossible.) @@ -206,6 +217,13 @@ main() { rm -rf "${tmpd}" "${tmp_output}" exit 1 fi + + if [ ! -z "$backend2" ]; then + if ! cat "${tmpf_vendor}" >>"${tmp_vendor_output}"; then + rm -rf "${tmpd}" "${tmp_vendor_output}" + exit 1 + fi + fi rm -rf "${tmpd}" # tmp_output and output are on the same filesystem, so POSIX guarantees @@ -217,6 +235,13 @@ main() { exit 1 fi + if [ ! -z "$backend2" ]; then + if ! mv -f "${tmp_vendor_output}" "${output/.tar.gz/-vendor.tar.gz}"; then + rm -f "${tmp_vendor_output}" + exit 1 + fi + fi + return ${rc} } diff --git a/package/ripgrep/find_vendor_licenses.sh b/support/scripts/find-vendor-licenses similarity index 64% rename from package/ripgrep/find_vendor_licenses.sh rename to support/scripts/find-vendor-licenses index fd8638e399..d48227e28f 100755 --- a/package/ripgrep/find_vendor_licenses.sh +++ b/support/scripts/find-vendor-licenses @@ -1,6 +1,11 @@ #!/bin/sh # to run from the extracted final sources -find VENDOR/ -type f \ + +if [ $# != 1 ] || [ ! -d "$1" ]; then + echo "usage: $0 " +fi + +find $1 -type f \ \( -iname '*license*' -o -iname 'licence*' -o -iname 'copying*' -o -iname 'copyright*' \) \ -a -not -name '*.a' | \ sort