From patchwork Mon Apr 2 13:09:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Hadjinlian X-Patchwork-Id: 894186 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="qbvZc2qq"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40FCHn5W4Wz9s0p for ; Mon, 2 Apr 2018 23:09:57 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id E0B252E5CF; Mon, 2 Apr 2018 13:09:55 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id fhqfpEhIhegQ; Mon, 2 Apr 2018 13:09:54 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 5FC41227E9; Mon, 2 Apr 2018 13:09:54 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id 049B31C2314 for ; Mon, 2 Apr 2018 13:09:50 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 01476227E9 for ; Mon, 2 Apr 2018 13:09:50 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TxVGNvbgS6dx for ; Mon, 2 Apr 2018 13:09:49 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr0-f194.google.com (mail-wr0-f194.google.com [209.85.128.194]) by silver.osuosl.org (Postfix) with ESMTPS id 04DDB22761 for ; Mon, 2 Apr 2018 13:09:49 +0000 (UTC) Received: by mail-wr0-f194.google.com with SMTP id y55so13691843wry.3 for ; Mon, 02 Apr 2018 06:09:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=EWaX5mGVfX01qFvW/Wj3LKx4829Wr6xBFQrckVvrEIE=; b=qbvZc2qqB4HImt9VFPkDG507AEnWR/zKlYXuiZ1l/xHBYZEyGBa2+qH8d0VHKfQ8ve zV6kv32v2E7tpRh7LWe7yjroaUmToAZvjpOzeGAyri1NSEgHmp7J/lLoKLVMAsSbi7o0 ARIpwgSL+nR0DvBocu5WLW2R25e0zZOezrl2wzBIwbMj47JRvUYKtbu5X94G5kbm+OtO TpQhOVrjqhxb3n1sDVS1CrWCpLucsR4w96KbyGNu6WOQ8zoE9eCVaTLiv8kHmFTB/TdD 27R5nJlRDWyE1j59D8CNYrCQffIaJ4KYPA1Rrr3+HkJ7rM3GBvSXXGpDvkE6edzkkBzl ZyPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=EWaX5mGVfX01qFvW/Wj3LKx4829Wr6xBFQrckVvrEIE=; b=AsouY+AfrjWn1MMnsKfUafuCW7qnnYWlSsA6xrgAWUnNLlzVNRWvPS3t6WAA+mYak6 lewHzPksLKbY3Cd29RvTlQ8z3MgK0v/FiIfdd0YV9XLnOwzUe75V8YWn+PLtq6uByE59 NAT15q2b0PoQF+QmSV0J0KVBNSs5hT9Fl/n2pOukawH+jPkx+1ZGedDwSAJoAhwZW84O n5ueyzyFd5Otuqs0U/6LUm6pfPi5jEc2aaut2eikYKUQV90Ruu+wtS4waaMMIF35QWRH W8d6RpiDUv4NxvnuU7aO4hVU3Bg7jZWzPCbrKvTbY0jXxivHb37IlLsoRuVNfJpIGjuy xr4A== X-Gm-Message-State: AElRT7GZRTlI2AuYFrTAhHt74dsl8isFCmtI/IG/EgxTr/2uLOUXRbpD V+1CXjhnrsxOxqADhZkLLIEKKBql X-Google-Smtp-Source: AIpwx4/6LB3/Q+yIuHZtfhw2UPCtjqUUXlpXsPtbhzK+8AyiTzHkgD7LJZ9XbpeN6fVFFOxHnaLCvw== X-Received: by 10.223.187.75 with SMTP id x11mr6150104wrg.217.1522674587109; Mon, 02 Apr 2018 06:09:47 -0700 (PDT) Received: from cerise.online.net ([195.154.228.195]) by smtp.gmail.com with ESMTPSA id p187sm1134507wme.8.2018.04.02.06.09.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 Apr 2018 06:09:46 -0700 (PDT) From: Maxime Hadjinlian To: buildroot@buildroot.org Date: Mon, 2 Apr 2018 15:09:34 +0200 Message-Id: <20180402130934.19274-11-maxime.hadjinlian@gmail.com> X-Mailer: git-send-email 2.16.2 In-Reply-To: <20180402130934.19274-1-maxime.hadjinlian@gmail.com> References: <20180402130934.19274-1-maxime.hadjinlian@gmail.com> Subject: [Buildroot] [v5 11/11] download: git: introduce cache feature X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Maxime Hadjinlian MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Now we keep the git clone that we download and generates our tarball from there. The main goal here is that if you change the version of a package (say Linux), instead of cloning all over again, you will simply 'git fetch' from the repo the missing objects, then generates the tarball again. This should speed the 'source' part of the build significantly. The drawback is that the DL_DIR will grow much larger; but time is more important than disk space nowadays. Signed-off-by: Maxime Hadjinlian --- v4 -> v5: - Fix comments's content --- support/download/git | 64 ++++++++++++++++++++++++++++++++-------------------- 1 file changed, 40 insertions(+), 24 deletions(-) diff --git a/support/download/git b/support/download/git index 58a2c6ad9d..0f8097b61f 100755 --- a/support/download/git +++ b/support/download/git @@ -39,29 +39,41 @@ _git() { eval ${GIT} "${@}" } -# Try a shallow clone, since it is faster than a full clone - but that only -# works if the version is a ref (tag or branch). Before trying to do a shallow -# clone we check if ${cset} is in the list provided by git ls-remote. If not -# we fall back on a full clone. +# We want to check if a cache of the git clone of this repo already exists. +git_cache="${BR2_DL_DIR}/${basename%%-*}/git" + +# If the cache directory doesn't exists, init a new repo, which will be +# fetch'ed later. +if [ ! -d "${git_cache}" ]; then + _git init "'${git_cache}'" + _git -C "'${git_cache}'" remote add origin "'${uri}'" +fi + +pushd "${git_cache}" >/dev/null + +_git remote set-url origin "'${uri}'" + +# Try to fetch with limited depth, since it is faster than a full clone - but +# that only works if the version is a ref (tag or branch). Before trying to do +# a shallow clone we check if ${cset} is in the list provided by git ls-remote. +# If not we fallback to a full fetch. # -# Messages for the type of clone used are provided to ease debugging in case of -# problems +# Messages for the type of clone used are provided to ease debugging in +# case of problems git_done=0 -if [ -n "$(_git ls-remote "'${uri}'" "'${cset}'" 2>&1)" ]; then - printf "Doing shallow clone\n" - if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${uri}'" "'${basename}'"; then +if [ -n "$(_git ls-remote origin "'${cset}'" 2>&1)" ]; then + printf "Doing a shallow fetch\n" + if _git fetch "${@}" --depth 1 origin "'${cset}'"; then git_done=1 else - printf "Shallow clone failed, falling back to doing a full clone\n" + printf "Shallow fetch failed, falling back to fetching all refs\n" fi fi if [ ${git_done} -eq 0 ]; then - printf "Doing full clone\n" - _git clone ${verbose} "${@}" "'${uri}'" "'${basename}'" + printf "Fetching all references\n" + _git fetch origin -t fi -pushd "${basename}" >/dev/null - # Try to get the special refs exposed by some forges (pull-requests for # github, changes for gerrit...). There is no easy way to know whether # the cset the user passed us is such a special ref or a tag or a sha1 @@ -86,20 +98,24 @@ if [ ${recurse} -eq 1 ]; then _git submodule update --init --recursive fi -# We do not want the .git dir; we keep other .git files, in case they -# are the only files in their directory. +# Generate the archive, sort with the C locale so that it is reproducible. +# We do not want the .git dir; we keep other .git files, in case they are the +# only files in their directory. # The .git dir would generate non reproducible tarballs as it depends on # the state of the remote server. It also would generate large tarballs # (gigabytes for some linux trees) when a full clone took place. -rm -rf .git +find . -not -type d \ + -and -not -path "./.git/*" >"${output}.list" +LC_ALL=C sort <"${output}.list" >"${output}.list.sorted" -popd >/dev/null - -# Generate the archive, sort with the C locale so that it is reproducible -find "${basename}" -not -type d >"${basename}.list" -LC_ALL=C sort <"${basename}.list" >"${basename}.list.sorted" # Create GNU-format tarballs, since that's the format of the tarballs on # sources.buildroot.org and used in the *.hash files -tar cf - --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \ - -T "${basename}.list.sorted" >"${output}.tar" +tar cf - --transform="s/^\.$/${basename}/" \ + --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \ + -T "${output}.list.sorted" >"${output}.tar" gzip -6 -n <"${output}.tar" >"${output}" + +rm -f "${output}.list" +rm -f "${output}.list.sorted" + +popd >/dev/null