From patchwork Tue Jul 4 16:22:11 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Hadjinlian X-Patchwork-Id: 784177 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3x28SY2zvPz9t0j for ; Wed, 5 Jul 2017 02:23:25 +1000 (AEST) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="VHpFY7RC"; dkim-atps=neutral Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 3EED587C67; Tue, 4 Jul 2017 16:23:16 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 0UD1uj_UYfuA; Tue, 4 Jul 2017 16:23:13 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id D4C7D87686; Tue, 4 Jul 2017 16:23:01 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id E47881CEA53 for ; Tue, 4 Jul 2017 16:22:31 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 574D925FAE for ; Tue, 4 Jul 2017 16:22:31 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IBj5uEKTTZ8B for ; Tue, 4 Jul 2017 16:22:30 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr0-f178.google.com (mail-wr0-f178.google.com [209.85.128.178]) by silver.osuosl.org (Postfix) with ESMTPS id 79CBB25C31 for ; Tue, 4 Jul 2017 16:22:30 +0000 (UTC) Received: by mail-wr0-f178.google.com with SMTP id 77so251470483wrb.1 for ; Tue, 04 Jul 2017 09:22:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=3k0d1JfpmUSzAs5mb49cHR8jt6TSc8Znk/pEBHzTEEc=; b=VHpFY7RCW9SUUA1vDq/e7USKYnC/Bcuo3RDCsWzk9/anSNLMYxKuhe2TRhMMZWLRVD 0rmo05936lT48SiHBzdhrXXTdPIijtxeVrN7OU7jYXzva50ZyQ6F0zac4Q9/hbkC+kJT BLsoxycS1lxPiv9mGGrDu8ClDK7Xdbfw0sBZq1fg50dd/uOimOGsbppPJWObM6EzsLjA r2ovDC26WyH8aJh4Q2mCwX407l8B6UjK/IWcETpI/RmkuIxuoDwixIXDMB0JlD+PuHU9 rBLdnlvAU5GXXV5iwmfhdD1UANgqKa1BxKA+1mTpjiPrnxS6FAxmAb125K3YowEtEMMT Stcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=3k0d1JfpmUSzAs5mb49cHR8jt6TSc8Znk/pEBHzTEEc=; b=I9raY/iz14osSnfhk3obLrF+m5WfZ6HHWdxgkVhIvBznFWH8CbsD2GUOJM0r/LXq7D dM7P3Ae3irlRkzArS7x3aP8RN892ZukftZFswOrii7GqdE+OaorIpiXytDdJYXWoueVK Lhg9SxrmxkYs7orY5gSa924cqvyAGLYEr6vw80Hlgck6Rb51kGzFdOP3jPFvWxuKKlxQ UnVH16mBXyRl8Lxym7c/iQSV3hMPSeGvlOX65/At7fErm/yKTBPMNO6hJXsWKqnm2pH7 wk8SPZjzNwU7zK4Cc+7HIBa13sp/DMCNZo+W7K5W0k1rc2MgyLrBZqIHrbcHkSxzxUCN i7qg== X-Gm-Message-State: AKS2vOzErWqdw9CfG++2Nrvpokc2sfb0H6FD6QfNYOWi7hDzLetLK4gl i0IQdM1vOm+pW+jyWxU= X-Received: by 10.223.132.133 with SMTP id 5mr18334687wrg.132.1499185348780; Tue, 04 Jul 2017 09:22:28 -0700 (PDT) Received: from cerise.bzh.lan (2a01cb0886107300297010e522b1c364.ipv6.abo.wanadoo.fr. [2a01:cb08:8610:7300:2970:10e5:22b1:c364]) by smtp.gmail.com with ESMTPSA id u187sm3582012wmd.26.2017.07.04.09.22.27 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 04 Jul 2017 09:22:27 -0700 (PDT) From: Maxime Hadjinlian To: buildroot@buildroot.org Date: Tue, 4 Jul 2017 18:22:11 +0200 Message-Id: <20170704162211.13238-14-maxime.hadjinlian@gmail.com> X-Mailer: git-send-email 2.13.2 In-Reply-To: <20170704162211.13238-1-maxime.hadjinlian@gmail.com> References: <20170704162211.13238-1-maxime.hadjinlian@gmail.com> Subject: [Buildroot] [PATCH 13/13] download: git: introduce cache feature X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Now we keep the git clone that we download and generates our tarball from there. The main goal here is that if you change the version of a package (say Linux), instead of cloning all over again, you will simply 'git fetch' from the repo the missing objects, then generates the tarball again. This should speed the 'source' part of the build significantly. The drawback is that the DL_DIR will grow much larger; but time is more important than disk space nowadays. Signed-off-by: Maxime Hadjinlian --- support/download/git | 69 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 40 insertions(+), 29 deletions(-) diff --git a/support/download/git b/support/download/git index a49e448e60..834345b53a 100755 --- a/support/download/git +++ b/support/download/git @@ -39,28 +39,34 @@ _git() { eval ${GIT} "${@}" } -# Try a shallow clone, since it is faster than a full clone - but that only -# works if the version is a ref (tag or branch). Before trying to do a shallow -# clone we check if ${cset} is in the list provided by git ls-remote. If not -# we fall back on a full clone. -# -# Messages for the type of clone used are provided to ease debugging in case of -# problems -git_done=0 -if [ -n "$(_git ls-remote "'${uri}'" "'${cset}'" 2>&1)" ]; then - printf "Doing shallow clone\n" - if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${uri}'" "'${basename}'"; then - git_done=1 - else - printf "Shallow clone failed, falling back to doing a full clone\n" +# We want to check if a cache of the git clone of this repo already exists. +git_cache="${BR2_DL_DIR}/${basename%%-*}/git" + +# If the cache directory already exists, don't try to clone. +if [ ! -d "${git_cache}" ]; then + # Try a shallow clone, since it is faster than a full clone - but that + # only works if the versionis a ref (tag or branch). Before trying to do a + # shallow clone we check if ${cset} is in the list provided by git + # ls-remote. If not we fall back on a full clone. + # + # Messages for the type of clone used are provided to ease debugging in + # case of problems + git_done=0 + if [ -n "$(_git ls-remote "'${uri}'" "'${cset}'" 2>&1)" ]; then + printf "Doing shallow clone\n" + if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${uri}'" "'${git_cache}'"; then + git_done=1 + else + printf "Shallow clone failed, falling back to doing a full clone\n" + fi + fi + if [ ${git_done} -eq 0 ]; then + printf "Doing full clone\n" + _git clone ${verbose} "${@}" "'${uri}'" "'${git_cache}'" fi -fi -if [ ${git_done} -eq 0 ]; then - printf "Doing full clone\n" - _git clone ${verbose} "${@}" "'${uri}'" "'${basename}'" fi -pushd "${basename}" >/dev/null +pushd "${git_cache}" >/dev/null # Try to get the special refs exposed by some forges (pull-requests for # github, changes for gerrit...). There is no easy way to know whether @@ -86,20 +92,25 @@ if [ ${recurse} -eq 1 ]; then _git submodule update --init --recursive fi -# We do not want the .git dir; we keep other .git files, in case they -# are the only files in their directory. +# Generate the archive, sort with the C locale so that it is reproducible +# We do not want the .git dir; we keep other .git +# files, in case they are the only files in their directory. # The .git dir would generate non reproducible tarballs as it depends on # the state of the remote server. It also would generate large tarballs # (gigabytes for some linux trees) when a full clone took place. -rm -rf .git +find . -not -type d \ + -and -not -path "./.git/*" >"${BR2_DL_DIR}/${basename}.list" +LC_ALL=C sort <"${BR2_DL_DIR}/${basename}.list" >"${BR2_DL_DIR}/${basename}.list.sorted" -popd >/dev/null - -# Generate the archive, sort with the C locale so that it is reproducible -find "${basename}" -not -type d >"${basename}.list" -LC_ALL=C sort <"${basename}.list" >"${basename}.list.sorted" # Create GNU-format tarballs, since that's the format of the tarballs on # sources.buildroot.org and used in the *.hash files -tar cf - --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \ - -T "${basename}.list.sorted" >"${output}.tar" +tar cf - --transform="s/^\./${basename}/" \ + --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \ + -T "${BR2_DL_DIR}/${basename}.list.sorted" >"${output}.tar" gzip -n <"${output}.tar" >"${output}" +tar tf "${output}" + +rm -f "${BR2_DL_DIR}/${basename}.list" +rm -f "${BR2_DL_DIR}/${basename}.list.sorted" + +popd >/dev/null