From patchwork Sat Mar 31 14:24:07 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Hadjinlian X-Patchwork-Id: 893793 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="G9WzYl+i"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40D12q4bFXz9rxs for ; Sun, 1 Apr 2018 01:24:35 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 106A989394; Sat, 31 Mar 2018 14:24:34 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4wzEME4BPdPy; Sat, 31 Mar 2018 14:24:31 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id A2AC689395; Sat, 31 Mar 2018 14:24:31 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by ash.osuosl.org (Postfix) with ESMTP id A8D9E1C16D3 for ; Sat, 31 Mar 2018 14:24:22 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id A63ED25F2A for ; Sat, 31 Mar 2018 14:24:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JPm83aYe5Ip0 for ; Sat, 31 Mar 2018 14:24:21 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm0-f68.google.com (mail-wm0-f68.google.com [74.125.82.68]) by silver.osuosl.org (Postfix) with ESMTPS id 122DD25D87 for ; Sat, 31 Mar 2018 14:24:21 +0000 (UTC) Received: by mail-wm0-f68.google.com with SMTP id x4so20470447wmh.5 for ; Sat, 31 Mar 2018 07:24:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=m8kLcTP5gJRqueCIWkgFFUfmpT67G5ke1601c+ZvDi8=; b=G9WzYl+ioTRjrlio1WjMJBK/X/orM0J+3/TobUmnO2UGUCpAK8eN4ajFsWGMRDrCrj icprI+o6cSMxI0zHFUk02MfobawPZ4IwpgM2lNqAWJQwhQK7J+2QJAwph4zlsk5AoUMH 9faehHhz8azeOUn/J44U6pqzG+z3HhNbsn440s3xrKTOCuO31L/5iZkABzbM+336okRi HPriQ8ybzDKoMxMxRok6Kkq5b+33GVKoqBkJA/3e8RBeZ93HZg1uemiJroP87U4ZUwC0 GmVrf1FKVNbUgNLVQul6yoDFQFXut1M5ORDKk30tlMcwRLzg5DJaW8NThFaDW8C5upnl yDNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=m8kLcTP5gJRqueCIWkgFFUfmpT67G5ke1601c+ZvDi8=; b=BSVF+nTrEVOVhFzm0VshxdJMYUYLew3QgTtfKCK9Uujl9Td34NziSWdNWAlXyPvbbP Ux2p2Gype0KEy0ZjV6vwjNzNR3JgXjj2EI+2Fu4FzvLXmDonvIZLSzyPiY0oE7jmVWS5 +fJTcKQPQH6pr2ELP/WFCsDAO3lB4xhaelUyzGvCi4DuRl7k0PIqvd0ZmTRak13oSNZV jeZk8jjwXqH4VH1ANGScca2rmGA9sHr5laQnWVmHa9CXG/K7LVquI/dL1uM/SxEuxpp3 jKwy6CBfoGn1AM6HrhvSR7p9Uwb/lKegUBt3msRtaPra8gDj3Tr0Aq6guUitaZNVlQgI 5R0w== X-Gm-Message-State: AElRT7EdwOT3WWVsXMFlOx9meK60H9UkxF9Xgva6M/aroKxjkaqODV8M 6g+F9kIWXN76ovqOfzuCI1WHCekW X-Google-Smtp-Source: AIpwx496594aaWP0dxbmpOzmETKV10C+mAKWYezapoXoS2PjVKSNi8/rulHQpjrtptWmANzhZm9yMA== X-Received: by 10.28.137.68 with SMTP id l65mr4993680wmd.57.1522506259175; Sat, 31 Mar 2018 07:24:19 -0700 (PDT) Received: from cerise.online.net ([195.154.228.195]) by smtp.gmail.com with ESMTPSA id o88sm11394409wrb.44.2018.03.31.07.24.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 31 Mar 2018 07:24:18 -0700 (PDT) From: Maxime Hadjinlian To: buildroot@buildroot.org Date: Sat, 31 Mar 2018 16:24:07 +0200 Message-Id: <20180331142407.9522-13-maxime.hadjinlian@gmail.com> X-Mailer: git-send-email 2.16.2 In-Reply-To: <20180331142407.9522-1-maxime.hadjinlian@gmail.com> References: <20180331142407.9522-1-maxime.hadjinlian@gmail.com> Subject: [Buildroot] [v3 13/13] download: git: introduce cache feature X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Yann E . Morin" , Maxime Hadjinlian MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Now we keep the git clone that we download and generates our tarball from there. The main goal here is that if you change the version of a package (say Linux), instead of cloning all over again, you will simply 'git fetch' from the repo the missing objects, then generates the tarball again. This should speed the 'source' part of the build significantly. The drawback is that the DL_DIR will grow much larger; but time is more important than disk space nowadays. Signed-off-by: Maxime Hadjinlian --- v1 -> v2: - Fix bad regex in the 'transform' option of tar (found by Peter Seiderer) v2 -> v3: - Change git fetch origin to use the uri of the package instead of the name of the default remote 'origin' (Thomas Petazzoni) --- support/download/git | 70 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 40 insertions(+), 30 deletions(-) diff --git a/support/download/git b/support/download/git index 58a2c6ad9d..301f7e792a 100755 --- a/support/download/git +++ b/support/download/git @@ -39,28 +39,34 @@ _git() { eval ${GIT} "${@}" } -# Try a shallow clone, since it is faster than a full clone - but that only -# works if the version is a ref (tag or branch). Before trying to do a shallow -# clone we check if ${cset} is in the list provided by git ls-remote. If not -# we fall back on a full clone. -# -# Messages for the type of clone used are provided to ease debugging in case of -# problems -git_done=0 -if [ -n "$(_git ls-remote "'${uri}'" "'${cset}'" 2>&1)" ]; then - printf "Doing shallow clone\n" - if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${uri}'" "'${basename}'"; then - git_done=1 - else - printf "Shallow clone failed, falling back to doing a full clone\n" +# We want to check if a cache of the git clone of this repo already exists. +git_cache="${BR2_DL_DIR}/${basename%%-*}/git" + +# If the cache directory already exists, don't try to clone. +if [ ! -d "${git_cache}" ]; then + # Try a shallow clone, since it is faster than a full clone - but that + # only works if the versionis a ref (tag or branch). Before trying to do a + # shallow clone we check if ${cset} is in the list provided by git + # ls-remote. If not we fall back on a full clone. + # + # Messages for the type of clone used are provided to ease debugging in + # case of problems + git_done=0 + if [ -n "$(_git ls-remote "'${uri}'" "'${cset}'" 2>&1)" ]; then + printf "Doing shallow clone\n" + if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${uri}'" "'${git_cache}'"; then + git_done=1 + else + printf "Shallow clone failed, falling back to doing a full clone\n" + fi + fi + if [ ${git_done} -eq 0 ]; then + printf "Doing full clone\n" + _git clone ${verbose} "${@}" "'${uri}'" "'${git_cache}'" fi -fi -if [ ${git_done} -eq 0 ]; then - printf "Doing full clone\n" - _git clone ${verbose} "${@}" "'${uri}'" "'${basename}'" fi -pushd "${basename}" >/dev/null +pushd "${git_cache}" >/dev/null # Try to get the special refs exposed by some forges (pull-requests for # github, changes for gerrit...). There is no easy way to know whether @@ -69,7 +75,7 @@ pushd "${basename}" >/dev/null # below, if there is an issue anyway. Since most of the cset we're gonna # have to clone are not such special refs, consign the output to oblivion # so as not to alarm unsuspecting users, but still trace it as a warning. -if ! _git fetch origin "'${cset}:${cset}'" >/dev/null 2>&1; then +if ! _git fetch "'${uri}'" "'${cset}:${cset}'" >/dev/null 2>&1; then printf "Could not fetch special ref '%s'; assuming it is not special.\n" "${cset}" fi @@ -86,20 +92,24 @@ if [ ${recurse} -eq 1 ]; then _git submodule update --init --recursive fi -# We do not want the .git dir; we keep other .git files, in case they -# are the only files in their directory. +# Generate the archive, sort with the C locale so that it is reproducible +# We do not want the .git dir; we keep other .git +# files, in case they are the only files in their directory. # The .git dir would generate non reproducible tarballs as it depends on # the state of the remote server. It also would generate large tarballs # (gigabytes for some linux trees) when a full clone took place. -rm -rf .git +find . -not -type d \ + -and -not -path "./.git/*" >"${BR2_DL_DIR}/${basename}.list" +LC_ALL=C sort <"${BR2_DL_DIR}/${basename}.list" >"${BR2_DL_DIR}/${basename}.list.sorted" -popd >/dev/null - -# Generate the archive, sort with the C locale so that it is reproducible -find "${basename}" -not -type d >"${basename}.list" -LC_ALL=C sort <"${basename}.list" >"${basename}.list.sorted" # Create GNU-format tarballs, since that's the format of the tarballs on # sources.buildroot.org and used in the *.hash files -tar cf - --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \ - -T "${basename}.list.sorted" >"${output}.tar" +tar cf - --transform="s/^\.$/${basename}/" \ + --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \ + -T "${BR2_DL_DIR}/${basename}.list.sorted" >"${output}.tar" gzip -6 -n <"${output}.tar" >"${output}" + +rm -f "${BR2_DL_DIR}/${basename}.list" +rm -f "${BR2_DL_DIR}/${basename}.list.sorted" + +popd >/dev/null