From patchwork Mon Apr 2 14:57:55 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Hadjinlian X-Patchwork-Id: 894275 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="l/KtHX5k"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40FFhh1W8lz9s28 for ; Tue, 3 Apr 2018 00:58:11 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 728AD88E15; Mon, 2 Apr 2018 14:58:10 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id cmpHm-30OrdG; Mon, 2 Apr 2018 14:58:09 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 1A22A88EC3; Mon, 2 Apr 2018 14:58:09 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 835F61C2314 for ; Mon, 2 Apr 2018 14:58:08 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 8037F88E4C for ; Mon, 2 Apr 2018 14:58:08 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LPcjdXgmNDBy for ; Mon, 2 Apr 2018 14:58:08 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr0-f193.google.com (mail-wr0-f193.google.com [209.85.128.193]) by whitealder.osuosl.org (Postfix) with ESMTPS id B05F088E15 for ; Mon, 2 Apr 2018 14:58:07 +0000 (UTC) Received: by mail-wr0-f193.google.com with SMTP id s18so14144623wrg.9 for ; Mon, 02 Apr 2018 07:58:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=E9Xhvp2bh0kk/wB7PMLKD99rK54Tanip0kadiUrgI0A=; b=l/KtHX5kS99bc7bCjruuqMnMZkYtFgs2IzQ6XB58mztgP8pOPmiUe9xuqdx/Ij2wTM CtV2Y0eHIVFqCgOS6iYTmqCsYhvRyEGj8Qlu/59Ju7IODNWLh8W142eJa01FUENeRz42 ivvehGZQbTuNA4bmWbTjffmrvBrqNeQtdwl/4LKZNe+wvUsqBZKEAJx9TXmMAsetiWuw 4W8n7SJqynz7LAhioHIY9Qpr+YAeeXHXSp0FXK7l5MaRI6kTo3A17gM825xInIBeHJEl cHfzZhT3tBdVEv+2NmCXkqmjVSD+P30+Ebkn4fmdKac4q5EZm8Who44lz8tgaPtiY8r6 g3yg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=E9Xhvp2bh0kk/wB7PMLKD99rK54Tanip0kadiUrgI0A=; b=RA+eMrP+FtKbOi8pTgS1Aeh1ZNFThNOW2DNIKdz1L3Rz+5fpiJG+Wi+KoVKFXV3/ol nlbaYjr/mtlbwSoAVisY575miLZCvrSOBPC6Fa9EJkBS/U/Dh+pWZER4SlN0PZJ8bSiw N2UOqgR23ghbWGm7g+Un6tIml9Kh84LonlJczt/K8VJWmDsSjDeKnjiS42aeUR7A0SsR MrkvktwTUi2pNNDr+ovY3XxTRa1bi+ok83ydVVrJUk60E4Js/jT7o3zUsUhmGcU0rzMs KyqIQl06Gp1k2udI/k2pLNEtcYG+ZSEYWf89vfbjbJMFHQytDzJhxAGN8HMo01BXZvrQ 0Big== X-Gm-Message-State: AElRT7Fed6wy6jp7Ht5y9mHWr2BaKHPXi+GqvMLBcQZ44TO6POMLg75i 6vNMVRYOJppzGEW3vboFxpwz0vfK X-Google-Smtp-Source: AIpwx48PbNAmD800zWJz5LU4QfMlSiByTO3CJ2o9sCs43uezAW5DXz2cX+MFWtoBxRHJDy0BMFh0pw== X-Received: by 10.223.144.227 with SMTP id i90mr6865205wri.100.1522681085657; Mon, 02 Apr 2018 07:58:05 -0700 (PDT) Received: from cerise.online.net ([195.154.228.195]) by smtp.gmail.com with ESMTPSA id s15sm756001wrg.28.2018.04.02.07.58.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 Apr 2018 07:58:04 -0700 (PDT) From: Maxime Hadjinlian To: buildroot@buildroot.org Date: Mon, 2 Apr 2018 16:57:55 +0200 Message-Id: <20180402145802.17611-1-maxime.hadjinlian@gmail.com> X-Mailer: git-send-email 2.16.2 Subject: [Buildroot] [PATCH 1/8] pkg-generic: add a subdirectory to the DL_DIR X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Maxime Hadjinlian MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" With all the previous changes, we are now ready to add a subdirectory to the DL_DIR. The structure will now be DL_DIR/PKG_NAME/{FILE1,FILE2} This is needed for multiple reasons: - Avoid patches with name like SHA1.patch laying flat in DL_DIR, which makes it hard to know to which packages they apply - Avoid possible collisions if two releases have the same name (e.g: v01.tar) - Allow the possibility to handle a git cache per package in the newly created subdirectory. Signed-off-by: Maxime Hadjinlian Reviewed-by: "Yann E. MORIN" --- package/pkg-generic.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index 01d61ac0ad..20ee73d438 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -432,7 +432,7 @@ endif $(2)_BASENAME = $$(if $$($(2)_VERSION),$(1)-$$($(2)_VERSION),$(1)) $(2)_BASENAME_RAW = $$(if $$($(2)_VERSION),$$($(2)_RAWNAME)-$$($(2)_VERSION),$$($(2)_RAWNAME)) -$(2)_DL_DIR = $$(DL_DIR) +$(2)_DL_DIR = $$(DL_DIR)/$$($(2)_RAWNAME) $(2)_DIR = $$(BUILD_DIR)/$$($(2)_BASENAME) ifndef $(2)_SUBDIR From patchwork Mon Apr 2 14:57:57 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Hadjinlian X-Patchwork-Id: 894279 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="sjcmAg+a"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40FFhs1LKqz9s2B for ; Tue, 3 Apr 2018 00:58:20 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id CBF3A8915D; Mon, 2 Apr 2018 14:58:16 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bIAXAB7f7YPB; Mon, 2 Apr 2018 14:58:12 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id AC1A388F52; Mon, 2 Apr 2018 14:58:12 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 704251C2314 for ; Mon, 2 Apr 2018 14:58:10 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 6DA3284CAB for ; Mon, 2 Apr 2018 14:58:10 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hRx-Oruwch4I for ; Mon, 2 Apr 2018 14:58:09 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr0-f193.google.com (mail-wr0-f193.google.com [209.85.128.193]) by fraxinus.osuosl.org (Postfix) with ESMTPS id E0D9784C36 for ; Mon, 2 Apr 2018 14:58:08 +0000 (UTC) Received: by mail-wr0-f193.google.com with SMTP id p53so14143786wrc.10 for ; Mon, 02 Apr 2018 07:58:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=OOAC1ZZzBHJEVwuZ7T/jz0fY7r0/Hbf1hhyHIaTG2Yc=; b=sjcmAg+aP+JuiUosLSLYfb1R1KH1rq9IoTdiYDMESWbWNzjOlwoJkgV13tX8C6cpFo EW4IoKjdlrGZ0t1WJ1/yD9pVpYlaz1lJaWO/wwIDoQ3Xs896iWJQpxxQmfMsCCGvMhME B+hqFRbITLSMiXMsM8w8YsLzC2OX2rUFKE2L6D16IkC8pwKgVGUxa2Jr1ptmkQbLMIef 8NRAKlLzl6ltYAPksUy87PPLWKjHA8bFNscf+e1/hv0S2I0PM3Iwr5Gf/ut3hW5JWN4P l4uDuwySw3HMUwWpycG3hsURvf9ArIIk02z/darZfuvTpoIDOU1z09KPIAeoeWoVFhLn 6AJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=OOAC1ZZzBHJEVwuZ7T/jz0fY7r0/Hbf1hhyHIaTG2Yc=; b=Z4yL8WvQyohn4c3+6nffGvpyBfLZJ6pjkwu30/18X7+7eoh9nBWZSZV0s0UKUgNYDP U4ebS+9cSOFG1fvTay6SpaNRvmDtZ0lmGctCcVM1P/yuy21jHvh4R7dhYCMg4umHnNWv S1dEtyiZO5jQGbSuMvz8GLp0mMhFxIGEnkVLlI8ROtekjLAFu73Ih/MiLgBb7cjFtvj2 vYwyxvjLRprRc+Y4WF/Egjd4SoAJ+6JNO5Nd3RCwNIGWJC9SSU/flj/pdlhqZHXWdrUg WbzZWjc789wPqxXyDGQsKxJwdrk8lAz3tsmP+GQgXVaE88tFnA+6J4Oky1jhKSZ7iGQ3 CLgQ== X-Gm-Message-State: AElRT7GARccmZMvMow+l7ipYwKIV7H33KELhRCPbfUXyijYZ6cTZ0r8a eat6aHEfHV+DLzjs+cQ7KPF0/DzN X-Google-Smtp-Source: AIpwx48chqeVfwL9wbv358mxthipbNOzo4oh9z09xddxzxtsOmOJ2IDqD+Uc1whJVbVxLzNdD8jHYA== X-Received: by 10.223.209.198 with SMTP id m6mr7269781wri.267.1522681087042; Mon, 02 Apr 2018 07:58:07 -0700 (PDT) Received: from cerise.online.net ([195.154.228.195]) by smtp.gmail.com with ESMTPSA id s15sm756001wrg.28.2018.04.02.07.58.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 Apr 2018 07:58:06 -0700 (PDT) From: Maxime Hadjinlian To: buildroot@buildroot.org Date: Mon, 2 Apr 2018 16:57:57 +0200 Message-Id: <20180402145802.17611-3-maxime.hadjinlian@gmail.com> X-Mailer: git-send-email 2.16.2 In-Reply-To: <20180402145802.17611-1-maxime.hadjinlian@gmail.com> References: <20180402145802.17611-1-maxime.hadjinlian@gmail.com> Subject: [Buildroot] [PATCH 3/8] pkg-download: support new subdir for mirrors X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Maxime Hadjinlian MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Since we introduced subdirectory to the DL_DIR, we need to support them in the PRIMARY and BACKUP mirror as they evolve to the new tree structure. We check first the new URI (with the subdir), and in case of failure, we go check without. By checking both URIs, we ensure that old mirror are usable. Also, add a missing qstrip call for BR2_BACKUP_SITE. Signed-off-by: Maxime Hadjinlian Reviewed-by: "Yann E. MORIN" --- v4 -> v5: - Use new DL_SUBDIR - Add missing qstrip --- package/pkg-download.mk | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/package/pkg-download.mk b/package/pkg-download.mk index 3f35661a16..7ca6fcc757 100644 --- a/package/pkg-download.mk +++ b/package/pkg-download.mk @@ -75,6 +75,7 @@ export BR_NO_CHECK_HASH_FOR = ifneq ($(call qstrip,$(BR2_PRIMARY_SITE)),) DOWNLOAD_URIS += \ + -u $(call getschemeplusuri,$(call qstrip,$(BR2_PRIMARY_SITE)/$($(PKG)_DL_SUBDIR)),urlencode) \ -u $(call getschemeplusuri,$(call qstrip,$(BR2_PRIMARY_SITE)),urlencode) endif @@ -83,7 +84,8 @@ DOWNLOAD_URIS += \ -u $($(PKG)_SITE_METHOD)+$(dir $(call qstrip,$(1))) ifneq ($(call qstrip,$(BR2_BACKUP_SITE)),) DOWNLOAD_URIS += \ - -u $(call getschemeplusuri,$(BR2_BACKUP_SITE),urlencode) + -u $(call getschemeplusuri,$(call qstrip,$(BR2_BACKUP_SITE)/$($(PKG)_DL_SUBDIR)),urlencode) \ + -u $(call getschemeplusuri,$(call qstrip,$(BR2_BACKUP_SITE)),urlencode) endif endif From patchwork Mon Apr 2 14:57:58 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Hadjinlian X-Patchwork-Id: 894280 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="esZK6E85"; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40FFhw2DKdz9s3D for ; Tue, 3 Apr 2018 00:58:24 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 72B6A89104; Mon, 2 Apr 2018 14:58:20 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id b2g7xJFFR25d; Mon, 2 Apr 2018 14:58:15 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 092EF8900C; Mon, 2 Apr 2018 14:58:15 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 1AC0E1C2314 for ; Mon, 2 Apr 2018 14:58:11 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 17D918A294 for ; Mon, 2 Apr 2018 14:58:11 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oNO+pOod3ovU for ; Mon, 2 Apr 2018 14:58:10 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr0-f194.google.com (mail-wr0-f194.google.com [209.85.128.194]) by hemlock.osuosl.org (Postfix) with ESMTPS id 407C08A28C for ; Mon, 2 Apr 2018 14:58:10 +0000 (UTC) Received: by mail-wr0-f194.google.com with SMTP id u46so14158608wrc.11 for ; Mon, 02 Apr 2018 07:58:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=BgArKtRCCxgm6wvLL6qCrXB1xGzCS4ie7/JDCiCPFHo=; b=esZK6E85FBaFcOnNR2/h6k6jvIW2Yr0XNApLO20/dAMU10XysIWfcf3oEVuEWwVd8F udV5CEYj3tDfYF13lrdQpZCEZs9qzNSqqaoObnXeP0LaqZSFbyQa0eCRqnWEgmUSotox TtLwKXOg2cHMpSux5sV1Lz4M/Zmx8pjYYSAN4OwJuLoQ8u1eIp0ONCYnP2tG4DlipoKB jasOtwEg4nDmWrZWQSoPXGLg+u8IkMcI1HokPj/tJ2X+zkdruXfqvZUv6GgqzMFxKDRo lLTcjILfos7X06MZFAdTfIhrRalc7LzRnfwx0ZAarEafw63SPLOfaDsd/Y0cGwrca0nk ZHrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=BgArKtRCCxgm6wvLL6qCrXB1xGzCS4ie7/JDCiCPFHo=; b=q0QXAQHKhPbH6RPetiLw68pcOQq4OGAUz2sgP7ZB2GoURyvv/228V++yQs7lP0Fo8n nf0rHfDba2Berz705c2mde5Q2A4vzNMdbbltqx2DU5ipPgyiG1ZJFS1/lCPrNd4jlvz6 xrXNMjex6bL2WX1snA0siiHFAlwEdsMEN2ZWNtR0F4n+gcFG/3N+L/tukfxHXx36IZLO 5Kpikf6jz1xeb/fX6nX2IUqwu4fodNvVOuA1oOO8bijY5PtwsArPqoXQ+FIpaYksc4ck wozFah5lj2USF7WpPx0h2HPStOi4+tVdgN0cTSwM+8AbDvQkycql6dcZJUNDBM9hr58S tZoA== X-Gm-Message-State: AElRT7EraskFfixIV5nriCyfyPy/9xl6HxVOgG4gQZBORZrs4XjC42A7 fQMBPn/GNA9465ceV9O/Vy2/8aqb X-Google-Smtp-Source: AIpwx48ik2f/O1HamuHed/0w3cCI4mM6U5S5R3/DyBN4actX5pO6LokpUTfNrnqEKvL2MByIe+M2wg== X-Received: by 10.223.135.237 with SMTP id c42mr7525114wrc.146.1522681088434; Mon, 02 Apr 2018 07:58:08 -0700 (PDT) Received: from cerise.online.net ([195.154.228.195]) by smtp.gmail.com with ESMTPSA id s15sm756001wrg.28.2018.04.02.07.58.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 Apr 2018 07:58:07 -0700 (PDT) From: Maxime Hadjinlian To: buildroot@buildroot.org Date: Mon, 2 Apr 2018 16:57:58 +0200 Message-Id: <20180402145802.17611-4-maxime.hadjinlian@gmail.com> X-Mailer: git-send-email 2.16.2 In-Reply-To: <20180402145802.17611-1-maxime.hadjinlian@gmail.com> References: <20180402145802.17611-1-maxime.hadjinlian@gmail.com> Subject: [Buildroot] [PATCH 4/8] package: share downloaded files for big packages X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas Petazzoni , "Yann E. MORIN" , Maxime Hadjinlian MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" From: "Yann E. MORIN" Theses packages are given as an example of the use of the *_DL_SUBDIR feature. There maybe other packages that would benefit from that feature, they would need to be added on a case by case basis. Signed-off-by: "Yann E. MORIN" Signed-off-by: Maxime Hadjinlian Cc: Maxime Hadjinlian Cc: Thomas Petazzoni --- v4 -> v5: - Change SAME_SOURCE_AS into DL_SUBDIR --- package/gcc/gcc-final/gcc-final.mk | 2 ++ package/gcc/gcc-initial/gcc-initial.mk | 5 +++++ package/linux-headers/linux-headers.mk | 3 +++ package/mesa3d-headers/mesa3d-headers.mk | 1 + 4 files changed, 11 insertions(+) diff --git a/package/gcc/gcc-final/gcc-final.mk b/package/gcc/gcc-final/gcc-final.mk index 213f3d7b66..b7b27aec13 100644 --- a/package/gcc/gcc-final/gcc-final.mk +++ b/package/gcc/gcc-final/gcc-final.mk @@ -8,6 +8,8 @@ GCC_FINAL_VERSION = $(GCC_VERSION) GCC_FINAL_SITE = $(GCC_SITE) GCC_FINAL_SOURCE = $(GCC_SOURCE) +HOST_GCC_FINAL_DL_SUBDIR = gcc + HOST_GCC_FINAL_DEPENDENCIES = \ $(HOST_GCC_COMMON_DEPENDENCIES) \ $(BR_LIBC) diff --git a/package/gcc/gcc-initial/gcc-initial.mk b/package/gcc/gcc-initial/gcc-initial.mk index c476b2faeb..9b20eb18f9 100644 --- a/package/gcc/gcc-initial/gcc-initial.mk +++ b/package/gcc/gcc-initial/gcc-initial.mk @@ -8,6 +8,11 @@ GCC_INITIAL_VERSION = $(GCC_VERSION) GCC_INITIAL_SITE = $(GCC_SITE) GCC_INITIAL_SOURCE = $(GCC_SOURCE) +# We do not have a 'gcc' package per-se; we only have two incarnations, +# gcc-initial and gcc-final. gcc-initial is just am internal step that +# users should not care about, while gcc-final is the one they shall see. +HOST_GCC_INITIAL_DL_SUBDIR = gcc + HOST_GCC_INITIAL_DEPENDENCIES = $(HOST_GCC_COMMON_DEPENDENCIES) HOST_GCC_INITIAL_EXCLUDES = $(HOST_GCC_EXCLUDES) diff --git a/package/linux-headers/linux-headers.mk b/package/linux-headers/linux-headers.mk index f1e3790608..954c6b7978 100644 --- a/package/linux-headers/linux-headers.mk +++ b/package/linux-headers/linux-headers.mk @@ -82,6 +82,9 @@ endif endif # ! BR2_KERNEL_HEADERS_AS_KERNEL +# linux-headers really is the same as the linux package +LINUX_HEADERS_DL_SUBDIR = linux + LINUX_HEADERS_LICENSE = GPL-2.0 LINUX_HEADERS_LICENSE_FILES = COPYING diff --git a/package/mesa3d-headers/mesa3d-headers.mk b/package/mesa3d-headers/mesa3d-headers.mk index b48d965d8a..19d1c6fdf1 100644 --- a/package/mesa3d-headers/mesa3d-headers.mk +++ b/package/mesa3d-headers/mesa3d-headers.mk @@ -15,6 +15,7 @@ endif MESA3D_HEADERS_VERSION = 18.0.0 MESA3D_HEADERS_SOURCE = mesa-$(MESA3D_HEADERS_VERSION).tar.xz MESA3D_HEADERS_SITE = https://mesa.freedesktop.org/archive +MESA3D_HEADERS_DL_SUBDIR = mesa3d MESA3D_HEADERS_LICENSE = MIT, SGI, Khronos MESA3D_HEADERS_LICENSE_FILES = docs/license.html From patchwork Mon Apr 2 14:57:59 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Hadjinlian X-Patchwork-Id: 894278 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="prAF87O2"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40FFhs3VbTz9s31 for ; Tue, 3 Apr 2018 00:58:20 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id C803E228ED; Mon, 2 Apr 2018 14:58:17 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id nlrhrqNikuub; Mon, 2 Apr 2018 14:58:17 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id EAE31227E9; Mon, 2 Apr 2018 14:58:16 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 3851F1C2314 for ; Mon, 2 Apr 2018 14:58:12 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 3305C88EDE for ; Mon, 2 Apr 2018 14:58:12 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id oArLBFpNhHwc for ; Mon, 2 Apr 2018 14:58:11 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr0-f195.google.com (mail-wr0-f195.google.com [209.85.128.195]) by whitealder.osuosl.org (Postfix) with ESMTPS id 5295E89008 for ; Mon, 2 Apr 2018 14:58:11 +0000 (UTC) Received: by mail-wr0-f195.google.com with SMTP id u46so14158686wrc.11 for ; Mon, 02 Apr 2018 07:58:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=iFGU6M36h4nNrksVYYkf/tqZsSNV7nx4mtkXPMhp/tM=; b=prAF87O2sph+nfkv9yuYWkaIQC568MIGU7VEAMUhGSZSQJLyHOIwTf/qQEzHImYgHi SkjfAz5mQNt9/eNTcXCVa9D4gAn0+PvooYrJb9bdKl3S7Ke4zroT+ZKn5OLPuXyuCMv5 aoEyyHLV4lj/U4Fgc1YYLc9ymU2pjSjUpIZzaFBf8Ea8G0KJl1ag8omRNaXo8fI6BMwr 2mY7FwGQFgTZ30xsW4nXfR8Ks1YYyrQpmfsUwoVeTZMfzkmgaWGpe4jKJDAQaJaS1gp/ bPw4v+ac1ctn44/zneXTeFB4Bl7CXLhG344wGpwCaQZIKuDukYkaujmPCtjkSUN9Jjrk YNEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=iFGU6M36h4nNrksVYYkf/tqZsSNV7nx4mtkXPMhp/tM=; b=CRbyD1rxsBDRvqUN3Wrs5jxjUTaAIHamG/Z/9Cbe1WCbYND0LIUu/xNxVie5JAjCs9 CzCISCj7Tayr9tTka6pApFxuDuCrEkN3QZ1mdOiAV9IRIYxXbIyEEzYOT3srMe0QArg8 JW9fV85mq1OO3Bsxq9IQOetfMA/BobT3CIgsw7ZaP7s6ZkIUJfgINmHaU4dT44V2LiR+ dzQIb3emWhg19YLZoXuKK8xpT2gI3/AQ3gqc7PlURftkUIG+lpSHF4zjCm6ULP3IYlUo GO8xEcR7YPiAlNXJhzrGXGXNC0mFKG7Of6D0g8xoehEHufWFRHvlGS4ufAmBWFTzh9Ob VDZg== X-Gm-Message-State: AElRT7GTkETUyQMXU17cIl7nxZB0dbptEdlyIxn0BC8WJR7+6/CZ2as1 LjlxgW54eOAxnlFBsg8eIUbRcGkz X-Google-Smtp-Source: AIpwx4+QuK+fZHF9OHD1zcJCiDPR9bqyVW5GKUXbG8W3t4ikutz5OHljkPdKvCZEvV/9iO6Qwjw8Lw== X-Received: by 10.223.175.201 with SMTP id y9mr6790765wrd.107.1522681089425; Mon, 02 Apr 2018 07:58:09 -0700 (PDT) Received: from cerise.online.net ([195.154.228.195]) by smtp.gmail.com with ESMTPSA id s15sm756001wrg.28.2018.04.02.07.58.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 Apr 2018 07:58:08 -0700 (PDT) From: Maxime Hadjinlian To: buildroot@buildroot.org Date: Mon, 2 Apr 2018 16:57:59 +0200 Message-Id: <20180402145802.17611-5-maxime.hadjinlian@gmail.com> X-Mailer: git-send-email 2.16.2 In-Reply-To: <20180402145802.17611-1-maxime.hadjinlian@gmail.com> References: <20180402145802.17611-1-maxime.hadjinlian@gmail.com> Subject: [Buildroot] [PATCH 5/8] help/manual: update help about the new $(LIBFOO_DL_DIR) X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Maxime Hadjinlian MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Signed-off-by: Maxime Hadjinlian Reviewed-by: "Yann E. MORIN" --- Config.in | 3 +++ docs/manual/adding-packages-generic.txt | 6 +++--- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/Config.in b/Config.in index 71b9f60e5b..c2a1925088 100644 --- a/Config.in +++ b/Config.in @@ -200,6 +200,9 @@ config BR2_DL_DIR If the Linux shell environment has defined the BR2_DL_DIR environment variable, then this overrides this configuration item. + The directory is organized with a subdirectory for each package. + Each package has its own $(LIBFOO_DL_DIR) variable that can be used + to find the correct path. The default is $(TOPDIR)/dl diff --git a/docs/manual/adding-packages-generic.txt b/docs/manual/adding-packages-generic.txt index 9d1428ad40..62906d92bb 100644 --- a/docs/manual/adding-packages-generic.txt +++ b/docs/manual/adding-packages-generic.txt @@ -263,7 +263,7 @@ information is (assuming the package name is +libfoo+) : the file using this URL. Otherwise, Buildroot will assume the file to be downloaded is located at +LIBFOO_SITE+. Buildroot will not do anything with those additional files, except download them: it will - be up to the package recipe to use them from +$(DL_DIR)+. + be up to the package recipe to use them from +$(LIBFOO_DL_DIR)+. * +LIBFOO_SITE_METHOD+ determines the method used to fetch or copy the package source code. In many cases, Buildroot guesses the method @@ -554,8 +554,8 @@ In the action definitions, you can use the following variables: * +$(@D)+, which contains the directory in which the package source code has been uncompressed. -* +$(DL_DIR)+ contains the path to the directory where all the downloads made - by Buildroot are stored. +* +$(LIBFOO_DL_DIR)+ contains the path to the directory where all the downloads + made by Buildroot for +libfoo+ are stored in. * +$(TARGET_CC)+, +$(TARGET_LD)+, etc. to get the target cross-compilation utilities From patchwork Mon Apr 2 14:58:00 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Hadjinlian X-Patchwork-Id: 894282 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="GnH866bp"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40FFhz1z1tz9s0w for ; Tue, 3 Apr 2018 00:58:27 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 10A2925EA4; Mon, 2 Apr 2018 14:58:23 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Af33U2+WBBAQ; Mon, 2 Apr 2018 14:58:18 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 9C88F22898; Mon, 2 Apr 2018 14:58:17 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 79C2D1C2314 for ; Mon, 2 Apr 2018 14:58:13 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 7004D89008 for ; Mon, 2 Apr 2018 14:58:13 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Yq3ow67-BdzR for ; Mon, 2 Apr 2018 14:58:12 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr0-f182.google.com (mail-wr0-f182.google.com [209.85.128.182]) by whitealder.osuosl.org (Postfix) with ESMTPS id 4240588EDE for ; Mon, 2 Apr 2018 14:58:12 +0000 (UTC) Received: by mail-wr0-f182.google.com with SMTP id d1so14146866wrj.13 for ; Mon, 02 Apr 2018 07:58:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=e9eAyqv7MmXiSc2V7hRCaD1/QMYtjZwCxvwu76ShRhQ=; b=GnH866bpT9lhYOMEjQbCK1IcUOpLn6vh3c6j0F+fyGa88b5ZOww1p1smlHk1UuHTHS d1q291UAQnzf0wrrUKC3djkPzVWAvAsv+GDusYyzGHtE2CpPN6EdRyrLrVfK/mgEH/XW J4IgSqBT3OBwxCplsB5bIPTjMnzmTskRf3eMKv3f+SMVurnetuzX+1Dsi/1pGbOesuPP 0Rmvqcp9+5fiIkC1+iDFGoAWFu7+wPurD79K24smo/ye1jaAEurXdWH24XzkA/no0DQ+ XwGyewfg33f6p5GDv8W238Ik5HB4vVslJ7aUpc3q9q0S4w28m6wQXG/Eb2OGbD3EvrT3 D61A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=e9eAyqv7MmXiSc2V7hRCaD1/QMYtjZwCxvwu76ShRhQ=; b=nx65443Xs1bMhKQ0f7XKyrYSHCnMt6R2WSafFNJpExU4AWC5YZ4lNN0syAsZcBaiqJ 9qaZhYaHbF+UPa3OdZ/Y0jM2ayJVUTt9nPMlqPC4LbQZzWDRXdRRTF9dYSY9kmQ/dKqu aiBExOHZ837qvzipXQylS7PczzchhNq2IKaGqa1XTh6lNqDpKie6fY6/AuBAsmYE/IXs K1FSU0KKE1X1qs3mCAI4POeoveIl84nhS+Nn0UbyIPudJlNHh0t4plMKZ1Ajq30gCBdc znYNYU7Z/6qZEOx3zbzZ2JzyzNhrMWmAzZuZwMzkkznoOiIVenblfupgabiMcrv2G6bh 7B8Q== X-Gm-Message-State: AElRT7EWCYFC2e2Dm/VgLz0gu10oUZihKAQ5bu7CiShV9HR0gTeHaVZE sG4CJLwh81sgOF84F3NzW1+rYKaJ X-Google-Smtp-Source: AIpwx4/mgErcyXr7xyc9lPBmtkO/S2E7/vnPf8htQZn/UImODk2vaf8H2ozbGik+npITvIvN6xKy1A== X-Received: by 10.223.190.138 with SMTP id i10mr7230468wrh.40.1522681090423; Mon, 02 Apr 2018 07:58:10 -0700 (PDT) Received: from cerise.online.net ([195.154.228.195]) by smtp.gmail.com with ESMTPSA id s15sm756001wrg.28.2018.04.02.07.58.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 Apr 2018 07:58:09 -0700 (PDT) From: Maxime Hadjinlian To: buildroot@buildroot.org Date: Mon, 2 Apr 2018 16:58:00 +0200 Message-Id: <20180402145802.17611-6-maxime.hadjinlian@gmail.com> X-Mailer: git-send-email 2.16.2 In-Reply-To: <20180402145802.17611-1-maxime.hadjinlian@gmail.com> References: <20180402145802.17611-1-maxime.hadjinlian@gmail.com> Subject: [Buildroot] [PATCH 6/8] download: add flock call before dl-wrapper X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Maxime Hadjinlian MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" In order to introduce the cache mechanisms, we need to have a lock on the $(LIBFOO_DL_DIR), otherwise it would be impossible to do parallel download (a shared DL_DIR for two buildroot instances). Signed-off-by: Maxime Hadjinlian Reviewed-by: "Yann E. MORIN" --- package/pkg-download.mk | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/package/pkg-download.mk b/package/pkg-download.mk index 7ca6fcc757..4fdb283acc 100644 --- a/package/pkg-download.mk +++ b/package/pkg-download.mk @@ -19,6 +19,7 @@ SSH := $(call qstrip,$(BR2_SSH)) export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES)) DL_WRAPPER = support/download/dl-wrapper +FLOCK = flock $($(PKG)_DL_DIR)/ # DL_DIR may have been set already from the environment ifeq ($(origin DL_DIR),undefined) @@ -93,7 +94,7 @@ define DOWNLOAD $(Q)mkdir -p $($(PKG)_DL_DIR) $(Q)$(if $(filter bzr cvs hg svn,$($(PKG)_SITE_METHOD)), BR_NO_CHECK_HASH_FOR=$(notdir $(call qstrip,$(1)))) \ - $(EXTRA_ENV) $(DL_WRAPPER) \ + $(EXTRA_ENV) $(FLOCK) $(DL_WRAPPER) \ -c '$($(PKG)_DL_VERSION)' \ -f '$(notdir $(1))' \ -H '$(PKGDIR)/$($(PKG)_RAWNAME).hash' \ From patchwork Mon Apr 2 14:58:01 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Hadjinlian X-Patchwork-Id: 894281 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.137; helo=fraxinus.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="FELYUv/M"; dkim-atps=neutral Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40FFhw5X05z9s3R for ; Tue, 3 Apr 2018 00:58:24 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 365B884E34; Mon, 2 Apr 2018 14:58:21 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hP9kZ5djQszt; Mon, 2 Apr 2018 14:58:18 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by fraxinus.osuosl.org (Postfix) with ESMTP id 91AC584CB1; Mon, 2 Apr 2018 14:58:18 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from fraxinus.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by ash.osuosl.org (Postfix) with ESMTP id 23B041C2314 for ; Mon, 2 Apr 2018 14:58:14 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by fraxinus.osuosl.org (Postfix) with ESMTP id 2062984CB1 for ; Mon, 2 Apr 2018 14:58:14 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from fraxinus.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id b2KXtpsHEs8I for ; Mon, 2 Apr 2018 14:58:13 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr0-f195.google.com (mail-wr0-f195.google.com [209.85.128.195]) by fraxinus.osuosl.org (Postfix) with ESMTPS id 4A52984CAB for ; Mon, 2 Apr 2018 14:58:13 +0000 (UTC) Received: by mail-wr0-f195.google.com with SMTP id o8so14181799wra.1 for ; Mon, 02 Apr 2018 07:58:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=CTiM/OqvMBUG5iaiGA8bl5n0j2yQn007YYemvkbS0Zc=; b=FELYUv/Md55N4HkgQCYTb5A2C2KD6IxqbKpNmoHkF5pxyOids0tjL7UuVh1ZQqlcxZ njwTMgpWc8VsIsL9l6n5ICmQZBKXu6thCFfFIpCmjO+yKYWMjKp/x52kn2dgFwDIb3Zk PaZ09MHNBE5O6B4Ws7QVxJv9E4l9AF88kx4aGF4nOAtLoIdDxpfgge/ENJdI44x9Qov/ 8ePbU6ODopTGIV9Ex1VNhYQ3LKFEWJFrVCiSPXJ92+jyPngp95ab4mcYWct/DVfD+cMa TNUY5sG06ISeb5bM/zlU3nG/wTXxHCqspRYFo39QGSxOffbpo9iiVOvbcea3USIkk7SZ dlFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=CTiM/OqvMBUG5iaiGA8bl5n0j2yQn007YYemvkbS0Zc=; b=k5HpOjujpRD6mou0aV2y/THoOULtw5qk1aBS+KWAQGNsAwGgfSL4pJApv1sLAz9aqp 3lzC/gvIeOXgIL0dI2YskpLgEa58Ih7/2Z6XDL69P0m8Xpu7cw8NTRpcY/eTsNjLxtlI GPBGTdnwajGuYU7gZ1gBZaqD+EBISPZH6CZKjemPwV3oVH7VxEfFL/LuC69RtRWd1iDd j54PzoGItOAOqnvjUyoY79N1vfpwXzU8Ut638uBim9yPCYixrscXgAYtTllH8GEx+xij ELYd9O5khvWXUUfqhCObfr9MV1sVyXeqatLh8S1lA6feLlcgVLUllgylb04OAfZvCDDv Dmew== X-Gm-Message-State: AElRT7EbgiBq/wqI2FyNTINkBk315qzZQZ8vH++BUbr3Gr6i9w09qRhi 99F84hlUTnSEI+Pe+CuWNDm31n9o X-Google-Smtp-Source: AIpwx48iQT5UcZcbr4veEm0F8U/TU3Uh+QMQGWQcaT8lPhCTLiOnyxLlVx8UtjSne2rY8YhUaWal8A== X-Received: by 10.223.210.13 with SMTP id g13mr6685112wri.59.1522681091457; Mon, 02 Apr 2018 07:58:11 -0700 (PDT) Received: from cerise.online.net ([195.154.228.195]) by smtp.gmail.com with ESMTPSA id s15sm756001wrg.28.2018.04.02.07.58.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 Apr 2018 07:58:10 -0700 (PDT) From: Maxime Hadjinlian To: buildroot@buildroot.org Date: Mon, 2 Apr 2018 16:58:01 +0200 Message-Id: <20180402145802.17611-7-maxime.hadjinlian@gmail.com> X-Mailer: git-send-email 2.16.2 In-Reply-To: <20180402145802.17611-1-maxime.hadjinlian@gmail.com> References: <20180402145802.17611-1-maxime.hadjinlian@gmail.com> Subject: [Buildroot] [PATCH 7/8] download: add missing '-d' option X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Maxime Hadjinlian MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" The infrastructure needs to give the 'dl_dir' to the dl-wrappe which in its turn needs to give it to the helper. It will only be used by the 'git' helper as of now. Signed-off-by: Maxime Hadjinlian --- package/pkg-download.mk | 1 + support/download/dl-wrapper | 6 ++++-- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/package/pkg-download.mk b/package/pkg-download.mk index 4fdb283acc..b7f1d43920 100644 --- a/package/pkg-download.mk +++ b/package/pkg-download.mk @@ -96,6 +96,7 @@ define DOWNLOAD BR_NO_CHECK_HASH_FOR=$(notdir $(call qstrip,$(1)))) \ $(EXTRA_ENV) $(FLOCK) $(DL_WRAPPER) \ -c '$($(PKG)_DL_VERSION)' \ + -d '$($(PKG)_DL_DIR)' \ -f '$(notdir $(1))' \ -H '$(PKGDIR)/$($(PKG)_RAWNAME).hash' \ -n '$($(PKG)_BASENAME_RAW)' \ diff --git a/support/download/dl-wrapper b/support/download/dl-wrapper index cd903f92d9..af2950ac3b 100755 --- a/support/download/dl-wrapper +++ b/support/download/dl-wrapper @@ -19,7 +19,7 @@ # We want to catch any unexpected failure, and exit immediately. set -e -export BR_BACKEND_DL_GETOPTS=":hc:o:n:N:H:ru:qf:e" +export BR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:ru:qf:e" main() { local OPT OPTARG @@ -27,10 +27,11 @@ main() { local -a uris # Parse our options; anything after '--' is for the backend - while getopts ":hc:o:n:N:H:rf:u:q" OPT; do + while getopts ":hc:d:o:n:N:H:rf:u:q" OPT; do case "${OPT}" in h) help; exit 0;; c) cset="${OPTARG}";; + d) dl_dir="${OPTARG}";; o) output="${OPTARG}";; n) raw_base_name="${OPTARG}";; N) base_name="${OPTARG}";; @@ -109,6 +110,7 @@ main() { if ! "${OLDPWD}/support/download/${backend}" \ $([ -n "${urlencode}" ] && printf %s '-e') \ -c "${cset}" \ + -d "${dl_dir}" \ -n "${raw_base_name}" \ -N "${raw_name}" \ -f "${filename}" \ From patchwork Mon Apr 2 14:58:02 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Maxime Hadjinlian X-Patchwork-Id: 894284 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=busybox.net (client-ip=140.211.166.136; helo=silver.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="ZOb2s4jK"; dkim-atps=neutral Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40FFj34xrTz9s28 for ; Tue, 3 Apr 2018 00:58:31 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 56EB722898; Mon, 2 Apr 2018 14:58:25 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bU-tb-HYJEYi; Mon, 2 Apr 2018 14:58:19 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by silver.osuosl.org (Postfix) with ESMTP id 9CD5325C0B; Mon, 2 Apr 2018 14:58:19 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id 7724D1C2314 for ; Mon, 2 Apr 2018 14:58:15 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 743448A2AB for ; Mon, 2 Apr 2018 14:58:15 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9PjLRD1qu6M1 for ; Mon, 2 Apr 2018 14:58:14 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wr0-f195.google.com (mail-wr0-f195.google.com [209.85.128.195]) by hemlock.osuosl.org (Postfix) with ESMTPS id 574A08A28C for ; Mon, 2 Apr 2018 14:58:14 +0000 (UTC) Received: by mail-wr0-f195.google.com with SMTP id 80so14186157wrb.2 for ; Mon, 02 Apr 2018 07:58:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=mveinkWErPya4/eJpVi0ptuBUCq4gitVMJPd+anRyQk=; b=ZOb2s4jKTqUN6DOTtdN7Yc+ii1Aje24ry6kFNNu8L807WQI0Zhz+AFOl6FqRH/TRtc HcpObWZRy/S1MZ8cT1hxjiFoL1IsV9cIRzpnKWpry7ddF24Q5l7cGd1cuQjOlH00g5wQ 0wwMlNTnhFu+NV+KtmDvf+ewViTzg1b8qzrWvU8AGCkOUxzdzCDjHnJY54ORL2+opS8C OkLWcqmfxbmwPodI699jjohEe9qaFXNnRM2KMZYsx0R2lcLbsm77syB1OPeA+Apau9uL aKlTMg2zVrQ8cg7NDZ7xiqQl0C2ZP/zHSbdI4LYBhYCFqOsQymOkIspeoIVXcJtBmfMA xEvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=mveinkWErPya4/eJpVi0ptuBUCq4gitVMJPd+anRyQk=; b=QC9FYJSguyopr63L9iHnE1jZiHjm8DNd2ZnfVxg1eFvDwoiYg1UPQvMU2rz+QClqUl 1BZrOO8bD+AH6TOX74MG4uwqPaeG8n/mJLXh8uURYDQyb5twBZG9VZ4UMX9/NAMLAjnT ZrDxXUXylK8L+xzaXU1OBFKp+sgBjpSQiRsc5tUg8Rg6U5BeFrwd8rCgQ65Mr5ivEFjT s0ukN9CDdCbiPVovYTOclt1fxKZBJqTUd0ytmzLu3AvSmp5dbWFioF9cZe7NBKVUvg3j ULGsvEuyroboWIdsnR5vnyGPAsIdeaXe2OmOpyl6wMO+judhBDdstvmuOZ0IjGuqIiVL 0lfA== X-Gm-Message-State: ALQs6tCXelYhNGn5Tq6uf+HOz8g/M+l/XYvlzYLEP2QhxjQgQT9dJPkK PhZcrUl3rzrrNv27F6G/9C19zp8A X-Google-Smtp-Source: AIpwx48SSg8et2IZQL/Uc37aVW94cIhcvuDpBruIyAN1w0Jy3dmBJbJ3Njeyd1q3lX0SLLRMjNTTNQ== X-Received: by 10.223.169.54 with SMTP id u51mr6513572wrc.71.1522681092441; Mon, 02 Apr 2018 07:58:12 -0700 (PDT) Received: from cerise.online.net ([195.154.228.195]) by smtp.gmail.com with ESMTPSA id s15sm756001wrg.28.2018.04.02.07.58.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 02 Apr 2018 07:58:11 -0700 (PDT) From: Maxime Hadjinlian To: buildroot@buildroot.org Date: Mon, 2 Apr 2018 16:58:02 +0200 Message-Id: <20180402145802.17611-8-maxime.hadjinlian@gmail.com> X-Mailer: git-send-email 2.16.2 In-Reply-To: <20180402145802.17611-1-maxime.hadjinlian@gmail.com> References: <20180402145802.17611-1-maxime.hadjinlian@gmail.com> Subject: [Buildroot] [PATCH 8/8] download: git: introduce cache feature X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.24 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Maxime Hadjinlian MIME-Version: 1.0 Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" Now we keep the git clone that we download and generates our tarball from there. The main goal here is that if you change the version of a package (say Linux), instead of cloning all over again, you will simply 'git fetch' from the repo the missing objects, then generates the tarball again. This should speed the 'source' part of the build significantly. The drawback is that the DL_DIR will grow much larger; but time is more important than disk space nowadays. Signed-off-by: Maxime Hadjinlian --- v4 -> v5: - Fix comments's content --- support/download/git | 65 +++++++++++++++++++++++++++++++++------------------- 1 file changed, 41 insertions(+), 24 deletions(-) diff --git a/support/download/git b/support/download/git index 58a2c6ad9d..58dbcd211c 100755 --- a/support/download/git +++ b/support/download/git @@ -25,6 +25,7 @@ while getopts "${BR_BACKEND_DL_GETOPTS}" OPT; do o) output="${OPTARG}";; u) uri="${OPTARG}";; c) cset="${OPTARG}";; + d) dl_dir="${OPTARG}";; n) basename="${OPTARG}";; :) printf "option '%s' expects a mandatory argument\n" "${OPTARG}"; exit 1;; \?) printf "unknown option '%s'\n" "${OPTARG}" >&2; exit 1;; @@ -39,29 +40,41 @@ _git() { eval ${GIT} "${@}" } -# Try a shallow clone, since it is faster than a full clone - but that only -# works if the version is a ref (tag or branch). Before trying to do a shallow -# clone we check if ${cset} is in the list provided by git ls-remote. If not -# we fall back on a full clone. +# We want to check if a cache of the git clone of this repo already exists. +git_cache="${dl_dir}/git" + +# If the cache directory doesn't exists, init a new repo, which will be +# fetch'ed later. +if [ ! -d "${git_cache}" ]; then + _git init "'${git_cache}'" + _git -C "'${git_cache}'" remote add origin "'${uri}'" +fi + +pushd "${git_cache}" >/dev/null + +_git remote set-url origin "'${uri}'" + +# Try to fetch with limited depth, since it is faster than a full clone - but +# that only works if the version is a ref (tag or branch). Before trying to do +# a shallow clone we check if ${cset} is in the list provided by git ls-remote. +# If not we fallback to a full fetch. # -# Messages for the type of clone used are provided to ease debugging in case of -# problems +# Messages for the type of clone used are provided to ease debugging in +# case of problems git_done=0 -if [ -n "$(_git ls-remote "'${uri}'" "'${cset}'" 2>&1)" ]; then - printf "Doing shallow clone\n" - if _git clone ${verbose} "${@}" --depth 1 -b "'${cset}'" "'${uri}'" "'${basename}'"; then +if [ -n "$(_git ls-remote origin "'${cset}'" 2>&1)" ]; then + printf "Doing a shallow fetch\n" + if _git fetch "${@}" --depth 1 origin "'${cset}'"; then git_done=1 else - printf "Shallow clone failed, falling back to doing a full clone\n" + printf "Shallow fetch failed, falling back to fetching all refs\n" fi fi if [ ${git_done} -eq 0 ]; then - printf "Doing full clone\n" - _git clone ${verbose} "${@}" "'${uri}'" "'${basename}'" + printf "Fetching all references\n" + _git fetch origin -t fi -pushd "${basename}" >/dev/null - # Try to get the special refs exposed by some forges (pull-requests for # github, changes for gerrit...). There is no easy way to know whether # the cset the user passed us is such a special ref or a tag or a sha1 @@ -86,20 +99,24 @@ if [ ${recurse} -eq 1 ]; then _git submodule update --init --recursive fi -# We do not want the .git dir; we keep other .git files, in case they -# are the only files in their directory. +# Generate the archive, sort with the C locale so that it is reproducible. +# We do not want the .git dir; we keep other .git files, in case they are the +# only files in their directory. # The .git dir would generate non reproducible tarballs as it depends on # the state of the remote server. It also would generate large tarballs # (gigabytes for some linux trees) when a full clone took place. -rm -rf .git +find . -not -type d \ + -and -not -path "./.git/*" >"${output}.list" +LC_ALL=C sort <"${output}.list" >"${output}.list.sorted" -popd >/dev/null - -# Generate the archive, sort with the C locale so that it is reproducible -find "${basename}" -not -type d >"${basename}.list" -LC_ALL=C sort <"${basename}.list" >"${basename}.list.sorted" # Create GNU-format tarballs, since that's the format of the tarballs on # sources.buildroot.org and used in the *.hash files -tar cf - --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \ - -T "${basename}.list.sorted" >"${output}.tar" +tar cf - --transform="s/^\.$/${basename}/" \ + --numeric-owner --owner=0 --group=0 --mtime="${date}" --format=gnu \ + -T "${output}.list.sorted" >"${output}.tar" gzip -6 -n <"${output}.tar" >"${output}" + +rm -f "${output}.list" +rm -f "${output}.list.sorted" + +popd >/dev/null