From patchwork Mon Jun 14 12:11:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lukas Zeller X-Patchwork-Id: 1491648 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=lists.openwrt.org (client-ip=2607:7c80:54:e::133; helo=bombadil.infradead.org; envelope-from=openwrt-devel-bounces+incoming=patchwork.ozlabs.org@lists.openwrt.org; receiver=) Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; secure) header.d=lists.infradead.org header.i=@lists.infradead.org header.a=rsa-sha256 header.s=bombadil.20210309 header.b=eHjEwv1A; dkim-atps=neutral Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:e::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G3VnS53Xdz9sXb for ; Mon, 14 Jun 2021 22:17:16 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: List-Subscribe:List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:To: Cc:Date:Message-Id:Subject:Mime-Version:From:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Owner; bh=TspN5gr2N4dznWFExPGqwyiowTDMKiUMu+4Lu6xSjsE=; b=eHj Ewv1AmAhl0AwgbeKmms01hEBv4U64dBQ/MGJwe1m9rCVOrY8PP9OP9aWgfVL4bc0JrcqltcGIIcNz SA/vb9mZey5Atz0uCclhSYTM/w8ZTT3ufKpH5tLkAClom151I5C4Ue7C6YWhkz+Gzclkl0p25wJKk iy58Rsi09ceJi+CZg50tWgY8fGJDr3eHbehTHWwdsa1/RB/ruPmFVSL7Dfb4SHFPOdp3og88ZanNo GL6aodOAQ343MCIPnA1LNZhgSsROFMhtbclLqwrAxnyaIOJqAXZDrWk1HerU8fhL4NsMLFgqeTuiK TTXsNVumGnMmAIV48st+O3E02VsUs8g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1lslRK-00E01o-Sq; Mon, 14 Jun 2021 12:11:38 +0000 Received: from plan442.nine.ch ([5.148.180.31]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1lslRH-00Dzzb-1H for openwrt-devel@lists.openwrt.org; Mon, 14 Jun 2021 12:11:36 +0000 Received: from fiber7.gleis70.ch ([85.195.224.34] helo=smtpclient.apple) by plan442.nine.ch with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lslRA-0002pk-3y; Mon, 14 Jun 2021 14:11:28 +0200 From: Lukas Zeller X-Mao-Original-Outgoing-Id: 645365487.53991-0cea246c5f326a814693298e123524dd Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.80.0.2.43\)) Subject: download: regression with USE_SOURCE_DIR and non-tarball packages? Message-Id: <6F4802D5-49F3-47D2-AC55-8A738F96C124@plan44.ch> Date: Mon, 14 Jun 2021 14:11:27 +0200 Cc: =?utf-8?q?Petr_=C5=A0tetiar?= To: openwrt-devel@lists.openwrt.org X-Mailer: Apple Mail (2.3654.80.0.2.43) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210614_051135_106608_2F113D57 X-CRM114-Status: GOOD ( 11.26 ) X-Spam-Score: 2.5 (++) X-Spam-Report: Spam detection software, running on the system "bombadil.infradead.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Hello Petr, a while ago (after switching projects from 19.07.3 to 19.07.6) I experienced unexpected rebuilds of packages. I asked about this in the forum [1] with no echo, so last week I dug deeper and think I fo [...] Content analysis details: (2.5 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at https://www.dnswl.org/, low trust [5.148.180.31 listed in list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record 3.2 FOUND_YOU I found you... X-BeenThere: openwrt-devel@lists.openwrt.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: OpenWrt Development List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "openwrt-devel" Errors-To: openwrt-devel-bounces+incoming=patchwork.ozlabs.org@lists.openwrt.org Hello Petr, a while ago (after switching projects from 19.07.3 to 19.07.6) I experienced unexpected rebuilds of packages. I asked about this in the forum [1] with no echo, so last week I dug deeper and think I found your commit from last november "4e19cbc - download: handle possibly invalid local tarballs" [2] causing what I see. I'm a build system newbie, so please apologize if I misunderstand things here. But it seems to me that this patch only works for packages that pull their upstream source from a tarball via download.pl, because only there a hash check prevents redownloads (causing rebuilds) in the last possible moment by exiting the script before the download would happen. For packages with PKG_SOURCE_PROTO=git the new line in package.mk `$(DL_DIR)/$(FILE): FORCE` forces re-cloning the upstream repo and thus rebuilding the entire package every time. Also, the USE_SOURCE_DIR mechanism does not work any more. Both git based packages and USE_SOURCE_DIR are central to my work (which is building automation devices based on OpenWrt, so most work is happening *in* my own packages, which are in git or, during hot development phases, via USE_SOURCE_DIR even directly linked dirs). I realize this is not a common situation for OpenWrt platform developers, but only for developers basing other stuff on OpenWrt. As said, I don't understand the details of the build system enough to actually propose a proper patch right into the heart of it. But below is my current workaround to re-enable USE_SOURCE_DIR and git-based packages, hopefully not affecting anything else I can't see. So far, works for me, but definitely needs a thorough look by someone who actually understands the whole thing... Lukas [1] https://forum.openwrt.org/t/does-19-07-6-trigger-package-downloads-rebuilds-more-frequently-than-19-07-3/92471 [2] http://lists.openwrt.org/pipermail/openwrt-devel/2020-November/032235.html --- a/include/package.mk +++ b/include/package.mk @@ -177,6 +177,12 @@ define Build/Exports/Default endef Build/Exports=$(Build/Exports/Default) +ifneq ($(wildcard $(PKG_BUILD_DIR)/.source_dir),) + FORCE_DL:= +else + FORCE_DL=$(if $(findstring git,$(PKG_SOURCE_PROTO)),,$(DL_DIR)/$(FILE): FORCE) +endif + define Build/CoreTargets STAMP_PREPARED:=$$(STAMP_PREPARED) STAMP_CONFIGURED:=$$(STAMP_CONFIGURED) @@ -185,7 +191,7 @@ define Build/CoreTargets $(call Build/Autoclean) $(call DefaultTargets) - $(DL_DIR)/$(FILE): FORCE + $(FORCE_DL) download: $(foreach hook,$(Hooks/Download),