From patchwork Tue Nov 29 02:36:13 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sam Bobroff X-Patchwork-Id: 700276 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3tSSNZ059Nz9vDp for ; Tue, 29 Nov 2016 13:36:29 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 26E4094E33; Tue, 29 Nov 2016 02:36:26 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Mah92z2z5C1X; Tue, 29 Nov 2016 02:36:24 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by hemlock.osuosl.org (Postfix) with ESMTP id 27AF294E34; Tue, 29 Nov 2016 02:36:24 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by ash.osuosl.org (Postfix) with ESMTP id ADEC31C002C for ; Tue, 29 Nov 2016 02:36:22 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id A7FD394E34 for ; Tue, 29 Nov 2016 02:36:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NUN+b81hdcUN for ; Tue, 29 Nov 2016 02:36:21 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by hemlock.osuosl.org (Postfix) with ESMTPS id 98EB994E33 for ; Tue, 29 Nov 2016 02:36:21 +0000 (UTC) Received: from pps.filterd (m0098410.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.17/8.16.0.17) with SMTP id uAT2XxLw041054 for ; Mon, 28 Nov 2016 21:36:21 -0500 Received: from e23smtp08.au.ibm.com (e23smtp08.au.ibm.com [202.81.31.141]) by mx0a-001b2d01.pphosted.com with ESMTP id 270vsgd9ac-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Mon, 28 Nov 2016 21:36:20 -0500 Received: from localhost by e23smtp08.au.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 29 Nov 2016 12:36:18 +1000 Received: from d23dlp02.au.ibm.com (202.81.31.213) by e23smtp08.au.ibm.com (202.81.31.205) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Tue, 29 Nov 2016 12:36:16 +1000 Received: from d23relay07.au.ibm.com (d23relay07.au.ibm.com [9.190.26.37]) by d23dlp02.au.ibm.com (Postfix) with ESMTP id F20EA2BB0055 for ; Tue, 29 Nov 2016 13:36:15 +1100 (EST) Received: from d23av04.au.ibm.com (d23av04.au.ibm.com [9.190.235.139]) by d23relay07.au.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id uAT2aFMJ34144490 for ; Tue, 29 Nov 2016 13:36:15 +1100 Received: from d23av04.au.ibm.com (localhost [127.0.0.1]) by d23av04.au.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id uAT2aFd0015293 for ; Tue, 29 Nov 2016 13:36:15 +1100 Received: from ozlabs.au.ibm.com (ozlabs.au.ibm.com [9.192.253.14]) by d23av04.au.ibm.com (8.14.4/8.14.4/NCO v10.0 AVin) with ESMTP id uAT2aF3F015290; Tue, 29 Nov 2016 13:36:15 +1100 Received: from tungsten.ozlabs.ibm.com (haven.au.ibm.com [9.192.254.114]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ozlabs.au.ibm.com (Postfix) with ESMTPSA id 1B1C6A01CF; Tue, 29 Nov 2016 13:36:15 +1100 (AEDT) Date: Tue, 29 Nov 2016 13:36:13 +1100 From: Sam Bobroff To: Thomas Petazzoni References: <20161128234621.0e512742@free-electrons.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20161128234621.0e512742@free-electrons.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-TM-AS-MML: disable X-Content-Scanned: Fidelis XPS MAILER x-cbid: 16112902-0048-0000-0000-000001E4F3C4 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 16112902-0049-0000-0000-000047130EDF Message-Id: <20161129023613.GB4657@tungsten.ozlabs.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2016-11-29_01:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1609300000 definitions=main-1611290041 Cc: "buildroot@uclibc.org" Subject: Re: [Buildroot] Investigating a parallel installation failure X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" On Mon, Nov 28, 2016 at 11:46:21PM +0100, Thomas Petazzoni wrote: > Hello, > > Tonight, I've been trying to investigate a build failure of > usb_modeswitch that occurred three times: > > http://autobuild.buildroot.net/results/06e/06e6f510415933b55eb8814284f63109dace3402/build-end.log > http://autobuild.buildroot.net/results/f3f/f3fa30a11b68721b81b5696b089f4a7528d580ad/build-end.log > http://autobuild.buildroot.net/results/b11/b11df0218500ff29c2a067f58f196a7b26c027f0/build-end.log > > Every time, the installation fails with: > > install: cannot stat `usb_modeswitch_dispatcher': No such file or directory > > What we can see is that usb_modeswitch_dispatcher is first built during > the build step: > > >>> usb_modeswitch 2.3.0 Building > [...] > /home/peko/autobuild/instance-1/output/host/usr/bin/i586-linux-gcc dispatcher.c jim/libjim.a -Ijim -o usb_modeswitch_dispatcher -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os -D_GNU_SOURCE -Wall -I. > [...] > > And then again during the install step, after which it tries to be > installed: > > >>> usb_modeswitch 2.3.0 Installing to target > [...] > /home/peko/autobuild/instance-1/output/host/usr/bin/i586-linux-gcc dispatcher.c jim/libjim.a -Ijim -o usb_modeswitch_dispatcher -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os > [...] > install -D --mode=755 usb_modeswitch_dispatcher /home/peko/autobuild/instance-1/output/target/usr/sbin/usb_modeswitch_dispatcher > install: cannot stat `usb_modeswitch_dispatcher': No such file or directory > > The reason why it gets rebuilt is because, in the Makefile: > > === > > dispatcher-static: jim/libjim.a dispatcher.c usb_modeswitch.string > $(CC) dispatcher.c $(LDFLAGS) jim/libjim.a -Ijim -o usb_modeswitch_dispatcher $(CFLAGS) > > install-static: dispatcher-static install-common > > === > > Since dispatcher-static is a phony target, it gets rebuilt everytime it > is depended on, which explains why usb_modeswitch_dispatcher is built > both at build time and install time. > > However, it does not explain why we have this race. What might explain > it, but I'm not sure to fully understand is that 'install-common' is > defined as follows: > > === > > install-common: $(PROG) usb_modeswitch_dispatcher > install -D --mode=755 usb_modeswitch $(SBINDIR)/usb_modeswitch > install -D --mode=755 usb_modeswitch.sh $(UDEVDIR)/usb_modeswitch > install -D --mode=644 usb_modeswitch.conf $(ETCDIR)/usb_modeswitch.conf > install -D --mode=644 usb_modeswitch.1 $(MANDIR)/usb_modeswitch.1 > install -D --mode=644 usb_modeswitch_dispatcher.1 $(MANDIR)/usb_modeswitch_dispatcher.1 > install -D --mode=755 usb_modeswitch_dispatcher $(SBINDIR)/usb_modeswitch_dispatcher > > === > > So install-common depends on the non-phy usb_modeswitch_dispatcher > target... which in fact doesn't exist anywhere in the Makefile! So if > the usb_modeswitch_dispatcher file has not been already produced, this > should lead to a "No such target" error message from make (which is not > what we get). > > So I can see that the dispatcher-static and install-common targets get > executed simultaneously, but it should rather trigger a "No such > target" message for the missing usb_modeswitch_dispatcher rather than a > failure during the installation of usb_modeswitch_dispatcher itself. > > Anyone understands what's going on here? > > Thomas I couldn't replicate this, but I tried hacking strace into the process so I could see if usb_modeswitch_dispatcher was being deleted during the build. It looks like it is, by the C compiler -- presumably removing the existing output file before it writes a new one. So you could see an interleaving where Make has seen the file but it's deleted either just before, or during, /usr/bin/install. And this can happen as long as Make thinks its OK to run those two rules in parallel. If that's correct, you could try converting the dependency on usb_modeswitch_dispatcher (which, as you point out, isn't really working anyway) into an order only dependency (or you could even just delete it -- it doesn't seem to do anything necessary unless you're running a "make install-static" without having first done a "make static" but buildroot does that). This really isn't fixing the whole problem (IMHO the makefile is quite broken) but it might hack around it by not forcing the unnecessary recompilation thus avoiding the racy delete. e.g. Good luck, Sam. --- a/Makefile 2016-11-29 13:29:55.238791205 +1100 +++ b/Makefile 2016-11-29 13:30:47.967090359 +1100 @@ -89,11 +89,11 @@ install: install-script -install-script: dispatcher-script install-common +install-script: install-common | dispatcher-script -install-shared: dispatcher-shared install-common +install-shared: install-common | dispatcher-shared -install-static: dispatcher-static install-common +install-static: install-common | dispatcher-static uninstall: $(RM) $(SBINDIR)/usb_modeswitch