From patchwork Sun Mar 1 21:25:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Titouan Christophe X-Patchwork-Id: 1247374 Return-Path: X-Original-To: incoming-buildroot@patchwork.ozlabs.org Delivered-To: patchwork-incoming-buildroot@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=busybox.net (client-ip=140.211.166.138; helo=whitealder.osuosl.org; envelope-from=buildroot-bounces@busybox.net; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=railnova.eu Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=railnova-eu.20150623.gappssmtp.com header.i=@railnova-eu.20150623.gappssmtp.com header.a=rsa-sha256 header.s=20150623 header.b=CYEwU2b+; dkim-atps=neutral Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 48VxCY5W3Tz9sSQ for ; Mon, 2 Mar 2020 08:26:01 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 08DA285A8B; Sun, 1 Mar 2020 21:25:58 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ejoe1kN6dlOH; Sun, 1 Mar 2020 21:25:56 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by whitealder.osuosl.org (Postfix) with ESMTP id 6C3F0856F4; Sun, 1 Mar 2020 21:25:56 +0000 (UTC) X-Original-To: buildroot@lists.busybox.net Delivered-To: buildroot@osuosl.org Received: from whitealder.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id 34B731BF3CC for ; Sun, 1 Mar 2020 21:25:44 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by whitealder.osuosl.org (Postfix) with ESMTP id 30C5F856F4 for ; Sun, 1 Mar 2020 21:25:44 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from whitealder.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JW98vcdAnNlV for ; Sun, 1 Mar 2020 21:25:43 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from mail-wm1-f68.google.com (mail-wm1-f68.google.com [209.85.128.68]) by whitealder.osuosl.org (Postfix) with ESMTPS id 3846F83079 for ; Sun, 1 Mar 2020 21:25:43 +0000 (UTC) Received: by mail-wm1-f68.google.com with SMTP id 6so571914wmi.5 for ; Sun, 01 Mar 2020 13:25:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=railnova-eu.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=+If828iZfvccxGCObZ1itPMWqBuqRsUA8eS1aJBpPAA=; b=CYEwU2b+B1wcAriBepDwnHEuZaC4HoMEKyhTlUlnQoXHtCOsYgG59DTazg5YwuVBDx CMbvlQ7PhkgjULY+x7yIKZImmBFfyOdTdMDXK9EDrEY4u8q3EM18IKq6GfaIge5Eei+K CfLtXDLbXhID70lLxDCoTGaLYlcVGB24FsOa3gDuiE26P1DS70BD2BokBt75QtanYxq8 TSl5+OFbVRVzQf4X6C/tRzubYHFR89A3wYK0pc7z/cC8Qqz+0pNSAS+XWKIIyPocn78B vjLs+R8YBn4UuKx/5uA5VaZUbkfxO4KQWzjKQFlCYI18rP2U4sjBTYztL1sIC1JKZb3l 3l1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=+If828iZfvccxGCObZ1itPMWqBuqRsUA8eS1aJBpPAA=; b=UCHL3JKgcKifWmcnUgOnFHiqK+MiOvGcH++JcKRXPyFqGuqIDLhnGfqxSjNvFzZexX fMKUw2WbN+CfHcKYj0r7uZTTdJWtkT4VFZ6fuNa0/5rGxvn3mFhMTsvLfL3pG+3ANn8h eOi17uD1gRBsbgGcpjlH6QHjPSZEvEbmoA1wZJuUgNFNQj8ZgLPxyx5i4TOM9MbB2fF7 lCsZhRcVNo/M1g6NSZyPdy28ACP9GMTUnmrpWdkRHBW0+BSnvmrmnNA348KMkL1uDlNE LlRInoQXgP2DEI9S401vyPADrcHs4AmhVBk5gJkc0ZBajpDPYjDa915UOewL6AicLHaG bkQQ== X-Gm-Message-State: APjAAAWZ9SOTY+sk33nKLTikDFuRstYqLA8XUVRAyPPx5TuE11JhFAZL IKj0nAJg68v9XzrbTheyYGhprGSGfm393GNN X-Google-Smtp-Source: APXvYqwDld4V4xrAffCo3r8jcOh3X2QFOxOcfe68H1xwZZcILBjnFUBHT5Lu59BMgOVygVr0IfMFeg== X-Received: by 2002:a1c:994b:: with SMTP id b72mr16717048wme.83.1583097941313; Sun, 01 Mar 2020 13:25:41 -0800 (PST) Received: from localhost.localdomain ([185.224.57.162]) by smtp.gmail.com with ESMTPSA id t13sm23568322wrw.19.2020.03.01.13.25.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 01 Mar 2020 13:25:40 -0800 (PST) From: Titouan Christophe To: buildroot@buildroot.org Date: Sun, 1 Mar 2020 22:25:28 +0100 Message-Id: <20200301212528.15754-1-titouan.christophe@railnova.eu> X-Mailer: git-send-email 2.24.1 MIME-Version: 1.0 Subject: [Buildroot] [PATCH 1/1] support/scripts/pkg-stats: clear multiprocessing pools after use X-BeenThere: buildroot@busybox.net X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion and development of buildroot List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Thomas De Schampheleire , Thomas Petazzoni , Titouan Christophe Errors-To: buildroot-bounces@busybox.net Sender: "buildroot" During the CVE checking phase, we can still see a huge amount of Python processes (actually 128) running on the host, even though the CVE step is entirely ran in the main thread. These are actually the worker processes spawned to check for the packages URL statuses and the latest versions from release-monitoring. This is because of an issue in Python's multiprocessing implementation: https://bugs.python.org/issue34172 The problem was already there before the CVE matching step was introduced, but because pkg-stat was terminating right after the release-monitoring step, it went unnoticed. Also, do not hold a reference to the multiprocessing pool from the Package class, as this is not needed. Signed-off-by: Titouan Christophe --- support/scripts/pkg-stats | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/support/scripts/pkg-stats b/support/scripts/pkg-stats index 7721d98459..4f653a6ad9 100755 --- a/support/scripts/pkg-stats +++ b/support/scripts/pkg-stats @@ -402,11 +402,13 @@ def check_url_status_worker(url, url_status): def check_package_urls(packages): - Package.pool = Pool(processes=64) + pool = Pool(processes=64) for pkg in packages: - pkg.url_worker = pkg.pool.apply_async(check_url_status_worker, (pkg.url, pkg.url_status)) + pkg.url_worker = pool.apply_async(check_url_status_worker, (pkg.url, pkg.url_status)) for pkg in packages: pkg.url_status = pkg.url_worker.get(timeout=3600) + del pkg.url_worker + pool.terminate() def release_monitoring_get_latest_version_by_distro(pool, name): @@ -479,6 +481,7 @@ def check_package_latest_version(packages): results = worker_pool.map(check_package_latest_version_worker, (pkg.name for pkg in packages)) for pkg, r in zip(packages, results): pkg.latest_version = r + worker_pool.terminate() del http_pool