From patchwork Mon Jul 20 10:42:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rohit Sarkar X-Patchwork-Id: 1332183 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4B9JGb5ZDMz9sR4 for ; Mon, 20 Jul 2020 20:43:03 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=T+s1MBv3; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 4B9JGb4PSgzDqd2 for ; Mon, 20 Jul 2020 20:43:03 +1000 (AEST) X-Original-To: patchwork@lists.ozlabs.org Delivered-To: patchwork@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=gmail.com (client-ip=2607:f8b0:4864:20::1042; helo=mail-pj1-x1042.google.com; envelope-from=rohitsarkar5398@gmail.com; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=T+s1MBv3; dkim-atps=neutral Received: from mail-pj1-x1042.google.com (mail-pj1-x1042.google.com [IPv6:2607:f8b0:4864:20::1042]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4B9JGF0fHczDqZG for ; Mon, 20 Jul 2020 20:42:44 +1000 (AEST) Received: by mail-pj1-x1042.google.com with SMTP id gc9so10028600pjb.2 for ; Mon, 20 Jul 2020 03:42:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=RAG1HCnwiSNHW9Pzc2h7XXhMB92+VFbd6sFrCma2RH0=; b=T+s1MBv3f+5maDzQ/6UEVWKUTwu8tuxvyS9z82SVfxIfsPQw06qwbfsYVyK59F3SjT 17FBpS94h+tshJ4fonsqTMxjaCx1nfMnQ5GBv5/woYUk1M/UpywDBq7bQcmflxtQ1iuS ULToJm9+PdtxEhScZX07V7D/PkZS+COaUTKsT81xMRkxKEhJjCNy0sZE6sRZoxuEVOYw EFK9mEwW29y1BrSBuXIurr8qzS8caT+PJYdUhrigl3kJF5lid2QcxXH7fe43lloDFg8t MyGDGaajprxEiOqbza9aJFiuVf8Q5qO9f7CEZbKw0FlJotQiWXac/XCcwVf4ErB8A/vU CtzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=RAG1HCnwiSNHW9Pzc2h7XXhMB92+VFbd6sFrCma2RH0=; b=cAaA2f9Ze2Sr5WA4pikluNqLE6YCO85nCXY9x4pSx30F0j0Mash3YWxvKW777YbaRd FYLt6aLRH9QRXfzXPsZ4noxS60NIQJpBdFhcJfbHMnxZgQtWjnQZnXSEkbjPP8V5QCn0 AcY19n8uKjVZIYJ+HWWtsoEKiorLtXHA+SOcF3wjyAxxdujevm/CdJZZcsGold7atdtk Av2QkFcAS6UaTU4j4xG10evpJ48XqfDPPCidwwxbyfzlDU1MtExXdCwi/aA9ztpJEyee 6VepVIr7SiDkEWEhs6RVzknWRKE36EgqkZPClYNjWzz7WKChIRhRLcC1sH4iFNfC+fdS nr0A== X-Gm-Message-State: AOAM5315aQqY7SWhRZ8PXoZaNMz2uSe5LBVLLaBc96ySDzAHzVcVBx5o q9Tkd6HdMrqHkhK1Rh25dPlypGsgX2jGxA== X-Google-Smtp-Source: ABdhPJwUrhn2+tnw/j2LuEyvhMsPD+/hxWR14J6W2y8uhsOx+WbAcw7SGRSiSa6Up3MVayjFl+CwKg== X-Received: by 2002:a17:90a:2a02:: with SMTP id i2mr22983490pjd.157.1595241761537; Mon, 20 Jul 2020 03:42:41 -0700 (PDT) Received: from localhost.localdomain ([103.252.169.174]) by smtp.gmail.com with ESMTPSA id c19sm11295069pjs.11.2020.07.20.03.42.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Jul 2020 03:42:40 -0700 (PDT) From: Rohit Sarkar To: patchwork@lists.ozlabs.org Subject: [PATCH v4 1/1] management: introduce replacerelations command Date: Mon, 20 Jul 2020 16:12:23 +0530 Message-Id: <20200720104223.6957-2-rohitsarkar5398@gmail.com> X-Mailer: git-send-email 2.23.0.385.gbc12974a89 In-Reply-To: <20200720104223.6957-1-rohitsarkar5398@gmail.com> References: <20200720104223.6957-1-rohitsarkar5398@gmail.com> MIME-Version: 1.0 X-BeenThere: patchwork@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Patchwork development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: ralf.ramsauer@oth-regensburg.de Errors-To: patchwork-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Patchwork" The replacerelations script is used to ingest relations into Patchwork's patch database. A patch groups file is taken as input, which on each line contains a space separated list of patchwork ids denoting a relation. All the existing relations in Patchwork's database are removed and the relations read from the patch groups file are ingested. Signed-off-by: Rohit Sarkar --- docs/deployment/management.rst | 28 ++++++++ .../management/commands/replacerelations.py | 72 +++++++++++++++++++ patchwork/tests/__init__.py | 1 + patchwork/tests/relations/patch-groups | 3 + .../relations/patch-groups-missing-patch-ids | 4 ++ patchwork/tests/test_management.py | 27 ++++++- 6 files changed, 134 insertions(+), 1 deletion(-) create mode 100644 patchwork/management/commands/replacerelations.py create mode 100644 patchwork/tests/relations/patch-groups create mode 100644 patchwork/tests/relations/patch-groups-missing-patch-ids diff --git a/docs/deployment/management.rst b/docs/deployment/management.rst index 9c57f19..1437550 100644 --- a/docs/deployment/management.rst +++ b/docs/deployment/management.rst @@ -116,6 +116,34 @@ the :ref:`deployment installation guide `. input mbox filename. If not supplied, a patch will be read from ``stdin``. +replacerelations +~~~~~~~~~~~~~~~~ + +.. program:: manage.py replacerelations + +Parse a patch groups file and store any relation found + +.. code-block:: shell + + ./manage.py replacerelations + +This is a script used to ingest relations into Patchwork. +A patch groups file contains on each line patch ids of patches that form a relation. +For example, consider the contents of a sample patch groups file: + + 1 3 5 + 2 + 7 10 11 12 + +In this case the script will identify 2 relations, (1, 3, 5) and (7, 10, 11, 12). +The single patch id "2" on the second line is ignored as a relation always consists of more than 1 patches. +Further if a patch id in the patch groups file does not exist in the database of the Patchwork instance, that patch id will be silently ignored while forming the relations +Running this script will remove all existing relations and replace them with the relations found in the file. + +.. option:: infile + + input patch groups file. + rehash ~~~~~~ diff --git a/patchwork/management/commands/replacerelations.py b/patchwork/management/commands/replacerelations.py new file mode 100644 index 0000000..0cc4d0a --- /dev/null +++ b/patchwork/management/commands/replacerelations.py @@ -0,0 +1,72 @@ +# Patchwork - automated patch tracking system +# Copyright (C) 2020 Rohit Sarkar +# +# SPDX-License-Identifier: GPL-2.0-or-later + +import logging +import os +import sys + +from django.db import transaction +from django.core.management.base import BaseCommand + +from patchwork.models import Patch +from patchwork.models import PatchRelation + +logger = logging.getLogger(__name__) + +class Command(BaseCommand): + help = 'Parse a relations file generated by PaStA and replace existing relations with the ones parsed' + + def add_arguments(self, parser): + parser.add_argument( + 'infile', + help='input relations filename') + + def handle(self, *args, **options): + verbosity = int(options['verbosity']) + if not verbosity: + level = logging.CRITICAL + elif verbosity == 1: + level = logging.ERROR + elif verbosity == 2: + level = logging.INFO + else: + level = logging.DEBUG + + logger.setLevel(level) + + path = args and args[0] or options['infile'] + if not os.path.exists(path): + logger.error('Invalid path: %s', path) + sys.exit(1) + + + with open(path, 'r') as f: + lines = f.readlines() + + # filter out trailing empty lines + while len(lines) and not lines[-1]: + lines.pop() + + relations = [line.split(' ') for line in lines] + + with transaction.atomic(): + PatchRelation.objects.all().delete() + count = len(relations) + ingested = 0 + logger.info('Parsing %d relations' % count) + for i, patch_ids in enumerate(relations): + related_patches = Patch.objects.filter(id__in=patch_ids) + + if len(related_patches) > 1: + relation = PatchRelation() + relation.save() + related_patches.update(related=relation) + ingested += 1 + + if i % 10 == 0: + self.stdout.write('%06d/%06d\r' % (i, count), ending='') + self.stdout.flush() + + self.stdout.write('Ingested %d relations' % ingested) diff --git a/patchwork/tests/__init__.py b/patchwork/tests/__init__.py index 8f78ea7..83358a6 100644 --- a/patchwork/tests/__init__.py +++ b/patchwork/tests/__init__.py @@ -8,3 +8,4 @@ import os TEST_MAIL_DIR = os.path.join(os.path.dirname(__file__), 'mail') TEST_PATCH_DIR = os.path.join(os.path.dirname(__file__), 'patches') TEST_FUZZ_DIR = os.path.join(os.path.dirname(__file__), 'fuzztests') +TEST_RELATIONS_DIR = os.path.join(os.path.dirname(__file__), 'relations') diff --git a/patchwork/tests/relations/patch-groups b/patchwork/tests/relations/patch-groups new file mode 100644 index 0000000..593cf0f --- /dev/null +++ b/patchwork/tests/relations/patch-groups @@ -0,0 +1,3 @@ +1 2 +3 4 5 +6 diff --git a/patchwork/tests/relations/patch-groups-missing-patch-ids b/patchwork/tests/relations/patch-groups-missing-patch-ids new file mode 100644 index 0000000..091b195 --- /dev/null +++ b/patchwork/tests/relations/patch-groups-missing-patch-ids @@ -0,0 +1,4 @@ +1 2 +3 4 5 7 +6 8 +9 10 diff --git a/patchwork/tests/test_management.py b/patchwork/tests/test_management.py index 66c6bad..905369d 100644 --- a/patchwork/tests/test_management.py +++ b/patchwork/tests/test_management.py @@ -11,7 +11,7 @@ from django.core.management import call_command from django.test import TestCase from patchwork import models -from patchwork.tests import TEST_MAIL_DIR +from patchwork.tests import TEST_MAIL_DIR, TEST_RELATIONS_DIR from patchwork.tests import utils @@ -124,3 +124,28 @@ class ParsearchiveTest(TestCase): self.assertIn('Processed 1 messages -->', out.getvalue()) self.assertIn(' 1 dropped', out.getvalue()) + +class ReplacerelationsTest(TestCase): + def test_invalid_path(self): + out = StringIO() + with self.assertRaises(SystemExit) as exc: + call_command('replacerelations', 'xyz123random', stdout=out) + self.assertEqual(exc.exception.code, 1) + + def test_valid_relations(self): + utils.create_patches(6) + out = StringIO() + call_command('replacerelations', + os.path.join(TEST_RELATIONS_DIR, + 'patch-groups'), + stdout=out) + + self.assertEqual(models.PatchRelation.objects.count(), 2) + + call_command('replacerelations', + os.path.join(TEST_RELATIONS_DIR, + 'patch-groups-missing-patch-ids'), + stdout=out) + + self.assertEqual(models.PatchRelation.objects.count(), 2) +