From patchwork Wed Sep 18 06:17:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Axtens X-Patchwork-Id: 1163756 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46Y8tX5fSnz9sCJ for ; Wed, 18 Sep 2019 16:18:28 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=axtens.net Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="Xa44/ra9"; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 46Y8tX34sKzF232 for ; Wed, 18 Sep 2019 16:18:28 +1000 (AEST) X-Original-To: patchwork@lists.ozlabs.org Delivered-To: patchwork@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=axtens.net (client-ip=2607:f8b0:4864:20::62e; helo=mail-pl1-x62e.google.com; envelope-from=dja@axtens.net; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=axtens.net Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="Xa44/ra9"; dkim-atps=neutral Received: from mail-pl1-x62e.google.com (mail-pl1-x62e.google.com [IPv6:2607:f8b0:4864:20::62e]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 46Y8tC3KlDzDqkk for ; Wed, 18 Sep 2019 16:18:11 +1000 (AEST) Received: by mail-pl1-x62e.google.com with SMTP id d22so2623273pll.7 for ; Tue, 17 Sep 2019 23:18:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axtens.net; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=X079ibtO5T1AfH5pj1CqwydSo8VDtcO99dyISg5lZ78=; b=Xa44/ra9hO43OXHoaZzkj0sus0t4geCNzPDWZGLi4HgNP5EMRA4kbRTziEQCHzIFU7 K0KM8nwgnvj6JEmMsUQ/4OPc4+eUo9PjYf0RSRpZQ2yQGL014uK8E9DxqsfxXAgUdkgi wxkJ7oID0T+owTFHhOfEygHTsB43WfkybBi+M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=X079ibtO5T1AfH5pj1CqwydSo8VDtcO99dyISg5lZ78=; b=onbrrTqt0MXYoWbezmsgnNppV2Yj+8/6A9hhgi1dUsyfhTTU8NX5n4eLng8SVzrVSi IfcssesbY8tZHeCF6lfl/oA1cUnWKC1wnm1dBqcZ7JAmnaJcKbydyCEftfHBoW/d8F4S w2mDtYU75AIZKT8YfeeyCKtuPy4k66hvtx7PZPKafdsHUEnFgAdbhfGB0EKTtYe7My/a nnZL/2bKiVo8ah1wd6HeiM/s2NCvFiEYiDxwdowbrScE3EhPmJuZd0JJAPKv8sXeMO4U F274/J9YKlRUytC1XyO3uwm4yymnJBncmd1irojp0W5y7xLBZ4N2tPXn8Fp0Ih1JZ+UK /x4A== X-Gm-Message-State: APjAAAVI/wCAwD8eiNt1TWpXhUBoIfpIczQvh7G0iGBC+dc4vh8k0B1j tXFDNGKWFa7wTbeccXECzhRp/DNk9fg= X-Google-Smtp-Source: APXvYqyFs6vEASZJ4JpVP2Bvx8WSBbRRLaeGe3HvJc8fbHwyRwHTZDo0iN4U5fggdmngUR2gcgB2Cg== X-Received: by 2002:a17:902:d70b:: with SMTP id w11mr2463410ply.313.1568787489265; Tue, 17 Sep 2019 23:18:09 -0700 (PDT) Received: from localhost (ppp167-251-205.static.internode.on.net. [59.167.251.205]) by smtp.gmail.com with ESMTPSA id l7sm10694441pga.92.2019.09.17.23.18.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Sep 2019 23:18:08 -0700 (PDT) From: Daniel Axtens To: patchwork@lists.ozlabs.org Subject: [PATCH 1/6] pyenv: also install requirements for python2 Date: Wed, 18 Sep 2019 16:17:26 +1000 Message-Id: <20190918061731.19142-2-dja@axtens.net> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190918061731.19142-1-dja@axtens.net> References: <20190918061731.19142-1-dja@axtens.net> MIME-Version: 1.0 X-BeenThere: patchwork@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Patchwork development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: patchwork-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Patchwork" The first time you do a migration with python3, you get a whole lot of seemingly null changes. This is a bit annoying so I use py2 to generate the changes. To do that, first fix the pyenv transition so requirements are still installed for python2. Signed-off-by: Daniel Axtens --- tools/docker/Dockerfile | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile index 35324b13c66a..78a97ffef9eb 100644 --- a/tools/docker/Dockerfile +++ b/tools/docker/Dockerfile @@ -48,12 +48,21 @@ RUN pyenv latest install 2.7 && \ pyenv latest install 3.6 && \ pyenv latest install 3.7 -RUN pyenv global $(pyenv versions --bare | tac) +# old python 2.7 +RUN pyenv global $(pyenv versions --bare) COPY requirements-*.txt /tmp/ RUN pip install tox tox-pyenv && \ pip install -r /tmp/requirements-dev.txt +# most recent python +RUN pyenv global $(pyenv versions --bare | tac) + +RUN pip install tox tox-pyenv && \ + pip install -r /tmp/requirements-dev.txt + + + # we deliberately leave the requirements files in tmp so we can # ping the user in entrypoint.sh if the change them! From patchwork Wed Sep 18 06:17:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Axtens X-Patchwork-Id: 1163757 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46Y8tz08lvz9sCJ for ; Wed, 18 Sep 2019 16:18:51 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=axtens.net Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="BFE++cp4"; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 46Y8ty50n6zF3Pn for ; Wed, 18 Sep 2019 16:18:50 +1000 (AEST) X-Original-To: patchwork@lists.ozlabs.org Delivered-To: patchwork@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=axtens.net (client-ip=2607:f8b0:4864:20::534; helo=mail-pg1-x534.google.com; envelope-from=dja@axtens.net; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=axtens.net Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="BFE++cp4"; dkim-atps=neutral Received: from mail-pg1-x534.google.com (mail-pg1-x534.google.com [IPv6:2607:f8b0:4864:20::534]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 46Y8tZ69xbzF2Nh for ; Wed, 18 Sep 2019 16:18:30 +1000 (AEST) Received: by mail-pg1-x534.google.com with SMTP id n9so3391078pgc.1 for ; Tue, 17 Sep 2019 23:18:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axtens.net; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gjcJDQ0JGHCbHMGCe+4LTbRjKqvA5IE21bSjQI8P5L0=; b=BFE++cp46n3/zkgBH+yWpEv/a3H0tmX0zrjwQkhybqhd2tUz5oqd1rQTNFnxNdrsI7 aWJknMFN3TBsMRCYRa5c932JOPnHBO68eDf0l3c4wJrxUu51NbFX+BJBK0cVI9BbQ5fQ LO2YOw9Q6bwsc46AVCkafi82oFrOGTM5gDg/k= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gjcJDQ0JGHCbHMGCe+4LTbRjKqvA5IE21bSjQI8P5L0=; b=ojYQ2UUPlgi6LLfQRLlLCjLgzuN3KEBTUuwK6dBsPcCzMXmKtqAy6OjLnsj9JJ+9GE maV0n4zyIRunjn6B43znfL2m0xcaNZOIQFgIPQO/GW9YRqpcr+9vsXPQ8W0aIaZuQ3WQ 4DpB7jHV3SOAml3zkUrn7kQnZhf2dSh7d6Ok0IXInpnT0Uv29jOny8thZ3TMQxDe1AM8 xDrEs3xxcedDlT/X2daUIYw9eipzi+yqr8BxnS+4GAL0I8Y/5AIfURd+JJ0kOqAV7V+W paUGrrvxAFvsmeTO2GZwgl8R1qsPQhPCZX5TkyXNQ0NePioTM/iFIwuHcPBI4rVu7Pbi zDqA== X-Gm-Message-State: APjAAAVDTa0lphO97rRkcwWQky/QU+A5uwamBLOmR3Gszcw+ClqpfkZ6 C3GgEO7qxS385TBLwNi/R96vtO3CHW8= X-Google-Smtp-Source: APXvYqwXPurW2M7R019jmNDRMvh8Tf1+3OI5tClKx0td+9HFoMQnwajHy/hobbYpJEqmAS212unXkQ== X-Received: by 2002:a63:d343:: with SMTP id u3mr2447145pgi.313.1568787507026; Tue, 17 Sep 2019 23:18:27 -0700 (PDT) Received: from localhost (ppp167-251-205.static.internode.on.net. [59.167.251.205]) by smtp.gmail.com with ESMTPSA id d69sm3868714pfd.175.2019.09.17.23.18.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Sep 2019 23:18:26 -0700 (PDT) From: Daniel Axtens To: patchwork@lists.ozlabs.org Subject: [PATCH 2/6] tests: make test suite pass with XML-RPC disabled Date: Wed, 18 Sep 2019 16:17:27 +1000 Message-Id: <20190918061731.19142-3-dja@axtens.net> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190918061731.19142-1-dja@axtens.net> References: <20190918061731.19142-1-dja@axtens.net> MIME-Version: 1.0 X-BeenThere: patchwork@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Patchwork development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: patchwork-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Patchwork" Handy for development purposes. Signed-off-by: Daniel Axtens Reviewed-by: Stephen Finucane --- patchwork/tests/test_about.py | 24 ++++++++++++++++++------ patchwork/tests/test_xmlrpc.py | 3 +++ 2 files changed, 21 insertions(+), 6 deletions(-) diff --git a/patchwork/tests/test_about.py b/patchwork/tests/test_about.py index 99892cb0539a..d8c35b9f3e5f 100644 --- a/patchwork/tests/test_about.py +++ b/patchwork/tests/test_about.py @@ -3,19 +3,31 @@ # # SPDX-License-Identifier: GPL-2.0-or-later +import unittest + +from django.conf import settings from django.test import TestCase from django.urls import reverse class AboutViewTest(TestCase): - def test_redirects(self): - for view in ['help', 'help-about', 'help-pwclient']: - requested_url = reverse(view) - redirect_url = reverse('about') + def _test_redirect(self, view): + requested_url = reverse(view) + redirect_url = reverse('about') - response = self.client.get(requested_url) - self.assertRedirects(response, redirect_url, 301) + response = self.client.get(requested_url) + self.assertRedirects(response, redirect_url, 301) + + def test_redirects(self): + for view in ['help', 'help-about']: + self._test_redirect(view) + + @unittest.skipUnless(settings.ENABLE_XMLRPC, + 'requires xmlrpc interface (use the ENABLE_XMLRPC ' + 'setting)') + def test_redirects_xmlrpc(self): + self._test_redirect('help-pwclient') def test_xmlrpc(self): with self.settings(ENABLE_XMLRPC=False): diff --git a/patchwork/tests/test_xmlrpc.py b/patchwork/tests/test_xmlrpc.py index 461dfe638fa1..79c6c848a0c2 100644 --- a/patchwork/tests/test_xmlrpc.py +++ b/patchwork/tests/test_xmlrpc.py @@ -42,6 +42,9 @@ class XMLRPCGenericTest(XMLRPCTest): self.rpc.patch_set(0, {}) +@unittest.skipUnless(settings.ENABLE_XMLRPC, + 'requires xmlrpc interface (use the ENABLE_XMLRPC ' + 'setting)') class XMLRPCAuthenticatedTest(LiveServerTestCase): def setUp(self): From patchwork Wed Sep 18 06:17:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Axtens X-Patchwork-Id: 1163758 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46Y8vV1BTSz9sCJ for ; Wed, 18 Sep 2019 16:19:18 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=axtens.net Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="dJBoE8Dk"; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 46Y8vT4KNszF3Xb for ; Wed, 18 Sep 2019 16:19:17 +1000 (AEST) X-Original-To: patchwork@lists.ozlabs.org Delivered-To: patchwork@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=axtens.net (client-ip=2607:f8b0:4864:20::441; helo=mail-pf1-x441.google.com; envelope-from=dja@axtens.net; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=axtens.net Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="dJBoE8Dk"; dkim-atps=neutral Received: from mail-pf1-x441.google.com (mail-pf1-x441.google.com [IPv6:2607:f8b0:4864:20::441]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 46Y8tt6k7PzF1Qy for ; Wed, 18 Sep 2019 16:18:46 +1000 (AEST) Received: by mail-pf1-x441.google.com with SMTP id y5so3665706pfo.4 for ; Tue, 17 Sep 2019 23:18:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axtens.net; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=q5trydgVI87zPDO24nwaA8pb1styFOfbn9am51fdXpE=; b=dJBoE8Dk1I1Ep/CHUYvwgyOTfME0bO3Nk/DyG2U83wh3umLhMxY0WXKiKC6c4CvbXC uFJhzTaHvDrR+U6eTQpsnpQRL5cUahNUPVx6UDIuLH3Lg/4AhhBuYIpNenWtpvrc5vGW d1HwziPcO+94Z1h12PLoXbrA32AU82gOpJ/as= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=q5trydgVI87zPDO24nwaA8pb1styFOfbn9am51fdXpE=; b=d1b7OVfX6Rr60s2UBPyOJEQqKrbExx9sY0PSXFkfdcnubN7gxvdIsBHrU8BPb0tZmX 6iD7j1b4pbC7vWlEnM43SLX5nNMkhMGJAeoNpL+s90zy9mzn3LEytsKab1s0o28FfQsm +3Wy5/Dm0NZBtF+RzJEb9GuKAXrhCCREYuEAtYIRBlCdmDJ808gN98x5KYLggEiBniYn jSRrNN1eGyDlfRii96MwWLqdN87EOD8YKmQ4lIH/GZpHVSF1lON4GJeanD0YQ6x5/aAj 4MpEz1NsoeroCYNdhPJybqXnXa4jSKbfoNhHNUztu8Ag2wuGlcWzHjJ1q4gNBFqKjL4G 3LNA== X-Gm-Message-State: APjAAAVmCUpVIgIVZ9w8XauSuCJUpjG6IHRI7ZnJrS7dAGBeEgN3ziGS DVivOzWMBiZ0ReNemwcMOy/kgNOkid4= X-Google-Smtp-Source: APXvYqwZ16eKciFExcGbBNSGYt/GlV6jEPXi4QW4m5qAfzxWO7SjZd0B1necG/U631NnrOlpT41cpA== X-Received: by 2002:a17:90a:fa01:: with SMTP id cm1mr1213365pjb.140.1568787523762; Tue, 17 Sep 2019 23:18:43 -0700 (PDT) Received: from localhost (ppp167-251-205.static.internode.on.net. [59.167.251.205]) by smtp.gmail.com with ESMTPSA id n9sm4494516pgf.64.2019.09.17.23.18.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Sep 2019 23:18:43 -0700 (PDT) From: Daniel Axtens To: patchwork@lists.ozlabs.org Subject: [PATCH 3/6] parsearchive, mail: use repr() to get a human readable exception Date: Wed, 18 Sep 2019 16:17:28 +1000 Message-Id: <20190918061731.19142-4-dja@axtens.net> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190918061731.19142-1-dja@axtens.net> References: <20190918061731.19142-1-dja@axtens.net> MIME-Version: 1.0 X-BeenThere: patchwork@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Patchwork development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: patchwork-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Patchwork" Currently if we have particular types of error in mail parsing in parsearchive or parsemail, we print exc.message, which doesn't always work: Traceback (most recent call last): File "/home/patchwork/patchwork/patchwork/management/commands/parsearchive.py", line 90, in handle obj = parse_mail(msg, options['list_id']) File "/home/patchwork/patchwork/patchwork/parser.py", line 961, in parse_mail raise ValueError("Missing 'Message-Id' header") ValueError: Missing 'Message-Id' header During handling of the above exception, another exception occurred: Traceback (most recent call last): File "manage.py", line 11, in execute_from_command_line(sys.argv) File "/opt/pyenv/versions/3.7.4/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line utility.execute() File "/opt/pyenv/versions/3.7.4/lib/python3.7/site-packages/django/core/management/__init__.py", line 375, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/opt/pyenv/versions/3.7.4/lib/python3.7/site-packages/django/core/management/base.py", line 323, in run_from_argv self.execute(*args, **cmd_options) File "/opt/pyenv/versions/3.7.4/lib/python3.7/site-packages/django/core/management/base.py", line 364, in execute output = self.handle(*args, **options) File "/home/patchwork/patchwork/patchwork/management/commands/parsearchive.py", line 100, in handle logger.warning('Invalid mail: %s', exc.message) AttributeError: 'ValueError' object has no attribute 'message' repr(exc) will work. Use it. Signed-off-by: Daniel Axtens Reviewed-by: Stephen Finucane --- patchwork/management/commands/parsearchive.py | 2 +- patchwork/management/commands/parsemail.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/patchwork/management/commands/parsearchive.py b/patchwork/management/commands/parsearchive.py index 37db2b0dc007..b7f1ea7313c2 100644 --- a/patchwork/management/commands/parsearchive.py +++ b/patchwork/management/commands/parsearchive.py @@ -97,7 +97,7 @@ class Command(BaseCommand): logger.warning('Duplicate mail for message ID %s', exc.msgid) except (ValueError, Exception) as exc: errors += 1 - logger.warning('Invalid mail: %s', exc.message) + logger.warning('Invalid mail: %s', repr(exc)) if verbosity < 3 and (i % 10) == 0: self.stdout.write('%06d/%06d\r' % (i, count), ending='') diff --git a/patchwork/management/commands/parsemail.py b/patchwork/management/commands/parsemail.py index b0871d211ad0..ea85e3aabb5c 100644 --- a/patchwork/management/commands/parsemail.py +++ b/patchwork/management/commands/parsemail.py @@ -70,6 +70,6 @@ class Command(base.BaseCommand): logger.warning('Duplicate mail for message ID %s', exc.msgid) except (ValueError, Exception) as exc: logger.exception('Error when parsing incoming email: %s', - exc.message, + repr(exc), extra={'mail': mail.as_string()}) sys.exit(1) From patchwork Wed Sep 18 06:17:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Axtens X-Patchwork-Id: 1163765 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46Y8wF26WKz9sN1 for ; Wed, 18 Sep 2019 16:19:57 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=axtens.net Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="QBdNhZXI"; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 46Y8wF0fS1zF4CP for ; Wed, 18 Sep 2019 16:19:57 +1000 (AEST) X-Original-To: patchwork@lists.ozlabs.org Delivered-To: patchwork@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=axtens.net (client-ip=2607:f8b0:4864:20::42d; helo=mail-pf1-x42d.google.com; envelope-from=dja@axtens.net; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=axtens.net Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="QBdNhZXI"; dkim-atps=neutral Received: from mail-pf1-x42d.google.com (mail-pf1-x42d.google.com [IPv6:2607:f8b0:4864:20::42d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 46Y8vG1MxgzF3ZP for ; Wed, 18 Sep 2019 16:19:06 +1000 (AEST) Received: by mail-pf1-x42d.google.com with SMTP id 205so3675772pfw.2 for ; Tue, 17 Sep 2019 23:19:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axtens.net; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yHvrENbeRUjzAIKTZdN6C5v92ubzb+R1dYW4XIbjU9U=; b=QBdNhZXI7PspCXJhFMIqLxPWJfHugYAFCMSsSP9nsYfxQGCu3YnqgAXlTrBNixlA/3 LaqMv5+PiwEBgtFp+pYKRqQF3Zr9Zf+oVPz3qDw1Zwr6MNN/AJ93NvX+9Z1i4jm8iPnh IMUqWjlY4SfkmIJU8EUnVNc5QgS0Zc6XnE7Jw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yHvrENbeRUjzAIKTZdN6C5v92ubzb+R1dYW4XIbjU9U=; b=HNnODlhoyOwcDwc952NpBU/4RO90hA/zDb9YtEviS04kxM7sPyQgzbQjFk2J7sSWlv fdV3Dxr0QRSc6umSaNUDV6hCej0i5+JcSICXqh03QZ60UXziPaR2dRbh3N24/ONbtC1H MSRtAk/o6gU8ZFJfDnprg/DUsZFOvpsz+I00BE7PKBKE9TvOcykbpVGEra8r3qZ12L/3 6qTKKME+ZAazYLLIBQpfyJ5FzQi+c3AWEBLkstYH+rSdB/yx9mpnPNtTtIQ+7mExn7Sn 0e/cIHhm42kMxTbltpuS/F4GE7S9iieWGvrpn0upJrbbxpzrxW7iTTyE6BuXT6UYKBM8 HX8w== X-Gm-Message-State: APjAAAXcb7ayE534AizieVt3bGsC8rPxwFrtPcb+Jlv1/VGHz59hjLmS 1zlszXNFCeA7MQqGG8FuOnVR0tkINbM= X-Google-Smtp-Source: APXvYqwfBb6LOTM4ij5isBa8BIqvFW9ldZGtZ7nZS5idQ8Oarsvd3yQ2i9svKHC9IcKgIzyQXwlv8g== X-Received: by 2002:a63:4902:: with SMTP id w2mr2476454pga.28.1568787542380; Tue, 17 Sep 2019 23:19:02 -0700 (PDT) Received: from localhost (ppp167-251-205.static.internode.on.net. [59.167.251.205]) by smtp.gmail.com with ESMTPSA id e6sm7654482pfl.146.2019.09.17.23.18.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Sep 2019 23:19:01 -0700 (PDT) From: Daniel Axtens To: patchwork@lists.ozlabs.org Subject: [PATCH 4/6] Begin live migration of diff and pull_url to Submission Date: Wed, 18 Sep 2019 16:17:29 +1000 Message-Id: <20190918061731.19142-5-dja@axtens.net> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190918061731.19142-1-dja@axtens.net> References: <20190918061731.19142-1-dja@axtens.net> MIME-Version: 1.0 X-BeenThere: patchwork@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Patchwork development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: patchwork-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Patchwork" This patch contains 2 migrations: 0037_prepare_old_diff_pull_url: - renames Patch's 'diff' to 'old_diff', while not affecting the db - likewise 'pull_url' to 'old_pull_url' This does no SQL changes, which you can verify with python manage.py sqlmigrate patchwork 0037 0038_submission_new_diff_pull_url: - creates a diff column in Submission, accessible via new_diff - likewise pull_url as new_pull_url livemigrate-v22-v23: (the version numbers are a bit arbitrary) - copies data from old_{pull_url,diff} to new_{pull_url,diff} as required. Can be run on a _running instance_ of patchwork. The rest of the code provides a bunch of pretty ugly glue code. It assumes both migrations have been run, and then allows pw to: - read old data from old_{pull_url,diff} - store new data in both the old and new fields The idea is that you should be able to: - bring down pw - apply this - do two fast migrations (one with no SQL, one that adds two columns) - restart patchwork, restart ingesting mail - do the live migration over time - proceed to the next patch By storing the data in duplicate, the hope is that you can back out the new code at any point and still have all of your data. Signed-off-by: Daniel Axtens --- patchwork/api/patch.py | 10 +- .../commands/livemigrate-v22-v23.py | 35 +++++++ .../0037_prepare_old_diff_pull_url.py | 35 +++++++ .../0038_submission_new_diff_pull_url.py | 25 +++++ patchwork/models.py | 93 ++++++++++++++++++- patchwork/views/xmlrpc.py | 5 +- 6 files changed, 197 insertions(+), 6 deletions(-) create mode 100644 patchwork/management/commands/livemigrate-v22-v23.py create mode 100644 patchwork/migrations/0037_prepare_old_diff_pull_url.py create mode 100644 patchwork/migrations/0038_submission_new_diff_pull_url.py diff --git a/patchwork/api/patch.py b/patchwork/api/patch.py index c9360308a56a..3866f92fe3dc 100644 --- a/patchwork/api/patch.py +++ b/patchwork/api/patch.py @@ -181,11 +181,15 @@ class PatchList(ListAPIView): ordering = 'id' def get_queryset(self): - return Patch.objects.all()\ + qs = Patch.objects.all()\ .prefetch_related('check_set')\ .select_related('project', 'state', 'submitter', 'delegate', - 'series')\ - .defer('content', 'diff', 'headers') + 'series') + if hasattr(Patch, 'old_diff'): + qs.defer('content', 'old_diff', 'headers') + else: + qs.defer('content', 'submission_ptr__new_diff', 'headers') + return qs class PatchDetail(RetrieveUpdateAPIView): diff --git a/patchwork/management/commands/livemigrate-v22-v23.py b/patchwork/management/commands/livemigrate-v22-v23.py new file mode 100644 index 000000000000..1079fb2f3997 --- /dev/null +++ b/patchwork/management/commands/livemigrate-v22-v23.py @@ -0,0 +1,35 @@ +# Patchwork - automated patch tracking system +# Copyright (C) 2019 IBM Corporation +# Author: Daniel Axtens +# +# SPDX-License-Identifier: GPL-2.0-or-later + +from django.core.management.base import BaseCommand +from django.db.models import Q + +from patchwork.models import Patch +from patchwork.models import Submission + + +class Command(BaseCommand): + help = 'Do the live migration for flattening patches' + + def handle(self, *args, **options): + if not hasattr(Submission, 'new_diff'): + print("Submission model is missing 'new_diff'. Have you applied the migration to add it?") + return + + diffs = Q(old_diff__isnull=False, submission_ptr__new_diff__isnull=True) + pull_urls = Q(old_pull_url__isnull=False, submission_ptr__new_pull_url__isnull=True) + query = Patch.objects.filter(diffs | pull_urls) + + count = query.count() + + for i, patch in enumerate(query.iterator()): + patch.new_diff = patch.old_diff + patch.new_pull_url = patch.old_pull_url + patch.save() + if (i % 10) == 0: + self.stdout.write('%06d/%06d\r' % (i, count), ending='') + self.stdout.flush() + self.stdout.write('\ndone') diff --git a/patchwork/migrations/0037_prepare_old_diff_pull_url.py b/patchwork/migrations/0037_prepare_old_diff_pull_url.py new file mode 100644 index 000000000000..83fd1beaac9c --- /dev/null +++ b/patchwork/migrations/0037_prepare_old_diff_pull_url.py @@ -0,0 +1,35 @@ +# -*- coding: utf-8 -*- +# Generated by Django 1.11.24 on 2019-09-18 01:49 +from __future__ import unicode_literals + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('patchwork', '0036_project_commit_url_format'), + ] + + operations = [ + migrations.AlterField( + model_name='patch', + name='diff', + field=models.TextField(blank=True, db_column='diff', null=True), + ), + migrations.RenameField( + model_name='patch', + old_name='diff', + new_name='old_diff', + ), + migrations.AlterField( + model_name='patch', + name='pull_url', + field=models.CharField(blank=True, db_column='pull_url', max_length=255, null=True), + ), + migrations.RenameField( + model_name='patch', + old_name='pull_url', + new_name='old_pull_url', + ), + ] diff --git a/patchwork/migrations/0038_submission_new_diff_pull_url.py b/patchwork/migrations/0038_submission_new_diff_pull_url.py new file mode 100644 index 000000000000..094854d863cb --- /dev/null +++ b/patchwork/migrations/0038_submission_new_diff_pull_url.py @@ -0,0 +1,25 @@ +# -*- coding: utf-8 -*- +# Generated by Django 1.11.24 on 2019-09-18 01:54 +from __future__ import unicode_literals + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('patchwork', '0037_prepare_old_diff_pull_url'), + ] + + operations = [ + migrations.AddField( + model_name='submission', + name='new_diff', + field=models.TextField(blank=True, db_column='diff', null=True), + ), + migrations.AddField( + model_name='submission', + name='new_pull_url', + field=models.CharField(blank=True, db_column='pull_url', max_length=255, null=True), + ), + ] diff --git a/patchwork/models.py b/patchwork/models.py index 32d1b3c2adc5..31f459e1a126 100644 --- a/patchwork/models.py +++ b/patchwork/models.py @@ -367,6 +367,11 @@ class Submission(FilenameMixin, EmailMixin, models.Model): name = models.CharField(max_length=255) + # patch + new_diff = models.TextField(null=True, blank=True, db_column='diff') + new_pull_url = models.CharField(max_length=255, null=True, blank=True, db_column='pull_url') + + @property def list_archive_url(self): if not self.project.list_archive_url_format: @@ -410,9 +415,9 @@ class CoverLetter(Submission): class Patch(Submission): # patch metadata - diff = models.TextField(null=True, blank=True) + old_diff = models.TextField(null=True, blank=True, db_column='diff') commit_ref = models.CharField(max_length=255, null=True, blank=True) - pull_url = models.CharField(max_length=255, null=True, blank=True) + old_pull_url = models.CharField(max_length=255, null=True, blank=True, db_column='pull_url') tags = models.ManyToManyField(Tag, through=PatchTag) # patchwork metadata @@ -479,6 +484,20 @@ class Patch(Submission): super(Patch, self).save(**kwargs) + # we don't have to worry about duplicating to the + # old fields: the setters do this for us. + if hasattr(self, '_diff'): + if hasattr(self.submission_ptr, 'new_diff'): + self.new_diff = self._diff + self.submission_ptr.save() + del self._diff + + if hasattr(self, '_pull_url'): + if hasattr(self.submission_ptr, 'new_pull_url'): + self.new_pull_url = self._diff + self.submission_ptr.save() + del self._pull_url + self.refresh_tag_counts() def is_editable(self, user): @@ -602,6 +621,76 @@ class Patch(Submission): ] + # transition goop + @property + def diff(self): + # three options: + # it's sitting around in our cached storage + # new field is not null + # old has field + # no point in null checking old + if hasattr(self, '_diff'): + return self._diff + try: + if self.submission_ptr.new_diff is not None: + return self.submission_ptr.new_diff + except: + pass + if hasattr(self, 'old_diff'): + return self.old_diff + return None + + @diff.setter + def diff(self, value): + try: + self.submission_ptr.diff = value + except: + self._diff = value + if hasattr(self, 'old_diff'): + self.old_diff = value + + @property + def pull_url(self): + if hasattr(self, '_pull_url'): + return self._pull_url + try: + if self.submission_ptr.new_pull_url is not None: + return self.submission_ptr.new_pull_url + except: + pass + if hasattr(self, 'old_pull_url'): + return self.old_pull_url + return None + + @pull_url.setter + def pull_url(self, value): + try: + self.submission_ptr.pull_url = value + except: + self._diff = value + if hasattr(self, 'old_pull_url'): + self.old_pull_url = value + + + def __init__(self, *args, **kwargs): + diff = None + if 'diff' in kwargs: + diff = kwargs['diff'] + del kwargs['diff'] + + pull_url = None + if 'pull_url' in kwargs: + pull_url = kwargs['pull_url'] + del kwargs['pull_url'] + + super(Patch, self).__init__(*args, **kwargs) + + if diff: + self.diff = diff + if pull_url: + self.pull_url = pull_url + + class Comment(EmailMixin, models.Model): # parent diff --git a/patchwork/views/xmlrpc.py b/patchwork/views/xmlrpc.py index f60725044ebe..a2e3d8985dd7 100644 --- a/patchwork/views/xmlrpc.py +++ b/patchwork/views/xmlrpc.py @@ -569,7 +569,10 @@ def patch_list(filt=None): # Only extract the relevant fields. This saves a big db load as we # no longer fetch content/headers/etc for potentially every patch # in a project. - patches = patches.defer('content', 'headers', 'diff') + if hasattr(Patch, 'old_diff'): + patches = patches.defer('content', 'headers', 'old_diff') + else: + patches = patches.defer('content', 'headers', 'submission_ptr__new_diff') return _get_objects(patch_to_dict, patches, max_count) From patchwork Wed Sep 18 06:17:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Axtens X-Patchwork-Id: 1163766 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46Y8x92GL5z9sQw for ; Wed, 18 Sep 2019 16:20:45 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=axtens.net Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="GsDwJCMd"; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 46Y8x83TWVzF3bV for ; Wed, 18 Sep 2019 16:20:44 +1000 (AEST) X-Original-To: patchwork@lists.ozlabs.org Delivered-To: patchwork@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=axtens.net (client-ip=2607:f8b0:4864:20::630; helo=mail-pl1-x630.google.com; envelope-from=dja@axtens.net; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=axtens.net Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="GsDwJCMd"; dkim-atps=neutral Received: from mail-pl1-x630.google.com (mail-pl1-x630.google.com [IPv6:2607:f8b0:4864:20::630]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 46Y8vZ4fPTzF4B8 for ; Wed, 18 Sep 2019 16:19:22 +1000 (AEST) Received: by mail-pl1-x630.google.com with SMTP id f19so2626516plr.3 for ; Tue, 17 Sep 2019 23:19:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axtens.net; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8wSu4h2AkZ4bfTJnntVe5L1OaUgBJc5d2oEHfSq4kFk=; b=GsDwJCMdfgj4ovb8O4q5geYzR3wCCtW8ERJqyRyv57uw0ErmII2XZkq5gES3wv/ZSj NRTD7YZRvMsE4WxoCW8SmXINc/ZqHLb/hMNd6E4/HCs4n7RgHMp0NA3dK5p9+xwEykGe yAb/anLAV7D1uSdWSHsvz63m8yeQLPLP44m4g= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8wSu4h2AkZ4bfTJnntVe5L1OaUgBJc5d2oEHfSq4kFk=; b=orP7mn1eM0I3Lgj/1G1U+34ydYNsztIhLXd7ovg1tO3fMSOOONKCNJMiS3v91DmbUr 5+uRNF/5WWCCWgN3dMQ1bC8Wc1Z0owu9TlPkQ9neAxbAMG1wjpuzwhaDkjr96FydlJmW KFlZJf/odbnOCWPY/qJ7Na2xUbJsN4qhbQTHsdkBE5dEWEIR3taBa5kSCy0AQjWg3q8z AuaNkGaePNKdhLwWidcx09bAgMabiTmm0wt9J47btTOWfnKdXH1rep9FIk5VMbinKgjx Lw5y1Sn63CyXxvair4T1RVPOHr4L+tdz7vBG2UaBiZkGWPRQNx4Orb2qGFhT1SeHzX91 LVwA== X-Gm-Message-State: APjAAAXxwa+5k8zUz8G9q691EFz1NPtUl9mrZiZMlyiW2wHpG3DOte/R klNO7gF4rwKP1SMzDGWElrGUs/s1r9k= X-Google-Smtp-Source: APXvYqz+912aDt5lx+uQOZiBbU8zFsn67sijfweaT5ZUEKzuNMeHKr6vRnrscBHdRpVHgOTmgZHnRg== X-Received: by 2002:a17:902:7003:: with SMTP id y3mr2539560plk.239.1568787559662; Tue, 17 Sep 2019 23:19:19 -0700 (PDT) Received: from localhost (ppp167-251-205.static.internode.on.net. [59.167.251.205]) by smtp.gmail.com with ESMTPSA id c11sm8875988pfj.114.2019.09.17.23.19.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Sep 2019 23:19:18 -0700 (PDT) From: Daniel Axtens To: patchwork@lists.ozlabs.org Subject: [PATCH 5/6] Migration part 2: drop old fields Date: Wed, 18 Sep 2019 16:17:30 +1000 Message-Id: <20190918061731.19142-6-dja@axtens.net> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190918061731.19142-1-dja@axtens.net> References: <20190918061731.19142-1-dja@axtens.net> MIME-Version: 1.0 X-BeenThere: patchwork@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Patchwork development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: patchwork-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Patchwork" Once you have run the live migration script, you can drop the old fields. There is a sanity check migration as well, to make sure that you don't drop the tables while you have unmigrated data. Once you have applied this, you can move on to applying the final stuff about dropping the cover letter table. This doesn't clean up the transition code, I figure we can do that once we have all the fields migrated and can kill off the Patch model entirely. Signed-off-by: Daniel Axtens --- patchwork/migrations/0039_sanity_check.py | 27 +++++++++++++++++++ .../migrations/0040_drop_old_diff_pull_url.py | 23 ++++++++++++++++ patchwork/models.py | 2 -- 3 files changed, 50 insertions(+), 2 deletions(-) create mode 100644 patchwork/migrations/0039_sanity_check.py create mode 100644 patchwork/migrations/0040_drop_old_diff_pull_url.py diff --git a/patchwork/migrations/0039_sanity_check.py b/patchwork/migrations/0039_sanity_check.py new file mode 100644 index 000000000000..e26b338f90b3 --- /dev/null +++ b/patchwork/migrations/0039_sanity_check.py @@ -0,0 +1,27 @@ +# -*- coding: utf-8 -*- +# Generated by Django 1.11.24 on 2019-09-18 10:34 +from __future__ import unicode_literals + +from django.db import migrations +from django.db.models import Q + +def sanity_check(apps, schema_editor): + Patch = apps.get_model("patchwork", "Patch") + db_alias = schema_editor.connection.alias + diffs = Q(old_diff__isnull=False, submission_ptr__new_diff__isnull=True) + pull_urls = Q(old_pull_url__isnull=False, submission_ptr__new_pull_url__isnull=True) + query = Patch.objects.using(db_alias).filter(diffs | pull_urls) + if query.exists(): + raise Exception("It seems not all data has been migrated! bailing out") + +class Migration(migrations.Migration): + + dependencies = [ + ('patchwork', '0038_submission_new_diff_pull_url'), + ] + + operations = [ + migrations.RunPython( + sanity_check, migrations.RunPython.noop, + ), + ] diff --git a/patchwork/migrations/0040_drop_old_diff_pull_url.py b/patchwork/migrations/0040_drop_old_diff_pull_url.py new file mode 100644 index 000000000000..088544b09d31 --- /dev/null +++ b/patchwork/migrations/0040_drop_old_diff_pull_url.py @@ -0,0 +1,23 @@ +# -*- coding: utf-8 -*- +# Generated by Django 1.11.24 on 2019-09-18 10:34 +from __future__ import unicode_literals + +from django.db import migrations + + +class Migration(migrations.Migration): + + dependencies = [ + ('patchwork', '0039_sanity_check'), + ] + + operations = [ + migrations.RemoveField( + model_name='patch', + name='old_diff', + ), + migrations.RemoveField( + model_name='patch', + name='old_pull_url', + ), + ] diff --git a/patchwork/models.py b/patchwork/models.py index 31f459e1a126..a96018004f75 100644 --- a/patchwork/models.py +++ b/patchwork/models.py @@ -415,9 +415,7 @@ class CoverLetter(Submission): class Patch(Submission): # patch metadata - old_diff = models.TextField(null=True, blank=True, db_column='diff') commit_ref = models.CharField(max_length=255, null=True, blank=True) - old_pull_url = models.CharField(max_length=255, null=True, blank=True, db_column='pull_url') tags = models.ManyToManyField(Tag, through=PatchTag) # patchwork metadata From patchwork Wed Sep 18 06:17:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Axtens X-Patchwork-Id: 1163767 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46Y8yJ352tz9sNw for ; Wed, 18 Sep 2019 16:21:44 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=axtens.net Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="Jv0cQqvd"; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 46Y8yJ1T9jzF4TM for ; Wed, 18 Sep 2019 16:21:44 +1000 (AEST) X-Original-To: patchwork@lists.ozlabs.org Delivered-To: patchwork@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=axtens.net (client-ip=2607:f8b0:4864:20::431; helo=mail-pf1-x431.google.com; envelope-from=dja@axtens.net; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=axtens.net Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="Jv0cQqvd"; dkim-atps=neutral Received: from mail-pf1-x431.google.com (mail-pf1-x431.google.com [IPv6:2607:f8b0:4864:20::431]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 46Y8vx361rzF4Cp for ; Wed, 18 Sep 2019 16:19:41 +1000 (AEST) Received: by mail-pf1-x431.google.com with SMTP id 205so3676686pfw.2 for ; Tue, 17 Sep 2019 23:19:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axtens.net; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=N/ZOdYdpvZxK4n8AU3Dh4Gvk15+h4IeM3wBb3FDfaec=; b=Jv0cQqvdvSRgextdRxDwo/w94x+kuz2bGVpGqqwqtD//BvsqE6tzb/5s3MRHCDMUEL J4ZEAt7qFbryjitmLdAlZ2ePLSorwhe8LsoqeU3cAKVaAj1WV+sKtNlRJN2AuHX/ftze p7Eix/PFJw6GfMiBooPdhA7jPAj/UNNgJoqhg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=N/ZOdYdpvZxK4n8AU3Dh4Gvk15+h4IeM3wBb3FDfaec=; b=o8F4lIkceAG52hNOldEp7W3XysbMonZDPF6yeImwk0PM8BT1RNVufO/iByUuuXfs5+ SbdWMZ1onpbUgyNJ2hm51ztRhHO7/ZgxMh7dgbk9EJECI11l7mEnVHNaG7iLVMSPsZFb bGJeSyp/vwF7I/2Hn05B1hlcKj4/+r6bQkzxKcWa0pNHFkeHddvGeYWLXQ1RtzhHBS7j lpJsBFigE6xkkA7l31tWwG98bT2TRXdd2jg4u3mzwikritHGyaMUFdav8NqXH6NgY0i3 hlrSRAZRhgPbicV1JIkfC1AxOKK+j18L7X7mI5cfmbBmgHM2I5i0KWpRq/YU0EiwLCe4 cQGQ== X-Gm-Message-State: APjAAAV+CfyB09LZK5z+19p3R6Ubq2IAXOM/MslRbY9fD4byjx53iCc4 7/N9g6FYWCJPx5WC0DJDqf3NuklPSrU= X-Google-Smtp-Source: APXvYqxJ8qXqu6FAAjahvVlHbVzwZ1eOC77+kDXSrE8Zkjmol1aIO7Pl+LMi+7C3q/3thZRyY+VciQ== X-Received: by 2002:a17:90a:3acf:: with SMTP id b73mr2032163pjc.88.1568787577515; Tue, 17 Sep 2019 23:19:37 -0700 (PDT) Received: from localhost (ppp167-251-205.static.internode.on.net. [59.167.251.205]) by smtp.gmail.com with ESMTPSA id bx18sm1063326pjb.26.2019.09.17.23.19.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Sep 2019 23:19:36 -0700 (PDT) From: Daniel Axtens To: patchwork@lists.ozlabs.org Subject: [PATCH 6/6] Burninate CoverLetter Date: Wed, 18 Sep 2019 16:17:31 +1000 Message-Id: <20190918061731.19142-7-dja@axtens.net> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190918061731.19142-1-dja@axtens.net> References: <20190918061731.19142-1-dja@axtens.net> MIME-Version: 1.0 X-BeenThere: patchwork@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Patchwork development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: patchwork-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Patchwork" A patch must have a diff or a pull_url. If a Submission does not have a diff or a pull_url, it must be a CoverLetter. This is enough to get rid of the CoverLetter model. We do have some work to do around the interactions with Series, which is currently super ugly, but it's good enough for a proof of concept. Signed-off-by: Daniel Axtens --- patchwork/admin.py | 3 -- patchwork/api/cover.py | 16 +++++----- patchwork/api/embedded.py | 2 +- patchwork/api/filters.py | 13 +++++++-- patchwork/management/commands/parsearchive.py | 4 +-- patchwork/migrations/0041_drop_coverletter.py | 29 +++++++++++++++++++ patchwork/models.py | 19 +++++------- patchwork/parser.py | 9 +++--- patchwork/signals.py | 4 +-- patchwork/templates/patchwork/submission.html | 10 +++---- patchwork/tests/api/test_cover.py | 5 ++-- patchwork/tests/test_series.py | 14 ++++----- patchwork/tests/utils.py | 6 ++-- patchwork/views/cover.py | 25 +++++++++++----- patchwork/views/patch.py | 1 + 15 files changed, 100 insertions(+), 60 deletions(-) create mode 100644 patchwork/migrations/0041_drop_coverletter.py diff --git a/patchwork/admin.py b/patchwork/admin.py index f9a94c6f5c07..48fe43b93c99 100644 --- a/patchwork/admin.py +++ b/patchwork/admin.py @@ -11,7 +11,6 @@ from django.db.models import Prefetch from patchwork.models import Bundle from patchwork.models import Check from patchwork.models import Comment -from patchwork.models import CoverLetter from patchwork.models import DelegationRule from patchwork.models import Patch from patchwork.models import Person @@ -83,8 +82,6 @@ class SubmissionAdmin(admin.ModelAdmin): admin.site.register(Submission, SubmissionAdmin) -admin.site.register(CoverLetter, SubmissionAdmin) - class PatchAdmin(admin.ModelAdmin): list_display = ('name', 'submitter', 'project', 'state', 'date', diff --git a/patchwork/api/cover.py b/patchwork/api/cover.py index caf9a386efa5..8eb65f8cb942 100644 --- a/patchwork/api/cover.py +++ b/patchwork/api/cover.py @@ -15,7 +15,7 @@ from patchwork.api.filters import CoverLetterFilterSet from patchwork.api.embedded import PersonSerializer from patchwork.api.embedded import ProjectSerializer from patchwork.api.embedded import SeriesSerializer -from patchwork.models import CoverLetter +from patchwork.models import Submission class CoverLetterListSerializer(BaseHyperlinkedModelSerializer): @@ -24,7 +24,7 @@ class CoverLetterListSerializer(BaseHyperlinkedModelSerializer): project = ProjectSerializer(read_only=True) submitter = PersonSerializer(read_only=True) mbox = SerializerMethodField() - series = SeriesSerializer(read_only=True) + series = SeriesSerializer(source='cl_series', read_only=True) comments = SerializerMethodField() def get_web_url(self, instance): @@ -49,7 +49,7 @@ class CoverLetterListSerializer(BaseHyperlinkedModelSerializer): return data class Meta: - model = CoverLetter + model = Submission fields = ('id', 'url', 'web_url', 'project', 'msgid', 'list_archive_url', 'date', 'name', 'submitter', 'mbox', 'series', 'comments') @@ -82,7 +82,7 @@ class CoverLetterDetailSerializer(CoverLetterListSerializer): return headers class Meta: - model = CoverLetter + model = Submission fields = CoverLetterListSerializer.Meta.fields + ( 'headers', 'content') read_only_fields = fields @@ -100,8 +100,8 @@ class CoverLetterList(ListAPIView): ordering = 'id' def get_queryset(self): - return CoverLetter.objects.all()\ - .select_related('project', 'submitter', 'series')\ + return Submission.objects.filter(new_diff__isnull=True, new_pull_url__isnull=True)\ + .select_related('project', 'submitter', 'cl_series')\ .defer('content', 'headers') @@ -111,5 +111,5 @@ class CoverLetterDetail(RetrieveAPIView): serializer_class = CoverLetterDetailSerializer def get_queryset(self): - return CoverLetter.objects.all()\ - .select_related('project', 'submitter', 'series') + return Submission.objects.filter(new_diff__isnull=True, new_pull_url__isnull=True)\ + .select_related('project', 'submitter', 'cl_series') diff --git a/patchwork/api/embedded.py b/patchwork/api/embedded.py index de4f31165ee7..a0196128bad4 100644 --- a/patchwork/api/embedded.py +++ b/patchwork/api/embedded.py @@ -107,7 +107,7 @@ class CoverLetterSerializer(SerializedRelatedField): class _Serializer(MboxMixin, WebURLMixin, BaseHyperlinkedModelSerializer): class Meta: - model = models.CoverLetter + model = models.Submission fields = ('id', 'url', 'web_url', 'msgid', 'list_archive_url', 'date', 'name', 'mbox') read_only_fields = fields diff --git a/patchwork/api/filters.py b/patchwork/api/filters.py index 37aca82d9ab2..4690fcab7479 100644 --- a/patchwork/api/filters.py +++ b/patchwork/api/filters.py @@ -15,13 +15,13 @@ from django.forms.widgets import MultipleHiddenInput from patchwork.compat import NAME_FIELD from patchwork.models import Bundle from patchwork.models import Check -from patchwork.models import CoverLetter from patchwork.models import Event from patchwork.models import Patch from patchwork.models import Person from patchwork.models import Project from patchwork.models import Series from patchwork.models import State +from patchwork.models import Submission # custom fields, filters @@ -161,8 +161,13 @@ class CoverLetterFilterSet(TimestampMixin, FilterSet): widget=MultipleHiddenInput) submitter = PersonFilter(queryset=Person.objects.all()) + @property + def qs(self): + parent = super(CoverLetterFilterSet, self).qs + return parent.filter(new_diff__isnull=True, new_pull_url__isnull=True) + class Meta: - model = CoverLetter + model = Submission fields = ('project', 'series', 'submitter') @@ -191,6 +196,8 @@ class CheckFilterSet(TimestampMixin, FilterSet): model = Check fields = ('user', 'state', 'context') +def get_coverletter_qs(request): + return Submission.objects.all().filter(new_diff__isnull=True, new_pull_url__isnull=True) class EventFilterSet(TimestampMixin, FilterSet): @@ -203,7 +210,7 @@ class EventFilterSet(TimestampMixin, FilterSet): widget=MultipleHiddenInput) patch = BaseFilter(queryset=Patch.objects.all(), widget=MultipleHiddenInput) - cover = BaseFilter(queryset=CoverLetter.objects.all(), + cover = BaseFilter(queryset=get_coverletter_qs, widget=MultipleHiddenInput) class Meta: diff --git a/patchwork/management/commands/parsearchive.py b/patchwork/management/commands/parsearchive.py index b7f1ea7313c2..39c640ccee8b 100644 --- a/patchwork/management/commands/parsearchive.py +++ b/patchwork/management/commands/parsearchive.py @@ -32,7 +32,7 @@ class Command(BaseCommand): def handle(self, *args, **options): results = { models.Patch: 0, - models.CoverLetter: 0, + models.Submission: 0, models.Comment: 0, } duplicates = 0 @@ -118,7 +118,7 @@ class Command(BaseCommand): ' %(errors)4d errors\n' 'Total: %(new)s new entries' % { 'total': count, - 'covers': results[models.CoverLetter], + 'covers': results[models.Submission], 'patches': results[models.Patch], 'comments': results[models.Comment], 'duplicates': duplicates, diff --git a/patchwork/migrations/0041_drop_coverletter.py b/patchwork/migrations/0041_drop_coverletter.py new file mode 100644 index 000000000000..d504676fd7fc --- /dev/null +++ b/patchwork/migrations/0041_drop_coverletter.py @@ -0,0 +1,29 @@ +# -*- coding: utf-8 -*- +# Generated by Django 1.11.24 on 2019-09-17 17:56 +from __future__ import unicode_literals + +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + dependencies = [ + ('patchwork', '0040_drop_old_diff_pull_url'), + ] + + operations = [ + migrations.AlterField( + model_name='event', + name='cover', + field=models.ForeignKey(blank=True, help_text=b'The cover letter that this event was created for.', null=True, on_delete=django.db.models.deletion.CASCADE, related_name='+', to='patchwork.Submission'), + ), + migrations.AlterField( + model_name='series', + name='cover_letter', + field=models.OneToOneField(null=True, on_delete=django.db.models.deletion.CASCADE, related_name='cl_series', to='patchwork.Submission'), + ), + migrations.DeleteModel( + name='CoverLetter', + ), + ] diff --git a/patchwork/models.py b/patchwork/models.py index a96018004f75..4b6466614fd3 100644 --- a/patchwork/models.py +++ b/patchwork/models.py @@ -389,6 +389,12 @@ class Submission(FilenameMixin, EmailMixin, models.Model): def __str__(self): return self.name + def get_absolute_url(self): + return reverse('cover-detail', kwargs={'cover_id': self.id}) + + def get_mbox_url(self): + return reverse('cover-mbox', kwargs={'cover_id': self.id}) + class Meta: ordering = ['date'] unique_together = [('msgid', 'project')] @@ -402,15 +408,6 @@ class Submission(FilenameMixin, EmailMixin, models.Model): ] -class CoverLetter(Submission): - - def get_absolute_url(self): - return reverse('cover-detail', kwargs={'cover_id': self.id}) - - def get_mbox_url(self): - return reverse('cover-mbox', kwargs={'cover_id': self.id}) - - @python_2_unicode_compatible class Patch(Submission): # patch metadata @@ -739,7 +736,7 @@ class Series(FilenameMixin, models.Model): blank=True, on_delete=models.CASCADE) # content - cover_letter = models.OneToOneField(CoverLetter, related_name='series', + cover_letter = models.OneToOneField(Submission, related_name='cl_series', null=True, on_delete=models.CASCADE) @@ -1030,7 +1027,7 @@ class Event(models.Model): on_delete=models.CASCADE, help_text='The series that this event was created for.') cover = models.ForeignKey( - CoverLetter, related_name='+', null=True, blank=True, + Submission, related_name='+', null=True, blank=True, on_delete=models.CASCADE, help_text='The cover letter that this event was created for.') diff --git a/patchwork/parser.py b/patchwork/parser.py index 7dc66bc05a5b..a122839f3f73 100644 --- a/patchwork/parser.py +++ b/patchwork/parser.py @@ -19,7 +19,6 @@ from django.db.utils import IntegrityError from django.utils import six from patchwork.models import Comment -from patchwork.models import CoverLetter from patchwork.models import DelegationRule from patchwork.models import get_default_initial_patch_state from patchwork.models import Patch @@ -1088,11 +1087,11 @@ def parse_mail(mail, list_id=None): if not is_comment: if not refs == []: try: - CoverLetter.objects.all().get(name=name) - except CoverLetter.DoesNotExist: + Submission.objects.get(name=name, new_diff__isnull=True, new_pull_url__isnull=True) + except Submission.DoesNotExist: # if no match, this is a new cover letter is_cover_letter = True - except CoverLetter.MultipleObjectsReturned: + except Submission.MultipleObjectsReturned: # if multiple cover letters are found, just ignore pass else: @@ -1135,7 +1134,7 @@ def parse_mail(mail, list_id=None): " in project %s!" % (msgid, project.name)) try: - cover_letter = CoverLetter.objects.create( + cover_letter = Submission.objects.create( msgid=msgid, project=project, name=name[:255], diff --git a/patchwork/signals.py b/patchwork/signals.py index 666199b68161..bbfdc0106fd7 100644 --- a/patchwork/signals.py +++ b/patchwork/signals.py @@ -10,11 +10,11 @@ from django.db.models.signals import pre_save from django.dispatch import receiver from patchwork.models import Check -from patchwork.models import CoverLetter from patchwork.models import Event from patchwork.models import Patch from patchwork.models import PatchChangeNotification from patchwork.models import Series +from patchwork.models import Submission @receiver(pre_save, sender=Patch) @@ -54,7 +54,7 @@ def patch_change_callback(sender, instance, raw, **kwargs): notification.save() -@receiver(post_save, sender=CoverLetter) +@receiver(post_save, sender=Submission) def create_cover_created_event(sender, instance, created, raw, **kwargs): def create_event(cover): diff --git a/patchwork/templates/patchwork/submission.html b/patchwork/templates/patchwork/submission.html index e79dd92497a4..c4175a30e9c5 100644 --- a/patchwork/templates/patchwork/submission.html +++ b/patchwork/templates/patchwork/submission.html @@ -68,12 +68,12 @@ function toggle_div(link_id, headers_id) -{% if submission.series %} +{% if series %} Series - - {{ submission.series }} + + {{ series }} @@ -85,7 +85,7 @@ function toggle_div(link_id, headers_id) >show