From patchwork Wed Sep 18 06:17:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Daniel Axtens X-Patchwork-Id: 1163755 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.ozlabs.org (lists.ozlabs.org [203.11.71.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 46Y8t411gkz9sCJ for ; Wed, 18 Sep 2019 16:18:04 +1000 (AEST) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=axtens.net Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="MadtHVkZ"; dkim-atps=neutral Received: from bilbo.ozlabs.org (lists.ozlabs.org [IPv6:2401:3900:2:1::3]) by lists.ozlabs.org (Postfix) with ESMTP id 46Y8t35qL7zF4T9 for ; Wed, 18 Sep 2019 16:18:03 +1000 (AEST) X-Original-To: patchwork@lists.ozlabs.org Delivered-To: patchwork@lists.ozlabs.org Authentication-Results: lists.ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=axtens.net (client-ip=2607:f8b0:4864:20::62e; helo=mail-pl1-x62e.google.com; envelope-from=dja@axtens.net; receiver=) Authentication-Results: lists.ozlabs.org; dmarc=none (p=none dis=none) header.from=axtens.net Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=axtens.net header.i=@axtens.net header.b="MadtHVkZ"; dkim-atps=neutral Received: from mail-pl1-x62e.google.com (mail-pl1-x62e.google.com [IPv6:2607:f8b0:4864:20::62e]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 46Y8sv3V4SzF32B for ; Wed, 18 Sep 2019 16:17:53 +1000 (AEST) Received: by mail-pl1-x62e.google.com with SMTP id e5so2608937pls.9 for ; Tue, 17 Sep 2019 23:17:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=axtens.net; s=google; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=twCIUjl4mF5ggIseqiLxdJxBS4imV2ukpGp3+vl8GdQ=; b=MadtHVkZD49LFSMPcC2Mhn+oXe/uTHqfClzkXoaqLYOYFGXXXZ/qQh020xF/afTzVz DXq4eResMd6yuChAbqffA28nPcAnja8b8nFz/q8hYzcRmqk6Vy2wLIYzgtCnAszmaXd5 roQfTxC1t4a8/I6mGgtrmXFIA9FwzLuo7vXEQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=twCIUjl4mF5ggIseqiLxdJxBS4imV2ukpGp3+vl8GdQ=; b=XtGLOV6UMZec+9VN1SQVK1RZciCgfCgGV5FImyWwCitl40H2FxS3IaB0FRFtdht/fY XtFlCLN3pqc5L2fiIhHH9jfXYqwqA7AtK+t2uHuCc8PvhXylVAl+w4oThiCQbpQD7ygO KNe0l7AuAMl72v/NKVShUOZeAQaO4UoaL54vnjEphwslrbLkMROSyQY66CqRfWOqZCKO hpoN+lhQ6BAfghNikm312vxgy0fLJUnOmzVlG6vNBYpVkwHfv7Dxlf5jP6Cdv/d/oPXb G5AjMU8XyyNSurpgZ0LrFapuFHD0DAeW38Z5c5zeZ2k8kjdFUqLxhB129Muz4c8fcasG XZuw== X-Gm-Message-State: APjAAAXyS3746k/RcSLVEB2WGMy0hKPnVLton+MC4oHMDK2xyiC3KSdn i7tEbKjXv+WMaxJyAD3uRnxrs1Kr9ro= X-Google-Smtp-Source: APXvYqzS/We1xP5XRaDIvmv+PcSoXrYPqGtvfgBdUWWpYWQ0QLY3UOxc+93L5LBPhHGmiYqcIYxTwQ== X-Received: by 2002:a17:902:a411:: with SMTP id p17mr2540551plq.108.1568787469745; Tue, 17 Sep 2019 23:17:49 -0700 (PDT) Received: from localhost (ppp167-251-205.static.internode.on.net. [59.167.251.205]) by smtp.gmail.com with ESMTPSA id 5sm2784289pgk.75.2019.09.17.23.17.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Sep 2019 23:17:48 -0700 (PDT) From: Daniel Axtens To: patchwork@lists.ozlabs.org Subject: [PATCH 0/6] A sketch of flattening the models with live migration Date: Wed, 18 Sep 2019 16:17:25 +1000 Message-Id: <20190918061731.19142-1-dja@axtens.net> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 X-BeenThere: patchwork@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Patchwork development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: patchwork-bounces+incoming=patchwork.ozlabs.org@lists.ozlabs.org Sender: "Patchwork" We want to flatten CoverLetter and Patch into Submission for performance and simplicity reasons. We also want to avoid some of the debacles of previous migrations where sites were offline for hours to perform the migration, or needed to drop the migration out of Python entirely and in to raw SQL. So this is a sketch or outline of an approach were we migrate diff and pull_url from Patch to Submission using live-ish migrations (not fully live, but with no data migrations in the schema changes) and then drop the CoverLetter model. The way this works is pretty straight forward and is documented in more detail in the patches - basically patches 1-3 are all minor fixups, then you bring down your instance, apply patch 4, do the schema migration, bring up your instance, perform the data migration in the background, then bring down your instance, apply patches 5 and 6, do those migrations, and then you're done. Total downtime should be very short (hopefully no more than a few minutes), depending on how long it takes to add columns to your database. Lots to do before this is a full approach - I still have to migrate all the rest of the fields, and then clean up everything that depends on a Patch model existing. But I feel like this at least demonstrates that we can do the really hard work of the migration - moving the data around - in a way that is friendly to sysadmins of large instances. Daniel Axtens (6): pyenv: also install requirements for python2 tests: make test suite pass with XML-RPC disabled parsearchive,mail: use repr() to get a human readable exception Begin live migration of diff and pull_url to Submission Migration part 2: drop old fields Burninate CoverLetter patchwork/admin.py | 3 - patchwork/api/cover.py | 16 +-- patchwork/api/embedded.py | 2 +- patchwork/api/filters.py | 13 ++- patchwork/api/patch.py | 10 +- .../commands/livemigrate-v22-v23.py | 35 ++++++ patchwork/management/commands/parsearchive.py | 6 +- patchwork/management/commands/parsemail.py | 2 +- .../0037_prepare_old_diff_pull_url.py | 35 ++++++ .../0038_submission_new_diff_pull_url.py | 25 ++++ patchwork/migrations/0039_sanity_check.py | 27 +++++ .../migrations/0040_drop_old_diff_pull_url.py | 23 ++++ patchwork/migrations/0041_drop_coverletter.py | 29 +++++ patchwork/models.py | 110 +++++++++++++++--- patchwork/parser.py | 9 +- patchwork/signals.py | 4 +- patchwork/templates/patchwork/submission.html | 10 +- patchwork/tests/api/test_cover.py | 5 +- patchwork/tests/test_about.py | 24 +++- patchwork/tests/test_series.py | 14 +-- patchwork/tests/test_xmlrpc.py | 3 + patchwork/tests/utils.py | 6 +- patchwork/views/cover.py | 25 ++-- patchwork/views/patch.py | 1 + patchwork/views/xmlrpc.py | 5 +- tools/docker/Dockerfile | 11 +- 26 files changed, 378 insertions(+), 75 deletions(-) create mode 100644 patchwork/management/commands/livemigrate-v22-v23.py create mode 100644 patchwork/migrations/0037_prepare_old_diff_pull_url.py create mode 100644 patchwork/migrations/0038_submission_new_diff_pull_url.py create mode 100644 patchwork/migrations/0039_sanity_check.py create mode 100644 patchwork/migrations/0040_drop_old_diff_pull_url.py create mode 100644 patchwork/migrations/0041_drop_coverletter.py