| Message ID | 20190918061731.19142-1-dja@axtens.net |
|---|---|
| Headers | show |
| Series | A sketch of flattening the models with live migration | expand |
Daniel Axtens <dja@axtens.net> writes: > We want to flatten CoverLetter and Patch into Submission for performance > and simplicity reasons. > > We also want to avoid some of the debacles of previous migrations where > sites were offline for hours to perform the migration, or needed to > drop the migration out of Python entirely and in to raw SQL. > > So this is a sketch or outline of an approach were we migrate diff > and pull_url from Patch to Submission using live-ish migrations (not > fully live, but with no data migrations in the schema changes) and > then drop the CoverLetter model. I now have a full implementation which I'm still cleaning up and rebasing in the lead up to an RFC: https://github.com/daxtens/patchwork/tree/flatten It does away with the Patch and CoverLetter models - everything is now a Submission and we have submission.is_patch()/Submission.patch_objects and submission.is_cover()/Submission.cover_objects instead. Daniel > > The way this works is pretty straight forward and is documented in > more detail in the patches - basically patches 1-3 are all minor > fixups, then you bring down your instance, apply patch 4, do the > schema migration, bring up your instance, perform the data migration > in the background, then bring down your instance, apply patches 5 and > 6, do those migrations, and then you're done. Total downtime should be > very short (hopefully no more than a few minutes), depending on how > long it takes to add columns to your database. > > Lots to do before this is a full approach - I still have to migrate > all the rest of the fields, and then clean up everything that depends > on a Patch model existing. But I feel like this at least demonstrates > that we can do the really hard work of the migration - moving the data > around - in a way that is friendly to sysadmins of large instances. > > Daniel Axtens (6): > pyenv: also install requirements for python2 > tests: make test suite pass with XML-RPC disabled > parsearchive,mail: use repr() to get a human readable exception > Begin live migration of diff and pull_url to Submission > Migration part 2: drop old fields > Burninate CoverLetter > > patchwork/admin.py | 3 - > patchwork/api/cover.py | 16 +-- > patchwork/api/embedded.py | 2 +- > patchwork/api/filters.py | 13 ++- > patchwork/api/patch.py | 10 +- > .../commands/livemigrate-v22-v23.py | 35 ++++++ > patchwork/management/commands/parsearchive.py | 6 +- > patchwork/management/commands/parsemail.py | 2 +- > .../0037_prepare_old_diff_pull_url.py | 35 ++++++ > .../0038_submission_new_diff_pull_url.py | 25 ++++ > patchwork/migrations/0039_sanity_check.py | 27 +++++ > .../migrations/0040_drop_old_diff_pull_url.py | 23 ++++ > patchwork/migrations/0041_drop_coverletter.py | 29 +++++ > patchwork/models.py | 110 +++++++++++++++--- > patchwork/parser.py | 9 +- > patchwork/signals.py | 4 +- > patchwork/templates/patchwork/submission.html | 10 +- > patchwork/tests/api/test_cover.py | 5 +- > patchwork/tests/test_about.py | 24 +++- > patchwork/tests/test_series.py | 14 +-- > patchwork/tests/test_xmlrpc.py | 3 + > patchwork/tests/utils.py | 6 +- > patchwork/views/cover.py | 25 ++-- > patchwork/views/patch.py | 1 + > patchwork/views/xmlrpc.py | 5 +- > tools/docker/Dockerfile | 11 +- > 26 files changed, 378 insertions(+), 75 deletions(-) > create mode 100644 patchwork/management/commands/livemigrate-v22-v23.py > create mode 100644 patchwork/migrations/0037_prepare_old_diff_pull_url.py > create mode 100644 patchwork/migrations/0038_submission_new_diff_pull_url.py > create mode 100644 patchwork/migrations/0039_sanity_check.py > create mode 100644 patchwork/migrations/0040_drop_old_diff_pull_url.py > create mode 100644 patchwork/migrations/0041_drop_coverletter.py > > -- > 2.20.1