diff mbox series

[v3,2/2] parser: Use a second query to weed out duplicate series

Message ID 20191130221432.73118-2-stephen@that.guru
State Superseded
Headers show
Series [v3,1/2] models: Use database constraints to prevent split Series | expand

Commit Message

Stephen Finucane Nov. 30, 2019, 10:14 p.m. UTC 
Annoyingly, not all email clients properly thread emails using the
message ID fields originally specified in RFC 822 [1]. Worse, some MTAs
(cough, outlook.com, cough) actually override what the client
configures, breaking the world in the process. Realising this is an
issue, Patchwork supports threading using arbitrary metadata in addition
to the RFC 822 metadata. Specifically, it uses a combination of
submitter and list-id extracted from the headers along with the series
version and total count metadata extracted from the subject. In addition
to this, we timebox things so that two or more series that match on all
of this metadata but which are sent some time apart from each other
aren't combined by accident. This does leave one edge case - duplicate
series received within the timebox will be combined. We've resigned
ourselves to this fact on the basis that it's extremely unlikely for all
of these things to go wrong at once.

Given all the above, there should be no reason that attempting to find
series by series markers should return more than one series. The
timeboxing will prevent us grouping similar looking series by accident
and the only other reason for this to happen is because we lost a race
and we should try again.

[1] https://tools.ietf.org/html/rfc822

Signed-off-by: Stephen Finucane <stephen@that.guru>
Cc: Daniel Axtens <dja@axtens.net>
---
 patchwork/parser.py | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
diff mbox series

Patch

diff --git a/patchwork/parser.py b/patchwork/parser.py
index 425684f6..563338ff 100644
--- a/patchwork/parser.py
+++ b/patchwork/parser.py
@@ -298,7 +298,8 @@  def _find_series_by_markers(project, mail, author):
         return Series.objects.get(
             submitter=author, project=project, version=version, total=total,
             date__range=[start_date, end_date])
-    except (Series.DoesNotExist, Series.MultipleObjectsReturned):
+    except Series.DoesNotExist:
+        # we're creating a new series
         return
 
 
@@ -1130,8 +1131,14 @@  def parse_mail(mail, list_id=None):
                             except SeriesReference.DoesNotExist:
                                 SeriesReference.objects.create(
                                     msgid=ref, project=project, series=series)
+
+                    # attempt to pull the series in again, raising an exception
+                    # if we lost the race when creating a series and force us
+                    # to go through this again
+                    series = find_series(project, mail, author)
+
                     break
-            except IntegrityError:
+            except (IntegrityError, Series.MultipleObjectsReturned):
                 # we lost the race so go again
                 logger.warning('Conflict while saving series. This is '
                                'probably because multiple patches belonging '