From patchwork Tue Oct 29 10:14:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 2003661 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Xd5kv02mMz1xwn for ; Tue, 29 Oct 2024 21:16:21 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 6AD2880BE0; Tue, 29 Oct 2024 10:16:19 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id b1zIbBR_3DeH; Tue, 29 Oct 2024 10:16:18 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.9.56; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 37B3580BB3 Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTPS id 37B3580BB3; Tue, 29 Oct 2024 10:16:18 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 167D1C08A6; Tue, 29 Oct 2024 10:16:18 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id E7544C08A3 for ; Tue, 29 Oct 2024 10:16:16 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id C05CB40B20 for ; Tue, 29 Oct 2024 10:16:16 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id IO4k6qPygmLC for ; Tue, 29 Oct 2024 10:16:15 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.128.66; helo=mail-wm1-f66.google.com; envelope-from=i.maximets.ovn@gmail.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp2.osuosl.org 1DF5B40B93 Authentication-Results: smtp2.osuosl.org; dmarc=none (p=none dis=none) header.from=ovn.org DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 1DF5B40B93 Received: from mail-wm1-f66.google.com (mail-wm1-f66.google.com [209.85.128.66]) by smtp2.osuosl.org (Postfix) with ESMTPS id 1DF5B40B93 for ; Tue, 29 Oct 2024 10:16:14 +0000 (UTC) Received: by mail-wm1-f66.google.com with SMTP id 5b1f17b1804b1-4315eac969aso36910505e9.1 for ; Tue, 29 Oct 2024 03:16:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730196973; x=1730801773; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=z3pnvXUB3BwHVCxzmDv42pKKA+rlqSMhDfldbkKIRIU=; b=ZO2WoDJrlsMgl0qyWdS42ujMw+ealbsoOF4fDw/9lZ7lrbcfUfsEQJa7B4xhSst0K1 XV+gEFoyjrrQGb3Vn4dEyPq4DT5vF/DkWBcmfdh1zNuVakH9iCcD7j4hlnQHwi1zJ5QW fU4zlhvYvihBP9BD7xOSi2ZmUtEbo1vmelD5A89hgfu9IKBABJtP8kKqeLh8HRJPEqrf 75jzzJa1P18c2oaMAJEZSy7fg2CkQXyz95Ch0Ki86M4zwPfiO9cz1xW9LPR4kq4gG/TT s+NxpH/VEv463nVRaV08qHvAbP8Gtlg1JA4x7jYSUvxJXFTuHn9aEq9koW8pPKYFXXHY ZJBA== X-Gm-Message-State: AOJu0YxNr+aVGdNVkS+MTWHqALclPu1ghya5/w/6xEF9bRwb6OMt5Kiz nhVBnFIboi7bCxShP6+mFwQFBrTWpUWx/xZAEZxK8FxghRjVmUJ2ISS817At X-Google-Smtp-Source: AGHT+IGzgeM8EIv0aW0yxxP/YNlciJ6i3mR6bRAa67NJ1BcbAr2CZmDStO2f6jUSo97YYszBhYfy8Q== X-Received: by 2002:a05:600c:5487:b0:424:a7f1:ba2 with SMTP id 5b1f17b1804b1-431b56364e0mr11787575e9.17.1730196972879; Tue, 29 Oct 2024 03:16:12 -0700 (PDT) Received: from im-t490s.redhat.com (ip-86-49-44-151.bb.vodafone.cz. [86.49.44.151]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-431b4594ec3sm20279685e9.1.2024.10.29.03.16.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Oct 2024 03:16:12 -0700 (PDT) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Tue, 29 Oct 2024 11:14:58 +0100 Message-ID: <20241029101608.2991596-1-i.maximets@ovn.org> X-Mailer: git-send-email 2.46.0 MIME-Version: 1.0 Subject: [ovs-dev] [PATCH 0/9] ipsec: Resiliency to Libreswan failures. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ilya Maximets Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This patch set is a result of debugging different Librewan issues for the past few weeks in an attempt to solve the problem where ovs-monitor-ipsec gets stuck forever while calling ipsec commands and cannot progress any further. Main parts here are the introduction of the reconciliation mechanism for the ipsec connections and termination of the stuck commands on timeout. This set also contains a lot of small changes that ultimately fix compatibility with multiple versions of Libreswan as well as improve visibility into what the ovs-monitor-ipsec process is doing by adding more verbose logging. For example, without the fist patch in the set, ovs-monitor-ipsec deadlocks both libreswan and itself with Libreswan 5 pretty easily: https://github.com/libreswan/libreswan/issues/1859 More details on addressed issues are in the commit messages. The last few patches in the set are adding a system test that stresses the reconciliation and various failure handling paths inside the monitor. Mainly because we do get a lot of failures from Libreswan while running the test. This test is currently actively used by Libreswan team to find and fix the root causes of multiple issues that triggered creation of this patch set. The intention for this patch set is to be backported to at least branch 3.3. But further down to 3.1 (or even 2.17 ?) may also be good. Luckily, the code is not that different on older branches. The set is tested with various versions of Libreswan including 3.32 (from Ubuntu 22.04), 4.5, 4.6, 4.9, 4.12, 4.14, 4.15 and 5.1. Without the set, only 4.5 and below work well enough, 4.9 - 4.15 are getting completely stuck with a few dozens of connections, and 5.1 deadlocks easily. With the set: 4.5 and below still work well, 5.1 works well, 4.9 - 4.15 can get into state with connectivity issues (libreswan issue that cannot be worked around externally), but it is much less likely to end up in this state and it affects only a couple individual connections instead of blocking the daemon as a whole. Also, 4.14 and 4.15 seems noticeably harder to get into that state (but still very possible). Ilya Maximets (9): ipsec: Add a helper function to run commands from the monitor. ipsec: libreswan: Reconcile missing connections periodically. ipsec: libreswan: Try to bring non-active connections up. ipsec: libreswan: Fix regexp for connections waiting on child SA. ipsec: libreswan: Avoid monitor hanging on stuck ipsec commands. ipsec: Make command timeout configurable. system-tests: Verbose cleanup of ports and namespaces. tests: ipsec: Add NxN + reconciliation test. tests: ipsec: Check that nodes can ping each other in the NxN test. ipsec/ovs-monitor-ipsec.in | 483 +++++++++++++++++++--------------- tests/system-common-macros.at | 7 +- tests/system-ipsec.at | 206 ++++++++++++++- 3 files changed, 463 insertions(+), 233 deletions(-)