From patchwork Tue Oct 29 10:14:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 2003662 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Xd5ky4Cb9z1xwn for ; Tue, 29 Oct 2024 21:16:26 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id B840760D53; Tue, 29 Oct 2024 10:16:24 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id au72CrJDiE3l; Tue, 29 Oct 2024 10:16:21 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=2605:bc80:3010:104::8cd3:938; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org A73E360A3B Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id A73E360A3B; Tue, 29 Oct 2024 10:16:21 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 86247C08A6; Tue, 29 Oct 2024 10:16:21 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 90E3CC08A8 for ; Tue, 29 Oct 2024 10:16:20 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 6A23D40B1F for ; Tue, 29 Oct 2024 10:16:20 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id ZKA3JUSY-PAG for ; Tue, 29 Oct 2024 10:16:18 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.128.68; helo=mail-wm1-f68.google.com; envelope-from=i.maximets.ovn@gmail.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp2.osuosl.org 83EE840AFB Authentication-Results: smtp2.osuosl.org; dmarc=none (p=none dis=none) header.from=ovn.org DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 83EE840AFB Received: from mail-wm1-f68.google.com (mail-wm1-f68.google.com [209.85.128.68]) by smtp2.osuosl.org (Postfix) with ESMTPS id 83EE840AFB for ; Tue, 29 Oct 2024 10:16:17 +0000 (UTC) Received: by mail-wm1-f68.google.com with SMTP id 5b1f17b1804b1-431ac30d379so15819535e9.1 for ; Tue, 29 Oct 2024 03:16:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730196975; x=1730801775; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sTc4JfznMqjHEBSgMU5MG3oR025MM5QTzNErnxhkfY4=; b=KYh2nUMz8+UpomI818/vGZg3kCKG4otE7HmekKko4uOUTeu6kTSGtsWbuD7nkVVlL+ luyp3HQtFCx+ej1YLJwYp2zdzAngjHDax4cd2UdLsHTJwBShCTy6behze8a7hEgb3AgU cA+Qd+Opok5zwKATXGdTVLFpo+rtjo0JGJfepayi/ymWFfIMzndHsTh/dfyXn4cm077H DXS2KVWQ/ehCOcH0X2MQ4yLKzOdbjCRBZBbkxvYUNrOmC9WTXmIhbpWJDriO4KJtgzGZ SdlsPQa5q73VXHLpSCI1Rxn/BQBR2inWGd/Bbdb1B2r8p/MawRIsEpwKsGDuYEIuXJ5h 3/8Q== X-Gm-Message-State: AOJu0YzdE9498xswlKhSUVnCuci8+WC8z8uaIe8d13sPOnjGezseHg2s hA+PD6XTsJXv553fsvLJRXrRRp0EexCXbGCVKNG1+qqUnRyJ7YJEP2rySlSY X-Google-Smtp-Source: AGHT+IGSvbILYQVs+eCFU6r9FFHVtk10c/zys09juaH+W1oogz9fvFT3y8x7DJA/jVZi+YXJd4231Q== X-Received: by 2002:a05:600c:1c82:b0:430:699b:7d22 with SMTP id 5b1f17b1804b1-4319ad14d14mr109825345e9.26.1730196975101; Tue, 29 Oct 2024 03:16:15 -0700 (PDT) Received: from im-t490s.redhat.com (ip-86-49-44-151.bb.vodafone.cz. [86.49.44.151]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-431b4594ec3sm20279685e9.1.2024.10.29.03.16.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Oct 2024 03:16:14 -0700 (PDT) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Tue, 29 Oct 2024 11:14:59 +0100 Message-ID: <20241029101608.2991596-2-i.maximets@ovn.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241029101608.2991596-1-i.maximets@ovn.org> References: <20241029101608.2991596-1-i.maximets@ovn.org> MIME-Version: 1.0 Subject: [ovs-dev] [PATCH 1/9] ipsec: Add a helper function to run commands from the monitor. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ilya Maximets Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Until now, functions that needed to call external programs like openssl or ipsec commands were using subprocess commands directly. Most of these calls had no failure checks or any logging making it hard to understand what is happening inside the daemon when something doesn't work as intended. Some commands also had a chance to not read the command output in full. That might sound like not a big problem, but in practice it causes ovs-monitor-ipsec to deadlock pluto and itself with certain versions of Libreswan (mainly Libreswan 5+). The order of events is following: 1. ovs-monitor-ipsec calls ipsec status redirecting the output to a pipe. 2. ipsec status calls ipsec whack. 3. ipsec whack connects to pluto and asks for status. 4. ovs-monitor-ipsec doesn't read the pipe in full. 5. ipsec whack blocks on write to the other side of the pipe when it runs out of buffer space. 6. pluto blocks on sendmsg to ipsec whack for the same reason. 7. ovs-monitor-ipsec calls another ipsec command and blocks on connection to pluto. In this scenario the running process is at the mercy of garbage collector and it doesn't run because we're blocked on calling another ipsec command. All the processes are completely blocked and will not do any work until ipsec whack is killed. With this change we're introducing a new function that will be used for all the external process execution commands and will read the full output before returning, avoiding the deadlock. It will also log all the failures as warnings and the commands themselves at the debug level. We'll be adding more logic into this function in later commits as well, so it will not stay that simple. Signed-off-by: Ilya Maximets --- ipsec/ovs-monitor-ipsec.in | 296 +++++++++++++++++-------------------- 1 file changed, 134 insertions(+), 162 deletions(-) diff --git a/ipsec/ovs-monitor-ipsec.in b/ipsec/ovs-monitor-ipsec.in index 37c509ac6..19a401609 100755 --- a/ipsec/ovs-monitor-ipsec.in +++ b/ipsec/ovs-monitor-ipsec.in @@ -84,6 +84,28 @@ monitor = None xfrm = None +def run_command(args, description=None): + """ This function runs the process args[0] with args[1:] arguments + and returns a tuple: return-code, stdout, stderr. """ + + if not description: + description = args[-1] + + vlog.dbg("Running %s" % args) + proc = subprocess.Popen(args, stdout=subprocess.PIPE, + stderr=subprocess.PIPE) + pout, perr = proc.communicate() + + if proc.returncode or len(perr): + vlog.warn("Failed to %s; exit code: %d" + % (description, proc.returncode)) + vlog.warn("cmdline: %s" % proc.args) + vlog.warn("stderr: %s" % perr) + vlog.warn("stdout: %s" % pout) + + return proc.returncode, pout or b'', perr or b'' + + class XFRM(object): """This class is a simple wrapper around ip-xfrm (8) command line utility. We are using this class only for informational purposes @@ -99,13 +121,14 @@ class XFRM(object): where is destination IPv4 address and is SELECTOR of the IPsec policy.""" policies = {} - proc = subprocess.Popen([self.IP, 'xfrm', 'policy'], - stdout=subprocess.PIPE) - while True: - line = proc.stdout.readline().strip().decode() - if line == '': - break - a = line.split(" ") + + ret, pout, perr = run_command([self.IP, 'xfrm', 'policy'], + "get XFRM policies") + if ret: + return policies + + for line in pout.decode().splitlines(): + a = line.strip().split(" ") if len(a) >= 4 and a[0] == "src" and a[2] == "dst": dst = (a[3].split("/"))[0] if dst not in policies: @@ -122,13 +145,14 @@ class XFRM(object): in a dictionary where is destination IPv4 address and is SELECTOR.""" securities = {} - proc = subprocess.Popen([self.IP, 'xfrm', 'state'], - stdout=subprocess.PIPE) - while True: - line = proc.stdout.readline().strip().decode() - if line == '': - break - a = line.split(" ") + + ret, pout, perr = run_command([self.IP, 'xfrm', 'state'], + "get XFRM state") + if ret: + return securities + + for line in pout.decode().splitlines(): + a = line.strip().split(" ") if len(a) >= 4 and a[0] == "sel" \ and a[1] == "src" and a[3] == "dst": remote_ip = a[4].rstrip().split("/")[0] @@ -242,7 +266,7 @@ conn prevent_unencrypted_vxlan f.close() vlog.info("Restarting StrongSwan") - subprocess.call([self.IPSEC, "restart"]) + run_command([self.IPSEC, "restart"], "restart StrongSwan") def get_active_conns(self): """This function parses output from 'ipsec status' command. @@ -252,13 +276,13 @@ conn prevent_unencrypted_vxlan sample line from the parsed outpus as . """ conns = {} - proc = subprocess.Popen([self.IPSEC, 'status'], stdout=subprocess.PIPE) + ret, pout, perr = run_command([self.IPSEC, 'status'], + "get active connections") + if ret: + return conns - while True: - line = proc.stdout.readline().strip().decode() - if line == '': - break - tunnel_name = line.split(":") + for line in pout.decode().splitlines(): + tunnel_name = line.strip().split(":") if len(tunnel_name) < 2: continue m = re.match(r"(.*)(-in-\d+|-out-\d+|-\d+).*", tunnel_name[0]) @@ -341,15 +365,11 @@ conn prevent_unencrypted_vxlan Once strongSwan vici bindings will be distributed with major Linux distributions this function could be simplified.""" vlog.info("Refreshing StrongSwan configuration") - proc = subprocess.Popen([self.IPSEC, "update"], - stdout=subprocess.PIPE, - stderr=subprocess.PIPE) - outs, errs = proc.communicate() - if proc.returncode != 0: - vlog.err("StrongSwan failed to update configuration:\n" - "%s \n %s" % (str(outs), str(errs))) - - subprocess.call([self.IPSEC, "rereadsecrets"]) + + run_command([self.IPSEC, "update"], + "update StrongSwan's configuration") + run_command([self.IPSEC, "rereadsecrets"], "re-read secrets") + # "ipsec update" command does not remove those tunnels that were # updated or that disappeared from the ipsec.conf file. So, we have # to manually remove them by calling "ipsec stroke down-nb " @@ -382,7 +402,8 @@ conn prevent_unencrypted_vxlan if not tunnel or tunnel.version != ver: vlog.info("%s is outdated %u" % (conn, ver)) - subprocess.call([self.IPSEC, "stroke", "down-nb", conn]) + run_command([self.IPSEC, "stroke", "down-nb", conn], + "stroke the outdated %s" % conn) class LibreSwanHelper(object): @@ -460,13 +481,11 @@ conn prevent_unencrypted_vxlan # Collect version infromation self.IPSEC = libreswan_root_prefix + "/usr/sbin/ipsec" self.IPSEC_AUTO = [self.IPSEC] - proc = subprocess.Popen([self.IPSEC, "--version"], - stdout=subprocess.PIPE, - encoding="latin1") - pout, perr = proc.communicate() - v = re.match("^Libreswan v?(.*)$", pout) + ret, pout, perr = run_command([self.IPSEC, "--version"], + "get Libreswan's version") try: + v = re.match("^Libreswan v?(.*)$", pout.decode().strip()) version = int(v.group(1).split(".")[0]) except: version = 0 @@ -513,7 +532,7 @@ conn prevent_unencrypted_vxlan f.close() vlog.info("Restarting LibreSwan") - subprocess.call([self.IPSEC, "restart"]) + run_command([self.IPSEC, "restart"], "restart Libreswan") def config_init(self): self.conf_file = open(self.IPSEC_CONF, "w") @@ -599,8 +618,10 @@ conn prevent_unencrypted_vxlan def refresh(self, monitor): vlog.info("Refreshing LibreSwan configuration") - subprocess.call(self.IPSEC_AUTO + ["--ctlsocket", self.IPSEC_CTL, - "--config", self.IPSEC_CONF, "--rereadsecrets"]) + run_command(self.IPSEC_AUTO + ["--ctlsocket", self.IPSEC_CTL, + "--config", self.IPSEC_CONF, + "--rereadsecrets"], + "re-read secrets") tunnels = set(monitor.tunnels.keys()) # Delete old connections @@ -627,9 +648,10 @@ conn prevent_unencrypted_vxlan if not tunnel or tunnel.version != ver: vlog.info("%s is outdated %u" % (conn, ver)) - subprocess.call(self.IPSEC_AUTO + ["--ctlsocket", - self.IPSEC_CTL, "--config", - self.IPSEC_CONF, "--delete", conn]) + run_command(self.IPSEC_AUTO + + ["--ctlsocket", self.IPSEC_CTL, + "--config", self.IPSEC_CONF, + "--delete", conn], "delete %s" % conn) elif ifname in tunnels: tunnels.remove(ifname) @@ -649,43 +671,43 @@ conn prevent_unencrypted_vxlan # Update shunt policy if changed if monitor.conf_in_use["skb_mark"] != monitor.conf["skb_mark"]: if monitor.conf["skb_mark"]: - subprocess.call(self.IPSEC_AUTO + + run_command(self.IPSEC_AUTO + ["--config", self.IPSEC_CONF, "--ctlsocket", self.IPSEC_CTL, "--add", "--asynchronous", "prevent_unencrypted_gre"]) - subprocess.call(self.IPSEC_AUTO + + run_command(self.IPSEC_AUTO + ["--config", self.IPSEC_CONF, "--ctlsocket", self.IPSEC_CTL, "--add", "--asynchronous", "prevent_unencrypted_geneve"]) - subprocess.call(self.IPSEC_AUTO + + run_command(self.IPSEC_AUTO + ["--config", self.IPSEC_CONF, "--ctlsocket", self.IPSEC_CTL, "--add", "--asynchronous", "prevent_unencrypted_stt"]) - subprocess.call(self.IPSEC_AUTO + + run_command(self.IPSEC_AUTO + ["--config", self.IPSEC_CONF, "--ctlsocket", self.IPSEC_CTL, "--add", "--asynchronous", "prevent_unencrypted_vxlan"]) else: - subprocess.call(self.IPSEC_AUTO + + run_command(self.IPSEC_AUTO + ["--config", self.IPSEC_CONF, "--ctlsocket", self.IPSEC_CTL, "--delete", "--asynchronous", "prevent_unencrypted_gre"]) - subprocess.call(self.IPSEC_AUTO + + run_command(self.IPSEC_AUTO + ["--config", self.IPSEC_CONF, "--ctlsocket", self.IPSEC_CTL, "--delete", "--asynchronous", "prevent_unencrypted_geneve"]) - subprocess.call(self.IPSEC_AUTO + + run_command(self.IPSEC_AUTO + ["--config", self.IPSEC_CONF, "--ctlsocket", self.IPSEC_CTL, "--delete", "--asynchronous", "prevent_unencrypted_stt"]) - subprocess.call(self.IPSEC_AUTO + + run_command(self.IPSEC_AUTO + ["--config", self.IPSEC_CONF, "--ctlsocket", self.IPSEC_CTL, "--delete", @@ -700,14 +722,13 @@ conn prevent_unencrypted_vxlan sample line from the parsed outpus as . """ conns = {} - proc = subprocess.Popen([self.IPSEC, 'status', '--ctlsocket', - self.IPSEC_CTL], stdout=subprocess.PIPE) - - while True: - line = proc.stdout.readline().strip().decode() - if line == '': - break + ret, pout, perr = run_command([self.IPSEC, 'status', + '--ctlsocket', self.IPSEC_CTL], + "get active connections") + if ret: + return conns + for line in pout.decode().splitlines(): m = re.search(r"#\d+: \"(.*)\".*", line) if not m: continue @@ -732,117 +753,72 @@ conn prevent_unencrypted_vxlan # the "ipsec auto --start" command is lost. Just retry to make sure # the command is received by LibreSwan. while True: - proc = subprocess.Popen(self.IPSEC_AUTO + - ["--config", self.IPSEC_CONF, - "--ctlsocket", self.IPSEC_CTL, - "--start", - "--asynchronous", conn], - stdout=subprocess.PIPE, - stderr=subprocess.PIPE) - perr = str(proc.stderr.read()) - pout = str(proc.stdout.read()) - if not re.match(r".*Connection refused.*", perr) and \ - not re.match(r".*need --listen.*", pout): + ret, pout, perr = run_command(self.IPSEC_AUTO + + ["--config", self.IPSEC_CONF, + "--ctlsocket", self.IPSEC_CTL, + "--start", + "--asynchronous", conn], + "start %s" % conn) + if not re.match(r".*Connection refused.*", perr.decode()) and \ + not re.match(r".*need --listen.*", pout.decode()): break - if re.match(r".*[F|f]ailed to initiate connection.*", pout): + if re.match(r".*[F|f]ailed to initiate connection.*", pout.decode()): vlog.err('Failed to initiate connection through' ' Interface %s.\n' % (conn.split('-')[0])) - vlog.err(pout) + vlog.err("stdout: %s" % pout) def _nss_clear_database(self): """Remove all OVS IPsec related state from the NSS database""" - try: - proc = subprocess.Popen(['certutil', '-L', '-d', - self.IPSEC_D], - stdout=subprocess.PIPE, - stderr=subprocess.PIPE, - universal_newlines=True) - lines = proc.stdout.readlines() - - for line in lines: - s = line.strip().split() - if len(s) < 1: - continue - name = s[0] - if name.startswith(self.CERT_PREFIX): - self._nss_delete_cert(name) - elif name.startswith(self.CERTKEY_PREFIX): - self._nss_delete_cert_and_key(name) + ret, pout, perr = run_command(['certutil', '-L', '-d', self.IPSEC_D], + "clear NSS database") + if ret: + return - except Exception as e: - vlog.err("Failed to clear NSS database.\n" + str(e)) + for line in pout.decode().splitlines(): + s = line.strip().split() + if len(s) < 1: + continue + name = s[0] + if name.startswith(self.CERT_PREFIX): + self._nss_delete_cert(name) + elif name.startswith(self.CERTKEY_PREFIX): + self._nss_delete_cert_and_key(name) def _nss_import_cert(self, cert, name, cert_type): """Cert_type is 'CT,,' for the CA certificate and 'P,P,P' for the normal certificate.""" - try: - proc = subprocess.Popen(['certutil', '-A', '-a', '-i', cert, - '-d', self.IPSEC_D, '-n', - name, '-t', cert_type], - stdout=subprocess.PIPE, - stderr=subprocess.PIPE) - proc.wait() - if proc.returncode: - raise Exception(proc.stderr.read()) - except Exception as e: - vlog.err("Failed to import certificate into NSS.\n" + str(e)) + run_command(['certutil', '-A', '-a', '-i', cert, '-d', self.IPSEC_D, + '-n', name, '-t', cert_type], + "import certificate %s into NSS" % name) def _nss_delete_cert(self, name): - try: - proc = subprocess.Popen(['certutil', '-D', '-d', - self.IPSEC_D, '-n', name], - stdout=subprocess.PIPE, - stderr=subprocess.PIPE) - proc.wait() - if proc.returncode: - raise Exception(proc.stderr.read()) - except Exception as e: - vlog.err("Failed to delete certificate from NSS.\n" + str(e)) + run_command(['certutil', '-D', '-d', self.IPSEC_D, '-n', name], + "delete certificate %s from NSS" % name) def _nss_import_cert_and_key(self, cert, key, name): - try: - # Avoid deleting other files - path = os.path.abspath('/tmp/%s.p12' % name) - if not path.startswith('/tmp/'): - raise Exception("Illegal certificate name!") - - # Create p12 file from pem files - proc = subprocess.Popen(['openssl', 'pkcs12', '-export', - '-in', cert, '-inkey', key, '-out', - path, '-name', name, '-passout', 'pass:'], - stdout=subprocess.PIPE, - stderr=subprocess.PIPE) - proc.wait() - if proc.returncode: - raise Exception(proc.stderr.read()) - - # Load p12 file to the database - proc = subprocess.Popen(['pk12util', '-i', path, '-d', - self.IPSEC_D, '-W', ''], - stdout=subprocess.PIPE, - stderr=subprocess.PIPE) - proc.wait() - if proc.returncode: - raise Exception(proc.stderr.read()) - - except Exception as e: - vlog.err("Import cert and key failed.\n" + str(e)) + # Avoid deleting other files + path = os.path.abspath('/tmp/%s.p12' % name) + if not path.startswith('/tmp/'): + vlog.err("Illegal certificate name '%s'!" % name) + return + + if run_command(['openssl', 'pkcs12', '-export', + '-in', cert, '-inkey', key, + '-out', path, '-name', name, + '-passout', 'pass:'], + "create p12 file from pem files")[0]: + return + + # Load p12 file to the database + run_command(['pk12util', '-i', path, '-d', self.IPSEC_D, '-W', ''], + "load p12 file to the NSS database") os.remove(path) def _nss_delete_cert_and_key(self, name): - try: - # Delete certificate and private key - proc = subprocess.Popen(['certutil', '-F', '-d', - self.IPSEC_D, '-n', name], - stdout=subprocess.PIPE, - stderr=subprocess.PIPE) - proc.wait() - if proc.returncode: - raise Exception(proc.stderr.read()) - - except Exception as e: - vlog.err("Delete cert and key failed.\n" + str(e)) + # Delete certificate and private key + run_command(['certutil', '-F', '-d', self.IPSEC_D, '-n', name], + "delete certificate and private key for %s" % name) class IPsecTunnel(object): @@ -1220,19 +1196,15 @@ class IPsecMonitor(object): self.ike_helper.refresh(self) def _get_cn_from_cert(self, cert): - try: - proc = subprocess.Popen(['openssl', 'x509', '-noout', '-subject', - '-nameopt', 'RFC2253', '-in', cert], - stdout=subprocess.PIPE, - stderr=subprocess.PIPE) - proc.wait() - if proc.returncode: - raise Exception(proc.stderr.read()) - m = re.search(r"CN=(.+?),", proc.stdout.readline().decode()) - if not m: - raise Exception("No CN in the certificate subject.") - except Exception as e: - vlog.warn(str(e)) + ret, pout, perr = run_command(['openssl', 'x509', '-noout', '-subject', + '-nameopt', 'RFC2253', '-in', cert], + "get certificate %s options" % cert) + if ret: + return None + + m = re.search(r"CN=(.+?),", pout.decode().strip()) + if not m: + vlog.warn("No CN in the certificate subject (%s)." % cert) return None return m.group(1) From patchwork Tue Oct 29 10:15:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 2003663 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Xd5l202jXz1xwn for ; Tue, 29 Oct 2024 21:16:30 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 2723A60D66; Tue, 29 Oct 2024 10:16:28 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id izhmc8paSzHF; Tue, 29 Oct 2024 10:16:24 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=2605:bc80:3010:104::8cd3:938; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 2B23F60B43 Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp3.osuosl.org (Postfix) with ESMTPS id 2B23F60B43; Tue, 29 Oct 2024 10:16:23 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id AE4DDC08A8; Tue, 29 Oct 2024 10:16:22 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 45F95C08A3 for ; Tue, 29 Oct 2024 10:16:21 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 31357608F8 for ; Tue, 29 Oct 2024 10:16:21 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id PeWOos4JKmfW for ; Tue, 29 Oct 2024 10:16:20 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.128.68; helo=mail-wm1-f68.google.com; envelope-from=i.maximets.ovn@gmail.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp3.osuosl.org A86B4608FA Authentication-Results: smtp3.osuosl.org; dmarc=none (p=none dis=none) header.from=ovn.org DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org A86B4608FA Received: from mail-wm1-f68.google.com (mail-wm1-f68.google.com [209.85.128.68]) by smtp3.osuosl.org (Postfix) with ESMTPS id A86B4608FA for ; Tue, 29 Oct 2024 10:16:19 +0000 (UTC) Received: by mail-wm1-f68.google.com with SMTP id 5b1f17b1804b1-4315baec69eso52922245e9.2 for ; Tue, 29 Oct 2024 03:16:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730196978; x=1730801778; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XcLX1R2RCUTDZJ0rGOyMKMdS9E0JbeYQugbPk0/UTlg=; b=q1Du1Wwzw6zAEtE9R5r4STQ0ks8me/HKn4Vxd2z/9jGkWEbV6vsTitfG/yClHAPVC3 BPw1iYG9yb5Zv0ML5aHl1l7yyQs+pV/QPqbsmHO9LT7OMezDPVAe7bSTVHfB3L7rJF67 VBIr87BtxKmhWJ2puoZDn8GQLAUaz0E3QMqYnIKicmYUxtMdixjhA0vvT2K3JSjLUeQ2 dyB/75s+XQ02otECT3QHUIfWt24r/sohW0WIBobXRYwTzECCYLO3NX9mxmJbggD+Coml 4ymbeQI1x2kQmnMtaFNgpFDJiTXwzUTZCIkIJvMa3m4vyfNbklxwl3L+0bYmRiHIPErB pKkQ== X-Gm-Message-State: AOJu0YwEGFI5rhom72+fx5IJKkuDFguwfXC9KXrW+ehxTxnw2o07m0/I 2WMfjNw9p1TgJX4l38JyWhMbKyCWVNeskyj85AjGAeBa2mlaUuuIw2QOUint X-Google-Smtp-Source: AGHT+IHVAVMiwoUjTMFjcMx6W+8Xkmf1im8rsfJvRbixEXzcJzc/tFnmGBMrRy25h190asWXlmVTLA== X-Received: by 2002:a05:600c:4708:b0:426:6308:e2f0 with SMTP id 5b1f17b1804b1-4319ad024d7mr110774825e9.26.1730196977399; Tue, 29 Oct 2024 03:16:17 -0700 (PDT) Received: from im-t490s.redhat.com (ip-86-49-44-151.bb.vodafone.cz. [86.49.44.151]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-431b4594ec3sm20279685e9.1.2024.10.29.03.16.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Oct 2024 03:16:17 -0700 (PDT) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Tue, 29 Oct 2024 11:15:00 +0100 Message-ID: <20241029101608.2991596-3-i.maximets@ovn.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241029101608.2991596-1-i.maximets@ovn.org> References: <20241029101608.2991596-1-i.maximets@ovn.org> MIME-Version: 1.0 Subject: [ovs-dev] [PATCH 2/9] ipsec: libreswan: Reconcile missing connections periodically. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ilya Maximets Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" There are cases where ipsec commands may fail to add new connections or remove the old ones. Unfortunately, this means that those connections may actually never be added or removed, since ovs-monitor-ipsec will not re-visit them, unless something else changes. Wake up the monitor periodically to check if something changed in the system or if some connections still need loading. This addresses two main use cases: 1. Connection failed to start for some reason and was not added to pluto or properly started. The logic will go over all the desired, loaded and active connections and make sure that any undesired connections are removed, non-loaded connections are loaded and non-active connections are brought UP. 2. If pluto re-starts it loads all the connections, but doesn't bring them up, because we're using route (ondemand) activation strategy. This change in this commit will notice all the loaded but not active connections and will bring them up. This helps avoiding packet drops on first packets until the connection activates. Choosing 15 seconds as an interval to wake up to give pluto some breathing room, i.e. a chance to activate the connections properly before we start poking them. And also if pluto is down, 15 second interval will create less spam in the logs. StrongSwan doesn't need such a logic, because it supports a single command 'ipsec update' that re-loads the config as a whole and figures out what configuration changes are needed. But since we're starting all the connections separately with Libreswan, we have to keep track and reconcile manually. Some more details of the logic are in the comments in the code. Signed-off-by: Ilya Maximets --- ipsec/ovs-monitor-ipsec.in | 178 ++++++++++++++++++++++++------------- 1 file changed, 116 insertions(+), 62 deletions(-) diff --git a/ipsec/ovs-monitor-ipsec.in b/ipsec/ovs-monitor-ipsec.in index 19a401609..09a29e2ca 100755 --- a/ipsec/ovs-monitor-ipsec.in +++ b/ipsec/ovs-monitor-ipsec.in @@ -295,6 +295,9 @@ conn prevent_unencrypted_vxlan return conns + def need_to_reconcile(self, monitor): + return False + def config_init(self): self.conf_file = open(self.IPSEC_CONF, "w") self.secrets_file = open(self.IPSEC_SECRETS, "w") @@ -622,51 +625,50 @@ conn prevent_unencrypted_vxlan "--config", self.IPSEC_CONF, "--rereadsecrets"], "re-read secrets") - tunnels = set(monitor.tunnels.keys()) - - # Delete old connections - conns_dict = self.get_active_conns() - for ifname, conns in conns_dict.items(): - tunnel = monitor.tunnels.get(ifname) - - for conn in conns: - # IPsec "connection" names must start with Interface name - if not conn.startswith(ifname): - vlog.err("%s does not start with %s" % (conn, ifname)) - continue - # version number should be the first integer after - # interface name in IPsec "connection" - try: - ver = int(re.findall(r'\d+', conn[len(ifname):])[0]) - except ValueError: - vlog.err("%s does not contain version number") - continue - except IndexError: - vlog.err("%s does not contain version number") - continue - - if not tunnel or tunnel.version != ver: - vlog.info("%s is outdated %u" % (conn, ver)) - run_command(self.IPSEC_AUTO + - ["--ctlsocket", self.IPSEC_CTL, - "--config", self.IPSEC_CONF, - "--delete", conn], "delete %s" % conn) - elif ifname in tunnels: - tunnels.remove(ifname) - - # Activate new connections - for name in tunnels: - ver = monitor.tunnels[name].version - - if monitor.tunnels[name].conf["tunnel_type"] == "gre": - conn = "%s-%s" % (name, ver) - self._start_ipsec_connection(conn) + loaded_conns = self.get_loaded_conns() + active_conns = self.get_active_conns() + + all_names = set(monitor.tunnels.keys()) | \ + set(loaded_conns.keys()) | \ + set(active_conns.keys()) + + for name in all_names: + desired = set(self.get_conn_names(monitor, name)) + loaded = set(loaded_conns.get(name, dict()).keys()) + active = set(active_conns.get(name, dict()).keys()) + + # Remove all the loaded or active but not desired connections. + for conn in loaded | active: + if conn not in desired: + self._delete_ipsec_connection(conn, "is outdated") + loaded.discard(conn) + active.discard(conn) + + # If not all desired are loaded, remove all the loaded and + # active for this tunnel and re-load only the desired ones. + # Need to do that, because connections for the same tunnel + # may share SAs. If one is loaded and the other is not, + # it means the second one failed, so the shared SA may be in + # a broken state. + if desired != loaded: + for conn in loaded | active: + self._delete_ipsec_connection(conn, "is half-loaded") + loaded.discard(conn) + active.discard(conn) + + for conn in desired: + vlog.info("Starting ipsec connection %s" % conn) + self._start_ipsec_connection(conn, "start") else: - conn_in = "%s-in-%s" % (name, ver) - conn_out = "%s-out-%s" % (name, ver) - self._start_ipsec_connection(conn_in) - self._start_ipsec_connection(conn_out) + # Ask pluto to bring UP connections that are loaded, + # but not active for some reason. + # + # desired == loaded and desired >= loaded + active, + # so loaded >= active + for conn in loaded - active: + vlog.info("Bringing up ipsec connection %s" % conn) + self._start_ipsec_connection(conn, "up") # Update shunt policy if changed if monitor.conf_in_use["skb_mark"] != monitor.conf["skb_mark"]: @@ -713,23 +715,26 @@ conn prevent_unencrypted_vxlan "--delete", "--asynchronous", "prevent_unencrypted_vxlan"]) monitor.conf_in_use["skb_mark"] = monitor.conf["skb_mark"] + vlog.info("Refreshing is done.") - def get_active_conns(self): + def get_conns_from_status(self, pattern): """This function parses output from 'ipsec status' command. It returns dictionary where is interface name (as in OVSDB) and is another dictionary. This another dictionary uses LibreSwan connection name as and more detailed - sample line from the parsed outpus as . """ + sample line from the parsed outpus as . 'pattern' should + be a regular expression that parses out the connection name. + Only the lines that match the pattern will be parsed. """ conns = {} ret, pout, perr = run_command([self.IPSEC, 'status', '--ctlsocket', self.IPSEC_CTL], - "get active connections") + "get ipsec status") if ret: return conns for line in pout.decode().splitlines(): - m = re.search(r"#\d+: \"(.*)\".*", line) + m = re.search(pattern, line) if not m: continue @@ -748,25 +753,73 @@ conn prevent_unencrypted_vxlan return conns - def _start_ipsec_connection(self, conn): - # In a corner case, LibreSwan daemon restarts for some reason and - # the "ipsec auto --start" command is lost. Just retry to make sure - # the command is received by LibreSwan. - while True: - ret, pout, perr = run_command(self.IPSEC_AUTO + - ["--config", self.IPSEC_CONF, - "--ctlsocket", self.IPSEC_CTL, - "--start", - "--asynchronous", conn], - "start %s" % conn) - if not re.match(r".*Connection refused.*", perr.decode()) and \ - not re.match(r".*need --listen.*", pout.decode()): - break + def get_active_conns(self): + return self.get_conns_from_status(r"#\d+: \"(.*)\".*") + + def get_loaded_conns(self): + return self.get_conns_from_status(r"\"(.*)\":.*(===|---).*") + + def get_conn_names(self, monitor, ifname): + conns = [] + if ifname not in monitor.tunnels: + return conns + + tunnel = monitor.tunnels.get(ifname) + ver = tunnel.version + + if tunnel.conf["tunnel_type"] == "gre": + conns.append("%s-%s" % (ifname, ver)) + else: + conns.append("%s-in-%s" % (ifname, ver)) + conns.append("%s-out-%s" % (ifname, ver)) + + return conns + + def need_to_reconcile(self, monitor): + conns_dict = self.get_active_conns() + for ifname, tunnel in monitor.tunnels.items(): + if ifname not in conns_dict: + vlog.info("Connection for port %s is not active, " + "need to reconcile" % ifname) + return True + + existing_conns = conns_dict.get(ifname) + desired_conns = self.get_conn_names(monitor, ifname) + + if set(existing_conns.keys()) != set(desired_conns): + vlog.info("Active connections for port %s %s do not match " + "desired %s, need to reconcile" + % (ifname, list(existing_conns.keys()), + desired_conns)) + return True + + return False + + def _delete_ipsec_connection(self, conn, reason): + vlog.info("%s %s, removing" % (conn, reason)) + run_command(self.IPSEC_AUTO + + ["--ctlsocket", self.IPSEC_CTL, + "--config", self.IPSEC_CONF, + "--delete", conn], "delete %s" % conn) + + def _start_ipsec_connection(self, conn, action): + ret, pout, perr = run_command(self.IPSEC_AUTO + + ["--config", self.IPSEC_CONF, + "--ctlsocket", self.IPSEC_CTL, + "--" + action, + "--asynchronous", conn], + "%s %s" % (action, conn)) if re.match(r".*[F|f]ailed to initiate connection.*", pout.decode()): vlog.err('Failed to initiate connection through' ' Interface %s.\n' % (conn.split('-')[0])) vlog.err("stdout: %s" % pout) + ret = 1 + + if ret: + # We don't know in which state the connection was left on + # failure. Try to clean it up. + self._delete_ipsec_connection(conn, "--%s failed" % action) def _nss_clear_database(self): """Remove all OVS IPsec related state from the NSS database""" @@ -1192,7 +1245,7 @@ class IPsecMonitor(object): self.ike_helper.clear_tunnel_state(self.tunnels[name]) del self.tunnels[name] - if needs_refresh: + if needs_refresh or self.ike_helper.need_to_reconcile(self): self.ike_helper.refresh(self) def _get_cn_from_cert(self, cert): @@ -1365,6 +1418,7 @@ def main(): poller = ovs.poller.Poller() unixctl_server.wait(poller) idl.wait(poller) + poller.timer_wait(15000) poller.block() unixctl_server.close() From patchwork Tue Oct 29 10:15:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 2003664 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Xd5l354QYz1xxC for ; Tue, 29 Oct 2024 21:16:31 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 1C42140F59; Tue, 29 Oct 2024 10:16:29 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id N1C4KiWckYHE; Tue, 29 Oct 2024 10:16:27 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.9.56; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org B42D740DEF Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id B42D740DEF; Tue, 29 Oct 2024 10:16:27 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 8387BC08A8; Tue, 29 Oct 2024 10:16:27 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 92D54C08A3 for ; Tue, 29 Oct 2024 10:16:26 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 5D27A80D91 for ; Tue, 29 Oct 2024 10:16:26 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id AxUl-LgpGR0U for ; Tue, 29 Oct 2024 10:16:24 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.167.67; helo=mail-lf1-f67.google.com; envelope-from=i.maximets.ovn@gmail.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp1.osuosl.org C1C6E80D69 Authentication-Results: smtp1.osuosl.org; dmarc=none (p=none dis=none) header.from=ovn.org DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org C1C6E80D69 Received: from mail-lf1-f67.google.com (mail-lf1-f67.google.com [209.85.167.67]) by smtp1.osuosl.org (Postfix) with ESMTPS id C1C6E80D69 for ; Tue, 29 Oct 2024 10:16:23 +0000 (UTC) Received: by mail-lf1-f67.google.com with SMTP id 2adb3069b0e04-539f4d8ef66so6870979e87.1 for ; Tue, 29 Oct 2024 03:16:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730196981; x=1730801781; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BMOy3Peu9UkbUk73zlOC4DDNp3HINhlNfVwEe3M71SI=; b=YRMhI+zEFufWWVEwseAHFC+v8mmBKI3tkiT9BjCHkMr+lXZqEegtq8OasdEQ2lwDrq IggQoB66S3d4gbEDdmHJybr6OWp6Nz6ZmAEB/xiy8esnr7WFn9yf1X3nnPtllmfm+iHc kmVozbKGl4UGUiG1/1QiPgMKuORrBvLHyp/INJ4zXWyOEQE3yGIcqSq2s0zLspHmcOUH B4hCZIfM7Z6gbO0uNe4YsVXUjQ8VLrYrVEYhayYMjv+EodCP1jPhxV9ML18sFkKtyr3F ewchpBIE1TDdls7ppQb4DcAOPUdoG4Vs2ZbKh0+xWPMtPYURdED7mYX8NCP1P4H02R2Q WWkQ== X-Gm-Message-State: AOJu0YzggqNaFlg17j4PzF4ixc7LiSX+lHZg2ZqsL6P7E5aL/szgKC8r JkHBIMeD+zZIKBcFmsFtD7LRpGC7YmJe/Fduj5iKEFvcDak+JaNSrp20AAHC X-Google-Smtp-Source: AGHT+IEbbJLjbrAXaSTEwJK8lS9DP0yTsjOf4ZeaTLY2Ci2fUz+lo3wP8Nn3C3a7sgVTce7o8Ad2Iw== X-Received: by 2002:a05:6512:1188:b0:539:fc26:74bd with SMTP id 2adb3069b0e04-53b3490ee7dmr4730175e87.27.1730196979570; Tue, 29 Oct 2024 03:16:19 -0700 (PDT) Received: from im-t490s.redhat.com (ip-86-49-44-151.bb.vodafone.cz. [86.49.44.151]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-431b4594ec3sm20279685e9.1.2024.10.29.03.16.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Oct 2024 03:16:19 -0700 (PDT) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Tue, 29 Oct 2024 11:15:01 +0100 Message-ID: <20241029101608.2991596-4-i.maximets@ovn.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241029101608.2991596-1-i.maximets@ovn.org> References: <20241029101608.2991596-1-i.maximets@ovn.org> MIME-Version: 1.0 Subject: [ovs-dev] [PATCH 3/9] ipsec: libreswan: Try to bring non-active connections up. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ilya Maximets Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Sometimes connections are getting loaded, but do not become active for some reason on a first try. We can try and bring them up manually. However, if they are still not active after that, it's better to just remove the connection and try to add them from scratch, as there must be some internal issue in libreswan that doesn't allow these connections to actually become active. Note: Once the "defunct" connection is removed, the second connection for the same tunnel will also be removed as "half-loaded". This ensures that all the shared SAs will also be cleaned up, so we can truly start from scratch. Signed-off-by: Ilya Maximets --- ipsec/ovs-monitor-ipsec.in | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/ipsec/ovs-monitor-ipsec.in b/ipsec/ovs-monitor-ipsec.in index 09a29e2ca..d667c08bc 100755 --- a/ipsec/ovs-monitor-ipsec.in +++ b/ipsec/ovs-monitor-ipsec.in @@ -514,6 +514,7 @@ conn prevent_unencrypted_vxlan self.IPSEC_D = "sql:" + libreswan_root_prefix + ipsec_d self.IPSEC_CTL = libreswan_root_prefix + ipsec_ctl self.conf_file = None + self.conns_not_active = set() self.secrets_file = None vlog.dbg("Using: " + self.IPSEC) vlog.dbg("Configuration file: " + self.IPSEC_CONF) @@ -638,6 +639,14 @@ conn prevent_unencrypted_vxlan loaded = set(loaded_conns.get(name, dict()).keys()) active = set(active_conns.get(name, dict()).keys()) + # Untrack connections that became active. + self.conns_not_active.difference_update(active) + # Remove connections that didn't become active after --start + # and another explicit --up. + for conn in self.conns_not_active & loaded: + self._delete_ipsec_connection(conn, "is defunct") + loaded.remove(conn) + # Remove all the loaded or active but not desired connections. for conn in loaded | active: if conn not in desired: @@ -668,6 +677,8 @@ conn prevent_unencrypted_vxlan # so loaded >= active for conn in loaded - active: vlog.info("Bringing up ipsec connection %s" % conn) + # On failure to --up it will be removed from the set. + self.conns_not_active.add(conn) self._start_ipsec_connection(conn, "up") # Update shunt policy if changed @@ -797,6 +808,7 @@ conn prevent_unencrypted_vxlan def _delete_ipsec_connection(self, conn, reason): vlog.info("%s %s, removing" % (conn, reason)) + self.conns_not_active.discard(conn) run_command(self.IPSEC_AUTO + ["--ctlsocket", self.IPSEC_CTL, "--config", self.IPSEC_CONF, From patchwork Tue Oct 29 10:15:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 2003666 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Xd5l94C0lz1xxC for ; Tue, 29 Oct 2024 21:16:37 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id C7E444076C; Tue, 29 Oct 2024 10:16:35 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id XasCG88ezoyy; Tue, 29 Oct 2024 10:16:34 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=2605:bc80:3010:104::8cd3:938; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 791424088E Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id 791424088E; Tue, 29 Oct 2024 10:16:34 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 22635C08AA; Tue, 29 Oct 2024 10:16:34 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 11974C08A9 for ; Tue, 29 Oct 2024 10:16:33 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 25E2860D67 for ; Tue, 29 Oct 2024 10:16:29 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id 0MrsnXA_cGw0 for ; Tue, 29 Oct 2024 10:16:27 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.167.66; helo=mail-lf1-f66.google.com; envelope-from=i.maximets.ovn@gmail.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp3.osuosl.org F2C1C60D5D Authentication-Results: smtp3.osuosl.org; dmarc=none (p=none dis=none) header.from=ovn.org DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org F2C1C60D5D Received: from mail-lf1-f66.google.com (mail-lf1-f66.google.com [209.85.167.66]) by smtp3.osuosl.org (Postfix) with ESMTPS id F2C1C60D5D for ; Tue, 29 Oct 2024 10:16:25 +0000 (UTC) Received: by mail-lf1-f66.google.com with SMTP id 2adb3069b0e04-539f0f9ee49so5593483e87.1 for ; Tue, 29 Oct 2024 03:16:25 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730196982; x=1730801782; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ExnzGMXQlKnNTYcuh0dGEVNPSlvLcVyJRnJ+AkcF2Fc=; b=MnHvf1f/6FPCn2N7HpTnhRztMDFOH0p7ZgurjuxwxEd+resLRu9NXnU2ONJN+8iKZk m77UMrqmNnuka1xKcIM7Lr7tN1iZfWU3S9UouvIrxwwVqFvqA/FEd6fqswC00S6qRt/t 2VPWp/iJmtNeHz0B0ZSYdkuz9x0uzv2IRuf1GqfWamyEO+71LZn5s5oXizM8a7E9jUax MRFkCLDHPMJ6kGaxfIv4A+vXg3LP/IcoClc9WoOTewJN2JEQ2NC/UvXRw/nCPHULHLwY uHZOs9FhfXP4tGPUkw5Cn9LLtaY33u5hfM3jh4ygX7vNR4tmsE+TbnUfD3q/uP88uPIN XU9Q== X-Gm-Message-State: AOJu0YzDPqYHt22r/SbjapWm2aNCUFY5nKPtp+XZPfOTQZMgY2hwIJUW P6gG7uYTaC5+YeHLc6VnuPLGmA8n9ITL3AELdfY7nXu29c/84uaknRHhEwWl X-Google-Smtp-Source: AGHT+IFoWG40VQX3zmBAigRPrsOGkKPrGJ/bYVr0//B9DlASwNxJwyLqZ/4sspRwWmlw4//248ZFvg== X-Received: by 2002:a05:6512:2385:b0:539:e14b:309c with SMTP id 2adb3069b0e04-53b348cb1c6mr4477163e87.19.1730196982092; Tue, 29 Oct 2024 03:16:22 -0700 (PDT) Received: from im-t490s.redhat.com (ip-86-49-44-151.bb.vodafone.cz. [86.49.44.151]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-431b4594ec3sm20279685e9.1.2024.10.29.03.16.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Oct 2024 03:16:21 -0700 (PDT) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Tue, 29 Oct 2024 11:15:02 +0100 Message-ID: <20241029101608.2991596-5-i.maximets@ovn.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241029101608.2991596-1-i.maximets@ovn.org> References: <20241029101608.2991596-1-i.maximets@ovn.org> MIME-Version: 1.0 Subject: [ovs-dev] [PATCH 4/9] ipsec: libreswan: Fix regexp for connections waiting on child SA. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ilya Maximets Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" These should be considered active, because pluto is waiting for the other side to react. We should not remove them or try to repair. Such connections have an extra text between the SA number and the name of the connection. Ideally, we would like not to parse the output of ipsec status, since it's very error prone, but there is, unfortunately, no other interface. Signed-off-by: Ilya Maximets --- ipsec/ovs-monitor-ipsec.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ipsec/ovs-monitor-ipsec.in b/ipsec/ovs-monitor-ipsec.in index d667c08bc..3b2057389 100755 --- a/ipsec/ovs-monitor-ipsec.in +++ b/ipsec/ovs-monitor-ipsec.in @@ -765,7 +765,7 @@ conn prevent_unencrypted_vxlan return conns def get_active_conns(self): - return self.get_conns_from_status(r"#\d+: \"(.*)\".*") + return self.get_conns_from_status(r"#\d+: .*\"(.*)\".*") def get_loaded_conns(self): return self.get_conns_from_status(r"\"(.*)\":.*(===|---).*") From patchwork Tue Oct 29 10:15:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 2003665 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Xd5l90nrqz1xwn for ; Tue, 29 Oct 2024 21:16:37 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 7F5C98121F; Tue, 29 Oct 2024 10:16:35 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id WfUkbkX8hGgn; Tue, 29 Oct 2024 10:16:32 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.9.56; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 3E84581224 Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTPS id 3E84581224; Tue, 29 Oct 2024 10:16:32 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id F072EC08A6; Tue, 29 Oct 2024 10:16:31 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 3C316C08B4 for ; Tue, 29 Oct 2024 10:16:30 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id CD7D940AF6 for ; Tue, 29 Oct 2024 10:16:28 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id jw-deEy3zn4o for ; Tue, 29 Oct 2024 10:16:27 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.167.65; helo=mail-lf1-f65.google.com; envelope-from=i.maximets.ovn@gmail.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp2.osuosl.org 6E34340B01 Authentication-Results: smtp2.osuosl.org; dmarc=none (p=none dis=none) header.from=ovn.org DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 6E34340B01 Received: from mail-lf1-f65.google.com (mail-lf1-f65.google.com [209.85.167.65]) by smtp2.osuosl.org (Postfix) with ESMTPS id 6E34340B01 for ; Tue, 29 Oct 2024 10:16:27 +0000 (UTC) Received: by mail-lf1-f65.google.com with SMTP id 2adb3069b0e04-539f84907caso5596280e87.3 for ; Tue, 29 Oct 2024 03:16:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730196985; x=1730801785; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ruUEmioI/vAPH/6Effwwy/4loMD2TIqW+V7yCVR/9Nk=; b=nsDkCm2e6slSGK/gJUodD7smuK7Q8cvnVmZbgeMW23XS6zZqXsZhhRyBmgjEwBKF9t M5QBiOsDZ/qaInXYpj9hyjM6WHSz7Ds4Ezsq6tvyPVZT1OyHVXGDxHdzEk+CHSgcxaSn xlK9JLKbgkz/xlk53CAaHlS+ZGo6QO3UM8VWIK5smRLOL5/lu+DJRqiCjN0JA3fc3j1T IGASB+fqzHOzwFwFuq3yZpZ+QWIjM/2ThvbD28Nh5/iw91ttBU0WVq2V044WekKKGwD3 opNOZbsIjlUQ2YBZ/pJ1ChweeFOsL4wK8uDnA6iXEMTqexu0NMTtEpaBYZdRLayXdJmp 03Tg== X-Gm-Message-State: AOJu0YwL/76r0pYne4pDgmd/roHkTdrulIcxRpnCtdPpN+Q7cSfUsuBR 1e9OfbVLLZ1y3WGjyj7h0++YadSTdIhMAn+22d0Ltfkezx0zh0TJ75IE2tAt X-Google-Smtp-Source: AGHT+IGttqMo9aOxiNhWuJTdeTi0GBqKFN1PPTRnP/yOQTAPnAUMYzgxt8vp4TCr4U0iQOk9ubIc5Q== X-Received: by 2002:a05:6512:3e26:b0:53b:2114:92a7 with SMTP id 2adb3069b0e04-53b34c8e61fmr5778743e87.52.1730196984488; Tue, 29 Oct 2024 03:16:24 -0700 (PDT) Received: from im-t490s.redhat.com (ip-86-49-44-151.bb.vodafone.cz. [86.49.44.151]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-431b4594ec3sm20279685e9.1.2024.10.29.03.16.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Oct 2024 03:16:24 -0700 (PDT) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Tue, 29 Oct 2024 11:15:03 +0100 Message-ID: <20241029101608.2991596-6-i.maximets@ovn.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241029101608.2991596-1-i.maximets@ovn.org> References: <20241029101608.2991596-1-i.maximets@ovn.org> MIME-Version: 1.0 Subject: [ovs-dev] [PATCH 5/9] ipsec: libreswan: Avoid monitor hanging on stuck ipsec commands. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ilya Maximets Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Multiple versions of Libreswan have an issue where ipsec --start command may get stuck forever. This issue affects many popular versions of Libreswan from 4.5 to 4.15, which are shipped in most modern distributions. When ipsec --start gets stuck, ovs-monitor-ipsec hangs and can't do anything else, so not olny this one but all other tunnels are also not being started. Add a timeout to the subprocess call, so we do not wait forever. Just introduced reconciliation process will clean things up and will try to re-add this connection later. Pluto may take a lot of time to process the --start request. Notably, the time depends on the retransmission timeout, which is 60 seconds by default. However, even at high scale, it doesn't take much more than that in tests. So, 120 second timeout should be a reasonable default value. Note: it is observed in practice that the process doesn't actually terminate for a long time, so we can't afford waiting for it. That's the main reason why we're not using the subprocess.run() with a timeout option here (it would wait). But also, because we'd had to catch the exception anyway. Reported-at: https://issues.redhat.com/browse/FDP-846 Signed-off-by: Ilya Maximets --- ipsec/ovs-monitor-ipsec.in | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/ipsec/ovs-monitor-ipsec.in b/ipsec/ovs-monitor-ipsec.in index 3b2057389..264b055e6 100755 --- a/ipsec/ovs-monitor-ipsec.in +++ b/ipsec/ovs-monitor-ipsec.in @@ -82,6 +82,7 @@ vlog = ovs.vlog.Vlog("ovs-monitor-ipsec") exiting = False monitor = None xfrm = None +TIEMOUT_EXPIRED = 37 def run_command(args, description=None): @@ -94,7 +95,16 @@ def run_command(args, description=None): vlog.dbg("Running %s" % args) proc = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) - pout, perr = proc.communicate() + try: + pout, perr = proc.communicate(timeout=120) + ret = proc.returncode + except subprocess.TimeoutExpired: + vlog.warn("Command timed out trying to %s." % description) + pout, perr = b'', b'' + # Just kill the process here. We can't afford waiting for it, + # as it may be stuck and may not actually be terminated. + proc.kill() + ret = TIEMOUT_EXPIRED if proc.returncode or len(perr): vlog.warn("Failed to %s; exit code: %d" @@ -103,7 +113,7 @@ def run_command(args, description=None): vlog.warn("stderr: %s" % perr) vlog.warn("stdout: %s" % pout) - return proc.returncode, pout or b'', perr or b'' + return ret, pout or b'', perr or b'' class XFRM(object): From patchwork Tue Oct 29 10:15:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 2003667 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Xd5lN3NJFz1xwn for ; Tue, 29 Oct 2024 21:16:48 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 82B4F81357; Tue, 29 Oct 2024 10:16:46 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id l1YdIbum_Xyg; Tue, 29 Oct 2024 10:16:43 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.9.56; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 6F53381398 Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTPS id 6F53381398; Tue, 29 Oct 2024 10:16:40 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 993A4C08A6; Tue, 29 Oct 2024 10:16:40 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1D2A6C08A9 for ; Tue, 29 Oct 2024 10:16:39 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 1F29960D5A for ; Tue, 29 Oct 2024 10:16:30 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id SBIqh_cGKwLi for ; Tue, 29 Oct 2024 10:16:29 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.128.67; helo=mail-wm1-f67.google.com; envelope-from=i.maximets.ovn@gmail.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp3.osuosl.org ECEE760D4E Authentication-Results: smtp3.osuosl.org; dmarc=none (p=none dis=none) header.from=ovn.org DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org ECEE760D4E Received: from mail-wm1-f67.google.com (mail-wm1-f67.google.com [209.85.128.67]) by smtp3.osuosl.org (Postfix) with ESMTPS id ECEE760D4E for ; Tue, 29 Oct 2024 10:16:28 +0000 (UTC) Received: by mail-wm1-f67.google.com with SMTP id 5b1f17b1804b1-4315e62afe0so54646745e9.1 for ; Tue, 29 Oct 2024 03:16:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730196987; x=1730801787; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PFSGRtxaqrBEsIz4+N1CsbhK2kmPw0W1sbAnOLkfGxw=; b=KvCdy7qWrPjb4OJOD5rYVla7f3CxSDnfAcMNmtwwb0q1DJXVcM+p4IeAEVaHkYyKFI M9ofbb4FlPKj/HMKQZOnVkPZGZ3Joic2C1HYCveebpqsNvWi92reSgdex8qq3og3pfzx K5FLiDYJdXQ198OcXbKG6pzBED6Ws2mNhI9C3pZTCbB+ohZ+Y639Run/+5Ak3xwJ4FQj kb980UO86VIsX4n51qIV/AnFXiXLw/2Qm0xYD7990HKLgA48m8s3E6iYpZQsoHSrbB4Y tT7jvTzFHtjpdNRYp+aUbkT6jfWa9AklmsGTi5ryjLq4BNJcdDGwzVBwRgxH/sEasHfx GcQA== X-Gm-Message-State: AOJu0Yzj5t3lDZZ3+zJzDAEtW7BZSWBJUNdnWoWJiip3XgiifnGnzFur WKeDrV8Lc0jJP+b3dA52/aGg2/AwtxUIqrX5mM/JpkaeYIpfzETDD/GNdkoo X-Google-Smtp-Source: AGHT+IGHliLzc58i6NAL7an1FcjAB0+BLesZH2IWqAPzUbcMLoOUgDRAx5t9OMCcWGYlKAPdHvnb3g== X-Received: by 2002:a05:600c:4e46:b0:430:52ec:1e2b with SMTP id 5b1f17b1804b1-4319ad2b89emr105838045e9.29.1730196986614; Tue, 29 Oct 2024 03:16:26 -0700 (PDT) Received: from im-t490s.redhat.com (ip-86-49-44-151.bb.vodafone.cz. [86.49.44.151]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-431b4594ec3sm20279685e9.1.2024.10.29.03.16.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Oct 2024 03:16:26 -0700 (PDT) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Tue, 29 Oct 2024 11:15:04 +0100 Message-ID: <20241029101608.2991596-7-i.maximets@ovn.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241029101608.2991596-1-i.maximets@ovn.org> References: <20241029101608.2991596-1-i.maximets@ovn.org> MIME-Version: 1.0 Subject: [ovs-dev] [PATCH 6/9] ipsec: Make command timeout configurable. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ilya Maximets Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add a new command line option --command-timeout that controls the command timeout. It is important to have this configurable, because the retransmit-timeout is configurable in Libreswan. Also, users may prefer the monitor to be more responsive. ovs-monitor-ipsec options are not documented anywhere, so not trying to address that here. Signed-off-by: Ilya Maximets --- ipsec/ovs-monitor-ipsec.in | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/ipsec/ovs-monitor-ipsec.in b/ipsec/ovs-monitor-ipsec.in index 264b055e6..2912075d7 100755 --- a/ipsec/ovs-monitor-ipsec.in +++ b/ipsec/ovs-monitor-ipsec.in @@ -82,6 +82,7 @@ vlog = ovs.vlog.Vlog("ovs-monitor-ipsec") exiting = False monitor = None xfrm = None +command_timeout = None TIEMOUT_EXPIRED = 37 @@ -96,7 +97,7 @@ def run_command(args, description=None): proc = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) try: - pout, perr = proc.communicate(timeout=120) + pout, perr = proc.communicate(timeout=command_timeout) ret = proc.returncode except subprocess.TimeoutExpired: vlog.warn("Command timed out trying to %s." % description) @@ -1380,6 +1381,10 @@ def main(): parser.add_argument("--ipsec-ctl", metavar="IPSEC-CTL", help="Use DIR/IPSEC-CTL as location for " " pluto ctl socket (libreswan only).") + parser.add_argument("--command-timeout", metavar="TIMEOUT", + type=int, default=120, + help="Timeout for external commands called by the " + "ovs-monitor-ipsec daemon, e.g. ipsec --start.") ovs.vlog.add_args(parser) ovs.daemon.add_args(parser) @@ -1389,11 +1394,13 @@ def main(): global monitor global xfrm + global command_timeout root_prefix = args.root_prefix if args.root_prefix else "" xfrm = XFRM(root_prefix) monitor = IPsecMonitor(root_prefix, args.ike_daemon, not args.no_restart_ike_daemon, args) + command_timeout = args.command_timeout remote = args.database schema_helper = ovs.db.idl.SchemaHelper() From patchwork Tue Oct 29 10:15:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 2003668 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Xd5lS4mtNz1xwn for ; Tue, 29 Oct 2024 21:16:52 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 1A40860F13; Tue, 29 Oct 2024 10:16:51 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id KTB37-L_nSnr; Tue, 29 Oct 2024 10:16:48 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.9.56; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 4AAF960A8F Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 4AAF960A8F; Tue, 29 Oct 2024 10:16:48 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id CEDEDC08A6; Tue, 29 Oct 2024 10:16:47 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 20CA7C08A9 for ; Tue, 29 Oct 2024 10:16:46 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id CA3A0811B0 for ; Tue, 29 Oct 2024 10:16:32 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id BgCABTnpDyRL for ; Tue, 29 Oct 2024 10:16:31 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.128.67; helo=mail-wm1-f67.google.com; envelope-from=i.maximets.ovn@gmail.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp1.osuosl.org 36BC8811BB Authentication-Results: smtp1.osuosl.org; dmarc=none (p=none dis=none) header.from=ovn.org DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 36BC8811BB Received: from mail-wm1-f67.google.com (mail-wm1-f67.google.com [209.85.128.67]) by smtp1.osuosl.org (Postfix) with ESMTPS id 36BC8811BB for ; Tue, 29 Oct 2024 10:16:31 +0000 (UTC) Received: by mail-wm1-f67.google.com with SMTP id 5b1f17b1804b1-4315baa51d8so52290425e9.0 for ; Tue, 29 Oct 2024 03:16:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730196989; x=1730801789; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OZebmDeZp5nc1MQzZgRIPjTSpmq5vUKcToSoeiDMkOY=; b=J1vi62+G0ZFGl7dYvnpHSWMXATgyH8Sqt6E9nbb85Mef6UFujhSDnR1ukBZTiJXu8r dGqgot5b63+WmhyOBNCe0z9j+yDuGah0r7l1iITJD57RL+NFymIcj9JqG6IHa/kr/4td US2Hj3RmkQCAoxG8xsB+8dfWfUUbcNTn0QGuCODkawA1KrE8xf5yNwxwU4sl+CMT6stt 5a6bcxx82/8DhK8j95DnrTuA0vSyIGV8wzgrVEWTvuvVTwfGgXc2wQnFDYKGH4+yTkhQ MMHELh27FqEuy9ECAv4/bghVliWRSVWjN+jjedmJYZ9O5jzyS/FJ1jJuhOOfqSAnXfhj 1z9g== X-Gm-Message-State: AOJu0YyLmA/RvKHVbexLq8cyWD7Qy0wD45kBUzx/wHEx69eN9BkPXzQp T/gctlqzDmqdQQSzOEDlc38ayJj5t/WqQu+sSNWtDx2BhrHdtsqoQiGIGJOk X-Google-Smtp-Source: AGHT+IGQIiuTjyIxMeWIPzeWGMx0cHCuMTdJBdIOnBqgQT/qesJ/OwI8DfOhqVne1WI1LkmJ2VNhgA== X-Received: by 2002:a05:600c:548b:b0:42c:b5f1:44ff with SMTP id 5b1f17b1804b1-4319ad0ca37mr94086255e9.24.1730196988999; Tue, 29 Oct 2024 03:16:28 -0700 (PDT) Received: from im-t490s.redhat.com (ip-86-49-44-151.bb.vodafone.cz. [86.49.44.151]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-431b4594ec3sm20279685e9.1.2024.10.29.03.16.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Oct 2024 03:16:28 -0700 (PDT) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Tue, 29 Oct 2024 11:15:05 +0100 Message-ID: <20241029101608.2991596-8-i.maximets@ovn.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241029101608.2991596-1-i.maximets@ovn.org> References: <20241029101608.2991596-1-i.maximets@ovn.org> MIME-Version: 1.0 Subject: [ovs-dev] [PATCH 7/9] system-tests: Verbose cleanup of ports and namespaces. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ilya Maximets Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Removal of ports and network namespaces can take a significant amount of time, and it is not clear if the test is stuck or actually doing something during that time. Add some logging to cleanup commands to see what is going on. Signed-off-by: Ilya Maximets --- tests/system-common-macros.at | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/tests/system-common-macros.at b/tests/system-common-macros.at index e9be021f3..ff86d15cd 100644 --- a/tests/system-common-macros.at +++ b/tests/system-common-macros.at @@ -2,10 +2,7 @@ # # Delete namespaces from the running OS m4_define([DEL_NAMESPACES], - [m4_foreach([ns], [$@], - [ip netns del ns -]) - ] + [m4_foreach([ns], [$@], [echo removing namespace ns; ip netns del ns])] ) # ADD_NAMESPACES(ns [, ns ... ]) @@ -72,7 +69,7 @@ m4_define([ADD_INT], # m4_define([ADD_VETH], [ AT_CHECK([ip link add $1 type veth peer name ovs-$1 || return 77]) - on_exit 'ip link del ovs-$1' + on_exit 'echo removing interface ovs-$1; ip link del ovs-$1' CONFIGURE_VETH_OFFLOADS([$1]) AT_CHECK([ip link set $1 netns $2]) AT_CHECK([ip link set dev ovs-$1 up]) From patchwork Tue Oct 29 10:15:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 2003669 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Xd5lj4PVcz1xwn for ; Tue, 29 Oct 2024 21:17:05 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id ACDD260AE3; Tue, 29 Oct 2024 10:17:03 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id Xsz6iKRpITFF; Tue, 29 Oct 2024 10:16:56 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.9.56; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 891F060EF5 Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 891F060EF5; Tue, 29 Oct 2024 10:16:52 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id CD74AC08A3; Tue, 29 Oct 2024 10:16:51 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 3F573C08A3 for ; Tue, 29 Oct 2024 10:16:50 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 588ED40917 for ; Tue, 29 Oct 2024 10:16:35 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id TPfxA-VwhlL7 for ; Tue, 29 Oct 2024 10:16:34 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.128.65; helo=mail-wm1-f65.google.com; envelope-from=i.maximets.ovn@gmail.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp4.osuosl.org AA68D408E5 Authentication-Results: smtp4.osuosl.org; dmarc=none (p=none dis=none) header.from=ovn.org DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org AA68D408E5 Received: from mail-wm1-f65.google.com (mail-wm1-f65.google.com [209.85.128.65]) by smtp4.osuosl.org (Postfix) with ESMTPS id AA68D408E5 for ; Tue, 29 Oct 2024 10:16:33 +0000 (UTC) Received: by mail-wm1-f65.google.com with SMTP id 5b1f17b1804b1-43161c0068bso50376415e9.1 for ; Tue, 29 Oct 2024 03:16:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730196992; x=1730801792; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zT8x1zKrfJ7/DWeOGoNIOmHE465d87ilfpOc473sMzw=; b=scWW7a7RaMVxMtlMKj0wD52kgnLD8EYgoSv2yA/LP/hD6xqrp74hE5pWvKm/+45AnD dj3/WDpykjx5BcWmvRCv+YLAmr4MazuLmz6978z+F+DgBEXgETvUAZQ+gaAwRxz4OQRE jIaYG8KChoVilTkkOkY4ZhmRveYkFV9tuI8hlIutJVmbFwB13hyHlG3o2f9iaA86y1Zd LMQvfjkMfrk+N8HDxn6MHJ/X0bdksQP7YvFg8oY+JeN0KkFqwdkU8eEyLNGfKdwp+Mgk c3+e5bEn/2jSs4O6fNgPH8UedZQVRRqcV7PKr1lLowY6zp6er7GiTM1n+R8JFyGM1Ndj fXTg== X-Gm-Message-State: AOJu0YzYIovg1bEjXDOEh1MRiGUlfaNakWa/WqC6/sCq8xsnvChwc8ao a8m/90ADsi3IkKDlcWQGVVS+mgGDENLU94tsNvpnkQfNQakky+wMLPgBNkEC X-Google-Smtp-Source: AGHT+IEI5SMAMSq4R8WFsO4DRE0aafHeM6SRep5H3KHGS23kG6LFx/Jw+whxCRb8/cdKKxRX6fOhNQ== X-Received: by 2002:a05:600c:5248:b0:42c:bf94:f9a6 with SMTP id 5b1f17b1804b1-4319ad1593bmr109638005e9.26.1730196991453; Tue, 29 Oct 2024 03:16:31 -0700 (PDT) Received: from im-t490s.redhat.com (ip-86-49-44-151.bb.vodafone.cz. [86.49.44.151]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-431b4594ec3sm20279685e9.1.2024.10.29.03.16.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Oct 2024 03:16:31 -0700 (PDT) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Tue, 29 Oct 2024 11:15:06 +0100 Message-ID: <20241029101608.2991596-9-i.maximets@ovn.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241029101608.2991596-1-i.maximets@ovn.org> References: <20241029101608.2991596-1-i.maximets@ovn.org> MIME-Version: 1.0 Subject: [ovs-dev] [PATCH 8/9] tests: ipsec: Add NxN + reconciliation test. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ilya Maximets Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add a test to check establishment of IPsec connections among multiple nodes and check the reconciliation logic along the way. The test: - Creates 20 network namespaces. - Starts Libreswan, OVS and ovs-monitor-ipsec in each of them. - Adds a geneve tunnel from each namespace to every other namespace. - Checks that each namespace has all the IPsec connections loaded. - Removes a few connections manually. - Checks that these connections are added back. Unfortunately, many widely used versions of Libreswan have issues of pluto crashing frequently. For that reason the test is trying to bring pluto back online once it finds a dead one. Also, since retransmit-timeout is 60 seconds and our command timeout is 120, we can't actually use the OVS_WAIT_UNTIL macro most of the time, so the checks are done in the custom loop that waits up to 300 seconds. Signed-off-by: Ilya Maximets --- tests/system-ipsec.at | 138 ++++++++++++++++++++++++++++++++++++++---- 1 file changed, 125 insertions(+), 13 deletions(-) diff --git a/tests/system-ipsec.at b/tests/system-ipsec.at index 1e155fece..5bb048cff 100644 --- a/tests/system-ipsec.at +++ b/tests/system-ipsec.at @@ -8,6 +8,18 @@ m4_define([IPSEC_SETUP_UNDERLAY], dnl Set up the underlay switch AT_CHECK([ovs-ofctl add-flow br0 "actions=normal"])]) +m4_define([START_PLUTO], [ + rm -f $ovs_base/$1/pluto.pid + mkdir -p $ovs_base/$1/ipsec.d + touch $ovs_base/$1/ipsec.conf + touch $ovs_base/$1/secrets + ipsec initnss --nssdir $ovs_base/$1/ipsec.d + NS_CHECK_EXEC([$1], [ipsec pluto --config $ovs_base/$1/ipsec.conf \ + --ipsecdir $ovs_base/$1 --nssdir $ovs_base/$1/ipsec.d \ + --logfile $ovs_base/$1/pluto.log --secretsfile $ovs_base/$1/secrets \ + --rundir $ovs_base/$1], [0], [], [stderr]) +]) + dnl IPSEC_ADD_NODE([namespace], [device], [address], [peer address])) dnl dnl Creates a dummy host that acts as an IPsec endpoint. Creates host in @@ -45,15 +57,8 @@ m4_define([IPSEC_ADD_NODE], on_exit "kill_ovs_vswitchd `cat $ovs_base/$1/vswitchd.pid`" dnl Start pluto - mkdir -p $ovs_base/$1/ipsec.d - touch $ovs_base/$1/ipsec.conf - touch $ovs_base/$1/secrets - ipsec initnss --nssdir $ovs_base/$1/ipsec.d - NS_CHECK_EXEC([$1], [ipsec pluto --config $ovs_base/$1/ipsec.conf \ - --ipsecdir $ovs_base/$1 --nssdir $ovs_base/$1/ipsec.d \ - --logfile $ovs_base/$1/pluto.log --secretsfile $ovs_base/$1/secrets \ - --rundir $ovs_base/$1], [0], [], [stderr]) - on_exit "kill `cat $ovs_base/$1/pluto.pid`" + START_PLUTO([$1]) + on_exit 'kill $(cat $ovs_base/$1/pluto.pid)' dnl Start ovs-monitor-ipsec NS_CHECK_EXEC([$1], [ovs-monitor-ipsec unix:${OVS_RUNDIR}/$1/db.sock\ @@ -110,16 +115,18 @@ m4_define([CHECK_LIBRESWAN], dnl IPSEC_STATUS_LOADED([]) dnl dnl Get number of loaded connections from ipsec status -m4_define([IPSEC_STATUS_LOADED], [ipsec --rundir $ovs_base/$1 status | \ +m4_define([IPSEC_STATUS_LOADED], [ + ipsec --rundir $ovs_base/$1 status | \ grep "Total IPsec connections" | \ - sed 's/[[0-9]]* *Total IPsec connections: loaded \([[0-2]]\), active \([[0-2]]\).*/\1/m']) + sed 's/[[0-9]]* *Total IPsec connections: loaded \([[0-9]]*\), active \([[0-9]]*\).*/\1/m']) dnl IPSEC_STATUS_ACTIVE([]) dnl dnl Get number of active connections from ipsec status -m4_define([IPSEC_STATUS_ACTIVE], [ipsec --rundir $ovs_base/$1 status | \ +m4_define([IPSEC_STATUS_ACTIVE], [ + ipsec --rundir $ovs_base/$1 status | \ grep "Total IPsec connections" | \ - sed 's/[[0-9]]* *Total IPsec connections: loaded \([[0-2]]\), active \([[0-2]]\).*/\2/m']) + sed 's/[[0-9]]* *Total IPsec connections: loaded \([[0-9]]*\), active \([[0-9]]*\).*/\2/m']) dnl CHECK_ESP_TRAFFIC() dnl @@ -401,3 +408,108 @@ CHECK_ESP_TRAFFIC OVS_TRAFFIC_VSWITCHD_STOP() AT_CLEANUP + +AT_SETUP([IPsec -- Libreswan NxN geneve tunnels + reconciliation]) +AT_KEYWORDS([ipsec libreswan scale reconciliation]) +dnl Note: Geneve test may not work on older kernels due to CVE-2020-25645 +dnl https://bugzilla.redhat.com/show_bug.cgi?id=1883988 + +CHECK_LIBRESWAN() +OVS_TRAFFIC_VSWITCHD_START() +IPSEC_SETUP_UNDERLAY() + +m4_define([NODES], [20]) + +dnl Set up fake hosts. +m4_for([id], [1], NODES, [1], [ + IPSEC_ADD_NODE([node-id], [p-id], 10.1.1.id, 10.1.1.254) + AT_CHECK([ovs-pki -b -d ${ovs_base} -l ${ovs_base}/ovs-pki.log \ + req -u node-id], [0], [stdout]) + AT_CHECK([ovs-pki -b -d ${ovs_base} -l ${ovs_base}/ovs-pki.log \ + self-sign node-id], [0], [stdout]) + AT_CHECK(OVS_VSCTL([node-id], set Open_vSwitch . \ + other_config:certificate=${ovs_base}/node-id-cert.pem \ + other_config:private_key=${ovs_base}/node-id-privkey.pem), + [0], [ignore], [ignore]) + on_exit "ipsec --rundir $ovs_base/node-id status > $ovs_base/node-id/status" +]) + +dnl Create a full mesh of tunnels. +m4_for([LEFT], [1], NODES, [1], [ + m4_for([RIGHT], [1], NODES, [1], [ + if test LEFT -ne RIGHT; then + AT_CHECK(OVS_VSCTL(node-LEFT, add-port br-ipsec tun-RIGHT \ + -- set Interface tun-RIGHT type=geneve options:remote_ip=10.1.1.RIGHT \ + options:remote_cert=${ovs_base}/node-RIGHT-cert.pem), + [0], [ignore], [ignore]) + fi +])]) + +m4_define([WAIT_FOR_LOADED_CONNS], [ + m4_for([id], [1], NODES, [1], [ + echo "================== node-id =========================" + iterations=0 + loaded=0 + dnl Using a custom loop instead of OVS_WAIT_UNTIL, because it may take + dnl much longer than a default timeout. The default retransmit timeout + dnl for pluto is 60 seconds. Also, we need to make sure pluto didn't + dnl crash in the process and revive it if it did, unfortunately. + while true; do + date + AT_CHECK([ipsec --rundir $ovs_base/node-id status 2>&1 \ + | grep -E "whack|Total"], [ignore], [stdout]) + if grep -E 'is Pluto running?|refused' stdout; then + echo "node-id: Pluto died, restarting..." + START_PLUTO([node-id]) + else + loaded=$(IPSEC_STATUS_LOADED(node-id)) + fi + if test "$loaded" -ne $(( (NODES - 1) * 2 )); then + sleep 3 + else + break + fi + let iterations=$iterations+1 + AT_CHECK([test $iterations -lt 100]) + done + ]) +]) + +dnl Wait for all the connections to be loaded to pluto. Not waiting for +dnl them to become active, because if pluto is down on one of the nodes, +dnl some connections may not become active until we revive it. Some +dnl connections may also never become active due to bugs in libreswan 4.x. +WAIT_FOR_LOADED_CONNS() + +AT_CHECK([ipsec auto --help], [ignore], [ignore], [stderr]) +auto=auto +if test -s stderr; then + auto= +fi + +dnl Remove connections for two tunnels. One fully and one partially. +AT_CHECK([ipsec $auto --ctlsocket $ovs_base/node-1/pluto.ctl \ + --config $ovs_base/node-1/ipsec.conf \ + --delete tun-5-out-1], [0], [stdout]) +AT_CHECK([ipsec $auto --ctlsocket $ovs_base/node-1/pluto.ctl \ + --config $ovs_base/node-1/ipsec.conf \ + --delete tun-2-in-1], [0], [stdout]) +AT_CHECK([ipsec $auto --ctlsocket $ovs_base/node-1/pluto.ctl \ + --config $ovs_base/node-1/ipsec.conf \ + --delete tun-2-out-1], [0], [stdout]) + +dnl Wait for the monitor to notice the missing connections. +OVS_WAIT_UNTIL([grep -q 'tun-2.*need to reconcile' \ + $ovs_base/node-1/ovs-monitor-ipsec.log]) + +dnl Wait for all the connections to be loaded back. +WAIT_FOR_LOADED_CONNS() + +dnl These are not necessary, but nice to have in the test log in +dnl order to spot pluto failures during the test. +grep -E 'timed out|outdated|half-loaded|defunct' \ + $ovs_base/node-*/ovs-monitor-ipsec.log +grep -E 'ABORT' $ovs_base/node-*/pluto.log + +OVS_TRAFFIC_VSWITCHD_STOP() +AT_CLEANUP From patchwork Tue Oct 29 10:15:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Maximets X-Patchwork-Id: 2003670 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4Xd5lz2F05z1xwn for ; Tue, 29 Oct 2024 21:17:19 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 514DA813CE; Tue, 29 Oct 2024 10:17:17 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id oKVfYDGnDG-F; Tue, 29 Oct 2024 10:17:15 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=2605:bc80:3010:104::8cd3:938; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org C768381397 Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTPS id C768381397; Tue, 29 Oct 2024 10:17:15 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 8E8C4C08A9; Tue, 29 Oct 2024 10:17:15 +0000 (UTC) X-Original-To: ovs-dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 2E916C08A3 for ; Tue, 29 Oct 2024 10:17:14 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 8D3C960E65 for ; Tue, 29 Oct 2024 10:16:43 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id tqAFFVagaifr for ; Tue, 29 Oct 2024 10:16:37 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.167.65; helo=mail-lf1-f65.google.com; envelope-from=i.maximets.ovn@gmail.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp3.osuosl.org 9ED0760DDF Authentication-Results: smtp3.osuosl.org; dmarc=none (p=none dis=none) header.from=ovn.org DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 9ED0760DDF Received: from mail-lf1-f65.google.com (mail-lf1-f65.google.com [209.85.167.65]) by smtp3.osuosl.org (Postfix) with ESMTPS id 9ED0760DDF for ; Tue, 29 Oct 2024 10:16:36 +0000 (UTC) Received: by mail-lf1-f65.google.com with SMTP id 2adb3069b0e04-539f6e1f756so5480538e87.0 for ; Tue, 29 Oct 2024 03:16:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730196994; x=1730801794; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=c3iL4ONEuzTwIaU4lY5jd0UVhjDSG21U2VmNIrUW7DY=; b=KXasGyNVsvmpfPmwHNMOwQ9294xY1ylshSed5BnIDU1bzAw/W9X7VTfSY1UPVZ9cGa /6eP+B9+dZwvUGUxi+pOXyhugW5rQ+ZoK7fdtHwRwUoHF7qMIrzLTK9dehutwjRifTCc N2Vim+hcTCWKNFirHxH9DTQcsOAbYQOjQ4CI9sGzUaRQGlPFZ9R9gT+sM02HeXnYblEx AXjUOODWyA3KoCFljQq2WR1hJX+rb2tSCyz/fA0fkIgvmRjiCmEAOUMMCxq8M4y/3dF+ MRWYLI+g9xUKWRAjifsZ0msQ/G1/ph3UQspJPWE7rAJTkh1zjdbyJJhxA4qcL9WrmOgY zLyA== X-Gm-Message-State: AOJu0YyawY6TZM4tr8au8ICMgS/5WJUMX9qe1g7+eEJs5ELU8rxKIC1z cMco6S1RQzy4LewE5VnUpT5v1RUaoJWEPc9L7eAmgTDTiNdDEvMT8fKbTh87 X-Google-Smtp-Source: AGHT+IGLg8becp1mhRu5uxe+2NszdjnWO5asgRXGDV34IrH5HHg1Gx2lzsyCdcRL3f3gnIgYrq8kdA== X-Received: by 2002:a05:6512:31c3:b0:52f:d69e:bb38 with SMTP id 2adb3069b0e04-53b348c3836mr5114965e87.2.1730196993929; Tue, 29 Oct 2024 03:16:33 -0700 (PDT) Received: from im-t490s.redhat.com (ip-86-49-44-151.bb.vodafone.cz. [86.49.44.151]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-431b4594ec3sm20279685e9.1.2024.10.29.03.16.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Oct 2024 03:16:33 -0700 (PDT) From: Ilya Maximets To: ovs-dev@openvswitch.org Date: Tue, 29 Oct 2024 11:15:07 +0100 Message-ID: <20241029101608.2991596-10-i.maximets@ovn.org> X-Mailer: git-send-email 2.46.0 In-Reply-To: <20241029101608.2991596-1-i.maximets@ovn.org> References: <20241029101608.2991596-1-i.maximets@ovn.org> MIME-Version: 1.0 Subject: [ovs-dev] [PATCH 9/9] tests: ipsec: Check that nodes can ping each other in the NxN test. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ilya Maximets Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Expand the NxN test with the network connectivity check between all the nodes. Unfortunately, we can't really run this test with Libreswan 4.x, since, due to internal issues in these versions, we are getting into states where everything is loaded and active, but no traffic can pass. This is an internal issue in Libreswan that we can't workaround from the outside. So, the fix is required in Libreswan itself. 4.5 and earlier versions seem to not be affected by this problem, at least not severely affected, but it's easier to just cut off all the 4.x versions from the test. 3.32 version from Ubuntu 22.04 and Libreswna 5.1 work just fine with this test. Test is relatively long, but it is very valuable, IMO. Besides stressing ovs-monitor-ipsec with various failure and asynchronous connection establishment conditions, which are important for OVS, it also was used to reproduce and fix several bugs in Libreswan 4.x. Unfortunately, not all the issues are understood and fixed yet. Signed-off-by: Ilya Maximets --- tests/system-ipsec.at | 82 ++++++++++++++++++++++++++++++++++++++----- 1 file changed, 74 insertions(+), 8 deletions(-) diff --git a/tests/system-ipsec.at b/tests/system-ipsec.at index 5bb048cff..9582bfcc1 100644 --- a/tests/system-ipsec.at +++ b/tests/system-ipsec.at @@ -71,7 +71,9 @@ m4_define([IPSEC_ADD_NODE], on_exit "kill `cat $ovs_base/$1/ovs-monitor-ipsec.pid`" dnl Set up OVS bridge - NS_EXEC([$1], [ovs-vsctl --db unix:$ovs_base/$1/db.sock add-br br-ipsec])] + NS_CHECK_EXEC([$1], + [ovs-vsctl --db unix:$ovs_base/$1/db.sock add-br br-ipsec \ + -- set-controller br-ipsec punix:$ovs_base/br-ipsec.$1.mgmt])] ) m4_define([IPSEC_ADD_NODE_LEFT], [IPSEC_ADD_NODE(left, p0, $1, $2)]) m4_define([IPSEC_ADD_NODE_RIGHT], [IPSEC_ADD_NODE(right, p1, $1, $2)]) @@ -429,7 +431,8 @@ m4_for([id], [1], NODES, [1], [ self-sign node-id], [0], [stdout]) AT_CHECK(OVS_VSCTL([node-id], set Open_vSwitch . \ other_config:certificate=${ovs_base}/node-id-cert.pem \ - other_config:private_key=${ovs_base}/node-id-privkey.pem), + other_config:private_key=${ovs_base}/node-id-privkey.pem \ + -- set bridge br-ipsec other-config:hwaddr=f2:ff:00:00:00:id), [0], [ignore], [ignore]) on_exit "ipsec --rundir $ovs_base/node-id status > $ovs_base/node-id/status" ]) @@ -445,11 +448,18 @@ m4_for([LEFT], [1], NODES, [1], [ fi ])]) +dnl These are not necessary, but nice to have in the test log in +dnl order to spot pluto failures during the test. +on_exit "grep -E 'timed out|outdated|half-loaded|defunct' \ + $ovs_base/node-*/ovs-monitor-ipsec.log" +on_exit "grep -E 'ABORT' $ovs_base/node-*/pluto.log" + m4_define([WAIT_FOR_LOADED_CONNS], [ m4_for([id], [1], NODES, [1], [ echo "================== node-id =========================" iterations=0 loaded=0 + active=0 dnl Using a custom loop instead of OVS_WAIT_UNTIL, because it may take dnl much longer than a default timeout. The default retransmit timeout dnl for pluto is 60 seconds. Also, we need to make sure pluto didn't @@ -463,8 +473,11 @@ m4_define([WAIT_FOR_LOADED_CONNS], [ START_PLUTO([node-id]) else loaded=$(IPSEC_STATUS_LOADED(node-id)) + m4_if([$1], [active], + [active=$(IPSEC_STATUS_ACTIVE(node-id))], [active=$loaded]) fi - if test "$loaded" -ne $(( (NODES - 1) * 2 )); then + if test "$loaded" -ne "$(( (NODES - 1) * 2 ))" -o \ + "$loaded" -ne "$active"; then sleep 3 else break @@ -505,11 +518,64 @@ OVS_WAIT_UNTIL([grep -q 'tun-2.*need to reconcile' \ dnl Wait for all the connections to be loaded back. WAIT_FOR_LOADED_CONNS() -dnl These are not necessary, but nice to have in the test log in -dnl order to spot pluto failures during the test. -grep -E 'timed out|outdated|half-loaded|defunct' \ - $ovs_base/node-*/ovs-monitor-ipsec.log -grep -E 'ABORT' $ovs_base/node-*/pluto.log +dnl Next section will check connectivity between all the nodes. +dnl Different versions of Libreswan 4.x have issues where connections +dnl are not being correctly established or never become active in a +dnl way that can not be mitigated from ovs-monitor-ipsec or the test. +dnl So, only checking connectivity for Libreswan 3- or 5+. +if ! (ipsec --version 2>&1 | grep -q 'Libreswan 4\.'); then + dnl Turn off IPv6 and add static ARP entries for all namespaces to avoid + dnl any broadcast / multicast traffic that would otherwise be multiplied + dnl by each node creating a traffic storm. Add specific OpenFlow rules + dnl to forward traffic to exact destinations without any MAC learning. + m4_for([LEFT], [1], NODES, [1], [ + NS_CHECK_EXEC([node-LEFT], [sysctl -w net.ipv6.conf.all.disable_ipv6=1], + [0], [ignore]) + AT_CHECK([ovs-ofctl del-flows unix:$ovs_base/br-ipsec.node-LEFT.mgmt]) + AT_CHECK([ovs-ofctl add-flow unix:$ovs_base/br-ipsec.node-LEFT.mgmt \ + "dl_dst=f2:ff:00:00:00:LEFT actions=LOCAL"]) + m4_for([RIGHT], [1], NODES, [1], [ + if test LEFT -ne RIGHT; then + NS_CHECK_EXEC([node-LEFT], + [ip neigh add 192.0.0.RIGHT lladdr f2:ff:00:00:00:RIGHT dev br-ipsec]) + AT_CHECK([ovs-ofctl add-flow unix:$ovs_base/br-ipsec.node-LEFT.mgmt \ + "dl_dst=f2:ff:00:00:00:RIGHT actions=tun-RIGHT"]) + fi + ]) + ]) + + dnl Bring up and add IP addresses for br-ipsec interface. + m4_for([id], [1], NODES, [1], [ + echo "================== node-id =========================" + NS_CHECK_EXEC([node-id], [ip addr add 192.0.0.id/24 dev br-ipsec]) + NS_CHECK_EXEC([node-id], [ip link set dev br-ipsec up]) + ]) + + dnl Wait for all the connections to be loaded and active. In case one of + dnl the pluto processes crashed some of the connections may never become + dnl active. But we did run this loop with a pluto reviving logic twice + dnl already, so the chances for pluto to be down here are much lower. + WAIT_FOR_LOADED_CONNS([active]) + + dnl Check the full mesh ping. + m4_for([LEFT], [1], NODES, [1], [ + m4_for([RIGHT], [1], NODES, [1], [ + if test LEFT -ne RIGHT; then + echo "====== ping: node-LEFT --> node-RIGHT ==========" + dnl Ping without checking in case connection will recover after the + dnl first packet. + NS_CHECK_EXEC([node-LEFT], + [ping -q -c 1 -W 2 192.0.0.RIGHT | FORMAT_PING], + [ignore], [stdout]) + dnl Now check. If this one fails, there is no actual connectivity. + NS_CHECK_EXEC([node-LEFT], + [ping -q -c 3 -i 0.1 -W 2 192.0.0.RIGHT | FORMAT_PING], + [0], [dnl +3 packets transmitted, 3 received, 0% packet loss, time 0ms +]) + fi + ])]) +fi OVS_TRAFFIC_VSWITCHD_STOP() AT_CLEANUP