From patchwork Fri Dec 2 17:31:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Vladislav Odintsov X-Patchwork-Id: 1711529 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=onfe9T0Z; dkim-atps=neutral Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4NP0PP5ZPqz23n4 for ; Sat, 3 Dec 2022 04:32:08 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 869A941C1E; Fri, 2 Dec 2022 17:32:05 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 869A941C1E Authentication-Results: smtp4.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20210112 header.b=onfe9T0Z X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PKH1mXRCncUd; Fri, 2 Dec 2022 17:32:04 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id 52C2B41BAE; Fri, 2 Dec 2022 17:32:03 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 52C2B41BAE Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 374F5C0033; Fri, 2 Dec 2022 17:32:03 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id EDFC0C002D for ; Fri, 2 Dec 2022 17:32:01 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id C6ADF41BAE for ; Fri, 2 Dec 2022 17:32:01 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org C6ADF41BAE X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id aq3ar3HU7pwL for ; Fri, 2 Dec 2022 17:32:00 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 14E1E41BA5 Received: from mail-ed1-x52a.google.com (mail-ed1-x52a.google.com [IPv6:2a00:1450:4864:20::52a]) by smtp4.osuosl.org (Postfix) with ESMTPS id 14E1E41BA5 for ; Fri, 2 Dec 2022 17:31:59 +0000 (UTC) Received: by mail-ed1-x52a.google.com with SMTP id z92so7419211ede.1 for ; Fri, 02 Dec 2022 09:31:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=uyGdhEUWa27lCkm5+lXeYiERcjlywdcppozO9BMJ8i8=; b=onfe9T0ZgW9n/P3var/x0p7v85qE3n969Jkgh6vnYPR6wUFcJPvRCg+AS6+kVo7dgd d21gaw9N+wuS/pQJNDRXBGnxG8y9CVyUWXKYCCQMnvMaWdrI3N7NnDP1gTHqDrDHEgqV 2oVwY40ykbnt+2xAz7XTfRMce/ewI6NwWZeJMw93ad4u375NPqyuj/K0XJfuNmx0mPL7 3FsNbC+eQ+n8ZSI89I0NpjxfGaUPmDA3U3A1E4UvwKFgRxksXzBl0XPhgB8xp2YFRn3m E8eeV5kCOx/oVT59YZgOqS53fvWMZGPmf6Xy7mQ03iWqJQmROh68IKD0JHwoY+N/muW4 wB+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=uyGdhEUWa27lCkm5+lXeYiERcjlywdcppozO9BMJ8i8=; b=5WbZNDa+C3AiwWE6jGU2bZLCYs8wPszBw1Y9iPqzDYIXF7D7hhkXINCvh9p5xrOb55 +u55ic3GYqhrjt94BLQy0PMZdyg79VWWrBZvJR/f9OBxngkUE1Xx+7tuF9OO86KQq/EK 5QnTHAtZmR8eF0P+917w0ZwDXK4eIKF+xxZsy95nSXqSHEFtoaMgP1r+Lk7FJRUEdRW+ /kMJof8IE3vsLcbp7G0B8RlLdTBzWczlCU9kV7egzGZCi/do5FCHQ0Jk6d/rQX8+JaN+ oJA6JrvLnx3BKQ6RqewqMngXvJ6wmOIO2Mj3nGHyeUqP5J9K/lgrGYsHbwoR6fts7kkh TM5A== X-Gm-Message-State: ANoB5pn5dIhxfq9AKbIwySHwUjgeKYvylMxNXdvN9Lhyyvh01V5UWHT5 WfPW+FeIKxosQo6xKz1eXsvn6OBeU9DwtD3L X-Google-Smtp-Source: AA0mqf6j5scCQv1brx4oVSDy6blol0sqrYtqX/ACcjOai1jNlL+RVFs+L7LsewLIHPxpt5GQq9TBQQ== X-Received: by 2002:a05:6402:2946:b0:468:febe:ebab with SMTP id ed6-20020a056402294600b00468febeebabmr17760783edb.337.1670002317624; Fri, 02 Dec 2022 09:31:57 -0800 (PST) Received: from ip-10-70-112-12.vpc-1e810be1.internal (c2-178-216-98-9.elastic.cloud.croc.ru. [178.216.98.9]) by smtp.gmail.com with ESMTPSA id c9-20020a170906762900b007be3aa82543sm3235776ejn.35.2022.12.02.09.31.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Dec 2022 09:31:57 -0800 (PST) From: Vladislav Odintsov To: dev@openvswitch.org Date: Fri, 2 Dec 2022 20:31:40 +0300 Message-Id: <20221202173147.3032702-1-odivlad@gmail.com> X-Mailer: git-send-email 2.36.1 MIME-Version: 1.0 Cc: Vladislav Odintsov , dragen15051@gmail.com Subject: [ovs-dev] [PATCH ovn 0/7] OVN IC bugfixes & proposals/questions X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Hi, we’ve met with an issue, where it was possible to create multiple similar routes within LR (same ip_prefix, nexthop, and route table). Initially the problem stared after OVN upgrade. We use python ovsdbapp library, and we found a problem in python-ovs, which is described here https://mail.openvswitch.org/pipermail/ovs-dev/2022-November/399722.html by my colleague Anton. @Terry Wilson, please take a look on this. The problem itself touches OVN and OVS. Sorry for the long read, but it seems that there are a couple of bugs in different places, part of which this RFC used to cover. How the issue was initially reproduced: 1. assume we have (at least) 2-Availability Zone OVN deployment (utilising ovn-ic infrastructure). 2. create transit switch in IC NB 3. create LR in each AZ, connect them to transit switch 4. create one logical switch with a VIF port attached to local OVS & connect this logical switch to LR (e.g. 192.168.0.1/24) 5. install in one AZ in LR 2 static routes with a create command (invoke next command twice): ovn-nbctl --id=@id create logical-router-static-route ip_prefix=1.2.3.4/32 nexthop=192.168.0.10 -- logical_router add lr1 static_routes @id From this time there is a couple of strange behaviour/bugs appear: 1. [possible problem] There is a duplicated route in the NB within a single LR. lflow is computed to have ECMP group with two similar routes: table=11(lr_in_ip_routing ), priority=97 , match=(reg7 == 0 && ip4.dst == 1.2.3.4/32), action=(ip.ttl--; flags.loopback = 1; reg8[0..15] = 1; reg8[16..31] = select(1, 2); table=12(lr_in_ip_routing_ecmp), priority=100 , match=(reg8[0..15] == 1 && reg8[16..31] == 1), action=(reg0 = 192.168.0.10; reg1 = 192.168.0.1; eth.src = d0:fe:00:00:00:04; outport = "subnet-45661000"; next;) table=12(lr_in_ip_routing_ecmp), priority=100 , match=(reg8[0..15] == 2 && reg8[16..31] == 1), action=(reg0 = 192.168.0.10; reg1 = 192.168.0.1; eth.src = d0:fe:00:00:00:04; outport = "subnet-45661000"; next;) Maybe, it’s better to have some kind of handling such routes? ovsdb index or some logic in ovn-northd? 2. [bug] There is a duplicated route advertisement in OVN_IC_Southbound:Route table. IMO, this should be fixed by adding a new index to this table for availability_zone, transit_switch, ip_prefix, nexthop and route_table; adding a logic to check if the route was already advertised (covered in Patch #7). 3. [bug] There is a constant same route learning. Each ovn-ic iteration on the opposite availability zone adds one new same route. It creates thousands of same routes each second. This bug is covered by Patch #7. 4. [possible problem] After multiple routes are learned to NB on the opposite availability zone, ovn-northd generates ecmp lflows. Same as in #1: one in lr_in_ip_routing with select() and thousands of same records in lr_in_ip_routing_ecmp. OVN allows installing UINT_MAX routes within ECMP group. 5. [OVS bug?] I'd like someone from OVS team to see on this. ovn-controller installed long-long openflow group rule (group #3): # ovn-appctl -t ovn-controller group-table-list | grep :3 | wc -c 797824 When I try to dump groups with ovs-ofctl dump-groups br-int, I get next error in console: # ovs-ofctl dump-groups br-int ovs-ofctl: OpenFlow packet receive failed (End of file) In ovs-vswitchd I see next error in logs and after this line ovs is restarted: 2022-11-16T15:21:29.898Z|00145|util|EMER|lib/ofp-msgs.c:995: assertion start_ofs <= UINT16_MAX failed in ofpmp_postappend() If I issue command again, sometimes it prints same error, but sometimes this one (I had on the dev machine another OVN LB, so there are excess groups): # ovs-ofctl dump-groups br-int NXST_GROUP_DESC reply (xid=0x2): flags=[more] group_id=3,type=select,selection_method=dp_hash,bucket=bucket_id:0,weight:100,actions=ct(commit,table=20,zone=NXM_NX_REG13[0..15],nat(dst=...),exec(load:0x1->NXM_NX_CT_LABEL[1])) group_id=1,type=select,selection_method=dp_hash,bucket=bucket_id:0,weight:100,actions=ct(commit,table=20,zone=NXM_NX_REG13[0..15],nat(dst=...),exec(load:0x1->NXM_NX_CT_LABEL[1])) 2022-11-17T17:53:41Z|00001|ofp_group|WARN|OpenFlow message bucket length 56 exceeds remaining buckets data size 40 NXST_GROUP_DESC reply (xid=0x2): ***decode error: OFPGMFC_BAD_BUCKET*** 00000000 01 11 a9 58 00 00 00 02-ff ff 00 00 00 00 23 20 |...X..........# | 00000010 00 00 00 08 00 00 00 00-a9 40 01 00 00 00 00 02 |.........@......| 00000020 a9 08 00 00 00 00 00 00-00 38 00 28 00 00 00 00 |.........8.(....| 00000030 ff ff 00 18 00 00 23 20-00 07 0c 0f 80 01 08 08 |......# ........| 00000040 00 00 00 00 00 00 00 01-ff ff 00 10 00 00 23 20 |..............# | 00000050 00 0e ff f8 14 00 00 00-00 00 00 08 00 64 00 00 |.............d..| 00000060 00 38 00 28 00 00 00 01-ff ff 00 18 00 00 23 20 |.8.(..........# | 00000070 00 07 0c 0f 80 01 08 08-00 00 00 00 00 00 00 02 |................| 00000080 ff ff 00 10 00 00 23 20-00 0e ff f8 14 00 00 00 |......# ........| 00000090 00 00 00 08 00 64 00 00-00 38 00 28 00 00 00 02 |.....d...8.(....| 000000a0 ff ff 00 18 00 00 23 20-00 07 0c 0f 80 01 08 08 |......# ........| 000000b0 00 00 00 00 00 00 00 03-ff ff 00 10 00 00 23 20 |..............# | 000000c0 00 0e ff f8 14 00 00 00-00 00 00 08 00 64 00 00 |.............d..| 000000d0 00 38 00 28 00 00 00 03-ff ff 00 18 00 00 23 20 |.8.(..........# | 000000e0 00 07 0c 0f 80 01 08 08-00 00 00 00 00 00 00 04 |................| 000000f0 ff ff 00 10 00 00 23 20-00 0e ff f8 14 00 00 00 |......# ........| 00000100 00 00 00 08 00 64 00 00-00 38 00 28 00 00 00 04 |.....d...8.(....| 00000110 ff ff 00 18 00 00 23 20-00 07 0c 0f 80 01 08 08 |......# ........| 00000120 00 00 00 00 00 00 00 05-ff ff 00 10 00 00 23 20 |..............# | 00000130 00 0e ff f8 14 00 00 00-00 00 00 08 00 64 00 00 |.............d..| 00000140 00 38 00 28 00 00 00 05-ff ff 00 18 00 00 23 20 |.8.(..........# | 00000150 00 07 0c 0f 80 01 08 08-00 00 00 00 00 00 00 06 |................| 00000160 ff ff 00 10 00 00 23 20-00 0e ff f8 14 00 00 00 |......# ........| 00000170 00 00 00 08 00 64 00 00-00 38 00 28 00 00 00 06 |.....d...8.(....| 00000180 ff ff 00 18 00 00 23 20-00 07 0c 0f 80 01 08 08 |......# ........| 00000190 00 00 00 00 00 00 00 07-ff ff 00 10 00 00 23 20 |..............# | 000001a0 00 0e ff f8 14 00 00 00-00 00 00 08 00 64 00 00 |.............d..| 000001b0 00 38 00 28 00 00 00 07-ff ff 00 18 00 00 23 20 |.8.(..........# | 000001c0 00 07 0c 0f 80 01 08 08-00 00 00 00 00 00 00 08 |................| 000001d0 ff ff 00 10 00 00 23 20-00 0e ff f8 14 00 00 00 |......# ........| 000001e0 00 00 00 08 00 64 00 00-00 38 00 28 00 00 00 08 |.....d...8.(....| 000001f0 ff ff 00 18 00 00 23 20-00 07 0c 0f 80 01 08 08 |......# ........| 00000200 00 00 00 00 00 00 00 09-ff ff 00 10 00 00 23 20 |..............# | 00000210 00 0e ff f8 14 00 00 00-00 00 00 08 00 64 00 00 |.............d..| 7. From this problem with groups-dump I have some questions: 1. Is there a limit for a buckets count in group? Or a limit for the group string length? 2. If yes, should OVN limit on its side the count of buckets in a group? (Patches #4 && #6). 8. Also I’ve tried to see from which values do these problem with dump-groups begin. I created in a for-loop in OVN multiple ECMP routes and see that starting from 1200 items in a group the error from last example appear. I tried to create 10k buckets and while it was configuring on my machine there were also next lines in logfile: 2022-11-17T18:23:30.992Z|00554|ovs_rcu(urcu6)|WARN|blocked 1000 ms waiting for main to quiesce 2022-11-17T18:23:31.992Z|00555|ovs_rcu(urcu6)|WARN|blocked 2000 ms waiting for main to quiesce 2022-11-17T18:23:33.993Z|00556|ovs_rcu(urcu6)|WARN|blocked 4001 ms waiting for main to quiesce When the routes finished creating, I've issued ovs-ofctl dump-groups br-int and there was just an error: # ovs-ofctl dump-groups br-int ovs-ofctl: OpenFlow packet receive failed (End of file) And OVS crashed. OVS 2.17.3 is used. My script: # cat ./repro.sh #!/bin/bash count=$1 echo "Creating ${count} same routes..." ovn-nbctl lr-route-del lr1 1.2.3.4/32 for i in $(seq 1 ${count}); do echo $i ovn-nbctl --id=@id create logical-router-static-route ip_prefix=1.2.3.4/32 nexthop=172.31.32.4 policy=dst-ip -- add logical-router vpc-FC7D6A54 static_routes @id done Thanks for reading this, I'm ready to provide any additional information to help investigate this. Vladislav Odintsov (7): ic: move routes_ad hmap insert to separate function ic: remove orphan ovn interconnection routes ic: lookup southbound port_binding only if needed actions: limit possible OF group bucket count ic: minor code improvements northd: limit ECMP group by 1024 members ic: prevent advertising/learning multiple same routes ic/ovn-ic.c | 123 ++++++++++++++++++++++++++++------------ lib/actions.c | 40 ++++++++++++- northd/northd.c | 2 +- ovn-ic-sb.ovsschema | 6 +- tests/ovn-ic.at | 133 ++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 263 insertions(+), 41 deletions(-)