| Message ID | 20211005202442.85322-3-odivlad@gmail.com |
|---|---|
| State | Changes Requested |
| Delegated to: | Han Zhou |
| Headers | show |
| Series | Add multiple routing tables support to Logical Routers | expand |
| Context | Check | Description |
|---|---|---|
| ovsrobot/apply-robot | fail | apply and check: fail |
Bleep bloop. Greetings Vladislav Odintsov, I am a robot and I have tried out your patch. Thanks for your contribution. I encountered some error that I wasn't expecting. See the details below. git-am: error: Failed to merge in the changes. hint: Use 'git am --show-current-patch' to see the failed patch Patch failed at 0001 northd, utils: support for RouteTables in LRs When you have resolved this problem, run "git am --continue". If you prefer to skip this patch, run "git am --skip" instead. To restore the original branch and stop patching, run "git am --abort". Please check this out. If you feel there has been an error, please email aconole@redhat.com Thanks, 0-day Robot
On Tue, Oct 5, 2021 at 1:26 PM Vladislav Odintsov <odivlad@gmail.com> wrote: > > This patch extends Logical Router's routing functionality. > Now user may create multiple routing tables within a Logical Router > and assign them to Logical Router Ports. > > Traffic coming from Logical Router Port with assigned route_table > is checked against global routes if any (Logical_Router_Static_Routes > whith empty route_table field), next against directly connected routes This is not accurate. The "directly connected routes" is NOT after the global routes. Their priority only depends on the prefix length. > and then Logical_Router_Static_Routes with same route_table value as > in Logical_Router_Port options:route_table field. > > A new Logical Router ingress table #10 is added - IN_IP_ROUTING_PRE. > In this table packets which come from LRPs with configured > options:route_table field are checked against inport and in OVS > register 7 unique non-zero value identifying route table is written. > > Then in 11th table IN_IP_ROUTING routes which have non-empty > `route_table` field are added with additional match on reg7 value > associated with appropriate route_table. > Hi Vladislav, First of all, sorry for the delayed review, and thanks for implementing this new feature. I have some questions regarding the feature itself. I remember that we had some discussion earlier for this feature, but it seems I misunderstood the feature you are implementing here. I thought by multiple routing tables you were trying to support something like VRF in physical routers, but this seems to be something different. According to your implementation, instead of assigning LRPs to different routing tables, a LRP can actually participate to both the global routing table and the table with a specific ID. For ingress, global routes are prefered over other routes; for egress (i.e. forwarding to the next hop), it doesn't even enforce any table-id check, so packet routed by a entry of table-X can go out of a LRP with table-id Y. Is my understanding correct about the desired behavior of this feature? If this is correct, it doesn't seem to be a common/standard requirement (or please educate me if I am wrong). Could you explain a little more about the actual use cases: what kind of real world problems need to be solved by this feature or how are you going to use this. For example, why would a port need to participate in both routing tables? It looks like what you really need is policy routing instead of multiple isolated routing tables. I understand that you already use policy routing to implement ACLs, so it is not convenient to combine other policies (e.g. inport based routing) into the policy routing stage. If that's the case, would it be more generic to support multiple policy routing stages? My concern to the current approach is that it is implemented for a very special use case. It makes the code more complex but when there is a slightly different requirement in the future it becomes insufficient. I am thinking that policy routing seems more flexible and has more potential to be made more generic. Maybe I will have a better understanding when I hear more detailed use cases and considerations from you. I haven't finished reviewing the code yet, but I have one comment regarding adding 100 to the priority of the global routes. For IPv6, the priority range from 0 to 120x2=240, so adding 100 is not enough. It would create overlapping priority ranges, and some table-id specific route entries may override the global routes. Thanks, Han > Signed-off-by: Vladislav Odintsov <odivlad@gmail.com> > Acked-by: Numan Siddique <numans@ovn.org> > --- > northd/northd.c | 159 ++++++++++++--- > northd/ovn-northd.8.xml | 63 ++++-- > ovn-nb.ovsschema | 5 +- > ovn-nb.xml | 30 +++ > tests/ovn-ic.at | 4 + > tests/ovn-nbctl.at | 196 +++++++++++++++++- > tests/ovn-northd.at | 76 ++++++- > tests/ovn.at | 441 +++++++++++++++++++++++++++++++++++++++- > utilities/ovn-nbctl.c | 134 +++++++++++- > 9 files changed, 1041 insertions(+), 67 deletions(-) > > diff --git a/northd/northd.c b/northd/northd.c > index 092eca829..6a020cb2e 100644 > --- a/northd/northd.c > +++ b/northd/northd.c > @@ -148,15 +148,16 @@ enum ovn_stage { > PIPELINE_STAGE(ROUTER, IN, ECMP_STATEFUL, 7, "lr_in_ecmp_stateful") \ > PIPELINE_STAGE(ROUTER, IN, ND_RA_OPTIONS, 8, "lr_in_nd_ra_options") \ > PIPELINE_STAGE(ROUTER, IN, ND_RA_RESPONSE, 9, "lr_in_nd_ra_response") \ > - PIPELINE_STAGE(ROUTER, IN, IP_ROUTING, 10, "lr_in_ip_routing") \ > - PIPELINE_STAGE(ROUTER, IN, IP_ROUTING_ECMP, 11, "lr_in_ip_routing_ecmp") \ > - PIPELINE_STAGE(ROUTER, IN, POLICY, 12, "lr_in_policy") \ > - PIPELINE_STAGE(ROUTER, IN, POLICY_ECMP, 13, "lr_in_policy_ecmp") \ > - PIPELINE_STAGE(ROUTER, IN, ARP_RESOLVE, 14, "lr_in_arp_resolve") \ > - PIPELINE_STAGE(ROUTER, IN, CHK_PKT_LEN , 15, "lr_in_chk_pkt_len") \ > - PIPELINE_STAGE(ROUTER, IN, LARGER_PKTS, 16, "lr_in_larger_pkts") \ > - PIPELINE_STAGE(ROUTER, IN, GW_REDIRECT, 17, "lr_in_gw_redirect") \ > - PIPELINE_STAGE(ROUTER, IN, ARP_REQUEST, 18, "lr_in_arp_request") \ > + PIPELINE_STAGE(ROUTER, IN, IP_ROUTING_PRE, 10, "lr_in_ip_routing_pre") \ > + PIPELINE_STAGE(ROUTER, IN, IP_ROUTING, 11, "lr_in_ip_routing") \ > + PIPELINE_STAGE(ROUTER, IN, IP_ROUTING_ECMP, 12, "lr_in_ip_routing_ecmp") \ > + PIPELINE_STAGE(ROUTER, IN, POLICY, 13, "lr_in_policy") \ > + PIPELINE_STAGE(ROUTER, IN, POLICY_ECMP, 14, "lr_in_policy_ecmp") \ > + PIPELINE_STAGE(ROUTER, IN, ARP_RESOLVE, 15, "lr_in_arp_resolve") \ > + PIPELINE_STAGE(ROUTER, IN, CHK_PKT_LEN, 16, "lr_in_chk_pkt_len") \ > + PIPELINE_STAGE(ROUTER, IN, LARGER_PKTS, 17, "lr_in_larger_pkts") \ > + PIPELINE_STAGE(ROUTER, IN, GW_REDIRECT, 18, "lr_in_gw_redirect") \ > + PIPELINE_STAGE(ROUTER, IN, ARP_REQUEST, 19, "lr_in_arp_request") \ > \ > /* Logical router egress stages. */ \ > PIPELINE_STAGE(ROUTER, OUT, UNDNAT, 0, "lr_out_undnat") \ > @@ -225,6 +226,7 @@ enum ovn_stage { > #define REG_NEXT_HOP_IPV6 "xxreg0" > #define REG_SRC_IPV4 "reg1" > #define REG_SRC_IPV6 "xxreg1" > +#define REG_ROUTE_TABLE_ID "reg7" > > #define REG_ORIG_TP_DPORT_ROUTER "reg9[16..31]" > > @@ -287,8 +289,9 @@ enum ovn_stage { > * | R6 | UNUSED | X | | G | IN_IP_ROUTING)| > * | | | R | | 1 | | > * +-----+--------------------------+ E | UNUSED | | | > - * | R7 | UNUSED | G | | | | > - * | | | 3 | | | | > + * | R7 | ROUTE_TABLE_ID | G | | | | > + * | | (>= IN_IP_ROUTING_PRE && | 3 | | | | > + * | | <= IN_IP_ROUTING) | | | | | > * +-----+--------------------------+---+-----------------+---+---------------+ > * | R8 | ECMP_GROUP_ID | | | > * | | ECMP_MEMBER_ID | X | | > @@ -8511,11 +8514,72 @@ cleanup: > ds_destroy(&actions); > } > > +static uint32_t > +route_table_add(struct simap *route_tables, const char *route_table_name) > +{ > + /* route table ids start from 1 */ > + uint32_t rtb_id = simap_count(route_tables) + 1; > + > + if (rtb_id == UINT16_MAX) { > + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1); > + VLOG_WARN_RL(&rl, "too many route tables for Logical Router."); > + return 0; > + } > + > + if (!simap_put(route_tables, route_table_name, rtb_id)) { > + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1); > + VLOG_WARN_RL(&rl, "Route table id unexpectedly appeared"); > + } > + > + return rtb_id; > +} > + > +static uint32_t > +get_route_table_id(struct simap *route_tables, const char *route_table_name) > +{ > + if (!route_table_name || !strlen(route_table_name)) { > + return 0; > + } > + > + uint32_t rtb_id = simap_get(route_tables, route_table_name); > + if (!rtb_id) { > + rtb_id = route_table_add(route_tables, route_table_name); > + } > + > + return rtb_id; > +} > + > +static void > +build_route_table_lflow(struct ovn_datapath *od, struct hmap *lflows, > + struct nbrec_logical_router_port *lrp, > + struct simap *route_tables) > +{ > + struct ds match = DS_EMPTY_INITIALIZER; > + struct ds actions = DS_EMPTY_INITIALIZER; > + > + const char *route_table_name = smap_get(&lrp->options, "route_table"); > + uint32_t rtb_id = get_route_table_id(route_tables, route_table_name); > + if (!rtb_id) { > + return; > + } > + > + ds_put_format(&match, "inport == \"%s\"", lrp->name); > + ds_put_format(&actions, "%s = %d; next;", > + REG_ROUTE_TABLE_ID, rtb_id); > + > + ovn_lflow_add(lflows, od, S_ROUTER_IN_IP_ROUTING_PRE, 100, > + ds_cstr(&match), ds_cstr(&actions)); > + > + ds_destroy(&match); > + ds_destroy(&actions); > +} > + > struct parsed_route { > struct ovs_list list_node; > struct in6_addr prefix; > unsigned int plen; > bool is_src_route; > + uint32_t route_table_id; > uint32_t hash; > const struct nbrec_logical_router_static_route *route; > bool ecmp_symmetric_reply; > @@ -8540,7 +8604,7 @@ find_static_route_outport(struct ovn_datapath *od, struct hmap *ports, > * Otherwise return NULL. */ > static struct parsed_route * > parsed_routes_add(struct ovn_datapath *od, struct hmap *ports, > - struct ovs_list *routes, > + struct ovs_list *routes, struct simap *route_tables, > const struct nbrec_logical_router_static_route *route, > struct hmap *bfd_connections) > { > @@ -8622,6 +8686,7 @@ parsed_routes_add(struct ovn_datapath *od, struct hmap *ports, > struct parsed_route *pr = xzalloc(sizeof *pr); > pr->prefix = prefix; > pr->plen = plen; > + pr->route_table_id = get_route_table_id(route_tables, route->route_table); > pr->is_src_route = (route->policy && !strcmp(route->policy, > "src-ip")); > pr->hash = route_hash(pr); > @@ -8655,6 +8720,7 @@ struct ecmp_groups_node { > struct in6_addr prefix; > unsigned int plen; > bool is_src_route; > + uint32_t route_table_id; > uint16_t route_count; > struct ovs_list route_list; /* Contains ecmp_route_list_node */ > }; > @@ -8663,7 +8729,7 @@ static void > ecmp_groups_add_route(struct ecmp_groups_node *group, > const struct parsed_route *route) > { > - if (group->route_count == UINT16_MAX) { > + if (group->route_count == UINT16_MAX) { > static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1); > VLOG_WARN_RL(&rl, "too many routes in a single ecmp group."); > return; > @@ -8692,6 +8758,7 @@ ecmp_groups_add(struct hmap *ecmp_groups, > eg->prefix = route->prefix; > eg->plen = route->plen; > eg->is_src_route = route->is_src_route; > + eg->route_table_id = route->route_table_id; > ovs_list_init(&eg->route_list); > ecmp_groups_add_route(eg, route); > > @@ -8705,7 +8772,8 @@ ecmp_groups_find(struct hmap *ecmp_groups, struct parsed_route *route) > HMAP_FOR_EACH_WITH_HASH (eg, hmap_node, route->hash, ecmp_groups) { > if (ipv6_addr_equals(&eg->prefix, &route->prefix) && > eg->plen == route->plen && > - eg->is_src_route == route->is_src_route) { > + eg->is_src_route == route->is_src_route && > + eg->route_table_id == route->route_table_id) { > return eg; > } > } > @@ -8752,7 +8820,8 @@ unique_routes_remove(struct hmap *unique_routes, > HMAP_FOR_EACH_WITH_HASH (ur, hmap_node, route->hash, unique_routes) { > if (ipv6_addr_equals(&route->prefix, &ur->route->prefix) && > route->plen == ur->route->plen && > - route->is_src_route == ur->route->is_src_route) { > + route->is_src_route == ur->route->is_src_route && > + route->route_table_id == ur->route->route_table_id) { > hmap_remove(unique_routes, &ur->hmap_node); > const struct parsed_route *existed_route = ur->route; > free(ur); > @@ -8790,9 +8859,9 @@ build_route_prefix_s(const struct in6_addr *prefix, unsigned int plen) > } > > static void > -build_route_match(const struct ovn_port *op_inport, const char *network_s, > - int plen, bool is_src_route, bool is_ipv4, struct ds *match, > - uint16_t *priority) > +build_route_match(const struct ovn_port *op_inport, uint32_t rtb_id, > + const char *network_s, int plen, bool is_src_route, > + bool is_ipv4, struct ds *match, uint16_t *priority) > { > const char *dir; > /* The priority here is calculated to implement longest-prefix-match > @@ -8808,6 +8877,15 @@ build_route_match(const struct ovn_port *op_inport, const char *network_s, > if (op_inport) { > ds_put_format(match, "inport == %s && ", op_inport->json_key); > } > + if (rtb_id) { > + ds_put_format(match, "%s == %d && ", REG_ROUTE_TABLE_ID, rtb_id); > + } else { > + /* Route-table assigned LRPs' routes should have lower priority > + * in order not to affect directly-connected global routes. > + * So, enlarge non-route-table routes priority by 100. > + */ > + *priority += 100; > + } > ds_put_format(match, "ip%s.%s == %s/%d", is_ipv4 ? "4" : "6", dir, > network_s, plen); > } > @@ -8946,7 +9024,7 @@ add_ecmp_symmetric_reply_flows(struct hmap *lflows, > out_port->lrp_networks.ea_s, > IN6_IS_ADDR_V4MAPPED(&route->prefix) ? "" : "xx", > port_ip, out_port->json_key); > - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_IP_ROUTING, 300, > + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_IP_ROUTING, 400, > ds_cstr(&match), ds_cstr(&actions), > &st_route->header_); > > @@ -8976,8 +9054,8 @@ build_ecmp_route_flow(struct hmap *lflows, struct ovn_datapath *od, > struct ds route_match = DS_EMPTY_INITIALIZER; > > char *prefix_s = build_route_prefix_s(&eg->prefix, eg->plen); > - build_route_match(NULL, prefix_s, eg->plen, eg->is_src_route, is_ipv4, > - &route_match, &priority); > + build_route_match(NULL, eg->route_table_id, prefix_s, eg->plen, > + eg->is_src_route, is_ipv4, &route_match, &priority); > free(prefix_s); > > struct ds actions = DS_EMPTY_INITIALIZER; > @@ -9052,8 +9130,8 @@ static void > add_route(struct hmap *lflows, struct ovn_datapath *od, > const struct ovn_port *op, const char *lrp_addr_s, > const char *network_s, int plen, const char *gateway, > - bool is_src_route, const struct ovsdb_idl_row *stage_hint, > - bool is_discard_route) > + bool is_src_route, const uint32_t rtb_id, > + const struct ovsdb_idl_row *stage_hint, bool is_discard_route) > { > bool is_ipv4 = strchr(network_s, '.') ? true : false; > struct ds match = DS_EMPTY_INITIALIZER; > @@ -9068,8 +9146,8 @@ add_route(struct hmap *lflows, struct ovn_datapath *od, > op_inport = op; > } > } > - build_route_match(op_inport, network_s, plen, is_src_route, is_ipv4, > - &match, &priority); > + build_route_match(op_inport, rtb_id, network_s, plen, is_src_route, > + is_ipv4, &match, &priority); > > struct ds common_actions = DS_EMPTY_INITIALIZER; > struct ds actions = DS_EMPTY_INITIALIZER; > @@ -9132,7 +9210,8 @@ build_static_route_flow(struct hmap *lflows, struct ovn_datapath *od, > char *prefix_s = build_route_prefix_s(&route_->prefix, route_->plen); > add_route(lflows, route_->is_discard_route ? od : out_port->od, out_port, > lrp_addr_s, prefix_s, route_->plen, route->nexthop, > - route_->is_src_route, &route->header_, route_->is_discard_route); > + route_->is_src_route, route_->route_table_id, &route->header_, > + route_->is_discard_route); > > free(prefix_s); > } > @@ -10584,6 +10663,17 @@ build_ND_RA_flows_for_lrouter(struct ovn_datapath *od, struct hmap *lflows) > } > } > > +/* Logical router ingress table IP_ROUTING_PRE: > + * by default goto next. (priority 0). */ > +static void > +build_ip_routing_pre_flows_for_lrouter(struct ovn_datapath *od, > + struct hmap *lflows) > +{ > + if (od->nbr) { > + ovn_lflow_add(lflows, od, S_ROUTER_IN_IP_ROUTING_PRE, 0, "1", "next;"); > + } > +} > + > /* Logical router ingress table IP_ROUTING : IP Routing. > * > * A packet that arrives at this table is an IP packet that should be > @@ -10609,14 +10699,14 @@ build_ip_routing_flows_for_lrouter_port( > for (int i = 0; i < op->lrp_networks.n_ipv4_addrs; i++) { > add_route(lflows, op->od, op, op->lrp_networks.ipv4_addrs[i].addr_s, > op->lrp_networks.ipv4_addrs[i].network_s, > - op->lrp_networks.ipv4_addrs[i].plen, NULL, false, > + op->lrp_networks.ipv4_addrs[i].plen, NULL, false, 0, > &op->nbrp->header_, false); > } > > for (int i = 0; i < op->lrp_networks.n_ipv6_addrs; i++) { > add_route(lflows, op->od, op, op->lrp_networks.ipv6_addrs[i].addr_s, > op->lrp_networks.ipv6_addrs[i].network_s, > - op->lrp_networks.ipv6_addrs[i].plen, NULL, false, > + op->lrp_networks.ipv6_addrs[i].plen, NULL, false, 0, > &op->nbrp->header_, false); > } > } else if (lsp_is_router(op->nbsp)) { > @@ -10639,7 +10729,7 @@ build_ip_routing_flows_for_lrouter_port( > add_route(lflows, peer->od, peer, > peer->lrp_networks.ipv4_addrs[0].addr_s, > laddrs->ipv4_addrs[k].network_s, > - laddrs->ipv4_addrs[k].plen, NULL, false, > + laddrs->ipv4_addrs[k].plen, NULL, false, 0, > &peer->nbrp->header_, false); > } > } > @@ -10659,10 +10749,17 @@ build_static_route_flows_for_lrouter( > struct hmap ecmp_groups = HMAP_INITIALIZER(&ecmp_groups); > struct hmap unique_routes = HMAP_INITIALIZER(&unique_routes); > struct ovs_list parsed_routes = OVS_LIST_INITIALIZER(&parsed_routes); > + struct simap route_tables = SIMAP_INITIALIZER(&route_tables); > struct ecmp_groups_node *group; > + > + for (int i = 0; i < od->nbr->n_ports; i++) { > + build_route_table_lflow(od, lflows, od->nbr->ports[i], > + &route_tables); > + } > + > for (int i = 0; i < od->nbr->n_static_routes; i++) { > struct parsed_route *route = > - parsed_routes_add(od, ports, &parsed_routes, > + parsed_routes_add(od, ports, &parsed_routes, &route_tables, > od->nbr->static_routes[i], bfd_connections); > if (!route) { > continue; > @@ -10695,6 +10792,7 @@ build_static_route_flows_for_lrouter( > ecmp_groups_destroy(&ecmp_groups); > unique_routes_destroy(&unique_routes); > parsed_routes_destroy(&parsed_routes); > + simap_destroy(&route_tables); > } > } > > @@ -12800,6 +12898,7 @@ build_lswitch_and_lrouter_iterate_by_od(struct ovn_datapath *od, > build_neigh_learning_flows_for_lrouter(od, lsi->lflows, &lsi->match, > &lsi->actions, lsi->meter_groups); > build_ND_RA_flows_for_lrouter(od, lsi->lflows); > + build_ip_routing_pre_flows_for_lrouter(od, lsi->lflows); > build_static_route_flows_for_lrouter(od, lsi->lflows, lsi->ports, > lsi->bfd_connections); > build_mcast_lookup_flows_for_lrouter(od, lsi->lflows, &lsi->match, > diff --git a/northd/ovn-northd.8.xml b/northd/ovn-northd.8.xml > index 39f4eaa0c..cc2e25367 100644 > --- a/northd/ovn-northd.8.xml > +++ b/northd/ovn-northd.8.xml > @@ -2899,7 +2899,7 @@ icmp6 { > > <p> > If ECMP routes with symmetric reply are configured in the > - <code>OVN_Northbound</code> database for a gateway router, a priority-300 > + <code>OVN_Northbound</code> database for a gateway router, a priority-400 > flow is added for each router port on which symmetric replies are > configured. The matching logic for these ports essentially reverses the > configured logic of the ECMP route. So for instance, a route with a > @@ -3245,7 +3245,35 @@ output; > </li> > </ul> > > - <h3>Ingress Table 10: IP Routing</h3> > + <h3>Ingress Table 10: IP Routing Pre</h3> > + > + <p> > + If a packet arrived at this table from Logical Router Port <var>P</var> > + which has <code>options:route_table</code> value set, a logical flow with > + match <code>inport == "<var>P</var>"</code> with priority 100 and action, > + setting unique-generated per-datapath 32-bit value (non-zero) in OVS > + register 7. This register is checked in next table. > + </p> > + > + <p> > + This table contains the following logical flows: > + </p> > + > + <ul> > + <li> > + <p> > + Priority-100 flow with match <code>inport == "LRP_NAME"</code> value > + and action, which set route table identifier in reg7. > + </p> > + > + <p> > + A priority-0 logical flow with match <code>1</code> has actions > + <code>next;</code>. > + </p> > + </li> > + </ul> > + > + <h3>Ingress Table 11: IP Routing</h3> > > <p> > A packet that arrives at this table is an IP packet that should be > @@ -3316,10 +3344,10 @@ output; > <p> > IPv4 routing table. For each route to IPv4 network <var>N</var> with > netmask <var>M</var>, on router port <var>P</var> with IP address > - <var>A</var> and Ethernet > - address <var>E</var>, a logical flow with match <code>ip4.dst == > - <var>N</var>/<var>M</var></code>, whose priority is the number of > - 1-bits in <var>M</var>, has the following actions: > + <var>A</var> and Ethernet address <var>E</var>, a logical flow with > + match <code>ip4.dst == <var>N</var>/<var>M</var></code>, whose > + priority is 100 + the number of 1-bits in <var>M</var>, has the > + following actions: > </p> > > <pre> > @@ -3382,6 +3410,13 @@ next; > If the address <var>A</var> is in the link-local scope, the > route will be limited to sending on the ingress port. > </p> > + > + <p> > + For routes with <code>route_table</code> value set > + <code>reg7 == id</code> is prefixed in logical flow match portion. > + Priority for routes with <code>route_table</code> value set is > + the number of 1-bits in <var>M</var>. > + </p> > </li> > > <li> > @@ -3408,7 +3443,7 @@ select(reg8[16..31], <var>MID1</var>, <var>MID2</var>, ...); > </li> > </ul> > > - <h3>Ingress Table 11: IP_ROUTING_ECMP</h3> > + <h3>Ingress Table 12: IP_ROUTING_ECMP</h3> > > <p> > This table implements the second part of IP routing for ECMP routes > @@ -3460,7 +3495,7 @@ outport = <var>P</var>; > </li> > </ul> > > - <h3>Ingress Table 12: Router policies</h3> > + <h3>Ingress Table 13: Router policies</h3> > <p> > This table adds flows for the logical router policies configured > on the logical router. Please see the > @@ -3532,7 +3567,7 @@ next; > </li> > </ul> > > - <h3>Ingress Table 13: ECMP handling for router policies</h3> > + <h3>Ingress Table 14: ECMP handling for router policies</h3> > <p> > This table handles the ECMP for the router policies configured > with multiple nexthops. > @@ -3576,7 +3611,7 @@ outport = <var>P</var> > </li> > </ul> > > - <h3>Ingress Table 14: ARP/ND Resolution</h3> > + <h3>Ingress Table 15: ARP/ND Resolution</h3> > > <p> > Any packet that reaches this table is an IP packet whose next-hop > @@ -3767,7 +3802,7 @@ outport = <var>P</var> > > </ul> > > - <h3>Ingress Table 15: Check packet length</h3> > + <h3>Ingress Table 16: Check packet length</h3> > > <p> > For distributed logical routers or gateway routers with gateway > @@ -3797,7 +3832,7 @@ REGBIT_PKT_LARGER = check_pkt_larger(<var>L</var>); next; > and advances to the next table. > </p> > > - <h3>Ingress Table 16: Handle larger packets</h3> > + <h3>Ingress Table 17: Handle larger packets</h3> > > <p> > For distributed logical routers or gateway routers with gateway port > @@ -3860,7 +3895,7 @@ icmp6 { > and advances to the next table. > </p> > > - <h3>Ingress Table 17: Gateway Redirect</h3> > + <h3>Ingress Table 18: Gateway Redirect</h3> > > <p> > For distributed logical routers where one or more of the logical router > @@ -3907,7 +3942,7 @@ icmp6 { > </li> > </ul> > > - <h3>Ingress Table 18: ARP Request</h3> > + <h3>Ingress Table 19: ARP Request</h3> > > <p> > In the common case where the Ethernet destination has been resolved, this > diff --git a/ovn-nb.ovsschema b/ovn-nb.ovsschema > index 2ac8ef3ea..a0a171e19 100644 > --- a/ovn-nb.ovsschema > +++ b/ovn-nb.ovsschema > @@ -1,7 +1,7 @@ > { > "name": "OVN_Northbound", > - "version": "5.32.1", > - "cksum": "2805328215 29734", > + "version": "5.33.1", > + "cksum": "3874993350 29785", > "tables": { > "NB_Global": { > "columns": { > @@ -387,6 +387,7 @@ > "isRoot": false}, > "Logical_Router_Static_Route": { > "columns": { > + "route_table": {"type": "string"}, > "ip_prefix": {"type": "string"}, > "policy": {"type": {"key": {"type": "string", > "enum": ["set", ["src-ip", > diff --git a/ovn-nb.xml b/ovn-nb.xml > index d8266ed4d..b2917c363 100644 > --- a/ovn-nb.xml > +++ b/ovn-nb.xml > @@ -2772,6 +2772,14 @@ > prefix according to RFC3663 > </p> > </column> > + > + <column name="options" key="route_table"> > + Designates lookup Logical_Router_Static_Routes with specified > + <code>route_table</code> value. Routes to directly connected networks > + from same Logical Router and routes without <code>route_table</code> > + option set have higher priority than routes with > + <code>route_table</code> option set. > + </column> > </group> > > <group title="Attachment"> > @@ -2891,6 +2899,28 @@ > </p> > </column> > > + <column name="route_table"> > + <p> > + Any string to place route to separate routing table. If Logical Router > + Port has configured value in <ref table="Logical_Router_Port" > + column="options" key="route_table"/> other than empty string, OVN > + performs route lookup for all packets entering Logical Router ingress > + pipeline from this port in the following manner: > + </p> > + > + <ul> > + <li> > + 1. First lookup among "global" routes: routes without > + <code>route_table</code> value set and routes to directly connected > + networks. > + </li> > + <li> > + 2. Next lookup among routes with same <code>route_table</code> value > + as specified in LRP's options:route_table field. > + </li> > + </ul> > + </column> > + > <column name="external_ids" key="ic-learned-route"> > <code>ovn-ic</code> populates this key if the route is learned from the > global <ref db="OVN_IC_Southbound"/> database. In this case the value > diff --git a/tests/ovn-ic.at b/tests/ovn-ic.at > index 32f4e9d02..3aab54362 100644 > --- a/tests/ovn-ic.at > +++ b/tests/ovn-ic.at > @@ -281,6 +281,7 @@ done > > AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr1], [0], [dnl > IPv4 Routes > +Route Table global: > 10.11.1.0/24 169.254.0.1 dst-ip > 10.11.2.0/24 169.254.100.2 dst-ip (learned) > 10.22.1.0/24 169.254.0.2 src-ip > @@ -299,6 +300,7 @@ ovn_as az1 ovn-nbctl set nb_global . options:ic-route-learn=false > OVS_WAIT_WHILE([ovn_as az1 ovn-nbctl lr-route-list lr1 | grep learned]) > AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr1], [0], [dnl > IPv4 Routes > +Route Table global: > 10.11.1.0/24 169.254.0.1 dst-ip > 10.22.1.0/24 169.254.0.2 src-ip > ]) > @@ -314,6 +316,7 @@ ovn_as az1 ovn-nbctl set nb_global . options:ic-route-adv=false > OVS_WAIT_WHILE([ovn_as az2 ovn-nbctl lr-route-list lr2 | grep learned]) > AT_CHECK([ovn_as az2 ovn-nbctl lr-route-list lr2], [0], [dnl > IPv4 Routes > +Route Table global: > 10.11.2.0/24 169.254.0.1 dst-ip > 10.22.2.0/24 169.254.0.2 src-ip > ]) > @@ -332,6 +335,7 @@ done > # Default route should NOT get advertised or learned, by default. > AT_CHECK([ovn_as az2 ovn-nbctl lr-route-list lr2], [0], [dnl > IPv4 Routes > +Route Table global: > 10.11.1.0/24 169.254.100.1 dst-ip (learned) > 10.11.2.0/24 169.254.0.1 dst-ip > 10.22.2.0/24 169.254.0.2 src-ip > diff --git a/tests/ovn-nbctl.at b/tests/ovn-nbctl.at > index 9b80ae410..ddb5536ce 100644 > --- a/tests/ovn-nbctl.at > +++ b/tests/ovn-nbctl.at > @@ -1520,6 +1520,7 @@ AT_CHECK([ovn-nbctl --ecmp --policy=src-ip lr-route-add lr0 20.0.0.0/24 11.0.0.1 > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > IPv4 Routes > +Route Table global: > 10.0.0.0/24 11.0.0.1 dst-ip > 10.0.1.0/24 11.0.1.1 dst-ip lp0 > 10.0.10.0/24 dst-ip lp0 > @@ -1534,6 +1535,7 @@ AT_CHECK([ovn-nbctl lrp-add lr0 lp1 f0:00:00:00:00:02 11.0.0.254/24]) > AT_CHECK([ovn-nbctl --may-exist lr-route-add lr0 10.0.0.111/24 11.0.0.1 lp1]) > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > IPv4 Routes > +Route Table global: > 10.0.0.0/24 11.0.0.1 dst-ip lp1 > 10.0.1.0/24 11.0.1.1 dst-ip lp0 > 10.0.10.0/24 dst-ip lp0 > @@ -1564,6 +1566,7 @@ AT_CHECK([ovn-nbctl --policy=src-ip lr-route-del lr0 9.16.1.0/24]) > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > IPv4 Routes > +Route Table global: > 10.0.0.0/24 11.0.0.1 dst-ip lp1 > 10.0.10.0/24 dst-ip lp0 > 10.0.0.0/24 11.0.0.2 src-ip > @@ -1575,6 +1578,7 @@ AT_CHECK([ovn-nbctl --policy=dst-ip lr-route-del lr0 10.0.0.0/24]) > AT_CHECK([ovn-nbctl --policy=src-ip lr-route-del lr0 10.0.0.0/24]) > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > IPv4 Routes > +Route Table global: > 10.0.10.0/24 dst-ip lp0 > 0.0.0.0/0 192.168.0.1 dst-ip > ]) > @@ -1585,6 +1589,7 @@ AT_CHECK([ovn-nbctl --policy=src-ip lr-route-add lr0 10.0.0.0/24 11.0.0.2]) > AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24]) > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > IPv4 Routes > +Route Table global: > 10.0.10.0/24 dst-ip lp0 > 0.0.0.0/0 192.168.0.1 dst-ip > ]) > @@ -1601,6 +1606,7 @@ AT_CHECK([ovn-nbctl --ecmp lr-route-add lr0 10.0.0.0/24 11.0.0.3]) > AT_CHECK([ovn-nbctl --ecmp lr-route-add lr0 10.0.0.0/24 11.0.0.4 lp0]) > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > IPv4 Routes > +Route Table global: > 10.0.0.0/24 11.0.0.1 dst-ip ecmp > 10.0.0.0/24 11.0.0.2 dst-ip ecmp > 10.0.0.0/24 11.0.0.3 dst-ip ecmp > @@ -1615,6 +1621,7 @@ dnl Delete ecmp routes > AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 11.0.0.1]) > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > IPv4 Routes > +Route Table global: > 10.0.0.0/24 11.0.0.2 dst-ip ecmp > 10.0.0.0/24 11.0.0.3 dst-ip ecmp > 10.0.0.0/24 11.0.0.4 dst-ip lp0 ecmp > @@ -1622,12 +1629,14 @@ IPv4 Routes > AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 11.0.0.2]) > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > IPv4 Routes > +Route Table global: > 10.0.0.0/24 11.0.0.3 dst-ip ecmp > 10.0.0.0/24 11.0.0.4 dst-ip lp0 ecmp > ]) > AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 11.0.0.4 lp0]) > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > IPv4 Routes > +Route Table global: > 10.0.0.0/24 11.0.0.3 dst-ip > ]) > AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 11.0.0.3]) > @@ -1641,6 +1650,7 @@ AT_CHECK([ovn-nbctl lr-route-add lr0 2001:0db8:1::/64 2001:0db8:0:f103::1]) > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > IPv6 Routes > +Route Table global: > 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > ::/0 2001:db8:0:f101::1 dst-ip > @@ -1650,6 +1660,7 @@ AT_CHECK([ovn-nbctl lr-route-del lr0 2001:0db8:0::/64]) > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > IPv6 Routes > +Route Table global: > 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > ::/0 2001:db8:0:f101::1 dst-ip > ]) > @@ -1677,11 +1688,13 @@ AT_CHECK([ovn-nbctl --may-exist --ecmp-symmetric-reply lr-route-add lr0 2003:0db > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > IPv4 Routes > +Route Table global: > 10.0.0.0/24 11.0.0.1 dst-ip > 10.0.1.0/24 11.0.1.1 dst-ip lp0 > 0.0.0.0/0 192.168.0.1 dst-ip > > IPv6 Routes > +Route Table global: > 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip ecmp > 2001:db8:1::/64 2001:db8:0:f103::2 dst-ip ecmp > @@ -1696,7 +1709,188 @@ AT_CHECK([ovn-nbctl lrp-add lr0 lr0-p0 00:00:01:01:02:03 192.168.10.1/24]) > bfd_uuid=$(ovn-nbctl create bfd logical_port=lr0-p0 dst_ip=100.0.0.50 status=down min_tx=250 min_rx=250 detect_mult=10) > AT_CHECK([ovn-nbctl lr-route-add lr0 100.0.0.0/24 192.168.0.1]) > route_uuid=$(fetch_column nb:logical_router_static_route _uuid ip_prefix="100.0.0.0/24") > -AT_CHECK([ovn-nbctl set logical_router_static_route $route_uuid bfd=$bfd_uuid])]) > +AT_CHECK([ovn-nbctl set logical_router_static_route $route_uuid bfd=$bfd_uuid]) > + > +check ovn-nbctl lr-route-del lr0 > +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > +]) > + > +dnl Check IPv4 routes in route table > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 0.0.0.0/0 192.168.0.1 > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 10.0.1.1/24 11.0.1.1 lp0 > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 10.0.0.1/24 11.0.0.1 > +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > +IPv4 Routes > +Route Table rtb-1: > + 10.0.0.0/24 11.0.0.1 dst-ip > + 10.0.1.0/24 11.0.1.1 dst-ip lp0 > + 0.0.0.0/0 192.168.0.1 dst-ip > +]) > + > +check ovn-nbctl lr-route-del lr0 > +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > +]) > + > +dnl Check IPv6 routes in route table > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 0:0:0:0:0:0:0:0/0 2001:0db8:0:f101::1 > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 2001:0db8:0::/64 2001:0db8:0:f102::1 lp0 > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 2001:0db8:1::/64 2001:0db8:0:f103::1 > + > +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > +IPv6 Routes > +Route Table rtb-1: > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > + ::/0 2001:db8:0:f101::1 dst-ip > +]) > + > +dnl Check IPv4 and IPv6 routes in route table > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 0.0.0.0/0 192.168.0.1 > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 10.0.1.1/24 11.0.1.1 lp0 > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 10.0.0.1/24 11.0.0.1 > + > +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > +IPv4 Routes > +Route Table rtb-1: > + 10.0.0.0/24 11.0.0.1 dst-ip > + 10.0.1.0/24 11.0.1.1 dst-ip lp0 > + 0.0.0.0/0 192.168.0.1 dst-ip > + > +IPv6 Routes > +Route Table rtb-1: > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > + ::/0 2001:db8:0:f101::1 dst-ip > +]) > + > +# Add routes in another route table > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 0.0.0.0/0 192.168.0.1 > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 10.0.1.1/24 11.0.1.1 lp0 > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 10.0.0.1/24 11.0.0.1 > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 0:0:0:0:0:0:0:0/0 2001:0db8:0:f101::1 > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 2001:0db8:0::/64 2001:0db8:0:f102::1 lp0 > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 2001:0db8:1::/64 2001:0db8:0:f103::1 > + > +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > +IPv4 Routes > +Route Table rtb-1: > + 10.0.0.0/24 11.0.0.1 dst-ip > + 10.0.1.0/24 11.0.1.1 dst-ip lp0 > + 0.0.0.0/0 192.168.0.1 dst-ip > + > +Route Table rtb-2: > + 10.0.0.0/24 11.0.0.1 dst-ip > + 10.0.1.0/24 11.0.1.1 dst-ip lp0 > + 0.0.0.0/0 192.168.0.1 dst-ip > + > +IPv6 Routes > +Route Table rtb-1: > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > + ::/0 2001:db8:0:f101::1 dst-ip > + > +Route Table rtb-2: > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > + ::/0 2001:db8:0:f101::1 dst-ip > +]) > + > +# Add routes to global route table > +check ovn-nbctl lr-route-add lr0 0.0.0.0/0 192.168.0.1 > +check ovn-nbctl lr-route-add lr0 10.0.1.1/24 11.0.1.1 lp0 > +check ovn-nbctl lr-route-add lr0 10.0.0.1/24 11.0.0.1 > +check ovn-nbctl lr-route-add lr0 0:0:0:0:0:0:0:0/0 2001:0db8:0:f101::1 > +check ovn-nbctl lr-route-add lr0 2001:0db8:0::/64 2001:0db8:0:f102::1 lp0 > +check check ovn-nbctl lr-route-add lr0 2001:0db8:1::/64 2001:0db8:0:f103::1 > + > +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > +IPv4 Routes > +Route Table global: > + 10.0.0.0/24 11.0.0.1 dst-ip > + 10.0.1.0/24 11.0.1.1 dst-ip lp0 > + 0.0.0.0/0 192.168.0.1 dst-ip > + > +Route Table rtb-1: > + 10.0.0.0/24 11.0.0.1 dst-ip > + 10.0.1.0/24 11.0.1.1 dst-ip lp0 > + 0.0.0.0/0 192.168.0.1 dst-ip > + > +Route Table rtb-2: > + 10.0.0.0/24 11.0.0.1 dst-ip > + 10.0.1.0/24 11.0.1.1 dst-ip lp0 > + 0.0.0.0/0 192.168.0.1 dst-ip > + > +IPv6 Routes > +Route Table global: > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > + ::/0 2001:db8:0:f101::1 dst-ip > + > +Route Table rtb-1: > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > + ::/0 2001:db8:0:f101::1 dst-ip > + > +Route Table rtb-2: > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > + ::/0 2001:db8:0:f101::1 dst-ip > +]) > + > +# delete IPv4 route from rtb-1 > +check ovn-nbctl --route-table=rtb-1 lr-route-del lr0 10.0.0.0/24 > +AT_CHECK([ovn-nbctl --route-table=rtb-1 lr-route-list lr0], [0], [dnl > +IPv4 Routes > +Route Table rtb-1: > + 10.0.1.0/24 11.0.1.1 dst-ip lp0 > + 0.0.0.0/0 192.168.0.1 dst-ip > + > +IPv6 Routes > +Route Table rtb-1: > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > + ::/0 2001:db8:0:f101::1 dst-ip > +]) > + > +# delete IPv6 route from rtb-2 > +check ovn-nbctl --route-table=rtb-2 lr-route-del lr0 2001:db8::/64 > +AT_CHECK([ovn-nbctl --route-table=rtb-2 lr-route-list lr0], [0], [dnl > +IPv4 Routes > +Route Table rtb-2: > + 10.0.0.0/24 11.0.0.1 dst-ip > + 10.0.1.0/24 11.0.1.1 dst-ip lp0 > + 0.0.0.0/0 192.168.0.1 dst-ip > + > +IPv6 Routes > +Route Table rtb-2: > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > + ::/0 2001:db8:0:f101::1 dst-ip > +]) > + > +check ovn-nbctl lr-route-del lr0 > + > +# ECMP route in route table > +check ovn-nbctl --route-table=rtb1 lr-route-add lr0 0.0.0.0/0 192.168.0.1 > +check ovn-nbctl --ecmp --route-table=rtb1 lr-route-add lr0 0.0.0.0/0 192.168.0.2 > + > +# Negative route table case: same prefix > +AT_CHECK([ovn-nbctl --route-table=rtb1 lr-route-add lr0 0.0.0.0/0 192.168.0.1], [1], [], [dnl > +ovn-nbctl: duplicate prefix: 0.0.0.0/0 (policy: dst-ip). Use option --ecmp to allow this for ECMP routing. > +]) > + > +# Negative route table case: same prefix & nexthop with ecmp > +AT_CHECK([ovn-nbctl --ecmp --route-table=rtb1 lr-route-add lr0 0.0.0.0/0 192.168.0.2], [1], [], [dnl > +ovn-nbctl: duplicate nexthop for the same ECMP route > +]) > + > +# Add routes to global route table > +check ovn-nbctl lrp-add lr0 lrp0 00:00:00:00:00:01 1.1.1.1/24 > +check ovn-nbctl lrp-set-options lrp0 route_table=rtb1 > +AT_CHECK([ovn-nbctl get logical-router-port lrp0 options:route_table], [0], [dnl > +rtb1 > +]) > +check `ovn-nbctl show lr0 | grep lrp0 -A3 | grep route_table=rtb1` > +]) > > dnl --------------------------------------------------------------------- > > diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at > index 3eebb55b6..e71e65bcc 100644 > --- a/tests/ovn-northd.at > +++ b/tests/ovn-northd.at > @@ -5111,7 +5111,7 @@ check ovn-nbctl --wait=sb --ecmp-symmetric-reply lr-route-add lr0 1.0.0.1 192.16 > > ovn-sbctl dump-flows lr0 > lr0flows > AT_CHECK([grep -e "lr_in_ip_routing.*select" lr0flows | sed 's/table=../table=??/' | sort], [0], [dnl > - table=??(lr_in_ip_routing ), priority=65 , match=(ip4.dst == 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = 1; reg8[[16..31]] = select(1, 2);) > + table=??(lr_in_ip_routing ), priority=165 , match=(ip4.dst == 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = 1; reg8[[16..31]] = select(1, 2);) > ]) > AT_CHECK([grep -e "lr_in_ip_routing_ecmp" lr0flows | sed 's/192\.168\.0\..0/192.168.0.??/' | sed 's/table=../table=??/' | sort], [0], [dnl > table=??(lr_in_ip_routing_ecmp), priority=100 , match=(reg8[[0..15]] == 1 && reg8[[16..31]] == 1), action=(reg0 = 192.168.0.??; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; next;) > @@ -5124,7 +5124,7 @@ check ovn-nbctl --wait=sb --ecmp-symmetric-reply lr-route-add lr0 1.0.0.1 192.16 > > ovn-sbctl dump-flows lr0 > lr0flows > AT_CHECK([grep -e "lr_in_ip_routing.*select" lr0flows | sed 's/table=../table=??/' | sort], [0], [dnl > - table=??(lr_in_ip_routing ), priority=65 , match=(ip4.dst == 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = 1; reg8[[16..31]] = select(1, 2);) > + table=??(lr_in_ip_routing ), priority=165 , match=(ip4.dst == 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = 1; reg8[[16..31]] = select(1, 2);) > ]) > AT_CHECK([grep -e "lr_in_ip_routing_ecmp" lr0flows | sed 's/192\.168\.0\..0/192.168.0.??/' | sed 's/table=../table=??/' | sort], [0], [dnl > table=??(lr_in_ip_routing_ecmp), priority=100 , match=(reg8[[0..15]] == 1 && reg8[[16..31]] == 1), action=(reg0 = 192.168.0.??; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; next;) > @@ -5139,14 +5139,14 @@ check ovn-nbctl --wait=sb lr-route-add lr0 1.0.0.0/24 192.168.0.10 > ovn-sbctl dump-flows lr0 > lr0flows > > AT_CHECK([grep -e "lr_in_ip_routing.*192.168.0.10" lr0flows | sed 's/table=../table=??/' | sort], [0], [dnl > - table=??(lr_in_ip_routing ), priority=49 , match=(ip4.dst == 1.0.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.10; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; flags.loopback = 1; next;) > + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == 1.0.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.10; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; flags.loopback = 1; next;) > ]) > > check ovn-nbctl --wait=sb lr-route-add lr0 2.0.0.0/24 lr0-public > > ovn-sbctl dump-flows lr0 > lr0flows > AT_CHECK([grep -e "lr_in_ip_routing.*2.0.0.0" lr0flows | sed 's/table=../table=??/' | sort], [0], [dnl > - table=??(lr_in_ip_routing ), priority=49 , match=(ip4.dst == 2.0.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; flags.loopback = 1; next;) > + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == 2.0.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; flags.loopback = 1; next;) > ]) > > AT_CLEANUP > @@ -5232,3 +5232,71 @@ AT_CHECK([grep lr_in_gw_redirect lrflows | grep cr-DR | sed 's/table=../table=?? > > AT_CLEANUP > ]) > + > + > +OVN_FOR_EACH_NORTHD([ > +AT_SETUP([route tables -- flows]) > +AT_KEYWORDS([route-tables-flows]) > +ovn_start > + > +check ovn-nbctl lr-add lr0 > +check ovn-nbctl lrp-add lr0 lrp0 00:00:00:00:00:01 192.168.0.1/24 > +check ovn-nbctl lrp-add lr0 lrp1 00:00:00:00:01:01 192.168.1.1/24 > +check ovn-nbctl lrp-add lr0 lrp2 00:00:00:00:02:01 192.168.2.1/24 > +check ovn-nbctl lrp-set-options lrp1 route_table=rtb-1 > +check ovn-nbctl lrp-set-options lrp2 route_table=rtb-2 > + > +check ovn-nbctl lr-route-add lr0 0.0.0.0/0 192.168.0.10 > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 192.168.0.0/24 192.168.1.10 > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 0.0.0.0/0 192.168.0.10 > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 1.1.1.1/32 192.168.0.20 > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 2.2.2.2/32 192.168.0.30 > +check ovn-nbctl --route-table=rtb-2 --ecmp lr-route-add lr0 2.2.2.2/32 192.168.0.31 > +check ovn-nbctl --wait=sb sync > + > +ovn-sbctl dump-flows lr0 > lr0flows > +AT_CAPTURE_FILE([lr0flows]) > + > +AT_CHECK([grep -e "lr_in_ip_routing_pre.*match=(1)" lr0flows | sed 's/table=../table=??/'], [0], [dnl > + table=??(lr_in_ip_routing_pre), priority=0 , match=(1), action=(next;) > +]) > + > +p1_reg=$(grep -oP "lr_in_ip_routing_pre.*lrp1.*action=\(reg7 = \K." lr0flows) > +p2_reg=$(grep -oP "lr_in_ip_routing_pre.*lrp2.*action=\(reg7 = \K." lr0flows) > +echo $p1_reg > +echo $p2_reg > + > +# exact register values are not predictable > +if [[ $p1_reg -eq 2 ] && [ $p2_reg -eq 1 ]]; then > + echo "swap reg values in dump" > + sed -i -r s'/^(.*lrp2.*action=\(reg7 = )(1)(.*)/\12\3/g' lr0flows # "reg7 = 1" -> "reg7 = 2" > + sed -i -r s'/^(.*lrp1.*action=\(reg7 = )(2)(.*)/\11\3/g' lr0flows # "reg7 = 2" -> "reg7 = 1" > + sed -i -r s'/^(.*match=\(reg7 == )(2)( &&.*lrp1.*)/\11\3/g' lr0flows # "reg7 == 2" -> "reg7 == 1" > + sed -i -r s'/^(.*match=\(reg7 == )(1)( &&.*lrp0.*)/\12\3/g' lr0flows # "reg7 == 1" -> "reg7 == 2" > +fi > + > +check test "$p1_reg" != "$p2_reg" -a $((p1_reg * p2_reg)) -eq 2 > + > +AT_CHECK([grep "lr_in_ip_routing_pre" lr0flows | sed 's/table=../table=??/' | sort], [0], [dnl > + table=??(lr_in_ip_routing_pre), priority=0 , match=(1), action=(next;) > + table=??(lr_in_ip_routing_pre), priority=100 , match=(inport == "lrp1"), action=(reg7 = 1; next;) > + table=??(lr_in_ip_routing_pre), priority=100 , match=(inport == "lrp2"), action=(reg7 = 2; next;) > +]) > + > +grep -e "(lr_in_ip_routing ).*outport" lr0flows > + > +AT_CHECK([grep -e "(lr_in_ip_routing ).*outport" lr0flows | sed 's/table=../table=??/' | sort], [0], [dnl > + table=??(lr_in_ip_routing ), priority=1 , match=(reg7 == 2 && ip4.dst == 0.0.0.0/0), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.10; reg1 = 192.168.0.1; eth.src = 00:00:00:00:00:01; outport = "lrp0"; flags.loopback = 1; next;) > + table=??(lr_in_ip_routing ), priority=101 , match=(ip4.dst == 0.0.0.0/0), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.10; reg1 = 192.168.0.1; eth.src = 00:00:00:00:00:01; outport = "lrp0"; flags.loopback = 1; next;) > + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == 192.168.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 = 192.168.0.1; eth.src = 00:00:00:00:00:01; outport = "lrp0"; flags.loopback = 1; next;) > + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == 192.168.1.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 = 192.168.1.1; eth.src = 00:00:00:00:01:01; outport = "lrp1"; flags.loopback = 1; next;) > + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == 192.168.2.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 = 192.168.2.1; eth.src = 00:00:00:00:02:01; outport = "lrp2"; flags.loopback = 1; next;) > + table=??(lr_in_ip_routing ), priority=229 , match=(inport == "lrp0" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe00:1; eth.src = 00:00:00:00:00:01; outport = "lrp0"; flags.loopback = 1; next;) > + table=??(lr_in_ip_routing ), priority=229 , match=(inport == "lrp1" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe00:101; eth.src = 00:00:00:00:01:01; outport = "lrp1"; flags.loopback = 1; next;) > + table=??(lr_in_ip_routing ), priority=229 , match=(inport == "lrp2" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe00:201; eth.src = 00:00:00:00:02:01; outport = "lrp2"; flags.loopback = 1; next;) > + table=??(lr_in_ip_routing ), priority=49 , match=(reg7 == 1 && ip4.dst == 192.168.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.1.10; reg1 = 192.168.1.1; eth.src = 00:00:00:00:01:01; outport = "lrp1"; flags.loopback = 1; next;) > + table=??(lr_in_ip_routing ), priority=65 , match=(reg7 == 2 && ip4.dst == 1.1.1.1/32), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.20; reg1 = 192.168.0.1; eth.src = 00:00:00:00:00:01; outport = "lrp0"; flags.loopback = 1; next;) > +]) > + > +AT_CLEANUP > +]) > diff --git a/tests/ovn.at b/tests/ovn.at > index 49ece8735..60783a14b 100644 > --- a/tests/ovn.at > +++ b/tests/ovn.at > @@ -18145,7 +18145,7 @@ eth_dst=00000000ff01 > ip_src=$(ip_to_hex 10 0 0 10) > ip_dst=$(ip_to_hex 172 168 0 101) > send_icmp_packet 1 1 $eth_src $eth_dst $ip_src $ip_dst c4c9 0000000000000000000000 > -AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | awk '/table=25, n_packets=1, n_bytes=45/{print $7" "$8}'],[0],[dnl > +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | awk '/table=26, n_packets=1, n_bytes=45/{print $7" "$8}'],[0],[dnl > priority=80,ip,reg15=0x3,metadata=0x3,nw_src=10.0.0.10 actions=drop > ]) > > @@ -22577,6 +22577,433 @@ OVN_CLEANUP([hv1]) > AT_CLEANUP > ]) > > + > +OVN_FOR_EACH_NORTHD([ > +AT_SETUP([route tables -- global routes]) > +ovn_start > + > +# Logical network: > +# ls1 (192.168.1.0/24) - lrp-lr1-ls1 - lr1 - lrp-lr1-ls2 - ls2 ( 192.168.2.0/24) > +# > +# ls1 has lsp11 (192.168.1.11) and ls2 has lsp21 (192.168.2.21) and lsp22 > +# (192.168.2.22) > +# > +# lrp-lr1-ls1 set options:route_table=rtb-1 > +# > +# Static routes on lr1: > +# 0.0.0.0/0 nexthop 192.168.2.21 > +# 1.1.1.1/32 nexthop 192.168.2.22 route_table=rtb-1 > +# > +# Test 1: > +# lsp11 send packet to 2.2.2.2 > +# > +# Expected result: > +# lsp21 should receive traffic, lsp22 should not receive any traffic > +# > +# Test 2: > +# lsp11 send packet to 1.1.1.1 > +# > +# Expected result: > +# lsp21 should receive traffic, lsp22 should not receive any traffic > + > +ovn-nbctl lr-add lr1 > + > +for i in 1 2; do > + ovn-nbctl ls-add ls${i} > + ovn-nbctl lrp-add lr1 lrp-lr1-ls${i} 00:00:00:01:0${i}:01 192.168.${i}.1/24 > + ovn-nbctl lsp-add ls${i} lsp-ls${i}-lr1 -- lsp-set-type lsp-ls${i}-lr1 router \ > + -- lsp-set-options lsp-ls${i}-lr1 router-port=lrp-lr1-ls${i} \ > + -- lsp-set-addresses lsp-ls${i}-lr1 router > +done > + > +# install static routes > +ovn-nbctl lr-route-add lr1 0.0.0.0/0 192.168.2.21 > +ovn-nbctl --route-table=rtb-1 lr-route-add lr1 1.1.1.1/32 192.168.2.22 > + > +# set lrp-lr1-ls1 route table > +ovn-nbctl lrp-set-options lrp-lr1-ls1 route_table=rtb-1 > + > +# Create logical ports > +ovn-nbctl lsp-add ls1 lsp11 -- \ > + lsp-set-addresses lsp11 "f0:00:00:00:01:11 192.168.1.11" > +ovn-nbctl lsp-add ls2 lsp21 -- \ > + lsp-set-addresses lsp21 "f0:00:00:00:02:21 192.168.2.21" > +ovn-nbctl lsp-add ls2 lsp22 -- \ > + lsp-set-addresses lsp22 "f0:00:00:00:02:22 192.168.2.22" > + > +net_add n1 > +sim_add hv1 > +as hv1 > +ovs-vsctl add-br br-phys > +ovn_attach n1 br-phys 192.168.0.1 > +ovs-vsctl -- add-port br-int hv1-vif1 -- \ > + set interface hv1-vif1 external-ids:iface-id=lsp11 \ > + options:tx_pcap=hv1/vif1-tx.pcap \ > + options:rxq_pcap=hv1/vif1-rx.pcap \ > + ofport-request=1 > + > +ovs-vsctl -- add-port br-int hv1-vif2 -- \ > + set interface hv1-vif2 external-ids:iface-id=lsp21 \ > + options:tx_pcap=hv1/vif2-tx.pcap \ > + options:rxq_pcap=hv1/vif2-rx.pcap \ > + ofport-request=2 > + > +ovs-vsctl -- add-port br-int hv1-vif3 -- \ > + set interface hv1-vif3 external-ids:iface-id=lsp22 \ > + options:tx_pcap=hv1/vif3-tx.pcap \ > + options:rxq_pcap=hv1/vif3-rx.pcap \ > + ofport-request=3 > + > +# wait for earlier changes to take effect > +check ovn-nbctl --wait=hv sync > +wait_for_ports_up > + > +for i in 1 2; do > + packet="inport==\"lsp11\" && eth.src==f0:00:00:00:01:11 && eth.dst==00:00:00:01:01:01 && > + ip4 && ip.ttl==64 && ip4.src==192.168.1.11 && ip4.dst==$i.$i.$i.$i && icmp" > + AT_CHECK([as hv1 ovs-appctl -t ovn-controller inject-pkt "$packet"]) > + > + # Assume all packets go to lsp21. > + exp_packet="eth.src==00:00:00:01:02:01 && eth.dst==f0:00:00:00:02:21 && > + ip4 && ip.ttl==63 && ip4.src==192.168.1.11 && ip4.dst==$i.$i.$i.$i && icmp" > + echo $exp_packet | ovstest test-ovn expr-to-packets >> expected_lsp21 > +done > +> expected_lsp22 > + > +# lsp21 should recieve 2 packets and lsp22 should recieve no packets > +OVS_WAIT_UNTIL([ > + rcv_n1=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" hv1/vif2-tx.pcap > lsp21.packets && cat lsp21.packets | wc -l` > + rcv_n2=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" hv1/vif3-tx.pcap > lsp22.packets && cat lsp22.packets | wc -l` > + echo $rcv_n1 $rcv_n2 > + test $rcv_n1 -eq 2 -a $rcv_n2 -eq 0]) > + > +for i in 1 2; do > + sort expected_lsp2$i > expout > + AT_CHECK([cat lsp2${i}.packets | sort], [0], [expout]) > +done > + > +OVN_CLEANUP([hv1]) > +AT_CLEANUP > +]) > + > + > +OVN_FOR_EACH_NORTHD([ > +AT_SETUP([route tables -- directly connected routes]) > +ovn_start > + > +# Logical network: > +# ls1 (192.168.1.0/24) - lrp-lr1-ls1 - lr1 - lrp-lr1-ls2 - ls2 ( 192.168.2.0/24) > +# > +# ls1 has lsp11 (192.168.1.11) and ls2 has lsp21 (192.168.2.21) > +# > +# lrp-lr1-ls1 set options:route_table=rtb-1 > +# > +# Static routes on lr1: > +# 192.168.2.0/25 nexthop 192.168.1.11 route_table=rtb-1 > +# > +# Test 1: > +# lsp11 send packet to 192.168.2.21 > +# > +# Expected result: > +# lsp21 should receive traffic, lsp11 should not receive any traffic > + > +ovn-nbctl lr-add lr1 > + > +for i in 1 2; do > + ovn-nbctl ls-add ls${i} > + ovn-nbctl lrp-add lr1 lrp-lr1-ls${i} 00:00:00:01:0${i}:01 192.168.${i}.1/24 > + ovn-nbctl lsp-add ls${i} lsp-ls${i}-lr1 -- lsp-set-type lsp-ls${i}-lr1 router \ > + -- lsp-set-options lsp-ls${i}-lr1 router-port=lrp-lr1-ls${i} \ > + -- lsp-set-addresses lsp-ls${i}-lr1 router > +done > + > +# install static route, which overrides directly-connected routes > +ovn-nbctl --route-table=rtb-1 lr-route-add lr1 192.168.2.0/25 192.168.1.11 > + > +# set lrp-lr1-ls1 route table > +ovn-nbctl lrp-set-options lrp-lr1-ls1 route_table=rtb-1 > + > +# Create logical ports > +ovn-nbctl lsp-add ls1 lsp11 -- \ > + lsp-set-addresses lsp11 "f0:00:00:00:01:11 192.168.1.11" > +ovn-nbctl lsp-add ls2 lsp21 -- \ > + lsp-set-addresses lsp21 "f0:00:00:00:02:21 192.168.2.21" > + > +net_add n1 > +sim_add hv1 > +as hv1 > +ovs-vsctl add-br br-phys > +ovn_attach n1 br-phys 192.168.0.1 > +ovs-vsctl -- add-port br-int hv1-vif1 -- \ > + set interface hv1-vif1 external-ids:iface-id=lsp11 \ > + options:tx_pcap=hv1/vif1-tx.pcap \ > + options:rxq_pcap=hv1/vif1-rx.pcap \ > + ofport-request=1 > + > +ovs-vsctl -- add-port br-int hv1-vif2 -- \ > + set interface hv1-vif2 external-ids:iface-id=lsp21 \ > + options:tx_pcap=hv1/vif2-tx.pcap \ > + options:rxq_pcap=hv1/vif2-rx.pcap \ > + ofport-request=2 > + > +# wait for earlier changes to take effect > +check ovn-nbctl --wait=hv sync > +wait_for_ports_up > + > +packet="inport==\"lsp11\" && eth.src==f0:00:00:00:01:11 && eth.dst==00:00:00:01:01:01 && > + ip4 && ip.ttl==64 && ip4.src==192.168.1.11 && ip4.dst==192.168.2.21 && icmp" > +AT_CHECK([as hv1 ovs-appctl -t ovn-controller inject-pkt "$packet"]) > + > +# Assume all packets go to lsp21. > +exp_packet="eth.src==00:00:00:01:02:01 && eth.dst==f0:00:00:00:02:21 && > + ip4 && ip.ttl==63 && ip4.src==192.168.1.11 && ip4.dst==192.168.2.21 && icmp" > +echo $exp_packet | ovstest test-ovn expr-to-packets >> expected_lsp21 > +> expected_lsp11 > + > +# lsp21 should recieve 1 icmp packet and lsp11 should recieve no packets > +OVS_WAIT_UNTIL([ > + rcv_n11=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" hv1/vif1-tx.pcap > lsp11.packets && cat lsp11.packets | wc -l` > + rcv_n21=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" hv1/vif2-tx.pcap > lsp21.packets && cat lsp21.packets | wc -l` > + echo $rcv_n11 $rcv_n21 > + test $rcv_n11 -eq 0 -a $rcv_n21 -eq 1]) > + > +for i in 11 21; do > + sort expected_lsp$i > expout > + AT_CHECK([cat lsp${i}.packets | sort], [0], [expout]) > +done > + > +OVN_CLEANUP([hv1]) > +AT_CLEANUP > +]) > + > + > +OVN_FOR_EACH_NORTHD([ > +AT_SETUP([route tables -- overlapping subnets]) > +ovn_start > + > +# Logical network: > +# > +# ls1 (192.168.1.0/24) - lrp-lr1-ls1 -\ /- lrp-lr1-ls2 - ls2 ( 192.168.2.0/24) > +# lr1 > +# ls3 (192.168.3.0/24) - lrp-lr1-ls3 -/ \- lrp-lr1-ls4 - ls4 ( 192.168.4.0/24) > +# > +# ls1 has lsp11 (192.168.1.11) > +# ls2 has lsp21 (192.168.2.21) > +# ls3 has lsp31 (192.168.3.31) > +# ls4 has lsp41 (192.168.4.41) > +# > +# lrp-lr1-ls1 set options:route_table=rtb-1 > +# lrp-lr1-ls2 set options:route_table=rtb-2 > +# > +# Static routes on lr1: > +# 10.0.0.0/24 nexthop 192.168.3.31 route_table=rtb-1 > +# 10.0.0.0/24 nexthop 192.168.4.41 route_table=rtb-2 > +# > +# Test 1: > +# lsp11 send packet to 10.0.0.1 > +# > +# Expected result: > +# lsp31 should receive traffic, lsp41 should not receive any traffic > +# > +# Test 2: > +# lsp21 send packet to 10.0.0.1 > +# > +# Expected result: > +# lsp41 should receive traffic, lsp31 should not receive any traffic > + > +ovn-nbctl lr-add lr1 > + > +# Create logical topology > +for i in $(seq 1 4); do > + ovn-nbctl ls-add ls${i} > + ovn-nbctl lrp-add lr1 lrp-lr1-ls${i} 00:00:00:01:0${i}:01 192.168.${i}.1/24 > + ovn-nbctl lsp-add ls${i} lsp-ls${i}-lr1 -- lsp-set-type lsp-ls${i}-lr1 router \ > + -- lsp-set-options lsp-ls${i}-lr1 router-port=lrp-lr1-ls${i} \ > + -- lsp-set-addresses lsp-ls${i}-lr1 router > + ovn-nbctl lsp-add ls$i lsp${i}1 -- \ > + lsp-set-addresses lsp${i}1 "f0:00:00:00:0${i}:1${i} 192.168.${i}.${i}1" > +done > + > +# install static routes > +ovn-nbctl --route-table=rtb-1 lr-route-add lr1 10.0.0.0/24 192.168.3.31 > +ovn-nbctl --route-table=rtb-2 lr-route-add lr1 10.0.0.0/24 192.168.4.41 > + > +# set lrp-lr1-ls{1,2} route tables > +ovn-nbctl lrp-set-options lrp-lr1-ls1 route_table=rtb-1 > +ovn-nbctl lrp-set-options lrp-lr1-ls2 route_table=rtb-2 > + > +net_add n1 > +sim_add hv1 > +as hv1 > +ovs-vsctl add-br br-phys > +ovn_attach n1 br-phys 192.168.0.1 > + > +for i in $(seq 1 4); do > + ovs-vsctl -- add-port br-int hv1-vif${i} -- \ > + set interface hv1-vif${i} external-ids:iface-id=lsp${i}1 \ > + options:tx_pcap=hv1/vif${i}-tx.pcap \ > + options:rxq_pcap=hv1/vif${i}-rx.pcap \ > + ofport-request=${i} > +done > + > +# wait for earlier changes to take effect > +check ovn-nbctl --wait=hv sync > +wait_for_ports_up > + > +# lsp31 should recieve packet coming from lsp11 > +# lsp41 should recieve packet coming from lsp21 > +for i in $(seq 1 2); do > + di=$(( i + 2)) # dst index > + ri=$(( 5 - i)) # reverse index > + packet="inport==\"lsp${i}1\" && eth.src==f0:00:00:00:0${i}:1${i} && > + eth.dst==00:00:00:01:0${i}:01 && ip4 && ip.ttl==64 && > + ip4.src==192.168.${i}.${i}1 && ip4.dst==10.0.0.1 && icmp" > + AT_CHECK([as hv1 ovs-appctl -t ovn-controller inject-pkt "$packet"]) > + > + # Assume all packets go to lsp${di}1. > + exp_packet="eth.src==00:00:00:01:0${di}:01 && eth.dst==f0:00:00:00:0${di}:1${di} && > + ip4 && ip.ttl==63 && ip4.src==192.168.${i}.${i}1 && ip4.dst==10.0.0.1 && icmp" > + echo $exp_packet | ovstest test-ovn expr-to-packets >> expected_lsp${di}1 > + > expected_lsp${ri}1 > + > + OVS_WAIT_UNTIL([ > + rcv_n1=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" hv1/vif${di}-tx.pcap > lsp${di}1.packets && cat lsp${di}1.packets | wc -l` > + rcv_n2=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" hv1/vif${ri}-tx.pcap > lsp${ri}1.packets && cat lsp${ri}1.packets | wc -l` > + echo $rcv_n1 $rcv_n2 > + test $rcv_n1 -eq 1 -a $rcv_n2 -eq 0]) > + > + for j in "${di}1" "${ri}1"; do > + sort expected_lsp${j} > expout > + AT_CHECK([cat lsp${j}.packets | sort], [0], [expout]) > + done > + > + # cleanup tx pcap files > + for j in "${di}1" "${ri}1"; do > + ovs-vsctl -- remove interface hv1-vif${di} options tx_pcap > + > hv1/vif${di}-tx.pcap > + ovs-vsctl -- set interface hv1-vif${di} external-ids:iface-id=lsp${di}1 \ > + options:tx_pcap=hv1/vif${di}-tx.pcap > + done > +done > + > +OVN_CLEANUP([hv1]) > +AT_CLEANUP > +]) > + > + > +OVN_FOR_EACH_NORTHD([ > +AT_SETUP([route tables IPv6 -- overlapping subnets]) > +ovn_start > + > +# Logical network: > +# > +# ls1 (2001:db8:1::/64) - lrp-lr1-ls1 -\ /- lrp-lr1-ls2 - ls2 (2001:db8:2::/64) > +# lr1 > +# ls3 (2001:db8:3::/64) - lrp-lr1-ls3 -/ \- lrp-lr1-ls4 - ls4 (2001:db8:4::/64) > +# > +# ls1 has lsp11 (2001:db8:1::11) > +# ls2 has lsp21 (2001:db8:2::21) > +# ls3 has lsp31 (2001:db8:3::31) > +# ls4 has lsp41 (2001:db8:4::41) > +# > +# lrp-lr1-ls1 set options:route_table=rtb-1 > +# lrp-lr1-ls2 set options:route_table=rtb-2 > +# > +# Static routes on lr1: > +# 2001:db8:2000::/64 nexthop 2001:db8:3::31 route_table=rtb-1 > +# 2001:db8:2000::/64 nexthop 2001:db8:3::41 route_table=rtb-2 > +# > +# Test 1: > +# lsp11 send packet to 2001:db8:2000::1 > +# > +# Expected result: > +# lsp31 should receive traffic, lsp41 should not receive any traffic > +# > +# Test 2: > +# lsp21 send packet to 2001:db8:2000::1 > +# > +# Expected result: > +# lsp41 should receive traffic, lsp31 should not receive any traffic > + > +ovn-nbctl lr-add lr1 > + > +# Create logical topology > +for i in $(seq 1 4); do > + ovn-nbctl ls-add ls${i} > + ovn-nbctl lrp-add lr1 lrp-lr1-ls${i} 00:00:00:01:0${i}:01 2001:db8:${i}::1/64 > + ovn-nbctl lsp-add ls${i} lsp-ls${i}-lr1 -- lsp-set-type lsp-ls${i}-lr1 router \ > + -- lsp-set-options lsp-ls${i}-lr1 router-port=lrp-lr1-ls${i} \ > + -- lsp-set-addresses lsp-ls${i}-lr1 router > + ovn-nbctl lsp-add ls$i lsp${i}1 -- \ > + lsp-set-addresses lsp${i}1 "f0:00:00:00:0${i}:1${i} 2001:db8:${i}::${i}1" > +done > + > +# install static routes > +ovn-nbctl --route-table=rtb-1 lr-route-add lr1 2001:db8:2000::/64 2001:db8:3::31 > +ovn-nbctl --route-table=rtb-2 lr-route-add lr1 2001:db8:2000::/64 2001:db8:4::41 > + > +# set lrp-lr1-ls{1,2} route tables > +ovn-nbctl lrp-set-options lrp-lr1-ls1 route_table=rtb-1 > +ovn-nbctl lrp-set-options lrp-lr1-ls2 route_table=rtb-2 > + > +net_add n1 > +sim_add hv1 > +as hv1 > +ovs-vsctl add-br br-phys > +ovn_attach n1 br-phys 192.168.0.1 > + > +for i in $(seq 1 4); do > + ovs-vsctl -- add-port br-int hv1-vif${i} -- \ > + set interface hv1-vif${i} external-ids:iface-id=lsp${i}1 \ > + options:tx_pcap=hv1/vif${i}-tx.pcap \ > + options:rxq_pcap=hv1/vif${i}-rx.pcap \ > + ofport-request=${i} > +done > + > +# wait for earlier changes to take effect > +AT_CHECK([ovn-nbctl --timeout=3 --wait=hv sync], [0], [ignore]) > + > +# lsp31 should recieve packet coming from lsp11 > +# lsp41 should recieve packet coming from lsp21 > +for i in $(seq 1 2); do > + di=$(( i + 2)) # dst index > + ri=$(( 5 - i)) # reverse index > + packet="inport==\"lsp${i}1\" && eth.src==f0:00:00:00:0${i}:1${i} && > + eth.dst==00:00:00:01:0${i}:01 && ip6 && ip.ttl==64 && > + ip6.src==2001:db8:${i}::${i}1 && ip6.dst==2001:db8:2000::1 && icmp6" > + AT_CHECK([as hv1 ovs-appctl -t ovn-controller inject-pkt "$packet"]) > + > + # Assume all packets go to lsp${di}1. > + exp_packet="eth.src==00:00:00:01:0${di}:01 && eth.dst==f0:00:00:00:0${di}:1${di} && ip6 && > + ip.ttl==63 && ip6.src==2001:db8:${i}::${i}1 && ip6.dst==2001:db8:2000::1 && icmp6" > + echo $exp_packet | ovstest test-ovn expr-to-packets >> expected_lsp${di}1 > + > expected_lsp${ri}1 > + > + OVS_WAIT_UNTIL([ > + rcv_n1=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" hv1/vif${di}-tx.pcap > lsp${di}1.packets && cat lsp${di}1.packets | wc -l` > + rcv_n2=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" hv1/vif${ri}-tx.pcap > lsp${ri}1.packets && cat lsp${ri}1.packets | wc -l` > + echo $rcv_n1 $rcv_n2 > + test $rcv_n1 -eq 1 -a $rcv_n2 -eq 0]) > + > + for j in "${di}1" "${ri}1"; do > + sort expected_lsp${j} > expout > + AT_CHECK([cat lsp${j}.packets | sort], [0], [expout]) > + done > + > + # cleanup tx pcap files > + for j in "${di}1" "${ri}1"; do > + ovs-vsctl -- remove interface hv1-vif${di} options tx_pcap > + > hv1/vif${di}-tx.pcap > + ovs-vsctl -- set interface hv1-vif${di} external-ids:iface-id=lsp${di}1 \ > + options:tx_pcap=hv1/vif${di}-tx.pcap > + done > +done > + > +OVN_CLEANUP([hv1]) > +AT_CLEANUP > +]) > + > + > OVN_FOR_EACH_NORTHD([ > AT_SETUP([forwarding group: 3 HVs, 1 LR, 2 LS]) > AT_KEYWORDS([forwarding-group]) > @@ -23332,7 +23759,7 @@ ovn-sbctl dump-flows > sbflows > AT_CAPTURE_FILE([sbflows]) > AT_CAPTURE_FILE([offlows]) > OVS_WAIT_UNTIL([ > - as hv1 ovs-ofctl dump-flows br-int table=20 > offlows > + as hv1 ovs-ofctl dump-flows br-int table=21 > offlows > test $(grep -c "load:0x64->NXM_NX_PKT_MARK" offlows) = 1 && \ > test $(grep -c "load:0x3->NXM_NX_PKT_MARK" offlows) = 1 && \ > test $(grep -c "load:0x4->NXM_NX_PKT_MARK" offlows) = 1 && \ > @@ -23425,12 +23852,12 @@ send_ipv4_pkt hv1 hv1-vif1 505400000003 00000000ff01 \ > $(ip_to_hex 10 0 0 3) $(ip_to_hex 172 168 0 120) > > OVS_WAIT_UNTIL([ > - test 1 -eq $(as hv1 ovs-ofctl dump-flows br-int table=20 | \ > + test 1 -eq $(as hv1 ovs-ofctl dump-flows br-int table=21 | \ > grep "load:0x2->NXM_NX_PKT_MARK" -c) > ]) > > AT_CHECK([ > - test 0 -eq $(as hv1 ovs-ofctl dump-flows br-int table=20 | \ > + test 0 -eq $(as hv1 ovs-ofctl dump-flows br-int table=21 | \ > grep "load:0x64->NXM_NX_PKT_MARK" -c) > ]) > > @@ -24133,7 +24560,7 @@ AT_CHECK([ > grep "priority=100" | \ > grep -c "ct(commit,zone=NXM_NX_REG11\\[[0..15\\]],.*exec(move:NXM_OF_ETH_SRC\\[[\\]]->NXM_NX_CT_LABEL\\[[32..79\\]],load:0x[[0-9]]->NXM_NX_CT_LABEL\\[[80..95\\]]))" > > - grep table=22 hv${hv}flows | \ > + grep table=23 hv${hv}flows | \ > grep "priority=200" | \ > grep -c "actions=move:NXM_NX_CT_LABEL\\[[32..79\\]]->NXM_OF_ETH_DST\\[[\\]]" > done; :], [0], [dnl > @@ -24258,7 +24685,7 @@ AT_CHECK([ > grep "priority=100" | \ > grep -c "ct(commit,zone=NXM_NX_REG11\\[[0..15\\]],.*exec(move:NXM_OF_ETH_SRC\\[[\\]]->NXM_NX_CT_LABEL\\[[32..79\\]],load:0x[[0-9]]->NXM_NX_CT_LABEL\\[[80..95\\]]))" > > - grep table=22 hv${hv}flows | \ > + grep table=23 hv${hv}flows | \ > grep "priority=200" | \ > grep -c "actions=move:NXM_NX_CT_LABEL\\[[32..79\\]]->NXM_OF_ETH_DST\\[[\\]]" > done; :], [0], [dnl > @@ -24880,7 +25307,7 @@ AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | grep "actions=controller" | grep > ]) > > # The packet should've been dropped in the lr_in_arp_resolve stage. > -AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | grep -E "table=22, n_packets=1,.* priority=1,ip,metadata=0x${sw_key},nw_dst=10.0.1.1 actions=drop" -c], [0], [dnl > +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | grep -E "table=23, n_packets=1,.* priority=1,ip,metadata=0x${sw_key},nw_dst=10.0.1.1 actions=drop" -c], [0], [dnl > 1 > ]) > > diff --git a/utilities/ovn-nbctl.c b/utilities/ovn-nbctl.c > index e34bb65f7..0ff10618b 100644 > --- a/utilities/ovn-nbctl.c > +++ b/utilities/ovn-nbctl.c > @@ -329,6 +329,8 @@ Logical router port commands:\n\ > add logical port PORT on ROUTER\n\ > lrp-set-gateway-chassis PORT CHASSIS [PRIORITY]\n\ > set gateway chassis for port PORT\n\ > + lrp-set-options PORT KEY=VALUE [KEY=VALUE]...\n\ > + set router port options\n\ > lrp-del-gateway-chassis PORT CHASSIS\n\ > delete gateway chassis from port PORT\n\ > lrp-get-gateway-chassis PORT\n\ > @@ -351,11 +353,17 @@ Logical router port commands:\n\ > ('overlay' or 'bridged')\n\ > \n\ > Route commands:\n\ > - [--policy=POLICY] [--ecmp] [--ecmp-symmetric-reply] lr-route-add ROUTER \n\ > - PREFIX NEXTHOP [PORT]\n\ > + [--policy=POLICY]\n\ > + [--ecmp]\n\ > + [--ecmp-symmetric-reply]\n\ > + [--route-table=ROUTE_TABLE]\n\ > + lr-route-add ROUTER PREFIX NEXTHOP [PORT]\n\ > add a route to ROUTER\n\ > - [--policy=POLICY] lr-route-del ROUTER [PREFIX [NEXTHOP [PORT]]]\n\ > + [--policy=POLICY]\n\ > + [--route-table=ROUTE_TABLE]\n\ > + lr-route-del ROUTER [PREFIX [NEXTHOP [PORT]]]\n\ > remove routes from ROUTER\n\ > + [--route-table=ROUTE_TABLE]\n\ > lr-route-list ROUTER print routes for ROUTER\n\ > \n\ > Policy commands:\n\ > @@ -743,6 +751,11 @@ print_lr(const struct nbrec_logical_router *lr, struct ds *s) > ds_put_cstr(s, "]\n"); > } > > + const char *route_table = smap_get(&lrp->options, "route_table"); > + if (route_table) { > + ds_put_format(s, " route-table: %s\n", route_table); > + } > + > if (lrp->n_gateway_chassis) { > const struct nbrec_gateway_chassis **gcs; > > @@ -862,6 +875,7 @@ nbctl_pre_show(struct ctl_context *ctx) > ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_name); > ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_mac); > ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_networks); > + ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_options); > ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_gateway_chassis); > > ovsdb_idl_add_column(ctx->idl, &nbrec_gateway_chassis_col_chassis_name); > @@ -4000,11 +4014,19 @@ nbctl_lr_policy_list(struct ctl_context *ctx) > > static struct nbrec_logical_router_static_route * > nbctl_lr_get_route(const struct nbrec_logical_router *lr, char *prefix, > - char *next_hop, bool is_src_route, bool ecmp) > + char *next_hop, bool is_src_route, bool ecmp, > + char *route_table) > { > for (int i = 0; i < lr->n_static_routes; i++) { > struct nbrec_logical_router_static_route *route = lr->static_routes[i]; > > + /* Strict compare for route_table. > + * If route_table was not specified, > + * lookup for routes with empty route_table value. */ > + if (strcmp(route->route_table, route_table ? route_table : "")) { > + continue; > + } > + > /* Compare route policy. */ > char *nb_policy = route->policy; > bool nb_is_src_route = false; > @@ -4060,6 +4082,8 @@ nbctl_pre_lr_route_add(struct ctl_context *ctx) > &nbrec_logical_router_static_route_col_bfd); > ovsdb_idl_add_column(ctx->idl, > &nbrec_logical_router_static_route_col_options); > + ovsdb_idl_add_column(ctx->idl, > + &nbrec_logical_router_static_route_col_route_table); > } > > static char * OVS_WARN_UNUSED_RESULT > @@ -4090,6 +4114,7 @@ nbctl_lr_route_add(struct ctl_context *ctx) > } > } > > + char *route_table = shash_find_data(&ctx->options, "--route-table"); > bool v6_prefix = false; > prefix = normalize_ipv4_prefix_str(ctx->argv[2]); > if (!prefix) { > @@ -4166,7 +4191,8 @@ nbctl_lr_route_add(struct ctl_context *ctx) > bool ecmp = shash_find(&ctx->options, "--ecmp") != NULL || > ecmp_symmetric_reply; > struct nbrec_logical_router_static_route *route = > - nbctl_lr_get_route(lr, prefix, next_hop, is_src_route, ecmp); > + nbctl_lr_get_route(lr, prefix, next_hop, is_src_route, ecmp, > + route_table); > > /* Validations for nexthop = "discard" */ > if (is_discard_route) { > @@ -4230,7 +4256,8 @@ nbctl_lr_route_add(struct ctl_context *ctx) > } > > struct nbrec_logical_router_static_route *discard_route = > - nbctl_lr_get_route(lr, prefix, "discard", is_src_route, true); > + nbctl_lr_get_route(lr, prefix, "discard", is_src_route, true, > + route_table); > if (discard_route) { > ctl_error(ctx, "discard nexthop for the same ECMP route exists."); > goto cleanup; > @@ -4246,6 +4273,9 @@ nbctl_lr_route_add(struct ctl_context *ctx) > if (policy) { > nbrec_logical_router_static_route_set_policy(route, policy); > } > + if (route_table) { > + nbrec_logical_router_static_route_set_route_table(route, route_table); > + } > > if (ecmp_symmetric_reply) { > const struct smap options = SMAP_CONST1(&options, > @@ -4289,6 +4319,8 @@ nbctl_pre_lr_route_del(struct ctl_context *ctx) > &nbrec_logical_router_static_route_col_nexthop); > ovsdb_idl_add_column(ctx->idl, > &nbrec_logical_router_static_route_col_output_port); > + ovsdb_idl_add_column(ctx->idl, > + &nbrec_logical_router_static_route_col_route_table); > > } > > @@ -4302,6 +4334,7 @@ nbctl_lr_route_del(struct ctl_context *ctx) > return; > } > > + const char *route_table = shash_find_data(&ctx->options, "--route-table"); > const char *policy = shash_find_data(&ctx->options, "--policy"); > bool is_src_route = false; > if (policy) { > @@ -4392,6 +4425,14 @@ nbctl_lr_route_del(struct ctl_context *ctx) > } > } > > + /* Strict compare for route_table. > + * If route_table was not specified, > + * lookup for routes with empty route_table value. */ > + if (strcmp(lr->static_routes[i]->route_table, > + route_table ? route_table : "")) { > + continue; > + } > + > /* Compare output_port, if specified. */ > if (output_port) { > char *rt_output_port = lr->static_routes[i]->output_port; > @@ -5115,6 +5156,41 @@ nbctl_pre_lrp_del_gateway_chassis(struct ctl_context *ctx) > ovsdb_idl_add_column(ctx->idl, &nbrec_gateway_chassis_col_chassis_name); > } > > +static void > +nbctl_pre_lrp_options(struct ctl_context *ctx) > +{ > + ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_name); > + ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_options); > +} > + > +static void > +nbctl_lrp_set_options(struct ctl_context *ctx) > +{ > + const char *id = ctx->argv[1]; > + const struct nbrec_logical_router_port *lrp = NULL; > + size_t i; > + struct smap options = SMAP_INITIALIZER(&options); > + > + char *error = lrp_by_name_or_uuid(ctx, id, true, &lrp); > + if (error) { > + ctx->error = error; > + return; > + } > + for (i = 2; i < ctx->argc; i++) { > + char *key, *value; > + value = xstrdup(ctx->argv[i]); > + key = strsep(&value, "="); > + if (value) { > + smap_add(&options, key, value); > + } > + free(key); > + } > + > + nbrec_logical_router_port_set_options(lrp, &options); > + > + smap_destroy(&options); > +} > + > /* Removes logical router port 'lrp->gateway_chassis[idx]'. */ > static void > remove_gc(const struct nbrec_logical_router_port *lrp, size_t idx) > @@ -5891,6 +5967,7 @@ route_cmp_details(const struct nbrec_logical_router_static_route *r1, > } > return r1->output_port ? 1 : -1; > } > + > struct ipv4_route { > int priority; > ovs_be32 addr; > @@ -5900,6 +5977,11 @@ struct ipv4_route { > static int > __ipv4_route_cmp(const struct ipv4_route *r1, const struct ipv4_route *r2) > { > + int rtb_cmp = strcmp(r1->route->route_table, > + r2->route->route_table); > + if (rtb_cmp) { > + return rtb_cmp; > + } > if (r1->priority != r2->priority) { > return r1->priority > r2->priority ? -1 : 1; > } > @@ -5931,6 +6013,11 @@ struct ipv6_route { > static int > __ipv6_route_cmp(const struct ipv6_route *r1, const struct ipv6_route *r2) > { > + int rtb_cmp = strcmp(r1->route->route_table, > + r2->route->route_table); > + if (rtb_cmp) { > + return rtb_cmp; > + } > if (r1->priority != r2->priority) { > return r1->priority > r2->priority ? -1 : 1; > } > @@ -6018,6 +6105,8 @@ nbctl_pre_lr_route_list(struct ctl_context *ctx) > &nbrec_logical_router_static_route_col_options); > ovsdb_idl_add_column(ctx->idl, > &nbrec_logical_router_static_route_col_bfd); > + ovsdb_idl_add_column(ctx->idl, > + &nbrec_logical_router_static_route_col_route_table); > } > > static void > @@ -6035,12 +6124,17 @@ nbctl_lr_route_list(struct ctl_context *ctx) > return; > } > > + char *route_table = shash_find_data(&ctx->options, "--route-table"); > + > ipv4_routes = xmalloc(sizeof *ipv4_routes * lr->n_static_routes); > ipv6_routes = xmalloc(sizeof *ipv6_routes * lr->n_static_routes); > > for (int i = 0; i < lr->n_static_routes; i++) { > const struct nbrec_logical_router_static_route *route > = lr->static_routes[i]; > + if (route_table && strcmp(route->route_table, route_table)) { > + continue; > + } > unsigned int plen; > ovs_be32 ipv4; > const char *policy = route->policy ? route->policy : "dst-ip"; > @@ -6081,6 +6175,7 @@ nbctl_lr_route_list(struct ctl_context *ctx) > if (n_ipv4_routes) { > ds_put_cstr(&ctx->output, "IPv4 Routes\n"); > } > + const struct nbrec_logical_router_static_route *route; > for (int i = 0; i < n_ipv4_routes; i++) { > bool ecmp = false; > if (i < n_ipv4_routes - 1 && > @@ -6091,6 +6186,15 @@ nbctl_lr_route_list(struct ctl_context *ctx) > &ipv4_routes[i - 1])) { > ecmp = true; > } > + > + route = ipv4_routes[i].route; > + if (!i || (i > 0 && strcmp(route->route_table, > + ipv4_routes[i - 1].route->route_table))) { > + ds_put_format(&ctx->output, "%sRoute Table %s:\n", i ? "\n" : "", > + strlen(route->route_table) ? route->route_table > + : "global"); > + } > + > print_route(ipv4_routes[i].route, &ctx->output, ecmp); > } > > @@ -6108,6 +6212,15 @@ nbctl_lr_route_list(struct ctl_context *ctx) > &ipv6_routes[i - 1])) { > ecmp = true; > } > + > + route = ipv6_routes[i].route; > + if (!i || (i > 0 && strcmp(route->route_table, > + ipv6_routes[i - 1].route->route_table))) { > + ds_put_format(&ctx->output, "%sRoute Table %s:\n", i ? "\n" : "", > + strlen(route->route_table) ? route->route_table > + : "global"); > + } > + > print_route(ipv6_routes[i].route, &ctx->output, ecmp); > } > > @@ -6926,6 +7039,8 @@ static const struct ctl_command_syntax nbctl_commands[] = { > "PORT CHASSIS [PRIORITY]", > nbctl_pre_lrp_set_gateway_chassis, nbctl_lrp_set_gateway_chassis, > NULL, "--may-exist", RW }, > + { "lrp-set-options", 1, INT_MAX, "PORT KEY=VALUE [KEY=VALUE]...", > + nbctl_pre_lrp_options, nbctl_lrp_set_options, NULL, "", RW }, > { "lrp-del-gateway-chassis", 2, 2, "PORT CHASSIS", > nbctl_pre_lrp_del_gateway_chassis, nbctl_lrp_del_gateway_chassis, > NULL, "", RW }, > @@ -6949,12 +7064,13 @@ static const struct ctl_command_syntax nbctl_commands[] = { > /* logical router route commands. */ > { "lr-route-add", 3, 4, "ROUTER PREFIX NEXTHOP [PORT]", > nbctl_pre_lr_route_add, nbctl_lr_route_add, NULL, > - "--may-exist,--ecmp,--ecmp-symmetric-reply,--policy=,--bfd?", RW }, > + "--may-exist,--ecmp,--ecmp-symmetric-reply,--policy=,--route-table=,--bfd?", > + RW }, > { "lr-route-del", 1, 4, "ROUTER [PREFIX [NEXTHOP [PORT]]]", > nbctl_pre_lr_route_del, nbctl_lr_route_del, NULL, > - "--if-exists,--policy=", RW }, > + "--if-exists,--policy=,--route-table=", RW }, > { "lr-route-list", 1, 1, "ROUTER", nbctl_pre_lr_route_list, > - nbctl_lr_route_list, NULL, "", RO }, > + nbctl_lr_route_list, NULL, "--route-table=", RO }, > > /* Policy commands */ > { "lr-policy-add", 4, INT_MAX, > -- > 2.30.0 > > _______________________________________________ > dev mailing list > dev@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Hi Han, Thanks for the review. Regards, Vladislav Odintsov > On 14 Oct 2021, at 08:13, Han Zhou <hzhou@ovn.org> wrote: > > > > On Tue, Oct 5, 2021 at 1:26 PM Vladislav Odintsov <odivlad@gmail.com <mailto:odivlad@gmail.com>> wrote: > > > > This patch extends Logical Router's routing functionality. > > Now user may create multiple routing tables within a Logical Router > > and assign them to Logical Router Ports. > > > > Traffic coming from Logical Router Port with assigned route_table > > is checked against global routes if any (Logical_Router_Static_Routes > > whith empty route_table field), next against directly connected routes > > This is not accurate. The "directly connected routes" is NOT after the global routes. Their priority only depends on the prefix length. > > > and then Logical_Router_Static_Routes with same route_table value as > > in Logical_Router_Port options:route_table field. > > > > A new Logical Router ingress table #10 is added - IN_IP_ROUTING_PRE. > > In this table packets which come from LRPs with configured > > options:route_table field are checked against inport and in OVS > > register 7 unique non-zero value identifying route table is written. > > > > Then in 11th table IN_IP_ROUTING routes which have non-empty > > `route_table` field are added with additional match on reg7 value > > associated with appropriate route_table. > > > > Hi Vladislav, > > First of all, sorry for the delayed review, and thanks for implementing this new feature. > > I have some questions regarding the feature itself. I remember that we had some discussion earlier for this feature, but it seems I misunderstood the feature you are implementing here. I thought by multiple routing tables you were trying to support something like VRF in physical routers, but this seems to be something different. According to your implementation, instead of assigning LRPs to different routing tables, a LRP can actually participate to both the global routing table and the table with a specific ID. For ingress, global routes are prefered over other routes; for egress (i.e. forwarding to the next hop), it doesn't even enforce any table-id check, so packet routed by a entry of table-X can go out of a LRP with table-id Y. Is my understanding correct about the desired behavior of this feature? Yes, your understanding is correct. This is not VRF. At first glance VRF can be done just by using LR-per-VRF without any code modifications (maybe there are corner cases, but it’s not something I’m currently working on). LRP can be optionally assigned to specific routing table name. This means that for LR ingress pipeline packets coming from this specific LRP would be checked against routes with same route_table value within appropriate LR. This is some kind of PBR, analog of "ip rule add iif <interface name> lookup <id>". There is one specific use-case, which requires special handling: directly-connected routes (subnet CIDRs from connected to this LR LRPs). These routes can’t be added manually by user, though routing between LRPs belonging to different routing tables is still needed. So, these routes should be added to global routing table to override routing for LRPs with configured routing tables. If for some reason user wants to prohibit IP connectivity to any LRP (honestly, I can’t imagine, why), it can be done by utilising OVN PBR with drop action or a route with "discard" nexthop. But I’d think in this case whether this LRP is needed in this LR. Also, if user wants to create routes, which apply for all LRPs from all routing tables, it can be done by adding new entries to global routing table. And the third reason to have global routing table - a fully backward-compatible solution. All users, who don’t need multiple routing tables support just continue using static routing in the same manner without any changes. > > If this is correct, it doesn't seem to be a common/standard requirement (or please educate me if I am wrong). Could you explain a little more about the actual use cases: what kind of real world problems need to be solved by this feature or how are you going to use this. For example, why would a port need to participate in both routing tables? It looks like what you really need is policy routing instead of multiple isolated routing tables. I understand that you already use policy routing to implement ACLs, so it is not convenient to combine other policies (e.g. inport based routing) into the policy routing stage. If that's the case, would it be more generic to support multiple policy routing stages? My concern to the current approach is that it is implemented for a very special use case. It makes the code more complex but when there is a slightly different requirement in the future it becomes insufficient. I am thinking that policy routing seems more flexible and has more potential to be made more generic. Maybe I will have a better understanding when I hear more detailed use cases and considerations from you. I can't agree here in it’s uncommon requirement. This implementation was inspired by AWS Route Tables feature [1]: within a VPC (LR in terms of OVN) user may create multiple routing tables and assign them to different subnets (LRPs) in multiple availability zones. Auto-generated directly-connected routes from LRPs CIDRs are working in the same manner as they do in AWS - apply to all subnets regardless of their association to routing table. GCP has similar behaviour: [2], Azure, I guess, too. Our public cloud (CROC Cloud Platform) supports AWS behaviour [3], so I primarily was oriented on it. Internally we already use this feature and currently it fits our use-case, but again I can't say it is specific. In my opinion having this feature to be implemented using PBR is less convenient and native for users, who are familiar with behaviour for mentioned above public cloud platforms, because configuring routes should be done in routes section. And adding route table property seems native in this route, not in PBR. Moreover, I _think_ using Logical_Router_Static_Route to extend this feature for OVN-Interconnection becomes quite easy comparing to PBR (though, I didn’t try the latter). > > I haven't finished reviewing the code yet, but I have one comment regarding adding 100 to the priority of the global routes. For IPv6, the priority range from 0 to 120x2=240, so adding 100 is not enough. It would create overlapping priority ranges, and some table-id specific route entries may override the global routes. Thanks, I’ll dig into this when you finish review. Let me know if I answered your questions or if you have new ones. Again many thanks for your time and digging into this patch series. 1: https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Route_Tables.html 2: https://cloud.google.com/vpc/docs/routes#subnet-routes 3: https://docs.cloud.croc.ru/en/services/networks/routetables.html > Thanks, > Han > > > Signed-off-by: Vladislav Odintsov <odivlad@gmail.com <mailto:odivlad@gmail.com>> > > Acked-by: Numan Siddique <numans@ovn.org <mailto:numans@ovn.org>> > > --- > > northd/northd.c | 159 ++++++++++++--- > > northd/ovn-northd.8.xml | 63 ++++-- > > ovn-nb.ovsschema | 5 +- > > ovn-nb.xml | 30 +++ > > tests/ovn-ic.at <http://ovn-ic.at/> | 4 + > > tests/ovn-nbctl.at <http://ovn-nbctl.at/> | 196 +++++++++++++++++- > > tests/ovn-northd.at <http://ovn-northd.at/> | 76 ++++++- > > tests/ovn.at <http://ovn.at/> | 441 +++++++++++++++++++++++++++++++++++++++- > > utilities/ovn-nbctl.c | 134 +++++++++++- > > 9 files changed, 1041 insertions(+), 67 deletions(-) > > > > diff --git a/northd/northd.c b/northd/northd.c > > index 092eca829..6a020cb2e 100644 > > --- a/northd/northd.c > > +++ b/northd/northd.c > > @@ -148,15 +148,16 @@ enum ovn_stage { > > PIPELINE_STAGE(ROUTER, IN, ECMP_STATEFUL, 7, "lr_in_ecmp_stateful") \ > > PIPELINE_STAGE(ROUTER, IN, ND_RA_OPTIONS, 8, "lr_in_nd_ra_options") \ > > PIPELINE_STAGE(ROUTER, IN, ND_RA_RESPONSE, 9, "lr_in_nd_ra_response") \ > > - PIPELINE_STAGE(ROUTER, IN, IP_ROUTING, 10, "lr_in_ip_routing") \ > > - PIPELINE_STAGE(ROUTER, IN, IP_ROUTING_ECMP, 11, "lr_in_ip_routing_ecmp") \ > > - PIPELINE_STAGE(ROUTER, IN, POLICY, 12, "lr_in_policy") \ > > - PIPELINE_STAGE(ROUTER, IN, POLICY_ECMP, 13, "lr_in_policy_ecmp") \ > > - PIPELINE_STAGE(ROUTER, IN, ARP_RESOLVE, 14, "lr_in_arp_resolve") \ > > - PIPELINE_STAGE(ROUTER, IN, CHK_PKT_LEN , 15, "lr_in_chk_pkt_len") \ > > - PIPELINE_STAGE(ROUTER, IN, LARGER_PKTS, 16, "lr_in_larger_pkts") \ > > - PIPELINE_STAGE(ROUTER, IN, GW_REDIRECT, 17, "lr_in_gw_redirect") \ > > - PIPELINE_STAGE(ROUTER, IN, ARP_REQUEST, 18, "lr_in_arp_request") \ > > + PIPELINE_STAGE(ROUTER, IN, IP_ROUTING_PRE, 10, "lr_in_ip_routing_pre") \ > > + PIPELINE_STAGE(ROUTER, IN, IP_ROUTING, 11, "lr_in_ip_routing") \ > > + PIPELINE_STAGE(ROUTER, IN, IP_ROUTING_ECMP, 12, "lr_in_ip_routing_ecmp") \ > > + PIPELINE_STAGE(ROUTER, IN, POLICY, 13, "lr_in_policy") \ > > + PIPELINE_STAGE(ROUTER, IN, POLICY_ECMP, 14, "lr_in_policy_ecmp") \ > > + PIPELINE_STAGE(ROUTER, IN, ARP_RESOLVE, 15, "lr_in_arp_resolve") \ > > + PIPELINE_STAGE(ROUTER, IN, CHK_PKT_LEN, 16, "lr_in_chk_pkt_len") \ > > + PIPELINE_STAGE(ROUTER, IN, LARGER_PKTS, 17, "lr_in_larger_pkts") \ > > + PIPELINE_STAGE(ROUTER, IN, GW_REDIRECT, 18, "lr_in_gw_redirect") \ > > + PIPELINE_STAGE(ROUTER, IN, ARP_REQUEST, 19, "lr_in_arp_request") \ > > \ > > /* Logical router egress stages. */ \ > > PIPELINE_STAGE(ROUTER, OUT, UNDNAT, 0, "lr_out_undnat") \ > > @@ -225,6 +226,7 @@ enum ovn_stage { > > #define REG_NEXT_HOP_IPV6 "xxreg0" > > #define REG_SRC_IPV4 "reg1" > > #define REG_SRC_IPV6 "xxreg1" > > +#define REG_ROUTE_TABLE_ID "reg7" > > > > #define REG_ORIG_TP_DPORT_ROUTER "reg9[16..31]" > > > > @@ -287,8 +289,9 @@ enum ovn_stage { > > * | R6 | UNUSED | X | | G | IN_IP_ROUTING)| > > * | | | R | | 1 | | > > * +-----+--------------------------+ E | UNUSED | | | > > - * | R7 | UNUSED | G | | | | > > - * | | | 3 | | | | > > + * | R7 | ROUTE_TABLE_ID | G | | | | > > + * | | (>= IN_IP_ROUTING_PRE && | 3 | | | | > > + * | | <= IN_IP_ROUTING) | | | | | > > * +-----+--------------------------+---+-----------------+---+---------------+ > > * | R8 | ECMP_GROUP_ID | | | > > * | | ECMP_MEMBER_ID | X | | > > @@ -8511,11 +8514,72 @@ cleanup: > > ds_destroy(&actions); > > } > > > > +static uint32_t > > +route_table_add(struct simap *route_tables, const char *route_table_name) > > +{ > > + /* route table ids start from 1 */ > > + uint32_t rtb_id = simap_count(route_tables) + 1; > > + > > + if (rtb_id == UINT16_MAX) { > > + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1); > > + VLOG_WARN_RL(&rl, "too many route tables for Logical Router."); > > + return 0; > > + } > > + > > + if (!simap_put(route_tables, route_table_name, rtb_id)) { > > + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1); > > + VLOG_WARN_RL(&rl, "Route table id unexpectedly appeared"); > > + } > > + > > + return rtb_id; > > +} > > + > > +static uint32_t > > +get_route_table_id(struct simap *route_tables, const char *route_table_name) > > +{ > > + if (!route_table_name || !strlen(route_table_name)) { > > + return 0; > > + } > > + > > + uint32_t rtb_id = simap_get(route_tables, route_table_name); > > + if (!rtb_id) { > > + rtb_id = route_table_add(route_tables, route_table_name); > > + } > > + > > + return rtb_id; > > +} > > + > > +static void > > +build_route_table_lflow(struct ovn_datapath *od, struct hmap *lflows, > > + struct nbrec_logical_router_port *lrp, > > + struct simap *route_tables) > > +{ > > + struct ds match = DS_EMPTY_INITIALIZER; > > + struct ds actions = DS_EMPTY_INITIALIZER; > > + > > + const char *route_table_name = smap_get(&lrp->options, "route_table"); > > + uint32_t rtb_id = get_route_table_id(route_tables, route_table_name); > > + if (!rtb_id) { > > + return; > > + } > > + > > + ds_put_format(&match, "inport == \"%s\"", lrp->name); > > + ds_put_format(&actions, "%s = %d; next;", > > + REG_ROUTE_TABLE_ID, rtb_id); > > + > > + ovn_lflow_add(lflows, od, S_ROUTER_IN_IP_ROUTING_PRE, 100, > > + ds_cstr(&match), ds_cstr(&actions)); > > + > > + ds_destroy(&match); > > + ds_destroy(&actions); > > +} > > + > > struct parsed_route { > > struct ovs_list list_node; > > struct in6_addr prefix; > > unsigned int plen; > > bool is_src_route; > > + uint32_t route_table_id; > > uint32_t hash; > > const struct nbrec_logical_router_static_route *route; > > bool ecmp_symmetric_reply; > > @@ -8540,7 +8604,7 @@ find_static_route_outport(struct ovn_datapath *od, struct hmap *ports, > > * Otherwise return NULL. */ > > static struct parsed_route * > > parsed_routes_add(struct ovn_datapath *od, struct hmap *ports, > > - struct ovs_list *routes, > > + struct ovs_list *routes, struct simap *route_tables, > > const struct nbrec_logical_router_static_route *route, > > struct hmap *bfd_connections) > > { > > @@ -8622,6 +8686,7 @@ parsed_routes_add(struct ovn_datapath *od, struct hmap *ports, > > struct parsed_route *pr = xzalloc(sizeof *pr); > > pr->prefix = prefix; > > pr->plen = plen; > > + pr->route_table_id = get_route_table_id(route_tables, route->route_table); > > pr->is_src_route = (route->policy && !strcmp(route->policy, > > "src-ip")); > > pr->hash = route_hash(pr); > > @@ -8655,6 +8720,7 @@ struct ecmp_groups_node { > > struct in6_addr prefix; > > unsigned int plen; > > bool is_src_route; > > + uint32_t route_table_id; > > uint16_t route_count; > > struct ovs_list route_list; /* Contains ecmp_route_list_node */ > > }; > > @@ -8663,7 +8729,7 @@ static void > > ecmp_groups_add_route(struct ecmp_groups_node *group, > > const struct parsed_route *route) > > { > > - if (group->route_count == UINT16_MAX) { > > + if (group->route_count == UINT16_MAX) { > > static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1); > > VLOG_WARN_RL(&rl, "too many routes in a single ecmp group."); > > return; > > @@ -8692,6 +8758,7 @@ ecmp_groups_add(struct hmap *ecmp_groups, > > eg->prefix = route->prefix; > > eg->plen = route->plen; > > eg->is_src_route = route->is_src_route; > > + eg->route_table_id = route->route_table_id; > > ovs_list_init(&eg->route_list); > > ecmp_groups_add_route(eg, route); > > > > @@ -8705,7 +8772,8 @@ ecmp_groups_find(struct hmap *ecmp_groups, struct parsed_route *route) > > HMAP_FOR_EACH_WITH_HASH (eg, hmap_node, route->hash, ecmp_groups) { > > if (ipv6_addr_equals(&eg->prefix, &route->prefix) && > > eg->plen == route->plen && > > - eg->is_src_route == route->is_src_route) { > > + eg->is_src_route == route->is_src_route && > > + eg->route_table_id == route->route_table_id) { > > return eg; > > } > > } > > @@ -8752,7 +8820,8 @@ unique_routes_remove(struct hmap *unique_routes, > > HMAP_FOR_EACH_WITH_HASH (ur, hmap_node, route->hash, unique_routes) { > > if (ipv6_addr_equals(&route->prefix, &ur->route->prefix) && > > route->plen == ur->route->plen && > > - route->is_src_route == ur->route->is_src_route) { > > + route->is_src_route == ur->route->is_src_route && > > + route->route_table_id == ur->route->route_table_id) { > > hmap_remove(unique_routes, &ur->hmap_node); > > const struct parsed_route *existed_route = ur->route; > > free(ur); > > @@ -8790,9 +8859,9 @@ build_route_prefix_s(const struct in6_addr *prefix, unsigned int plen) > > } > > > > static void > > -build_route_match(const struct ovn_port *op_inport, const char *network_s, > > - int plen, bool is_src_route, bool is_ipv4, struct ds *match, > > - uint16_t *priority) > > +build_route_match(const struct ovn_port *op_inport, uint32_t rtb_id, > > + const char *network_s, int plen, bool is_src_route, > > + bool is_ipv4, struct ds *match, uint16_t *priority) > > { > > const char *dir; > > /* The priority here is calculated to implement longest-prefix-match > > @@ -8808,6 +8877,15 @@ build_route_match(const struct ovn_port *op_inport, const char *network_s, > > if (op_inport) { > > ds_put_format(match, "inport == %s && ", op_inport->json_key); > > } > > + if (rtb_id) { > > + ds_put_format(match, "%s == %d && ", REG_ROUTE_TABLE_ID, rtb_id); > > + } else { > > + /* Route-table assigned LRPs' routes should have lower priority > > + * in order not to affect directly-connected global routes. > > + * So, enlarge non-route-table routes priority by 100. > > + */ > > + *priority += 100; > > + } > > ds_put_format(match, "ip%s.%s == %s/%d", is_ipv4 ? "4" : "6", dir, > > network_s, plen); > > } > > @@ -8946,7 +9024,7 @@ add_ecmp_symmetric_reply_flows(struct hmap *lflows, > > out_port->lrp_networks.ea_s, > > IN6_IS_ADDR_V4MAPPED(&route->prefix) ? "" : "xx", > > port_ip, out_port->json_key); > > - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_IP_ROUTING, 300, > > + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_IP_ROUTING, 400, > > ds_cstr(&match), ds_cstr(&actions), > > &st_route->header_); > > > > @@ -8976,8 +9054,8 @@ build_ecmp_route_flow(struct hmap *lflows, struct ovn_datapath *od, > > struct ds route_match = DS_EMPTY_INITIALIZER; > > > > char *prefix_s = build_route_prefix_s(&eg->prefix, eg->plen); > > - build_route_match(NULL, prefix_s, eg->plen, eg->is_src_route, is_ipv4, > > - &route_match, &priority); > > + build_route_match(NULL, eg->route_table_id, prefix_s, eg->plen, > > + eg->is_src_route, is_ipv4, &route_match, &priority); > > free(prefix_s); > > > > struct ds actions = DS_EMPTY_INITIALIZER; > > @@ -9052,8 +9130,8 @@ static void > > add_route(struct hmap *lflows, struct ovn_datapath *od, > > const struct ovn_port *op, const char *lrp_addr_s, > > const char *network_s, int plen, const char *gateway, > > - bool is_src_route, const struct ovsdb_idl_row *stage_hint, > > - bool is_discard_route) > > + bool is_src_route, const uint32_t rtb_id, > > + const struct ovsdb_idl_row *stage_hint, bool is_discard_route) > > { > > bool is_ipv4 = strchr(network_s, '.') ? true : false; > > struct ds match = DS_EMPTY_INITIALIZER; > > @@ -9068,8 +9146,8 @@ add_route(struct hmap *lflows, struct ovn_datapath *od, > > op_inport = op; > > } > > } > > - build_route_match(op_inport, network_s, plen, is_src_route, is_ipv4, > > - &match, &priority); > > + build_route_match(op_inport, rtb_id, network_s, plen, is_src_route, > > + is_ipv4, &match, &priority); > > > > struct ds common_actions = DS_EMPTY_INITIALIZER; > > struct ds actions = DS_EMPTY_INITIALIZER; > > @@ -9132,7 +9210,8 @@ build_static_route_flow(struct hmap *lflows, struct ovn_datapath *od, > > char *prefix_s = build_route_prefix_s(&route_->prefix, route_->plen); > > add_route(lflows, route_->is_discard_route ? od : out_port->od, out_port, > > lrp_addr_s, prefix_s, route_->plen, route->nexthop, > > - route_->is_src_route, &route->header_, route_->is_discard_route); > > + route_->is_src_route, route_->route_table_id, &route->header_, > > + route_->is_discard_route); > > > > free(prefix_s); > > } > > @@ -10584,6 +10663,17 @@ build_ND_RA_flows_for_lrouter(struct ovn_datapath *od, struct hmap *lflows) > > } > > } > > > > +/* Logical router ingress table IP_ROUTING_PRE: > > + * by default goto next. (priority 0). */ > > +static void > > +build_ip_routing_pre_flows_for_lrouter(struct ovn_datapath *od, > > + struct hmap *lflows) > > +{ > > + if (od->nbr) { > > + ovn_lflow_add(lflows, od, S_ROUTER_IN_IP_ROUTING_PRE, 0, "1", "next;"); > > + } > > +} > > + > > /* Logical router ingress table IP_ROUTING : IP Routing. > > * > > * A packet that arrives at this table is an IP packet that should be > > @@ -10609,14 +10699,14 @@ build_ip_routing_flows_for_lrouter_port( > > for (int i = 0; i < op->lrp_networks.n_ipv4_addrs; i++) { > > add_route(lflows, op->od, op, op->lrp_networks.ipv4_addrs[i].addr_s, > > op->lrp_networks.ipv4_addrs[i].network_s, > > - op->lrp_networks.ipv4_addrs[i].plen, NULL, false, > > + op->lrp_networks.ipv4_addrs[i].plen, NULL, false, 0, > > &op->nbrp->header_, false); > > } > > > > for (int i = 0; i < op->lrp_networks.n_ipv6_addrs; i++) { > > add_route(lflows, op->od, op, op->lrp_networks.ipv6_addrs[i].addr_s, > > op->lrp_networks.ipv6_addrs[i].network_s, > > - op->lrp_networks.ipv6_addrs[i].plen, NULL, false, > > + op->lrp_networks.ipv6_addrs[i].plen, NULL, false, 0, > > &op->nbrp->header_, false); > > } > > } else if (lsp_is_router(op->nbsp)) { > > @@ -10639,7 +10729,7 @@ build_ip_routing_flows_for_lrouter_port( > > add_route(lflows, peer->od, peer, > > peer->lrp_networks.ipv4_addrs[0].addr_s, > > laddrs->ipv4_addrs[k].network_s, > > - laddrs->ipv4_addrs[k].plen, NULL, false, > > + laddrs->ipv4_addrs[k].plen, NULL, false, 0, > > &peer->nbrp->header_, false); > > } > > } > > @@ -10659,10 +10749,17 @@ build_static_route_flows_for_lrouter( > > struct hmap ecmp_groups = HMAP_INITIALIZER(&ecmp_groups); > > struct hmap unique_routes = HMAP_INITIALIZER(&unique_routes); > > struct ovs_list parsed_routes = OVS_LIST_INITIALIZER(&parsed_routes); > > + struct simap route_tables = SIMAP_INITIALIZER(&route_tables); > > struct ecmp_groups_node *group; > > + > > + for (int i = 0; i < od->nbr->n_ports; i++) { > > + build_route_table_lflow(od, lflows, od->nbr->ports[i], > > + &route_tables); > > + } > > + > > for (int i = 0; i < od->nbr->n_static_routes; i++) { > > struct parsed_route *route = > > - parsed_routes_add(od, ports, &parsed_routes, > > + parsed_routes_add(od, ports, &parsed_routes, &route_tables, > > od->nbr->static_routes[i], bfd_connections); > > if (!route) { > > continue; > > @@ -10695,6 +10792,7 @@ build_static_route_flows_for_lrouter( > > ecmp_groups_destroy(&ecmp_groups); > > unique_routes_destroy(&unique_routes); > > parsed_routes_destroy(&parsed_routes); > > + simap_destroy(&route_tables); > > } > > } > > > > @@ -12800,6 +12898,7 @@ build_lswitch_and_lrouter_iterate_by_od(struct ovn_datapath *od, > > build_neigh_learning_flows_for_lrouter(od, lsi->lflows, &lsi->match, > > &lsi->actions, lsi->meter_groups); > > build_ND_RA_flows_for_lrouter(od, lsi->lflows); > > + build_ip_routing_pre_flows_for_lrouter(od, lsi->lflows); > > build_static_route_flows_for_lrouter(od, lsi->lflows, lsi->ports, > > lsi->bfd_connections); > > build_mcast_lookup_flows_for_lrouter(od, lsi->lflows, &lsi->match, > > diff --git a/northd/ovn-northd.8.xml b/northd/ovn-northd.8.xml > > index 39f4eaa0c..cc2e25367 100644 > > --- a/northd/ovn-northd.8.xml > > +++ b/northd/ovn-northd.8.xml > > @@ -2899,7 +2899,7 @@ icmp6 { > > > > <p> > > If ECMP routes with symmetric reply are configured in the > > - <code>OVN_Northbound</code> database for a gateway router, a priority-300 > > + <code>OVN_Northbound</code> database for a gateway router, a priority-400 > > flow is added for each router port on which symmetric replies are > > configured. The matching logic for these ports essentially reverses the > > configured logic of the ECMP route. So for instance, a route with a > > @@ -3245,7 +3245,35 @@ output; > > </li> > > </ul> > > > > - <h3>Ingress Table 10: IP Routing</h3> > > + <h3>Ingress Table 10: IP Routing Pre</h3> > > + > > + <p> > > + If a packet arrived at this table from Logical Router Port <var>P</var> > > + which has <code>options:route_table</code> value set, a logical flow with > > + match <code>inport == "<var>P</var>"</code> with priority 100 and action, > > + setting unique-generated per-datapath 32-bit value (non-zero) in OVS > > + register 7. This register is checked in next table. > > + </p> > > + > > + <p> > > + This table contains the following logical flows: > > + </p> > > + > > + <ul> > > + <li> > > + <p> > > + Priority-100 flow with match <code>inport == "LRP_NAME"</code> value > > + and action, which set route table identifier in reg7. > > + </p> > > + > > + <p> > > + A priority-0 logical flow with match <code>1</code> has actions > > + <code>next;</code>. > > + </p> > > + </li> > > + </ul> > > + > > + <h3>Ingress Table 11: IP Routing</h3> > > > > <p> > > A packet that arrives at this table is an IP packet that should be > > @@ -3316,10 +3344,10 @@ output; > > <p> > > IPv4 routing table. For each route to IPv4 network <var>N</var> with > > netmask <var>M</var>, on router port <var>P</var> with IP address > > - <var>A</var> and Ethernet > > - address <var>E</var>, a logical flow with match <code>ip4.dst == > > - <var>N</var>/<var>M</var></code>, whose priority is the number of > > - 1-bits in <var>M</var>, has the following actions: > > + <var>A</var> and Ethernet address <var>E</var>, a logical flow with > > + match <code>ip4.dst == <var>N</var>/<var>M</var></code>, whose > > + priority is 100 + the number of 1-bits in <var>M</var>, has the > > + following actions: > > </p> > > > > <pre> > > @@ -3382,6 +3410,13 @@ next; > > If the address <var>A</var> is in the link-local scope, the > > route will be limited to sending on the ingress port. > > </p> > > + > > + <p> > > + For routes with <code>route_table</code> value set > > + <code>reg7 == id</code> is prefixed in logical flow match portion. > > + Priority for routes with <code>route_table</code> value set is > > + the number of 1-bits in <var>M</var>. > > + </p> > > </li> > > > > <li> > > @@ -3408,7 +3443,7 @@ select(reg8[16..31], <var>MID1</var>, <var>MID2</var>, ...); > > </li> > > </ul> > > > > - <h3>Ingress Table 11: IP_ROUTING_ECMP</h3> > > + <h3>Ingress Table 12: IP_ROUTING_ECMP</h3> > > > > <p> > > This table implements the second part of IP routing for ECMP routes > > @@ -3460,7 +3495,7 @@ outport = <var>P</var>; > > </li> > > </ul> > > > > - <h3>Ingress Table 12: Router policies</h3> > > + <h3>Ingress Table 13: Router policies</h3> > > <p> > > This table adds flows for the logical router policies configured > > on the logical router. Please see the > > @@ -3532,7 +3567,7 @@ next; > > </li> > > </ul> > > > > - <h3>Ingress Table 13: ECMP handling for router policies</h3> > > + <h3>Ingress Table 14: ECMP handling for router policies</h3> > > <p> > > This table handles the ECMP for the router policies configured > > with multiple nexthops. > > @@ -3576,7 +3611,7 @@ outport = <var>P</var> > > </li> > > </ul> > > > > - <h3>Ingress Table 14: ARP/ND Resolution</h3> > > + <h3>Ingress Table 15: ARP/ND Resolution</h3> > > > > <p> > > Any packet that reaches this table is an IP packet whose next-hop > > @@ -3767,7 +3802,7 @@ outport = <var>P</var> > > > > </ul> > > > > - <h3>Ingress Table 15: Check packet length</h3> > > + <h3>Ingress Table 16: Check packet length</h3> > > > > <p> > > For distributed logical routers or gateway routers with gateway > > @@ -3797,7 +3832,7 @@ REGBIT_PKT_LARGER = check_pkt_larger(<var>L</var>); next; > > and advances to the next table. > > </p> > > > > - <h3>Ingress Table 16: Handle larger packets</h3> > > + <h3>Ingress Table 17: Handle larger packets</h3> > > > > <p> > > For distributed logical routers or gateway routers with gateway port > > @@ -3860,7 +3895,7 @@ icmp6 { > > and advances to the next table. > > </p> > > > > - <h3>Ingress Table 17: Gateway Redirect</h3> > > + <h3>Ingress Table 18: Gateway Redirect</h3> > > > > <p> > > For distributed logical routers where one or more of the logical router > > @@ -3907,7 +3942,7 @@ icmp6 { > > </li> > > </ul> > > > > - <h3>Ingress Table 18: ARP Request</h3> > > + <h3>Ingress Table 19: ARP Request</h3> > > > > <p> > > In the common case where the Ethernet destination has been resolved, this > > diff --git a/ovn-nb.ovsschema b/ovn-nb.ovsschema > > index 2ac8ef3ea..a0a171e19 100644 > > --- a/ovn-nb.ovsschema > > +++ b/ovn-nb.ovsschema > > @@ -1,7 +1,7 @@ > > { > > "name": "OVN_Northbound", > > - "version": "5.32.1", > > - "cksum": "2805328215 29734", > > + "version": "5.33.1", > > + "cksum": "3874993350 29785", > > "tables": { > > "NB_Global": { > > "columns": { > > @@ -387,6 +387,7 @@ > > "isRoot": false}, > > "Logical_Router_Static_Route": { > > "columns": { > > + "route_table": {"type": "string"}, > > "ip_prefix": {"type": "string"}, > > "policy": {"type": {"key": {"type": "string", > > "enum": ["set", ["src-ip", > > diff --git a/ovn-nb.xml b/ovn-nb.xml > > index d8266ed4d..b2917c363 100644 > > --- a/ovn-nb.xml > > +++ b/ovn-nb.xml > > @@ -2772,6 +2772,14 @@ > > prefix according to RFC3663 > > </p> > > </column> > > + > > + <column name="options" key="route_table"> > > + Designates lookup Logical_Router_Static_Routes with specified > > + <code>route_table</code> value. Routes to directly connected networks > > + from same Logical Router and routes without <code>route_table</code> > > + option set have higher priority than routes with > > + <code>route_table</code> option set. > > + </column> > > </group> > > > > <group title="Attachment"> > > @@ -2891,6 +2899,28 @@ > > </p> > > </column> > > > > + <column name="route_table"> > > + <p> > > + Any string to place route to separate routing table. If Logical Router > > + Port has configured value in <ref table="Logical_Router_Port" > > + column="options" key="route_table"/> other than empty string, OVN > > + performs route lookup for all packets entering Logical Router ingress > > + pipeline from this port in the following manner: > > + </p> > > + > > + <ul> > > + <li> > > + 1. First lookup among "global" routes: routes without > > + <code>route_table</code> value set and routes to directly connected > > + networks. > > + </li> > > + <li> > > + 2. Next lookup among routes with same <code>route_table</code> value > > + as specified in LRP's options:route_table field. > > + </li> > > + </ul> > > + </column> > > + > > <column name="external_ids" key="ic-learned-route"> > > <code>ovn-ic</code> populates this key if the route is learned from the > > global <ref db="OVN_IC_Southbound"/> database. In this case the value > > diff --git a/tests/ovn-ic.at <http://ovn-ic.at/> b/tests/ovn-ic.at <http://ovn-ic.at/> > > index 32f4e9d02..3aab54362 100644 > > --- a/tests/ovn-ic.at <http://ovn-ic.at/> > > +++ b/tests/ovn-ic.at <http://ovn-ic.at/> > > @@ -281,6 +281,7 @@ done > > > > AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr1], [0], [dnl > > IPv4 Routes > > +Route Table global: > > 10.11.1.0/24 <http://10.11.1.0/24> 169.254.0.1 dst-ip > > 10.11.2.0/24 <http://10.11.2.0/24> 169.254.100.2 dst-ip (learned) > > 10.22.1.0/24 <http://10.22.1.0/24> 169.254.0.2 src-ip > > @@ -299,6 +300,7 @@ ovn_as az1 ovn-nbctl set nb_global . options:ic-route-learn=false > > OVS_WAIT_WHILE([ovn_as az1 ovn-nbctl lr-route-list lr1 | grep learned]) > > AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr1], [0], [dnl > > IPv4 Routes > > +Route Table global: > > 10.11.1.0/24 <http://10.11.1.0/24> 169.254.0.1 dst-ip > > 10.22.1.0/24 <http://10.22.1.0/24> 169.254.0.2 src-ip > > ]) > > @@ -314,6 +316,7 @@ ovn_as az1 ovn-nbctl set nb_global . options:ic-route-adv=false > > OVS_WAIT_WHILE([ovn_as az2 ovn-nbctl lr-route-list lr2 | grep learned]) > > AT_CHECK([ovn_as az2 ovn-nbctl lr-route-list lr2], [0], [dnl > > IPv4 Routes > > +Route Table global: > > 10.11.2.0/24 <http://10.11.2.0/24> 169.254.0.1 dst-ip > > 10.22.2.0/24 <http://10.22.2.0/24> 169.254.0.2 src-ip > > ]) > > @@ -332,6 +335,7 @@ done > > # Default route should NOT get advertised or learned, by default. > > AT_CHECK([ovn_as az2 ovn-nbctl lr-route-list lr2], [0], [dnl > > IPv4 Routes > > +Route Table global: > > 10.11.1.0/24 <http://10.11.1.0/24> 169.254.100.1 dst-ip (learned) > > 10.11.2.0/24 <http://10.11.2.0/24> 169.254.0.1 dst-ip > > 10.22.2.0/24 <http://10.22.2.0/24> 169.254.0.2 src-ip > > diff --git a/tests/ovn-nbctl.at <http://ovn-nbctl.at/> b/tests/ovn-nbctl.at <http://ovn-nbctl.at/> > > index 9b80ae410..ddb5536ce 100644 > > --- a/tests/ovn-nbctl.at <http://ovn-nbctl.at/> > > +++ b/tests/ovn-nbctl.at <http://ovn-nbctl.at/> > > @@ -1520,6 +1520,7 @@ AT_CHECK([ovn-nbctl --ecmp --policy=src-ip lr-route-add lr0 20.0.0.0/24 <http://20.0.0.0/24> 11.0.0.1 > > > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > IPv4 Routes > > +Route Table global: > > 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.1 dst-ip > > 10.0.1.0/24 <http://10.0.1.0/24> 11.0.1.1 dst-ip lp0 > > 10.0.10.0/24 <http://10.0.10.0/24> dst-ip lp0 > > @@ -1534,6 +1535,7 @@ AT_CHECK([ovn-nbctl lrp-add lr0 lp1 f0:00:00:00:00:02 11.0.0.254/24 <http://11.0.0.254/24>]) > > AT_CHECK([ovn-nbctl --may-exist lr-route-add lr0 10.0.0.111/24 <http://10.0.0.111/24> 11.0.0.1 lp1]) > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > IPv4 Routes > > +Route Table global: > > 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.1 dst-ip lp1 > > 10.0.1.0/24 <http://10.0.1.0/24> 11.0.1.1 dst-ip lp0 > > 10.0.10.0/24 <http://10.0.10.0/24> dst-ip lp0 > > @@ -1564,6 +1566,7 @@ AT_CHECK([ovn-nbctl --policy=src-ip lr-route-del lr0 9.16.1.0/24 <http://9.16.1.0/24>]) > > > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > IPv4 Routes > > +Route Table global: > > 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.1 dst-ip lp1 > > 10.0.10.0/24 <http://10.0.10.0/24> dst-ip lp0 > > 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.2 src-ip > > @@ -1575,6 +1578,7 @@ AT_CHECK([ovn-nbctl --policy=dst-ip lr-route-del lr0 10.0.0.0/24 <http://10.0.0.0/24>]) > > AT_CHECK([ovn-nbctl --policy=src-ip lr-route-del lr0 10.0.0.0/24 <http://10.0.0.0/24>]) > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > IPv4 Routes > > +Route Table global: > > 10.0.10.0/24 <http://10.0.10.0/24> dst-ip lp0 > > 0.0.0.0/0 <http://0.0.0.0/0> 192.168.0.1 dst-ip > > ]) > > @@ -1585,6 +1589,7 @@ AT_CHECK([ovn-nbctl --policy=src-ip lr-route-add lr0 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.2]) > > AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 <http://10.0.0.0/24>]) > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > IPv4 Routes > > +Route Table global: > > 10.0.10.0/24 <http://10.0.10.0/24> dst-ip lp0 > > 0.0.0.0/0 <http://0.0.0.0/0> 192.168.0.1 dst-ip > > ]) > > @@ -1601,6 +1606,7 @@ AT_CHECK([ovn-nbctl --ecmp lr-route-add lr0 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.3]) > > AT_CHECK([ovn-nbctl --ecmp lr-route-add lr0 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.4 lp0]) > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > IPv4 Routes > > +Route Table global: > > 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.1 dst-ip ecmp > > 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.2 dst-ip ecmp > > 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.3 dst-ip ecmp > > @@ -1615,6 +1621,7 @@ dnl Delete ecmp routes > > AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.1]) > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > IPv4 Routes > > +Route Table global: > > 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.2 dst-ip ecmp > > 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.3 dst-ip ecmp > > 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.4 dst-ip lp0 ecmp > > @@ -1622,12 +1629,14 @@ IPv4 Routes > > AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.2]) > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > IPv4 Routes > > +Route Table global: > > 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.3 dst-ip ecmp > > 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.4 dst-ip lp0 ecmp > > ]) > > AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.4 lp0]) > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > IPv4 Routes > > +Route Table global: > > 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.3 dst-ip > > ]) > > AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.3]) > > @@ -1641,6 +1650,7 @@ AT_CHECK([ovn-nbctl lr-route-add lr0 2001:0db8:1::/64 2001:0db8:0:f103::1]) > > > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > IPv6 Routes > > +Route Table global: > > 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > > 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > > ::/0 2001:db8:0:f101::1 dst-ip > > @@ -1650,6 +1660,7 @@ AT_CHECK([ovn-nbctl lr-route-del lr0 2001:0db8:0::/64]) > > > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > IPv6 Routes > > +Route Table global: > > 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > > ::/0 2001:db8:0:f101::1 dst-ip > > ]) > > @@ -1677,11 +1688,13 @@ AT_CHECK([ovn-nbctl --may-exist --ecmp-symmetric-reply lr-route-add lr0 2003:0db > > > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > IPv4 Routes > > +Route Table global: > > 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.1 dst-ip > > 10.0.1.0/24 <http://10.0.1.0/24> 11.0.1.1 dst-ip lp0 > > 0.0.0.0/0 <http://0.0.0.0/0> 192.168.0.1 dst-ip > > > > IPv6 Routes > > +Route Table global: > > 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > > 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip ecmp > > 2001:db8:1::/64 2001:db8:0:f103::2 dst-ip ecmp > > @@ -1696,7 +1709,188 @@ AT_CHECK([ovn-nbctl lrp-add lr0 lr0-p0 00:00:01:01:02:03 192.168.10.1/24 <http://192.168.10.1/24>]) > > bfd_uuid=$(ovn-nbctl create bfd logical_port=lr0-p0 dst_ip=100.0.0.50 status=down min_tx=250 min_rx=250 detect_mult=10) > > AT_CHECK([ovn-nbctl lr-route-add lr0 100.0.0.0/24 <http://100.0.0.0/24> 192.168.0.1]) > > route_uuid=$(fetch_column nb:logical_router_static_route _uuid ip_prefix="100.0.0.0/24 <http://100.0.0.0/24>") > > -AT_CHECK([ovn-nbctl set logical_router_static_route $route_uuid bfd=$bfd_uuid])]) > > +AT_CHECK([ovn-nbctl set logical_router_static_route $route_uuid bfd=$bfd_uuid]) > > + > > +check ovn-nbctl lr-route-del lr0 > > +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > +]) > > + > > +dnl Check IPv4 routes in route table > > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 0.0.0.0/0 <http://0.0.0.0/0> 192.168.0.1 > > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 10.0.1.1/24 <http://10.0.1.1/24> 11.0.1.1 lp0 > > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 10.0.0.1/24 <http://10.0.0.1/24> 11.0.0.1 > > +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > +IPv4 Routes > > +Route Table rtb-1: > > + 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.1 dst-ip > > + 10.0.1.0/24 <http://10.0.1.0/24> 11.0.1.1 dst-ip lp0 > > + 0.0.0.0/0 <http://0.0.0.0/0> 192.168.0.1 dst-ip > > +]) > > + > > +check ovn-nbctl lr-route-del lr0 > > +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > +]) > > + > > +dnl Check IPv6 routes in route table > > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 0:0:0:0:0:0:0:0/0 2001:0db8:0:f101::1 > > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 2001:0db8:0::/64 2001:0db8:0:f102::1 lp0 > > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 2001:0db8:1::/64 2001:0db8:0:f103::1 > > + > > +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > +IPv6 Routes > > +Route Table rtb-1: > > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > > + ::/0 2001:db8:0:f101::1 dst-ip > > +]) > > + > > +dnl Check IPv4 and IPv6 routes in route table > > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 0.0.0.0/0 <http://0.0.0.0/0> 192.168.0.1 > > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 10.0.1.1/24 <http://10.0.1.1/24> 11.0.1.1 lp0 > > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 10.0.0.1/24 <http://10.0.0.1/24> 11.0.0.1 > > + > > +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > +IPv4 Routes > > +Route Table rtb-1: > > + 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.1 dst-ip > > + 10.0.1.0/24 <http://10.0.1.0/24> 11.0.1.1 dst-ip lp0 > > + 0.0.0.0/0 <http://0.0.0.0/0> 192.168.0.1 dst-ip > > + > > +IPv6 Routes > > +Route Table rtb-1: > > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > > + ::/0 2001:db8:0:f101::1 dst-ip > > +]) > > + > > +# Add routes in another route table > > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 0.0.0.0/0 <http://0.0.0.0/0> 192.168.0.1 > > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 10.0.1.1/24 <http://10.0.1.1/24> 11.0.1.1 lp0 > > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 10.0.0.1/24 <http://10.0.0.1/24> 11.0.0.1 > > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 0:0:0:0:0:0:0:0/0 2001:0db8:0:f101::1 > > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 2001:0db8:0::/64 2001:0db8:0:f102::1 lp0 > > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 2001:0db8:1::/64 2001:0db8:0:f103::1 > > + > > +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > +IPv4 Routes > > +Route Table rtb-1: > > + 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.1 dst-ip > > + 10.0.1.0/24 <http://10.0.1.0/24> 11.0.1.1 dst-ip lp0 > > + 0.0.0.0/0 <http://0.0.0.0/0> 192.168.0.1 dst-ip > > + > > +Route Table rtb-2: > > + 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.1 dst-ip > > + 10.0.1.0/24 <http://10.0.1.0/24> 11.0.1.1 dst-ip lp0 > > + 0.0.0.0/0 <http://0.0.0.0/0> 192.168.0.1 dst-ip > > + > > +IPv6 Routes > > +Route Table rtb-1: > > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > > + ::/0 2001:db8:0:f101::1 dst-ip > > + > > +Route Table rtb-2: > > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > > + ::/0 2001:db8:0:f101::1 dst-ip > > +]) > > + > > +# Add routes to global route table > > +check ovn-nbctl lr-route-add lr0 0.0.0.0/0 <http://0.0.0.0/0> 192.168.0.1 > > +check ovn-nbctl lr-route-add lr0 10.0.1.1/24 <http://10.0.1.1/24> 11.0.1.1 lp0 > > +check ovn-nbctl lr-route-add lr0 10.0.0.1/24 <http://10.0.0.1/24> 11.0.0.1 > > +check ovn-nbctl lr-route-add lr0 0:0:0:0:0:0:0:0/0 2001:0db8:0:f101::1 > > +check ovn-nbctl lr-route-add lr0 2001:0db8:0::/64 2001:0db8:0:f102::1 lp0 > > +check check ovn-nbctl lr-route-add lr0 2001:0db8:1::/64 2001:0db8:0:f103::1 > > + > > +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > +IPv4 Routes > > +Route Table global: > > + 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.1 dst-ip > > + 10.0.1.0/24 <http://10.0.1.0/24> 11.0.1.1 dst-ip lp0 > > + 0.0.0.0/0 <http://0.0.0.0/0> 192.168.0.1 dst-ip > > + > > +Route Table rtb-1: > > + 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.1 dst-ip > > + 10.0.1.0/24 <http://10.0.1.0/24> 11.0.1.1 dst-ip lp0 > > + 0.0.0.0/0 <http://0.0.0.0/0> 192.168.0.1 dst-ip > > + > > +Route Table rtb-2: > > + 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.1 dst-ip > > + 10.0.1.0/24 <http://10.0.1.0/24> 11.0.1.1 dst-ip lp0 > > + 0.0.0.0/0 <http://0.0.0.0/0> 192.168.0.1 dst-ip > > + > > +IPv6 Routes > > +Route Table global: > > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > > + ::/0 2001:db8:0:f101::1 dst-ip > > + > > +Route Table rtb-1: > > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > > + ::/0 2001:db8:0:f101::1 dst-ip > > + > > +Route Table rtb-2: > > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > > + ::/0 2001:db8:0:f101::1 dst-ip > > +]) > > + > > +# delete IPv4 route from rtb-1 > > +check ovn-nbctl --route-table=rtb-1 lr-route-del lr0 10.0.0.0/24 <http://10.0.0.0/24> > > +AT_CHECK([ovn-nbctl --route-table=rtb-1 lr-route-list lr0], [0], [dnl > > +IPv4 Routes > > +Route Table rtb-1: > > + 10.0.1.0/24 <http://10.0.1.0/24> 11.0.1.1 dst-ip lp0 > > + 0.0.0.0/0 <http://0.0.0.0/0> 192.168.0.1 dst-ip > > + > > +IPv6 Routes > > +Route Table rtb-1: > > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > > + ::/0 2001:db8:0:f101::1 dst-ip > > +]) > > + > > +# delete IPv6 route from rtb-2 > > +check ovn-nbctl --route-table=rtb-2 lr-route-del lr0 2001:db8::/64 > > +AT_CHECK([ovn-nbctl --route-table=rtb-2 lr-route-list lr0], [0], [dnl > > +IPv4 Routes > > +Route Table rtb-2: > > + 10.0.0.0/24 <http://10.0.0.0/24> 11.0.0.1 dst-ip > > + 10.0.1.0/24 <http://10.0.1.0/24> 11.0.1.1 dst-ip lp0 > > + 0.0.0.0/0 <http://0.0.0.0/0> 192.168.0.1 dst-ip > > + > > +IPv6 Routes > > +Route Table rtb-2: > > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > > + ::/0 2001:db8:0:f101::1 dst-ip > > +]) > > + > > +check ovn-nbctl lr-route-del lr0 > > + > > +# ECMP route in route table > > +check ovn-nbctl --route-table=rtb1 lr-route-add lr0 0.0.0.0/0 <http://0.0.0.0/0> 192.168.0.1 > > +check ovn-nbctl --ecmp --route-table=rtb1 lr-route-add lr0 0.0.0.0/0 <http://0.0.0.0/0> 192.168.0.2 > > + > > +# Negative route table case: same prefix > > +AT_CHECK([ovn-nbctl --route-table=rtb1 lr-route-add lr0 0.0.0.0/0 <http://0.0.0.0/0> 192.168.0.1], [1], [], [dnl > > +ovn-nbctl: duplicate prefix: 0.0.0.0/0 <http://0.0.0.0/0> (policy: dst-ip). Use option --ecmp to allow this for ECMP routing. > > +]) > > + > > +# Negative route table case: same prefix & nexthop with ecmp > > +AT_CHECK([ovn-nbctl --ecmp --route-table=rtb1 lr-route-add lr0 0.0.0.0/0 <http://0.0.0.0/0> 192.168.0.2], [1], [], [dnl > > +ovn-nbctl: duplicate nexthop for the same ECMP route > > +]) > > + > > +# Add routes to global route table > > +check ovn-nbctl lrp-add lr0 lrp0 00:00:00:00:00:01 1.1.1.1/24 <http://1.1.1.1/24> > > +check ovn-nbctl lrp-set-options lrp0 route_table=rtb1 > > +AT_CHECK([ovn-nbctl get logical-router-port lrp0 options:route_table], [0], [dnl > > +rtb1 > > +]) > > +check `ovn-nbctl show lr0 | grep lrp0 -A3 | grep route_table=rtb1` > > +]) > > > > dnl --------------------------------------------------------------------- > > > > diff --git a/tests/ovn-northd.at <http://ovn-northd.at/> b/tests/ovn-northd.at <http://ovn-northd.at/> > > index 3eebb55b6..e71e65bcc 100644 > > --- a/tests/ovn-northd.at <http://ovn-northd.at/> > > +++ b/tests/ovn-northd.at <http://ovn-northd.at/> > > @@ -5111,7 +5111,7 @@ check ovn-nbctl --wait=sb --ecmp-symmetric-reply lr-route-add lr0 1.0.0.1 192.16 > > > > ovn-sbctl dump-flows lr0 > lr0flows > > AT_CHECK([grep -e "lr_in_ip_routing.*select" lr0flows | sed 's/table=../table=??/' | sort], [0], [dnl > > - table=??(lr_in_ip_routing ), priority=65 , match=(ip4.dst == 1.0.0.1/32 <http://1.0.0.1/32>), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = 1; reg8[[16..31]] = select(1, 2);) > > + table=??(lr_in_ip_routing ), priority=165 , match=(ip4.dst == 1.0.0.1/32 <http://1.0.0.1/32>), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = 1; reg8[[16..31]] = select(1, 2);) > > ]) > > AT_CHECK([grep -e "lr_in_ip_routing_ecmp" lr0flows | sed 's/192\.168\.0\..0/192.168.0.??/ <http://192.168.0.0/??/>' | sed 's/table=../table=??/' | sort], [0], [dnl > > table=??(lr_in_ip_routing_ecmp), priority=100 , match=(reg8[[0..15]] == 1 && reg8[[16..31]] == 1), action=(reg0 = 192.168.0.??; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; next;) > > @@ -5124,7 +5124,7 @@ check ovn-nbctl --wait=sb --ecmp-symmetric-reply lr-route-add lr0 1.0.0.1 192.16 > > > > ovn-sbctl dump-flows lr0 > lr0flows > > AT_CHECK([grep -e "lr_in_ip_routing.*select" lr0flows | sed 's/table=../table=??/' | sort], [0], [dnl > > - table=??(lr_in_ip_routing ), priority=65 , match=(ip4.dst == 1.0.0.1/32 <http://1.0.0.1/32>), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = 1; reg8[[16..31]] = select(1, 2);) > > + table=??(lr_in_ip_routing ), priority=165 , match=(ip4.dst == 1.0.0.1/32 <http://1.0.0.1/32>), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = 1; reg8[[16..31]] = select(1, 2);) > > ]) > > AT_CHECK([grep -e "lr_in_ip_routing_ecmp" lr0flows | sed 's/192\.168\.0\..0/192.168.0.??/ <http://192.168.0.0/??/>' | sed 's/table=../table=??/' | sort], [0], [dnl > > table=??(lr_in_ip_routing_ecmp), priority=100 , match=(reg8[[0..15]] == 1 && reg8[[16..31]] == 1), action=(reg0 = 192.168.0.??; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; next;) > > @@ -5139,14 +5139,14 @@ check ovn-nbctl --wait=sb lr-route-add lr0 1.0.0.0/24 <http://1.0.0.0/24> 192.168.0.10 > > ovn-sbctl dump-flows lr0 > lr0flows > > > > AT_CHECK([grep -e "lr_in_ip_routing.*192.168.0.10" lr0flows | sed 's/table=../table=??/' | sort], [0], [dnl > > - table=??(lr_in_ip_routing ), priority=49 , match=(ip4.dst == 1.0.0.0/24 <http://1.0.0.0/24>), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.10; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == 1.0.0.0/24 <http://1.0.0.0/24>), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.10; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; flags.loopback = 1; next;) > > ]) > > > > check ovn-nbctl --wait=sb lr-route-add lr0 2.0.0.0/24 <http://2.0.0.0/24> lr0-public > > > > ovn-sbctl dump-flows lr0 > lr0flows > > AT_CHECK([grep -e "lr_in_ip_routing.*2.0.0.0" lr0flows | sed 's/table=../table=??/' | sort], [0], [dnl > > - table=??(lr_in_ip_routing ), priority=49 , match=(ip4.dst == 2.0.0.0/24 <http://2.0.0.0/24>), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == 2.0.0.0/24 <http://2.0.0.0/24>), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; flags.loopback = 1; next;) > > ]) > > > > AT_CLEANUP > > @@ -5232,3 +5232,71 @@ AT_CHECK([grep lr_in_gw_redirect lrflows | grep cr-DR | sed 's/table=../table=?? > > > > AT_CLEANUP > > ]) > > + > > + > > +OVN_FOR_EACH_NORTHD([ > > +AT_SETUP([route tables -- flows]) > > +AT_KEYWORDS([route-tables-flows]) > > +ovn_start > > + > > +check ovn-nbctl lr-add lr0 > > +check ovn-nbctl lrp-add lr0 lrp0 00:00:00:00:00:01 192.168.0.1/24 <http://192.168.0.1/24> > > +check ovn-nbctl lrp-add lr0 lrp1 00:00:00:00:01:01 192.168.1.1/24 <http://192.168.1.1/24> > > +check ovn-nbctl lrp-add lr0 lrp2 00:00:00:00:02:01 192.168.2.1/24 <http://192.168.2.1/24> > > +check ovn-nbctl lrp-set-options lrp1 route_table=rtb-1 > > +check ovn-nbctl lrp-set-options lrp2 route_table=rtb-2 > > + > > +check ovn-nbctl lr-route-add lr0 0.0.0.0/0 <http://0.0.0.0/0> 192.168.0.10 > > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 192.168.0.0/24 <http://192.168.0.0/24> 192.168.1.10 > > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 0.0.0.0/0 <http://0.0.0.0/0> 192.168.0.10 > > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 1.1.1.1/32 <http://1.1.1.1/32> 192.168.0.20 > > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 2.2.2.2/32 <http://2.2.2.2/32> 192.168.0.30 > > +check ovn-nbctl --route-table=rtb-2 --ecmp lr-route-add lr0 2.2.2.2/32 <http://2.2.2.2/32> 192.168.0.31 > > +check ovn-nbctl --wait=sb sync > > + > > +ovn-sbctl dump-flows lr0 > lr0flows > > +AT_CAPTURE_FILE([lr0flows]) > > + > > +AT_CHECK([grep -e "lr_in_ip_routing_pre.*match=(1)" lr0flows | sed 's/table=../table=??/'], [0], [dnl > > + table=??(lr_in_ip_routing_pre), priority=0 , match=(1), action=(next;) > > +]) > > + > > +p1_reg=$(grep -oP "lr_in_ip_routing_pre.*lrp1.*action=\(reg7 = \K." lr0flows) > > +p2_reg=$(grep -oP "lr_in_ip_routing_pre.*lrp2.*action=\(reg7 = \K." lr0flows) > > +echo $p1_reg > > +echo $p2_reg > > + > > +# exact register values are not predictable > > +if [[ $p1_reg -eq 2 ] && [ $p2_reg -eq 1 ]]; then > > + echo "swap reg values in dump" > > + sed -i -r s'/^(.*lrp2.*action=\(reg7 = )(1)(.*)/\12\3/g' lr0flows # "reg7 = 1" -> "reg7 = 2" > > + sed -i -r s'/^(.*lrp1.*action=\(reg7 = )(2)(.*)/\11\3/g' lr0flows # "reg7 = 2" -> "reg7 = 1" > > + sed -i -r s'/^(.*match=\(reg7 == )(2)( &&.*lrp1.*)/\11\3/g' lr0flows # "reg7 == 2" -> "reg7 == 1" > > + sed -i -r s'/^(.*match=\(reg7 == )(1)( &&.*lrp0.*)/\12\3/g' lr0flows # "reg7 == 1" -> "reg7 == 2" > > +fi > > + > > +check test "$p1_reg" != "$p2_reg" -a $((p1_reg * p2_reg)) -eq 2 > > + > > +AT_CHECK([grep "lr_in_ip_routing_pre" lr0flows | sed 's/table=../table=??/' | sort], [0], [dnl > > + table=??(lr_in_ip_routing_pre), priority=0 , match=(1), action=(next;) > > + table=??(lr_in_ip_routing_pre), priority=100 , match=(inport == "lrp1"), action=(reg7 = 1; next;) > > + table=??(lr_in_ip_routing_pre), priority=100 , match=(inport == "lrp2"), action=(reg7 = 2; next;) > > +]) > > + > > +grep -e "(lr_in_ip_routing ).*outport" lr0flows > > + > > +AT_CHECK([grep -e "(lr_in_ip_routing ).*outport" lr0flows | sed 's/table=../table=??/' | sort], [0], [dnl > > + table=??(lr_in_ip_routing ), priority=1 , match=(reg7 == 2 && ip4.dst == 0.0.0.0/0 <http://0.0.0.0/0>), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.10; reg1 = 192.168.0.1; eth.src = 00:00:00:00:00:01; outport = "lrp0"; flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=101 , match=(ip4.dst == 0.0.0.0/0 <http://0.0.0.0/0>), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.10; reg1 = 192.168.0.1; eth.src = 00:00:00:00:00:01; outport = "lrp0"; flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == 192.168.0.0/24 <http://192.168.0.0/24>), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 = 192.168.0.1; eth.src = 00:00:00:00:00:01; outport = "lrp0"; flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == 192.168.1.0/24 <http://192.168.1.0/24>), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 = 192.168.1.1; eth.src = 00:00:00:00:01:01; outport = "lrp1"; flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == 192.168.2.0/24 <http://192.168.2.0/24>), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 = 192.168.2.1; eth.src = 00:00:00:00:02:01; outport = "lrp2"; flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=229 , match=(inport == "lrp0" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe00:1; eth.src = 00:00:00:00:00:01; outport = "lrp0"; flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=229 , match=(inport == "lrp1" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe00:101; eth.src = 00:00:00:00:01:01; outport = "lrp1"; flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=229 , match=(inport == "lrp2" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe00:201; eth.src = 00:00:00:00:02:01; outport = "lrp2"; flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=49 , match=(reg7 == 1 && ip4.dst == 192.168.0.0/24 <http://192.168.0.0/24>), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.1.10; reg1 = 192.168.1.1; eth.src = 00:00:00:00:01:01; outport = "lrp1"; flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=65 , match=(reg7 == 2 && ip4.dst == 1.1.1.1/32 <http://1.1.1.1/32>), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.20; reg1 = 192.168.0.1; eth.src = 00:00:00:00:00:01; outport = "lrp0"; flags.loopback = 1; next;) > > +]) > > + > > +AT_CLEANUP > > +]) > > diff --git a/tests/ovn.at <http://ovn.at/> b/tests/ovn.at <http://ovn.at/> > > index 49ece8735..60783a14b 100644 > > --- a/tests/ovn.at <http://ovn.at/> > > +++ b/tests/ovn.at <http://ovn.at/> > > @@ -18145,7 +18145,7 @@ eth_dst=00000000ff01 > > ip_src=$(ip_to_hex 10 0 0 10) > > ip_dst=$(ip_to_hex 172 168 0 101) > > send_icmp_packet 1 1 $eth_src $eth_dst $ip_src $ip_dst c4c9 0000000000000000000000 > > -AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | awk '/table=25, n_packets=1, n_bytes=45/{print $7" "$8}'],[0],[dnl > > +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | awk '/table=26, n_packets=1, n_bytes=45/{print $7" "$8}'],[0],[dnl > > priority=80,ip,reg15=0x3,metadata=0x3,nw_src=10.0.0.10 actions=drop > > ]) > > > > @@ -22577,6 +22577,433 @@ OVN_CLEANUP([hv1]) > > AT_CLEANUP > > ]) > > > > + > > +OVN_FOR_EACH_NORTHD([ > > +AT_SETUP([route tables -- global routes]) > > +ovn_start > > + > > +# Logical network: > > +# ls1 (192.168.1.0/24 <http://192.168.1.0/24>) - lrp-lr1-ls1 - lr1 - lrp-lr1-ls2 - ls2 (192.168.2.0/24 <http://192.168.2.0/24>) > > +# > > +# ls1 has lsp11 (192.168.1.11) and ls2 has lsp21 (192.168.2.21) and lsp22 > > +# (192.168.2.22) > > +# > > +# lrp-lr1-ls1 set options:route_table=rtb-1 > > +# > > +# Static routes on lr1: > > +# 0.0.0.0/0 <http://0.0.0.0/0> nexthop 192.168.2.21 > > +# 1.1.1.1/32 <http://1.1.1.1/32> nexthop 192.168.2.22 route_table=rtb-1 > > +# > > +# Test 1: > > +# lsp11 send packet to 2.2.2.2 > > +# > > +# Expected result: > > +# lsp21 should receive traffic, lsp22 should not receive any traffic > > +# > > +# Test 2: > > +# lsp11 send packet to 1.1.1.1 > > +# > > +# Expected result: > > +# lsp21 should receive traffic, lsp22 should not receive any traffic > > + > > +ovn-nbctl lr-add lr1 > > + > > +for i in 1 2; do > > + ovn-nbctl ls-add ls${i} > > + ovn-nbctl lrp-add lr1 lrp-lr1-ls${i} 00:00:00:01:0${i}:01 192.168.${i}.1/24 > > + ovn-nbctl lsp-add ls${i} lsp-ls${i}-lr1 -- lsp-set-type lsp-ls${i}-lr1 router \ > > + -- lsp-set-options lsp-ls${i}-lr1 router-port=lrp-lr1-ls${i} \ > > + -- lsp-set-addresses lsp-ls${i}-lr1 router > > +done > > + > > +# install static routes > > +ovn-nbctl lr-route-add lr1 0.0.0.0/0 <http://0.0.0.0/0> 192.168.2.21 > > +ovn-nbctl --route-table=rtb-1 lr-route-add lr1 1.1.1.1/32 <http://1.1.1.1/32> 192.168.2.22 > > + > > +# set lrp-lr1-ls1 route table > > +ovn-nbctl lrp-set-options lrp-lr1-ls1 route_table=rtb-1 > > + > > +# Create logical ports > > +ovn-nbctl lsp-add ls1 lsp11 -- \ > > + lsp-set-addresses lsp11 "f0:00:00:00:01:11 192.168.1.11" > > +ovn-nbctl lsp-add ls2 lsp21 -- \ > > + lsp-set-addresses lsp21 "f0:00:00:00:02:21 192.168.2.21" > > +ovn-nbctl lsp-add ls2 lsp22 -- \ > > + lsp-set-addresses lsp22 "f0:00:00:00:02:22 192.168.2.22" > > + > > +net_add n1 > > +sim_add hv1 > > +as hv1 > > +ovs-vsctl add-br br-phys > > +ovn_attach n1 br-phys 192.168.0.1 > > +ovs-vsctl -- add-port br-int hv1-vif1 -- \ > > + set interface hv1-vif1 external-ids:iface-id=lsp11 \ > > + options:tx_pcap=hv1/vif1-tx.pcap \ > > + options:rxq_pcap=hv1/vif1-rx.pcap \ > > + ofport-request=1 > > + > > +ovs-vsctl -- add-port br-int hv1-vif2 -- \ > > + set interface hv1-vif2 external-ids:iface-id=lsp21 \ > > + options:tx_pcap=hv1/vif2-tx.pcap \ > > + options:rxq_pcap=hv1/vif2-rx.pcap \ > > + ofport-request=2 > > + > > +ovs-vsctl -- add-port br-int hv1-vif3 -- \ > > + set interface hv1-vif3 external-ids:iface-id=lsp22 \ > > + options:tx_pcap=hv1/vif3-tx.pcap \ > > + options:rxq_pcap=hv1/vif3-rx.pcap \ > > + ofport-request=3 > > + > > +# wait for earlier changes to take effect > > +check ovn-nbctl --wait=hv sync > > +wait_for_ports_up > > + > > +for i in 1 2; do > > + packet="inport==\"lsp11\" && eth.src==f0:00:00:00:01:11 && eth.dst==00:00:00:01:01:01 && > > + ip4 && ip.ttl==64 && ip4.src==192.168.1.11 && ip4.dst==$i.$i.$i.$i && icmp" > > + AT_CHECK([as hv1 ovs-appctl -t ovn-controller inject-pkt "$packet"]) > > + > > + # Assume all packets go to lsp21. > > + exp_packet="eth.src==00:00:00:01:02:01 && eth.dst==f0:00:00:00:02:21 && > > + ip4 && ip.ttl==63 && ip4.src==192.168.1.11 && ip4.dst==$i.$i.$i.$i && icmp" > > + echo $exp_packet | ovstest test-ovn expr-to-packets >> expected_lsp21 > > +done > > +> expected_lsp22 > > + > > +# lsp21 should recieve 2 packets and lsp22 should recieve no packets > > +OVS_WAIT_UNTIL([ > > + rcv_n1=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in <http://ovs-pcap.in/>" hv1/vif2-tx.pcap > lsp21.packets && cat lsp21.packets | wc -l` > > + rcv_n2=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in <http://ovs-pcap.in/>" hv1/vif3-tx.pcap > lsp22.packets && cat lsp22.packets | wc -l` > > + echo $rcv_n1 $rcv_n2 > > + test $rcv_n1 -eq 2 -a $rcv_n2 -eq 0]) > > + > > +for i in 1 2; do > > + sort expected_lsp2$i > expout > > + AT_CHECK([cat lsp2${i}.packets | sort], [0], [expout]) > > +done > > + > > +OVN_CLEANUP([hv1]) > > +AT_CLEANUP > > +]) > > + > > + > > +OVN_FOR_EACH_NORTHD([ > > +AT_SETUP([route tables -- directly connected routes]) > > +ovn_start > > + > > +# Logical network: > > +# ls1 (192.168.1.0/24 <http://192.168.1.0/24>) - lrp-lr1-ls1 - lr1 - lrp-lr1-ls2 - ls2 (192.168.2.0/24 <http://192.168.2.0/24>) > > +# > > +# ls1 has lsp11 (192.168.1.11) and ls2 has lsp21 (192.168.2.21) > > +# > > +# lrp-lr1-ls1 set options:route_table=rtb-1 > > +# > > +# Static routes on lr1: > > +# 192.168.2.0/25 <http://192.168.2.0/25> nexthop 192.168.1.11 route_table=rtb-1 > > +# > > +# Test 1: > > +# lsp11 send packet to 192.168.2.21 > > +# > > +# Expected result: > > +# lsp21 should receive traffic, lsp11 should not receive any traffic > > + > > +ovn-nbctl lr-add lr1 > > + > > +for i in 1 2; do > > + ovn-nbctl ls-add ls${i} > > + ovn-nbctl lrp-add lr1 lrp-lr1-ls${i} 00:00:00:01:0${i}:01 192.168.${i}.1/24 > > + ovn-nbctl lsp-add ls${i} lsp-ls${i}-lr1 -- lsp-set-type lsp-ls${i}-lr1 router \ > > + -- lsp-set-options lsp-ls${i}-lr1 router-port=lrp-lr1-ls${i} \ > > + -- lsp-set-addresses lsp-ls${i}-lr1 router > > +done > > + > > +# install static route, which overrides directly-connected routes > > +ovn-nbctl --route-table=rtb-1 lr-route-add lr1 192.168.2.0/25 <http://192.168.2.0/25> 192.168.1.11 > > + > > +# set lrp-lr1-ls1 route table > > +ovn-nbctl lrp-set-options lrp-lr1-ls1 route_table=rtb-1 > > + > > +# Create logical ports > > +ovn-nbctl lsp-add ls1 lsp11 -- \ > > + lsp-set-addresses lsp11 "f0:00:00:00:01:11 192.168.1.11" > > +ovn-nbctl lsp-add ls2 lsp21 -- \ > > + lsp-set-addresses lsp21 "f0:00:00:00:02:21 192.168.2.21" > > + > > +net_add n1 > > +sim_add hv1 > > +as hv1 > > +ovs-vsctl add-br br-phys > > +ovn_attach n1 br-phys 192.168.0.1 > > +ovs-vsctl -- add-port br-int hv1-vif1 -- \ > > + set interface hv1-vif1 external-ids:iface-id=lsp11 \ > > + options:tx_pcap=hv1/vif1-tx.pcap \ > > + options:rxq_pcap=hv1/vif1-rx.pcap \ > > + ofport-request=1 > > + > > +ovs-vsctl -- add-port br-int hv1-vif2 -- \ > > + set interface hv1-vif2 external-ids:iface-id=lsp21 \ > > + options:tx_pcap=hv1/vif2-tx.pcap \ > > + options:rxq_pcap=hv1/vif2-rx.pcap \ > > + ofport-request=2 > > + > > +# wait for earlier changes to take effect > > +check ovn-nbctl --wait=hv sync > > +wait_for_ports_up > > + > > +packet="inport==\"lsp11\" && eth.src==f0:00:00:00:01:11 && eth.dst==00:00:00:01:01:01 && > > + ip4 && ip.ttl==64 && ip4.src==192.168.1.11 && ip4.dst==192.168.2.21 && icmp" > > +AT_CHECK([as hv1 ovs-appctl -t ovn-controller inject-pkt "$packet"]) > > + > > +# Assume all packets go to lsp21. > > +exp_packet="eth.src==00:00:00:01:02:01 && eth.dst==f0:00:00:00:02:21 && > > + ip4 && ip.ttl==63 && ip4.src==192.168.1.11 && ip4.dst==192.168.2.21 && icmp" > > +echo $exp_packet | ovstest test-ovn expr-to-packets >> expected_lsp21 > > +> expected_lsp11 > > + > > +# lsp21 should recieve 1 icmp packet and lsp11 should recieve no packets > > +OVS_WAIT_UNTIL([ > > + rcv_n11=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in <http://ovs-pcap.in/>" hv1/vif1-tx.pcap > lsp11.packets && cat lsp11.packets | wc -l` > > + rcv_n21=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in <http://ovs-pcap.in/>" hv1/vif2-tx.pcap > lsp21.packets && cat lsp21.packets | wc -l` > > + echo $rcv_n11 $rcv_n21 > > + test $rcv_n11 -eq 0 -a $rcv_n21 -eq 1]) > > + > > +for i in 11 21; do > > + sort expected_lsp$i > expout > > + AT_CHECK([cat lsp${i}.packets | sort], [0], [expout]) > > +done > > + > > +OVN_CLEANUP([hv1]) > > +AT_CLEANUP > > +]) > > + > > + > > +OVN_FOR_EACH_NORTHD([ > > +AT_SETUP([route tables -- overlapping subnets]) > > +ovn_start > > + > > +# Logical network: > > +# > > +# ls1 (192.168.1.0/24 <http://192.168.1.0/24>) - lrp-lr1-ls1 -\ /- lrp-lr1-ls2 - ls2 (192.168.2.0/24 <http://192.168.2.0/24>) > > +# lr1 > > +# ls3 (192.168.3.0/24 <http://192.168.3.0/24>) - lrp-lr1-ls3 -/ \- lrp-lr1-ls4 - ls4 (192.168.4.0/24 <http://192.168.4.0/24>) > > +# > > +# ls1 has lsp11 (192.168.1.11) > > +# ls2 has lsp21 (192.168.2.21) > > +# ls3 has lsp31 (192.168.3.31) > > +# ls4 has lsp41 (192.168.4.41) > > +# > > +# lrp-lr1-ls1 set options:route_table=rtb-1 > > +# lrp-lr1-ls2 set options:route_table=rtb-2 > > +# > > +# Static routes on lr1: > > +# 10.0.0.0/24 <http://10.0.0.0/24> nexthop 192.168.3.31 route_table=rtb-1 > > +# 10.0.0.0/24 <http://10.0.0.0/24> nexthop 192.168.4.41 route_table=rtb-2 > > +# > > +# Test 1: > > +# lsp11 send packet to 10.0.0.1 > > +# > > +# Expected result: > > +# lsp31 should receive traffic, lsp41 should not receive any traffic > > +# > > +# Test 2: > > +# lsp21 send packet to 10.0.0.1 > > +# > > +# Expected result: > > +# lsp41 should receive traffic, lsp31 should not receive any traffic > > + > > +ovn-nbctl lr-add lr1 > > + > > +# Create logical topology > > +for i in $(seq 1 4); do > > + ovn-nbctl ls-add ls${i} > > + ovn-nbctl lrp-add lr1 lrp-lr1-ls${i} 00:00:00:01:0${i}:01 192.168.${i}.1/24 > > + ovn-nbctl lsp-add ls${i} lsp-ls${i}-lr1 -- lsp-set-type lsp-ls${i}-lr1 router \ > > + -- lsp-set-options lsp-ls${i}-lr1 router-port=lrp-lr1-ls${i} \ > > + -- lsp-set-addresses lsp-ls${i}-lr1 router > > + ovn-nbctl lsp-add ls$i lsp${i}1 -- \ > > + lsp-set-addresses lsp${i}1 "f0:00:00:00:0${i}:1${i} 192.168.${i}.${i}1" > > +done > > + > > +# install static routes > > +ovn-nbctl --route-table=rtb-1 lr-route-add lr1 10.0.0.0/24 <http://10.0.0.0/24> 192.168.3.31 > > +ovn-nbctl --route-table=rtb-2 lr-route-add lr1 10.0.0.0/24 <http://10.0.0.0/24> 192.168.4.41 > > + > > +# set lrp-lr1-ls{1,2} route tables > > +ovn-nbctl lrp-set-options lrp-lr1-ls1 route_table=rtb-1 > > +ovn-nbctl lrp-set-options lrp-lr1-ls2 route_table=rtb-2 > > + > > +net_add n1 > > +sim_add hv1 > > +as hv1 > > +ovs-vsctl add-br br-phys > > +ovn_attach n1 br-phys 192.168.0.1 > > + > > +for i in $(seq 1 4); do > > + ovs-vsctl -- add-port br-int hv1-vif${i} -- \ > > + set interface hv1-vif${i} external-ids:iface-id=lsp${i}1 \ > > + options:tx_pcap=hv1/vif${i}-tx.pcap \ > > + options:rxq_pcap=hv1/vif${i}-rx.pcap \ > > + ofport-request=${i} > > +done > > + > > +# wait for earlier changes to take effect > > +check ovn-nbctl --wait=hv sync > > +wait_for_ports_up > > + > > +# lsp31 should recieve packet coming from lsp11 > > +# lsp41 should recieve packet coming from lsp21 > > +for i in $(seq 1 2); do > > + di=$(( i + 2)) # dst index > > + ri=$(( 5 - i)) # reverse index > > + packet="inport==\"lsp${i}1\" && eth.src==f0:00:00:00:0${i}:1${i} && > > + eth.dst==00:00:00:01:0${i}:01 && ip4 && ip.ttl==64 && > > + ip4.src==192.168.${i}.${i}1 && ip4.dst==10.0.0.1 && icmp" > > + AT_CHECK([as hv1 ovs-appctl -t ovn-controller inject-pkt "$packet"]) > > + > > + # Assume all packets go to lsp${di}1. > > + exp_packet="eth.src==00:00:00:01:0${di}:01 && eth.dst==f0:00:00:00:0${di}:1${di} && > > + ip4 && ip.ttl==63 && ip4.src==192.168.${i}.${i}1 && ip4.dst==10.0.0.1 && icmp" > > + echo $exp_packet | ovstest test-ovn expr-to-packets >> expected_lsp${di}1 > > + > expected_lsp${ri}1 > > + > > + OVS_WAIT_UNTIL([ > > + rcv_n1=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in <http://ovs-pcap.in/>" hv1/vif${di}-tx.pcap > lsp${di}1.packets && cat lsp${di}1.packets | wc -l` > > + rcv_n2=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in <http://ovs-pcap.in/>" hv1/vif${ri}-tx.pcap > lsp${ri}1.packets && cat lsp${ri}1.packets | wc -l` > > + echo $rcv_n1 $rcv_n2 > > + test $rcv_n1 -eq 1 -a $rcv_n2 -eq 0]) > > + > > + for j in "${di}1" "${ri}1"; do > > + sort expected_lsp${j} > expout > > + AT_CHECK([cat lsp${j}.packets | sort], [0], [expout]) > > + done > > + > > + # cleanup tx pcap files > > + for j in "${di}1" "${ri}1"; do > > + ovs-vsctl -- remove interface hv1-vif${di} options tx_pcap > > + > hv1/vif${di}-tx.pcap > > + ovs-vsctl -- set interface hv1-vif${di} external-ids:iface-id=lsp${di}1 \ > > + options:tx_pcap=hv1/vif${di}-tx.pcap > > + done > > +done > > + > > +OVN_CLEANUP([hv1]) > > +AT_CLEANUP > > +]) > > + > > + > > +OVN_FOR_EACH_NORTHD([ > > +AT_SETUP([route tables IPv6 -- overlapping subnets]) > > +ovn_start > > + > > +# Logical network: > > +# > > +# ls1 (2001:db8:1::/64) - lrp-lr1-ls1 -\ /- lrp-lr1-ls2 - ls2 (2001:db8:2::/64) > > +# lr1 > > +# ls3 (2001:db8:3::/64) - lrp-lr1-ls3 -/ \- lrp-lr1-ls4 - ls4 (2001:db8:4::/64) > > +# > > +# ls1 has lsp11 (2001:db8:1::11) > > +# ls2 has lsp21 (2001:db8:2::21) > > +# ls3 has lsp31 (2001:db8:3::31) > > +# ls4 has lsp41 (2001:db8:4::41) > > +# > > +# lrp-lr1-ls1 set options:route_table=rtb-1 > > +# lrp-lr1-ls2 set options:route_table=rtb-2 > > +# > > +# Static routes on lr1: > > +# 2001:db8:2000::/64 nexthop 2001:db8:3::31 route_table=rtb-1 > > +# 2001:db8:2000::/64 nexthop 2001:db8:3::41 route_table=rtb-2 > > +# > > +# Test 1: > > +# lsp11 send packet to 2001:db8:2000::1 > > +# > > +# Expected result: > > +# lsp31 should receive traffic, lsp41 should not receive any traffic > > +# > > +# Test 2: > > +# lsp21 send packet to 2001:db8:2000::1 > > +# > > +# Expected result: > > +# lsp41 should receive traffic, lsp31 should not receive any traffic > > + > > +ovn-nbctl lr-add lr1 > > + > > +# Create logical topology > > +for i in $(seq 1 4); do > > + ovn-nbctl ls-add ls${i} > > + ovn-nbctl lrp-add lr1 lrp-lr1-ls${i} 00:00:00:01:0${i}:01 2001:db8:${i}::1/64 > > + ovn-nbctl lsp-add ls${i} lsp-ls${i}-lr1 -- lsp-set-type lsp-ls${i}-lr1 router \ > > + -- lsp-set-options lsp-ls${i}-lr1 router-port=lrp-lr1-ls${i} \ > > + -- lsp-set-addresses lsp-ls${i}-lr1 router > > + ovn-nbctl lsp-add ls$i lsp${i}1 -- \ > > + lsp-set-addresses lsp${i}1 "f0:00:00:00:0${i}:1${i} 2001:db8:${i}::${i}1" > > +done > > + > > +# install static routes > > +ovn-nbctl --route-table=rtb-1 lr-route-add lr1 2001:db8:2000::/64 2001:db8:3::31 > > +ovn-nbctl --route-table=rtb-2 lr-route-add lr1 2001:db8:2000::/64 2001:db8:4::41 > > + > > +# set lrp-lr1-ls{1,2} route tables > > +ovn-nbctl lrp-set-options lrp-lr1-ls1 route_table=rtb-1 > > +ovn-nbctl lrp-set-options lrp-lr1-ls2 route_table=rtb-2 > > + > > +net_add n1 > > +sim_add hv1 > > +as hv1 > > +ovs-vsctl add-br br-phys > > +ovn_attach n1 br-phys 192.168.0.1 > > + > > +for i in $(seq 1 4); do > > + ovs-vsctl -- add-port br-int hv1-vif${i} -- \ > > + set interface hv1-vif${i} external-ids:iface-id=lsp${i}1 \ > > + options:tx_pcap=hv1/vif${i}-tx.pcap \ > > + options:rxq_pcap=hv1/vif${i}-rx.pcap \ > > + ofport-request=${i} > > +done > > + > > +# wait for earlier changes to take effect > > +AT_CHECK([ovn-nbctl --timeout=3 --wait=hv sync], [0], [ignore]) > > + > > +# lsp31 should recieve packet coming from lsp11 > > +# lsp41 should recieve packet coming from lsp21 > > +for i in $(seq 1 2); do > > + di=$(( i + 2)) # dst index > > + ri=$(( 5 - i)) # reverse index > > + packet="inport==\"lsp${i}1\" && eth.src==f0:00:00:00:0${i}:1${i} && > > + eth.dst==00:00:00:01:0${i}:01 && ip6 && ip.ttl==64 && > > + ip6.src==2001:db8:${i}::${i}1 && ip6.dst==2001:db8:2000::1 && icmp6" > > + AT_CHECK([as hv1 ovs-appctl -t ovn-controller inject-pkt "$packet"]) > > + > > + # Assume all packets go to lsp${di}1. > > + exp_packet="eth.src==00:00:00:01:0${di}:01 && eth.dst==f0:00:00:00:0${di}:1${di} && ip6 && > > + ip.ttl==63 && ip6.src==2001:db8:${i}::${i}1 && ip6.dst==2001:db8:2000::1 && icmp6" > > + echo $exp_packet | ovstest test-ovn expr-to-packets >> expected_lsp${di}1 > > + > expected_lsp${ri}1 > > + > > + OVS_WAIT_UNTIL([ > > + rcv_n1=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in <http://ovs-pcap.in/>" hv1/vif${di}-tx.pcap > lsp${di}1.packets && cat lsp${di}1.packets | wc -l` > > + rcv_n2=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in <http://ovs-pcap.in/>" hv1/vif${ri}-tx.pcap > lsp${ri}1.packets && cat lsp${ri}1.packets | wc -l` > > + echo $rcv_n1 $rcv_n2 > > + test $rcv_n1 -eq 1 -a $rcv_n2 -eq 0]) > > + > > + for j in "${di}1" "${ri}1"; do > > + sort expected_lsp${j} > expout > > + AT_CHECK([cat lsp${j}.packets | sort], [0], [expout]) > > + done > > + > > + # cleanup tx pcap files > > + for j in "${di}1" "${ri}1"; do > > + ovs-vsctl -- remove interface hv1-vif${di} options tx_pcap > > + > hv1/vif${di}-tx.pcap > > + ovs-vsctl -- set interface hv1-vif${di} external-ids:iface-id=lsp${di}1 \ > > + options:tx_pcap=hv1/vif${di}-tx.pcap > > + done > > +done > > + > > +OVN_CLEANUP([hv1]) > > +AT_CLEANUP > > +]) > > + > > + > > OVN_FOR_EACH_NORTHD([ > > AT_SETUP([forwarding group: 3 HVs, 1 LR, 2 LS]) > > AT_KEYWORDS([forwarding-group]) > > @@ -23332,7 +23759,7 @@ ovn-sbctl dump-flows > sbflows > > AT_CAPTURE_FILE([sbflows]) > > AT_CAPTURE_FILE([offlows]) > > OVS_WAIT_UNTIL([ > > - as hv1 ovs-ofctl dump-flows br-int table=20 > offlows > > + as hv1 ovs-ofctl dump-flows br-int table=21 > offlows > > test $(grep -c "load:0x64->NXM_NX_PKT_MARK" offlows) = 1 && \ > > test $(grep -c "load:0x3->NXM_NX_PKT_MARK" offlows) = 1 && \ > > test $(grep -c "load:0x4->NXM_NX_PKT_MARK" offlows) = 1 && \ > > @@ -23425,12 +23852,12 @@ send_ipv4_pkt hv1 hv1-vif1 505400000003 00000000ff01 \ > > $(ip_to_hex 10 0 0 3) $(ip_to_hex 172 168 0 120) > > > > OVS_WAIT_UNTIL([ > > - test 1 -eq $(as hv1 ovs-ofctl dump-flows br-int table=20 | \ > > + test 1 -eq $(as hv1 ovs-ofctl dump-flows br-int table=21 | \ > > grep "load:0x2->NXM_NX_PKT_MARK" -c) > > ]) > > > > AT_CHECK([ > > - test 0 -eq $(as hv1 ovs-ofctl dump-flows br-int table=20 | \ > > + test 0 -eq $(as hv1 ovs-ofctl dump-flows br-int table=21 | \ > > grep "load:0x64->NXM_NX_PKT_MARK" -c) > > ]) > > > > @@ -24133,7 +24560,7 @@ AT_CHECK([ > > grep "priority=100" | \ > > grep -c "ct(commit,zone=NXM_NX_REG11\\[[0..15\\]],.*exec(move:NXM_OF_ETH_SRC\\[[\\]]->NXM_NX_CT_LABEL\\[[32..79\\]],load:0x[[0-9]]->NXM_NX_CT_LABEL\\[[80..95\\]]))" > > > > - grep table=22 hv${hv}flows | \ > > + grep table=23 hv${hv}flows | \ > > grep "priority=200" | \ > > grep -c "actions=move:NXM_NX_CT_LABEL\\[[32..79\\]]->NXM_OF_ETH_DST\\[[\\]]" > > done; :], [0], [dnl > > @@ -24258,7 +24685,7 @@ AT_CHECK([ > > grep "priority=100" | \ > > grep -c "ct(commit,zone=NXM_NX_REG11\\[[0..15\\]],.*exec(move:NXM_OF_ETH_SRC\\[[\\]]->NXM_NX_CT_LABEL\\[[32..79\\]],load:0x[[0-9]]->NXM_NX_CT_LABEL\\[[80..95\\]]))" > > > > - grep table=22 hv${hv}flows | \ > > + grep table=23 hv${hv}flows | \ > > grep "priority=200" | \ > > grep -c "actions=move:NXM_NX_CT_LABEL\\[[32..79\\]]->NXM_OF_ETH_DST\\[[\\]]" > > done; :], [0], [dnl > > @@ -24880,7 +25307,7 @@ AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | grep "actions=controller" | grep > > ]) > > > > # The packet should've been dropped in the lr_in_arp_resolve stage. > > -AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | grep -E "table=22, n_packets=1,.* priority=1,ip,metadata=0x${sw_key},nw_dst=10.0.1.1 actions=drop" -c], [0], [dnl > > +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | grep -E "table=23, n_packets=1,.* priority=1,ip,metadata=0x${sw_key},nw_dst=10.0.1.1 actions=drop" -c], [0], [dnl > > 1 > > ]) > > > > diff --git a/utilities/ovn-nbctl.c b/utilities/ovn-nbctl.c > > index e34bb65f7..0ff10618b 100644 > > --- a/utilities/ovn-nbctl.c > > +++ b/utilities/ovn-nbctl.c > > @@ -329,6 +329,8 @@ Logical router port commands:\n\ > > add logical port PORT on ROUTER\n\ > > lrp-set-gateway-chassis PORT CHASSIS [PRIORITY]\n\ > > set gateway chassis for port PORT\n\ > > + lrp-set-options PORT KEY=VALUE [KEY=VALUE]...\n\ > > + set router port options\n\ > > lrp-del-gateway-chassis PORT CHASSIS\n\ > > delete gateway chassis from port PORT\n\ > > lrp-get-gateway-chassis PORT\n\ > > @@ -351,11 +353,17 @@ Logical router port commands:\n\ > > ('overlay' or 'bridged')\n\ > > \n\ > > Route commands:\n\ > > - [--policy=POLICY] [--ecmp] [--ecmp-symmetric-reply] lr-route-add ROUTER \n\ > > - PREFIX NEXTHOP [PORT]\n\ > > + [--policy=POLICY]\n\ > > + [--ecmp]\n\ > > + [--ecmp-symmetric-reply]\n\ > > + [--route-table=ROUTE_TABLE]\n\ > > + lr-route-add ROUTER PREFIX NEXTHOP [PORT]\n\ > > add a route to ROUTER\n\ > > - [--policy=POLICY] lr-route-del ROUTER [PREFIX [NEXTHOP [PORT]]]\n\ > > + [--policy=POLICY]\n\ > > + [--route-table=ROUTE_TABLE]\n\ > > + lr-route-del ROUTER [PREFIX [NEXTHOP [PORT]]]\n\ > > remove routes from ROUTER\n\ > > + [--route-table=ROUTE_TABLE]\n\ > > lr-route-list ROUTER print routes for ROUTER\n\ > > \n\ > > Policy commands:\n\ > > @@ -743,6 +751,11 @@ print_lr(const struct nbrec_logical_router *lr, struct ds *s) > > ds_put_cstr(s, "]\n"); > > } > > > > + const char *route_table = smap_get(&lrp->options, "route_table"); > > + if (route_table) { > > + ds_put_format(s, " route-table: %s\n", route_table); > > + } > > + > > if (lrp->n_gateway_chassis) { > > const struct nbrec_gateway_chassis **gcs; > > > > @@ -862,6 +875,7 @@ nbctl_pre_show(struct ctl_context *ctx) > > ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_name); > > ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_mac); > > ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_networks); > > + ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_options); > > ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_gateway_chassis); > > > > ovsdb_idl_add_column(ctx->idl, &nbrec_gateway_chassis_col_chassis_name); > > @@ -4000,11 +4014,19 @@ nbctl_lr_policy_list(struct ctl_context *ctx) > > > > static struct nbrec_logical_router_static_route * > > nbctl_lr_get_route(const struct nbrec_logical_router *lr, char *prefix, > > - char *next_hop, bool is_src_route, bool ecmp) > > + char *next_hop, bool is_src_route, bool ecmp, > > + char *route_table) > > { > > for (int i = 0; i < lr->n_static_routes; i++) { > > struct nbrec_logical_router_static_route *route = lr->static_routes[i]; > > > > + /* Strict compare for route_table. > > + * If route_table was not specified, > > + * lookup for routes with empty route_table value. */ > > + if (strcmp(route->route_table, route_table ? route_table : "")) { > > + continue; > > + } > > + > > /* Compare route policy. */ > > char *nb_policy = route->policy; > > bool nb_is_src_route = false; > > @@ -4060,6 +4082,8 @@ nbctl_pre_lr_route_add(struct ctl_context *ctx) > > &nbrec_logical_router_static_route_col_bfd); > > ovsdb_idl_add_column(ctx->idl, > > &nbrec_logical_router_static_route_col_options); > > + ovsdb_idl_add_column(ctx->idl, > > + &nbrec_logical_router_static_route_col_route_table); > > } > > > > static char * OVS_WARN_UNUSED_RESULT > > @@ -4090,6 +4114,7 @@ nbctl_lr_route_add(struct ctl_context *ctx) > > } > > } > > > > + char *route_table = shash_find_data(&ctx->options, "--route-table"); > > bool v6_prefix = false; > > prefix = normalize_ipv4_prefix_str(ctx->argv[2]); > > if (!prefix) { > > @@ -4166,7 +4191,8 @@ nbctl_lr_route_add(struct ctl_context *ctx) > > bool ecmp = shash_find(&ctx->options, "--ecmp") != NULL || > > ecmp_symmetric_reply; > > struct nbrec_logical_router_static_route *route = > > - nbctl_lr_get_route(lr, prefix, next_hop, is_src_route, ecmp); > > + nbctl_lr_get_route(lr, prefix, next_hop, is_src_route, ecmp, > > + route_table); > > > > /* Validations for nexthop = "discard" */ > > if (is_discard_route) { > > @@ -4230,7 +4256,8 @@ nbctl_lr_route_add(struct ctl_context *ctx) > > } > > > > struct nbrec_logical_router_static_route *discard_route = > > - nbctl_lr_get_route(lr, prefix, "discard", is_src_route, true); > > + nbctl_lr_get_route(lr, prefix, "discard", is_src_route, true, > > + route_table); > > if (discard_route) { > > ctl_error(ctx, "discard nexthop for the same ECMP route exists."); > > goto cleanup; > > @@ -4246,6 +4273,9 @@ nbctl_lr_route_add(struct ctl_context *ctx) > > if (policy) { > > nbrec_logical_router_static_route_set_policy(route, policy); > > } > > + if (route_table) { > > + nbrec_logical_router_static_route_set_route_table(route, route_table); > > + } > > > > if (ecmp_symmetric_reply) { > > const struct smap options = SMAP_CONST1(&options, > > @@ -4289,6 +4319,8 @@ nbctl_pre_lr_route_del(struct ctl_context *ctx) > > &nbrec_logical_router_static_route_col_nexthop); > > ovsdb_idl_add_column(ctx->idl, > > &nbrec_logical_router_static_route_col_output_port); > > + ovsdb_idl_add_column(ctx->idl, > > + &nbrec_logical_router_static_route_col_route_table); > > > > } > > > > @@ -4302,6 +4334,7 @@ nbctl_lr_route_del(struct ctl_context *ctx) > > return; > > } > > > > + const char *route_table = shash_find_data(&ctx->options, "--route-table"); > > const char *policy = shash_find_data(&ctx->options, "--policy"); > > bool is_src_route = false; > > if (policy) { > > @@ -4392,6 +4425,14 @@ nbctl_lr_route_del(struct ctl_context *ctx) > > } > > } > > > > + /* Strict compare for route_table. > > + * If route_table was not specified, > > + * lookup for routes with empty route_table value. */ > > + if (strcmp(lr->static_routes[i]->route_table, > > + route_table ? route_table : "")) { > > + continue; > > + } > > + > > /* Compare output_port, if specified. */ > > if (output_port) { > > char *rt_output_port = lr->static_routes[i]->output_port; > > @@ -5115,6 +5156,41 @@ nbctl_pre_lrp_del_gateway_chassis(struct ctl_context *ctx) > > ovsdb_idl_add_column(ctx->idl, &nbrec_gateway_chassis_col_chassis_name); > > } > > > > +static void > > +nbctl_pre_lrp_options(struct ctl_context *ctx) > > +{ > > + ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_name); > > + ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_options); > > +} > > + > > +static void > > +nbctl_lrp_set_options(struct ctl_context *ctx) > > +{ > > + const char *id = ctx->argv[1]; > > + const struct nbrec_logical_router_port *lrp = NULL; > > + size_t i; > > + struct smap options = SMAP_INITIALIZER(&options); > > + > > + char *error = lrp_by_name_or_uuid(ctx, id, true, &lrp); > > + if (error) { > > + ctx->error = error; > > + return; > > + } > > + for (i = 2; i < ctx->argc; i++) { > > + char *key, *value; > > + value = xstrdup(ctx->argv[i]); > > + key = strsep(&value, "="); > > + if (value) { > > + smap_add(&options, key, value); > > + } > > + free(key); > > + } > > + > > + nbrec_logical_router_port_set_options(lrp, &options); > > + > > + smap_destroy(&options); > > +} > > + > > /* Removes logical router port 'lrp->gateway_chassis[idx]'. */ > > static void > > remove_gc(const struct nbrec_logical_router_port *lrp, size_t idx) > > @@ -5891,6 +5967,7 @@ route_cmp_details(const struct nbrec_logical_router_static_route *r1, > > } > > return r1->output_port ? 1 : -1; > > } > > + > > struct ipv4_route { > > int priority; > > ovs_be32 addr; > > @@ -5900,6 +5977,11 @@ struct ipv4_route { > > static int > > __ipv4_route_cmp(const struct ipv4_route *r1, const struct ipv4_route *r2) > > { > > + int rtb_cmp = strcmp(r1->route->route_table, > > + r2->route->route_table); > > + if (rtb_cmp) { > > + return rtb_cmp; > > + } > > if (r1->priority != r2->priority) { > > return r1->priority > r2->priority ? -1 : 1; > > } > > @@ -5931,6 +6013,11 @@ struct ipv6_route { > > static int > > __ipv6_route_cmp(const struct ipv6_route *r1, const struct ipv6_route *r2) > > { > > + int rtb_cmp = strcmp(r1->route->route_table, > > + r2->route->route_table); > > + if (rtb_cmp) { > > + return rtb_cmp; > > + } > > if (r1->priority != r2->priority) { > > return r1->priority > r2->priority ? -1 : 1; > > } > > @@ -6018,6 +6105,8 @@ nbctl_pre_lr_route_list(struct ctl_context *ctx) > > &nbrec_logical_router_static_route_col_options); > > ovsdb_idl_add_column(ctx->idl, > > &nbrec_logical_router_static_route_col_bfd); > > + ovsdb_idl_add_column(ctx->idl, > > + &nbrec_logical_router_static_route_col_route_table); > > } > > > > static void > > @@ -6035,12 +6124,17 @@ nbctl_lr_route_list(struct ctl_context *ctx) > > return; > > } > > > > + char *route_table = shash_find_data(&ctx->options, "--route-table"); > > + > > ipv4_routes = xmalloc(sizeof *ipv4_routes * lr->n_static_routes); > > ipv6_routes = xmalloc(sizeof *ipv6_routes * lr->n_static_routes); > > > > for (int i = 0; i < lr->n_static_routes; i++) { > > const struct nbrec_logical_router_static_route *route > > = lr->static_routes[i]; > > + if (route_table && strcmp(route->route_table, route_table)) { > > + continue; > > + } > > unsigned int plen; > > ovs_be32 ipv4; > > const char *policy = route->policy ? route->policy : "dst-ip"; > > @@ -6081,6 +6175,7 @@ nbctl_lr_route_list(struct ctl_context *ctx) > > if (n_ipv4_routes) { > > ds_put_cstr(&ctx->output, "IPv4 Routes\n"); > > } > > + const struct nbrec_logical_router_static_route *route; > > for (int i = 0; i < n_ipv4_routes; i++) { > > bool ecmp = false; > > if (i < n_ipv4_routes - 1 && > > @@ -6091,6 +6186,15 @@ nbctl_lr_route_list(struct ctl_context *ctx) > > &ipv4_routes[i - 1])) { > > ecmp = true; > > } > > + > > + route = ipv4_routes[i].route; > > + if (!i || (i > 0 && strcmp(route->route_table, > > + ipv4_routes[i - 1].route->route_table))) { > > + ds_put_format(&ctx->output, "%sRoute Table %s:\n", i ? "\n" : "", > > + strlen(route->route_table) ? route->route_table > > + : "global"); > > + } > > + > > print_route(ipv4_routes[i].route, &ctx->output, ecmp); > > } > > > > @@ -6108,6 +6212,15 @@ nbctl_lr_route_list(struct ctl_context *ctx) > > &ipv6_routes[i - 1])) { > > ecmp = true; > > } > > + > > + route = ipv6_routes[i].route; > > + if (!i || (i > 0 && strcmp(route->route_table, > > + ipv6_routes[i - 1].route->route_table))) { > > + ds_put_format(&ctx->output, "%sRoute Table %s:\n", i ? "\n" : "", > > + strlen(route->route_table) ? route->route_table > > + : "global"); > > + } > > + > > print_route(ipv6_routes[i].route, &ctx->output, ecmp); > > } > > > > @@ -6926,6 +7039,8 @@ static const struct ctl_command_syntax nbctl_commands[] = { > > "PORT CHASSIS [PRIORITY]", > > nbctl_pre_lrp_set_gateway_chassis, nbctl_lrp_set_gateway_chassis, > > NULL, "--may-exist", RW }, > > + { "lrp-set-options", 1, INT_MAX, "PORT KEY=VALUE [KEY=VALUE]...", > > + nbctl_pre_lrp_options, nbctl_lrp_set_options, NULL, "", RW }, > > { "lrp-del-gateway-chassis", 2, 2, "PORT CHASSIS", > > nbctl_pre_lrp_del_gateway_chassis, nbctl_lrp_del_gateway_chassis, > > NULL, "", RW }, > > @@ -6949,12 +7064,13 @@ static const struct ctl_command_syntax nbctl_commands[] = { > > /* logical router route commands. */ > > { "lr-route-add", 3, 4, "ROUTER PREFIX NEXTHOP [PORT]", > > nbctl_pre_lr_route_add, nbctl_lr_route_add, NULL, > > - "--may-exist,--ecmp,--ecmp-symmetric-reply,--policy=,--bfd?", RW }, > > + "--may-exist,--ecmp,--ecmp-symmetric-reply,--policy=,--route-table=,--bfd?", > > + RW }, > > { "lr-route-del", 1, 4, "ROUTER [PREFIX [NEXTHOP [PORT]]]", > > nbctl_pre_lr_route_del, nbctl_lr_route_del, NULL, > > - "--if-exists,--policy=", RW }, > > + "--if-exists,--policy=,--route-table=", RW }, > > { "lr-route-list", 1, 1, "ROUTER", nbctl_pre_lr_route_list, > > - nbctl_lr_route_list, NULL, "", RO }, > > + nbctl_lr_route_list, NULL, "--route-table=", RO }, > > > > /* Policy commands */ > > { "lr-policy-add", 4, INT_MAX, > > -- > > 2.30.0 > > > > _______________________________________________ > > dev mailing list > > dev@openvswitch.org <mailto:dev@openvswitch.org> > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev <https://mail.openvswitch.org/mailman/listinfo/ovs-dev>
On Thu, Oct 14, 2021 at 12:58 AM Vladislav Odintsov <odivlad@gmail.com> wrote: > > Hi Han, > > Thanks for the review. > > Regards, > Vladislav Odintsov > > On 14 Oct 2021, at 08:13, Han Zhou <hzhou@ovn.org> wrote: > > > > On Tue, Oct 5, 2021 at 1:26 PM Vladislav Odintsov <odivlad@gmail.com> wrote: > > > > This patch extends Logical Router's routing functionality. > > Now user may create multiple routing tables within a Logical Router > > and assign them to Logical Router Ports. > > > > Traffic coming from Logical Router Port with assigned route_table > > is checked against global routes if any (Logical_Router_Static_Routes > > whith empty route_table field), next against directly connected routes > > This is not accurate. The "directly connected routes" is NOT after the global routes. Their priority only depends on the prefix length. > > > and then Logical_Router_Static_Routes with same route_table value as > > in Logical_Router_Port options:route_table field. > > > > A new Logical Router ingress table #10 is added - IN_IP_ROUTING_PRE. > > In this table packets which come from LRPs with configured > > options:route_table field are checked against inport and in OVS > > register 7 unique non-zero value identifying route table is written. > > > > Then in 11th table IN_IP_ROUTING routes which have non-empty > > `route_table` field are added with additional match on reg7 value > > associated with appropriate route_table. > > > > Hi Vladislav, > > First of all, sorry for the delayed review, and thanks for implementing this new feature. > > I have some questions regarding the feature itself. I remember that we had some discussion earlier for this feature, but it seems I misunderstood the feature you are implementing here. I thought by multiple routing tables you were trying to support something like VRF in physical routers, but this seems to be something different. According to your implementation, instead of assigning LRPs to different routing tables, a LRP can actually participate to both the global routing table and the table with a specific ID. For ingress, global routes are prefered over other routes; for egress (i.e. forwarding to the next hop), it doesn't even enforce any table-id check, so packet routed by a entry of table-X can go out of a LRP with table-id Y. Is my understanding correct about the desired behavior of this feature? > > > Yes, your understanding is correct. > This is not VRF. At first glance VRF can be done just by using LR-per-VRF without any code modifications (maybe there are corner cases, but it’s not something I’m currently working on). I agree VRF can be achieved by just separate LRs. I am trying to understand why multiple LRs wouldn't satisfy your use case here. > LRP can be optionally assigned to specific routing table name. This means that for LR ingress pipeline packets coming from this specific LRP would be checked against routes with same route_table value within appropriate LR. This is some kind of PBR, analog of "ip rule add iif <interface name> lookup <id>". > There is one specific use-case, which requires special handling: directly-connected routes (subnet CIDRs from connected to this LR LRPs). These routes can’t be added manually by user, though routing between LRPs belonging to different routing tables is still needed. So, these routes should be added to global routing table to override routing for LRPs with configured routing tables. If for some reason user wants to prohibit IP connectivity to any LRP (honestly, I can’t imagine, why), it can be done by utilising OVN PBR with drop action or a route with "discard" nexthop. But I’d think in this case whether this LRP is needed in this LR. > Also, if user wants to create routes, which apply for all LRPs from all routing tables, it can be done by adding new entries to global routing table. > And the third reason to have global routing table - a fully backward-compatible solution. All users, who don’t need multiple routing tables support just continue using static routing in the same manner without any changes. > > > If this is correct, it doesn't seem to be a common/standard requirement (or please educate me if I am wrong). Could you explain a little more about the actual use cases: what kind of real world problems need to be solved by this feature or how are you going to use this. For example, why would a port need to participate in both routing tables? It looks like what you really need is policy routing instead of multiple isolated routing tables. I understand that you already use policy routing to implement ACLs, so it is not convenient to combine other policies (e.g. inport based routing) into the policy routing stage. If that's the case, would it be more generic to support multiple policy routing stages? My concern to the current approach is that it is implemented for a very special use case. It makes the code more complex but when there is a slightly different requirement in the future it becomes insufficient. I am thinking that policy routing seems more flexible and has more potential to be made more generic. Maybe I will have a better understanding when I hear more detailed use cases and considerations from you. > > > I can't agree here in it’s uncommon requirement. > This implementation was inspired by AWS Route Tables feature [1]: within a VPC (LR in terms of OVN) user may create multiple routing tables and assign them to different subnets (LRPs) in multiple availability zones. Auto-generated directly-connected routes from LRPs CIDRs are working in the same manner as they do in AWS - apply to all subnets regardless of their association to routing table. GCP has similar behaviour: [2], Azure, I guess, too. > Our public cloud (CROC Cloud Platform) supports AWS behaviour [3], so I primarily was oriented on it. Internally we already use this feature and currently it fits our use-case, but again I can't say it is specific. If it is for AWS/GCP alike routing features, then from what I understand what's implemented in this patch is a little different. To implement a VPC model in OVN, I think it is ok to have multiple LRs instead of only one LR for each VPC. So naturally I would use a LR to map to AWS's routing table. In AWS's document (the link you provided), it says: "A subnet can only be associated with one route table at a time" So basically in AWS a subnet is either attached to the default/main route table or a custom table, but not both, right? However, in your use case, a LRP (maps to a subnet) attaches to both "main" and a custom table, which seems not common to me. Or did I miss something? > > In my opinion having this feature to be implemented using PBR is less convenient and native for users, who are familiar with behaviour for mentioned above public cloud platforms, because configuring routes should be done in routes section. And adding route table property seems native in this route, not in PBR. Moreover, I _think_ using Logical_Router_Static_Route to extend this feature for OVN-Interconnection becomes quite easy comparing to PBR (though, I didn’t try the latter). I agree if it is just AWS -like requirement, PBR is less convenient. I am trying to understand if it can be achieved with separate LRs. If not, what's special about the requirement, and is the current approach providing a solution common enough so that more use cases can also benefit from? Could you clarify a little more? Thanks again. Han > > > I haven't finished reviewing the code yet, but I have one comment regarding adding 100 to the priority of the global routes. For IPv6, the priority range from 0 to 120x2=240, so adding 100 is not enough. It would create overlapping priority ranges, and some table-id specific route entries may override the global routes. > > > Thanks, I’ll dig into this when you finish review. > > > Let me know if I answered your questions or if you have new ones. > Again many thanks for your time and digging into this patch series. > > 1: https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Route_Tables.html > 2: https://cloud.google.com/vpc/docs/routes#subnet-routes > 3: https://docs.cloud.croc.ru/en/services/networks/routetables.html > > Thanks, > Han > > > Signed-off-by: Vladislav Odintsov <odivlad@gmail.com> > > Acked-by: Numan Siddique <numans@ovn.org> > > --- > > northd/northd.c | 159 ++++++++++++--- > > northd/ovn-northd.8.xml | 63 ++++-- > > ovn-nb.ovsschema | 5 +- > > ovn-nb.xml | 30 +++ > > tests/ovn-ic.at | 4 + > > tests/ovn-nbctl.at | 196 +++++++++++++++++- > > tests/ovn-northd.at | 76 ++++++- > > tests/ovn.at | 441 +++++++++++++++++++++++++++++++++++++++- > > utilities/ovn-nbctl.c | 134 +++++++++++- > > 9 files changed, 1041 insertions(+), 67 deletions(-) > > > > diff --git a/northd/northd.c b/northd/northd.c > > index 092eca829..6a020cb2e 100644 > > --- a/northd/northd.c > > +++ b/northd/northd.c > > @@ -148,15 +148,16 @@ enum ovn_stage { > > PIPELINE_STAGE(ROUTER, IN, ECMP_STATEFUL, 7, "lr_in_ecmp_stateful") \ > > PIPELINE_STAGE(ROUTER, IN, ND_RA_OPTIONS, 8, "lr_in_nd_ra_options") \ > > PIPELINE_STAGE(ROUTER, IN, ND_RA_RESPONSE, 9, "lr_in_nd_ra_response") \ > > - PIPELINE_STAGE(ROUTER, IN, IP_ROUTING, 10, "lr_in_ip_routing") \ > > - PIPELINE_STAGE(ROUTER, IN, IP_ROUTING_ECMP, 11, "lr_in_ip_routing_ecmp") \ > > - PIPELINE_STAGE(ROUTER, IN, POLICY, 12, "lr_in_policy") \ > > - PIPELINE_STAGE(ROUTER, IN, POLICY_ECMP, 13, "lr_in_policy_ecmp") \ > > - PIPELINE_STAGE(ROUTER, IN, ARP_RESOLVE, 14, "lr_in_arp_resolve") \ > > - PIPELINE_STAGE(ROUTER, IN, CHK_PKT_LEN , 15, "lr_in_chk_pkt_len") \ > > - PIPELINE_STAGE(ROUTER, IN, LARGER_PKTS, 16, "lr_in_larger_pkts") \ > > - PIPELINE_STAGE(ROUTER, IN, GW_REDIRECT, 17, "lr_in_gw_redirect") \ > > - PIPELINE_STAGE(ROUTER, IN, ARP_REQUEST, 18, "lr_in_arp_request") \ > > + PIPELINE_STAGE(ROUTER, IN, IP_ROUTING_PRE, 10, "lr_in_ip_routing_pre") \ > > + PIPELINE_STAGE(ROUTER, IN, IP_ROUTING, 11, "lr_in_ip_routing") \ > > + PIPELINE_STAGE(ROUTER, IN, IP_ROUTING_ECMP, 12, "lr_in_ip_routing_ecmp") \ > > + PIPELINE_STAGE(ROUTER, IN, POLICY, 13, "lr_in_policy") \ > > + PIPELINE_STAGE(ROUTER, IN, POLICY_ECMP, 14, "lr_in_policy_ecmp") \ > > + PIPELINE_STAGE(ROUTER, IN, ARP_RESOLVE, 15, "lr_in_arp_resolve") \ > > + PIPELINE_STAGE(ROUTER, IN, CHK_PKT_LEN, 16, "lr_in_chk_pkt_len") \ > > + PIPELINE_STAGE(ROUTER, IN, LARGER_PKTS, 17, "lr_in_larger_pkts") \ > > + PIPELINE_STAGE(ROUTER, IN, GW_REDIRECT, 18, "lr_in_gw_redirect") \ > > + PIPELINE_STAGE(ROUTER, IN, ARP_REQUEST, 19, "lr_in_arp_request") \ > > \ > > /* Logical router egress stages. */ \ > > PIPELINE_STAGE(ROUTER, OUT, UNDNAT, 0, "lr_out_undnat") \ > > @@ -225,6 +226,7 @@ enum ovn_stage { > > #define REG_NEXT_HOP_IPV6 "xxreg0" > > #define REG_SRC_IPV4 "reg1" > > #define REG_SRC_IPV6 "xxreg1" > > +#define REG_ROUTE_TABLE_ID "reg7" > > > > #define REG_ORIG_TP_DPORT_ROUTER "reg9[16..31]" > > > > @@ -287,8 +289,9 @@ enum ovn_stage { > > * | R6 | UNUSED | X | | G | IN_IP_ROUTING)| > > * | | | R | | 1 | | > > * +-----+--------------------------+ E | UNUSED | | | > > - * | R7 | UNUSED | G | | | | > > - * | | | 3 | | | | > > + * | R7 | ROUTE_TABLE_ID | G | | | | > > + * | | (>= IN_IP_ROUTING_PRE && | 3 | | | | > > + * | | <= IN_IP_ROUTING) | | | | | > > * +-----+--------------------------+---+-----------------+---+---------------+ > > * | R8 | ECMP_GROUP_ID | | | > > * | | ECMP_MEMBER_ID | X | | > > @@ -8511,11 +8514,72 @@ cleanup: > > ds_destroy(&actions); > > } > > > > +static uint32_t > > +route_table_add(struct simap *route_tables, const char *route_table_name) > > +{ > > + /* route table ids start from 1 */ > > + uint32_t rtb_id = simap_count(route_tables) + 1; > > + > > + if (rtb_id == UINT16_MAX) { > > + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1); > > + VLOG_WARN_RL(&rl, "too many route tables for Logical Router."); > > + return 0; > > + } > > + > > + if (!simap_put(route_tables, route_table_name, rtb_id)) { > > + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1); > > + VLOG_WARN_RL(&rl, "Route table id unexpectedly appeared"); > > + } > > + > > + return rtb_id; > > +} > > + > > +static uint32_t > > +get_route_table_id(struct simap *route_tables, const char *route_table_name) > > +{ > > + if (!route_table_name || !strlen(route_table_name)) { > > + return 0; > > + } > > + > > + uint32_t rtb_id = simap_get(route_tables, route_table_name); > > + if (!rtb_id) { > > + rtb_id = route_table_add(route_tables, route_table_name); > > + } > > + > > + return rtb_id; > > +} > > + > > +static void > > +build_route_table_lflow(struct ovn_datapath *od, struct hmap *lflows, > > + struct nbrec_logical_router_port *lrp, > > + struct simap *route_tables) > > +{ > > + struct ds match = DS_EMPTY_INITIALIZER; > > + struct ds actions = DS_EMPTY_INITIALIZER; > > + > > + const char *route_table_name = smap_get(&lrp->options, "route_table"); > > + uint32_t rtb_id = get_route_table_id(route_tables, route_table_name); > > + if (!rtb_id) { > > + return; > > + } > > + > > + ds_put_format(&match, "inport == \"%s\"", lrp->name); > > + ds_put_format(&actions, "%s = %d; next;", > > + REG_ROUTE_TABLE_ID, rtb_id); > > + > > + ovn_lflow_add(lflows, od, S_ROUTER_IN_IP_ROUTING_PRE, 100, > > + ds_cstr(&match), ds_cstr(&actions)); > > + > > + ds_destroy(&match); > > + ds_destroy(&actions); > > +} > > + > > struct parsed_route { > > struct ovs_list list_node; > > struct in6_addr prefix; > > unsigned int plen; > > bool is_src_route; > > + uint32_t route_table_id; > > uint32_t hash; > > const struct nbrec_logical_router_static_route *route; > > bool ecmp_symmetric_reply; > > @@ -8540,7 +8604,7 @@ find_static_route_outport(struct ovn_datapath *od, struct hmap *ports, > > * Otherwise return NULL. */ > > static struct parsed_route * > > parsed_routes_add(struct ovn_datapath *od, struct hmap *ports, > > - struct ovs_list *routes, > > + struct ovs_list *routes, struct simap *route_tables, > > const struct nbrec_logical_router_static_route *route, > > struct hmap *bfd_connections) > > { > > @@ -8622,6 +8686,7 @@ parsed_routes_add(struct ovn_datapath *od, struct hmap *ports, > > struct parsed_route *pr = xzalloc(sizeof *pr); > > pr->prefix = prefix; > > pr->plen = plen; > > + pr->route_table_id = get_route_table_id(route_tables, route->route_table); > > pr->is_src_route = (route->policy && !strcmp(route->policy, > > "src-ip")); > > pr->hash = route_hash(pr); > > @@ -8655,6 +8720,7 @@ struct ecmp_groups_node { > > struct in6_addr prefix; > > unsigned int plen; > > bool is_src_route; > > + uint32_t route_table_id; > > uint16_t route_count; > > struct ovs_list route_list; /* Contains ecmp_route_list_node */ > > }; > > @@ -8663,7 +8729,7 @@ static void > > ecmp_groups_add_route(struct ecmp_groups_node *group, > > const struct parsed_route *route) > > { > > - if (group->route_count == UINT16_MAX) { > > + if (group->route_count == UINT16_MAX) { > > static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1); > > VLOG_WARN_RL(&rl, "too many routes in a single ecmp group."); > > return; > > @@ -8692,6 +8758,7 @@ ecmp_groups_add(struct hmap *ecmp_groups, > > eg->prefix = route->prefix; > > eg->plen = route->plen; > > eg->is_src_route = route->is_src_route; > > + eg->route_table_id = route->route_table_id; > > ovs_list_init(&eg->route_list); > > ecmp_groups_add_route(eg, route); > > > > @@ -8705,7 +8772,8 @@ ecmp_groups_find(struct hmap *ecmp_groups, struct parsed_route *route) > > HMAP_FOR_EACH_WITH_HASH (eg, hmap_node, route->hash, ecmp_groups) { > > if (ipv6_addr_equals(&eg->prefix, &route->prefix) && > > eg->plen == route->plen && > > - eg->is_src_route == route->is_src_route) { > > + eg->is_src_route == route->is_src_route && > > + eg->route_table_id == route->route_table_id) { > > return eg; > > } > > } > > @@ -8752,7 +8820,8 @@ unique_routes_remove(struct hmap *unique_routes, > > HMAP_FOR_EACH_WITH_HASH (ur, hmap_node, route->hash, unique_routes) { > > if (ipv6_addr_equals(&route->prefix, &ur->route->prefix) && > > route->plen == ur->route->plen && > > - route->is_src_route == ur->route->is_src_route) { > > + route->is_src_route == ur->route->is_src_route && > > + route->route_table_id == ur->route->route_table_id) { > > hmap_remove(unique_routes, &ur->hmap_node); > > const struct parsed_route *existed_route = ur->route; > > free(ur); > > @@ -8790,9 +8859,9 @@ build_route_prefix_s(const struct in6_addr *prefix, unsigned int plen) > > } > > > > static void > > -build_route_match(const struct ovn_port *op_inport, const char *network_s, > > - int plen, bool is_src_route, bool is_ipv4, struct ds *match, > > - uint16_t *priority) > > +build_route_match(const struct ovn_port *op_inport, uint32_t rtb_id, > > + const char *network_s, int plen, bool is_src_route, > > + bool is_ipv4, struct ds *match, uint16_t *priority) > > { > > const char *dir; > > /* The priority here is calculated to implement longest-prefix-match > > @@ -8808,6 +8877,15 @@ build_route_match(const struct ovn_port *op_inport, const char *network_s, > > if (op_inport) { > > ds_put_format(match, "inport == %s && ", op_inport->json_key); > > } > > + if (rtb_id) { > > + ds_put_format(match, "%s == %d && ", REG_ROUTE_TABLE_ID, rtb_id); > > + } else { > > + /* Route-table assigned LRPs' routes should have lower priority > > + * in order not to affect directly-connected global routes. > > + * So, enlarge non-route-table routes priority by 100. > > + */ > > + *priority += 100; > > + } > > ds_put_format(match, "ip%s.%s == %s/%d", is_ipv4 ? "4" : "6", dir, > > network_s, plen); > > } > > @@ -8946,7 +9024,7 @@ add_ecmp_symmetric_reply_flows(struct hmap *lflows, > > out_port->lrp_networks.ea_s, > > IN6_IS_ADDR_V4MAPPED(&route->prefix) ? "" : "xx", > > port_ip, out_port->json_key); > > - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_IP_ROUTING, 300, > > + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_IP_ROUTING, 400, > > ds_cstr(&match), ds_cstr(&actions), > > &st_route->header_); > > > > @@ -8976,8 +9054,8 @@ build_ecmp_route_flow(struct hmap *lflows, struct ovn_datapath *od, > > struct ds route_match = DS_EMPTY_INITIALIZER; > > > > char *prefix_s = build_route_prefix_s(&eg->prefix, eg->plen); > > - build_route_match(NULL, prefix_s, eg->plen, eg->is_src_route, is_ipv4, > > - &route_match, &priority); > > + build_route_match(NULL, eg->route_table_id, prefix_s, eg->plen, > > + eg->is_src_route, is_ipv4, &route_match, &priority); > > free(prefix_s); > > > > struct ds actions = DS_EMPTY_INITIALIZER; > > @@ -9052,8 +9130,8 @@ static void > > add_route(struct hmap *lflows, struct ovn_datapath *od, > > const struct ovn_port *op, const char *lrp_addr_s, > > const char *network_s, int plen, const char *gateway, > > - bool is_src_route, const struct ovsdb_idl_row *stage_hint, > > - bool is_discard_route) > > + bool is_src_route, const uint32_t rtb_id, > > + const struct ovsdb_idl_row *stage_hint, bool is_discard_route) > > { > > bool is_ipv4 = strchr(network_s, '.') ? true : false; > > struct ds match = DS_EMPTY_INITIALIZER; > > @@ -9068,8 +9146,8 @@ add_route(struct hmap *lflows, struct ovn_datapath *od, > > op_inport = op; > > } > > } > > - build_route_match(op_inport, network_s, plen, is_src_route, is_ipv4, > > - &match, &priority); > > + build_route_match(op_inport, rtb_id, network_s, plen, is_src_route, > > + is_ipv4, &match, &priority); > > > > struct ds common_actions = DS_EMPTY_INITIALIZER; > > struct ds actions = DS_EMPTY_INITIALIZER; > > @@ -9132,7 +9210,8 @@ build_static_route_flow(struct hmap *lflows, struct ovn_datapath *od, > > char *prefix_s = build_route_prefix_s(&route_->prefix, route_->plen); > > add_route(lflows, route_->is_discard_route ? od : out_port->od, out_port, > > lrp_addr_s, prefix_s, route_->plen, route->nexthop, > > - route_->is_src_route, &route->header_, route_->is_discard_route); > > + route_->is_src_route, route_->route_table_id, &route->header_, > > + route_->is_discard_route); > > > > free(prefix_s); > > } > > @@ -10584,6 +10663,17 @@ build_ND_RA_flows_for_lrouter(struct ovn_datapath *od, struct hmap *lflows) > > } > > } > > > > +/* Logical router ingress table IP_ROUTING_PRE: > > + * by default goto next. (priority 0). */ > > +static void > > +build_ip_routing_pre_flows_for_lrouter(struct ovn_datapath *od, > > + struct hmap *lflows) > > +{ > > + if (od->nbr) { > > + ovn_lflow_add(lflows, od, S_ROUTER_IN_IP_ROUTING_PRE, 0, "1", "next;"); > > + } > > +} > > + > > /* Logical router ingress table IP_ROUTING : IP Routing. > > * > > * A packet that arrives at this table is an IP packet that should be > > @@ -10609,14 +10699,14 @@ build_ip_routing_flows_for_lrouter_port( > > for (int i = 0; i < op->lrp_networks.n_ipv4_addrs; i++) { > > add_route(lflows, op->od, op, op->lrp_networks.ipv4_addrs[i].addr_s, > > op->lrp_networks.ipv4_addrs[i].network_s, > > - op->lrp_networks.ipv4_addrs[i].plen, NULL, false, > > + op->lrp_networks.ipv4_addrs[i].plen, NULL, false, 0, > > &op->nbrp->header_, false); > > } > > > > for (int i = 0; i < op->lrp_networks.n_ipv6_addrs; i++) { > > add_route(lflows, op->od, op, op->lrp_networks.ipv6_addrs[i].addr_s, > > op->lrp_networks.ipv6_addrs[i].network_s, > > - op->lrp_networks.ipv6_addrs[i].plen, NULL, false, > > + op->lrp_networks.ipv6_addrs[i].plen, NULL, false, 0, > > &op->nbrp->header_, false); > > } > > } else if (lsp_is_router(op->nbsp)) { > > @@ -10639,7 +10729,7 @@ build_ip_routing_flows_for_lrouter_port( > > add_route(lflows, peer->od, peer, > > peer->lrp_networks.ipv4_addrs[0].addr_s, > > laddrs->ipv4_addrs[k].network_s, > > - laddrs->ipv4_addrs[k].plen, NULL, false, > > + laddrs->ipv4_addrs[k].plen, NULL, false, 0, > > &peer->nbrp->header_, false); > > } > > } > > @@ -10659,10 +10749,17 @@ build_static_route_flows_for_lrouter( > > struct hmap ecmp_groups = HMAP_INITIALIZER(&ecmp_groups); > > struct hmap unique_routes = HMAP_INITIALIZER(&unique_routes); > > struct ovs_list parsed_routes = OVS_LIST_INITIALIZER(&parsed_routes); > > + struct simap route_tables = SIMAP_INITIALIZER(&route_tables); > > struct ecmp_groups_node *group; > > + > > + for (int i = 0; i < od->nbr->n_ports; i++) { > > + build_route_table_lflow(od, lflows, od->nbr->ports[i], > > + &route_tables); > > + } > > + > > for (int i = 0; i < od->nbr->n_static_routes; i++) { > > struct parsed_route *route = > > - parsed_routes_add(od, ports, &parsed_routes, > > + parsed_routes_add(od, ports, &parsed_routes, &route_tables, > > od->nbr->static_routes[i], bfd_connections); > > if (!route) { > > continue; > > @@ -10695,6 +10792,7 @@ build_static_route_flows_for_lrouter( > > ecmp_groups_destroy(&ecmp_groups); > > unique_routes_destroy(&unique_routes); > > parsed_routes_destroy(&parsed_routes); > > + simap_destroy(&route_tables); > > } > > } > > > > @@ -12800,6 +12898,7 @@ build_lswitch_and_lrouter_iterate_by_od(struct ovn_datapath *od, > > build_neigh_learning_flows_for_lrouter(od, lsi->lflows, &lsi->match, > > &lsi->actions, lsi->meter_groups); > > build_ND_RA_flows_for_lrouter(od, lsi->lflows); > > + build_ip_routing_pre_flows_for_lrouter(od, lsi->lflows); > > build_static_route_flows_for_lrouter(od, lsi->lflows, lsi->ports, > > lsi->bfd_connections); > > build_mcast_lookup_flows_for_lrouter(od, lsi->lflows, &lsi->match, > > diff --git a/northd/ovn-northd.8.xml b/northd/ovn-northd.8.xml > > index 39f4eaa0c..cc2e25367 100644 > > --- a/northd/ovn-northd.8.xml > > +++ b/northd/ovn-northd.8.xml > > @@ -2899,7 +2899,7 @@ icmp6 { > > > > <p> > > If ECMP routes with symmetric reply are configured in the > > - <code>OVN_Northbound</code> database for a gateway router, a priority-300 > > + <code>OVN_Northbound</code> database for a gateway router, a priority-400 > > flow is added for each router port on which symmetric replies are > > configured. The matching logic for these ports essentially reverses the > > configured logic of the ECMP route. So for instance, a route with a > > @@ -3245,7 +3245,35 @@ output; > > </li> > > </ul> > > > > - <h3>Ingress Table 10: IP Routing</h3> > > + <h3>Ingress Table 10: IP Routing Pre</h3> > > + > > + <p> > > + If a packet arrived at this table from Logical Router Port <var>P</var> > > + which has <code>options:route_table</code> value set, a logical flow with > > + match <code>inport == "<var>P</var>"</code> with priority 100 and action, > > + setting unique-generated per-datapath 32-bit value (non-zero) in OVS > > + register 7. This register is checked in next table. > > + </p> > > + > > + <p> > > + This table contains the following logical flows: > > + </p> > > + > > + <ul> > > + <li> > > + <p> > > + Priority-100 flow with match <code>inport == "LRP_NAME"</code> value > > + and action, which set route table identifier in reg7. > > + </p> > > + > > + <p> > > + A priority-0 logical flow with match <code>1</code> has actions > > + <code>next;</code>. > > + </p> > > + </li> > > + </ul> > > + > > + <h3>Ingress Table 11: IP Routing</h3> > > > > <p> > > A packet that arrives at this table is an IP packet that should be > > @@ -3316,10 +3344,10 @@ output; > > <p> > > IPv4 routing table. For each route to IPv4 network <var>N</var> with > > netmask <var>M</var>, on router port <var>P</var> with IP address > > - <var>A</var> and Ethernet > > - address <var>E</var>, a logical flow with match <code>ip4.dst == > > - <var>N</var>/<var>M</var></code>, whose priority is the number of > > - 1-bits in <var>M</var>, has the following actions: > > + <var>A</var> and Ethernet address <var>E</var>, a logical flow with > > + match <code>ip4.dst == <var>N</var>/<var>M</var></code>, whose > > + priority is 100 + the number of 1-bits in <var>M</var>, has the > > + following actions: > > </p> > > > > <pre> > > @@ -3382,6 +3410,13 @@ next; > > If the address <var>A</var> is in the link-local scope, the > > route will be limited to sending on the ingress port. > > </p> > > + > > + <p> > > + For routes with <code>route_table</code> value set > > + <code>reg7 == id</code> is prefixed in logical flow match portion. > > + Priority for routes with <code>route_table</code> value set is > > + the number of 1-bits in <var>M</var>. > > + </p> > > </li> > > > > <li> > > @@ -3408,7 +3443,7 @@ select(reg8[16..31], <var>MID1</var>, <var>MID2</var>, ...); > > </li> > > </ul> > > > > - <h3>Ingress Table 11: IP_ROUTING_ECMP</h3> > > + <h3>Ingress Table 12: IP_ROUTING_ECMP</h3> > > > > <p> > > This table implements the second part of IP routing for ECMP routes > > @@ -3460,7 +3495,7 @@ outport = <var>P</var>; > > </li> > > </ul> > > > > - <h3>Ingress Table 12: Router policies</h3> > > + <h3>Ingress Table 13: Router policies</h3> > > <p> > > This table adds flows for the logical router policies configured > > on the logical router. Please see the > > @@ -3532,7 +3567,7 @@ next; > > </li> > > </ul> > > > > - <h3>Ingress Table 13: ECMP handling for router policies</h3> > > + <h3>Ingress Table 14: ECMP handling for router policies</h3> > > <p> > > This table handles the ECMP for the router policies configured > > with multiple nexthops. > > @@ -3576,7 +3611,7 @@ outport = <var>P</var> > > </li> > > </ul> > > > > - <h3>Ingress Table 14: ARP/ND Resolution</h3> > > + <h3>Ingress Table 15: ARP/ND Resolution</h3> > > > > <p> > > Any packet that reaches this table is an IP packet whose next-hop > > @@ -3767,7 +3802,7 @@ outport = <var>P</var> > > > > </ul> > > > > - <h3>Ingress Table 15: Check packet length</h3> > > + <h3>Ingress Table 16: Check packet length</h3> > > > > <p> > > For distributed logical routers or gateway routers with gateway > > @@ -3797,7 +3832,7 @@ REGBIT_PKT_LARGER = check_pkt_larger(<var>L</var>); next; > > and advances to the next table. > > </p> > > > > - <h3>Ingress Table 16: Handle larger packets</h3> > > + <h3>Ingress Table 17: Handle larger packets</h3> > > > > <p> > > For distributed logical routers or gateway routers with gateway port > > @@ -3860,7 +3895,7 @@ icmp6 { > > and advances to the next table. > > </p> > > > > - <h3>Ingress Table 17: Gateway Redirect</h3> > > + <h3>Ingress Table 18: Gateway Redirect</h3> > > > > <p> > > For distributed logical routers where one or more of the logical router > > @@ -3907,7 +3942,7 @@ icmp6 { > > </li> > > </ul> > > > > - <h3>Ingress Table 18: ARP Request</h3> > > + <h3>Ingress Table 19: ARP Request</h3> > > > > <p> > > In the common case where the Ethernet destination has been resolved, this > > diff --git a/ovn-nb.ovsschema b/ovn-nb.ovsschema > > index 2ac8ef3ea..a0a171e19 100644 > > --- a/ovn-nb.ovsschema > > +++ b/ovn-nb.ovsschema > > @@ -1,7 +1,7 @@ > > { > > "name": "OVN_Northbound", > > - "version": "5.32.1", > > - "cksum": "2805328215 29734", > > + "version": "5.33.1", > > + "cksum": "3874993350 29785", > > "tables": { > > "NB_Global": { > > "columns": { > > @@ -387,6 +387,7 @@ > > "isRoot": false}, > > "Logical_Router_Static_Route": { > > "columns": { > > + "route_table": {"type": "string"}, > > "ip_prefix": {"type": "string"}, > > "policy": {"type": {"key": {"type": "string", > > "enum": ["set", ["src-ip", > > diff --git a/ovn-nb.xml b/ovn-nb.xml > > index d8266ed4d..b2917c363 100644 > > --- a/ovn-nb.xml > > +++ b/ovn-nb.xml > > @@ -2772,6 +2772,14 @@ > > prefix according to RFC3663 > > </p> > > </column> > > + > > + <column name="options" key="route_table"> > > + Designates lookup Logical_Router_Static_Routes with specified > > + <code>route_table</code> value. Routes to directly connected networks > > + from same Logical Router and routes without <code>route_table</code> > > + option set have higher priority than routes with > > + <code>route_table</code> option set. > > + </column> > > </group> > > > > <group title="Attachment"> > > @@ -2891,6 +2899,28 @@ > > </p> > > </column> > > > > + <column name="route_table"> > > + <p> > > + Any string to place route to separate routing table. If Logical Router > > + Port has configured value in <ref table="Logical_Router_Port" > > + column="options" key="route_table"/> other than empty string, OVN > > + performs route lookup for all packets entering Logical Router ingress > > + pipeline from this port in the following manner: > > + </p> > > + > > + <ul> > > + <li> > > + 1. First lookup among "global" routes: routes without > > + <code>route_table</code> value set and routes to directly connected > > + networks. > > + </li> > > + <li> > > + 2. Next lookup among routes with same <code>route_table</code> value > > + as specified in LRP's options:route_table field. > > + </li> > > + </ul> > > + </column> > > + > > <column name="external_ids" key="ic-learned-route"> > > <code>ovn-ic</code> populates this key if the route is learned from the > > global <ref db="OVN_IC_Southbound"/> database. In this case the value > > diff --git a/tests/ovn-ic.at b/tests/ovn-ic.at > > index 32f4e9d02..3aab54362 100644 > > --- a/tests/ovn-ic.at > > +++ b/tests/ovn-ic.at > > @@ -281,6 +281,7 @@ done > > > > AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr1], [0], [dnl > > IPv4 Routes > > +Route Table global: > > 10.11.1.0/24 169.254.0.1 dst-ip > > 10.11.2.0/24 169.254.100.2 dst-ip (learned) > > 10.22.1.0/24 169.254.0.2 src-ip > > @@ -299,6 +300,7 @@ ovn_as az1 ovn-nbctl set nb_global . options:ic-route-learn=false > > OVS_WAIT_WHILE([ovn_as az1 ovn-nbctl lr-route-list lr1 | grep learned]) > > AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr1], [0], [dnl > > IPv4 Routes > > +Route Table global: > > 10.11.1.0/24 169.254.0.1 dst-ip > > 10.22.1.0/24 169.254.0.2 src-ip > > ]) > > @@ -314,6 +316,7 @@ ovn_as az1 ovn-nbctl set nb_global . options:ic-route-adv=false > > OVS_WAIT_WHILE([ovn_as az2 ovn-nbctl lr-route-list lr2 | grep learned]) > > AT_CHECK([ovn_as az2 ovn-nbctl lr-route-list lr2], [0], [dnl > > IPv4 Routes > > +Route Table global: > > 10.11.2.0/24 169.254.0.1 dst-ip > > 10.22.2.0/24 169.254.0.2 src-ip > > ]) > > @@ -332,6 +335,7 @@ done > > # Default route should NOT get advertised or learned, by default. > > AT_CHECK([ovn_as az2 ovn-nbctl lr-route-list lr2], [0], [dnl > > IPv4 Routes > > +Route Table global: > > 10.11.1.0/24 169.254.100.1 dst-ip (learned) > > 10.11.2.0/24 169.254.0.1 dst-ip > > 10.22.2.0/24 169.254.0.2 src-ip > > diff --git a/tests/ovn-nbctl.at b/tests/ovn-nbctl.at > > index 9b80ae410..ddb5536ce 100644 > > --- a/tests/ovn-nbctl.at > > +++ b/tests/ovn-nbctl.at > > @@ -1520,6 +1520,7 @@ AT_CHECK([ovn-nbctl --ecmp --policy=src-ip lr-route-add lr0 20.0.0.0/24 11.0.0.1 > > > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > IPv4 Routes > > +Route Table global: > > 10.0.0.0/24 11.0.0.1 dst-ip > > 10.0.1.0/24 11.0.1.1 dst-ip lp0 > > 10.0.10.0/24 dst-ip lp0 > > @@ -1534,6 +1535,7 @@ AT_CHECK([ovn-nbctl lrp-add lr0 lp1 f0:00:00:00:00:02 11.0.0.254/24]) > > AT_CHECK([ovn-nbctl --may-exist lr-route-add lr0 10.0.0.111/24 11.0.0.1 lp1]) > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > IPv4 Routes > > +Route Table global: > > 10.0.0.0/24 11.0.0.1 dst-ip lp1 > > 10.0.1.0/24 11.0.1.1 dst-ip lp0 > > 10.0.10.0/24 dst-ip lp0 > > @@ -1564,6 +1566,7 @@ AT_CHECK([ovn-nbctl --policy=src-ip lr-route-del lr0 9.16.1.0/24]) > > > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > IPv4 Routes > > +Route Table global: > > 10.0.0.0/24 11.0.0.1 dst-ip lp1 > > 10.0.10.0/24 dst-ip lp0 > > 10.0.0.0/24 11.0.0.2 src-ip > > @@ -1575,6 +1578,7 @@ AT_CHECK([ovn-nbctl --policy=dst-ip lr-route-del lr0 10.0.0.0/24]) > > AT_CHECK([ovn-nbctl --policy=src-ip lr-route-del lr0 10.0.0.0/24]) > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > IPv4 Routes > > +Route Table global: > > 10.0.10.0/24 dst-ip lp0 > > 0.0.0.0/0 192.168.0.1 dst-ip > > ]) > > @@ -1585,6 +1589,7 @@ AT_CHECK([ovn-nbctl --policy=src-ip lr-route-add lr0 10.0.0.0/24 11.0.0.2]) > > AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24]) > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > IPv4 Routes > > +Route Table global: > > 10.0.10.0/24 dst-ip lp0 > > 0.0.0.0/0 192.168.0.1 dst-ip > > ]) > > @@ -1601,6 +1606,7 @@ AT_CHECK([ovn-nbctl --ecmp lr-route-add lr0 10.0.0.0/24 11.0.0.3]) > > AT_CHECK([ovn-nbctl --ecmp lr-route-add lr0 10.0.0.0/24 11.0.0.4 lp0]) > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > IPv4 Routes > > +Route Table global: > > 10.0.0.0/24 11.0.0.1 dst-ip ecmp > > 10.0.0.0/24 11.0.0.2 dst-ip ecmp > > 10.0.0.0/24 11.0.0.3 dst-ip ecmp > > @@ -1615,6 +1621,7 @@ dnl Delete ecmp routes > > AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 11.0.0.1]) > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > IPv4 Routes > > +Route Table global: > > 10.0.0.0/24 11.0.0.2 dst-ip ecmp > > 10.0.0.0/24 11.0.0.3 dst-ip ecmp > > 10.0.0.0/24 11.0.0.4 dst-ip lp0 ecmp > > @@ -1622,12 +1629,14 @@ IPv4 Routes > > AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 11.0.0.2]) > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > IPv4 Routes > > +Route Table global: > > 10.0.0.0/24 11.0.0.3 dst-ip ecmp > > 10.0.0.0/24 11.0.0.4 dst-ip lp0 ecmp > > ]) > > AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 11.0.0.4 lp0]) > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > IPv4 Routes > > +Route Table global: > > 10.0.0.0/24 11.0.0.3 dst-ip > > ]) > > AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 11.0.0.3]) > > @@ -1641,6 +1650,7 @@ AT_CHECK([ovn-nbctl lr-route-add lr0 2001:0db8:1::/64 2001:0db8:0:f103::1]) > > > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > IPv6 Routes > > +Route Table global: > > 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > > 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > > ::/0 2001:db8:0:f101::1 dst-ip > > @@ -1650,6 +1660,7 @@ AT_CHECK([ovn-nbctl lr-route-del lr0 2001:0db8:0::/64]) > > > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > IPv6 Routes > > +Route Table global: > > 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > > ::/0 2001:db8:0:f101::1 dst-ip > > ]) > > @@ -1677,11 +1688,13 @@ AT_CHECK([ovn-nbctl --may-exist --ecmp-symmetric-reply lr-route-add lr0 2003:0db > > > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > IPv4 Routes > > +Route Table global: > > 10.0.0.0/24 11.0.0.1 dst-ip > > 10.0.1.0/24 11.0.1.1 dst-ip lp0 > > 0.0.0.0/0 192.168.0.1 dst-ip > > > > IPv6 Routes > > +Route Table global: > > 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > > 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip ecmp > > 2001:db8:1::/64 2001:db8:0:f103::2 dst-ip ecmp > > @@ -1696,7 +1709,188 @@ AT_CHECK([ovn-nbctl lrp-add lr0 lr0-p0 00:00:01:01:02:03 192.168.10.1/24]) > > bfd_uuid=$(ovn-nbctl create bfd logical_port=lr0-p0 dst_ip=100.0.0.50 status=down min_tx=250 min_rx=250 detect_mult=10) > > AT_CHECK([ovn-nbctl lr-route-add lr0 100.0.0.0/24 192.168.0.1]) > > route_uuid=$(fetch_column nb:logical_router_static_route _uuid ip_prefix="100.0.0.0/24") > > -AT_CHECK([ovn-nbctl set logical_router_static_route $route_uuid bfd=$bfd_uuid])]) > > +AT_CHECK([ovn-nbctl set logical_router_static_route $route_uuid bfd=$bfd_uuid]) > > + > > +check ovn-nbctl lr-route-del lr0 > > +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > +]) > > + > > +dnl Check IPv4 routes in route table > > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 0.0.0.0/0 192.168.0.1 > > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 10.0.1.1/24 11.0.1.1 lp0 > > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 10.0.0.1/24 11.0.0.1 > > +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > +IPv4 Routes > > +Route Table rtb-1: > > + 10.0.0.0/24 11.0.0.1 dst-ip > > + 10.0.1.0/24 11.0.1.1 dst-ip lp0 > > + 0.0.0.0/0 192.168.0.1 dst-ip > > +]) > > + > > +check ovn-nbctl lr-route-del lr0 > > +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > +]) > > + > > +dnl Check IPv6 routes in route table > > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 0:0:0:0:0:0:0:0/0 2001:0db8:0:f101::1 > > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 2001:0db8:0::/64 2001:0db8:0:f102::1 lp0 > > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 2001:0db8:1::/64 2001:0db8:0:f103::1 > > + > > +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > +IPv6 Routes > > +Route Table rtb-1: > > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > > + ::/0 2001:db8:0:f101::1 dst-ip > > +]) > > + > > +dnl Check IPv4 and IPv6 routes in route table > > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 0.0.0.0/0 192.168.0.1 > > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 10.0.1.1/24 11.0.1.1 lp0 > > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 10.0.0.1/24 11.0.0.1 > > + > > +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > +IPv4 Routes > > +Route Table rtb-1: > > + 10.0.0.0/24 11.0.0.1 dst-ip > > + 10.0.1.0/24 11.0.1.1 dst-ip lp0 > > + 0.0.0.0/0 192.168.0.1 dst-ip > > + > > +IPv6 Routes > > +Route Table rtb-1: > > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > > + ::/0 2001:db8:0:f101::1 dst-ip > > +]) > > + > > +# Add routes in another route table > > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 0.0.0.0/0 192.168.0.1 > > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 10.0.1.1/24 11.0.1.1 lp0 > > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 10.0.0.1/24 11.0.0.1 > > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 0:0:0:0:0:0:0:0/0 2001:0db8:0:f101::1 > > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 2001:0db8:0::/64 2001:0db8:0:f102::1 lp0 > > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 2001:0db8:1::/64 2001:0db8:0:f103::1 > > + > > +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > +IPv4 Routes > > +Route Table rtb-1: > > + 10.0.0.0/24 11.0.0.1 dst-ip > > + 10.0.1.0/24 11.0.1.1 dst-ip lp0 > > + 0.0.0.0/0 192.168.0.1 dst-ip > > + > > +Route Table rtb-2: > > + 10.0.0.0/24 11.0.0.1 dst-ip > > + 10.0.1.0/24 11.0.1.1 dst-ip lp0 > > + 0.0.0.0/0 192.168.0.1 dst-ip > > + > > +IPv6 Routes > > +Route Table rtb-1: > > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > > + ::/0 2001:db8:0:f101::1 dst-ip > > + > > +Route Table rtb-2: > > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > > + ::/0 2001:db8:0:f101::1 dst-ip > > +]) > > + > > +# Add routes to global route table > > +check ovn-nbctl lr-route-add lr0 0.0.0.0/0 192.168.0.1 > > +check ovn-nbctl lr-route-add lr0 10.0.1.1/24 11.0.1.1 lp0 > > +check ovn-nbctl lr-route-add lr0 10.0.0.1/24 11.0.0.1 > > +check ovn-nbctl lr-route-add lr0 0:0:0:0:0:0:0:0/0 2001:0db8:0:f101::1 > > +check ovn-nbctl lr-route-add lr0 2001:0db8:0::/64 2001:0db8:0:f102::1 lp0 > > +check check ovn-nbctl lr-route-add lr0 2001:0db8:1::/64 2001:0db8:0:f103::1 > > + > > +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > > +IPv4 Routes > > +Route Table global: > > + 10.0.0.0/24 11.0.0.1 dst-ip > > + 10.0.1.0/24 11.0.1.1 dst-ip lp0 > > + 0.0.0.0/0 192.168.0.1 dst-ip > > + > > +Route Table rtb-1: > > + 10.0.0.0/24 11.0.0.1 dst-ip > > + 10.0.1.0/24 11.0.1.1 dst-ip lp0 > > + 0.0.0.0/0 192.168.0.1 dst-ip > > + > > +Route Table rtb-2: > > + 10.0.0.0/24 11.0.0.1 dst-ip > > + 10.0.1.0/24 11.0.1.1 dst-ip lp0 > > + 0.0.0.0/0 192.168.0.1 dst-ip > > + > > +IPv6 Routes > > +Route Table global: > > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > > + ::/0 2001:db8:0:f101::1 dst-ip > > + > > +Route Table rtb-1: > > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > > + ::/0 2001:db8:0:f101::1 dst-ip > > + > > +Route Table rtb-2: > > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > > + ::/0 2001:db8:0:f101::1 dst-ip > > +]) > > + > > +# delete IPv4 route from rtb-1 > > +check ovn-nbctl --route-table=rtb-1 lr-route-del lr0 10.0.0.0/24 > > +AT_CHECK([ovn-nbctl --route-table=rtb-1 lr-route-list lr0], [0], [dnl > > +IPv4 Routes > > +Route Table rtb-1: > > + 10.0.1.0/24 11.0.1.1 dst-ip lp0 > > + 0.0.0.0/0 192.168.0.1 dst-ip > > + > > +IPv6 Routes > > +Route Table rtb-1: > > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > > + ::/0 2001:db8:0:f101::1 dst-ip > > +]) > > + > > +# delete IPv6 route from rtb-2 > > +check ovn-nbctl --route-table=rtb-2 lr-route-del lr0 2001:db8::/64 > > +AT_CHECK([ovn-nbctl --route-table=rtb-2 lr-route-list lr0], [0], [dnl > > +IPv4 Routes > > +Route Table rtb-2: > > + 10.0.0.0/24 11.0.0.1 dst-ip > > + 10.0.1.0/24 11.0.1.1 dst-ip lp0 > > + 0.0.0.0/0 192.168.0.1 dst-ip > > + > > +IPv6 Routes > > +Route Table rtb-2: > > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > > + ::/0 2001:db8:0:f101::1 dst-ip > > +]) > > + > > +check ovn-nbctl lr-route-del lr0 > > + > > +# ECMP route in route table > > +check ovn-nbctl --route-table=rtb1 lr-route-add lr0 0.0.0.0/0 192.168.0.1 > > +check ovn-nbctl --ecmp --route-table=rtb1 lr-route-add lr0 0.0.0.0/0 192.168.0.2 > > + > > +# Negative route table case: same prefix > > +AT_CHECK([ovn-nbctl --route-table=rtb1 lr-route-add lr0 0.0.0.0/0 192.168.0.1], [1], [], [dnl > > +ovn-nbctl: duplicate prefix: 0.0.0.0/0 (policy: dst-ip). Use option --ecmp to allow this for ECMP routing. > > +]) > > + > > +# Negative route table case: same prefix & nexthop with ecmp > > +AT_CHECK([ovn-nbctl --ecmp --route-table=rtb1 lr-route-add lr0 0.0.0.0/0 192.168.0.2], [1], [], [dnl > > +ovn-nbctl: duplicate nexthop for the same ECMP route > > +]) > > + > > +# Add routes to global route table > > +check ovn-nbctl lrp-add lr0 lrp0 00:00:00:00:00:01 1.1.1.1/24 > > +check ovn-nbctl lrp-set-options lrp0 route_table=rtb1 > > +AT_CHECK([ovn-nbctl get logical-router-port lrp0 options:route_table], [0], [dnl > > +rtb1 > > +]) > > +check `ovn-nbctl show lr0 | grep lrp0 -A3 | grep route_table=rtb1` > > +]) > > > > dnl --------------------------------------------------------------------- > > > > diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at > > index 3eebb55b6..e71e65bcc 100644 > > --- a/tests/ovn-northd.at > > +++ b/tests/ovn-northd.at > > @@ -5111,7 +5111,7 @@ check ovn-nbctl --wait=sb --ecmp-symmetric-reply lr-route-add lr0 1.0.0.1 192.16 > > > > ovn-sbctl dump-flows lr0 > lr0flows > > AT_CHECK([grep -e "lr_in_ip_routing.*select" lr0flows | sed 's/table=../table=??/' | sort], [0], [dnl > > - table=??(lr_in_ip_routing ), priority=65 , match=(ip4.dst == 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = 1; reg8[[16..31]] = select(1, 2);) > > + table=??(lr_in_ip_routing ), priority=165 , match=(ip4.dst == 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = 1; reg8[[16..31]] = select(1, 2);) > > ]) > > AT_CHECK([grep -e "lr_in_ip_routing_ecmp" lr0flows | sed 's/192\.168\.0\..0/192.168.0.??/' | sed 's/table=../table=??/' | sort], [0], [dnl > > table=??(lr_in_ip_routing_ecmp), priority=100 , match=(reg8[[0..15]] == 1 && reg8[[16..31]] == 1), action=(reg0 = 192.168.0.??; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; next;) > > @@ -5124,7 +5124,7 @@ check ovn-nbctl --wait=sb --ecmp-symmetric-reply lr-route-add lr0 1.0.0.1 192.16 > > > > ovn-sbctl dump-flows lr0 > lr0flows > > AT_CHECK([grep -e "lr_in_ip_routing.*select" lr0flows | sed 's/table=../table=??/' | sort], [0], [dnl > > - table=??(lr_in_ip_routing ), priority=65 , match=(ip4.dst == 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = 1; reg8[[16..31]] = select(1, 2);) > > + table=??(lr_in_ip_routing ), priority=165 , match=(ip4.dst == 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = 1; reg8[[16..31]] = select(1, 2);) > > ]) > > AT_CHECK([grep -e "lr_in_ip_routing_ecmp" lr0flows | sed 's/192\.168\.0\..0/192.168.0.??/' | sed 's/table=../table=??/' | sort], [0], [dnl > > table=??(lr_in_ip_routing_ecmp), priority=100 , match=(reg8[[0..15]] == 1 && reg8[[16..31]] == 1), action=(reg0 = 192.168.0.??; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; next;) > > @@ -5139,14 +5139,14 @@ check ovn-nbctl --wait=sb lr-route-add lr0 1.0.0.0/24 192.168.0.10 > > ovn-sbctl dump-flows lr0 > lr0flows > > > > AT_CHECK([grep -e "lr_in_ip_routing.*192.168.0.10" lr0flows | sed 's/table=../table=??/' | sort], [0], [dnl > > - table=??(lr_in_ip_routing ), priority=49 , match=(ip4.dst == 1.0.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.10; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == 1.0.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.10; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; flags.loopback = 1; next;) > > ]) > > > > check ovn-nbctl --wait=sb lr-route-add lr0 2.0.0.0/24 lr0-public > > > > ovn-sbctl dump-flows lr0 > lr0flows > > AT_CHECK([grep -e "lr_in_ip_routing.*2.0.0.0" lr0flows | sed 's/table=../table=??/' | sort], [0], [dnl > > - table=??(lr_in_ip_routing ), priority=49 , match=(ip4.dst == 2.0.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == 2.0.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; flags.loopback = 1; next;) > > ]) > > > > AT_CLEANUP > > @@ -5232,3 +5232,71 @@ AT_CHECK([grep lr_in_gw_redirect lrflows | grep cr-DR | sed 's/table=../table=?? > > > > AT_CLEANUP > > ]) > > + > > + > > +OVN_FOR_EACH_NORTHD([ > > +AT_SETUP([route tables -- flows]) > > +AT_KEYWORDS([route-tables-flows]) > > +ovn_start > > + > > +check ovn-nbctl lr-add lr0 > > +check ovn-nbctl lrp-add lr0 lrp0 00:00:00:00:00:01 192.168.0.1/24 > > +check ovn-nbctl lrp-add lr0 lrp1 00:00:00:00:01:01 192.168.1.1/24 > > +check ovn-nbctl lrp-add lr0 lrp2 00:00:00:00:02:01 192.168.2.1/24 > > +check ovn-nbctl lrp-set-options lrp1 route_table=rtb-1 > > +check ovn-nbctl lrp-set-options lrp2 route_table=rtb-2 > > + > > +check ovn-nbctl lr-route-add lr0 0.0.0.0/0 192.168.0.10 > > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 192.168.0.0/24 192.168.1.10 > > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 0.0.0.0/0 192.168.0.10 > > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 1.1.1.1/32 192.168.0.20 > > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 2.2.2.2/32 192.168.0.30 > > +check ovn-nbctl --route-table=rtb-2 --ecmp lr-route-add lr0 2.2.2.2/32 192.168.0.31 > > +check ovn-nbctl --wait=sb sync > > + > > +ovn-sbctl dump-flows lr0 > lr0flows > > +AT_CAPTURE_FILE([lr0flows]) > > + > > +AT_CHECK([grep -e "lr_in_ip_routing_pre.*match=(1)" lr0flows | sed 's/table=../table=??/'], [0], [dnl > > + table=??(lr_in_ip_routing_pre), priority=0 , match=(1), action=(next;) > > +]) > > + > > +p1_reg=$(grep -oP "lr_in_ip_routing_pre.*lrp1.*action=\(reg7 = \K." lr0flows) > > +p2_reg=$(grep -oP "lr_in_ip_routing_pre.*lrp2.*action=\(reg7 = \K." lr0flows) > > +echo $p1_reg > > +echo $p2_reg > > + > > +# exact register values are not predictable > > +if [[ $p1_reg -eq 2 ] && [ $p2_reg -eq 1 ]]; then > > + echo "swap reg values in dump" > > + sed -i -r s'/^(.*lrp2.*action=\(reg7 = )(1)(.*)/\12\3/g' lr0flows # "reg7 = 1" -> "reg7 = 2" > > + sed -i -r s'/^(.*lrp1.*action=\(reg7 = )(2)(.*)/\11\3/g' lr0flows # "reg7 = 2" -> "reg7 = 1" > > + sed -i -r s'/^(.*match=\(reg7 == )(2)( &&.*lrp1.*)/\11\3/g' lr0flows # "reg7 == 2" -> "reg7 == 1" > > + sed -i -r s'/^(.*match=\(reg7 == )(1)( &&.*lrp0.*)/\12\3/g' lr0flows # "reg7 == 1" -> "reg7 == 2" > > +fi > > + > > +check test "$p1_reg" != "$p2_reg" -a $((p1_reg * p2_reg)) -eq 2 > > + > > +AT_CHECK([grep "lr_in_ip_routing_pre" lr0flows | sed 's/table=../table=??/' | sort], [0], [dnl > > + table=??(lr_in_ip_routing_pre), priority=0 , match=(1), action=(next;) > > + table=??(lr_in_ip_routing_pre), priority=100 , match=(inport == "lrp1"), action=(reg7 = 1; next;) > > + table=??(lr_in_ip_routing_pre), priority=100 , match=(inport == "lrp2"), action=(reg7 = 2; next;) > > +]) > > + > > +grep -e "(lr_in_ip_routing ).*outport" lr0flows > > + > > +AT_CHECK([grep -e "(lr_in_ip_routing ).*outport" lr0flows | sed 's/table=../table=??/' | sort], [0], [dnl > > + table=??(lr_in_ip_routing ), priority=1 , match=(reg7 == 2 && ip4.dst == 0.0.0.0/0), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.10; reg1 = 192.168.0.1; eth.src = 00:00:00:00:00:01; outport = "lrp0"; flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=101 , match=(ip4.dst == 0.0.0.0/0), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.10; reg1 = 192.168.0.1; eth.src = 00:00:00:00:00:01; outport = "lrp0"; flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == 192.168.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 = 192.168.0.1; eth.src = 00:00:00:00:00:01; outport = "lrp0"; flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == 192.168.1.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 = 192.168.1.1; eth.src = 00:00:00:00:01:01; outport = "lrp1"; flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == 192.168.2.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 = 192.168.2.1; eth.src = 00:00:00:00:02:01; outport = "lrp2"; flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=229 , match=(inport == "lrp0" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe00:1; eth.src = 00:00:00:00:00:01; outport = "lrp0"; flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=229 , match=(inport == "lrp1" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe00:101; eth.src = 00:00:00:00:01:01; outport = "lrp1"; flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=229 , match=(inport == "lrp2" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe00:201; eth.src = 00:00:00:00:02:01; outport = "lrp2"; flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=49 , match=(reg7 == 1 && ip4.dst == 192.168.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.1.10; reg1 = 192.168.1.1; eth.src = 00:00:00:00:01:01; outport = "lrp1"; flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=65 , match=(reg7 == 2 && ip4.dst == 1.1.1.1/32), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.20; reg1 = 192.168.0.1; eth.src = 00:00:00:00:00:01; outport = "lrp0"; flags.loopback = 1; next;) > > +]) > > + > > +AT_CLEANUP > > +]) > > diff --git a/tests/ovn.at b/tests/ovn.at > > index 49ece8735..60783a14b 100644 > > --- a/tests/ovn.at > > +++ b/tests/ovn.at > > @@ -18145,7 +18145,7 @@ eth_dst=00000000ff01 > > ip_src=$(ip_to_hex 10 0 0 10) > > ip_dst=$(ip_to_hex 172 168 0 101) > > send_icmp_packet 1 1 $eth_src $eth_dst $ip_src $ip_dst c4c9 0000000000000000000000 > > -AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | awk '/table=25, n_packets=1, n_bytes=45/{print $7" "$8}'],[0],[dnl > > +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | awk '/table=26, n_packets=1, n_bytes=45/{print $7" "$8}'],[0],[dnl > > priority=80,ip,reg15=0x3,metadata=0x3,nw_src=10.0.0.10 actions=drop > > ]) > > > > @@ -22577,6 +22577,433 @@ OVN_CLEANUP([hv1]) > > AT_CLEANUP > > ]) > > > > + > > +OVN_FOR_EACH_NORTHD([ > > +AT_SETUP([route tables -- global routes]) > > +ovn_start > > + > > +# Logical network: > > +# ls1 (192.168.1.0/24) - lrp-lr1-ls1 - lr1 - lrp-lr1-ls2 - ls2 ( 192.168.2.0/24) > > +# > > +# ls1 has lsp11 (192.168.1.11) and ls2 has lsp21 (192.168.2.21) and lsp22 > > +# (192.168.2.22) > > +# > > +# lrp-lr1-ls1 set options:route_table=rtb-1 > > +# > > +# Static routes on lr1: > > +# 0.0.0.0/0 nexthop 192.168.2.21 > > +# 1.1.1.1/32 nexthop 192.168.2.22 route_table=rtb-1 > > +# > > +# Test 1: > > +# lsp11 send packet to 2.2.2.2 > > +# > > +# Expected result: > > +# lsp21 should receive traffic, lsp22 should not receive any traffic > > +# > > +# Test 2: > > +# lsp11 send packet to 1.1.1.1 > > +# > > +# Expected result: > > +# lsp21 should receive traffic, lsp22 should not receive any traffic > > + > > +ovn-nbctl lr-add lr1 > > + > > +for i in 1 2; do > > + ovn-nbctl ls-add ls${i} > > + ovn-nbctl lrp-add lr1 lrp-lr1-ls${i} 00:00:00:01:0${i}:01 192.168.${i}.1/24 > > + ovn-nbctl lsp-add ls${i} lsp-ls${i}-lr1 -- lsp-set-type lsp-ls${i}-lr1 router \ > > + -- lsp-set-options lsp-ls${i}-lr1 router-port=lrp-lr1-ls${i} \ > > + -- lsp-set-addresses lsp-ls${i}-lr1 router > > +done > > + > > +# install static routes > > +ovn-nbctl lr-route-add lr1 0.0.0.0/0 192.168.2.21 > > +ovn-nbctl --route-table=rtb-1 lr-route-add lr1 1.1.1.1/32 192.168.2.22 > > + > > +# set lrp-lr1-ls1 route table > > +ovn-nbctl lrp-set-options lrp-lr1-ls1 route_table=rtb-1 > > + > > +# Create logical ports > > +ovn-nbctl lsp-add ls1 lsp11 -- \ > > + lsp-set-addresses lsp11 "f0:00:00:00:01:11 192.168.1.11" > > +ovn-nbctl lsp-add ls2 lsp21 -- \ > > + lsp-set-addresses lsp21 "f0:00:00:00:02:21 192.168.2.21" > > +ovn-nbctl lsp-add ls2 lsp22 -- \ > > + lsp-set-addresses lsp22 "f0:00:00:00:02:22 192.168.2.22" > > + > > +net_add n1 > > +sim_add hv1 > > +as hv1 > > +ovs-vsctl add-br br-phys > > +ovn_attach n1 br-phys 192.168.0.1 > > +ovs-vsctl -- add-port br-int hv1-vif1 -- \ > > + set interface hv1-vif1 external-ids:iface-id=lsp11 \ > > + options:tx_pcap=hv1/vif1-tx.pcap \ > > + options:rxq_pcap=hv1/vif1-rx.pcap \ > > + ofport-request=1 > > + > > +ovs-vsctl -- add-port br-int hv1-vif2 -- \ > > + set interface hv1-vif2 external-ids:iface-id=lsp21 \ > > + options:tx_pcap=hv1/vif2-tx.pcap \ > > + options:rxq_pcap=hv1/vif2-rx.pcap \ > > + ofport-request=2 > > + > > +ovs-vsctl -- add-port br-int hv1-vif3 -- \ > > + set interface hv1-vif3 external-ids:iface-id=lsp22 \ > > + options:tx_pcap=hv1/vif3-tx.pcap \ > > + options:rxq_pcap=hv1/vif3-rx.pcap \ > > + ofport-request=3 > > + > > +# wait for earlier changes to take effect > > +check ovn-nbctl --wait=hv sync > > +wait_for_ports_up > > + > > +for i in 1 2; do > > + packet="inport==\"lsp11\" && eth.src==f0:00:00:00:01:11 && eth.dst==00:00:00:01:01:01 && > > + ip4 && ip.ttl==64 && ip4.src==192.168.1.11 && ip4.dst==$i.$i.$i.$i && icmp" > > + AT_CHECK([as hv1 ovs-appctl -t ovn-controller inject-pkt "$packet"]) > > + > > + # Assume all packets go to lsp21. > > + exp_packet="eth.src==00:00:00:01:02:01 && eth.dst==f0:00:00:00:02:21 && > > + ip4 && ip.ttl==63 && ip4.src==192.168.1.11 && ip4.dst==$i.$i.$i.$i && icmp" > > + echo $exp_packet | ovstest test-ovn expr-to-packets >> expected_lsp21 > > +done > > +> expected_lsp22 > > + > > +# lsp21 should recieve 2 packets and lsp22 should recieve no packets > > +OVS_WAIT_UNTIL([ > > + rcv_n1=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" hv1/vif2-tx.pcap > lsp21.packets && cat lsp21.packets | wc -l` > > + rcv_n2=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" hv1/vif3-tx.pcap > lsp22.packets && cat lsp22.packets | wc -l` > > + echo $rcv_n1 $rcv_n2 > > + test $rcv_n1 -eq 2 -a $rcv_n2 -eq 0]) > > + > > +for i in 1 2; do > > + sort expected_lsp2$i > expout > > + AT_CHECK([cat lsp2${i}.packets | sort], [0], [expout]) > > +done > > + > > +OVN_CLEANUP([hv1]) > > +AT_CLEANUP > > +]) > > + > > + > > +OVN_FOR_EACH_NORTHD([ > > +AT_SETUP([route tables -- directly connected routes]) > > +ovn_start > > + > > +# Logical network: > > +# ls1 (192.168.1.0/24) - lrp-lr1-ls1 - lr1 - lrp-lr1-ls2 - ls2 ( 192.168.2.0/24) > > +# > > +# ls1 has lsp11 (192.168.1.11) and ls2 has lsp21 (192.168.2.21) > > +# > > +# lrp-lr1-ls1 set options:route_table=rtb-1 > > +# > > +# Static routes on lr1: > > +# 192.168.2.0/25 nexthop 192.168.1.11 route_table=rtb-1 > > +# > > +# Test 1: > > +# lsp11 send packet to 192.168.2.21 > > +# > > +# Expected result: > > +# lsp21 should receive traffic, lsp11 should not receive any traffic > > + > > +ovn-nbctl lr-add lr1 > > + > > +for i in 1 2; do > > + ovn-nbctl ls-add ls${i} > > + ovn-nbctl lrp-add lr1 lrp-lr1-ls${i} 00:00:00:01:0${i}:01 192.168.${i}.1/24 > > + ovn-nbctl lsp-add ls${i} lsp-ls${i}-lr1 -- lsp-set-type lsp-ls${i}-lr1 router \ > > + -- lsp-set-options lsp-ls${i}-lr1 router-port=lrp-lr1-ls${i} \ > > + -- lsp-set-addresses lsp-ls${i}-lr1 router > > +done > > + > > +# install static route, which overrides directly-connected routes > > +ovn-nbctl --route-table=rtb-1 lr-route-add lr1 192.168.2.0/25 192.168.1.11 > > + > > +# set lrp-lr1-ls1 route table > > +ovn-nbctl lrp-set-options lrp-lr1-ls1 route_table=rtb-1 > > + > > +# Create logical ports > > +ovn-nbctl lsp-add ls1 lsp11 -- \ > > + lsp-set-addresses lsp11 "f0:00:00:00:01:11 192.168.1.11" > > +ovn-nbctl lsp-add ls2 lsp21 -- \ > > + lsp-set-addresses lsp21 "f0:00:00:00:02:21 192.168.2.21" > > + > > +net_add n1 > > +sim_add hv1 > > +as hv1 > > +ovs-vsctl add-br br-phys > > +ovn_attach n1 br-phys 192.168.0.1 > > +ovs-vsctl -- add-port br-int hv1-vif1 -- \ > > + set interface hv1-vif1 external-ids:iface-id=lsp11 \ > > + options:tx_pcap=hv1/vif1-tx.pcap \ > > + options:rxq_pcap=hv1/vif1-rx.pcap \ > > + ofport-request=1 > > + > > +ovs-vsctl -- add-port br-int hv1-vif2 -- \ > > + set interface hv1-vif2 external-ids:iface-id=lsp21 \ > > + options:tx_pcap=hv1/vif2-tx.pcap \ > > + options:rxq_pcap=hv1/vif2-rx.pcap \ > > + ofport-request=2 > > + > > +# wait for earlier changes to take effect > > +check ovn-nbctl --wait=hv sync > > +wait_for_ports_up > > + > > +packet="inport==\"lsp11\" && eth.src==f0:00:00:00:01:11 && eth.dst==00:00:00:01:01:01 && > > + ip4 && ip.ttl==64 && ip4.src==192.168.1.11 && ip4.dst==192.168.2.21 && icmp" > > +AT_CHECK([as hv1 ovs-appctl -t ovn-controller inject-pkt "$packet"]) > > + > > +# Assume all packets go to lsp21. > > +exp_packet="eth.src==00:00:00:01:02:01 && eth.dst==f0:00:00:00:02:21 && > > + ip4 && ip.ttl==63 && ip4.src==192.168.1.11 && ip4.dst==192.168.2.21 && icmp" > > +echo $exp_packet | ovstest test-ovn expr-to-packets >> expected_lsp21 > > +> expected_lsp11 > > + > > +# lsp21 should recieve 1 icmp packet and lsp11 should recieve no packets > > +OVS_WAIT_UNTIL([ > > + rcv_n11=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" hv1/vif1-tx.pcap > lsp11.packets && cat lsp11.packets | wc -l` > > + rcv_n21=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" hv1/vif2-tx.pcap > lsp21.packets && cat lsp21.packets | wc -l` > > + echo $rcv_n11 $rcv_n21 > > + test $rcv_n11 -eq 0 -a $rcv_n21 -eq 1]) > > + > > +for i in 11 21; do > > + sort expected_lsp$i > expout > > + AT_CHECK([cat lsp${i}.packets | sort], [0], [expout]) > > +done > > + > > +OVN_CLEANUP([hv1]) > > +AT_CLEANUP > > +]) > > + > > + > > +OVN_FOR_EACH_NORTHD([ > > +AT_SETUP([route tables -- overlapping subnets]) > > +ovn_start > > + > > +# Logical network: > > +# > > +# ls1 (192.168.1.0/24) - lrp-lr1-ls1 -\ /- lrp-lr1-ls2 - ls2 ( 192.168.2.0/24) > > +# lr1 > > +# ls3 (192.168.3.0/24) - lrp-lr1-ls3 -/ \- lrp-lr1-ls4 - ls4 ( 192.168.4.0/24) > > +# > > +# ls1 has lsp11 (192.168.1.11) > > +# ls2 has lsp21 (192.168.2.21) > > +# ls3 has lsp31 (192.168.3.31) > > +# ls4 has lsp41 (192.168.4.41) > > +# > > +# lrp-lr1-ls1 set options:route_table=rtb-1 > > +# lrp-lr1-ls2 set options:route_table=rtb-2 > > +# > > +# Static routes on lr1: > > +# 10.0.0.0/24 nexthop 192.168.3.31 route_table=rtb-1 > > +# 10.0.0.0/24 nexthop 192.168.4.41 route_table=rtb-2 > > +# > > +# Test 1: > > +# lsp11 send packet to 10.0.0.1 > > +# > > +# Expected result: > > +# lsp31 should receive traffic, lsp41 should not receive any traffic > > +# > > +# Test 2: > > +# lsp21 send packet to 10.0.0.1 > > +# > > +# Expected result: > > +# lsp41 should receive traffic, lsp31 should not receive any traffic > > + > > +ovn-nbctl lr-add lr1 > > + > > +# Create logical topology > > +for i in $(seq 1 4); do > > + ovn-nbctl ls-add ls${i} > > + ovn-nbctl lrp-add lr1 lrp-lr1-ls${i} 00:00:00:01:0${i}:01 192.168.${i}.1/24 > > + ovn-nbctl lsp-add ls${i} lsp-ls${i}-lr1 -- lsp-set-type lsp-ls${i}-lr1 router \ > > + -- lsp-set-options lsp-ls${i}-lr1 router-port=lrp-lr1-ls${i} \ > > + -- lsp-set-addresses lsp-ls${i}-lr1 router > > + ovn-nbctl lsp-add ls$i lsp${i}1 -- \ > > + lsp-set-addresses lsp${i}1 "f0:00:00:00:0${i}:1${i} 192.168.${i}.${i}1" > > +done > > + > > +# install static routes > > +ovn-nbctl --route-table=rtb-1 lr-route-add lr1 10.0.0.0/24 192.168.3.31 > > +ovn-nbctl --route-table=rtb-2 lr-route-add lr1 10.0.0.0/24 192.168.4.41 > > + > > +# set lrp-lr1-ls{1,2} route tables > > +ovn-nbctl lrp-set-options lrp-lr1-ls1 route_table=rtb-1 > > +ovn-nbctl lrp-set-options lrp-lr1-ls2 route_table=rtb-2 > > + > > +net_add n1 > > +sim_add hv1 > > +as hv1 > > +ovs-vsctl add-br br-phys > > +ovn_attach n1 br-phys 192.168.0.1 > > + > > +for i in $(seq 1 4); do > > + ovs-vsctl -- add-port br-int hv1-vif${i} -- \ > > + set interface hv1-vif${i} external-ids:iface-id=lsp${i}1 \ > > + options:tx_pcap=hv1/vif${i}-tx.pcap \ > > + options:rxq_pcap=hv1/vif${i}-rx.pcap \ > > + ofport-request=${i} > > +done > > + > > +# wait for earlier changes to take effect > > +check ovn-nbctl --wait=hv sync > > +wait_for_ports_up > > + > > +# lsp31 should recieve packet coming from lsp11 > > +# lsp41 should recieve packet coming from lsp21 > > +for i in $(seq 1 2); do > > + di=$(( i + 2)) # dst index > > + ri=$(( 5 - i)) # reverse index > > + packet="inport==\"lsp${i}1\" && eth.src==f0:00:00:00:0${i}:1${i} && > > + eth.dst==00:00:00:01:0${i}:01 && ip4 && ip.ttl==64 && > > + ip4.src==192.168.${i}.${i}1 && ip4.dst==10.0.0.1 && icmp" > > + AT_CHECK([as hv1 ovs-appctl -t ovn-controller inject-pkt "$packet"]) > > + > > + # Assume all packets go to lsp${di}1. > > + exp_packet="eth.src==00:00:00:01:0${di}:01 && eth.dst==f0:00:00:00:0${di}:1${di} && > > + ip4 && ip.ttl==63 && ip4.src==192.168.${i}.${i}1 && ip4.dst==10.0.0.1 && icmp" > > + echo $exp_packet | ovstest test-ovn expr-to-packets >> expected_lsp${di}1 > > + > expected_lsp${ri}1 > > + > > + OVS_WAIT_UNTIL([ > > + rcv_n1=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" hv1/vif${di}-tx.pcap > lsp${di}1.packets && cat lsp${di}1.packets | wc -l` > > + rcv_n2=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" hv1/vif${ri}-tx.pcap > lsp${ri}1.packets && cat lsp${ri}1.packets | wc -l` > > + echo $rcv_n1 $rcv_n2 > > + test $rcv_n1 -eq 1 -a $rcv_n2 -eq 0]) > > + > > + for j in "${di}1" "${ri}1"; do > > + sort expected_lsp${j} > expout > > + AT_CHECK([cat lsp${j}.packets | sort], [0], [expout]) > > + done > > + > > + # cleanup tx pcap files > > + for j in "${di}1" "${ri}1"; do > > + ovs-vsctl -- remove interface hv1-vif${di} options tx_pcap > > + > hv1/vif${di}-tx.pcap > > + ovs-vsctl -- set interface hv1-vif${di} external-ids:iface-id=lsp${di}1 \ > > + options:tx_pcap=hv1/vif${di}-tx.pcap > > + done > > +done > > + > > +OVN_CLEANUP([hv1]) > > +AT_CLEANUP > > +]) > > + > > + > > +OVN_FOR_EACH_NORTHD([ > > +AT_SETUP([route tables IPv6 -- overlapping subnets]) > > +ovn_start > > + > > +# Logical network: > > +# > > +# ls1 (2001:db8:1::/64) - lrp-lr1-ls1 -\ /- lrp-lr1-ls2 - ls2 (2001:db8:2::/64) > > +# lr1 > > +# ls3 (2001:db8:3::/64) - lrp-lr1-ls3 -/ \- lrp-lr1-ls4 - ls4 (2001:db8:4::/64) > > +# > > +# ls1 has lsp11 (2001:db8:1::11) > > +# ls2 has lsp21 (2001:db8:2::21) > > +# ls3 has lsp31 (2001:db8:3::31) > > +# ls4 has lsp41 (2001:db8:4::41) > > +# > > +# lrp-lr1-ls1 set options:route_table=rtb-1 > > +# lrp-lr1-ls2 set options:route_table=rtb-2 > > +# > > +# Static routes on lr1: > > +# 2001:db8:2000::/64 nexthop 2001:db8:3::31 route_table=rtb-1 > > +# 2001:db8:2000::/64 nexthop 2001:db8:3::41 route_table=rtb-2 > > +# > > +# Test 1: > > +# lsp11 send packet to 2001:db8:2000::1 > > +# > > +# Expected result: > > +# lsp31 should receive traffic, lsp41 should not receive any traffic > > +# > > +# Test 2: > > +# lsp21 send packet to 2001:db8:2000::1 > > +# > > +# Expected result: > > +# lsp41 should receive traffic, lsp31 should not receive any traffic > > + > > +ovn-nbctl lr-add lr1 > > + > > +# Create logical topology > > +for i in $(seq 1 4); do > > + ovn-nbctl ls-add ls${i} > > + ovn-nbctl lrp-add lr1 lrp-lr1-ls${i} 00:00:00:01:0${i}:01 2001:db8:${i}::1/64 > > + ovn-nbctl lsp-add ls${i} lsp-ls${i}-lr1 -- lsp-set-type lsp-ls${i}-lr1 router \ > > + -- lsp-set-options lsp-ls${i}-lr1 router-port=lrp-lr1-ls${i} \ > > + -- lsp-set-addresses lsp-ls${i}-lr1 router > > + ovn-nbctl lsp-add ls$i lsp${i}1 -- \ > > + lsp-set-addresses lsp${i}1 "f0:00:00:00:0${i}:1${i} 2001:db8:${i}::${i}1" > > +done > > + > > +# install static routes > > +ovn-nbctl --route-table=rtb-1 lr-route-add lr1 2001:db8:2000::/64 2001:db8:3::31 > > +ovn-nbctl --route-table=rtb-2 lr-route-add lr1 2001:db8:2000::/64 2001:db8:4::41 > > + > > +# set lrp-lr1-ls{1,2} route tables > > +ovn-nbctl lrp-set-options lrp-lr1-ls1 route_table=rtb-1 > > +ovn-nbctl lrp-set-options lrp-lr1-ls2 route_table=rtb-2 > > + > > +net_add n1 > > +sim_add hv1 > > +as hv1 > > +ovs-vsctl add-br br-phys > > +ovn_attach n1 br-phys 192.168.0.1 > > + > > +for i in $(seq 1 4); do > > + ovs-vsctl -- add-port br-int hv1-vif${i} -- \ > > + set interface hv1-vif${i} external-ids:iface-id=lsp${i}1 \ > > + options:tx_pcap=hv1/vif${i}-tx.pcap \ > > + options:rxq_pcap=hv1/vif${i}-rx.pcap \ > > + ofport-request=${i} > > +done > > + > > +# wait for earlier changes to take effect > > +AT_CHECK([ovn-nbctl --timeout=3 --wait=hv sync], [0], [ignore]) > > + > > +# lsp31 should recieve packet coming from lsp11 > > +# lsp41 should recieve packet coming from lsp21 > > +for i in $(seq 1 2); do > > + di=$(( i + 2)) # dst index > > + ri=$(( 5 - i)) # reverse index > > + packet="inport==\"lsp${i}1\" && eth.src==f0:00:00:00:0${i}:1${i} && > > + eth.dst==00:00:00:01:0${i}:01 && ip6 && ip.ttl==64 && > > + ip6.src==2001:db8:${i}::${i}1 && ip6.dst==2001:db8:2000::1 && icmp6" > > + AT_CHECK([as hv1 ovs-appctl -t ovn-controller inject-pkt "$packet"]) > > + > > + # Assume all packets go to lsp${di}1. > > + exp_packet="eth.src==00:00:00:01:0${di}:01 && eth.dst==f0:00:00:00:0${di}:1${di} && ip6 && > > + ip.ttl==63 && ip6.src==2001:db8:${i}::${i}1 && ip6.dst==2001:db8:2000::1 && icmp6" > > + echo $exp_packet | ovstest test-ovn expr-to-packets >> expected_lsp${di}1 > > + > expected_lsp${ri}1 > > + > > + OVS_WAIT_UNTIL([ > > + rcv_n1=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" hv1/vif${di}-tx.pcap > lsp${di}1.packets && cat lsp${di}1.packets | wc -l` > > + rcv_n2=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" hv1/vif${ri}-tx.pcap > lsp${ri}1.packets && cat lsp${ri}1.packets | wc -l` > > + echo $rcv_n1 $rcv_n2 > > + test $rcv_n1 -eq 1 -a $rcv_n2 -eq 0]) > > + > > + for j in "${di}1" "${ri}1"; do > > + sort expected_lsp${j} > expout > > + AT_CHECK([cat lsp${j}.packets | sort], [0], [expout]) > > + done > > + > > + # cleanup tx pcap files > > + for j in "${di}1" "${ri}1"; do > > + ovs-vsctl -- remove interface hv1-vif${di} options tx_pcap > > + > hv1/vif${di}-tx.pcap > > + ovs-vsctl -- set interface hv1-vif${di} external-ids:iface-id=lsp${di}1 \ > > + options:tx_pcap=hv1/vif${di}-tx.pcap > > + done > > +done > > + > > +OVN_CLEANUP([hv1]) > > +AT_CLEANUP > > +]) > > + > > + > > OVN_FOR_EACH_NORTHD([ > > AT_SETUP([forwarding group: 3 HVs, 1 LR, 2 LS]) > > AT_KEYWORDS([forwarding-group]) > > @@ -23332,7 +23759,7 @@ ovn-sbctl dump-flows > sbflows > > AT_CAPTURE_FILE([sbflows]) > > AT_CAPTURE_FILE([offlows]) > > OVS_WAIT_UNTIL([ > > - as hv1 ovs-ofctl dump-flows br-int table=20 > offlows > > + as hv1 ovs-ofctl dump-flows br-int table=21 > offlows > > test $(grep -c "load:0x64->NXM_NX_PKT_MARK" offlows) = 1 && \ > > test $(grep -c "load:0x3->NXM_NX_PKT_MARK" offlows) = 1 && \ > > test $(grep -c "load:0x4->NXM_NX_PKT_MARK" offlows) = 1 && \ > > @@ -23425,12 +23852,12 @@ send_ipv4_pkt hv1 hv1-vif1 505400000003 00000000ff01 \ > > $(ip_to_hex 10 0 0 3) $(ip_to_hex 172 168 0 120) > > > > OVS_WAIT_UNTIL([ > > - test 1 -eq $(as hv1 ovs-ofctl dump-flows br-int table=20 | \ > > + test 1 -eq $(as hv1 ovs-ofctl dump-flows br-int table=21 | \ > > grep "load:0x2->NXM_NX_PKT_MARK" -c) > > ]) > > > > AT_CHECK([ > > - test 0 -eq $(as hv1 ovs-ofctl dump-flows br-int table=20 | \ > > + test 0 -eq $(as hv1 ovs-ofctl dump-flows br-int table=21 | \ > > grep "load:0x64->NXM_NX_PKT_MARK" -c) > > ]) > > > > @@ -24133,7 +24560,7 @@ AT_CHECK([ > > grep "priority=100" | \ > > grep -c "ct(commit,zone=NXM_NX_REG11\\[[0..15\\]],.*exec(move:NXM_OF_ETH_SRC\\[[\\]]->NXM_NX_CT_LABEL\\[[32..79\\]],load:0x[[0-9]]->NXM_NX_CT_LABEL\\[[80..95\\]]))" > > > > - grep table=22 hv${hv}flows | \ > > + grep table=23 hv${hv}flows | \ > > grep "priority=200" | \ > > grep -c "actions=move:NXM_NX_CT_LABEL\\[[32..79\\]]->NXM_OF_ETH_DST\\[[\\]]" > > done; :], [0], [dnl > > @@ -24258,7 +24685,7 @@ AT_CHECK([ > > grep "priority=100" | \ > > grep -c "ct(commit,zone=NXM_NX_REG11\\[[0..15\\]],.*exec(move:NXM_OF_ETH_SRC\\[[\\]]->NXM_NX_CT_LABEL\\[[32..79\\]],load:0x[[0-9]]->NXM_NX_CT_LABEL\\[[80..95\\]]))" > > > > - grep table=22 hv${hv}flows | \ > > + grep table=23 hv${hv}flows | \ > > grep "priority=200" | \ > > grep -c "actions=move:NXM_NX_CT_LABEL\\[[32..79\\]]->NXM_OF_ETH_DST\\[[\\]]" > > done; :], [0], [dnl > > @@ -24880,7 +25307,7 @@ AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | grep "actions=controller" | grep > > ]) > > > > # The packet should've been dropped in the lr_in_arp_resolve stage. > > -AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | grep -E "table=22, n_packets=1,.* priority=1,ip,metadata=0x${sw_key},nw_dst=10.0.1.1 actions=drop" -c], [0], [dnl > > +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | grep -E "table=23, n_packets=1,.* priority=1,ip,metadata=0x${sw_key},nw_dst=10.0.1.1 actions=drop" -c], [0], [dnl > > 1 > > ]) > > > > diff --git a/utilities/ovn-nbctl.c b/utilities/ovn-nbctl.c > > index e34bb65f7..0ff10618b 100644 > > --- a/utilities/ovn-nbctl.c > > +++ b/utilities/ovn-nbctl.c > > @@ -329,6 +329,8 @@ Logical router port commands:\n\ > > add logical port PORT on ROUTER\n\ > > lrp-set-gateway-chassis PORT CHASSIS [PRIORITY]\n\ > > set gateway chassis for port PORT\n\ > > + lrp-set-options PORT KEY=VALUE [KEY=VALUE]...\n\ > > + set router port options\n\ > > lrp-del-gateway-chassis PORT CHASSIS\n\ > > delete gateway chassis from port PORT\n\ > > lrp-get-gateway-chassis PORT\n\ > > @@ -351,11 +353,17 @@ Logical router port commands:\n\ > > ('overlay' or 'bridged')\n\ > > \n\ > > Route commands:\n\ > > - [--policy=POLICY] [--ecmp] [--ecmp-symmetric-reply] lr-route-add ROUTER \n\ > > - PREFIX NEXTHOP [PORT]\n\ > > + [--policy=POLICY]\n\ > > + [--ecmp]\n\ > > + [--ecmp-symmetric-reply]\n\ > > + [--route-table=ROUTE_TABLE]\n\ > > + lr-route-add ROUTER PREFIX NEXTHOP [PORT]\n\ > > add a route to ROUTER\n\ > > - [--policy=POLICY] lr-route-del ROUTER [PREFIX [NEXTHOP [PORT]]]\n\ > > + [--policy=POLICY]\n\ > > + [--route-table=ROUTE_TABLE]\n\ > > + lr-route-del ROUTER [PREFIX [NEXTHOP [PORT]]]\n\ > > remove routes from ROUTER\n\ > > + [--route-table=ROUTE_TABLE]\n\ > > lr-route-list ROUTER print routes for ROUTER\n\ > > \n\ > > Policy commands:\n\ > > @@ -743,6 +751,11 @@ print_lr(const struct nbrec_logical_router *lr, struct ds *s) > > ds_put_cstr(s, "]\n"); > > } > > > > + const char *route_table = smap_get(&lrp->options, "route_table"); > > + if (route_table) { > > + ds_put_format(s, " route-table: %s\n", route_table); > > + } > > + > > if (lrp->n_gateway_chassis) { > > const struct nbrec_gateway_chassis **gcs; > > > > @@ -862,6 +875,7 @@ nbctl_pre_show(struct ctl_context *ctx) > > ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_name); > > ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_mac); > > ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_networks); > > + ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_options); > > ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_gateway_chassis); > > > > ovsdb_idl_add_column(ctx->idl, &nbrec_gateway_chassis_col_chassis_name); > > @@ -4000,11 +4014,19 @@ nbctl_lr_policy_list(struct ctl_context *ctx) > > > > static struct nbrec_logical_router_static_route * > > nbctl_lr_get_route(const struct nbrec_logical_router *lr, char *prefix, > > - char *next_hop, bool is_src_route, bool ecmp) > > + char *next_hop, bool is_src_route, bool ecmp, > > + char *route_table) > > { > > for (int i = 0; i < lr->n_static_routes; i++) { > > struct nbrec_logical_router_static_route *route = lr->static_routes[i]; > > > > + /* Strict compare for route_table. > > + * If route_table was not specified, > > + * lookup for routes with empty route_table value. */ > > + if (strcmp(route->route_table, route_table ? route_table : "")) { > > + continue; > > + } > > + > > /* Compare route policy. */ > > char *nb_policy = route->policy; > > bool nb_is_src_route = false; > > @@ -4060,6 +4082,8 @@ nbctl_pre_lr_route_add(struct ctl_context *ctx) > > &nbrec_logical_router_static_route_col_bfd); > > ovsdb_idl_add_column(ctx->idl, > > &nbrec_logical_router_static_route_col_options); > > + ovsdb_idl_add_column(ctx->idl, > > + &nbrec_logical_router_static_route_col_route_table); > > } > > > > static char * OVS_WARN_UNUSED_RESULT > > @@ -4090,6 +4114,7 @@ nbctl_lr_route_add(struct ctl_context *ctx) > > } > > } > > > > + char *route_table = shash_find_data(&ctx->options, "--route-table"); > > bool v6_prefix = false; > > prefix = normalize_ipv4_prefix_str(ctx->argv[2]); > > if (!prefix) { > > @@ -4166,7 +4191,8 @@ nbctl_lr_route_add(struct ctl_context *ctx) > > bool ecmp = shash_find(&ctx->options, "--ecmp") != NULL || > > ecmp_symmetric_reply; > > struct nbrec_logical_router_static_route *route = > > - nbctl_lr_get_route(lr, prefix, next_hop, is_src_route, ecmp); > > + nbctl_lr_get_route(lr, prefix, next_hop, is_src_route, ecmp, > > + route_table); > > > > /* Validations for nexthop = "discard" */ > > if (is_discard_route) { > > @@ -4230,7 +4256,8 @@ nbctl_lr_route_add(struct ctl_context *ctx) > > } > > > > struct nbrec_logical_router_static_route *discard_route = > > - nbctl_lr_get_route(lr, prefix, "discard", is_src_route, true); > > + nbctl_lr_get_route(lr, prefix, "discard", is_src_route, true, > > + route_table); > > if (discard_route) { > > ctl_error(ctx, "discard nexthop for the same ECMP route exists."); > > goto cleanup; > > @@ -4246,6 +4273,9 @@ nbctl_lr_route_add(struct ctl_context *ctx) > > if (policy) { > > nbrec_logical_router_static_route_set_policy(route, policy); > > } > > + if (route_table) { > > + nbrec_logical_router_static_route_set_route_table(route, route_table); > > + } > > > > if (ecmp_symmetric_reply) { > > const struct smap options = SMAP_CONST1(&options, > > @@ -4289,6 +4319,8 @@ nbctl_pre_lr_route_del(struct ctl_context *ctx) > > &nbrec_logical_router_static_route_col_nexthop); > > ovsdb_idl_add_column(ctx->idl, > > &nbrec_logical_router_static_route_col_output_port); > > + ovsdb_idl_add_column(ctx->idl, > > + &nbrec_logical_router_static_route_col_route_table); > > > > } > > > > @@ -4302,6 +4334,7 @@ nbctl_lr_route_del(struct ctl_context *ctx) > > return; > > } > > > > + const char *route_table = shash_find_data(&ctx->options, "--route-table"); > > const char *policy = shash_find_data(&ctx->options, "--policy"); > > bool is_src_route = false; > > if (policy) { > > @@ -4392,6 +4425,14 @@ nbctl_lr_route_del(struct ctl_context *ctx) > > } > > } > > > > + /* Strict compare for route_table. > > + * If route_table was not specified, > > + * lookup for routes with empty route_table value. */ > > + if (strcmp(lr->static_routes[i]->route_table, > > + route_table ? route_table : "")) { > > + continue; > > + } > > + > > /* Compare output_port, if specified. */ > > if (output_port) { > > char *rt_output_port = lr->static_routes[i]->output_port; > > @@ -5115,6 +5156,41 @@ nbctl_pre_lrp_del_gateway_chassis(struct ctl_context *ctx) > > ovsdb_idl_add_column(ctx->idl, &nbrec_gateway_chassis_col_chassis_name); > > } > > > > +static void > > +nbctl_pre_lrp_options(struct ctl_context *ctx) > > +{ > > + ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_name); > > + ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_options); > > +} > > + > > +static void > > +nbctl_lrp_set_options(struct ctl_context *ctx) > > +{ > > + const char *id = ctx->argv[1]; > > + const struct nbrec_logical_router_port *lrp = NULL; > > + size_t i; > > + struct smap options = SMAP_INITIALIZER(&options); > > + > > + char *error = lrp_by_name_or_uuid(ctx, id, true, &lrp); > > + if (error) { > > + ctx->error = error; > > + return; > > + } > > + for (i = 2; i < ctx->argc; i++) { > > + char *key, *value; > > + value = xstrdup(ctx->argv[i]); > > + key = strsep(&value, "="); > > + if (value) { > > + smap_add(&options, key, value); > > + } > > + free(key); > > + } > > + > > + nbrec_logical_router_port_set_options(lrp, &options); > > + > > + smap_destroy(&options); > > +} > > + > > /* Removes logical router port 'lrp->gateway_chassis[idx]'. */ > > static void > > remove_gc(const struct nbrec_logical_router_port *lrp, size_t idx) > > @@ -5891,6 +5967,7 @@ route_cmp_details(const struct nbrec_logical_router_static_route *r1, > > } > > return r1->output_port ? 1 : -1; > > } > > + > > struct ipv4_route { > > int priority; > > ovs_be32 addr; > > @@ -5900,6 +5977,11 @@ struct ipv4_route { > > static int > > __ipv4_route_cmp(const struct ipv4_route *r1, const struct ipv4_route *r2) > > { > > + int rtb_cmp = strcmp(r1->route->route_table, > > + r2->route->route_table); > > + if (rtb_cmp) { > > + return rtb_cmp; > > + } > > if (r1->priority != r2->priority) { > > return r1->priority > r2->priority ? -1 : 1; > > } > > @@ -5931,6 +6013,11 @@ struct ipv6_route { > > static int > > __ipv6_route_cmp(const struct ipv6_route *r1, const struct ipv6_route *r2) > > { > > + int rtb_cmp = strcmp(r1->route->route_table, > > + r2->route->route_table); > > + if (rtb_cmp) { > > + return rtb_cmp; > > + } > > if (r1->priority != r2->priority) { > > return r1->priority > r2->priority ? -1 : 1; > > } > > @@ -6018,6 +6105,8 @@ nbctl_pre_lr_route_list(struct ctl_context *ctx) > > &nbrec_logical_router_static_route_col_options); > > ovsdb_idl_add_column(ctx->idl, > > &nbrec_logical_router_static_route_col_bfd); > > + ovsdb_idl_add_column(ctx->idl, > > + &nbrec_logical_router_static_route_col_route_table); > > } > > > > static void > > @@ -6035,12 +6124,17 @@ nbctl_lr_route_list(struct ctl_context *ctx) > > return; > > } > > > > + char *route_table = shash_find_data(&ctx->options, "--route-table"); > > + > > ipv4_routes = xmalloc(sizeof *ipv4_routes * lr->n_static_routes); > > ipv6_routes = xmalloc(sizeof *ipv6_routes * lr->n_static_routes); > > > > for (int i = 0; i < lr->n_static_routes; i++) { > > const struct nbrec_logical_router_static_route *route > > = lr->static_routes[i]; > > + if (route_table && strcmp(route->route_table, route_table)) { > > + continue; > > + } > > unsigned int plen; > > ovs_be32 ipv4; > > const char *policy = route->policy ? route->policy : "dst-ip"; > > @@ -6081,6 +6175,7 @@ nbctl_lr_route_list(struct ctl_context *ctx) > > if (n_ipv4_routes) { > > ds_put_cstr(&ctx->output, "IPv4 Routes\n"); > > } > > + const struct nbrec_logical_router_static_route *route; > > for (int i = 0; i < n_ipv4_routes; i++) { > > bool ecmp = false; > > if (i < n_ipv4_routes - 1 && > > @@ -6091,6 +6186,15 @@ nbctl_lr_route_list(struct ctl_context *ctx) > > &ipv4_routes[i - 1])) { > > ecmp = true; > > } > > + > > + route = ipv4_routes[i].route; > > + if (!i || (i > 0 && strcmp(route->route_table, > > + ipv4_routes[i - 1].route->route_table))) { > > + ds_put_format(&ctx->output, "%sRoute Table %s:\n", i ? "\n" : "", > > + strlen(route->route_table) ? route->route_table > > + : "global"); > > + } > > + > > print_route(ipv4_routes[i].route, &ctx->output, ecmp); > > } > > > > @@ -6108,6 +6212,15 @@ nbctl_lr_route_list(struct ctl_context *ctx) > > &ipv6_routes[i - 1])) { > > ecmp = true; > > } > > + > > + route = ipv6_routes[i].route; > > + if (!i || (i > 0 && strcmp(route->route_table, > > + ipv6_routes[i - 1].route->route_table))) { > > + ds_put_format(&ctx->output, "%sRoute Table %s:\n", i ? "\n" : "", > > + strlen(route->route_table) ? route->route_table > > + : "global"); > > + } > > + > > print_route(ipv6_routes[i].route, &ctx->output, ecmp); > > } > > > > @@ -6926,6 +7039,8 @@ static const struct ctl_command_syntax nbctl_commands[] = { > > "PORT CHASSIS [PRIORITY]", > > nbctl_pre_lrp_set_gateway_chassis, nbctl_lrp_set_gateway_chassis, > > NULL, "--may-exist", RW }, > > + { "lrp-set-options", 1, INT_MAX, "PORT KEY=VALUE [KEY=VALUE]...", > > + nbctl_pre_lrp_options, nbctl_lrp_set_options, NULL, "", RW }, > > { "lrp-del-gateway-chassis", 2, 2, "PORT CHASSIS", > > nbctl_pre_lrp_del_gateway_chassis, nbctl_lrp_del_gateway_chassis, > > NULL, "", RW }, > > @@ -6949,12 +7064,13 @@ static const struct ctl_command_syntax nbctl_commands[] = { > > /* logical router route commands. */ > > { "lr-route-add", 3, 4, "ROUTER PREFIX NEXTHOP [PORT]", > > nbctl_pre_lr_route_add, nbctl_lr_route_add, NULL, > > - "--may-exist,--ecmp,--ecmp-symmetric-reply,--policy=,--bfd?", RW }, > > + "--may-exist,--ecmp,--ecmp-symmetric-reply,--policy=,--route-table=,--bfd?", > > + RW }, > > { "lr-route-del", 1, 4, "ROUTER [PREFIX [NEXTHOP [PORT]]]", > > nbctl_pre_lr_route_del, nbctl_lr_route_del, NULL, > > - "--if-exists,--policy=", RW }, > > + "--if-exists,--policy=,--route-table=", RW }, > > { "lr-route-list", 1, 1, "ROUTER", nbctl_pre_lr_route_list, > > - nbctl_lr_route_list, NULL, "", RO }, > > + nbctl_lr_route_list, NULL, "--route-table=", RO }, > > > > /* Policy commands */ > > { "lr-policy-add", 4, INT_MAX, > > -- > > 2.30.0 > > > > _______________________________________________ > > dev mailing list > > dev@openvswitch.org > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > >
Regards, Vladislav Odintsov > On 15 Oct 2021, at 08:42, Han Zhou <hzhou@ovn.org> wrote: > > On Thu, Oct 14, 2021 at 12:58 AM Vladislav Odintsov <odivlad@gmail.com> > wrote: >> >> Hi Han, >> >> Thanks for the review. >> >> Regards, >> Vladislav Odintsov >> >> On 14 Oct 2021, at 08:13, Han Zhou <hzhou@ovn.org> wrote: >> >> >> >> On Tue, Oct 5, 2021 at 1:26 PM Vladislav Odintsov <odivlad@gmail.com> > wrote: >>>> >>>> This patch extends Logical Router's routing functionality. >>>> Now user may create multiple routing tables within a Logical Router >>>> and assign them to Logical Router Ports. >>>> >>>> Traffic coming from Logical Router Port with assigned route_table >>>> is checked against global routes if any (Logical_Router_Static_Routes >>>> whith empty route_table field), next against directly connected routes >>> >>> This is not accurate. The "directly connected routes" is NOT after the >>> global routes. Their priority only depends on the prefix length. >> >>>> and then Logical_Router_Static_Routes with same route_table value as >>>> in Logical_Router_Port options:route_table field. >>>> >>>> A new Logical Router ingress table #10 is added - IN_IP_ROUTING_PRE. >>>> In this table packets which come from LRPs with configured >>>> options:route_table field are checked against inport and in OVS >>>> register 7 unique non-zero value identifying route table is written. >>>> >>>> Then in 11th table IN_IP_ROUTING routes which have non-empty >>>> `route_table` field are added with additional match on reg7 value >>>> associated with appropriate route_table. >>>> >>> >>> Hi Vladislav, >>> >>> First of all, sorry for the delayed review, and thanks for implementing >>> this new feature. >>> >>> I have some questions regarding the feature itself. I remember that we >>> had some discussion earlier for this feature, but it seems I misunderstood >>> the feature you are implementing here. I thought by multiple routing tables >>> you were trying to support something like VRF in physical routers, but this >>> seems to be something different. According to your implementation, instead >>> of assigning LRPs to different routing tables, a LRP can actually >>> participate to both the global routing table and the table with a specific >>> ID. For ingress, global routes are prefered over other routes; for egress >>> (i.e. forwarding to the next hop), it doesn't even enforce any table-id >>> check, so packet routed by a entry of table-X can go out of a LRP with >>> table-id Y. Is my understanding correct about the desired behavior of this >>> feature? >>> >>> >> Yes, your understanding is correct. >> This is not VRF. At first glance VRF can be done just by using LR-per-VRF >> without any code modifications (maybe there are corner cases, but it’s not >> something I’m currently working on). >> > I agree VRF can be achieved by just separate LRs. I am trying to understand > why multiple LRs wouldn't satisfy your use case here. > >> LRP can be optionally assigned to specific routing table name. This means >> that for LR ingress pipeline packets coming from this specific LRP would be >> checked against routes with same route_table value within appropriate LR. >> This is some kind of PBR, analog of "ip rule add iif <interface name> >> lookup <id>". >> There is one specific use-case, which requires special handling: >> directly-connected routes (subnet CIDRs from connected to this LR LRPs). >> These routes can’t be added manually by user, though routing between LRPs >> belonging to different routing tables is still needed. So, these routes >> should be added to global routing table to override routing for LRPs with >> configured routing tables. If for some reason user wants to prohibit IP >> connectivity to any LRP (honestly, I can’t imagine, why), it can be done by >> utilising OVN PBR with drop action or a route with "discard" nexthop. But >> I’d think in this case whether this LRP is needed in this LR. >> Also, if user wants to create routes, which apply for all LRPs from all >> routing tables, it can be done by adding new entries to global routing >> table. >> And the third reason to have global routing table - a fully >> backward-compatible solution. All users, who don’t need multiple routing >> tables support just continue using static routing in the same manner >> without any changes. >> >> >>> If this is correct, it doesn't seem to be a common/standard requirement >>> (or please educate me if I am wrong). Could you explain a little more about >>> the actual use cases: what kind of real world problems need to be solved by >>> this feature or how are you going to use this. For example, why would a >>> port need to participate in both routing tables? It looks like what you >>> really need is policy routing instead of multiple isolated routing tables. >>> I understand that you already use policy routing to implement ACLs, so it >>> is not convenient to combine other policies (e.g. inport based routing) >>> into the policy routing stage. If that's the case, would it be more generic >>> to support multiple policy routing stages? My concern to the current >>> approach is that it is implemented for a very special use case. It makes >>> the code more complex but when there is a slightly different requirement in >>> the future it becomes insufficient. I am thinking that policy routing seems >>> more flexible and has more potential to be made more generic. Maybe I will >>> have a better understanding when I hear more detailed use cases and >>> considerations from you. >> >> >> I can't agree here in it’s uncommon requirement. >> This implementation was inspired by AWS Route Tables feature [1]: within >> a VPC (LR in terms of OVN) user may create multiple routing tables and >> assign them to different subnets (LRPs) in multiple availability zones. >> Auto-generated directly-connected routes from LRPs CIDRs are working in the >> same manner as they do in AWS - apply to all subnets regardless of their >> association to routing table. GCP has similar behaviour: [2], Azure, I >> guess, too. >> Our public cloud (CROC Cloud Platform) supports AWS behaviour [3], so I >> primarily was oriented on it. Internally we already use this feature and >> currently it fits our use-case, but again I can't say it is specific. >> > If it is for AWS/GCP alike routing features, then from what I understand > what's implemented in this patch is a little different. To implement a VPC > model in OVN, I think it is ok to have multiple LRs instead of only one LR > for each VPC. So naturally I would use a LR to map to AWS's routing table. > In AWS's document (the link you provided), it says: > > "A subnet can only be associated with one route table at a time" > > So basically in AWS a subnet is either attached to the default/main route > table or a custom table, but not both, right? However, in your use case, a > LRP (maps to a subnet) attaches to both "main" and a custom table, which > seems not common to me. Or did I miss something? That’s true about AWS, but there is still a bit not accurate about OVN. Global routing table in OVN terms is not that AWS main route table is. Main route table is just a configuration hint for users for implicit route tables association with subnets. Implicitly-associated via main routing table subnets routing functions the same manner as a normal explicit route_table-subnet association. Global routing table in OVN is just a list of routes with higher priority than routes with configured "route_table". I do not offer to configure both tables at the same time. But it is _possible_ to do if required for some reason (for instance to configure some service chaining or just internal VPC services like metadata/another internal APIs, access to another services). Normally, if we talk about AWS Route Table to OVN, it is mostly one-to-one mapping, except "Local route": Example: AWS Route Table rtb-xxx: 172.31.0.0/16: local route (VPC CIDR for subnets) 0.0.0.0/0: igw-XXX (internet gateway) AWS Route Table rtb-yyy: 172.31.0.0/16: local route (VPC CIDR for subnets) 5.5.5.5/32: instance-xxx This maps to OVN configuration (for one LR with one subnet 172.31.0.0/24): implicit route (not present in logical router static routes table): 172.31.0.0/24: LRP-subnet-xxx - has highest priority over other route table-oriented routes and can be threat as placed in global routing table Normal static routes: ip_prefix: 0.0.0.0/0, nexthop: <IP for edge LR’s LRP>, route_table: rtb-xxx ip_prefix: 5.5.5.5/32, nexthop: <IP of some LSP, which belongs to VM/container via which route is built>, route_table: rtb-yyy I guess, I understood the reason for misunderstanding: the global routing table, which I referred earlier is a routing table which has no value in "route_table" field and directly-connected routes at the same time. Latter have no records in logical router static routes, but I still referred them as a routes from "global routing table". I can think about terminology here, if it’s a root cause for misunderstanding. What do you think? > >> >> In my opinion having this feature to be implemented using PBR is less >> convenient and native for users, who are familiar with behaviour for >> mentioned above public cloud platforms, because configuring routes should >> be done in routes section. And adding route table property seems native in >> this route, not in PBR. Moreover, I _think_ using >> Logical_Router_Static_Route to extend this feature for OVN-Interconnection >> becomes quite easy comparing to PBR (though, I didn’t try the latter). >> > I agree if it is just AWS -like requirement, PBR is less convenient. > > I am trying to understand if it can be achieved with separate LRs. If not, > what's special about the requirement, and is the current approach providing > a solution common enough so that more use cases can also benefit from? > Could you clarify a little more? Thanks again. > That was our initial approach - to use separate LRs for each Route Table. We rejected that solution because met some difficulties and blocker. See below: Brief topology description if using LRs per Route Table: Imagine 2 subnets in VPC in 1st AZ, one in another AZ. Each subnet in it’s own Route Table (LR). All subnets must have IP connectivity, so we have to somehow interconnect these Route Table LRs. [BLOCKER] It is impossible to place route in route table 1 via VM from subnet assiciated to route table 2 if using per-RTB LR. Because in RTB-2 LR we have to add route from RTB 1 and this breaks route table isolation. Route Table 2 LR will start looking up routes and there could be routes from another route tables. This breaks the idea of having LR per Route Table completely. Here we rejected this solution and moved to adding support for route tables in OVN. But some more cons: 1. complex network topology. Interconnecting all LRs even with some transit switch is harder than having one LR and all VPC-related configuration is done in one place (in AZ). 2. Directly-connected routes. In case we have multiple Route Table LRs, we have to add route for each subnet from another LR. In case one LR per VPC all such routes are installed automatically and learnt via ovn-ic. 3. PBR, LBs. It’s much easier to implement PBR and configuring Load Balancers in one LR, than in multiple. 4. There could be very many LRs, LRPs, LSPs (datapaths in SB) - excess database records, huge database growth. 5. Extra client code complexity (updating routes required configuring routes in many LRs on different availibility zones); 5. We couldn’t use ovn-ic routes learning, because VPC requires out-of-box IP connectivity between subnets, and if connect Route Table LRs between AZs, because connecting multiple LRs from different Route Tables would learn routes without binding to route table. This requires additional isolation at the transit switches level. 6. There were some more problems. Here are listed some, which I could refresh in my mind. From our half-of-a-year experience using LR per VPC is very comfortable and it looks quite extendable it terms of network features. Let me know if this makes sense. > Han > >> >> >>> I haven't finished reviewing the code yet, but I have one comment >>> regarding adding 100 to the priority of the global routes. For IPv6, the >>> priority range from 0 to 120x2=240, so adding 100 is not enough. It would >>> create overlapping priority ranges, and some table-id specific route >>> entries may override the global routes. >> >> >> Thanks, I’ll dig into this when you finish review. >> >> >> Let me know if I answered your questions or if you have new ones. >> Again many thanks for your time and digging into this patch series. >> >> 1: https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Route_Tables.html >> 2: https://cloud.google.com/vpc/docs/routes#subnet-routes >> 3: https://docs.cloud.croc.ru/en/services/networks/routetables.html >> >> Thanks, >> Han >>> >>>> Signed-off-by: Vladislav Odintsov <odivlad@gmail.com> >>>> Acked-by: Numan Siddique <numans@ovn.org> >>>> --- >>>> northd/northd.c | 159 ++++++++++++--- >>>> northd/ovn-northd.8.xml | 63 ++++-- >>>> ovn-nb.ovsschema | 5 +- >>>> ovn-nb.xml | 30 +++ >>>> tests/ovn-ic.at | 4 + >>>> tests/ovn-nbctl.at | 196 +++++++++++++++++- >>>> tests/ovn-northd.at | 76 ++++++- >>>> tests/ovn.at | 441 +++++++++++++++++++++++++++++++++++++++- >>>> utilities/ovn-nbctl.c | 134 +++++++++++- >>>> 9 files changed, 1041 insertions(+), 67 deletions(-) >>>> >>>> diff --git a/northd/northd.c b/northd/northd.c >>>> index 092eca829..6a020cb2e 100644 >>>> --- a/northd/northd.c >>>> +++ b/northd/northd.c >>>> @@ -148,15 +148,16 @@ enum ovn_stage { >>>> PIPELINE_STAGE(ROUTER, IN, ECMP_STATEFUL, 7, >> "lr_in_ecmp_stateful") \ >>>> PIPELINE_STAGE(ROUTER, IN, ND_RA_OPTIONS, 8, >> "lr_in_nd_ra_options") \ >>>> PIPELINE_STAGE(ROUTER, IN, ND_RA_RESPONSE, 9, >> "lr_in_nd_ra_response") \ >>>> - PIPELINE_STAGE(ROUTER, IN, IP_ROUTING, 10, >> "lr_in_ip_routing") \ >>>> - PIPELINE_STAGE(ROUTER, IN, IP_ROUTING_ECMP, 11, >> "lr_in_ip_routing_ecmp") \ >>>> - PIPELINE_STAGE(ROUTER, IN, POLICY, 12, "lr_in_policy") >> \ >>>> - PIPELINE_STAGE(ROUTER, IN, POLICY_ECMP, 13, >> "lr_in_policy_ecmp") \ >>>> - PIPELINE_STAGE(ROUTER, IN, ARP_RESOLVE, 14, >> "lr_in_arp_resolve") \ >>>> - PIPELINE_STAGE(ROUTER, IN, CHK_PKT_LEN , 15, >> "lr_in_chk_pkt_len") \ >>>> - PIPELINE_STAGE(ROUTER, IN, LARGER_PKTS, 16, >> "lr_in_larger_pkts") \ >>>> - PIPELINE_STAGE(ROUTER, IN, GW_REDIRECT, 17, >> "lr_in_gw_redirect") \ >>>> - PIPELINE_STAGE(ROUTER, IN, ARP_REQUEST, 18, >> "lr_in_arp_request") \ >>>> + PIPELINE_STAGE(ROUTER, IN, IP_ROUTING_PRE, 10, >> "lr_in_ip_routing_pre") \ >>>> + PIPELINE_STAGE(ROUTER, IN, IP_ROUTING, 11, >> "lr_in_ip_routing") \ >>>> + PIPELINE_STAGE(ROUTER, IN, IP_ROUTING_ECMP, 12, >> "lr_in_ip_routing_ecmp") \ >>>> + PIPELINE_STAGE(ROUTER, IN, POLICY, 13, "lr_in_policy") >> \ >>>> + PIPELINE_STAGE(ROUTER, IN, POLICY_ECMP, 14, >> "lr_in_policy_ecmp") \ >>>> + PIPELINE_STAGE(ROUTER, IN, ARP_RESOLVE, 15, >> "lr_in_arp_resolve") \ >>>> + PIPELINE_STAGE(ROUTER, IN, CHK_PKT_LEN, 16, >> "lr_in_chk_pkt_len") \ >>>> + PIPELINE_STAGE(ROUTER, IN, LARGER_PKTS, 17, >> "lr_in_larger_pkts") \ >>>> + PIPELINE_STAGE(ROUTER, IN, GW_REDIRECT, 18, >> "lr_in_gw_redirect") \ >>>> + PIPELINE_STAGE(ROUTER, IN, ARP_REQUEST, 19, >> "lr_in_arp_request") \ >>>> \ >>>> /* Logical router egress stages. */ \ >>>> PIPELINE_STAGE(ROUTER, OUT, UNDNAT, 0, "lr_out_undnat") >> \ >>>> @@ -225,6 +226,7 @@ enum ovn_stage { >>>> #define REG_NEXT_HOP_IPV6 "xxreg0" >>>> #define REG_SRC_IPV4 "reg1" >>>> #define REG_SRC_IPV6 "xxreg1" >>>> +#define REG_ROUTE_TABLE_ID "reg7" >>>> >>>> #define REG_ORIG_TP_DPORT_ROUTER "reg9[16..31]" >>>> >>>> @@ -287,8 +289,9 @@ enum ovn_stage { >>>> * | R6 | UNUSED | X | | G | >> IN_IP_ROUTING)| >>>> * | | | R | | 1 | >> | >>>> * +-----+--------------------------+ E | UNUSED | | >> | >>>> - * | R7 | UNUSED | G | | | >> | >>>> - * | | | 3 | | | >> | >>>> + * | R7 | ROUTE_TABLE_ID | G | | | >> | >>>> + * | | (>= IN_IP_ROUTING_PRE && | 3 | | | >> | >>>> + * | | <= IN_IP_ROUTING) | | | | >> | >>>> * >> +-----+--------------------------+---+-----------------+---+---------------+ >>>> * | R8 | ECMP_GROUP_ID | | | >>>> * | | ECMP_MEMBER_ID | X | | >>>> @@ -8511,11 +8514,72 @@ cleanup: >>>> ds_destroy(&actions); >>>> } >>>> >>>> +static uint32_t >>>> +route_table_add(struct simap *route_tables, const char >> *route_table_name) >>>> +{ >>>> + /* route table ids start from 1 */ >>>> + uint32_t rtb_id = simap_count(route_tables) + 1; >>>> + >>>> + if (rtb_id == UINT16_MAX) { >>>> + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1); >>>> + VLOG_WARN_RL(&rl, "too many route tables for Logical Router."); >>>> + return 0; >>>> + } >>>> + >>>> + if (!simap_put(route_tables, route_table_name, rtb_id)) { >>>> + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1); >>>> + VLOG_WARN_RL(&rl, "Route table id unexpectedly appeared"); >>>> + } >>>> + >>>> + return rtb_id; >>>> +} >>>> + >>>> +static uint32_t >>>> +get_route_table_id(struct simap *route_tables, const char >> *route_table_name) >>>> +{ >>>> + if (!route_table_name || !strlen(route_table_name)) { >>>> + return 0; >>>> + } >>>> + >>>> + uint32_t rtb_id = simap_get(route_tables, route_table_name); >>>> + if (!rtb_id) { >>>> + rtb_id = route_table_add(route_tables, route_table_name); >>>> + } >>>> + >>>> + return rtb_id; >>>> +} >>>> + >>>> +static void >>>> +build_route_table_lflow(struct ovn_datapath *od, struct hmap *lflows, >>>> + struct nbrec_logical_router_port *lrp, >>>> + struct simap *route_tables) >>>> +{ >>>> + struct ds match = DS_EMPTY_INITIALIZER; >>>> + struct ds actions = DS_EMPTY_INITIALIZER; >>>> + >>>> + const char *route_table_name = smap_get(&lrp->options, >> "route_table"); >>>> + uint32_t rtb_id = get_route_table_id(route_tables, >> route_table_name); >>>> + if (!rtb_id) { >>>> + return; >>>> + } >>>> + >>>> + ds_put_format(&match, "inport == \"%s\"", lrp->name); >>>> + ds_put_format(&actions, "%s = %d; next;", >>>> + REG_ROUTE_TABLE_ID, rtb_id); >>>> + >>>> + ovn_lflow_add(lflows, od, S_ROUTER_IN_IP_ROUTING_PRE, 100, >>>> + ds_cstr(&match), ds_cstr(&actions)); >>>> + >>>> + ds_destroy(&match); >>>> + ds_destroy(&actions); >>>> +} >>>> + >>>> struct parsed_route { >>>> struct ovs_list list_node; >>>> struct in6_addr prefix; >>>> unsigned int plen; >>>> bool is_src_route; >>>> + uint32_t route_table_id; >>>> uint32_t hash; >>>> const struct nbrec_logical_router_static_route *route; >>>> bool ecmp_symmetric_reply; >>>> @@ -8540,7 +8604,7 @@ find_static_route_outport(struct ovn_datapath >> *od, struct hmap *ports, >>>> * Otherwise return NULL. */ >>>> static struct parsed_route * >>>> parsed_routes_add(struct ovn_datapath *od, struct hmap *ports, >>>> - struct ovs_list *routes, >>>> + struct ovs_list *routes, struct simap *route_tables, >>>> const struct nbrec_logical_router_static_route >> *route, >>>> struct hmap *bfd_connections) >>>> { >>>> @@ -8622,6 +8686,7 @@ parsed_routes_add(struct ovn_datapath *od, struct >> hmap *ports, >>>> struct parsed_route *pr = xzalloc(sizeof *pr); >>>> pr->prefix = prefix; >>>> pr->plen = plen; >>>> + pr->route_table_id = get_route_table_id(route_tables, >> route->route_table); >>>> pr->is_src_route = (route->policy && !strcmp(route->policy, >>>> "src-ip")); >>>> pr->hash = route_hash(pr); >>>> @@ -8655,6 +8720,7 @@ struct ecmp_groups_node { >>>> struct in6_addr prefix; >>>> unsigned int plen; >>>> bool is_src_route; >>>> + uint32_t route_table_id; >>>> uint16_t route_count; >>>> struct ovs_list route_list; /* Contains ecmp_route_list_node */ >>>> }; >>>> @@ -8663,7 +8729,7 @@ static void >>>> ecmp_groups_add_route(struct ecmp_groups_node *group, >>>> const struct parsed_route *route) >>>> { >>>> - if (group->route_count == UINT16_MAX) { >>>> + if (group->route_count == UINT16_MAX) { >>>> static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1); >>>> VLOG_WARN_RL(&rl, "too many routes in a single ecmp group."); >>>> return; >>>> @@ -8692,6 +8758,7 @@ ecmp_groups_add(struct hmap *ecmp_groups, >>>> eg->prefix = route->prefix; >>>> eg->plen = route->plen; >>>> eg->is_src_route = route->is_src_route; >>>> + eg->route_table_id = route->route_table_id; >>>> ovs_list_init(&eg->route_list); >>>> ecmp_groups_add_route(eg, route); >>>> >>>> @@ -8705,7 +8772,8 @@ ecmp_groups_find(struct hmap *ecmp_groups, struct >> parsed_route *route) >>>> HMAP_FOR_EACH_WITH_HASH (eg, hmap_node, route->hash, ecmp_groups) { >>>> if (ipv6_addr_equals(&eg->prefix, &route->prefix) && >>>> eg->plen == route->plen && >>>> - eg->is_src_route == route->is_src_route) { >>>> + eg->is_src_route == route->is_src_route && >>>> + eg->route_table_id == route->route_table_id) { >>>> return eg; >>>> } >>>> } >>>> @@ -8752,7 +8820,8 @@ unique_routes_remove(struct hmap *unique_routes, >>>> HMAP_FOR_EACH_WITH_HASH (ur, hmap_node, route->hash, >> unique_routes) { >>>> if (ipv6_addr_equals(&route->prefix, &ur->route->prefix) && >>>> route->plen == ur->route->plen && >>>> - route->is_src_route == ur->route->is_src_route) { >>>> + route->is_src_route == ur->route->is_src_route && >>>> + route->route_table_id == ur->route->route_table_id) { >>>> hmap_remove(unique_routes, &ur->hmap_node); >>>> const struct parsed_route *existed_route = ur->route; >>>> free(ur); >>>> @@ -8790,9 +8859,9 @@ build_route_prefix_s(const struct in6_addr >> *prefix, unsigned int plen) >>>> } >>>> >>>> static void >>>> -build_route_match(const struct ovn_port *op_inport, const char >> *network_s, >>>> - int plen, bool is_src_route, bool is_ipv4, struct ds >> *match, >>>> - uint16_t *priority) >>>> +build_route_match(const struct ovn_port *op_inport, uint32_t rtb_id, >>>> + const char *network_s, int plen, bool is_src_route, >>>> + bool is_ipv4, struct ds *match, uint16_t *priority) >>>> { >>>> const char *dir; >>>> /* The priority here is calculated to implement >> longest-prefix-match >>>> @@ -8808,6 +8877,15 @@ build_route_match(const struct ovn_port >> *op_inport, const char *network_s, >>>> if (op_inport) { >>>> ds_put_format(match, "inport == %s && ", op_inport->json_key); >>>> } >>>> + if (rtb_id) { >>>> + ds_put_format(match, "%s == %d && ", REG_ROUTE_TABLE_ID, >> rtb_id); >>>> + } else { >>>> + /* Route-table assigned LRPs' routes should have lower priority >>>> + * in order not to affect directly-connected global routes. >>>> + * So, enlarge non-route-table routes priority by 100. >>>> + */ >>>> + *priority += 100; >>>> + } >>>> ds_put_format(match, "ip%s.%s == %s/%d", is_ipv4 ? "4" : "6", dir, >>>> network_s, plen); >>>> } >>>> @@ -8946,7 +9024,7 @@ add_ecmp_symmetric_reply_flows(struct hmap >> *lflows, >>>> out_port->lrp_networks.ea_s, >>>> IN6_IS_ADDR_V4MAPPED(&route->prefix) ? "" : "xx", >>>> port_ip, out_port->json_key); >>>> - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_IP_ROUTING, 300, >>>> + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_IP_ROUTING, 400, >>>> ds_cstr(&match), ds_cstr(&actions), >>>> &st_route->header_); >>>> >>>> @@ -8976,8 +9054,8 @@ build_ecmp_route_flow(struct hmap *lflows, struct >> ovn_datapath *od, >>>> struct ds route_match = DS_EMPTY_INITIALIZER; >>>> >>>> char *prefix_s = build_route_prefix_s(&eg->prefix, eg->plen); >>>> - build_route_match(NULL, prefix_s, eg->plen, eg->is_src_route, >> is_ipv4, >>>> - &route_match, &priority); >>>> + build_route_match(NULL, eg->route_table_id, prefix_s, eg->plen, >>>> + eg->is_src_route, is_ipv4, &route_match, >> &priority); >>>> free(prefix_s); >>>> >>>> struct ds actions = DS_EMPTY_INITIALIZER; >>>> @@ -9052,8 +9130,8 @@ static void >>>> add_route(struct hmap *lflows, struct ovn_datapath *od, >>>> const struct ovn_port *op, const char *lrp_addr_s, >>>> const char *network_s, int plen, const char *gateway, >>>> - bool is_src_route, const struct ovsdb_idl_row *stage_hint, >>>> - bool is_discard_route) >>>> + bool is_src_route, const uint32_t rtb_id, >>>> + const struct ovsdb_idl_row *stage_hint, bool >> is_discard_route) >>>> { >>>> bool is_ipv4 = strchr(network_s, '.') ? true : false; >>>> struct ds match = DS_EMPTY_INITIALIZER; >>>> @@ -9068,8 +9146,8 @@ add_route(struct hmap *lflows, struct >> ovn_datapath *od, >>>> op_inport = op; >>>> } >>>> } >>>> - build_route_match(op_inport, network_s, plen, is_src_route, >> is_ipv4, >>>> - &match, &priority); >>>> + build_route_match(op_inport, rtb_id, network_s, plen, is_src_route, >>>> + is_ipv4, &match, &priority); >>>> >>>> struct ds common_actions = DS_EMPTY_INITIALIZER; >>>> struct ds actions = DS_EMPTY_INITIALIZER; >>>> @@ -9132,7 +9210,8 @@ build_static_route_flow(struct hmap *lflows, >> struct ovn_datapath *od, >>>> char *prefix_s = build_route_prefix_s(&route_->prefix, >> route_->plen); >>>> add_route(lflows, route_->is_discard_route ? od : out_port->od, >> out_port, >>>> lrp_addr_s, prefix_s, route_->plen, route->nexthop, >>>> - route_->is_src_route, &route->header_, >> route_->is_discard_route); >>>> + route_->is_src_route, route_->route_table_id, >> &route->header_, >>>> + route_->is_discard_route); >>>> >>>> free(prefix_s); >>>> } >>>> @@ -10584,6 +10663,17 @@ build_ND_RA_flows_for_lrouter(struct >> ovn_datapath *od, struct hmap *lflows) >>>> } >>>> } >>>> >>>> +/* Logical router ingress table IP_ROUTING_PRE: >>>> + * by default goto next. (priority 0). */ >>>> +static void >>>> +build_ip_routing_pre_flows_for_lrouter(struct ovn_datapath *od, >>>> + struct hmap *lflows) >>>> +{ >>>> + if (od->nbr) { >>>> + ovn_lflow_add(lflows, od, S_ROUTER_IN_IP_ROUTING_PRE, 0, "1", >> "next;"); >>>> + } >>>> +} >>>> + >>>> /* Logical router ingress table IP_ROUTING : IP Routing. >>>> * >>>> * A packet that arrives at this table is an IP packet that should be >>>> @@ -10609,14 +10699,14 @@ build_ip_routing_flows_for_lrouter_port( >>>> for (int i = 0; i < op->lrp_networks.n_ipv4_addrs; i++) { >>>> add_route(lflows, op->od, op, >> op->lrp_networks.ipv4_addrs[i].addr_s, >>>> op->lrp_networks.ipv4_addrs[i].network_s, >>>> - op->lrp_networks.ipv4_addrs[i].plen, NULL, false, >>>> + op->lrp_networks.ipv4_addrs[i].plen, NULL, >> false, 0, >>>> &op->nbrp->header_, false); >>>> } >>>> >>>> for (int i = 0; i < op->lrp_networks.n_ipv6_addrs; i++) { >>>> add_route(lflows, op->od, op, >> op->lrp_networks.ipv6_addrs[i].addr_s, >>>> op->lrp_networks.ipv6_addrs[i].network_s, >>>> - op->lrp_networks.ipv6_addrs[i].plen, NULL, false, >>>> + op->lrp_networks.ipv6_addrs[i].plen, NULL, >> false, 0, >>>> &op->nbrp->header_, false); >>>> } >>>> } else if (lsp_is_router(op->nbsp)) { >>>> @@ -10639,7 +10729,7 @@ build_ip_routing_flows_for_lrouter_port( >>>> add_route(lflows, peer->od, peer, >>>> peer->lrp_networks.ipv4_addrs[0].addr_s, >>>> laddrs->ipv4_addrs[k].network_s, >>>> - laddrs->ipv4_addrs[k].plen, NULL, false, >>>> + laddrs->ipv4_addrs[k].plen, NULL, false, >> 0, >>>> &peer->nbrp->header_, false); >>>> } >>>> } >>>> @@ -10659,10 +10749,17 @@ build_static_route_flows_for_lrouter( >>>> struct hmap ecmp_groups = HMAP_INITIALIZER(&ecmp_groups); >>>> struct hmap unique_routes = HMAP_INITIALIZER(&unique_routes); >>>> struct ovs_list parsed_routes = >> OVS_LIST_INITIALIZER(&parsed_routes); >>>> + struct simap route_tables = SIMAP_INITIALIZER(&route_tables); >>>> struct ecmp_groups_node *group; >>>> + >>>> + for (int i = 0; i < od->nbr->n_ports; i++) { >>>> + build_route_table_lflow(od, lflows, od->nbr->ports[i], >>>> + &route_tables); >>>> + } >>>> + >>>> for (int i = 0; i < od->nbr->n_static_routes; i++) { >>>> struct parsed_route *route = >>>> - parsed_routes_add(od, ports, &parsed_routes, >>>> + parsed_routes_add(od, ports, &parsed_routes, >> &route_tables, >>>> od->nbr->static_routes[i], >> bfd_connections); >>>> if (!route) { >>>> continue; >>>> @@ -10695,6 +10792,7 @@ build_static_route_flows_for_lrouter( >>>> ecmp_groups_destroy(&ecmp_groups); >>>> unique_routes_destroy(&unique_routes); >>>> parsed_routes_destroy(&parsed_routes); >>>> + simap_destroy(&route_tables); >>>> } >>>> } >>>> >>>> @@ -12800,6 +12898,7 @@ build_lswitch_and_lrouter_iterate_by_od(struct >> ovn_datapath *od, >>>> build_neigh_learning_flows_for_lrouter(od, lsi->lflows, >> &lsi->match, >>>> &lsi->actions, >> lsi->meter_groups); >>>> build_ND_RA_flows_for_lrouter(od, lsi->lflows); >>>> + build_ip_routing_pre_flows_for_lrouter(od, lsi->lflows); >>>> build_static_route_flows_for_lrouter(od, lsi->lflows, lsi->ports, >>>> lsi->bfd_connections); >>>> build_mcast_lookup_flows_for_lrouter(od, lsi->lflows, &lsi->match, >>>> diff --git a/northd/ovn-northd.8.xml b/northd/ovn-northd.8.xml >>>> index 39f4eaa0c..cc2e25367 100644 >>>> --- a/northd/ovn-northd.8.xml >>>> +++ b/northd/ovn-northd.8.xml >>>> @@ -2899,7 +2899,7 @@ icmp6 { >>>> >>>> <p> >>>> If ECMP routes with symmetric reply are configured in the >>>> - <code>OVN_Northbound</code> database for a gateway router, a >> priority-300 >>>> + <code>OVN_Northbound</code> database for a gateway router, a >> priority-400 >>>> flow is added for each router port on which symmetric replies are >>>> configured. The matching logic for these ports essentially >> reverses the >>>> configured logic of the ECMP route. So for instance, a route >> with a >>>> @@ -3245,7 +3245,35 @@ output; >>>> </li> >>>> </ul> >>>> >>>> - <h3>Ingress Table 10: IP Routing</h3> >>>> + <h3>Ingress Table 10: IP Routing Pre</h3> >>>> + >>>> + <p> >>>> + If a packet arrived at this table from Logical Router Port >> <var>P</var> >>>> + which has <code>options:route_table</code> value set, a logical >> flow with >>>> + match <code>inport == "<var>P</var>"</code> with priority 100 >> and action, >>>> + setting unique-generated per-datapath 32-bit value (non-zero) in >> OVS >>>> + register 7. This register is checked in next table. >>>> + </p> >>>> + >>>> + <p> >>>> + This table contains the following logical flows: >>>> + </p> >>>> + >>>> + <ul> >>>> + <li> >>>> + <p> >>>> + Priority-100 flow with match <code>inport == >> "LRP_NAME"</code> value >>>> + and action, which set route table identifier in reg7. >>>> + </p> >>>> + >>>> + <p> >>>> + A priority-0 logical flow with match <code>1</code> has >> actions >>>> + <code>next;</code>. >>>> + </p> >>>> + </li> >>>> + </ul> >>>> + >>>> + <h3>Ingress Table 11: IP Routing</h3> >>>> >>>> <p> >>>> A packet that arrives at this table is an IP packet that should >> be >>>> @@ -3316,10 +3344,10 @@ output; >>>> <p> >>>> IPv4 routing table. For each route to IPv4 network >> <var>N</var> with >>>> netmask <var>M</var>, on router port <var>P</var> with IP >> address >>>> - <var>A</var> and Ethernet >>>> - address <var>E</var>, a logical flow with match >> <code>ip4.dst == >>>> - <var>N</var>/<var>M</var></code>, whose priority is the >> number of >>>> - 1-bits in <var>M</var>, has the following actions: >>>> + <var>A</var> and Ethernet address <var>E</var>, a logical >> flow with >>>> + match <code>ip4.dst == <var>N</var>/<var>M</var></code>, >> whose >>>> + priority is 100 + the number of 1-bits in <var>M</var>, has >> the >>>> + following actions: >>>> </p> >>>> >>>> <pre> >>>> @@ -3382,6 +3410,13 @@ next; >>>> If the address <var>A</var> is in the link-local scope, the >>>> route will be limited to sending on the ingress port. >>>> </p> >>>> + >>>> + <p> >>>> + For routes with <code>route_table</code> value set >>>> + <code>reg7 == id</code> is prefixed in logical flow match >> portion. >>>> + Priority for routes with <code>route_table</code> value set >> is >>>> + the number of 1-bits in <var>M</var>. >>>> + </p> >>>> </li> >>>> >>>> <li> >>>> @@ -3408,7 +3443,7 @@ select(reg8[16..31], <var>MID1</var>, >> <var>MID2</var>, ...); >>>> </li> >>>> </ul> >>>> >>>> - <h3>Ingress Table 11: IP_ROUTING_ECMP</h3> >>>> + <h3>Ingress Table 12: IP_ROUTING_ECMP</h3> >>>> >>>> <p> >>>> This table implements the second part of IP routing for ECMP >> routes >>>> @@ -3460,7 +3495,7 @@ outport = <var>P</var>; >>>> </li> >>>> </ul> >>>> >>>> - <h3>Ingress Table 12: Router policies</h3> >>>> + <h3>Ingress Table 13: Router policies</h3> >>>> <p> >>>> This table adds flows for the logical router policies configured >>>> on the logical router. Please see the >>>> @@ -3532,7 +3567,7 @@ next; >>>> </li> >>>> </ul> >>>> >>>> - <h3>Ingress Table 13: ECMP handling for router policies</h3> >>>> + <h3>Ingress Table 14: ECMP handling for router policies</h3> >>>> <p> >>>> This table handles the ECMP for the router policies configured >>>> with multiple nexthops. >>>> @@ -3576,7 +3611,7 @@ outport = <var>P</var> >>>> </li> >>>> </ul> >>>> >>>> - <h3>Ingress Table 14: ARP/ND Resolution</h3> >>>> + <h3>Ingress Table 15: ARP/ND Resolution</h3> >>>> >>>> <p> >>>> Any packet that reaches this table is an IP packet whose next-hop >>>> @@ -3767,7 +3802,7 @@ outport = <var>P</var> >>>> >>>> </ul> >>>> >>>> - <h3>Ingress Table 15: Check packet length</h3> >>>> + <h3>Ingress Table 16: Check packet length</h3> >>>> >>>> <p> >>>> For distributed logical routers or gateway routers with gateway >>>> @@ -3797,7 +3832,7 @@ REGBIT_PKT_LARGER = >> check_pkt_larger(<var>L</var>); next; >>>> and advances to the next table. >>>> </p> >>>> >>>> - <h3>Ingress Table 16: Handle larger packets</h3> >>>> + <h3>Ingress Table 17: Handle larger packets</h3> >>>> >>>> <p> >>>> For distributed logical routers or gateway routers with gateway >> port >>>> @@ -3860,7 +3895,7 @@ icmp6 { >>>> and advances to the next table. >>>> </p> >>>> >>>> - <h3>Ingress Table 17: Gateway Redirect</h3> >>>> + <h3>Ingress Table 18: Gateway Redirect</h3> >>>> >>>> <p> >>>> For distributed logical routers where one or more of the logical >> router >>>> @@ -3907,7 +3942,7 @@ icmp6 { >>>> </li> >>>> </ul> >>>> >>>> - <h3>Ingress Table 18: ARP Request</h3> >>>> + <h3>Ingress Table 19: ARP Request</h3> >>>> >>>> <p> >>>> In the common case where the Ethernet destination has been >> resolved, this >>>> diff --git a/ovn-nb.ovsschema b/ovn-nb.ovsschema >>>> index 2ac8ef3ea..a0a171e19 100644 >>>> --- a/ovn-nb.ovsschema >>>> +++ b/ovn-nb.ovsschema >>>> @@ -1,7 +1,7 @@ >>>> { >>>> "name": "OVN_Northbound", >>>> - "version": "5.32.1", >>>> - "cksum": "2805328215 29734", >>>> + "version": "5.33.1", >>>> + "cksum": "3874993350 29785", >>>> "tables": { >>>> "NB_Global": { >>>> "columns": { >>>> @@ -387,6 +387,7 @@ >>>> "isRoot": false}, >>>> "Logical_Router_Static_Route": { >>>> "columns": { >>>> + "route_table": {"type": "string"}, >>>> "ip_prefix": {"type": "string"}, >>>> "policy": {"type": {"key": {"type": "string", >>>> "enum": ["set", ["src-ip", >>>> diff --git a/ovn-nb.xml b/ovn-nb.xml >>>> index d8266ed4d..b2917c363 100644 >>>> --- a/ovn-nb.xml >>>> +++ b/ovn-nb.xml >>>> @@ -2772,6 +2772,14 @@ >>>> prefix according to RFC3663 >>>> </p> >>>> </column> >>>> + >>>> + <column name="options" key="route_table"> >>>> + Designates lookup Logical_Router_Static_Routes with specified >>>> + <code>route_table</code> value. Routes to directly connected >> networks >>>> + from same Logical Router and routes without >> <code>route_table</code> >>>> + option set have higher priority than routes with >>>> + <code>route_table</code> option set. >>>> + </column> >>>> </group> >>>> >>>> <group title="Attachment"> >>>> @@ -2891,6 +2899,28 @@ >>>> </p> >>>> </column> >>>> >>>> + <column name="route_table"> >>>> + <p> >>>> + Any string to place route to separate routing table. If >> Logical Router >>>> + Port has configured value in <ref table="Logical_Router_Port" >>>> + column="options" key="route_table"/> other than empty string, >> OVN >>>> + performs route lookup for all packets entering Logical Router >> ingress >>>> + pipeline from this port in the following manner: >>>> + </p> >>>> + >>>> + <ul> >>>> + <li> >>>> + 1. First lookup among "global" routes: routes without >>>> + <code>route_table</code> value set and routes to directly >> connected >>>> + networks. >>>> + </li> >>>> + <li> >>>> + 2. Next lookup among routes with same >> <code>route_table</code> value >>>> + as specified in LRP's options:route_table field. >>>> + </li> >>>> + </ul> >>>> + </column> >>>> + >>>> <column name="external_ids" key="ic-learned-route"> >>>> <code>ovn-ic</code> populates this key if the route is learned >> from the >>>> global <ref db="OVN_IC_Southbound"/> database. In this case the >> value >>>> diff --git a/tests/ovn-ic.at b/tests/ovn-ic.at >>>> index 32f4e9d02..3aab54362 100644 >>>> --- a/tests/ovn-ic.at >>>> +++ b/tests/ovn-ic.at >>>> @@ -281,6 +281,7 @@ done >>>> >>>> AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr1], [0], [dnl >>>> IPv4 Routes >>>> +Route Table global: >>>> 10.11.1.0/24 169.254.0.1 dst-ip >>>> 10.11.2.0/24 169.254.100.2 dst-ip (learned) >>>> 10.22.1.0/24 169.254.0.2 src-ip >>>> @@ -299,6 +300,7 @@ ovn_as az1 ovn-nbctl set nb_global . >> options:ic-route-learn=false >>>> OVS_WAIT_WHILE([ovn_as az1 ovn-nbctl lr-route-list lr1 | grep learned]) >>>> AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr1], [0], [dnl >>>> IPv4 Routes >>>> +Route Table global: >>>> 10.11.1.0/24 169.254.0.1 dst-ip >>>> 10.22.1.0/24 169.254.0.2 src-ip >>>> ]) >>>> @@ -314,6 +316,7 @@ ovn_as az1 ovn-nbctl set nb_global . >> options:ic-route-adv=false >>>> OVS_WAIT_WHILE([ovn_as az2 ovn-nbctl lr-route-list lr2 | grep learned]) >>>> AT_CHECK([ovn_as az2 ovn-nbctl lr-route-list lr2], [0], [dnl >>>> IPv4 Routes >>>> +Route Table global: >>>> 10.11.2.0/24 169.254.0.1 dst-ip >>>> 10.22.2.0/24 169.254.0.2 src-ip >>>> ]) >>>> @@ -332,6 +335,7 @@ done >>>> # Default route should NOT get advertised or learned, by default. >>>> AT_CHECK([ovn_as az2 ovn-nbctl lr-route-list lr2], [0], [dnl >>>> IPv4 Routes >>>> +Route Table global: >>>> 10.11.1.0/24 169.254.100.1 dst-ip (learned) >>>> 10.11.2.0/24 169.254.0.1 dst-ip >>>> 10.22.2.0/24 169.254.0.2 src-ip >>>> diff --git a/tests/ovn-nbctl.at b/tests/ovn-nbctl.at >>>> index 9b80ae410..ddb5536ce 100644 >>>> --- a/tests/ovn-nbctl.at >>>> +++ b/tests/ovn-nbctl.at >>>> @@ -1520,6 +1520,7 @@ AT_CHECK([ovn-nbctl --ecmp --policy=src-ip >> lr-route-add lr0 20.0.0.0/24 11.0.0.1 >>>> >>>> AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >>>> IPv4 Routes >>>> +Route Table global: >>>> 10.0.0.0/24 11.0.0.1 dst-ip >>>> 10.0.1.0/24 11.0.1.1 dst-ip lp0 >>>> 10.0.10.0/24 dst-ip lp0 >>>> @@ -1534,6 +1535,7 @@ AT_CHECK([ovn-nbctl lrp-add lr0 lp1 >> f0:00:00:00:00:02 11.0.0.254/24]) >>>> AT_CHECK([ovn-nbctl --may-exist lr-route-add lr0 10.0.0.111/24 >> 11.0.0.1 lp1]) >>>> AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >>>> IPv4 Routes >>>> +Route Table global: >>>> 10.0.0.0/24 11.0.0.1 dst-ip lp1 >>>> 10.0.1.0/24 11.0.1.1 dst-ip lp0 >>>> 10.0.10.0/24 dst-ip lp0 >>>> @@ -1564,6 +1566,7 @@ AT_CHECK([ovn-nbctl --policy=src-ip lr-route-del >> lr0 9.16.1.0/24]) >>>> >>>> AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >>>> IPv4 Routes >>>> +Route Table global: >>>> 10.0.0.0/24 11.0.0.1 dst-ip lp1 >>>> 10.0.10.0/24 dst-ip lp0 >>>> 10.0.0.0/24 11.0.0.2 src-ip >>>> @@ -1575,6 +1578,7 @@ AT_CHECK([ovn-nbctl --policy=dst-ip lr-route-del >> lr0 10.0.0.0/24]) >>>> AT_CHECK([ovn-nbctl --policy=src-ip lr-route-del lr0 10.0.0.0/24]) >>>> AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >>>> IPv4 Routes >>>> +Route Table global: >>>> 10.0.10.0/24 dst-ip lp0 >>>> 0.0.0.0/0 192.168.0.1 dst-ip >>>> ]) >>>> @@ -1585,6 +1589,7 @@ AT_CHECK([ovn-nbctl --policy=src-ip lr-route-add >> lr0 10.0.0.0/24 11.0.0.2]) >>>> AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24]) >>>> AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >>>> IPv4 Routes >>>> +Route Table global: >>>> 10.0.10.0/24 dst-ip lp0 >>>> 0.0.0.0/0 192.168.0.1 dst-ip >>>> ]) >>>> @@ -1601,6 +1606,7 @@ AT_CHECK([ovn-nbctl --ecmp lr-route-add lr0 >> 10.0.0.0/24 11.0.0.3]) >>>> AT_CHECK([ovn-nbctl --ecmp lr-route-add lr0 10.0.0.0/24 11.0.0.4 lp0]) >>>> AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >>>> IPv4 Routes >>>> +Route Table global: >>>> 10.0.0.0/24 11.0.0.1 dst-ip ecmp >>>> 10.0.0.0/24 11.0.0.2 dst-ip ecmp >>>> 10.0.0.0/24 11.0.0.3 dst-ip ecmp >>>> @@ -1615,6 +1621,7 @@ dnl Delete ecmp routes >>>> AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 11.0.0.1]) >>>> AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >>>> IPv4 Routes >>>> +Route Table global: >>>> 10.0.0.0/24 11.0.0.2 dst-ip ecmp >>>> 10.0.0.0/24 11.0.0.3 dst-ip ecmp >>>> 10.0.0.0/24 11.0.0.4 dst-ip lp0 ecmp >>>> @@ -1622,12 +1629,14 @@ IPv4 Routes >>>> AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 11.0.0.2]) >>>> AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >>>> IPv4 Routes >>>> +Route Table global: >>>> 10.0.0.0/24 11.0.0.3 dst-ip ecmp >>>> 10.0.0.0/24 11.0.0.4 dst-ip lp0 ecmp >>>> ]) >>>> AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 11.0.0.4 lp0]) >>>> AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >>>> IPv4 Routes >>>> +Route Table global: >>>> 10.0.0.0/24 11.0.0.3 dst-ip >>>> ]) >>>> AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 11.0.0.3]) >>>> @@ -1641,6 +1650,7 @@ AT_CHECK([ovn-nbctl lr-route-add lr0 >> 2001:0db8:1::/64 2001:0db8:0:f103::1]) >>>> >>>> AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >>>> IPv6 Routes >>>> +Route Table global: >>>> 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 >>>> 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip >>>> ::/0 2001:db8:0:f101::1 dst-ip >>>> @@ -1650,6 +1660,7 @@ AT_CHECK([ovn-nbctl lr-route-del lr0 >> 2001:0db8:0::/64]) >>>> >>>> AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >>>> IPv6 Routes >>>> +Route Table global: >>>> 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip >>>> ::/0 2001:db8:0:f101::1 dst-ip >>>> ]) >>>> @@ -1677,11 +1688,13 @@ AT_CHECK([ovn-nbctl --may-exist >> --ecmp-symmetric-reply lr-route-add lr0 2003:0db >>>> >>>> AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >>>> IPv4 Routes >>>> +Route Table global: >>>> 10.0.0.0/24 11.0.0.1 dst-ip >>>> 10.0.1.0/24 11.0.1.1 dst-ip lp0 >>>> 0.0.0.0/0 192.168.0.1 dst-ip >>>> >>>> IPv6 Routes >>>> +Route Table global: >>>> 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 >>>> 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip ecmp >>>> 2001:db8:1::/64 2001:db8:0:f103::2 dst-ip ecmp >>>> @@ -1696,7 +1709,188 @@ AT_CHECK([ovn-nbctl lrp-add lr0 lr0-p0 >> 00:00:01:01:02:03 192.168.10.1/24]) >>>> bfd_uuid=$(ovn-nbctl create bfd logical_port=lr0-p0 dst_ip=100.0.0.50 >> status=down min_tx=250 min_rx=250 detect_mult=10) >>>> AT_CHECK([ovn-nbctl lr-route-add lr0 100.0.0.0/24 192.168.0.1]) >>>> route_uuid=$(fetch_column nb:logical_router_static_route _uuid >> ip_prefix="100.0.0.0/24") >>>> -AT_CHECK([ovn-nbctl set logical_router_static_route $route_uuid >> bfd=$bfd_uuid])]) >>>> +AT_CHECK([ovn-nbctl set logical_router_static_route $route_uuid >> bfd=$bfd_uuid]) >>>> + >>>> +check ovn-nbctl lr-route-del lr0 >>>> +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >>>> +]) >>>> + >>>> +dnl Check IPv4 routes in route table >>>> +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 0.0.0.0/0 >> 192.168.0.1 >>>> +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 10.0.1.1/24 >> 11.0.1.1 lp0 >>>> +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 10.0.0.1/24 >> 11.0.0.1 >>>> +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >>>> +IPv4 Routes >>>> +Route Table rtb-1: >>>> + 10.0.0.0/24 11.0.0.1 dst-ip >>>> + 10.0.1.0/24 11.0.1.1 dst-ip lp0 >>>> + 0.0.0.0/0 192.168.0.1 dst-ip >>>> +]) >>>> + >>>> +check ovn-nbctl lr-route-del lr0 >>>> +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >>>> +]) >>>> + >>>> +dnl Check IPv6 routes in route table >>>> +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 0:0:0:0:0:0:0:0/0 >> 2001:0db8:0:f101::1 >>>> +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 2001:0db8:0::/64 >> 2001:0db8:0:f102::1 lp0 >>>> +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 2001:0db8:1::/64 >> 2001:0db8:0:f103::1 >>>> + >>>> +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >>>> +IPv6 Routes >>>> +Route Table rtb-1: >>>> + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 >>>> + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip >>>> + ::/0 2001:db8:0:f101::1 dst-ip >>>> +]) >>>> + >>>> +dnl Check IPv4 and IPv6 routes in route table >>>> +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 0.0.0.0/0 >> 192.168.0.1 >>>> +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 10.0.1.1/24 >> 11.0.1.1 lp0 >>>> +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 10.0.0.1/24 >> 11.0.0.1 >>>> + >>>> +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >>>> +IPv4 Routes >>>> +Route Table rtb-1: >>>> + 10.0.0.0/24 11.0.0.1 dst-ip >>>> + 10.0.1.0/24 11.0.1.1 dst-ip lp0 >>>> + 0.0.0.0/0 192.168.0.1 dst-ip >>>> + >>>> +IPv6 Routes >>>> +Route Table rtb-1: >>>> + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 >>>> + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip >>>> + ::/0 2001:db8:0:f101::1 dst-ip >>>> +]) >>>> + >>>> +# Add routes in another route table >>>> +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 0.0.0.0/0 >> 192.168.0.1 >>>> +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 10.0.1.1/24 >> 11.0.1.1 lp0 >>>> +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 10.0.0.1/24 >> 11.0.0.1 >>>> +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 0:0:0:0:0:0:0:0/0 >> 2001:0db8:0:f101::1 >>>> +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 2001:0db8:0::/64 >> 2001:0db8:0:f102::1 lp0 >>>> +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 2001:0db8:1::/64 >> 2001:0db8:0:f103::1 >>>> + >>>> +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >>>> +IPv4 Routes >>>> +Route Table rtb-1: >>>> + 10.0.0.0/24 11.0.0.1 dst-ip >>>> + 10.0.1.0/24 11.0.1.1 dst-ip lp0 >>>> + 0.0.0.0/0 192.168.0.1 dst-ip >>>> + >>>> +Route Table rtb-2: >>>> + 10.0.0.0/24 11.0.0.1 dst-ip >>>> + 10.0.1.0/24 11.0.1.1 dst-ip lp0 >>>> + 0.0.0.0/0 192.168.0.1 dst-ip >>>> + >>>> +IPv6 Routes >>>> +Route Table rtb-1: >>>> + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 >>>> + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip >>>> + ::/0 2001:db8:0:f101::1 dst-ip >>>> + >>>> +Route Table rtb-2: >>>> + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 >>>> + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip >>>> + ::/0 2001:db8:0:f101::1 dst-ip >>>> +]) >>>> + >>>> +# Add routes to global route table >>>> +check ovn-nbctl lr-route-add lr0 0.0.0.0/0 192.168.0.1 >>>> +check ovn-nbctl lr-route-add lr0 10.0.1.1/24 11.0.1.1 lp0 >>>> +check ovn-nbctl lr-route-add lr0 10.0.0.1/24 11.0.0.1 >>>> +check ovn-nbctl lr-route-add lr0 0:0:0:0:0:0:0:0/0 2001:0db8:0:f101::1 >>>> +check ovn-nbctl lr-route-add lr0 2001:0db8:0::/64 2001:0db8:0:f102::1 >> lp0 >>>> +check check ovn-nbctl lr-route-add lr0 2001:0db8:1::/64 >> 2001:0db8:0:f103::1 >>>> + >>>> +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >>>> +IPv4 Routes >>>> +Route Table global: >>>> + 10.0.0.0/24 11.0.0.1 dst-ip >>>> + 10.0.1.0/24 11.0.1.1 dst-ip lp0 >>>> + 0.0.0.0/0 192.168.0.1 dst-ip >>>> + >>>> +Route Table rtb-1: >>>> + 10.0.0.0/24 11.0.0.1 dst-ip >>>> + 10.0.1.0/24 11.0.1.1 dst-ip lp0 >>>> + 0.0.0.0/0 192.168.0.1 dst-ip >>>> + >>>> +Route Table rtb-2: >>>> + 10.0.0.0/24 11.0.0.1 dst-ip >>>> + 10.0.1.0/24 11.0.1.1 dst-ip lp0 >>>> + 0.0.0.0/0 192.168.0.1 dst-ip >>>> + >>>> +IPv6 Routes >>>> +Route Table global: >>>> + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 >>>> + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip >>>> + ::/0 2001:db8:0:f101::1 dst-ip >>>> + >>>> +Route Table rtb-1: >>>> + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 >>>> + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip >>>> + ::/0 2001:db8:0:f101::1 dst-ip >>>> + >>>> +Route Table rtb-2: >>>> + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 >>>> + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip >>>> + ::/0 2001:db8:0:f101::1 dst-ip >>>> +]) >>>> + >>>> +# delete IPv4 route from rtb-1 >>>> +check ovn-nbctl --route-table=rtb-1 lr-route-del lr0 10.0.0.0/24 >>>> +AT_CHECK([ovn-nbctl --route-table=rtb-1 lr-route-list lr0], [0], [dnl >>>> +IPv4 Routes >>>> +Route Table rtb-1: >>>> + 10.0.1.0/24 11.0.1.1 dst-ip lp0 >>>> + 0.0.0.0/0 192.168.0.1 dst-ip >>>> + >>>> +IPv6 Routes >>>> +Route Table rtb-1: >>>> + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 >>>> + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip >>>> + ::/0 2001:db8:0:f101::1 dst-ip >>>> +]) >>>> + >>>> +# delete IPv6 route from rtb-2 >>>> +check ovn-nbctl --route-table=rtb-2 lr-route-del lr0 2001:db8::/64 >>>> +AT_CHECK([ovn-nbctl --route-table=rtb-2 lr-route-list lr0], [0], [dnl >>>> +IPv4 Routes >>>> +Route Table rtb-2: >>>> + 10.0.0.0/24 11.0.0.1 dst-ip >>>> + 10.0.1.0/24 11.0.1.1 dst-ip lp0 >>>> + 0.0.0.0/0 192.168.0.1 dst-ip >>>> + >>>> +IPv6 Routes >>>> +Route Table rtb-2: >>>> + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip >>>> + ::/0 2001:db8:0:f101::1 dst-ip >>>> +]) >>>> + >>>> +check ovn-nbctl lr-route-del lr0 >>>> + >>>> +# ECMP route in route table >>>> +check ovn-nbctl --route-table=rtb1 lr-route-add lr0 0.0.0.0/0 >> 192.168.0.1 >>>> +check ovn-nbctl --ecmp --route-table=rtb1 lr-route-add lr0 0.0.0.0/0 >> 192.168.0.2 >>>> + >>>> +# Negative route table case: same prefix >>>> +AT_CHECK([ovn-nbctl --route-table=rtb1 lr-route-add lr0 0.0.0.0/0 >> 192.168.0.1], [1], [], [dnl >>>> +ovn-nbctl: duplicate prefix: 0.0.0.0/0 (policy: dst-ip). Use option >> --ecmp to allow this for ECMP routing. >>>> +]) >>>> + >>>> +# Negative route table case: same prefix & nexthop with ecmp >>>> +AT_CHECK([ovn-nbctl --ecmp --route-table=rtb1 lr-route-add lr0 >> 0.0.0.0/0 192.168.0.2], [1], [], [dnl >>>> +ovn-nbctl: duplicate nexthop for the same ECMP route >>>> +]) >>>> + >>>> +# Add routes to global route table >>>> +check ovn-nbctl lrp-add lr0 lrp0 00:00:00:00:00:01 1.1.1.1/24 >>>> +check ovn-nbctl lrp-set-options lrp0 route_table=rtb1 >>>> +AT_CHECK([ovn-nbctl get logical-router-port lrp0 options:route_table], >> [0], [dnl >>>> +rtb1 >>>> +]) >>>> +check `ovn-nbctl show lr0 | grep lrp0 -A3 | grep route_table=rtb1` >>>> +]) >>>> >>>> dnl >> --------------------------------------------------------------------- >>>> >>>> diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at >>>> index 3eebb55b6..e71e65bcc 100644 >>>> --- a/tests/ovn-northd.at >>>> +++ b/tests/ovn-northd.at >>>> @@ -5111,7 +5111,7 @@ check ovn-nbctl --wait=sb --ecmp-symmetric-reply >> lr-route-add lr0 1.0.0.1 192.16 >>>> >>>> ovn-sbctl dump-flows lr0 > lr0flows >>>> AT_CHECK([grep -e "lr_in_ip_routing.*select" lr0flows | sed >> 's/table=../table=??/' | sort], [0], [dnl >>>> - table=??(lr_in_ip_routing ), priority=65 , match=(ip4.dst == >> 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = 1; >> reg8[[16..31]] = select(1, 2);) >>>> + table=??(lr_in_ip_routing ), priority=165 , match=(ip4.dst == >> 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = 1; >> reg8[[16..31]] = select(1, 2);) >>>> ]) >>>> AT_CHECK([grep -e "lr_in_ip_routing_ecmp" lr0flows | sed >> 's/192\.168\.0\..0/192.168.0.??/' | sed 's/table=../table=??/' | sort], >> [0], [dnl >>>> table=??(lr_in_ip_routing_ecmp), priority=100 , >> match=(reg8[[0..15]] == 1 && reg8[[16..31]] == 1), action=(reg0 = >> 192.168.0.??; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = >> "lr0-public"; next;) >>>> @@ -5124,7 +5124,7 @@ check ovn-nbctl --wait=sb --ecmp-symmetric-reply >> lr-route-add lr0 1.0.0.1 192.16 >>>> >>>> ovn-sbctl dump-flows lr0 > lr0flows >>>> AT_CHECK([grep -e "lr_in_ip_routing.*select" lr0flows | sed >> 's/table=../table=??/' | sort], [0], [dnl >>>> - table=??(lr_in_ip_routing ), priority=65 , match=(ip4.dst == >> 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = 1; >> reg8[[16..31]] = select(1, 2);) >>>> + table=??(lr_in_ip_routing ), priority=165 , match=(ip4.dst == >> 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = 1; >> reg8[[16..31]] = select(1, 2);) >>>> ]) >>>> AT_CHECK([grep -e "lr_in_ip_routing_ecmp" lr0flows | sed >> 's/192\.168\.0\..0/192.168.0.??/' | sed 's/table=../table=??/' | sort], >> [0], [dnl >>>> table=??(lr_in_ip_routing_ecmp), priority=100 , >> match=(reg8[[0..15]] == 1 && reg8[[16..31]] == 1), action=(reg0 = >> 192.168.0.??; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = >> "lr0-public"; next;) >>>> @@ -5139,14 +5139,14 @@ check ovn-nbctl --wait=sb lr-route-add lr0 >> 1.0.0.0/24 192.168.0.10 >>>> ovn-sbctl dump-flows lr0 > lr0flows >>>> >>>> AT_CHECK([grep -e "lr_in_ip_routing.*192.168.0.10" lr0flows | sed >> 's/table=../table=??/' | sort], [0], [dnl >>>> - table=??(lr_in_ip_routing ), priority=49 , match=(ip4.dst == >> 1.0.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.10; reg1 >> = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; >> flags.loopback = 1; next;) >>>> + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == >> 1.0.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.10; reg1 >> = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; >> flags.loopback = 1; next;) >>>> ]) >>>> >>>> check ovn-nbctl --wait=sb lr-route-add lr0 2.0.0.0/24 lr0-public >>>> >>>> ovn-sbctl dump-flows lr0 > lr0flows >>>> AT_CHECK([grep -e "lr_in_ip_routing.*2.0.0.0" lr0flows | sed >> 's/table=../table=??/' | sort], [0], [dnl >>>> - table=??(lr_in_ip_routing ), priority=49 , match=(ip4.dst == >> 2.0.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 = >> 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; >> flags.loopback = 1; next;) >>>> + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == >> 2.0.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 = >> 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; >> flags.loopback = 1; next;) >>>> ]) >>>> >>>> AT_CLEANUP >>>> @@ -5232,3 +5232,71 @@ AT_CHECK([grep lr_in_gw_redirect lrflows | grep >> cr-DR | sed 's/table=../table=?? >>>> >>>> AT_CLEANUP >>>> ]) >>>> + >>>> + >>>> +OVN_FOR_EACH_NORTHD([ >>>> +AT_SETUP([route tables -- flows]) >>>> +AT_KEYWORDS([route-tables-flows]) >>>> +ovn_start >>>> + >>>> +check ovn-nbctl lr-add lr0 >>>> +check ovn-nbctl lrp-add lr0 lrp0 00:00:00:00:00:01 192.168.0.1/24 >>>> +check ovn-nbctl lrp-add lr0 lrp1 00:00:00:00:01:01 192.168.1.1/24 >>>> +check ovn-nbctl lrp-add lr0 lrp2 00:00:00:00:02:01 192.168.2.1/24 >>>> +check ovn-nbctl lrp-set-options lrp1 route_table=rtb-1 >>>> +check ovn-nbctl lrp-set-options lrp2 route_table=rtb-2 >>>> + >>>> +check ovn-nbctl lr-route-add lr0 0.0.0.0/0 192.168.0.10 >>>> +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 192.168.0.0/24 >> 192.168.1.10 >>>> +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 0.0.0.0/0 >> 192.168.0.10 >>>> +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 1.1.1.1/32 >> 192.168.0.20 >>>> +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 2.2.2.2/32 >> 192.168.0.30 >>>> +check ovn-nbctl --route-table=rtb-2 --ecmp lr-route-add lr0 2.2.2.2/32 >> 192.168.0.31 >>>> +check ovn-nbctl --wait=sb sync >>>> + >>>> +ovn-sbctl dump-flows lr0 > lr0flows >>>> +AT_CAPTURE_FILE([lr0flows]) >>>> + >>>> +AT_CHECK([grep -e "lr_in_ip_routing_pre.*match=(1)" lr0flows | sed >> 's/table=../table=??/'], [0], [dnl >>>> + table=??(lr_in_ip_routing_pre), priority=0 , match=(1), >> action=(next;) >>>> +]) >>>> + >>>> +p1_reg=$(grep -oP "lr_in_ip_routing_pre.*lrp1.*action=\(reg7 = \K." >> lr0flows) >>>> +p2_reg=$(grep -oP "lr_in_ip_routing_pre.*lrp2.*action=\(reg7 = \K." >> lr0flows) >>>> +echo $p1_reg >>>> +echo $p2_reg >>>> + >>>> +# exact register values are not predictable >>>> +if [[ $p1_reg -eq 2 ] && [ $p2_reg -eq 1 ]]; then >>>> + echo "swap reg values in dump" >>>> + sed -i -r s'/^(.*lrp2.*action=\(reg7 = )(1)(.*)/\12\3/g' lr0flows # >> "reg7 = 1" -> "reg7 = 2" >>>> + sed -i -r s'/^(.*lrp1.*action=\(reg7 = )(2)(.*)/\11\3/g' lr0flows # >> "reg7 = 2" -> "reg7 = 1" >>>> + sed -i -r s'/^(.*match=\(reg7 == )(2)( &&.*lrp1.*)/\11\3/g' lr0flows >> # "reg7 == 2" -> "reg7 == 1" >>>> + sed -i -r s'/^(.*match=\(reg7 == )(1)( &&.*lrp0.*)/\12\3/g' lr0flows >> # "reg7 == 1" -> "reg7 == 2" >>>> +fi >>>> + >>>> +check test "$p1_reg" != "$p2_reg" -a $((p1_reg * p2_reg)) -eq 2 >>>> + >>>> +AT_CHECK([grep "lr_in_ip_routing_pre" lr0flows | sed >> 's/table=../table=??/' | sort], [0], [dnl >>>> + table=??(lr_in_ip_routing_pre), priority=0 , match=(1), >> action=(next;) >>>> + table=??(lr_in_ip_routing_pre), priority=100 , match=(inport == >> "lrp1"), action=(reg7 = 1; next;) >>>> + table=??(lr_in_ip_routing_pre), priority=100 , match=(inport == >> "lrp2"), action=(reg7 = 2; next;) >>>> +]) >>>> + >>>> +grep -e "(lr_in_ip_routing ).*outport" lr0flows >>>> + >>>> +AT_CHECK([grep -e "(lr_in_ip_routing ).*outport" lr0flows | sed >> 's/table=../table=??/' | sort], [0], [dnl >>>> + table=??(lr_in_ip_routing ), priority=1 , match=(reg7 == 2 && >> ip4.dst == 0.0.0.0/0), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = >> 192.168.0.10; reg1 = 192.168.0.1; eth.src = 00:00:00:00:00:01; outport = >> "lrp0"; flags.loopback = 1; next;) >>>> + table=??(lr_in_ip_routing ), priority=101 , match=(ip4.dst == >> 0.0.0.0/0), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.10; reg1 >> = 192.168.0.1; eth.src = 00:00:00:00:00:01; outport = "lrp0"; >> flags.loopback = 1; next;) >>>> + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == >> 192.168.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 >> = 192.168.0.1; eth.src = 00:00:00:00:00:01; outport = "lrp0"; >> flags.loopback = 1; next;) >>>> + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == >> 192.168.1.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 >> = 192.168.1.1; eth.src = 00:00:00:00:01:01; outport = "lrp1"; >> flags.loopback = 1; next;) >>>> + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == >> 192.168.2.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 >> = 192.168.2.1; eth.src = 00:00:00:00:02:01; outport = "lrp2"; >> flags.loopback = 1; next;) >>>> + table=??(lr_in_ip_routing ), priority=229 , match=(inport == >> "lrp0" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; >> xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe00:1; eth.src = >> 00:00:00:00:00:01; outport = "lrp0"; flags.loopback = 1; next;) >>>> + table=??(lr_in_ip_routing ), priority=229 , match=(inport == >> "lrp1" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; >> xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe00:101; eth.src = >> 00:00:00:00:01:01; outport = "lrp1"; flags.loopback = 1; next;) >>>> + table=??(lr_in_ip_routing ), priority=229 , match=(inport == >> "lrp2" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; >> xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe00:201; eth.src = >> 00:00:00:00:02:01; outport = "lrp2"; flags.loopback = 1; next;) >>>> + table=??(lr_in_ip_routing ), priority=49 , match=(reg7 == 1 && >> ip4.dst == 192.168.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = >> 192.168.1.10; reg1 = 192.168.1.1; eth.src = 00:00:00:00:01:01; outport = >> "lrp1"; flags.loopback = 1; next;) >>>> + table=??(lr_in_ip_routing ), priority=65 , match=(reg7 == 2 && >> ip4.dst == 1.1.1.1/32), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = >> 192.168.0.20; reg1 = 192.168.0.1; eth.src = 00:00:00:00:00:01; outport = >> "lrp0"; flags.loopback = 1; next;) >>>> +]) >>>> + >>>> +AT_CLEANUP >>>> +]) >>>> diff --git a/tests/ovn.at b/tests/ovn.at >>>> index 49ece8735..60783a14b 100644 >>>> --- a/tests/ovn.at >>>> +++ b/tests/ovn.at >>>> @@ -18145,7 +18145,7 @@ eth_dst=00000000ff01 >>>> ip_src=$(ip_to_hex 10 0 0 10) >>>> ip_dst=$(ip_to_hex 172 168 0 101) >>>> send_icmp_packet 1 1 $eth_src $eth_dst $ip_src $ip_dst c4c9 >> 0000000000000000000000 >>>> -AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | awk '/table=25, >> n_packets=1, n_bytes=45/{print $7" "$8}'],[0],[dnl >>>> +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | awk '/table=26, >> n_packets=1, n_bytes=45/{print $7" "$8}'],[0],[dnl >>>> priority=80,ip,reg15=0x3,metadata=0x3,nw_src=10.0.0.10 actions=drop >>>> ]) >>>> >>>> @@ -22577,6 +22577,433 @@ OVN_CLEANUP([hv1]) >>>> AT_CLEANUP >>>> ]) >>>> >>>> + >>>> +OVN_FOR_EACH_NORTHD([ >>>> +AT_SETUP([route tables -- global routes]) >>>> +ovn_start >>>> + >>>> +# Logical network: >>>> +# ls1 (192.168.1.0/24) - lrp-lr1-ls1 - lr1 - lrp-lr1-ls2 - ls2 ( >> 192.168.2.0/24) >>>> +# >>>> +# ls1 has lsp11 (192.168.1.11) and ls2 has lsp21 (192.168.2.21) and >> lsp22 >>>> +# (192.168.2.22) >>>> +# >>>> +# lrp-lr1-ls1 set options:route_table=rtb-1 >>>> +# >>>> +# Static routes on lr1: >>>> +# 0.0.0.0/0 nexthop 192.168.2.21 >>>> +# 1.1.1.1/32 nexthop 192.168.2.22 route_table=rtb-1 >>>> +# >>>> +# Test 1: >>>> +# lsp11 send packet to 2.2.2.2 >>>> +# >>>> +# Expected result: >>>> +# lsp21 should receive traffic, lsp22 should not receive any traffic >>>> +# >>>> +# Test 2: >>>> +# lsp11 send packet to 1.1.1.1 >>>> +# >>>> +# Expected result: >>>> +# lsp21 should receive traffic, lsp22 should not receive any traffic >>>> + >>>> +ovn-nbctl lr-add lr1 >>>> + >>>> +for i in 1 2; do >>>> + ovn-nbctl ls-add ls${i} >>>> + ovn-nbctl lrp-add lr1 lrp-lr1-ls${i} 00:00:00:01:0${i}:01 >> 192.168.${i}.1/24 >>>> + ovn-nbctl lsp-add ls${i} lsp-ls${i}-lr1 -- lsp-set-type >> lsp-ls${i}-lr1 router \ >>>> + -- lsp-set-options lsp-ls${i}-lr1 router-port=lrp-lr1-ls${i} \ >>>> + -- lsp-set-addresses lsp-ls${i}-lr1 router >>>> +done >>>> + >>>> +# install static routes >>>> +ovn-nbctl lr-route-add lr1 0.0.0.0/0 192.168.2.21 >>>> +ovn-nbctl --route-table=rtb-1 lr-route-add lr1 1.1.1.1/32 192.168.2.22 >>>> + >>>> +# set lrp-lr1-ls1 route table >>>> +ovn-nbctl lrp-set-options lrp-lr1-ls1 route_table=rtb-1 >>>> + >>>> +# Create logical ports >>>> +ovn-nbctl lsp-add ls1 lsp11 -- \ >>>> + lsp-set-addresses lsp11 "f0:00:00:00:01:11 192.168.1.11" >>>> +ovn-nbctl lsp-add ls2 lsp21 -- \ >>>> + lsp-set-addresses lsp21 "f0:00:00:00:02:21 192.168.2.21" >>>> +ovn-nbctl lsp-add ls2 lsp22 -- \ >>>> + lsp-set-addresses lsp22 "f0:00:00:00:02:22 192.168.2.22" >>>> + >>>> +net_add n1 >>>> +sim_add hv1 >>>> +as hv1 >>>> +ovs-vsctl add-br br-phys >>>> +ovn_attach n1 br-phys 192.168.0.1 >>>> +ovs-vsctl -- add-port br-int hv1-vif1 -- \ >>>> + set interface hv1-vif1 external-ids:iface-id=lsp11 \ >>>> + options:tx_pcap=hv1/vif1-tx.pcap \ >>>> + options:rxq_pcap=hv1/vif1-rx.pcap \ >>>> + ofport-request=1 >>>> + >>>> +ovs-vsctl -- add-port br-int hv1-vif2 -- \ >>>> + set interface hv1-vif2 external-ids:iface-id=lsp21 \ >>>> + options:tx_pcap=hv1/vif2-tx.pcap \ >>>> + options:rxq_pcap=hv1/vif2-rx.pcap \ >>>> + ofport-request=2 >>>> + >>>> +ovs-vsctl -- add-port br-int hv1-vif3 -- \ >>>> + set interface hv1-vif3 external-ids:iface-id=lsp22 \ >>>> + options:tx_pcap=hv1/vif3-tx.pcap \ >>>> + options:rxq_pcap=hv1/vif3-rx.pcap \ >>>> + ofport-request=3 >>>> + >>>> +# wait for earlier changes to take effect >>>> +check ovn-nbctl --wait=hv sync >>>> +wait_for_ports_up >>>> + >>>> +for i in 1 2; do >>>> + packet="inport==\"lsp11\" && eth.src==f0:00:00:00:01:11 && >> eth.dst==00:00:00:01:01:01 && >>>> + ip4 && ip.ttl==64 && ip4.src==192.168.1.11 && >> ip4.dst==$i.$i.$i.$i && icmp" >>>> + AT_CHECK([as hv1 ovs-appctl -t ovn-controller inject-pkt >> "$packet"]) >>>> + >>>> + # Assume all packets go to lsp21. >>>> + exp_packet="eth.src==00:00:00:01:02:01 && >> eth.dst==f0:00:00:00:02:21 && >>>> + ip4 && ip.ttl==63 && ip4.src==192.168.1.11 && >> ip4.dst==$i.$i.$i.$i && icmp" >>>> + echo $exp_packet | ovstest test-ovn expr-to-packets >> >> expected_lsp21 >>>> +done >>>> +> expected_lsp22 >>>> + >>>> +# lsp21 should recieve 2 packets and lsp22 should recieve no packets >>>> +OVS_WAIT_UNTIL([ >>>> + rcv_n1=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" >> hv1/vif2-tx.pcap > lsp21.packets && cat lsp21.packets | wc -l` >>>> + rcv_n2=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" >> hv1/vif3-tx.pcap > lsp22.packets && cat lsp22.packets | wc -l` >>>> + echo $rcv_n1 $rcv_n2 >>>> + test $rcv_n1 -eq 2 -a $rcv_n2 -eq 0]) >>>> + >>>> +for i in 1 2; do >>>> + sort expected_lsp2$i > expout >>>> + AT_CHECK([cat lsp2${i}.packets | sort], [0], [expout]) >>>> +done >>>> + >>>> +OVN_CLEANUP([hv1]) >>>> +AT_CLEANUP >>>> +]) >>>> + >>>> + >>>> +OVN_FOR_EACH_NORTHD([ >>>> +AT_SETUP([route tables -- directly connected routes]) >>>> +ovn_start >>>> + >>>> +# Logical network: >>>> +# ls1 (192.168.1.0/24) - lrp-lr1-ls1 - lr1 - lrp-lr1-ls2 - ls2 ( >> 192.168.2.0/24) >>>> +# >>>> +# ls1 has lsp11 (192.168.1.11) and ls2 has lsp21 (192.168.2.21) >>>> +# >>>> +# lrp-lr1-ls1 set options:route_table=rtb-1 >>>> +# >>>> +# Static routes on lr1: >>>> +# 192.168.2.0/25 nexthop 192.168.1.11 route_table=rtb-1 >>>> +# >>>> +# Test 1: >>>> +# lsp11 send packet to 192.168.2.21 >>>> +# >>>> +# Expected result: >>>> +# lsp21 should receive traffic, lsp11 should not receive any traffic >>>> + >>>> +ovn-nbctl lr-add lr1 >>>> + >>>> +for i in 1 2; do >>>> + ovn-nbctl ls-add ls${i} >>>> + ovn-nbctl lrp-add lr1 lrp-lr1-ls${i} 00:00:00:01:0${i}:01 >> 192.168.${i}.1/24 >>>> + ovn-nbctl lsp-add ls${i} lsp-ls${i}-lr1 -- lsp-set-type >> lsp-ls${i}-lr1 router \ >>>> + -- lsp-set-options lsp-ls${i}-lr1 router-port=lrp-lr1-ls${i} \ >>>> + -- lsp-set-addresses lsp-ls${i}-lr1 router >>>> +done >>>> + >>>> +# install static route, which overrides directly-connected routes >>>> +ovn-nbctl --route-table=rtb-1 lr-route-add lr1 192.168.2.0/25 >> 192.168.1.11 >>>> + >>>> +# set lrp-lr1-ls1 route table >>>> +ovn-nbctl lrp-set-options lrp-lr1-ls1 route_table=rtb-1 >>>> + >>>> +# Create logical ports >>>> +ovn-nbctl lsp-add ls1 lsp11 -- \ >>>> + lsp-set-addresses lsp11 "f0:00:00:00:01:11 192.168.1.11" >>>> +ovn-nbctl lsp-add ls2 lsp21 -- \ >>>> + lsp-set-addresses lsp21 "f0:00:00:00:02:21 192.168.2.21" >>>> + >>>> +net_add n1 >>>> +sim_add hv1 >>>> +as hv1 >>>> +ovs-vsctl add-br br-phys >>>> +ovn_attach n1 br-phys 192.168.0.1 >>>> +ovs-vsctl -- add-port br-int hv1-vif1 -- \ >>>> + set interface hv1-vif1 external-ids:iface-id=lsp11 \ >>>> + options:tx_pcap=hv1/vif1-tx.pcap \ >>>> + options:rxq_pcap=hv1/vif1-rx.pcap \ >>>> + ofport-request=1 >>>> + >>>> +ovs-vsctl -- add-port br-int hv1-vif2 -- \ >>>> + set interface hv1-vif2 external-ids:iface-id=lsp21 \ >>>> + options:tx_pcap=hv1/vif2-tx.pcap \ >>>> + options:rxq_pcap=hv1/vif2-rx.pcap \ >>>> + ofport-request=2 >>>> + >>>> +# wait for earlier changes to take effect >>>> +check ovn-nbctl --wait=hv sync >>>> +wait_for_ports_up >>>> + >>>> +packet="inport==\"lsp11\" && eth.src==f0:00:00:00:01:11 && >> eth.dst==00:00:00:01:01:01 && >>>> + ip4 && ip.ttl==64 && ip4.src==192.168.1.11 && >> ip4.dst==192.168.2.21 && icmp" >>>> +AT_CHECK([as hv1 ovs-appctl -t ovn-controller inject-pkt "$packet"]) >>>> + >>>> +# Assume all packets go to lsp21. >>>> +exp_packet="eth.src==00:00:00:01:02:01 && eth.dst==f0:00:00:00:02:21 && >>>> + ip4 && ip.ttl==63 && ip4.src==192.168.1.11 && >> ip4.dst==192.168.2.21 && icmp" >>>> +echo $exp_packet | ovstest test-ovn expr-to-packets >> expected_lsp21 >>>> +> expected_lsp11 >>>> + >>>> +# lsp21 should recieve 1 icmp packet and lsp11 should recieve no >> packets >>>> +OVS_WAIT_UNTIL([ >>>> + rcv_n11=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" >> hv1/vif1-tx.pcap > lsp11.packets && cat lsp11.packets | wc -l` >>>> + rcv_n21=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" >> hv1/vif2-tx.pcap > lsp21.packets && cat lsp21.packets | wc -l` >>>> + echo $rcv_n11 $rcv_n21 >>>> + test $rcv_n11 -eq 0 -a $rcv_n21 -eq 1]) >>>> + >>>> +for i in 11 21; do >>>> + sort expected_lsp$i > expout >>>> + AT_CHECK([cat lsp${i}.packets | sort], [0], [expout]) >>>> +done >>>> + >>>> +OVN_CLEANUP([hv1]) >>>> +AT_CLEANUP >>>> +]) >>>> + >>>> + >>>> +OVN_FOR_EACH_NORTHD([ >>>> +AT_SETUP([route tables -- overlapping subnets]) >>>> +ovn_start >>>> + >>>> +# Logical network: >>>> +# >>>> +# ls1 (192.168.1.0/24) - lrp-lr1-ls1 -\ /- lrp-lr1-ls2 - ls2 ( >> 192.168.2.0/24) >>>> +# lr1 >>>> +# ls3 (192.168.3.0/24) - lrp-lr1-ls3 -/ \- lrp-lr1-ls4 - ls4 ( >> 192.168.4.0/24) >>>> +# >>>> +# ls1 has lsp11 (192.168.1.11) >>>> +# ls2 has lsp21 (192.168.2.21) >>>> +# ls3 has lsp31 (192.168.3.31) >>>> +# ls4 has lsp41 (192.168.4.41) >>>> +# >>>> +# lrp-lr1-ls1 set options:route_table=rtb-1 >>>> +# lrp-lr1-ls2 set options:route_table=rtb-2 >>>> +# >>>> +# Static routes on lr1: >>>> +# 10.0.0.0/24 nexthop 192.168.3.31 route_table=rtb-1 >>>> +# 10.0.0.0/24 nexthop 192.168.4.41 route_table=rtb-2 >>>> +# >>>> +# Test 1: >>>> +# lsp11 send packet to 10.0.0.1 >>>> +# >>>> +# Expected result: >>>> +# lsp31 should receive traffic, lsp41 should not receive any traffic >>>> +# >>>> +# Test 2: >>>> +# lsp21 send packet to 10.0.0.1 >>>> +# >>>> +# Expected result: >>>> +# lsp41 should receive traffic, lsp31 should not receive any traffic >>>> + >>>> +ovn-nbctl lr-add lr1 >>>> + >>>> +# Create logical topology >>>> +for i in $(seq 1 4); do >>>> + ovn-nbctl ls-add ls${i} >>>> + ovn-nbctl lrp-add lr1 lrp-lr1-ls${i} 00:00:00:01:0${i}:01 >> 192.168.${i}.1/24 >>>> + ovn-nbctl lsp-add ls${i} lsp-ls${i}-lr1 -- lsp-set-type >> lsp-ls${i}-lr1 router \ >>>> + -- lsp-set-options lsp-ls${i}-lr1 router-port=lrp-lr1-ls${i} \ >>>> + -- lsp-set-addresses lsp-ls${i}-lr1 router >>>> + ovn-nbctl lsp-add ls$i lsp${i}1 -- \ >>>> + lsp-set-addresses lsp${i}1 "f0:00:00:00:0${i}:1${i} >> 192.168.${i}.${i}1" >>>> +done >>>> + >>>> +# install static routes >>>> +ovn-nbctl --route-table=rtb-1 lr-route-add lr1 10.0.0.0/24 192.168.3.31 >>>> +ovn-nbctl --route-table=rtb-2 lr-route-add lr1 10.0.0.0/24 192.168.4.41 >>>> + >>>> +# set lrp-lr1-ls{1,2} route tables >>>> +ovn-nbctl lrp-set-options lrp-lr1-ls1 route_table=rtb-1 >>>> +ovn-nbctl lrp-set-options lrp-lr1-ls2 route_table=rtb-2 >>>> + >>>> +net_add n1 >>>> +sim_add hv1 >>>> +as hv1 >>>> +ovs-vsctl add-br br-phys >>>> +ovn_attach n1 br-phys 192.168.0.1 >>>> + >>>> +for i in $(seq 1 4); do >>>> + ovs-vsctl -- add-port br-int hv1-vif${i} -- \ >>>> + set interface hv1-vif${i} external-ids:iface-id=lsp${i}1 \ >>>> + options:tx_pcap=hv1/vif${i}-tx.pcap \ >>>> + options:rxq_pcap=hv1/vif${i}-rx.pcap \ >>>> + ofport-request=${i} >>>> +done >>>> + >>>> +# wait for earlier changes to take effect >>>> +check ovn-nbctl --wait=hv sync >>>> +wait_for_ports_up >>>> + >>>> +# lsp31 should recieve packet coming from lsp11 >>>> +# lsp41 should recieve packet coming from lsp21 >>>> +for i in $(seq 1 2); do >>>> + di=$(( i + 2)) # dst index >>>> + ri=$(( 5 - i)) # reverse index >>>> + packet="inport==\"lsp${i}1\" && eth.src==f0:00:00:00:0${i}:1${i} && >>>> + eth.dst==00:00:00:01:0${i}:01 && ip4 && ip.ttl==64 && >>>> + ip4.src==192.168.${i}.${i}1 && ip4.dst==10.0.0.1 && icmp" >>>> + AT_CHECK([as hv1 ovs-appctl -t ovn-controller inject-pkt >> "$packet"]) >>>> + >>>> + # Assume all packets go to lsp${di}1. >>>> + exp_packet="eth.src==00:00:00:01:0${di}:01 && >> eth.dst==f0:00:00:00:0${di}:1${di} && >>>> + ip4 && ip.ttl==63 && ip4.src==192.168.${i}.${i}1 && >> ip4.dst==10.0.0.1 && icmp" >>>> + echo $exp_packet | ovstest test-ovn expr-to-packets >> >> expected_lsp${di}1 >>>> + > expected_lsp${ri}1 >>>> + >>>> + OVS_WAIT_UNTIL([ >>>> + rcv_n1=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" >> hv1/vif${di}-tx.pcap > lsp${di}1.packets && cat lsp${di}1.packets | wc -l` >>>> + rcv_n2=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" >> hv1/vif${ri}-tx.pcap > lsp${ri}1.packets && cat lsp${ri}1.packets | wc -l` >>>> + echo $rcv_n1 $rcv_n2 >>>> + test $rcv_n1 -eq 1 -a $rcv_n2 -eq 0]) >>>> + >>>> + for j in "${di}1" "${ri}1"; do >>>> + sort expected_lsp${j} > expout >>>> + AT_CHECK([cat lsp${j}.packets | sort], [0], [expout]) >>>> + done >>>> + >>>> + # cleanup tx pcap files >>>> + for j in "${di}1" "${ri}1"; do >>>> + ovs-vsctl -- remove interface hv1-vif${di} options tx_pcap >>>> + > hv1/vif${di}-tx.pcap >>>> + ovs-vsctl -- set interface hv1-vif${di} >> external-ids:iface-id=lsp${di}1 \ >>>> + options:tx_pcap=hv1/vif${di}-tx.pcap >>>> + done >>>> +done >>>> + >>>> +OVN_CLEANUP([hv1]) >>>> +AT_CLEANUP >>>> +]) >>>> + >>>> + >>>> +OVN_FOR_EACH_NORTHD([ >>>> +AT_SETUP([route tables IPv6 -- overlapping subnets]) >>>> +ovn_start >>>> + >>>> +# Logical network: >>>> +# >>>> +# ls1 (2001:db8:1::/64) - lrp-lr1-ls1 -\ /- lrp-lr1-ls2 - ls2 >> (2001:db8:2::/64) >>>> +# lr1 >>>> +# ls3 (2001:db8:3::/64) - lrp-lr1-ls3 -/ \- lrp-lr1-ls4 - ls4 >> (2001:db8:4::/64) >>>> +# >>>> +# ls1 has lsp11 (2001:db8:1::11) >>>> +# ls2 has lsp21 (2001:db8:2::21) >>>> +# ls3 has lsp31 (2001:db8:3::31) >>>> +# ls4 has lsp41 (2001:db8:4::41) >>>> +# >>>> +# lrp-lr1-ls1 set options:route_table=rtb-1 >>>> +# lrp-lr1-ls2 set options:route_table=rtb-2 >>>> +# >>>> +# Static routes on lr1: >>>> +# 2001:db8:2000::/64 nexthop 2001:db8:3::31 route_table=rtb-1 >>>> +# 2001:db8:2000::/64 nexthop 2001:db8:3::41 route_table=rtb-2 >>>> +# >>>> +# Test 1: >>>> +# lsp11 send packet to 2001:db8:2000::1 >>>> +# >>>> +# Expected result: >>>> +# lsp31 should receive traffic, lsp41 should not receive any traffic >>>> +# >>>> +# Test 2: >>>> +# lsp21 send packet to 2001:db8:2000::1 >>>> +# >>>> +# Expected result: >>>> +# lsp41 should receive traffic, lsp31 should not receive any traffic >>>> + >>>> +ovn-nbctl lr-add lr1 >>>> + >>>> +# Create logical topology >>>> +for i in $(seq 1 4); do >>>> + ovn-nbctl ls-add ls${i} >>>> + ovn-nbctl lrp-add lr1 lrp-lr1-ls${i} 00:00:00:01:0${i}:01 >> 2001:db8:${i}::1/64 >>>> + ovn-nbctl lsp-add ls${i} lsp-ls${i}-lr1 -- lsp-set-type >> lsp-ls${i}-lr1 router \ >>>> + -- lsp-set-options lsp-ls${i}-lr1 router-port=lrp-lr1-ls${i} \ >>>> + -- lsp-set-addresses lsp-ls${i}-lr1 router >>>> + ovn-nbctl lsp-add ls$i lsp${i}1 -- \ >>>> + lsp-set-addresses lsp${i}1 "f0:00:00:00:0${i}:1${i} >> 2001:db8:${i}::${i}1" >>>> +done >>>> + >>>> +# install static routes >>>> +ovn-nbctl --route-table=rtb-1 lr-route-add lr1 2001:db8:2000::/64 >> 2001:db8:3::31 >>>> +ovn-nbctl --route-table=rtb-2 lr-route-add lr1 2001:db8:2000::/64 >> 2001:db8:4::41 >>>> + >>>> +# set lrp-lr1-ls{1,2} route tables >>>> +ovn-nbctl lrp-set-options lrp-lr1-ls1 route_table=rtb-1 >>>> +ovn-nbctl lrp-set-options lrp-lr1-ls2 route_table=rtb-2 >>>> + >>>> +net_add n1 >>>> +sim_add hv1 >>>> +as hv1 >>>> +ovs-vsctl add-br br-phys >>>> +ovn_attach n1 br-phys 192.168.0.1 >>>> + >>>> +for i in $(seq 1 4); do >>>> + ovs-vsctl -- add-port br-int hv1-vif${i} -- \ >>>> + set interface hv1-vif${i} external-ids:iface-id=lsp${i}1 \ >>>> + options:tx_pcap=hv1/vif${i}-tx.pcap \ >>>> + options:rxq_pcap=hv1/vif${i}-rx.pcap \ >>>> + ofport-request=${i} >>>> +done >>>> + >>>> +# wait for earlier changes to take effect >>>> +AT_CHECK([ovn-nbctl --timeout=3 --wait=hv sync], [0], [ignore]) >>>> + >>>> +# lsp31 should recieve packet coming from lsp11 >>>> +# lsp41 should recieve packet coming from lsp21 >>>> +for i in $(seq 1 2); do >>>> + di=$(( i + 2)) # dst index >>>> + ri=$(( 5 - i)) # reverse index >>>> + packet="inport==\"lsp${i}1\" && eth.src==f0:00:00:00:0${i}:1${i} && >>>> + eth.dst==00:00:00:01:0${i}:01 && ip6 && ip.ttl==64 && >>>> + ip6.src==2001:db8:${i}::${i}1 && ip6.dst==2001:db8:2000::1 >> && icmp6" >>>> + AT_CHECK([as hv1 ovs-appctl -t ovn-controller inject-pkt >> "$packet"]) >>>> + >>>> + # Assume all packets go to lsp${di}1. >>>> + exp_packet="eth.src==00:00:00:01:0${di}:01 && >> eth.dst==f0:00:00:00:0${di}:1${di} && ip6 && >>>> + ip.ttl==63 && ip6.src==2001:db8:${i}::${i}1 && >> ip6.dst==2001:db8:2000::1 && icmp6" >>>> + echo $exp_packet | ovstest test-ovn expr-to-packets >> >> expected_lsp${di}1 >>>> + > expected_lsp${ri}1 >>>> + >>>> + OVS_WAIT_UNTIL([ >>>> + rcv_n1=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" >> hv1/vif${di}-tx.pcap > lsp${di}1.packets && cat lsp${di}1.packets | wc -l` >>>> + rcv_n2=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" >> hv1/vif${ri}-tx.pcap > lsp${ri}1.packets && cat lsp${ri}1.packets | wc -l` >>>> + echo $rcv_n1 $rcv_n2 >>>> + test $rcv_n1 -eq 1 -a $rcv_n2 -eq 0]) >>>> + >>>> + for j in "${di}1" "${ri}1"; do >>>> + sort expected_lsp${j} > expout >>>> + AT_CHECK([cat lsp${j}.packets | sort], [0], [expout]) >>>> + done >>>> + >>>> + # cleanup tx pcap files >>>> + for j in "${di}1" "${ri}1"; do >>>> + ovs-vsctl -- remove interface hv1-vif${di} options tx_pcap >>>> + > hv1/vif${di}-tx.pcap >>>> + ovs-vsctl -- set interface hv1-vif${di} >> external-ids:iface-id=lsp${di}1 \ >>>> + options:tx_pcap=hv1/vif${di}-tx.pcap >>>> + done >>>> +done >>>> + >>>> +OVN_CLEANUP([hv1]) >>>> +AT_CLEANUP >>>> +]) >>>> + >>>> + >>>> OVN_FOR_EACH_NORTHD([ >>>> AT_SETUP([forwarding group: 3 HVs, 1 LR, 2 LS]) >>>> AT_KEYWORDS([forwarding-group]) >>>> @@ -23332,7 +23759,7 @@ ovn-sbctl dump-flows > sbflows >>>> AT_CAPTURE_FILE([sbflows]) >>>> AT_CAPTURE_FILE([offlows]) >>>> OVS_WAIT_UNTIL([ >>>> - as hv1 ovs-ofctl dump-flows br-int table=20 > offlows >>>> + as hv1 ovs-ofctl dump-flows br-int table=21 > offlows >>>> test $(grep -c "load:0x64->NXM_NX_PKT_MARK" offlows) = 1 && \ >>>> test $(grep -c "load:0x3->NXM_NX_PKT_MARK" offlows) = 1 && \ >>>> test $(grep -c "load:0x4->NXM_NX_PKT_MARK" offlows) = 1 && \ >>>> @@ -23425,12 +23852,12 @@ send_ipv4_pkt hv1 hv1-vif1 505400000003 >> 00000000ff01 \ >>>> $(ip_to_hex 10 0 0 3) $(ip_to_hex 172 168 0 120) >>>> >>>> OVS_WAIT_UNTIL([ >>>> - test 1 -eq $(as hv1 ovs-ofctl dump-flows br-int table=20 | \ >>>> + test 1 -eq $(as hv1 ovs-ofctl dump-flows br-int table=21 | \ >>>> grep "load:0x2->NXM_NX_PKT_MARK" -c) >>>> ]) >>>> >>>> AT_CHECK([ >>>> - test 0 -eq $(as hv1 ovs-ofctl dump-flows br-int table=20 | \ >>>> + test 0 -eq $(as hv1 ovs-ofctl dump-flows br-int table=21 | \ >>>> grep "load:0x64->NXM_NX_PKT_MARK" -c) >>>> ]) >>>> >>>> @@ -24133,7 +24560,7 @@ AT_CHECK([ >>>> grep "priority=100" | \ >>>> grep -c >> "ct(commit,zone=NXM_NX_REG11\\[[0..15\\]],.*exec(move:NXM_OF_ETH_SRC\\[[\\]]->NXM_NX_CT_LABEL\\[[32..79\\]],load:0x[[0-9]]->NXM_NX_CT_LABEL\\[[80..95\\]]))" >>>> >>>> - grep table=22 hv${hv}flows | \ >>>> + grep table=23 hv${hv}flows | \ >>>> grep "priority=200" | \ >>>> grep -c >> "actions=move:NXM_NX_CT_LABEL\\[[32..79\\]]->NXM_OF_ETH_DST\\[[\\]]" >>>> done; :], [0], [dnl >>>> @@ -24258,7 +24685,7 @@ AT_CHECK([ >>>> grep "priority=100" | \ >>>> grep -c >> "ct(commit,zone=NXM_NX_REG11\\[[0..15\\]],.*exec(move:NXM_OF_ETH_SRC\\[[\\]]->NXM_NX_CT_LABEL\\[[32..79\\]],load:0x[[0-9]]->NXM_NX_CT_LABEL\\[[80..95\\]]))" >>>> >>>> - grep table=22 hv${hv}flows | \ >>>> + grep table=23 hv${hv}flows | \ >>>> grep "priority=200" | \ >>>> grep -c >> "actions=move:NXM_NX_CT_LABEL\\[[32..79\\]]->NXM_OF_ETH_DST\\[[\\]]" >>>> done; :], [0], [dnl >>>> @@ -24880,7 +25307,7 @@ AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | >> grep "actions=controller" | grep >>>> ]) >>>> >>>> # The packet should've been dropped in the lr_in_arp_resolve stage. >>>> -AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | grep -E "table=22, >> n_packets=1,.* priority=1,ip,metadata=0x${sw_key},nw_dst=10.0.1.1 >> actions=drop" -c], [0], [dnl >>>> +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | grep -E "table=23, >> n_packets=1,.* priority=1,ip,metadata=0x${sw_key},nw_dst=10.0.1.1 >> actions=drop" -c], [0], [dnl >>>> 1 >>>> ]) >>>> >>>> diff --git a/utilities/ovn-nbctl.c b/utilities/ovn-nbctl.c >>>> index e34bb65f7..0ff10618b 100644 >>>> --- a/utilities/ovn-nbctl.c >>>> +++ b/utilities/ovn-nbctl.c >>>> @@ -329,6 +329,8 @@ Logical router port commands:\n\ >>>> add logical port PORT on ROUTER\n\ >>>> lrp-set-gateway-chassis PORT CHASSIS [PRIORITY]\n\ >>>> set gateway chassis for port PORT\n\ >>>> + lrp-set-options PORT KEY=VALUE [KEY=VALUE]...\n\ >>>> + set router port options\n\ >>>> lrp-del-gateway-chassis PORT CHASSIS\n\ >>>> delete gateway chassis from port PORT\n\ >>>> lrp-get-gateway-chassis PORT\n\ >>>> @@ -351,11 +353,17 @@ Logical router port commands:\n\ >>>> ('overlay' or 'bridged')\n\ >>>> \n\ >>>> Route commands:\n\ >>>> - [--policy=POLICY] [--ecmp] [--ecmp-symmetric-reply] lr-route-add >> ROUTER \n\ >>>> - PREFIX NEXTHOP [PORT]\n\ >>>> + [--policy=POLICY]\n\ >>>> + [--ecmp]\n\ >>>> + [--ecmp-symmetric-reply]\n\ >>>> + [--route-table=ROUTE_TABLE]\n\ >>>> + lr-route-add ROUTER PREFIX NEXTHOP [PORT]\n\ >>>> add a route to ROUTER\n\ >>>> - [--policy=POLICY] lr-route-del ROUTER [PREFIX [NEXTHOP [PORT]]]\n\ >>>> + [--policy=POLICY]\n\ >>>> + [--route-table=ROUTE_TABLE]\n\ >>>> + lr-route-del ROUTER [PREFIX [NEXTHOP [PORT]]]\n\ >>>> remove routes from ROUTER\n\ >>>> + [--route-table=ROUTE_TABLE]\n\ >>>> lr-route-list ROUTER print routes for ROUTER\n\ >>>> \n\ >>>> Policy commands:\n\ >>>> @@ -743,6 +751,11 @@ print_lr(const struct nbrec_logical_router *lr, >> struct ds *s) >>>> ds_put_cstr(s, "]\n"); >>>> } >>>> >>>> + const char *route_table = smap_get(&lrp->options, >> "route_table"); >>>> + if (route_table) { >>>> + ds_put_format(s, " route-table: %s\n", route_table); >>>> + } >>>> + >>>> if (lrp->n_gateway_chassis) { >>>> const struct nbrec_gateway_chassis **gcs; >>>> >>>> @@ -862,6 +875,7 @@ nbctl_pre_show(struct ctl_context *ctx) >>>> ovsdb_idl_add_column(ctx->idl, >> &nbrec_logical_router_port_col_name); >>>> ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_mac); >>>> ovsdb_idl_add_column(ctx->idl, >> &nbrec_logical_router_port_col_networks); >>>> + ovsdb_idl_add_column(ctx->idl, >> &nbrec_logical_router_port_col_options); >>>> ovsdb_idl_add_column(ctx->idl, >> &nbrec_logical_router_port_col_gateway_chassis); >>>> >>>> ovsdb_idl_add_column(ctx->idl, >> &nbrec_gateway_chassis_col_chassis_name); >>>> @@ -4000,11 +4014,19 @@ nbctl_lr_policy_list(struct ctl_context *ctx) >>>> >>>> static struct nbrec_logical_router_static_route * >>>> nbctl_lr_get_route(const struct nbrec_logical_router *lr, char *prefix, >>>> - char *next_hop, bool is_src_route, bool ecmp) >>>> + char *next_hop, bool is_src_route, bool ecmp, >>>> + char *route_table) >>>> { >>>> for (int i = 0; i < lr->n_static_routes; i++) { >>>> struct nbrec_logical_router_static_route *route = >> lr->static_routes[i]; >>>> >>>> + /* Strict compare for route_table. >>>> + * If route_table was not specified, >>>> + * lookup for routes with empty route_table value. */ >>>> + if (strcmp(route->route_table, route_table ? route_table : >> "")) { >>>> + continue; >>>> + } >>>> + >>>> /* Compare route policy. */ >>>> char *nb_policy = route->policy; >>>> bool nb_is_src_route = false; >>>> @@ -4060,6 +4082,8 @@ nbctl_pre_lr_route_add(struct ctl_context *ctx) >>>> &nbrec_logical_router_static_route_col_bfd); >>>> ovsdb_idl_add_column(ctx->idl, >>>> >> &nbrec_logical_router_static_route_col_options); >>>> + ovsdb_idl_add_column(ctx->idl, >>>> + >> &nbrec_logical_router_static_route_col_route_table); >>>> } >>>> >>>> static char * OVS_WARN_UNUSED_RESULT >>>> @@ -4090,6 +4114,7 @@ nbctl_lr_route_add(struct ctl_context *ctx) >>>> } >>>> } >>>> >>>> + char *route_table = shash_find_data(&ctx->options, >> "--route-table"); >>>> bool v6_prefix = false; >>>> prefix = normalize_ipv4_prefix_str(ctx->argv[2]); >>>> if (!prefix) { >>>> @@ -4166,7 +4191,8 @@ nbctl_lr_route_add(struct ctl_context *ctx) >>>> bool ecmp = shash_find(&ctx->options, "--ecmp") != NULL || >>>> ecmp_symmetric_reply; >>>> struct nbrec_logical_router_static_route *route = >>>> - nbctl_lr_get_route(lr, prefix, next_hop, is_src_route, ecmp); >>>> + nbctl_lr_get_route(lr, prefix, next_hop, is_src_route, ecmp, >>>> + route_table); >>>> >>>> /* Validations for nexthop = "discard" */ >>>> if (is_discard_route) { >>>> @@ -4230,7 +4256,8 @@ nbctl_lr_route_add(struct ctl_context *ctx) >>>> } >>>> >>>> struct nbrec_logical_router_static_route *discard_route = >>>> - nbctl_lr_get_route(lr, prefix, "discard", is_src_route, true); >>>> + nbctl_lr_get_route(lr, prefix, "discard", is_src_route, true, >>>> + route_table); >>>> if (discard_route) { >>>> ctl_error(ctx, "discard nexthop for the same ECMP route >> exists."); >>>> goto cleanup; >>>> @@ -4246,6 +4273,9 @@ nbctl_lr_route_add(struct ctl_context *ctx) >>>> if (policy) { >>>> nbrec_logical_router_static_route_set_policy(route, policy); >>>> } >>>> + if (route_table) { >>>> + nbrec_logical_router_static_route_set_route_table(route, >> route_table); >>>> + } >>>> >>>> if (ecmp_symmetric_reply) { >>>> const struct smap options = SMAP_CONST1(&options, >>>> @@ -4289,6 +4319,8 @@ nbctl_pre_lr_route_del(struct ctl_context *ctx) >>>> >> &nbrec_logical_router_static_route_col_nexthop); >>>> ovsdb_idl_add_column(ctx->idl, >>>> >> &nbrec_logical_router_static_route_col_output_port); >>>> + ovsdb_idl_add_column(ctx->idl, >>>> + >> &nbrec_logical_router_static_route_col_route_table); >>>> >>>> } >>>> >>>> @@ -4302,6 +4334,7 @@ nbctl_lr_route_del(struct ctl_context *ctx) >>>> return; >>>> } >>>> >>>> + const char *route_table = shash_find_data(&ctx->options, >> "--route-table"); >>>> const char *policy = shash_find_data(&ctx->options, "--policy"); >>>> bool is_src_route = false; >>>> if (policy) { >>>> @@ -4392,6 +4425,14 @@ nbctl_lr_route_del(struct ctl_context *ctx) >>>> } >>>> } >>>> >>>> + /* Strict compare for route_table. >>>> + * If route_table was not specified, >>>> + * lookup for routes with empty route_table value. */ >>>> + if (strcmp(lr->static_routes[i]->route_table, >>>> + route_table ? route_table : "")) { >>>> + continue; >>>> + } >>>> + >>>> /* Compare output_port, if specified. */ >>>> if (output_port) { >>>> char *rt_output_port = lr->static_routes[i]->output_port; >>>> @@ -5115,6 +5156,41 @@ nbctl_pre_lrp_del_gateway_chassis(struct >> ctl_context *ctx) >>>> ovsdb_idl_add_column(ctx->idl, >> &nbrec_gateway_chassis_col_chassis_name); >>>> } >>>> >>>> +static void >>>> +nbctl_pre_lrp_options(struct ctl_context *ctx) >>>> +{ >>>> + ovsdb_idl_add_column(ctx->idl, >> &nbrec_logical_router_port_col_name); >>>> + ovsdb_idl_add_column(ctx->idl, >> &nbrec_logical_router_port_col_options); >>>> +} >>>> + >>>> +static void >>>> +nbctl_lrp_set_options(struct ctl_context *ctx) >>>> +{ >>>> + const char *id = ctx->argv[1]; >>>> + const struct nbrec_logical_router_port *lrp = NULL; >>>> + size_t i; >>>> + struct smap options = SMAP_INITIALIZER(&options); >>>> + >>>> + char *error = lrp_by_name_or_uuid(ctx, id, true, &lrp); >>>> + if (error) { >>>> + ctx->error = error; >>>> + return; >>>> + } >>>> + for (i = 2; i < ctx->argc; i++) { >>>> + char *key, *value; >>>> + value = xstrdup(ctx->argv[i]); >>>> + key = strsep(&value, "="); >>>> + if (value) { >>>> + smap_add(&options, key, value); >>>> + } >>>> + free(key); >>>> + } >>>> + >>>> + nbrec_logical_router_port_set_options(lrp, &options); >>>> + >>>> + smap_destroy(&options); >>>> +} >>>> + >>>> /* Removes logical router port 'lrp->gateway_chassis[idx]'. */ >>>> static void >>>> remove_gc(const struct nbrec_logical_router_port *lrp, size_t idx) >>>> @@ -5891,6 +5967,7 @@ route_cmp_details(const struct >> nbrec_logical_router_static_route *r1, >>>> } >>>> return r1->output_port ? 1 : -1; >>>> } >>>> + >>>> struct ipv4_route { >>>> int priority; >>>> ovs_be32 addr; >>>> @@ -5900,6 +5977,11 @@ struct ipv4_route { >>>> static int >>>> __ipv4_route_cmp(const struct ipv4_route *r1, const struct ipv4_route >> *r2) >>>> { >>>> + int rtb_cmp = strcmp(r1->route->route_table, >>>> + r2->route->route_table); >>>> + if (rtb_cmp) { >>>> + return rtb_cmp; >>>> + } >>>> if (r1->priority != r2->priority) { >>>> return r1->priority > r2->priority ? -1 : 1; >>>> } >>>> @@ -5931,6 +6013,11 @@ struct ipv6_route { >>>> static int >>>> __ipv6_route_cmp(const struct ipv6_route *r1, const struct ipv6_route >> *r2) >>>> { >>>> + int rtb_cmp = strcmp(r1->route->route_table, >>>> + r2->route->route_table); >>>> + if (rtb_cmp) { >>>> + return rtb_cmp; >>>> + } >>>> if (r1->priority != r2->priority) { >>>> return r1->priority > r2->priority ? -1 : 1; >>>> } >>>> @@ -6018,6 +6105,8 @@ nbctl_pre_lr_route_list(struct ctl_context *ctx) >>>> >> &nbrec_logical_router_static_route_col_options); >>>> ovsdb_idl_add_column(ctx->idl, >>>> &nbrec_logical_router_static_route_col_bfd); >>>> + ovsdb_idl_add_column(ctx->idl, >>>> + >> &nbrec_logical_router_static_route_col_route_table); >>>> } >>>> >>>> static void >>>> @@ -6035,12 +6124,17 @@ nbctl_lr_route_list(struct ctl_context *ctx) >>>> return; >>>> } >>>> >>>> + char *route_table = shash_find_data(&ctx->options, >> "--route-table"); >>>> + >>>> ipv4_routes = xmalloc(sizeof *ipv4_routes * lr->n_static_routes); >>>> ipv6_routes = xmalloc(sizeof *ipv6_routes * lr->n_static_routes); >>>> >>>> for (int i = 0; i < lr->n_static_routes; i++) { >>>> const struct nbrec_logical_router_static_route *route >>>> = lr->static_routes[i]; >>>> + if (route_table && strcmp(route->route_table, route_table)) { >>>> + continue; >>>> + } >>>> unsigned int plen; >>>> ovs_be32 ipv4; >>>> const char *policy = route->policy ? route->policy : "dst-ip"; >>>> @@ -6081,6 +6175,7 @@ nbctl_lr_route_list(struct ctl_context *ctx) >>>> if (n_ipv4_routes) { >>>> ds_put_cstr(&ctx->output, "IPv4 Routes\n"); >>>> } >>>> + const struct nbrec_logical_router_static_route *route; >>>> for (int i = 0; i < n_ipv4_routes; i++) { >>>> bool ecmp = false; >>>> if (i < n_ipv4_routes - 1 && >>>> @@ -6091,6 +6186,15 @@ nbctl_lr_route_list(struct ctl_context *ctx) >>>> &ipv4_routes[i - 1])) { >>>> ecmp = true; >>>> } >>>> + >>>> + route = ipv4_routes[i].route; >>>> + if (!i || (i > 0 && strcmp(route->route_table, >>>> + ipv4_routes[i - >> 1].route->route_table))) { >>>> + ds_put_format(&ctx->output, "%sRoute Table %s:\n", i ? >> "\n" : "", >>>> + strlen(route->route_table) ? >> route->route_table >>>> + : "global"); >>>> + } >>>> + >>>> print_route(ipv4_routes[i].route, &ctx->output, ecmp); >>>> } >>>> >>>> @@ -6108,6 +6212,15 @@ nbctl_lr_route_list(struct ctl_context *ctx) >>>> &ipv6_routes[i - 1])) { >>>> ecmp = true; >>>> } >>>> + >>>> + route = ipv6_routes[i].route; >>>> + if (!i || (i > 0 && strcmp(route->route_table, >>>> + ipv6_routes[i - >> 1].route->route_table))) { >>>> + ds_put_format(&ctx->output, "%sRoute Table %s:\n", i ? >> "\n" : "", >>>> + strlen(route->route_table) ? >> route->route_table >>>> + : "global"); >>>> + } >>>> + >>>> print_route(ipv6_routes[i].route, &ctx->output, ecmp); >>>> } >>>> >>>> @@ -6926,6 +7039,8 @@ static const struct ctl_command_syntax >> nbctl_commands[] = { >>>> "PORT CHASSIS [PRIORITY]", >>>> nbctl_pre_lrp_set_gateway_chassis, nbctl_lrp_set_gateway_chassis, >>>> NULL, "--may-exist", RW }, >>>> + { "lrp-set-options", 1, INT_MAX, "PORT KEY=VALUE [KEY=VALUE]...", >>>> + nbctl_pre_lrp_options, nbctl_lrp_set_options, NULL, "", RW }, >>>> { "lrp-del-gateway-chassis", 2, 2, "PORT CHASSIS", >>>> nbctl_pre_lrp_del_gateway_chassis, nbctl_lrp_del_gateway_chassis, >>>> NULL, "", RW }, >>>> @@ -6949,12 +7064,13 @@ static const struct ctl_command_syntax >> nbctl_commands[] = { >>>> /* logical router route commands. */ >>>> { "lr-route-add", 3, 4, "ROUTER PREFIX NEXTHOP [PORT]", >>>> nbctl_pre_lr_route_add, nbctl_lr_route_add, NULL, >>>> - "--may-exist,--ecmp,--ecmp-symmetric-reply,--policy=,--bfd?", RW >> }, >>>> + >> "--may-exist,--ecmp,--ecmp-symmetric-reply,--policy=,--route-table=,--bfd?", >>>> + RW }, >>>> { "lr-route-del", 1, 4, "ROUTER [PREFIX [NEXTHOP [PORT]]]", >>>> nbctl_pre_lr_route_del, nbctl_lr_route_del, NULL, >>>> - "--if-exists,--policy=", RW }, >>>> + "--if-exists,--policy=,--route-table=", RW }, >>>> { "lr-route-list", 1, 1, "ROUTER", nbctl_pre_lr_route_list, >>>> - nbctl_lr_route_list, NULL, "", RO }, >>>> + nbctl_lr_route_list, NULL, "--route-table=", RO }, >>>> >>>> /* Policy commands */ >>>> { "lr-policy-add", 4, INT_MAX, >>>> -- >>>> 2.30.0 >>>> >>>> _______________________________________________ >>>> dev mailing list >>>> dev@openvswitch.org >>>> https://mail.openvswitch.org/mailman/listinfo/ovs-dev >>> >>> > _______________________________________________ > dev mailing list > dev@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
On Fri, Oct 15, 2021 at 2:36 AM Vladislav Odintsov <odivlad@gmail.com> wrote: > > > Regards, > Vladislav Odintsov > > On 15 Oct 2021, at 08:42, Han Zhou <hzhou@ovn.org> wrote: > > On Thu, Oct 14, 2021 at 12:58 AM Vladislav Odintsov <odivlad@gmail.com> > wrote: > > > Hi Han, > > Thanks for the review. > > Regards, > Vladislav Odintsov > > On 14 Oct 2021, at 08:13, Han Zhou <hzhou@ovn.org> wrote: > > > > On Tue, Oct 5, 2021 at 1:26 PM Vladislav Odintsov <odivlad@gmail.com> > > wrote: > > > This patch extends Logical Router's routing functionality. > Now user may create multiple routing tables within a Logical Router > and assign them to Logical Router Ports. > > Traffic coming from Logical Router Port with assigned route_table > is checked against global routes if any (Logical_Router_Static_Routes > whith empty route_table field), next against directly connected routes > > > This is not accurate. The "directly connected routes" is NOT after the > > global routes. Their priority only depends on the prefix length. > > > and then Logical_Router_Static_Routes with same route_table value as > in Logical_Router_Port options:route_table field. > > A new Logical Router ingress table #10 is added - IN_IP_ROUTING_PRE. > In this table packets which come from LRPs with configured > options:route_table field are checked against inport and in OVS > register 7 unique non-zero value identifying route table is written. > > Then in 11th table IN_IP_ROUTING routes which have non-empty > `route_table` field are added with additional match on reg7 value > associated with appropriate route_table. > > > Hi Vladislav, > > First of all, sorry for the delayed review, and thanks for implementing > > this new feature. > > > I have some questions regarding the feature itself. I remember that we > > had some discussion earlier for this feature, but it seems I misunderstood > the feature you are implementing here. I thought by multiple routing tables > you were trying to support something like VRF in physical routers, but this > seems to be something different. According to your implementation, instead > of assigning LRPs to different routing tables, a LRP can actually > participate to both the global routing table and the table with a specific > ID. For ingress, global routes are prefered over other routes; for egress > (i.e. forwarding to the next hop), it doesn't even enforce any table-id > check, so packet routed by a entry of table-X can go out of a LRP with > table-id Y. Is my understanding correct about the desired behavior of this > feature? > > > > Yes, your understanding is correct. > This is not VRF. At first glance VRF can be done just by using LR-per-VRF > > without any code modifications (maybe there are corner cases, but it’s not > something I’m currently working on). > > I agree VRF can be achieved by just separate LRs. I am trying to understand > why multiple LRs wouldn't satisfy your use case here. > > LRP can be optionally assigned to specific routing table name. This means > > that for LR ingress pipeline packets coming from this specific LRP would be > checked against routes with same route_table value within appropriate LR. > This is some kind of PBR, analog of "ip rule add iif <interface name> > lookup <id>". > > There is one specific use-case, which requires special handling: > > directly-connected routes (subnet CIDRs from connected to this LR LRPs). > These routes can’t be added manually by user, though routing between LRPs > belonging to different routing tables is still needed. So, these routes > should be added to global routing table to override routing for LRPs with > configured routing tables. If for some reason user wants to prohibit IP > connectivity to any LRP (honestly, I can’t imagine, why), it can be done by > utilising OVN PBR with drop action or a route with "discard" nexthop. But > I’d think in this case whether this LRP is needed in this LR. > > Also, if user wants to create routes, which apply for all LRPs from all > > routing tables, it can be done by adding new entries to global routing > table. > > And the third reason to have global routing table - a fully > > backward-compatible solution. All users, who don’t need multiple routing > tables support just continue using static routing in the same manner > without any changes. > > > > If this is correct, it doesn't seem to be a common/standard requirement > > (or please educate me if I am wrong). Could you explain a little more about > the actual use cases: what kind of real world problems need to be solved by > this feature or how are you going to use this. For example, why would a > port need to participate in both routing tables? It looks like what you > really need is policy routing instead of multiple isolated routing tables. > I understand that you already use policy routing to implement ACLs, so it > is not convenient to combine other policies (e.g. inport based routing) > into the policy routing stage. If that's the case, would it be more generic > to support multiple policy routing stages? My concern to the current > approach is that it is implemented for a very special use case. It makes > the code more complex but when there is a slightly different requirement in > the future it becomes insufficient. I am thinking that policy routing seems > more flexible and has more potential to be made more generic. Maybe I will > have a better understanding when I hear more detailed use cases and > considerations from you. > > > > I can't agree here in it’s uncommon requirement. > This implementation was inspired by AWS Route Tables feature [1]: within > > a VPC (LR in terms of OVN) user may create multiple routing tables and > assign them to different subnets (LRPs) in multiple availability zones. > Auto-generated directly-connected routes from LRPs CIDRs are working in the > same manner as they do in AWS - apply to all subnets regardless of their > association to routing table. GCP has similar behaviour: [2], Azure, I > guess, too. > > Our public cloud (CROC Cloud Platform) supports AWS behaviour [3], so I > > primarily was oriented on it. Internally we already use this feature and > currently it fits our use-case, but again I can't say it is specific. > > If it is for AWS/GCP alike routing features, then from what I understand > what's implemented in this patch is a little different. To implement a VPC > model in OVN, I think it is ok to have multiple LRs instead of only one LR > for each VPC. So naturally I would use a LR to map to AWS's routing table. > In AWS's document (the link you provided), it says: > > "A subnet can only be associated with one route table at a time" > > So basically in AWS a subnet is either attached to the default/main route > table or a custom table, but not both, right? However, in your use case, a > LRP (maps to a subnet) attaches to both "main" and a custom table, which > seems not common to me. Or did I miss something? > > > That’s true about AWS, but there is still a bit not accurate about OVN. > Global routing table in OVN terms is not that AWS main route table is. > Main route table is just a configuration hint for users for implicit route > tables association with subnets. > Implicitly-associated via main routing table subnets routing functions the > same manner as a normal explicit route_table-subnet association. > > Global routing table in OVN is just a list of routes with higher priority > than routes with configured "route_table". > > I do not offer to configure both tables at the same time. But it is > _possible_ to do if required for some reason (for instance to configure > some service chaining or just internal VPC services like metadata/another > internal APIs, access to another services). > Normally, if we talk about AWS Route Table to OVN, it is mostly one-to-one > mapping, except "Local route": > > Example: > AWS Route Table rtb-xxx: > 172.31.0.0/16: local route (VPC CIDR for subnets) > 0.0.0.0/0: igw-XXX (internet gateway) > > AWS Route Table rtb-yyy: > 172.31.0.0/16: local route (VPC CIDR for subnets) > 5.5.5.5/32: instance-xxx > > This maps to OVN configuration (for one LR with one subnet 172.31.0.0/24): > > implicit route (not present in logical router static routes table): > 172.31.0.0/24: LRP-subnet-xxx - has highest priority over other route > table-oriented routes and can be threat as placed in global routing table > What do you mean by highest priority here? It all depends on the prefix length, right? If you have a static route entry that says: 172.31.0.0./25 -> xyz, then this route will have higher priority than the directly connected subnet. > > Normal static routes: > ip_prefix: 0.0.0.0/0, nexthop: <IP for edge LR’s LRP>, route_table: > rtb-xxx > ip_prefix: 5.5.5.5/32, nexthop: <IP of some LSP, which belongs to > VM/container via which route is built>, route_table: rtb-yyy > > I guess, I understood the reason for misunderstanding: the global routing > table, which I referred earlier is a routing table which has no value in > "route_table" field and directly-connected routes at the same time. Latter > have no records in logical router static routes, but I still referred them > as a routes from "global routing table". I can think about terminology > here, if it’s a root cause for misunderstanding. What do you think? > Thanks for explaining. In fact I understand what you mean about "global routing table", but I think you are implementing something different from what AWS provides, primarily because you use a single LR instead of LR per routing table. I understand there are reasons why you want to do it this way, but here I can think of some challenges of your approach. For example, in AWS we could do: route table main: subnet S0: 172.31.0.0/24 routes: 172.31.0.0/16: local route table R1: subnet S1: 172.31.1.0/24 routes: 172.31.0.0/16: local 172.31.0.0/24: <some FW/GW> Packet from S1 to S0 will go to FW/GW, where it may be dropped/forwarded to S0/or redirected to something outside of AWS ... While in your approach with OVN, both subnets will be under the same logical router, and with different table IDs assigned to the LRPs and routes. But when a VM under S1 sends a packet to S0, it will go to the destination directly because you are prioritizing the direct-connection routes and they are under the same global route table, and there is no way to force it to go through some FW/GW from the custom route table. You can add a route to achieve this in the global table, because in your design the global routes are still applicable to all LRPs and has higher priority, but it would impact all the LRPs/subnets, while in the AWS design above it is supposed to affect subnets under R1 only. In addition, for my understanding, S0 and S1 in AWS cannot communicate directly, unless there are some specific routes on both route tables to route them to some gateway. In other words, there are some isolations between route tables. However, in your design with OVN, it is more of policy routing without isolation at all. I am not saying that your design is wrong, but just saying it implements very different behavior than AWS, and I am trying to understand your real use cases to have a better understanding of the feature you implemented. (You explained that you just wanted the AWS behavior, but it looks not exactly the case). > > > In my opinion having this feature to be implemented using PBR is less > > convenient and native for users, who are familiar with behaviour for > mentioned above public cloud platforms, because configuring routes should > be done in routes section. And adding route table property seems native in > this route, not in PBR. Moreover, I _think_ using > Logical_Router_Static_Route to extend this feature for OVN-Interconnection > becomes quite easy comparing to PBR (though, I didn’t try the latter). > > I agree if it is just AWS -like requirement, PBR is less convenient. > > I am trying to understand if it can be achieved with separate LRs. If not, > what's special about the requirement, and is the current approach providing > a solution common enough so that more use cases can also benefit from? > Could you clarify a little more? Thanks again. > > That was our initial approach - to use separate LRs for each Route Table. > We rejected that solution because met some difficulties and blocker. See > below: > > Brief topology description if using LRs per Route Table: > Imagine 2 subnets in VPC in 1st AZ, one in another AZ. Each subnet in it’s > own Route Table (LR). > All subnets must have IP connectivity, so we have to somehow interconnect > these Route Table LRs. > That's exactly the same case for AWS, right? If you want direct connectivity, you would just put them under the same route table in AWS (or same LR in OVN), right? > > [BLOCKER] It is impossible to place route in route table 1 via VM from > subnet assiciated to route table 2 if using per-RTB LR. Because in RTB-2 LR > we have to add route from RTB 1 and this breaks route table isolation. > Route Table 2 LR will start looking up routes and there could be routes > from another route tables. This breaks the idea of having LR per Route > Table completely. Here we rejected this solution and moved to adding > support for route tables in OVN. > > Would it be just the same problem in AWS? Why would a user route a subnet of RTB-1 to a VM under RTB-2, and at the same time want route table isolation? With the single LR solution in OVN, you would not get the isolation between the tables, because 1) all LRPs still share the global routing table, 2) for output to nexthop there is no distinction between the output interfaces. > But some more cons: > > 1. complex network topology. Interconnecting all LRs even with some > transit switch is harder than having one LR and all VPC-related > configuration is done in one place (in AZ). > 2. Directly-connected routes. In case we have multiple Route Table LRs, we > have to add route for each subnet from another LR. In case one LR per VPC > all such routes are installed automatically and learnt via ovn-ic. > 3. PBR, LBs. It’s much easier to implement PBR and configuring Load > Balancers in one LR, than in multiple. > 4. There could be very many LRs, LRPs, LSPs (datapaths in SB) - excess > database records, huge database growth. > 5. Extra client code complexity (updating routes required configuring > routes in many LRs on different availibility zones); > 5. We couldn’t use ovn-ic routes learning, because VPC requires out-of-box > IP connectivity between subnets, and if connect Route Table LRs between > AZs, because connecting multiple LRs from different Route Tables would > learn routes without binding to route table. This requires additional > isolation at the transit switches level. > 6. There were some more problems. Here are listed some, which I could > refresh in my mind. > > From our half-of-a-year experience using LR per VPC is very comfortable > and it looks quite extendable it terms of network features. > > Let me know if this makes sense. > > Thanks for the list of the cons! Most of the cons you listed above seem to apply to the AWS model, too, right? It is ok to have different requirements than AWS, but we'd better define it clearly and it would be helpful with some typical use cases that solve specific problems. I have more information now after your explanation, but still not really clear under what scenarios would this feature be used. @Numan Siddique <numans@ovn.org> @Mark Michelson <mmichels@redhat.com> could you let me know your thoughts on this, too? Would you see similar use cases in Redhat customers? Or does this design match well with your potential use cases? I'd like to be careful when implementing something like this and make sure we are doing it the right way. Thanks, Han > Han > > > > I haven't finished reviewing the code yet, but I have one comment > > regarding adding 100 to the priority of the global routes. For IPv6, the > priority range from 0 to 120x2=240, so adding 100 is not enough. It would > create overlapping priority ranges, and some table-id specific route > entries may override the global routes. > > > > Thanks, I’ll dig into this when you finish review. > > > Let me know if I answered your questions or if you have new ones. > Again many thanks for your time and digging into this patch series. > > 1: https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Route_Tables.html > 2: https://cloud.google.com/vpc/docs/routes#subnet-routes > 3: https://docs.cloud.croc.ru/en/services/networks/routetables.html > > Thanks, > Han > > > Signed-off-by: Vladislav Odintsov <odivlad@gmail.com> > Acked-by: Numan Siddique <numans@ovn.org> > --- > northd/northd.c | 159 ++++++++++++--- > northd/ovn-northd.8.xml | 63 ++++-- > ovn-nb.ovsschema | 5 +- > ovn-nb.xml | 30 +++ > tests/ovn-ic.at | 4 + > tests/ovn-nbctl.at | 196 +++++++++++++++++- > tests/ovn-northd.at | 76 ++++++- > tests/ovn.at | 441 +++++++++++++++++++++++++++++++++++++++- > utilities/ovn-nbctl.c | 134 +++++++++++- > 9 files changed, 1041 insertions(+), 67 deletions(-) > > diff --git a/northd/northd.c b/northd/northd.c > index 092eca829..6a020cb2e 100644 > --- a/northd/northd.c > +++ b/northd/northd.c > @@ -148,15 +148,16 @@ enum ovn_stage { > PIPELINE_STAGE(ROUTER, IN, ECMP_STATEFUL, 7, > > "lr_in_ecmp_stateful") \ > > PIPELINE_STAGE(ROUTER, IN, ND_RA_OPTIONS, 8, > > "lr_in_nd_ra_options") \ > > PIPELINE_STAGE(ROUTER, IN, ND_RA_RESPONSE, 9, > > "lr_in_nd_ra_response") \ > > - PIPELINE_STAGE(ROUTER, IN, IP_ROUTING, 10, > > "lr_in_ip_routing") \ > > - PIPELINE_STAGE(ROUTER, IN, IP_ROUTING_ECMP, 11, > > "lr_in_ip_routing_ecmp") \ > > - PIPELINE_STAGE(ROUTER, IN, POLICY, 12, "lr_in_policy") > > \ > > - PIPELINE_STAGE(ROUTER, IN, POLICY_ECMP, 13, > > "lr_in_policy_ecmp") \ > > - PIPELINE_STAGE(ROUTER, IN, ARP_RESOLVE, 14, > > "lr_in_arp_resolve") \ > > - PIPELINE_STAGE(ROUTER, IN, CHK_PKT_LEN , 15, > > "lr_in_chk_pkt_len") \ > > - PIPELINE_STAGE(ROUTER, IN, LARGER_PKTS, 16, > > "lr_in_larger_pkts") \ > > - PIPELINE_STAGE(ROUTER, IN, GW_REDIRECT, 17, > > "lr_in_gw_redirect") \ > > - PIPELINE_STAGE(ROUTER, IN, ARP_REQUEST, 18, > > "lr_in_arp_request") \ > > + PIPELINE_STAGE(ROUTER, IN, IP_ROUTING_PRE, 10, > > "lr_in_ip_routing_pre") \ > > + PIPELINE_STAGE(ROUTER, IN, IP_ROUTING, 11, > > "lr_in_ip_routing") \ > > + PIPELINE_STAGE(ROUTER, IN, IP_ROUTING_ECMP, 12, > > "lr_in_ip_routing_ecmp") \ > > + PIPELINE_STAGE(ROUTER, IN, POLICY, 13, "lr_in_policy") > > \ > > + PIPELINE_STAGE(ROUTER, IN, POLICY_ECMP, 14, > > "lr_in_policy_ecmp") \ > > + PIPELINE_STAGE(ROUTER, IN, ARP_RESOLVE, 15, > > "lr_in_arp_resolve") \ > > + PIPELINE_STAGE(ROUTER, IN, CHK_PKT_LEN, 16, > > "lr_in_chk_pkt_len") \ > > + PIPELINE_STAGE(ROUTER, IN, LARGER_PKTS, 17, > > "lr_in_larger_pkts") \ > > + PIPELINE_STAGE(ROUTER, IN, GW_REDIRECT, 18, > > "lr_in_gw_redirect") \ > > + PIPELINE_STAGE(ROUTER, IN, ARP_REQUEST, 19, > > "lr_in_arp_request") \ > > \ > /* Logical router egress stages. */ \ > PIPELINE_STAGE(ROUTER, OUT, UNDNAT, 0, "lr_out_undnat") > > \ > > @@ -225,6 +226,7 @@ enum ovn_stage { > #define REG_NEXT_HOP_IPV6 "xxreg0" > #define REG_SRC_IPV4 "reg1" > #define REG_SRC_IPV6 "xxreg1" > +#define REG_ROUTE_TABLE_ID "reg7" > > #define REG_ORIG_TP_DPORT_ROUTER "reg9[16..31]" > > @@ -287,8 +289,9 @@ enum ovn_stage { > * | R6 | UNUSED | X | | G | > > IN_IP_ROUTING)| > > * | | | R | | 1 | > > | > > * +-----+--------------------------+ E | UNUSED | | > > | > > - * | R7 | UNUSED | G | | | > > | > > - * | | | 3 | | | > > | > > + * | R7 | ROUTE_TABLE_ID | G | | | > > | > > + * | | (>= IN_IP_ROUTING_PRE && | 3 | | | > > | > > + * | | <= IN_IP_ROUTING) | | | | > > | > > * > > > +-----+--------------------------+---+-----------------+---+---------------+ > > * | R8 | ECMP_GROUP_ID | | | > * | | ECMP_MEMBER_ID | X | | > @@ -8511,11 +8514,72 @@ cleanup: > ds_destroy(&actions); > } > > +static uint32_t > +route_table_add(struct simap *route_tables, const char > > *route_table_name) > > +{ > + /* route table ids start from 1 */ > + uint32_t rtb_id = simap_count(route_tables) + 1; > + > + if (rtb_id == UINT16_MAX) { > + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1); > + VLOG_WARN_RL(&rl, "too many route tables for Logical Router."); > + return 0; > + } > + > + if (!simap_put(route_tables, route_table_name, rtb_id)) { > + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1); > + VLOG_WARN_RL(&rl, "Route table id unexpectedly appeared"); > + } > + > + return rtb_id; > +} > + > +static uint32_t > +get_route_table_id(struct simap *route_tables, const char > > *route_table_name) > > +{ > + if (!route_table_name || !strlen(route_table_name)) { > + return 0; > + } > + > + uint32_t rtb_id = simap_get(route_tables, route_table_name); > + if (!rtb_id) { > + rtb_id = route_table_add(route_tables, route_table_name); > + } > + > + return rtb_id; > +} > + > +static void > +build_route_table_lflow(struct ovn_datapath *od, struct hmap *lflows, > + struct nbrec_logical_router_port *lrp, > + struct simap *route_tables) > +{ > + struct ds match = DS_EMPTY_INITIALIZER; > + struct ds actions = DS_EMPTY_INITIALIZER; > + > + const char *route_table_name = smap_get(&lrp->options, > > "route_table"); > > + uint32_t rtb_id = get_route_table_id(route_tables, > > route_table_name); > > + if (!rtb_id) { > + return; > + } > + > + ds_put_format(&match, "inport == \"%s\"", lrp->name); > + ds_put_format(&actions, "%s = %d; next;", > + REG_ROUTE_TABLE_ID, rtb_id); > + > + ovn_lflow_add(lflows, od, S_ROUTER_IN_IP_ROUTING_PRE, 100, > + ds_cstr(&match), ds_cstr(&actions)); > + > + ds_destroy(&match); > + ds_destroy(&actions); > +} > + > struct parsed_route { > struct ovs_list list_node; > struct in6_addr prefix; > unsigned int plen; > bool is_src_route; > + uint32_t route_table_id; > uint32_t hash; > const struct nbrec_logical_router_static_route *route; > bool ecmp_symmetric_reply; > @@ -8540,7 +8604,7 @@ find_static_route_outport(struct ovn_datapath > > *od, struct hmap *ports, > > * Otherwise return NULL. */ > static struct parsed_route * > parsed_routes_add(struct ovn_datapath *od, struct hmap *ports, > - struct ovs_list *routes, > + struct ovs_list *routes, struct simap *route_tables, > const struct nbrec_logical_router_static_route > > *route, > > struct hmap *bfd_connections) > { > @@ -8622,6 +8686,7 @@ parsed_routes_add(struct ovn_datapath *od, struct > > hmap *ports, > > struct parsed_route *pr = xzalloc(sizeof *pr); > pr->prefix = prefix; > pr->plen = plen; > + pr->route_table_id = get_route_table_id(route_tables, > > route->route_table); > > pr->is_src_route = (route->policy && !strcmp(route->policy, > "src-ip")); > pr->hash = route_hash(pr); > @@ -8655,6 +8720,7 @@ struct ecmp_groups_node { > struct in6_addr prefix; > unsigned int plen; > bool is_src_route; > + uint32_t route_table_id; > uint16_t route_count; > struct ovs_list route_list; /* Contains ecmp_route_list_node */ > }; > @@ -8663,7 +8729,7 @@ static void > ecmp_groups_add_route(struct ecmp_groups_node *group, > const struct parsed_route *route) > { > - if (group->route_count == UINT16_MAX) { > + if (group->route_count == UINT16_MAX) { > static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1); > VLOG_WARN_RL(&rl, "too many routes in a single ecmp group."); > return; > @@ -8692,6 +8758,7 @@ ecmp_groups_add(struct hmap *ecmp_groups, > eg->prefix = route->prefix; > eg->plen = route->plen; > eg->is_src_route = route->is_src_route; > + eg->route_table_id = route->route_table_id; > ovs_list_init(&eg->route_list); > ecmp_groups_add_route(eg, route); > > @@ -8705,7 +8772,8 @@ ecmp_groups_find(struct hmap *ecmp_groups, struct > > parsed_route *route) > > HMAP_FOR_EACH_WITH_HASH (eg, hmap_node, route->hash, ecmp_groups) { > if (ipv6_addr_equals(&eg->prefix, &route->prefix) && > eg->plen == route->plen && > - eg->is_src_route == route->is_src_route) { > + eg->is_src_route == route->is_src_route && > + eg->route_table_id == route->route_table_id) { > return eg; > } > } > @@ -8752,7 +8820,8 @@ unique_routes_remove(struct hmap *unique_routes, > HMAP_FOR_EACH_WITH_HASH (ur, hmap_node, route->hash, > > unique_routes) { > > if (ipv6_addr_equals(&route->prefix, &ur->route->prefix) && > route->plen == ur->route->plen && > - route->is_src_route == ur->route->is_src_route) { > + route->is_src_route == ur->route->is_src_route && > + route->route_table_id == ur->route->route_table_id) { > hmap_remove(unique_routes, &ur->hmap_node); > const struct parsed_route *existed_route = ur->route; > free(ur); > @@ -8790,9 +8859,9 @@ build_route_prefix_s(const struct in6_addr > > *prefix, unsigned int plen) > > } > > static void > -build_route_match(const struct ovn_port *op_inport, const char > > *network_s, > > - int plen, bool is_src_route, bool is_ipv4, struct ds > > *match, > > - uint16_t *priority) > +build_route_match(const struct ovn_port *op_inport, uint32_t rtb_id, > + const char *network_s, int plen, bool is_src_route, > + bool is_ipv4, struct ds *match, uint16_t *priority) > { > const char *dir; > /* The priority here is calculated to implement > > longest-prefix-match > > @@ -8808,6 +8877,15 @@ build_route_match(const struct ovn_port > > *op_inport, const char *network_s, > > if (op_inport) { > ds_put_format(match, "inport == %s && ", op_inport->json_key); > } > + if (rtb_id) { > + ds_put_format(match, "%s == %d && ", REG_ROUTE_TABLE_ID, > > rtb_id); > > + } else { > + /* Route-table assigned LRPs' routes should have lower priority > + * in order not to affect directly-connected global routes. > + * So, enlarge non-route-table routes priority by 100. > + */ > + *priority += 100; > + } > ds_put_format(match, "ip%s.%s == %s/%d", is_ipv4 ? "4" : "6", dir, > network_s, plen); > } > @@ -8946,7 +9024,7 @@ add_ecmp_symmetric_reply_flows(struct hmap > > *lflows, > > out_port->lrp_networks.ea_s, > IN6_IS_ADDR_V4MAPPED(&route->prefix) ? "" : "xx", > port_ip, out_port->json_key); > - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_IP_ROUTING, 300, > + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_IP_ROUTING, 400, > ds_cstr(&match), ds_cstr(&actions), > &st_route->header_); > > @@ -8976,8 +9054,8 @@ build_ecmp_route_flow(struct hmap *lflows, struct > > ovn_datapath *od, > > struct ds route_match = DS_EMPTY_INITIALIZER; > > char *prefix_s = build_route_prefix_s(&eg->prefix, eg->plen); > - build_route_match(NULL, prefix_s, eg->plen, eg->is_src_route, > > is_ipv4, > > - &route_match, &priority); > + build_route_match(NULL, eg->route_table_id, prefix_s, eg->plen, > + eg->is_src_route, is_ipv4, &route_match, > > &priority); > > free(prefix_s); > > struct ds actions = DS_EMPTY_INITIALIZER; > @@ -9052,8 +9130,8 @@ static void > add_route(struct hmap *lflows, struct ovn_datapath *od, > const struct ovn_port *op, const char *lrp_addr_s, > const char *network_s, int plen, const char *gateway, > - bool is_src_route, const struct ovsdb_idl_row *stage_hint, > - bool is_discard_route) > + bool is_src_route, const uint32_t rtb_id, > + const struct ovsdb_idl_row *stage_hint, bool > > is_discard_route) > > { > bool is_ipv4 = strchr(network_s, '.') ? true : false; > struct ds match = DS_EMPTY_INITIALIZER; > @@ -9068,8 +9146,8 @@ add_route(struct hmap *lflows, struct > > ovn_datapath *od, > > op_inport = op; > } > } > - build_route_match(op_inport, network_s, plen, is_src_route, > > is_ipv4, > > - &match, &priority); > + build_route_match(op_inport, rtb_id, network_s, plen, is_src_route, > + is_ipv4, &match, &priority); > > struct ds common_actions = DS_EMPTY_INITIALIZER; > struct ds actions = DS_EMPTY_INITIALIZER; > @@ -9132,7 +9210,8 @@ build_static_route_flow(struct hmap *lflows, > > struct ovn_datapath *od, > > char *prefix_s = build_route_prefix_s(&route_->prefix, > > route_->plen); > > add_route(lflows, route_->is_discard_route ? od : out_port->od, > > out_port, > > lrp_addr_s, prefix_s, route_->plen, route->nexthop, > - route_->is_src_route, &route->header_, > > route_->is_discard_route); > > + route_->is_src_route, route_->route_table_id, > > &route->header_, > > + route_->is_discard_route); > > free(prefix_s); > } > @@ -10584,6 +10663,17 @@ build_ND_RA_flows_for_lrouter(struct > > ovn_datapath *od, struct hmap *lflows) > > } > } > > +/* Logical router ingress table IP_ROUTING_PRE: > + * by default goto next. (priority 0). */ > +static void > +build_ip_routing_pre_flows_for_lrouter(struct ovn_datapath *od, > + struct hmap *lflows) > +{ > + if (od->nbr) { > + ovn_lflow_add(lflows, od, S_ROUTER_IN_IP_ROUTING_PRE, 0, "1", > > "next;"); > > + } > +} > + > /* Logical router ingress table IP_ROUTING : IP Routing. > * > * A packet that arrives at this table is an IP packet that should be > @@ -10609,14 +10699,14 @@ build_ip_routing_flows_for_lrouter_port( > for (int i = 0; i < op->lrp_networks.n_ipv4_addrs; i++) { > add_route(lflows, op->od, op, > > op->lrp_networks.ipv4_addrs[i].addr_s, > > op->lrp_networks.ipv4_addrs[i].network_s, > - op->lrp_networks.ipv4_addrs[i].plen, NULL, false, > + op->lrp_networks.ipv4_addrs[i].plen, NULL, > > false, 0, > > &op->nbrp->header_, false); > } > > for (int i = 0; i < op->lrp_networks.n_ipv6_addrs; i++) { > add_route(lflows, op->od, op, > > op->lrp_networks.ipv6_addrs[i].addr_s, > > op->lrp_networks.ipv6_addrs[i].network_s, > - op->lrp_networks.ipv6_addrs[i].plen, NULL, false, > + op->lrp_networks.ipv6_addrs[i].plen, NULL, > > false, 0, > > &op->nbrp->header_, false); > } > } else if (lsp_is_router(op->nbsp)) { > @@ -10639,7 +10729,7 @@ build_ip_routing_flows_for_lrouter_port( > add_route(lflows, peer->od, peer, > peer->lrp_networks.ipv4_addrs[0].addr_s, > laddrs->ipv4_addrs[k].network_s, > - laddrs->ipv4_addrs[k].plen, NULL, false, > + laddrs->ipv4_addrs[k].plen, NULL, false, > > 0, > > &peer->nbrp->header_, false); > } > } > @@ -10659,10 +10749,17 @@ build_static_route_flows_for_lrouter( > struct hmap ecmp_groups = HMAP_INITIALIZER(&ecmp_groups); > struct hmap unique_routes = HMAP_INITIALIZER(&unique_routes); > struct ovs_list parsed_routes = > > OVS_LIST_INITIALIZER(&parsed_routes); > > + struct simap route_tables = SIMAP_INITIALIZER(&route_tables); > struct ecmp_groups_node *group; > + > + for (int i = 0; i < od->nbr->n_ports; i++) { > + build_route_table_lflow(od, lflows, od->nbr->ports[i], > + &route_tables); > + } > + > for (int i = 0; i < od->nbr->n_static_routes; i++) { > struct parsed_route *route = > - parsed_routes_add(od, ports, &parsed_routes, > + parsed_routes_add(od, ports, &parsed_routes, > > &route_tables, > > od->nbr->static_routes[i], > > bfd_connections); > > if (!route) { > continue; > @@ -10695,6 +10792,7 @@ build_static_route_flows_for_lrouter( > ecmp_groups_destroy(&ecmp_groups); > unique_routes_destroy(&unique_routes); > parsed_routes_destroy(&parsed_routes); > + simap_destroy(&route_tables); > } > } > > @@ -12800,6 +12898,7 @@ build_lswitch_and_lrouter_iterate_by_od(struct > > ovn_datapath *od, > > build_neigh_learning_flows_for_lrouter(od, lsi->lflows, > > &lsi->match, > > &lsi->actions, > > lsi->meter_groups); > > build_ND_RA_flows_for_lrouter(od, lsi->lflows); > + build_ip_routing_pre_flows_for_lrouter(od, lsi->lflows); > build_static_route_flows_for_lrouter(od, lsi->lflows, lsi->ports, > lsi->bfd_connections); > build_mcast_lookup_flows_for_lrouter(od, lsi->lflows, &lsi->match, > diff --git a/northd/ovn-northd.8.xml b/northd/ovn-northd.8.xml > index 39f4eaa0c..cc2e25367 100644 > --- a/northd/ovn-northd.8.xml > +++ b/northd/ovn-northd.8.xml > @@ -2899,7 +2899,7 @@ icmp6 { > > <p> > If ECMP routes with symmetric reply are configured in the > - <code>OVN_Northbound</code> database for a gateway router, a > > priority-300 > > + <code>OVN_Northbound</code> database for a gateway router, a > > priority-400 > > flow is added for each router port on which symmetric replies are > configured. The matching logic for these ports essentially > > reverses the > > configured logic of the ECMP route. So for instance, a route > > with a > > @@ -3245,7 +3245,35 @@ output; > </li> > </ul> > > - <h3>Ingress Table 10: IP Routing</h3> > + <h3>Ingress Table 10: IP Routing Pre</h3> > + > + <p> > + If a packet arrived at this table from Logical Router Port > > <var>P</var> > > + which has <code>options:route_table</code> value set, a logical > > flow with > > + match <code>inport == "<var>P</var>"</code> with priority 100 > > and action, > > + setting unique-generated per-datapath 32-bit value (non-zero) in > > OVS > > + register 7. This register is checked in next table. > + </p> > + > + <p> > + This table contains the following logical flows: > + </p> > + > + <ul> > + <li> > + <p> > + Priority-100 flow with match <code>inport == > > "LRP_NAME"</code> value > > + and action, which set route table identifier in reg7. > + </p> > + > + <p> > + A priority-0 logical flow with match <code>1</code> has > > actions > > + <code>next;</code>. > + </p> > + </li> > + </ul> > + > + <h3>Ingress Table 11: IP Routing</h3> > > <p> > A packet that arrives at this table is an IP packet that should > > be > > @@ -3316,10 +3344,10 @@ output; > <p> > IPv4 routing table. For each route to IPv4 network > > <var>N</var> with > > netmask <var>M</var>, on router port <var>P</var> with IP > > address > > - <var>A</var> and Ethernet > - address <var>E</var>, a logical flow with match > > <code>ip4.dst == > > - <var>N</var>/<var>M</var></code>, whose priority is the > > number of > > - 1-bits in <var>M</var>, has the following actions: > + <var>A</var> and Ethernet address <var>E</var>, a logical > > flow with > > + match <code>ip4.dst == <var>N</var>/<var>M</var></code>, > > whose > > + priority is 100 + the number of 1-bits in <var>M</var>, has > > the > > + following actions: > </p> > > <pre> > @@ -3382,6 +3410,13 @@ next; > If the address <var>A</var> is in the link-local scope, the > route will be limited to sending on the ingress port. > </p> > + > + <p> > + For routes with <code>route_table</code> value set > + <code>reg7 == id</code> is prefixed in logical flow match > > portion. > > + Priority for routes with <code>route_table</code> value set > > is > > + the number of 1-bits in <var>M</var>. > + </p> > </li> > > <li> > @@ -3408,7 +3443,7 @@ select(reg8[16..31], <var>MID1</var>, > > <var>MID2</var>, ...); > > </li> > </ul> > > - <h3>Ingress Table 11: IP_ROUTING_ECMP</h3> > + <h3>Ingress Table 12: IP_ROUTING_ECMP</h3> > > <p> > This table implements the second part of IP routing for ECMP > > routes > > @@ -3460,7 +3495,7 @@ outport = <var>P</var>; > </li> > </ul> > > - <h3>Ingress Table 12: Router policies</h3> > + <h3>Ingress Table 13: Router policies</h3> > <p> > This table adds flows for the logical router policies configured > on the logical router. Please see the > @@ -3532,7 +3567,7 @@ next; > </li> > </ul> > > - <h3>Ingress Table 13: ECMP handling for router policies</h3> > + <h3>Ingress Table 14: ECMP handling for router policies</h3> > <p> > This table handles the ECMP for the router policies configured > with multiple nexthops. > @@ -3576,7 +3611,7 @@ outport = <var>P</var> > </li> > </ul> > > - <h3>Ingress Table 14: ARP/ND Resolution</h3> > + <h3>Ingress Table 15: ARP/ND Resolution</h3> > > <p> > Any packet that reaches this table is an IP packet whose next-hop > @@ -3767,7 +3802,7 @@ outport = <var>P</var> > > </ul> > > - <h3>Ingress Table 15: Check packet length</h3> > + <h3>Ingress Table 16: Check packet length</h3> > > <p> > For distributed logical routers or gateway routers with gateway > @@ -3797,7 +3832,7 @@ REGBIT_PKT_LARGER = > > check_pkt_larger(<var>L</var>); next; > > and advances to the next table. > </p> > > - <h3>Ingress Table 16: Handle larger packets</h3> > + <h3>Ingress Table 17: Handle larger packets</h3> > > <p> > For distributed logical routers or gateway routers with gateway > > port > > @@ -3860,7 +3895,7 @@ icmp6 { > and advances to the next table. > </p> > > - <h3>Ingress Table 17: Gateway Redirect</h3> > + <h3>Ingress Table 18: Gateway Redirect</h3> > > <p> > For distributed logical routers where one or more of the logical > > router > > @@ -3907,7 +3942,7 @@ icmp6 { > </li> > </ul> > > - <h3>Ingress Table 18: ARP Request</h3> > + <h3>Ingress Table 19: ARP Request</h3> > > <p> > In the common case where the Ethernet destination has been > > resolved, this > > diff --git a/ovn-nb.ovsschema b/ovn-nb.ovsschema > index 2ac8ef3ea..a0a171e19 100644 > --- a/ovn-nb.ovsschema > +++ b/ovn-nb.ovsschema > @@ -1,7 +1,7 @@ > { > "name": "OVN_Northbound", > - "version": "5.32.1", > - "cksum": "2805328215 29734", > + "version": "5.33.1", > + "cksum": "3874993350 29785", > "tables": { > "NB_Global": { > "columns": { > @@ -387,6 +387,7 @@ > "isRoot": false}, > "Logical_Router_Static_Route": { > "columns": { > + "route_table": {"type": "string"}, > "ip_prefix": {"type": "string"}, > "policy": {"type": {"key": {"type": "string", > "enum": ["set", ["src-ip", > diff --git a/ovn-nb.xml b/ovn-nb.xml > index d8266ed4d..b2917c363 100644 > --- a/ovn-nb.xml > +++ b/ovn-nb.xml > @@ -2772,6 +2772,14 @@ > prefix according to RFC3663 > </p> > </column> > + > + <column name="options" key="route_table"> > + Designates lookup Logical_Router_Static_Routes with specified > + <code>route_table</code> value. Routes to directly connected > > networks > > + from same Logical Router and routes without > > <code>route_table</code> > > + option set have higher priority than routes with > + <code>route_table</code> option set. > + </column> > </group> > > <group title="Attachment"> > @@ -2891,6 +2899,28 @@ > </p> > </column> > > + <column name="route_table"> > + <p> > + Any string to place route to separate routing table. If > > Logical Router > > + Port has configured value in <ref table="Logical_Router_Port" > + column="options" key="route_table"/> other than empty string, > > OVN > > + performs route lookup for all packets entering Logical Router > > ingress > > + pipeline from this port in the following manner: > + </p> > + > + <ul> > + <li> > + 1. First lookup among "global" routes: routes without > + <code>route_table</code> value set and routes to directly > > connected > > + networks. > + </li> > + <li> > + 2. Next lookup among routes with same > > <code>route_table</code> value > > + as specified in LRP's options:route_table field. > + </li> > + </ul> > + </column> > + > <column name="external_ids" key="ic-learned-route"> > <code>ovn-ic</code> populates this key if the route is learned > > from the > > global <ref db="OVN_IC_Southbound"/> database. In this case the > > value > > diff --git a/tests/ovn-ic.at b/tests/ovn-ic.at > index 32f4e9d02..3aab54362 100644 > --- a/tests/ovn-ic.at > +++ b/tests/ovn-ic.at > @@ -281,6 +281,7 @@ done > > AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr1], [0], [dnl > IPv4 Routes > +Route Table global: > 10.11.1.0/24 169.254.0.1 dst-ip > 10.11.2.0/24 169.254.100.2 dst-ip (learned) > 10.22.1.0/24 169.254.0.2 src-ip > @@ -299,6 +300,7 @@ ovn_as az1 ovn-nbctl set nb_global . > > options:ic-route-learn=false > > OVS_WAIT_WHILE([ovn_as az1 ovn-nbctl lr-route-list lr1 | grep learned]) > AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr1], [0], [dnl > IPv4 Routes > +Route Table global: > 10.11.1.0/24 169.254.0.1 dst-ip > 10.22.1.0/24 169.254.0.2 src-ip > ]) > @@ -314,6 +316,7 @@ ovn_as az1 ovn-nbctl set nb_global . > > options:ic-route-adv=false > > OVS_WAIT_WHILE([ovn_as az2 ovn-nbctl lr-route-list lr2 | grep learned]) > AT_CHECK([ovn_as az2 ovn-nbctl lr-route-list lr2], [0], [dnl > IPv4 Routes > +Route Table global: > 10.11.2.0/24 169.254.0.1 dst-ip > 10.22.2.0/24 169.254.0.2 src-ip > ]) > @@ -332,6 +335,7 @@ done > # Default route should NOT get advertised or learned, by default. > AT_CHECK([ovn_as az2 ovn-nbctl lr-route-list lr2], [0], [dnl > IPv4 Routes > +Route Table global: > 10.11.1.0/24 169.254.100.1 dst-ip (learned) > 10.11.2.0/24 169.254.0.1 dst-ip > 10.22.2.0/24 169.254.0.2 src-ip > diff --git a/tests/ovn-nbctl.at b/tests/ovn-nbctl.at > index 9b80ae410..ddb5536ce 100644 > --- a/tests/ovn-nbctl.at > +++ b/tests/ovn-nbctl.at > @@ -1520,6 +1520,7 @@ AT_CHECK([ovn-nbctl --ecmp --policy=src-ip > > lr-route-add lr0 20.0.0.0/24 11.0.0.1 > > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > IPv4 Routes > +Route Table global: > 10.0.0.0/24 11.0.0.1 dst-ip > 10.0.1.0/24 11.0.1.1 dst-ip lp0 > 10.0.10.0/24 dst-ip lp0 > @@ -1534,6 +1535,7 @@ AT_CHECK([ovn-nbctl lrp-add lr0 lp1 > > f0:00:00:00:00:02 11.0.0.254/24]) > > AT_CHECK([ovn-nbctl --may-exist lr-route-add lr0 10.0.0.111/24 > > 11.0.0.1 lp1]) > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > IPv4 Routes > +Route Table global: > 10.0.0.0/24 11.0.0.1 dst-ip lp1 > 10.0.1.0/24 11.0.1.1 dst-ip lp0 > 10.0.10.0/24 dst-ip lp0 > @@ -1564,6 +1566,7 @@ AT_CHECK([ovn-nbctl --policy=src-ip lr-route-del > > lr0 9.16.1.0/24]) > > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > IPv4 Routes > +Route Table global: > 10.0.0.0/24 11.0.0.1 dst-ip lp1 > 10.0.10.0/24 dst-ip lp0 > 10.0.0.0/24 11.0.0.2 src-ip > @@ -1575,6 +1578,7 @@ AT_CHECK([ovn-nbctl --policy=dst-ip lr-route-del > > lr0 10.0.0.0/24]) > > AT_CHECK([ovn-nbctl --policy=src-ip lr-route-del lr0 10.0.0.0/24]) > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > IPv4 Routes > +Route Table global: > 10.0.10.0/24 dst-ip lp0 > 0.0.0.0/0 192.168.0.1 dst-ip > ]) > @@ -1585,6 +1589,7 @@ AT_CHECK([ovn-nbctl --policy=src-ip lr-route-add > > lr0 10.0.0.0/24 11.0.0.2]) > > AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24]) > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > IPv4 Routes > +Route Table global: > 10.0.10.0/24 dst-ip lp0 > 0.0.0.0/0 192.168.0.1 dst-ip > ]) > @@ -1601,6 +1606,7 @@ AT_CHECK([ovn-nbctl --ecmp lr-route-add lr0 > > 10.0.0.0/24 11.0.0.3]) > > AT_CHECK([ovn-nbctl --ecmp lr-route-add lr0 10.0.0.0/24 11.0.0.4 lp0]) > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > IPv4 Routes > +Route Table global: > 10.0.0.0/24 11.0.0.1 dst-ip ecmp > 10.0.0.0/24 11.0.0.2 dst-ip ecmp > 10.0.0.0/24 11.0.0.3 dst-ip ecmp > @@ -1615,6 +1621,7 @@ dnl Delete ecmp routes > AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 11.0.0.1]) > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > IPv4 Routes > +Route Table global: > 10.0.0.0/24 11.0.0.2 dst-ip ecmp > 10.0.0.0/24 11.0.0.3 dst-ip ecmp > 10.0.0.0/24 11.0.0.4 dst-ip lp0 ecmp > @@ -1622,12 +1629,14 @@ IPv4 Routes > AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 11.0.0.2]) > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > IPv4 Routes > +Route Table global: > 10.0.0.0/24 11.0.0.3 dst-ip ecmp > 10.0.0.0/24 11.0.0.4 dst-ip lp0 ecmp > ]) > AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 11.0.0.4 lp0]) > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > IPv4 Routes > +Route Table global: > 10.0.0.0/24 11.0.0.3 dst-ip > ]) > AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 11.0.0.3]) > @@ -1641,6 +1650,7 @@ AT_CHECK([ovn-nbctl lr-route-add lr0 > > 2001:0db8:1::/64 2001:0db8:0:f103::1]) > > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > IPv6 Routes > +Route Table global: > 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > ::/0 2001:db8:0:f101::1 dst-ip > @@ -1650,6 +1660,7 @@ AT_CHECK([ovn-nbctl lr-route-del lr0 > > 2001:0db8:0::/64]) > > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > IPv6 Routes > +Route Table global: > 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > ::/0 2001:db8:0:f101::1 dst-ip > ]) > @@ -1677,11 +1688,13 @@ AT_CHECK([ovn-nbctl --may-exist > > --ecmp-symmetric-reply lr-route-add lr0 2003:0db > > > AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > IPv4 Routes > +Route Table global: > 10.0.0.0/24 11.0.0.1 dst-ip > 10.0.1.0/24 11.0.1.1 dst-ip lp0 > 0.0.0.0/0 192.168.0.1 dst-ip > > IPv6 Routes > +Route Table global: > 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip ecmp > 2001:db8:1::/64 2001:db8:0:f103::2 dst-ip ecmp > @@ -1696,7 +1709,188 @@ AT_CHECK([ovn-nbctl lrp-add lr0 lr0-p0 > > 00:00:01:01:02:03 192.168.10.1/24]) > > bfd_uuid=$(ovn-nbctl create bfd logical_port=lr0-p0 dst_ip=100.0.0.50 > > status=down min_tx=250 min_rx=250 detect_mult=10) > > AT_CHECK([ovn-nbctl lr-route-add lr0 100.0.0.0/24 192.168.0.1]) > route_uuid=$(fetch_column nb:logical_router_static_route _uuid > > ip_prefix="100.0.0.0/24") > > -AT_CHECK([ovn-nbctl set logical_router_static_route $route_uuid > > bfd=$bfd_uuid])]) > > +AT_CHECK([ovn-nbctl set logical_router_static_route $route_uuid > > bfd=$bfd_uuid]) > > + > +check ovn-nbctl lr-route-del lr0 > +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > +]) > + > +dnl Check IPv4 routes in route table > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 0.0.0.0/0 > > 192.168.0.1 > > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 10.0.1.1/24 > > 11.0.1.1 lp0 > > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 10.0.0.1/24 > > 11.0.0.1 > > +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > +IPv4 Routes > +Route Table rtb-1: > + 10.0.0.0/24 11.0.0.1 dst-ip > + 10.0.1.0/24 11.0.1.1 dst-ip lp0 > + 0.0.0.0/0 192.168.0.1 dst-ip > +]) > + > +check ovn-nbctl lr-route-del lr0 > +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > +]) > + > +dnl Check IPv6 routes in route table > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 0:0:0:0:0:0:0:0/0 > > 2001:0db8:0:f101::1 > > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 2001:0db8:0::/64 > > 2001:0db8:0:f102::1 lp0 > > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 2001:0db8:1::/64 > > 2001:0db8:0:f103::1 > > + > +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > +IPv6 Routes > +Route Table rtb-1: > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > + ::/0 2001:db8:0:f101::1 dst-ip > +]) > + > +dnl Check IPv4 and IPv6 routes in route table > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 0.0.0.0/0 > > 192.168.0.1 > > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 10.0.1.1/24 > > 11.0.1.1 lp0 > > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 10.0.0.1/24 > > 11.0.0.1 > > + > +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > +IPv4 Routes > +Route Table rtb-1: > + 10.0.0.0/24 11.0.0.1 dst-ip > + 10.0.1.0/24 11.0.1.1 dst-ip lp0 > + 0.0.0.0/0 192.168.0.1 dst-ip > + > +IPv6 Routes > +Route Table rtb-1: > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > + ::/0 2001:db8:0:f101::1 dst-ip > +]) > + > +# Add routes in another route table > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 0.0.0.0/0 > > 192.168.0.1 > > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 10.0.1.1/24 > > 11.0.1.1 lp0 > > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 10.0.0.1/24 > > 11.0.0.1 > > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 0:0:0:0:0:0:0:0/0 > > 2001:0db8:0:f101::1 > > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 2001:0db8:0::/64 > > 2001:0db8:0:f102::1 lp0 > > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 2001:0db8:1::/64 > > 2001:0db8:0:f103::1 > > + > +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > +IPv4 Routes > +Route Table rtb-1: > + 10.0.0.0/24 11.0.0.1 dst-ip > + 10.0.1.0/24 11.0.1.1 dst-ip lp0 > + 0.0.0.0/0 192.168.0.1 dst-ip > + > +Route Table rtb-2: > + 10.0.0.0/24 11.0.0.1 dst-ip > + 10.0.1.0/24 11.0.1.1 dst-ip lp0 > + 0.0.0.0/0 192.168.0.1 dst-ip > + > +IPv6 Routes > +Route Table rtb-1: > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > + ::/0 2001:db8:0:f101::1 dst-ip > + > +Route Table rtb-2: > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > + ::/0 2001:db8:0:f101::1 dst-ip > +]) > + > +# Add routes to global route table > +check ovn-nbctl lr-route-add lr0 0.0.0.0/0 192.168.0.1 > +check ovn-nbctl lr-route-add lr0 10.0.1.1/24 11.0.1.1 lp0 > +check ovn-nbctl lr-route-add lr0 10.0.0.1/24 11.0.0.1 > +check ovn-nbctl lr-route-add lr0 0:0:0:0:0:0:0:0/0 2001:0db8:0:f101::1 > +check ovn-nbctl lr-route-add lr0 2001:0db8:0::/64 2001:0db8:0:f102::1 > > lp0 > > +check check ovn-nbctl lr-route-add lr0 2001:0db8:1::/64 > > 2001:0db8:0:f103::1 > > + > +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl > +IPv4 Routes > +Route Table global: > + 10.0.0.0/24 11.0.0.1 dst-ip > + 10.0.1.0/24 11.0.1.1 dst-ip lp0 > + 0.0.0.0/0 192.168.0.1 dst-ip > + > +Route Table rtb-1: > + 10.0.0.0/24 11.0.0.1 dst-ip > + 10.0.1.0/24 11.0.1.1 dst-ip lp0 > + 0.0.0.0/0 192.168.0.1 dst-ip > + > +Route Table rtb-2: > + 10.0.0.0/24 11.0.0.1 dst-ip > + 10.0.1.0/24 11.0.1.1 dst-ip lp0 > + 0.0.0.0/0 192.168.0.1 dst-ip > + > +IPv6 Routes > +Route Table global: > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > + ::/0 2001:db8:0:f101::1 dst-ip > + > +Route Table rtb-1: > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > + ::/0 2001:db8:0:f101::1 dst-ip > + > +Route Table rtb-2: > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > + ::/0 2001:db8:0:f101::1 dst-ip > +]) > + > +# delete IPv4 route from rtb-1 > +check ovn-nbctl --route-table=rtb-1 lr-route-del lr0 10.0.0.0/24 > +AT_CHECK([ovn-nbctl --route-table=rtb-1 lr-route-list lr0], [0], [dnl > +IPv4 Routes > +Route Table rtb-1: > + 10.0.1.0/24 11.0.1.1 dst-ip lp0 > + 0.0.0.0/0 192.168.0.1 dst-ip > + > +IPv6 Routes > +Route Table rtb-1: > + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > + ::/0 2001:db8:0:f101::1 dst-ip > +]) > + > +# delete IPv6 route from rtb-2 > +check ovn-nbctl --route-table=rtb-2 lr-route-del lr0 2001:db8::/64 > +AT_CHECK([ovn-nbctl --route-table=rtb-2 lr-route-list lr0], [0], [dnl > +IPv4 Routes > +Route Table rtb-2: > + 10.0.0.0/24 11.0.0.1 dst-ip > + 10.0.1.0/24 11.0.1.1 dst-ip lp0 > + 0.0.0.0/0 192.168.0.1 dst-ip > + > +IPv6 Routes > +Route Table rtb-2: > + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip > + ::/0 2001:db8:0:f101::1 dst-ip > +]) > + > +check ovn-nbctl lr-route-del lr0 > + > +# ECMP route in route table > +check ovn-nbctl --route-table=rtb1 lr-route-add lr0 0.0.0.0/0 > > 192.168.0.1 > > +check ovn-nbctl --ecmp --route-table=rtb1 lr-route-add lr0 0.0.0.0/0 > > 192.168.0.2 > > + > +# Negative route table case: same prefix > +AT_CHECK([ovn-nbctl --route-table=rtb1 lr-route-add lr0 0.0.0.0/0 > > 192.168.0.1], [1], [], [dnl > > +ovn-nbctl: duplicate prefix: 0.0.0.0/0 (policy: dst-ip). Use option > > --ecmp to allow this for ECMP routing. > > +]) > + > +# Negative route table case: same prefix & nexthop with ecmp > +AT_CHECK([ovn-nbctl --ecmp --route-table=rtb1 lr-route-add lr0 > > 0.0.0.0/0 192.168.0.2], [1], [], [dnl > > +ovn-nbctl: duplicate nexthop for the same ECMP route > +]) > + > +# Add routes to global route table > +check ovn-nbctl lrp-add lr0 lrp0 00:00:00:00:00:01 1.1.1.1/24 > +check ovn-nbctl lrp-set-options lrp0 route_table=rtb1 > +AT_CHECK([ovn-nbctl get logical-router-port lrp0 options:route_table], > > [0], [dnl > > +rtb1 > +]) > +check `ovn-nbctl show lr0 | grep lrp0 -A3 | grep route_table=rtb1` > +]) > > dnl > > --------------------------------------------------------------------- > > > diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at > index 3eebb55b6..e71e65bcc 100644 > --- a/tests/ovn-northd.at > +++ b/tests/ovn-northd.at > @@ -5111,7 +5111,7 @@ check ovn-nbctl --wait=sb --ecmp-symmetric-reply > > lr-route-add lr0 1.0.0.1 192.16 > > > ovn-sbctl dump-flows lr0 > lr0flows > AT_CHECK([grep -e "lr_in_ip_routing.*select" lr0flows | sed > > 's/table=../table=??/' | sort], [0], [dnl > > - table=??(lr_in_ip_routing ), priority=65 , match=(ip4.dst == > > 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = 1; > reg8[[16..31]] = select(1, 2);) > > + table=??(lr_in_ip_routing ), priority=165 , match=(ip4.dst == > > 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = 1; > reg8[[16..31]] = select(1, 2);) > > ]) > AT_CHECK([grep -e "lr_in_ip_routing_ecmp" lr0flows | sed > > 's/192\.168\.0\..0/192.168.0.??/' | sed 's/table=../table=??/' | sort], > [0], [dnl > > table=??(lr_in_ip_routing_ecmp), priority=100 , > > match=(reg8[[0..15]] == 1 && reg8[[16..31]] == 1), action=(reg0 = > 192.168.0.??; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = > "lr0-public"; next;) > > @@ -5124,7 +5124,7 @@ check ovn-nbctl --wait=sb --ecmp-symmetric-reply > > lr-route-add lr0 1.0.0.1 192.16 > > > ovn-sbctl dump-flows lr0 > lr0flows > AT_CHECK([grep -e "lr_in_ip_routing.*select" lr0flows | sed > > 's/table=../table=??/' | sort], [0], [dnl > > - table=??(lr_in_ip_routing ), priority=65 , match=(ip4.dst == > > 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = 1; > reg8[[16..31]] = select(1, 2);) > > + table=??(lr_in_ip_routing ), priority=165 , match=(ip4.dst == > > 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = 1; > reg8[[16..31]] = select(1, 2);) > > ]) > AT_CHECK([grep -e "lr_in_ip_routing_ecmp" lr0flows | sed > > 's/192\.168\.0\..0/192.168.0.??/' | sed 's/table=../table=??/' | sort], > [0], [dnl > > table=??(lr_in_ip_routing_ecmp), priority=100 , > > match=(reg8[[0..15]] == 1 && reg8[[16..31]] == 1), action=(reg0 = > 192.168.0.??; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = > "lr0-public"; next;) > > @@ -5139,14 +5139,14 @@ check ovn-nbctl --wait=sb lr-route-add lr0 > > 1.0.0.0/24 192.168.0.10 > > ovn-sbctl dump-flows lr0 > lr0flows > > AT_CHECK([grep -e "lr_in_ip_routing.*192.168.0.10" lr0flows | sed > > 's/table=../table=??/' | sort], [0], [dnl > > - table=??(lr_in_ip_routing ), priority=49 , match=(ip4.dst == > > 1.0.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.10; > reg1 > = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; > flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == > > 1.0.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.10; > reg1 > = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; > flags.loopback = 1; next;) > > ]) > > check ovn-nbctl --wait=sb lr-route-add lr0 2.0.0.0/24 lr0-public > > ovn-sbctl dump-flows lr0 > lr0flows > AT_CHECK([grep -e "lr_in_ip_routing.*2.0.0.0" lr0flows | sed > > 's/table=../table=??/' | sort], [0], [dnl > > - table=??(lr_in_ip_routing ), priority=49 , match=(ip4.dst == > > 2.0.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 = > 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; > flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == > > 2.0.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 = > 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; > flags.loopback = 1; next;) > > ]) > > AT_CLEANUP > @@ -5232,3 +5232,71 @@ AT_CHECK([grep lr_in_gw_redirect lrflows | grep > > cr-DR | sed 's/table=../table=?? > > > AT_CLEANUP > ]) > + > + > +OVN_FOR_EACH_NORTHD([ > +AT_SETUP([route tables -- flows]) > +AT_KEYWORDS([route-tables-flows]) > +ovn_start > + > +check ovn-nbctl lr-add lr0 > +check ovn-nbctl lrp-add lr0 lrp0 00:00:00:00:00:01 192.168.0.1/24 > +check ovn-nbctl lrp-add lr0 lrp1 00:00:00:00:01:01 192.168.1.1/24 > +check ovn-nbctl lrp-add lr0 lrp2 00:00:00:00:02:01 192.168.2.1/24 > +check ovn-nbctl lrp-set-options lrp1 route_table=rtb-1 > +check ovn-nbctl lrp-set-options lrp2 route_table=rtb-2 > + > +check ovn-nbctl lr-route-add lr0 0.0.0.0/0 192.168.0.10 > +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 192.168.0.0/24 > > 192.168.1.10 > > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 0.0.0.0/0 > > 192.168.0.10 > > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 1.1.1.1/32 > > 192.168.0.20 > > +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 2.2.2.2/32 > > 192.168.0.30 > > +check ovn-nbctl --route-table=rtb-2 --ecmp lr-route-add lr0 2.2.2.2/32 > > 192.168.0.31 > > +check ovn-nbctl --wait=sb sync > + > +ovn-sbctl dump-flows lr0 > lr0flows > +AT_CAPTURE_FILE([lr0flows]) > + > +AT_CHECK([grep -e "lr_in_ip_routing_pre.*match=(1)" lr0flows | sed > > 's/table=../table=??/'], [0], [dnl > > + table=??(lr_in_ip_routing_pre), priority=0 , match=(1), > > action=(next;) > > +]) > + > +p1_reg=$(grep -oP "lr_in_ip_routing_pre.*lrp1.*action=\(reg7 = \K." > > lr0flows) > > +p2_reg=$(grep -oP "lr_in_ip_routing_pre.*lrp2.*action=\(reg7 = \K." > > lr0flows) > > +echo $p1_reg > +echo $p2_reg > + > +# exact register values are not predictable > +if [[ $p1_reg -eq 2 ] && [ $p2_reg -eq 1 ]]; then > + echo "swap reg values in dump" > + sed -i -r s'/^(.*lrp2.*action=\(reg7 = )(1)(.*)/\12\3/g' lr0flows # > > "reg7 = 1" -> "reg7 = 2" > > + sed -i -r s'/^(.*lrp1.*action=\(reg7 = )(2)(.*)/\11\3/g' lr0flows # > > "reg7 = 2" -> "reg7 = 1" > > + sed -i -r s'/^(.*match=\(reg7 == )(2)( &&.*lrp1.*)/\11\3/g' lr0flows > > # "reg7 == 2" -> "reg7 == 1" > > + sed -i -r s'/^(.*match=\(reg7 == )(1)( &&.*lrp0.*)/\12\3/g' lr0flows > > # "reg7 == 1" -> "reg7 == 2" > > +fi > + > +check test "$p1_reg" != "$p2_reg" -a $((p1_reg * p2_reg)) -eq 2 > + > +AT_CHECK([grep "lr_in_ip_routing_pre" lr0flows | sed > > 's/table=../table=??/' | sort], [0], [dnl > > + table=??(lr_in_ip_routing_pre), priority=0 , match=(1), > > action=(next;) > > + table=??(lr_in_ip_routing_pre), priority=100 , match=(inport == > > "lrp1"), action=(reg7 = 1; next;) > > + table=??(lr_in_ip_routing_pre), priority=100 , match=(inport == > > "lrp2"), action=(reg7 = 2; next;) > > +]) > + > +grep -e "(lr_in_ip_routing ).*outport" lr0flows > + > +AT_CHECK([grep -e "(lr_in_ip_routing ).*outport" lr0flows | sed > > 's/table=../table=??/' | sort], [0], [dnl > > + table=??(lr_in_ip_routing ), priority=1 , match=(reg7 == 2 && > > ip4.dst == 0.0.0.0/0), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = > 192.168.0.10; reg1 = 192.168.0.1; eth.src = 00:00:00:00:00:01; outport = > "lrp0"; flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=101 , match=(ip4.dst == > > 0.0.0.0/0), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.10; reg1 > = 192.168.0.1; eth.src = 00:00:00:00:00:01; outport = "lrp0"; > flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == > > 192.168.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 > = 192.168.0.1; eth.src = 00:00:00:00:00:01; outport = "lrp0"; > flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == > > 192.168.1.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 > = 192.168.1.1; eth.src = 00:00:00:00:01:01; outport = "lrp1"; > flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == > > 192.168.2.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 > = 192.168.2.1; eth.src = 00:00:00:00:02:01; outport = "lrp2"; > flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=229 , match=(inport == > > "lrp0" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; > xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe00:1; eth.src = > 00:00:00:00:00:01; outport = "lrp0"; flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=229 , match=(inport == > > "lrp1" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; > xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe00:101; eth.src = > 00:00:00:00:01:01; outport = "lrp1"; flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=229 , match=(inport == > > "lrp2" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; > xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe00:201; eth.src = > 00:00:00:00:02:01; outport = "lrp2"; flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=49 , match=(reg7 == 1 && > > ip4.dst == 192.168.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = > 192.168.1.10; reg1 = 192.168.1.1; eth.src = 00:00:00:00:01:01; outport = > "lrp1"; flags.loopback = 1; next;) > > + table=??(lr_in_ip_routing ), priority=65 , match=(reg7 == 2 && > > ip4.dst == 1.1.1.1/32), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = > 192.168.0.20; reg1 = 192.168.0.1; eth.src = 00:00:00:00:00:01; outport = > "lrp0"; flags.loopback = 1; next;) > > +]) > + > +AT_CLEANUP > +]) > diff --git a/tests/ovn.at b/tests/ovn.at > index 49ece8735..60783a14b 100644 > --- a/tests/ovn.at > +++ b/tests/ovn.at > @@ -18145,7 +18145,7 @@ eth_dst=00000000ff01 > ip_src=$(ip_to_hex 10 0 0 10) > ip_dst=$(ip_to_hex 172 168 0 101) > send_icmp_packet 1 1 $eth_src $eth_dst $ip_src $ip_dst c4c9 > > 0000000000000000000000 > > -AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | awk '/table=25, > > n_packets=1, n_bytes=45/{print $7" "$8}'],[0],[dnl > > +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | awk '/table=26, > > n_packets=1, n_bytes=45/{print $7" "$8}'],[0],[dnl > > priority=80,ip,reg15=0x3,metadata=0x3,nw_src=10.0.0.10 actions=drop > ]) > > @@ -22577,6 +22577,433 @@ OVN_CLEANUP([hv1]) > AT_CLEANUP > ]) > > + > +OVN_FOR_EACH_NORTHD([ > +AT_SETUP([route tables -- global routes]) > +ovn_start > + > +# Logical network: > +# ls1 (192.168.1.0/24) - lrp-lr1-ls1 - lr1 - lrp-lr1-ls2 - ls2 ( > > 192.168.2.0/24) > > +# > +# ls1 has lsp11 (192.168.1.11) and ls2 has lsp21 (192.168.2.21) and > > lsp22 > > +# (192.168.2.22) > +# > +# lrp-lr1-ls1 set options:route_table=rtb-1 > +# > +# Static routes on lr1: > +# 0.0.0.0/0 nexthop 192.168.2.21 > +# 1.1.1.1/32 nexthop 192.168.2.22 route_table=rtb-1 > +# > +# Test 1: > +# lsp11 send packet to 2.2.2.2 > +# > +# Expected result: > +# lsp21 should receive traffic, lsp22 should not receive any traffic > +# > +# Test 2: > +# lsp11 send packet to 1.1.1.1 > +# > +# Expected result: > +# lsp21 should receive traffic, lsp22 should not receive any traffic > + > +ovn-nbctl lr-add lr1 > + > +for i in 1 2; do > + ovn-nbctl ls-add ls${i} > + ovn-nbctl lrp-add lr1 lrp-lr1-ls${i} 00:00:00:01:0${i}:01 > > 192.168.${i}.1/24 > > + ovn-nbctl lsp-add ls${i} lsp-ls${i}-lr1 -- lsp-set-type > > lsp-ls${i}-lr1 router \ > > + -- lsp-set-options lsp-ls${i}-lr1 router-port=lrp-lr1-ls${i} \ > + -- lsp-set-addresses lsp-ls${i}-lr1 router > +done > + > +# install static routes > +ovn-nbctl lr-route-add lr1 0.0.0.0/0 192.168.2.21 > +ovn-nbctl --route-table=rtb-1 lr-route-add lr1 1.1.1.1/32 192.168.2.22 > + > +# set lrp-lr1-ls1 route table > +ovn-nbctl lrp-set-options lrp-lr1-ls1 route_table=rtb-1 > + > +# Create logical ports > +ovn-nbctl lsp-add ls1 lsp11 -- \ > + lsp-set-addresses lsp11 "f0:00:00:00:01:11 192.168.1.11" > +ovn-nbctl lsp-add ls2 lsp21 -- \ > + lsp-set-addresses lsp21 "f0:00:00:00:02:21 192.168.2.21" > +ovn-nbctl lsp-add ls2 lsp22 -- \ > + lsp-set-addresses lsp22 "f0:00:00:00:02:22 192.168.2.22" > + > +net_add n1 > +sim_add hv1 > +as hv1 > +ovs-vsctl add-br br-phys > +ovn_attach n1 br-phys 192.168.0.1 > +ovs-vsctl -- add-port br-int hv1-vif1 -- \ > + set interface hv1-vif1 external-ids:iface-id=lsp11 \ > + options:tx_pcap=hv1/vif1-tx.pcap \ > + options:rxq_pcap=hv1/vif1-rx.pcap \ > + ofport-request=1 > + > +ovs-vsctl -- add-port br-int hv1-vif2 -- \ > + set interface hv1-vif2 external-ids:iface-id=lsp21 \ > + options:tx_pcap=hv1/vif2-tx.pcap \ > + options:rxq_pcap=hv1/vif2-rx.pcap \ > + ofport-request=2 > + > +ovs-vsctl -- add-port br-int hv1-vif3 -- \ > + set interface hv1-vif3 external-ids:iface-id=lsp22 \ > + options:tx_pcap=hv1/vif3-tx.pcap \ > + options:rxq_pcap=hv1/vif3-rx.pcap \ > + ofport-request=3 > + > +# wait for earlier changes to take effect > +check ovn-nbctl --wait=hv sync > +wait_for_ports_up > + > +for i in 1 2; do > + packet="inport==\"lsp11\" && eth.src==f0:00:00:00:01:11 && > > eth.dst==00:00:00:01:01:01 && > > + ip4 && ip.ttl==64 && ip4.src==192.168.1.11 && > > ip4.dst==$i.$i.$i.$i && icmp" > > + AT_CHECK([as hv1 ovs-appctl -t ovn-controller inject-pkt > > "$packet"]) > > + > + # Assume all packets go to lsp21. > + exp_packet="eth.src==00:00:00:01:02:01 && > > eth.dst==f0:00:00:00:02:21 && > > + ip4 && ip.ttl==63 && ip4.src==192.168.1.11 && > > ip4.dst==$i.$i.$i.$i && icmp" > > + echo $exp_packet | ovstest test-ovn expr-to-packets >> > > expected_lsp21 > > +done > +> expected_lsp22 > + > +# lsp21 should recieve 2 packets and lsp22 should recieve no packets > +OVS_WAIT_UNTIL([ > + rcv_n1=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" > > hv1/vif2-tx.pcap > lsp21.packets && cat lsp21.packets | wc -l` > > + rcv_n2=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" > > hv1/vif3-tx.pcap > lsp22.packets && cat lsp22.packets | wc -l` > > + echo $rcv_n1 $rcv_n2 > + test $rcv_n1 -eq 2 -a $rcv_n2 -eq 0]) > + > +for i in 1 2; do > + sort expected_lsp2$i > expout > + AT_CHECK([cat lsp2${i}.packets | sort], [0], [expout]) > +done > + > +OVN_CLEANUP([hv1]) > +AT_CLEANUP > +]) > + > + > +OVN_FOR_EACH_NORTHD([ > +AT_SETUP([route tables -- directly connected routes]) > +ovn_start > + > +# Logical network: > +# ls1 (192.168.1.0/24) - lrp-lr1-ls1 - lr1 - lrp-lr1-ls2 - ls2 ( > > 192.168.2.0/24) > > +# > +# ls1 has lsp11 (192.168.1.11) and ls2 has lsp21 (192.168.2.21) > +# > +# lrp-lr1-ls1 set options:route_table=rtb-1 > +# > +# Static routes on lr1: > +# 192.168.2.0/25 nexthop 192.168.1.11 route_table=rtb-1 > +# > +# Test 1: > +# lsp11 send packet to 192.168.2.21 > +# > +# Expected result: > +# lsp21 should receive traffic, lsp11 should not receive any traffic > + > +ovn-nbctl lr-add lr1 > + > +for i in 1 2; do > + ovn-nbctl ls-add ls${i} > + ovn-nbctl lrp-add lr1 lrp-lr1-ls${i} 00:00:00:01:0${i}:01 > > 192.168.${i}.1/24 > > + ovn-nbctl lsp-add ls${i} lsp-ls${i}-lr1 -- lsp-set-type > > lsp-ls${i}-lr1 router \ > > + -- lsp-set-options lsp-ls${i}-lr1 router-port=lrp-lr1-ls${i} \ > + -- lsp-set-addresses lsp-ls${i}-lr1 router > +done > + > +# install static route, which overrides directly-connected routes > +ovn-nbctl --route-table=rtb-1 lr-route-add lr1 192.168.2.0/25 > > 192.168.1.11 > > + > +# set lrp-lr1-ls1 route table > +ovn-nbctl lrp-set-options lrp-lr1-ls1 route_table=rtb-1 > + > +# Create logical ports > +ovn-nbctl lsp-add ls1 lsp11 -- \ > + lsp-set-addresses lsp11 "f0:00:00:00:01:11 192.168.1.11" > +ovn-nbctl lsp-add ls2 lsp21 -- \ > + lsp-set-addresses lsp21 "f0:00:00:00:02:21 192.168.2.21" > + > +net_add n1 > +sim_add hv1 > +as hv1 > +ovs-vsctl add-br br-phys > +ovn_attach n1 br-phys 192.168.0.1 > +ovs-vsctl -- add-port br-int hv1-vif1 -- \ > + set interface hv1-vif1 external-ids:iface-id=lsp11 \ > + options:tx_pcap=hv1/vif1-tx.pcap \ > + options:rxq_pcap=hv1/vif1-rx.pcap \ > + ofport-request=1 > + > +ovs-vsctl -- add-port br-int hv1-vif2 -- \ > + set interface hv1-vif2 external-ids:iface-id=lsp21 \ > + options:tx_pcap=hv1/vif2-tx.pcap \ > + options:rxq_pcap=hv1/vif2-rx.pcap \ > + ofport-request=2 > + > +# wait for earlier changes to take effect > +check ovn-nbctl --wait=hv sync > +wait_for_ports_up > + > +packet="inport==\"lsp11\" && eth.src==f0:00:00:00:01:11 && > > eth.dst==00:00:00:01:01:01 && > > + ip4 && ip.ttl==64 && ip4.src==192.168.1.11 && > > ip4.dst==192.168.2.21 && icmp" > > +AT_CHECK([as hv1 ovs-appctl -t ovn-controller inject-pkt "$packet"]) > + > +# Assume all packets go to lsp21. > +exp_packet="eth.src==00:00:00:01:02:01 && eth.dst==f0:00:00:00:02:21 && > + ip4 && ip.ttl==63 && ip4.src==192.168.1.11 && > > ip4.dst==192.168.2.21 && icmp" > > +echo $exp_packet | ovstest test-ovn expr-to-packets >> expected_lsp21 > +> expected_lsp11 > + > +# lsp21 should recieve 1 icmp packet and lsp11 should recieve no > > packets > > +OVS_WAIT_UNTIL([ > + rcv_n11=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" > > hv1/vif1-tx.pcap > lsp11.packets && cat lsp11.packets | wc -l` > > + rcv_n21=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" > > hv1/vif2-tx.pcap > lsp21.packets && cat lsp21.packets | wc -l` > > + echo $rcv_n11 $rcv_n21 > + test $rcv_n11 -eq 0 -a $rcv_n21 -eq 1]) > + > +for i in 11 21; do > + sort expected_lsp$i > expout > + AT_CHECK([cat lsp${i}.packets | sort], [0], [expout]) > +done > + > +OVN_CLEANUP([hv1]) > +AT_CLEANUP > +]) > + > + > +OVN_FOR_EACH_NORTHD([ > +AT_SETUP([route tables -- overlapping subnets]) > +ovn_start > + > +# Logical network: > +# > +# ls1 (192.168.1.0/24) - lrp-lr1-ls1 -\ /- lrp-lr1-ls2 - ls2 ( > > 192.168.2.0/24) > > +# lr1 > +# ls3 (192.168.3.0/24) - lrp-lr1-ls3 -/ \- lrp-lr1-ls4 - ls4 ( > > 192.168.4.0/24) > > +# > +# ls1 has lsp11 (192.168.1.11) > +# ls2 has lsp21 (192.168.2.21) > +# ls3 has lsp31 (192.168.3.31) > +# ls4 has lsp41 (192.168.4.41) > +# > +# lrp-lr1-ls1 set options:route_table=rtb-1 > +# lrp-lr1-ls2 set options:route_table=rtb-2 > +# > +# Static routes on lr1: > +# 10.0.0.0/24 nexthop 192.168.3.31 route_table=rtb-1 > +# 10.0.0.0/24 nexthop 192.168.4.41 route_table=rtb-2 > +# > +# Test 1: > +# lsp11 send packet to 10.0.0.1 > +# > +# Expected result: > +# lsp31 should receive traffic, lsp41 should not receive any traffic > +# > +# Test 2: > +# lsp21 send packet to 10.0.0.1 > +# > +# Expected result: > +# lsp41 should receive traffic, lsp31 should not receive any traffic > + > +ovn-nbctl lr-add lr1 > + > +# Create logical topology > +for i in $(seq 1 4); do > + ovn-nbctl ls-add ls${i} > + ovn-nbctl lrp-add lr1 lrp-lr1-ls${i} 00:00:00:01:0${i}:01 > > 192.168.${i}.1/24 > > + ovn-nbctl lsp-add ls${i} lsp-ls${i}-lr1 -- lsp-set-type > > lsp-ls${i}-lr1 router \ > > + -- lsp-set-options lsp-ls${i}-lr1 router-port=lrp-lr1-ls${i} \ > + -- lsp-set-addresses lsp-ls${i}-lr1 router > + ovn-nbctl lsp-add ls$i lsp${i}1 -- \ > + lsp-set-addresses lsp${i}1 "f0:00:00:00:0${i}:1${i} > > 192.168.${i}.${i}1" > > +done > + > +# install static routes > +ovn-nbctl --route-table=rtb-1 lr-route-add lr1 10.0.0.0/24 192.168.3.31 > +ovn-nbctl --route-table=rtb-2 lr-route-add lr1 10.0.0.0/24 192.168.4.41 > + > +# set lrp-lr1-ls{1,2} route tables > +ovn-nbctl lrp-set-options lrp-lr1-ls1 route_table=rtb-1 > +ovn-nbctl lrp-set-options lrp-lr1-ls2 route_table=rtb-2 > + > +net_add n1 > +sim_add hv1 > +as hv1 > +ovs-vsctl add-br br-phys > +ovn_attach n1 br-phys 192.168.0.1 > + > +for i in $(seq 1 4); do > + ovs-vsctl -- add-port br-int hv1-vif${i} -- \ > + set interface hv1-vif${i} external-ids:iface-id=lsp${i}1 \ > + options:tx_pcap=hv1/vif${i}-tx.pcap \ > + options:rxq_pcap=hv1/vif${i}-rx.pcap \ > + ofport-request=${i} > +done > + > +# wait for earlier changes to take effect > +check ovn-nbctl --wait=hv sync > +wait_for_ports_up > + > +# lsp31 should recieve packet coming from lsp11 > +# lsp41 should recieve packet coming from lsp21 > +for i in $(seq 1 2); do > + di=$(( i + 2)) # dst index > + ri=$(( 5 - i)) # reverse index > + packet="inport==\"lsp${i}1\" && eth.src==f0:00:00:00:0${i}:1${i} && > + eth.dst==00:00:00:01:0${i}:01 && ip4 && ip.ttl==64 && > + ip4.src==192.168.${i}.${i}1 && ip4.dst==10.0.0.1 && icmp" > + AT_CHECK([as hv1 ovs-appctl -t ovn-controller inject-pkt > > "$packet"]) > > + > + # Assume all packets go to lsp${di}1. > + exp_packet="eth.src==00:00:00:01:0${di}:01 && > > eth.dst==f0:00:00:00:0${di}:1${di} && > > + ip4 && ip.ttl==63 && ip4.src==192.168.${i}.${i}1 && > > ip4.dst==10.0.0.1 && icmp" > > + echo $exp_packet | ovstest test-ovn expr-to-packets >> > > expected_lsp${di}1 > > + > expected_lsp${ri}1 > + > + OVS_WAIT_UNTIL([ > + rcv_n1=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" > > hv1/vif${di}-tx.pcap > lsp${di}1.packets && cat lsp${di}1.packets | wc -l` > > + rcv_n2=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" > > hv1/vif${ri}-tx.pcap > lsp${ri}1.packets && cat lsp${ri}1.packets | wc -l` > > + echo $rcv_n1 $rcv_n2 > + test $rcv_n1 -eq 1 -a $rcv_n2 -eq 0]) > + > + for j in "${di}1" "${ri}1"; do > + sort expected_lsp${j} > expout > + AT_CHECK([cat lsp${j}.packets | sort], [0], [expout]) > + done > + > + # cleanup tx pcap files > + for j in "${di}1" "${ri}1"; do > + ovs-vsctl -- remove interface hv1-vif${di} options tx_pcap > + > hv1/vif${di}-tx.pcap > + ovs-vsctl -- set interface hv1-vif${di} > > external-ids:iface-id=lsp${di}1 \ > > + options:tx_pcap=hv1/vif${di}-tx.pcap > + done > +done > + > +OVN_CLEANUP([hv1]) > +AT_CLEANUP > +]) > + > + > +OVN_FOR_EACH_NORTHD([ > +AT_SETUP([route tables IPv6 -- overlapping subnets]) > +ovn_start > + > +# Logical network: > +# > +# ls1 (2001:db8:1::/64) - lrp-lr1-ls1 -\ /- lrp-lr1-ls2 - ls2 > > (2001:db8:2::/64) > > +# lr1 > +# ls3 (2001:db8:3::/64) - lrp-lr1-ls3 -/ \- lrp-lr1-ls4 - ls4 > > (2001:db8:4::/64) > > +# > +# ls1 has lsp11 (2001:db8:1::11) > +# ls2 has lsp21 (2001:db8:2::21) > +# ls3 has lsp31 (2001:db8:3::31) > +# ls4 has lsp41 (2001:db8:4::41) > +# > +# lrp-lr1-ls1 set options:route_table=rtb-1 > +# lrp-lr1-ls2 set options:route_table=rtb-2 > +# > +# Static routes on lr1: > +# 2001:db8:2000::/64 nexthop 2001:db8:3::31 route_table=rtb-1 > +# 2001:db8:2000::/64 nexthop 2001:db8:3::41 route_table=rtb-2 > +# > +# Test 1: > +# lsp11 send packet to 2001:db8:2000::1 > +# > +# Expected result: > +# lsp31 should receive traffic, lsp41 should not receive any traffic > +# > +# Test 2: > +# lsp21 send packet to 2001:db8:2000::1 > +# > +# Expected result: > +# lsp41 should receive traffic, lsp31 should not receive any traffic > + > +ovn-nbctl lr-add lr1 > + > +# Create logical topology > +for i in $(seq 1 4); do > + ovn-nbctl ls-add ls${i} > + ovn-nbctl lrp-add lr1 lrp-lr1-ls${i} 00:00:00:01:0${i}:01 > > 2001:db8:${i}::1/64 > > + ovn-nbctl lsp-add ls${i} lsp-ls${i}-lr1 -- lsp-set-type > > lsp-ls${i}-lr1 router \ > > + -- lsp-set-options lsp-ls${i}-lr1 router-port=lrp-lr1-ls${i} \ > + -- lsp-set-addresses lsp-ls${i}-lr1 router > + ovn-nbctl lsp-add ls$i lsp${i}1 -- \ > + lsp-set-addresses lsp${i}1 "f0:00:00:00:0${i}:1${i} > > 2001:db8:${i}::${i}1" > > +done > + > +# install static routes > +ovn-nbctl --route-table=rtb-1 lr-route-add lr1 2001:db8:2000::/64 > > 2001:db8:3::31 > > +ovn-nbctl --route-table=rtb-2 lr-route-add lr1 2001:db8:2000::/64 > > 2001:db8:4::41 > > + > +# set lrp-lr1-ls{1,2} route tables > +ovn-nbctl lrp-set-options lrp-lr1-ls1 route_table=rtb-1 > +ovn-nbctl lrp-set-options lrp-lr1-ls2 route_table=rtb-2 > + > +net_add n1 > +sim_add hv1 > +as hv1 > +ovs-vsctl add-br br-phys > +ovn_attach n1 br-phys 192.168.0.1 > + > +for i in $(seq 1 4); do > + ovs-vsctl -- add-port br-int hv1-vif${i} -- \ > + set interface hv1-vif${i} external-ids:iface-id=lsp${i}1 \ > + options:tx_pcap=hv1/vif${i}-tx.pcap \ > + options:rxq_pcap=hv1/vif${i}-rx.pcap \ > + ofport-request=${i} > +done > + > +# wait for earlier changes to take effect > +AT_CHECK([ovn-nbctl --timeout=3 --wait=hv sync], [0], [ignore]) > + > +# lsp31 should recieve packet coming from lsp11 > +# lsp41 should recieve packet coming from lsp21 > +for i in $(seq 1 2); do > + di=$(( i + 2)) # dst index > + ri=$(( 5 - i)) # reverse index > + packet="inport==\"lsp${i}1\" && eth.src==f0:00:00:00:0${i}:1${i} && > + eth.dst==00:00:00:01:0${i}:01 && ip6 && ip.ttl==64 && > + ip6.src==2001:db8:${i}::${i}1 && ip6.dst==2001:db8:2000::1 > > && icmp6" > > + AT_CHECK([as hv1 ovs-appctl -t ovn-controller inject-pkt > > "$packet"]) > > + > + # Assume all packets go to lsp${di}1. > + exp_packet="eth.src==00:00:00:01:0${di}:01 && > > eth.dst==f0:00:00:00:0${di}:1${di} && ip6 && > > + ip.ttl==63 && ip6.src==2001:db8:${i}::${i}1 && > > ip6.dst==2001:db8:2000::1 && icmp6" > > + echo $exp_packet | ovstest test-ovn expr-to-packets >> > > expected_lsp${di}1 > > + > expected_lsp${ri}1 > + > + OVS_WAIT_UNTIL([ > + rcv_n1=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" > > hv1/vif${di}-tx.pcap > lsp${di}1.packets && cat lsp${di}1.packets | wc -l` > > + rcv_n2=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" > > hv1/vif${ri}-tx.pcap > lsp${ri}1.packets && cat lsp${ri}1.packets | wc -l` > > + echo $rcv_n1 $rcv_n2 > + test $rcv_n1 -eq 1 -a $rcv_n2 -eq 0]) > + > + for j in "${di}1" "${ri}1"; do > + sort expected_lsp${j} > expout > + AT_CHECK([cat lsp${j}.packets | sort], [0], [expout]) > + done > + > + # cleanup tx pcap files > + for j in "${di}1" "${ri}1"; do > + ovs-vsctl -- remove interface hv1-vif${di} options tx_pcap > + > hv1/vif${di}-tx.pcap > + ovs-vsctl -- set interface hv1-vif${di} > > external-ids:iface-id=lsp${di}1 \ > > + options:tx_pcap=hv1/vif${di}-tx.pcap > + done > +done > + > +OVN_CLEANUP([hv1]) > +AT_CLEANUP > +]) > + > + > OVN_FOR_EACH_NORTHD([ > AT_SETUP([forwarding group: 3 HVs, 1 LR, 2 LS]) > AT_KEYWORDS([forwarding-group]) > @@ -23332,7 +23759,7 @@ ovn-sbctl dump-flows > sbflows > AT_CAPTURE_FILE([sbflows]) > AT_CAPTURE_FILE([offlows]) > OVS_WAIT_UNTIL([ > - as hv1 ovs-ofctl dump-flows br-int table=20 > offlows > + as hv1 ovs-ofctl dump-flows br-int table=21 > offlows > test $(grep -c "load:0x64->NXM_NX_PKT_MARK" offlows) = 1 && \ > test $(grep -c "load:0x3->NXM_NX_PKT_MARK" offlows) = 1 && \ > test $(grep -c "load:0x4->NXM_NX_PKT_MARK" offlows) = 1 && \ > @@ -23425,12 +23852,12 @@ send_ipv4_pkt hv1 hv1-vif1 505400000003 > > 00000000ff01 \ > > $(ip_to_hex 10 0 0 3) $(ip_to_hex 172 168 0 120) > > OVS_WAIT_UNTIL([ > - test 1 -eq $(as hv1 ovs-ofctl dump-flows br-int table=20 | \ > + test 1 -eq $(as hv1 ovs-ofctl dump-flows br-int table=21 | \ > grep "load:0x2->NXM_NX_PKT_MARK" -c) > ]) > > AT_CHECK([ > - test 0 -eq $(as hv1 ovs-ofctl dump-flows br-int table=20 | \ > + test 0 -eq $(as hv1 ovs-ofctl dump-flows br-int table=21 | \ > grep "load:0x64->NXM_NX_PKT_MARK" -c) > ]) > > @@ -24133,7 +24560,7 @@ AT_CHECK([ > grep "priority=100" | \ > grep -c > > > "ct(commit,zone=NXM_NX_REG11\\[[0..15\\]],.*exec(move:NXM_OF_ETH_SRC\\[[\\]]->NXM_NX_CT_LABEL\\[[32..79\\]],load:0x[[0-9]]->NXM_NX_CT_LABEL\\[[80..95\\]]))" > > > - grep table=22 hv${hv}flows | \ > + grep table=23 hv${hv}flows | \ > grep "priority=200" | \ > grep -c > > "actions=move:NXM_NX_CT_LABEL\\[[32..79\\]]->NXM_OF_ETH_DST\\[[\\]]" > > done; :], [0], [dnl > @@ -24258,7 +24685,7 @@ AT_CHECK([ > grep "priority=100" | \ > grep -c > > > "ct(commit,zone=NXM_NX_REG11\\[[0..15\\]],.*exec(move:NXM_OF_ETH_SRC\\[[\\]]->NXM_NX_CT_LABEL\\[[32..79\\]],load:0x[[0-9]]->NXM_NX_CT_LABEL\\[[80..95\\]]))" > > > - grep table=22 hv${hv}flows | \ > + grep table=23 hv${hv}flows | \ > grep "priority=200" | \ > grep -c > > "actions=move:NXM_NX_CT_LABEL\\[[32..79\\]]->NXM_OF_ETH_DST\\[[\\]]" > > done; :], [0], [dnl > @@ -24880,7 +25307,7 @@ AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | > > grep "actions=controller" | grep > > ]) > > # The packet should've been dropped in the lr_in_arp_resolve stage. > -AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | grep -E "table=22, > > n_packets=1,.* priority=1,ip,metadata=0x${sw_key},nw_dst=10.0.1.1 > actions=drop" -c], [0], [dnl > > +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | grep -E "table=23, > > n_packets=1,.* priority=1,ip,metadata=0x${sw_key},nw_dst=10.0.1.1 > actions=drop" -c], [0], [dnl > > 1 > ]) > > diff --git a/utilities/ovn-nbctl.c b/utilities/ovn-nbctl.c > index e34bb65f7..0ff10618b 100644 > --- a/utilities/ovn-nbctl.c > +++ b/utilities/ovn-nbctl.c > @@ -329,6 +329,8 @@ Logical router port commands:\n\ > add logical port PORT on ROUTER\n\ > lrp-set-gateway-chassis PORT CHASSIS [PRIORITY]\n\ > set gateway chassis for port PORT\n\ > + lrp-set-options PORT KEY=VALUE [KEY=VALUE]...\n\ > + set router port options\n\ > lrp-del-gateway-chassis PORT CHASSIS\n\ > delete gateway chassis from port PORT\n\ > lrp-get-gateway-chassis PORT\n\ > @@ -351,11 +353,17 @@ Logical router port commands:\n\ > ('overlay' or 'bridged')\n\ > \n\ > Route commands:\n\ > - [--policy=POLICY] [--ecmp] [--ecmp-symmetric-reply] lr-route-add > > ROUTER \n\ > > - PREFIX NEXTHOP [PORT]\n\ > + [--policy=POLICY]\n\ > + [--ecmp]\n\ > + [--ecmp-symmetric-reply]\n\ > + [--route-table=ROUTE_TABLE]\n\ > + lr-route-add ROUTER PREFIX NEXTHOP [PORT]\n\ > add a route to ROUTER\n\ > - [--policy=POLICY] lr-route-del ROUTER [PREFIX [NEXTHOP [PORT]]]\n\ > + [--policy=POLICY]\n\ > + [--route-table=ROUTE_TABLE]\n\ > + lr-route-del ROUTER [PREFIX [NEXTHOP [PORT]]]\n\ > remove routes from ROUTER\n\ > + [--route-table=ROUTE_TABLE]\n\ > lr-route-list ROUTER print routes for ROUTER\n\ > \n\ > Policy commands:\n\ > @@ -743,6 +751,11 @@ print_lr(const struct nbrec_logical_router *lr, > > struct ds *s) > > ds_put_cstr(s, "]\n"); > } > > + const char *route_table = smap_get(&lrp->options, > > "route_table"); > > + if (route_table) { > + ds_put_format(s, " route-table: %s\n", route_table); > + } > + > if (lrp->n_gateway_chassis) { > const struct nbrec_gateway_chassis **gcs; > > @@ -862,6 +875,7 @@ nbctl_pre_show(struct ctl_context *ctx) > ovsdb_idl_add_column(ctx->idl, > > &nbrec_logical_router_port_col_name); > > ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_mac); > ovsdb_idl_add_column(ctx->idl, > > &nbrec_logical_router_port_col_networks); > > + ovsdb_idl_add_column(ctx->idl, > > &nbrec_logical_router_port_col_options); > > ovsdb_idl_add_column(ctx->idl, > > &nbrec_logical_router_port_col_gateway_chassis); > > > ovsdb_idl_add_column(ctx->idl, > > &nbrec_gateway_chassis_col_chassis_name); > > @@ -4000,11 +4014,19 @@ nbctl_lr_policy_list(struct ctl_context *ctx) > > static struct nbrec_logical_router_static_route * > nbctl_lr_get_route(const struct nbrec_logical_router *lr, char *prefix, > - char *next_hop, bool is_src_route, bool ecmp) > + char *next_hop, bool is_src_route, bool ecmp, > + char *route_table) > { > for (int i = 0; i < lr->n_static_routes; i++) { > struct nbrec_logical_router_static_route *route = > > lr->static_routes[i]; > > > + /* Strict compare for route_table. > + * If route_table was not specified, > + * lookup for routes with empty route_table value. */ > + if (strcmp(route->route_table, route_table ? route_table : > > "")) { > > + continue; > + } > + > /* Compare route policy. */ > char *nb_policy = route->policy; > bool nb_is_src_route = false; > @@ -4060,6 +4082,8 @@ nbctl_pre_lr_route_add(struct ctl_context *ctx) > &nbrec_logical_router_static_route_col_bfd); > ovsdb_idl_add_column(ctx->idl, > > &nbrec_logical_router_static_route_col_options); > > + ovsdb_idl_add_column(ctx->idl, > + > > &nbrec_logical_router_static_route_col_route_table); > > } > > static char * OVS_WARN_UNUSED_RESULT > @@ -4090,6 +4114,7 @@ nbctl_lr_route_add(struct ctl_context *ctx) > } > } > > + char *route_table = shash_find_data(&ctx->options, > > "--route-table"); > > bool v6_prefix = false; > prefix = normalize_ipv4_prefix_str(ctx->argv[2]); > if (!prefix) { > @@ -4166,7 +4191,8 @@ nbctl_lr_route_add(struct ctl_context *ctx) > bool ecmp = shash_find(&ctx->options, "--ecmp") != NULL || > ecmp_symmetric_reply; > struct nbrec_logical_router_static_route *route = > - nbctl_lr_get_route(lr, prefix, next_hop, is_src_route, ecmp); > + nbctl_lr_get_route(lr, prefix, next_hop, is_src_route, ecmp, > + route_table); > > /* Validations for nexthop = "discard" */ > if (is_discard_route) { > @@ -4230,7 +4256,8 @@ nbctl_lr_route_add(struct ctl_context *ctx) > } > > struct nbrec_logical_router_static_route *discard_route = > - nbctl_lr_get_route(lr, prefix, "discard", is_src_route, true); > + nbctl_lr_get_route(lr, prefix, "discard", is_src_route, true, > + route_table); > if (discard_route) { > ctl_error(ctx, "discard nexthop for the same ECMP route > > exists."); > > goto cleanup; > @@ -4246,6 +4273,9 @@ nbctl_lr_route_add(struct ctl_context *ctx) > if (policy) { > nbrec_logical_router_static_route_set_policy(route, policy); > } > + if (route_table) { > + nbrec_logical_router_static_route_set_route_table(route, > > route_table); > > + } > > if (ecmp_symmetric_reply) { > const struct smap options = SMAP_CONST1(&options, > @@ -4289,6 +4319,8 @@ nbctl_pre_lr_route_del(struct ctl_context *ctx) > > &nbrec_logical_router_static_route_col_nexthop); > > ovsdb_idl_add_column(ctx->idl, > > &nbrec_logical_router_static_route_col_output_port); > > + ovsdb_idl_add_column(ctx->idl, > + > > &nbrec_logical_router_static_route_col_route_table); > > > } > > @@ -4302,6 +4334,7 @@ nbctl_lr_route_del(struct ctl_context *ctx) > return; > } > > + const char *route_table = shash_find_data(&ctx->options, > > "--route-table"); > > const char *policy = shash_find_data(&ctx->options, "--policy"); > bool is_src_route = false; > if (policy) { > @@ -4392,6 +4425,14 @@ nbctl_lr_route_del(struct ctl_context *ctx) > } > } > > + /* Strict compare for route_table. > + * If route_table was not specified, > + * lookup for routes with empty route_table value. */ > + if (strcmp(lr->static_routes[i]->route_table, > + route_table ? route_table : "")) { > + continue; > + } > + > /* Compare output_port, if specified. */ > if (output_port) { > char *rt_output_port = lr->static_routes[i]->output_port; > @@ -5115,6 +5156,41 @@ nbctl_pre_lrp_del_gateway_chassis(struct > > ctl_context *ctx) > > ovsdb_idl_add_column(ctx->idl, > > &nbrec_gateway_chassis_col_chassis_name); > > } > > +static void > +nbctl_pre_lrp_options(struct ctl_context *ctx) > +{ > + ovsdb_idl_add_column(ctx->idl, > > &nbrec_logical_router_port_col_name); > > + ovsdb_idl_add_column(ctx->idl, > > &nbrec_logical_router_port_col_options); > > +} > + > +static void > +nbctl_lrp_set_options(struct ctl_context *ctx) > +{ > + const char *id = ctx->argv[1]; > + const struct nbrec_logical_router_port *lrp = NULL; > + size_t i; > + struct smap options = SMAP_INITIALIZER(&options); > + > + char *error = lrp_by_name_or_uuid(ctx, id, true, &lrp); > + if (error) { > + ctx->error = error; > + return; > + } > + for (i = 2; i < ctx->argc; i++) { > + char *key, *value; > + value = xstrdup(ctx->argv[i]); > + key = strsep(&value, "="); > + if (value) { > + smap_add(&options, key, value); > + } > + free(key); > + } > + > + nbrec_logical_router_port_set_options(lrp, &options); > + > + smap_destroy(&options); > +} > + > /* Removes logical router port 'lrp->gateway_chassis[idx]'. */ > static void > remove_gc(const struct nbrec_logical_router_port *lrp, size_t idx) > @@ -5891,6 +5967,7 @@ route_cmp_details(const struct > > nbrec_logical_router_static_route *r1, > > } > return r1->output_port ? 1 : -1; > } > + > struct ipv4_route { > int priority; > ovs_be32 addr; > @@ -5900,6 +5977,11 @@ struct ipv4_route { > static int > __ipv4_route_cmp(const struct ipv4_route *r1, const struct ipv4_route > > *r2) > > { > + int rtb_cmp = strcmp(r1->route->route_table, > + r2->route->route_table); > + if (rtb_cmp) { > + return rtb_cmp; > + } > if (r1->priority != r2->priority) { > return r1->priority > r2->priority ? -1 : 1; > } > @@ -5931,6 +6013,11 @@ struct ipv6_route { > static int > __ipv6_route_cmp(const struct ipv6_route *r1, const struct ipv6_route > > *r2) > > { > + int rtb_cmp = strcmp(r1->route->route_table, > + r2->route->route_table); > + if (rtb_cmp) { > + return rtb_cmp; > + } > if (r1->priority != r2->priority) { > return r1->priority > r2->priority ? -1 : 1; > } > @@ -6018,6 +6105,8 @@ nbctl_pre_lr_route_list(struct ctl_context *ctx) > > &nbrec_logical_router_static_route_col_options); > > ovsdb_idl_add_column(ctx->idl, > &nbrec_logical_router_static_route_col_bfd); > + ovsdb_idl_add_column(ctx->idl, > + > > &nbrec_logical_router_static_route_col_route_table); > > } > > static void > @@ -6035,12 +6124,17 @@ nbctl_lr_route_list(struct ctl_context *ctx) > return; > } > > + char *route_table = shash_find_data(&ctx->options, > > "--route-table"); > > + > ipv4_routes = xmalloc(sizeof *ipv4_routes * lr->n_static_routes); > ipv6_routes = xmalloc(sizeof *ipv6_routes * lr->n_static_routes); > > for (int i = 0; i < lr->n_static_routes; i++) { > const struct nbrec_logical_router_static_route *route > = lr->static_routes[i]; > + if (route_table && strcmp(route->route_table, route_table)) { > + continue; > + } > unsigned int plen; > ovs_be32 ipv4; > const char *policy = route->policy ? route->policy : "dst-ip"; > @@ -6081,6 +6175,7 @@ nbctl_lr_route_list(struct ctl_context *ctx) > if (n_ipv4_routes) { > ds_put_cstr(&ctx->output, "IPv4 Routes\n"); > } > + const struct nbrec_logical_router_static_route *route; > for (int i = 0; i < n_ipv4_routes; i++) { > bool ecmp = false; > if (i < n_ipv4_routes - 1 && > @@ -6091,6 +6186,15 @@ nbctl_lr_route_list(struct ctl_context *ctx) > &ipv4_routes[i - 1])) { > ecmp = true; > } > + > + route = ipv4_routes[i].route; > + if (!i || (i > 0 && strcmp(route->route_table, > + ipv4_routes[i - > > 1].route->route_table))) { > > + ds_put_format(&ctx->output, "%sRoute Table %s:\n", i ? > > "\n" : "", > > + strlen(route->route_table) ? > > route->route_table > > + : "global"); > + } > + > print_route(ipv4_routes[i].route, &ctx->output, ecmp); > } > > @@ -6108,6 +6212,15 @@ nbctl_lr_route_list(struct ctl_context *ctx) > &ipv6_routes[i - 1])) { > ecmp = true; > } > + > + route = ipv6_routes[i].route; > + if (!i || (i > 0 && strcmp(route->route_table, > + ipv6_routes[i - > > 1].route->route_table))) { > > + ds_put_format(&ctx->output, "%sRoute Table %s:\n", i ? > > "\n" : "", > > + strlen(route->route_table) ? > > route->route_table > > + : "global"); > + } > + > print_route(ipv6_routes[i].route, &ctx->output, ecmp); > } > > @@ -6926,6 +7039,8 @@ static const struct ctl_command_syntax > > nbctl_commands[] = { > > "PORT CHASSIS [PRIORITY]", > nbctl_pre_lrp_set_gateway_chassis, nbctl_lrp_set_gateway_chassis, > NULL, "--may-exist", RW }, > + { "lrp-set-options", 1, INT_MAX, "PORT KEY=VALUE [KEY=VALUE]...", > + nbctl_pre_lrp_options, nbctl_lrp_set_options, NULL, "", RW }, > { "lrp-del-gateway-chassis", 2, 2, "PORT CHASSIS", > nbctl_pre_lrp_del_gateway_chassis, nbctl_lrp_del_gateway_chassis, > NULL, "", RW }, > @@ -6949,12 +7064,13 @@ static const struct ctl_command_syntax > > nbctl_commands[] = { > > /* logical router route commands. */ > { "lr-route-add", 3, 4, "ROUTER PREFIX NEXTHOP [PORT]", > nbctl_pre_lr_route_add, nbctl_lr_route_add, NULL, > - "--may-exist,--ecmp,--ecmp-symmetric-reply,--policy=,--bfd?", RW > > }, > > + > > > "--may-exist,--ecmp,--ecmp-symmetric-reply,--policy=,--route-table=,--bfd?", > > + RW }, > { "lr-route-del", 1, 4, "ROUTER [PREFIX [NEXTHOP [PORT]]]", > nbctl_pre_lr_route_del, nbctl_lr_route_del, NULL, > - "--if-exists,--policy=", RW }, > + "--if-exists,--policy=,--route-table=", RW }, > { "lr-route-list", 1, 1, "ROUTER", nbctl_pre_lr_route_list, > - nbctl_lr_route_list, NULL, "", RO }, > + nbctl_lr_route_list, NULL, "--route-table=", RO }, > > /* Policy commands */ > { "lr-policy-add", 4, INT_MAX, > -- > 2.30.0 > > _______________________________________________ > dev mailing list > dev@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > > _______________________________________________ > dev mailing list > dev@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > >
regards, Vladislav Odintsov > On 16 Oct 2021, at 03:20, Han Zhou <hzhou@ovn.org> wrote: > > On Fri, Oct 15, 2021 at 2:36 AM Vladislav Odintsov <odivlad@gmail.com> > wrote: > >> >> >> Regards, >> Vladislav Odintsov >> >> On 15 Oct 2021, at 08:42, Han Zhou <hzhou@ovn.org> wrote: >> >> On Thu, Oct 14, 2021 at 12:58 AM Vladislav Odintsov <odivlad@gmail.com> >> wrote: >> >> >> Hi Han, >> >> Thanks for the review. >> >> Regards, >> Vladislav Odintsov >> >> On 14 Oct 2021, at 08:13, Han Zhou <hzhou@ovn.org> wrote: >> >> >> >> On Tue, Oct 5, 2021 at 1:26 PM Vladislav Odintsov <odivlad@gmail.com> >> >> wrote: >> >> >> This patch extends Logical Router's routing functionality. >> Now user may create multiple routing tables within a Logical Router >> and assign them to Logical Router Ports. >> >> Traffic coming from Logical Router Port with assigned route_table >> is checked against global routes if any (Logical_Router_Static_Routes >> whith empty route_table field), next against directly connected routes >> >> >> This is not accurate. The "directly connected routes" is NOT after the >> >> global routes. Their priority only depends on the prefix length. >> >> >> and then Logical_Router_Static_Routes with same route_table value as >> in Logical_Router_Port options:route_table field. >> >> A new Logical Router ingress table #10 is added - IN_IP_ROUTING_PRE. >> In this table packets which come from LRPs with configured >> options:route_table field are checked against inport and in OVS >> register 7 unique non-zero value identifying route table is written. >> >> Then in 11th table IN_IP_ROUTING routes which have non-empty >> `route_table` field are added with additional match on reg7 value >> associated with appropriate route_table. >> >> >> Hi Vladislav, >> >> First of all, sorry for the delayed review, and thanks for implementing >> >> this new feature. >> >> >> I have some questions regarding the feature itself. I remember that we >> >> had some discussion earlier for this feature, but it seems I misunderstood >> the feature you are implementing here. I thought by multiple routing tables >> you were trying to support something like VRF in physical routers, but this >> seems to be something different. According to your implementation, instead >> of assigning LRPs to different routing tables, a LRP can actually >> participate to both the global routing table and the table with a specific >> ID. For ingress, global routes are prefered over other routes; for egress >> (i.e. forwarding to the next hop), it doesn't even enforce any table-id >> check, so packet routed by a entry of table-X can go out of a LRP with >> table-id Y. Is my understanding correct about the desired behavior of this >> feature? >> >> >> >> Yes, your understanding is correct. >> This is not VRF. At first glance VRF can be done just by using LR-per-VRF >> >> without any code modifications (maybe there are corner cases, but it’s not >> something I’m currently working on). >> >> I agree VRF can be achieved by just separate LRs. I am trying to understand >> why multiple LRs wouldn't satisfy your use case here. >> >> LRP can be optionally assigned to specific routing table name. This means >> >> that for LR ingress pipeline packets coming from this specific LRP would be >> checked against routes with same route_table value within appropriate LR. >> This is some kind of PBR, analog of "ip rule add iif <interface name> >> lookup <id>". >> >> There is one specific use-case, which requires special handling: >> >> directly-connected routes (subnet CIDRs from connected to this LR LRPs). >> These routes can’t be added manually by user, though routing between LRPs >> belonging to different routing tables is still needed. So, these routes >> should be added to global routing table to override routing for LRPs with >> configured routing tables. If for some reason user wants to prohibit IP >> connectivity to any LRP (honestly, I can’t imagine, why), it can be done by >> utilising OVN PBR with drop action or a route with "discard" nexthop. But >> I’d think in this case whether this LRP is needed in this LR. >> >> Also, if user wants to create routes, which apply for all LRPs from all >> >> routing tables, it can be done by adding new entries to global routing >> table. >> >> And the third reason to have global routing table - a fully >> >> backward-compatible solution. All users, who don’t need multiple routing >> tables support just continue using static routing in the same manner >> without any changes. >> >> >> >> If this is correct, it doesn't seem to be a common/standard requirement >> >> (or please educate me if I am wrong). Could you explain a little more about >> the actual use cases: what kind of real world problems need to be solved by >> this feature or how are you going to use this. For example, why would a >> port need to participate in both routing tables? It looks like what you >> really need is policy routing instead of multiple isolated routing tables. >> I understand that you already use policy routing to implement ACLs, so it >> is not convenient to combine other policies (e.g. inport based routing) >> into the policy routing stage. If that's the case, would it be more generic >> to support multiple policy routing stages? My concern to the current >> approach is that it is implemented for a very special use case. It makes >> the code more complex but when there is a slightly different requirement in >> the future it becomes insufficient. I am thinking that policy routing seems >> more flexible and has more potential to be made more generic. Maybe I will >> have a better understanding when I hear more detailed use cases and >> considerations from you. >> >> >> >> I can't agree here in it’s uncommon requirement. >> This implementation was inspired by AWS Route Tables feature [1]: within >> >> a VPC (LR in terms of OVN) user may create multiple routing tables and >> assign them to different subnets (LRPs) in multiple availability zones. >> Auto-generated directly-connected routes from LRPs CIDRs are working in the >> same manner as they do in AWS - apply to all subnets regardless of their >> association to routing table. GCP has similar behaviour: [2], Azure, I >> guess, too. >> >> Our public cloud (CROC Cloud Platform) supports AWS behaviour [3], so I >> >> primarily was oriented on it. Internally we already use this feature and >> currently it fits our use-case, but again I can't say it is specific. >> >> If it is for AWS/GCP alike routing features, then from what I understand >> what's implemented in this patch is a little different. To implement a VPC >> model in OVN, I think it is ok to have multiple LRs instead of only one LR >> for each VPC. So naturally I would use a LR to map to AWS's routing table. >> In AWS's document (the link you provided), it says: >> >> "A subnet can only be associated with one route table at a time" >> >> So basically in AWS a subnet is either attached to the default/main route >> table or a custom table, but not both, right? However, in your use case, a >> LRP (maps to a subnet) attaches to both "main" and a custom table, which >> seems not common to me. Or did I miss something? >> >> >> That’s true about AWS, but there is still a bit not accurate about OVN. >> Global routing table in OVN terms is not that AWS main route table is. >> Main route table is just a configuration hint for users for implicit route >> tables association with subnets. >> Implicitly-associated via main routing table subnets routing functions the >> same manner as a normal explicit route_table-subnet association. >> >> Global routing table in OVN is just a list of routes with higher priority >> than routes with configured "route_table". >> >> I do not offer to configure both tables at the same time. But it is >> _possible_ to do if required for some reason (for instance to configure >> some service chaining or just internal VPC services like metadata/another >> internal APIs, access to another services). >> Normally, if we talk about AWS Route Table to OVN, it is mostly one-to-one >> mapping, except "Local route": >> > >> Example: >> AWS Route Table rtb-xxx: >> 172.31.0.0/16: local route (VPC CIDR for subnets) >> 0.0.0.0/0: igw-XXX (internet gateway) >> >> AWS Route Table rtb-yyy: >> 172.31.0.0/16: local route (VPC CIDR for subnets) >> 5.5.5.5/32: instance-xxx >> >> This maps to OVN configuration (for one LR with one subnet 172.31.0.0/24): >> >> implicit route (not present in logical router static routes table): >> 172.31.0.0/24: LRP-subnet-xxx - has highest priority over other route >> table-oriented routes and can be threat as placed in global routing table >> > > What do you mean by highest priority here? It all depends on the prefix > length, right? If you have a static route entry that says: 172.31.0.0./25 > -> xyz, then this route will have higher priority than the directly > connected subnet. > Yes, it is true, but only for routes within “global” routing table. I left this to save previous behaviour. It is impossible to create a static route within any non-global routing table which overrides directly connected routes, because priority for “global” routes is added by 100 (though, it should add >2*128 to support same behavior for IPv6 as you already pointed). > >> >> Normal static routes: >> ip_prefix: 0.0.0.0/0, nexthop: <IP for edge LR’s LRP>, route_table: >> rtb-xxx >> ip_prefix: 5.5.5.5/32, nexthop: <IP of some LSP, which belongs to >> VM/container via which route is built>, route_table: rtb-yyy >> >> I guess, I understood the reason for misunderstanding: the global routing >> table, which I referred earlier is a routing table which has no value in >> "route_table" field and directly-connected routes at the same time. Latter >> have no records in logical router static routes, but I still referred them >> as a routes from "global routing table". I can think about terminology >> here, if it’s a root cause for misunderstanding. What do you think? >> > > Thanks for explaining. In fact I understand what you mean about "global > routing table", but I think you are implementing something different from > what AWS provides, primarily because you use a single LR instead of LR per > routing table. I understand there are reasons why you want to do it this > way, but here I can think of some challenges of your approach. For example, > in AWS we could do: > > route table main: > subnet S0: > 172.31.0.0/24 > routes: > 172.31.0.0/16: local > > route table R1: > subnet S1: > 172.31.1.0/24 > routes: > 172.31.0.0/16: local > 172.31.0.0/24: <some FW/GW> Wow. It’s a brand-new behaviour for AWS, about which I was not aware, as it appeared 1.5 months ago [1]. Previously, when I was writing this patch series in AWS such static route was impossible to add. You couldn’t add routes, which are the same or more specific than VPC CIDR block. Though in GCP you can’t create same or more specific than subnet route [2]. I think I can try to update my patch to support AWS approach here. But all this is relevant to single LR per VPC (not per route table because of described earlier blocker). Do you think it is reasonable with such inputs? > > Packet from S1 to S0 will go to FW/GW, where it may be dropped/forwarded to > S0/or redirected to something outside of AWS ... > > While in your approach with OVN, both subnets will be under the same > logical router, and with different table IDs assigned to the LRPs and > routes. But when a VM under S1 sends a packet to S0, it will go to the > destination directly because you are prioritizing the direct-connection > routes and they are under the same global route table, and there is no way > to force it to go through some FW/GW from the custom route table. You can > add a route to achieve this in the global table, because in your design the > global routes are still applicable to all LRPs and has higher priority, but > it would impact all the LRPs/subnets, while in the AWS design above it is > supposed to affect subnets under R1 only. > > In addition, for my understanding, S0 and S1 in AWS cannot communicate > directly, unless there are some specific routes on both route tables to > route them to some gateway. In other words, there are some isolations > between route tables. However, in your design with OVN, it is more of S0 and S1 can communicate with each other being in different route tables without any specific setup, the default unremovable “Local route” ensures this. User only can override "local" route with only existing subnet’s (more specific than "local route") CIDR to send traffic from S0 to S1 through some specific appliance(s) in S2. But basic IP connectivity always remains. Regardless of route tables subnet associations. So I’m still sure that one LR per VPC suites this scenario better, as I can’t imagine why to place subnets, which don’t need connectivity in one VPC. > policy routing without isolation at all. I am not saying that your design > is wrong, but just saying it implements very different behavior than AWS, > and I am trying to understand your real use cases to have a better > understanding of the feature you implemented. (You explained that you just > wanted the AWS behavior, but it looks not exactly the case). > Will fix places where the behaviour has difference with AWS. And thanks you for pointing such places. > >> >> >> In my opinion having this feature to be implemented using PBR is less >> >> convenient and native for users, who are familiar with behaviour for >> mentioned above public cloud platforms, because configuring routes should >> be done in routes section. And adding route table property seems native in >> this route, not in PBR. Moreover, I _think_ using >> Logical_Router_Static_Route to extend this feature for OVN-Interconnection >> becomes quite easy comparing to PBR (though, I didn’t try the latter). >> >> I agree if it is just AWS -like requirement, PBR is less convenient. >> >> I am trying to understand if it can be achieved with separate LRs. If not, >> what's special about the requirement, and is the current approach providing >> a solution common enough so that more use cases can also benefit from? >> Could you clarify a little more? Thanks again. >> >> That was our initial approach - to use separate LRs for each Route Table. >> We rejected that solution because met some difficulties and blocker. See >> below: >> >> Brief topology description if using LRs per Route Table: >> Imagine 2 subnets in VPC in 1st AZ, one in another AZ. Each subnet in it’s >> own Route Table (LR). >> All subnets must have IP connectivity, so we have to somehow interconnect >> these Route Table LRs. >> > > That's exactly the same case for AWS, right? If you want direct > connectivity, you would just put them under the same route table in AWS (or > same LR in OVN), right? > I’m sorry, my example was not good enough. Let’s remove AZ’s from that case as they’re not relevant to discussion. So, we’ve got 2 subnets S0 and S1, and suppose, they’ve got different route tables assigned to them. If we place them in a single LR, assign different route tables, they would have connectivity "out-of-box". Same in AWS: because "local route" is added automatically, can’t be removed or modified and ensures IP connectivity between all VPC subnets. > >> >> [BLOCKER] It is impossible to place route in route table 1 via VM from >> subnet assiciated to route table 2 if using per-RTB LR. Because in RTB-2 LR >> we have to add route from RTB 1 and this breaks route table isolation. >> Route Table 2 LR will start looking up routes and there could be routes >> from another route tables. This breaks the idea of having LR per Route >> Table completely. Here we rejected this solution and moved to adding >> support for route tables in OVN. >> >> > Would it be just the same problem in AWS? Why would a user route a subnet > of RTB-1 to a VM under RTB-2, and at the same time want route table > isolation? > With the single LR solution in OVN, you would not get the isolation between > the tables, because 1) all LRPs still share the global routing table, 2) > for output to nexthop there is no distinction between the output interfaces. > No, such problem does not exist. By route tables "isolation" I meant that routing rules configured in one routing table must not affect routing in another one. In this scenario user may want to route traffic to internet from S0 (RTB-0) through instance in S1 (RTB-1). For instance it can be a gateway service (FW, WAF, IPS, etc) in S1, which provides secure access for clients from VPC to the internet with NAT. So, configuration should be: - 0.0.0.0/0 via GW-INSTANCE in RTB-0 - 0.0.0.0/0 via edge LR in RTB-1 So, instances from all subnets with rtb-0 assigned route table will be routed to internet through GW-INSTANCE, which can analyse/drop/do whatever things with traffic. This is a small example to show how different route tables can be used in a single VPC. With multiple OVN LRs per VPC in AZ such topology can’t be created, see: In LR for RTB-0 we can add route 0.0.0.0/0 via <LR0-to-LR1 link IP>; bun in LR for RTB-1 we can only add route to internet through either edge-LR or GW-INSTANCE. Not both. So, this case can’t be implemented currently in OVN. Hope this helps to understand my problem. > >> But some more cons: >> >> 1. complex network topology. Interconnecting all LRs even with some >> transit switch is harder than having one LR and all VPC-related >> configuration is done in one place (in AZ). >> 2. Directly-connected routes. In case we have multiple Route Table LRs, we >> have to add route for each subnet from another LR. In case one LR per VPC >> all such routes are installed automatically and learnt via ovn-ic. >> 3. PBR, LBs. It’s much easier to implement PBR and configuring Load >> Balancers in one LR, than in multiple. >> 4. There could be very many LRs, LRPs, LSPs (datapaths in SB) - excess >> database records, huge database growth. >> 5. Extra client code complexity (updating routes required configuring >> routes in many LRs on different availibility zones); >> 5. We couldn’t use ovn-ic routes learning, because VPC requires out-of-box >> IP connectivity between subnets, and if connect Route Table LRs between >> AZs, because connecting multiple LRs from different Route Tables would >> learn routes without binding to route table. This requires additional >> isolation at the transit switches level. >> 6. There were some more problems. Here are listed some, which I could >> refresh in my mind. >> >> From our half-of-a-year experience using LR per VPC is very comfortable >> and it looks quite extendable it terms of network features. >> >> Let me know if this makes sense. >> >> > Thanks for the list of the cons! Most of the cons you listed above seem to > apply to the AWS model, too, right? It is ok to have different requirements Honestly, I can’t imagine these bullets to "AWS model". Maybe only ovn-ic, which requires modifications in ovn-ic codebase to support route_tables in routes. But this work is already done. > than AWS, but we'd better define it clearly and it would be helpful with > some typical use cases that solve specific problems. I have more > information now after your explanation, but still not really clear under > what scenarios would this feature be used. > > @Numan Siddique <numans@ovn.org> @Mark Michelson <mmichels@redhat.com> > could you let me know your thoughts on this, too? Would you see similar use > cases in Redhat customers? Or does this design match well with your > potential use cases? I'd like to be careful when implementing something > like this and make sure we are doing it the right way. One comment here from my side: by "this design" we should talk about "behaviour similar to AWS/other public platforms route tables support". I think that it is a good feature for opensource SDN, which can be futher used by opensource cloud platforms like OpenStack to extend routing capabilities. > > Thanks, > Han 1: https://aws.amazon.com/blogs/aws/inspect-subnet-to-subnet-traffic-with-amazon-vpc-more-specific-routing/ 2: https://cloud.google.com/vpc/docs/routes#subnet-routes > >> Han >> >> >> >> I haven't finished reviewing the code yet, but I have one comment >> >> regarding adding 100 to the priority of the global routes. For IPv6, the >> priority range from 0 to 120x2=240, so adding 100 is not enough. It would >> create overlapping priority ranges, and some table-id specific route >> entries may override the global routes. >> >> >> >> Thanks, I’ll dig into this when you finish review. >> >> >> Let me know if I answered your questions or if you have new ones. >> Again many thanks for your time and digging into this patch series. >> >> 1: https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Route_Tables.html >> 2: https://cloud.google.com/vpc/docs/routes#subnet-routes >> 3: https://docs.cloud.croc.ru/en/services/networks/routetables.html >> >> Thanks, >> Han >> >> >> Signed-off-by: Vladislav Odintsov <odivlad@gmail.com> >> Acked-by: Numan Siddique <numans@ovn.org> >> --- >> northd/northd.c | 159 ++++++++++++--- >> northd/ovn-northd.8.xml | 63 ++++-- >> ovn-nb.ovsschema | 5 +- >> ovn-nb.xml | 30 +++ >> tests/ovn-ic.at | 4 + >> tests/ovn-nbctl.at | 196 +++++++++++++++++- >> tests/ovn-northd.at | 76 ++++++- >> tests/ovn.at | 441 +++++++++++++++++++++++++++++++++++++++- >> utilities/ovn-nbctl.c | 134 +++++++++++- >> 9 files changed, 1041 insertions(+), 67 deletions(-) >> >> diff --git a/northd/northd.c b/northd/northd.c >> index 092eca829..6a020cb2e 100644 >> --- a/northd/northd.c >> +++ b/northd/northd.c >> @@ -148,15 +148,16 @@ enum ovn_stage { >> PIPELINE_STAGE(ROUTER, IN, ECMP_STATEFUL, 7, >> >> "lr_in_ecmp_stateful") \ >> >> PIPELINE_STAGE(ROUTER, IN, ND_RA_OPTIONS, 8, >> >> "lr_in_nd_ra_options") \ >> >> PIPELINE_STAGE(ROUTER, IN, ND_RA_RESPONSE, 9, >> >> "lr_in_nd_ra_response") \ >> >> - PIPELINE_STAGE(ROUTER, IN, IP_ROUTING, 10, >> >> "lr_in_ip_routing") \ >> >> - PIPELINE_STAGE(ROUTER, IN, IP_ROUTING_ECMP, 11, >> >> "lr_in_ip_routing_ecmp") \ >> >> - PIPELINE_STAGE(ROUTER, IN, POLICY, 12, "lr_in_policy") >> >> \ >> >> - PIPELINE_STAGE(ROUTER, IN, POLICY_ECMP, 13, >> >> "lr_in_policy_ecmp") \ >> >> - PIPELINE_STAGE(ROUTER, IN, ARP_RESOLVE, 14, >> >> "lr_in_arp_resolve") \ >> >> - PIPELINE_STAGE(ROUTER, IN, CHK_PKT_LEN , 15, >> >> "lr_in_chk_pkt_len") \ >> >> - PIPELINE_STAGE(ROUTER, IN, LARGER_PKTS, 16, >> >> "lr_in_larger_pkts") \ >> >> - PIPELINE_STAGE(ROUTER, IN, GW_REDIRECT, 17, >> >> "lr_in_gw_redirect") \ >> >> - PIPELINE_STAGE(ROUTER, IN, ARP_REQUEST, 18, >> >> "lr_in_arp_request") \ >> >> + PIPELINE_STAGE(ROUTER, IN, IP_ROUTING_PRE, 10, >> >> "lr_in_ip_routing_pre") \ >> >> + PIPELINE_STAGE(ROUTER, IN, IP_ROUTING, 11, >> >> "lr_in_ip_routing") \ >> >> + PIPELINE_STAGE(ROUTER, IN, IP_ROUTING_ECMP, 12, >> >> "lr_in_ip_routing_ecmp") \ >> >> + PIPELINE_STAGE(ROUTER, IN, POLICY, 13, "lr_in_policy") >> >> \ >> >> + PIPELINE_STAGE(ROUTER, IN, POLICY_ECMP, 14, >> >> "lr_in_policy_ecmp") \ >> >> + PIPELINE_STAGE(ROUTER, IN, ARP_RESOLVE, 15, >> >> "lr_in_arp_resolve") \ >> >> + PIPELINE_STAGE(ROUTER, IN, CHK_PKT_LEN, 16, >> >> "lr_in_chk_pkt_len") \ >> >> + PIPELINE_STAGE(ROUTER, IN, LARGER_PKTS, 17, >> >> "lr_in_larger_pkts") \ >> >> + PIPELINE_STAGE(ROUTER, IN, GW_REDIRECT, 18, >> >> "lr_in_gw_redirect") \ >> >> + PIPELINE_STAGE(ROUTER, IN, ARP_REQUEST, 19, >> >> "lr_in_arp_request") \ >> >> \ >> /* Logical router egress stages. */ \ >> PIPELINE_STAGE(ROUTER, OUT, UNDNAT, 0, "lr_out_undnat") >> >> \ >> >> @@ -225,6 +226,7 @@ enum ovn_stage { >> #define REG_NEXT_HOP_IPV6 "xxreg0" >> #define REG_SRC_IPV4 "reg1" >> #define REG_SRC_IPV6 "xxreg1" >> +#define REG_ROUTE_TABLE_ID "reg7" >> >> #define REG_ORIG_TP_DPORT_ROUTER "reg9[16..31]" >> >> @@ -287,8 +289,9 @@ enum ovn_stage { >> * | R6 | UNUSED | X | | G | >> >> IN_IP_ROUTING)| >> >> * | | | R | | 1 | >> >> | >> >> * +-----+--------------------------+ E | UNUSED | | >> >> | >> >> - * | R7 | UNUSED | G | | | >> >> | >> >> - * | | | 3 | | | >> >> | >> >> + * | R7 | ROUTE_TABLE_ID | G | | | >> >> | >> >> + * | | (>= IN_IP_ROUTING_PRE && | 3 | | | >> >> | >> >> + * | | <= IN_IP_ROUTING) | | | | >> >> | >> >> * >> >> >> +-----+--------------------------+---+-----------------+---+---------------+ >> >> * | R8 | ECMP_GROUP_ID | | | >> * | | ECMP_MEMBER_ID | X | | >> @@ -8511,11 +8514,72 @@ cleanup: >> ds_destroy(&actions); >> } >> >> +static uint32_t >> +route_table_add(struct simap *route_tables, const char >> >> *route_table_name) >> >> +{ >> + /* route table ids start from 1 */ >> + uint32_t rtb_id = simap_count(route_tables) + 1; >> + >> + if (rtb_id == UINT16_MAX) { >> + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1); >> + VLOG_WARN_RL(&rl, "too many route tables for Logical Router."); >> + return 0; >> + } >> + >> + if (!simap_put(route_tables, route_table_name, rtb_id)) { >> + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1); >> + VLOG_WARN_RL(&rl, "Route table id unexpectedly appeared"); >> + } >> + >> + return rtb_id; >> +} >> + >> +static uint32_t >> +get_route_table_id(struct simap *route_tables, const char >> >> *route_table_name) >> >> +{ >> + if (!route_table_name || !strlen(route_table_name)) { >> + return 0; >> + } >> + >> + uint32_t rtb_id = simap_get(route_tables, route_table_name); >> + if (!rtb_id) { >> + rtb_id = route_table_add(route_tables, route_table_name); >> + } >> + >> + return rtb_id; >> +} >> + >> +static void >> +build_route_table_lflow(struct ovn_datapath *od, struct hmap *lflows, >> + struct nbrec_logical_router_port *lrp, >> + struct simap *route_tables) >> +{ >> + struct ds match = DS_EMPTY_INITIALIZER; >> + struct ds actions = DS_EMPTY_INITIALIZER; >> + >> + const char *route_table_name = smap_get(&lrp->options, >> >> "route_table"); >> >> + uint32_t rtb_id = get_route_table_id(route_tables, >> >> route_table_name); >> >> + if (!rtb_id) { >> + return; >> + } >> + >> + ds_put_format(&match, "inport == \"%s\"", lrp->name); >> + ds_put_format(&actions, "%s = %d; next;", >> + REG_ROUTE_TABLE_ID, rtb_id); >> + >> + ovn_lflow_add(lflows, od, S_ROUTER_IN_IP_ROUTING_PRE, 100, >> + ds_cstr(&match), ds_cstr(&actions)); >> + >> + ds_destroy(&match); >> + ds_destroy(&actions); >> +} >> + >> struct parsed_route { >> struct ovs_list list_node; >> struct in6_addr prefix; >> unsigned int plen; >> bool is_src_route; >> + uint32_t route_table_id; >> uint32_t hash; >> const struct nbrec_logical_router_static_route *route; >> bool ecmp_symmetric_reply; >> @@ -8540,7 +8604,7 @@ find_static_route_outport(struct ovn_datapath >> >> *od, struct hmap *ports, >> >> * Otherwise return NULL. */ >> static struct parsed_route * >> parsed_routes_add(struct ovn_datapath *od, struct hmap *ports, >> - struct ovs_list *routes, >> + struct ovs_list *routes, struct simap *route_tables, >> const struct nbrec_logical_router_static_route >> >> *route, >> >> struct hmap *bfd_connections) >> { >> @@ -8622,6 +8686,7 @@ parsed_routes_add(struct ovn_datapath *od, struct >> >> hmap *ports, >> >> struct parsed_route *pr = xzalloc(sizeof *pr); >> pr->prefix = prefix; >> pr->plen = plen; >> + pr->route_table_id = get_route_table_id(route_tables, >> >> route->route_table); >> >> pr->is_src_route = (route->policy && !strcmp(route->policy, >> "src-ip")); >> pr->hash = route_hash(pr); >> @@ -8655,6 +8720,7 @@ struct ecmp_groups_node { >> struct in6_addr prefix; >> unsigned int plen; >> bool is_src_route; >> + uint32_t route_table_id; >> uint16_t route_count; >> struct ovs_list route_list; /* Contains ecmp_route_list_node */ >> }; >> @@ -8663,7 +8729,7 @@ static void >> ecmp_groups_add_route(struct ecmp_groups_node *group, >> const struct parsed_route *route) >> { >> - if (group->route_count == UINT16_MAX) { >> + if (group->route_count == UINT16_MAX) { >> static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1); >> VLOG_WARN_RL(&rl, "too many routes in a single ecmp group."); >> return; >> @@ -8692,6 +8758,7 @@ ecmp_groups_add(struct hmap *ecmp_groups, >> eg->prefix = route->prefix; >> eg->plen = route->plen; >> eg->is_src_route = route->is_src_route; >> + eg->route_table_id = route->route_table_id; >> ovs_list_init(&eg->route_list); >> ecmp_groups_add_route(eg, route); >> >> @@ -8705,7 +8772,8 @@ ecmp_groups_find(struct hmap *ecmp_groups, struct >> >> parsed_route *route) >> >> HMAP_FOR_EACH_WITH_HASH (eg, hmap_node, route->hash, ecmp_groups) { >> if (ipv6_addr_equals(&eg->prefix, &route->prefix) && >> eg->plen == route->plen && >> - eg->is_src_route == route->is_src_route) { >> + eg->is_src_route == route->is_src_route && >> + eg->route_table_id == route->route_table_id) { >> return eg; >> } >> } >> @@ -8752,7 +8820,8 @@ unique_routes_remove(struct hmap *unique_routes, >> HMAP_FOR_EACH_WITH_HASH (ur, hmap_node, route->hash, >> >> unique_routes) { >> >> if (ipv6_addr_equals(&route->prefix, &ur->route->prefix) && >> route->plen == ur->route->plen && >> - route->is_src_route == ur->route->is_src_route) { >> + route->is_src_route == ur->route->is_src_route && >> + route->route_table_id == ur->route->route_table_id) { >> hmap_remove(unique_routes, &ur->hmap_node); >> const struct parsed_route *existed_route = ur->route; >> free(ur); >> @@ -8790,9 +8859,9 @@ build_route_prefix_s(const struct in6_addr >> >> *prefix, unsigned int plen) >> >> } >> >> static void >> -build_route_match(const struct ovn_port *op_inport, const char >> >> *network_s, >> >> - int plen, bool is_src_route, bool is_ipv4, struct ds >> >> *match, >> >> - uint16_t *priority) >> +build_route_match(const struct ovn_port *op_inport, uint32_t rtb_id, >> + const char *network_s, int plen, bool is_src_route, >> + bool is_ipv4, struct ds *match, uint16_t *priority) >> { >> const char *dir; >> /* The priority here is calculated to implement >> >> longest-prefix-match >> >> @@ -8808,6 +8877,15 @@ build_route_match(const struct ovn_port >> >> *op_inport, const char *network_s, >> >> if (op_inport) { >> ds_put_format(match, "inport == %s && ", op_inport->json_key); >> } >> + if (rtb_id) { >> + ds_put_format(match, "%s == %d && ", REG_ROUTE_TABLE_ID, >> >> rtb_id); >> >> + } else { >> + /* Route-table assigned LRPs' routes should have lower priority >> + * in order not to affect directly-connected global routes. >> + * So, enlarge non-route-table routes priority by 100. >> + */ >> + *priority += 100; >> + } >> ds_put_format(match, "ip%s.%s == %s/%d", is_ipv4 ? "4" : "6", dir, >> network_s, plen); >> } >> @@ -8946,7 +9024,7 @@ add_ecmp_symmetric_reply_flows(struct hmap >> >> *lflows, >> >> out_port->lrp_networks.ea_s, >> IN6_IS_ADDR_V4MAPPED(&route->prefix) ? "" : "xx", >> port_ip, out_port->json_key); >> - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_IP_ROUTING, 300, >> + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_IP_ROUTING, 400, >> ds_cstr(&match), ds_cstr(&actions), >> &st_route->header_); >> >> @@ -8976,8 +9054,8 @@ build_ecmp_route_flow(struct hmap *lflows, struct >> >> ovn_datapath *od, >> >> struct ds route_match = DS_EMPTY_INITIALIZER; >> >> char *prefix_s = build_route_prefix_s(&eg->prefix, eg->plen); >> - build_route_match(NULL, prefix_s, eg->plen, eg->is_src_route, >> >> is_ipv4, >> >> - &route_match, &priority); >> + build_route_match(NULL, eg->route_table_id, prefix_s, eg->plen, >> + eg->is_src_route, is_ipv4, &route_match, >> >> &priority); >> >> free(prefix_s); >> >> struct ds actions = DS_EMPTY_INITIALIZER; >> @@ -9052,8 +9130,8 @@ static void >> add_route(struct hmap *lflows, struct ovn_datapath *od, >> const struct ovn_port *op, const char *lrp_addr_s, >> const char *network_s, int plen, const char *gateway, >> - bool is_src_route, const struct ovsdb_idl_row *stage_hint, >> - bool is_discard_route) >> + bool is_src_route, const uint32_t rtb_id, >> + const struct ovsdb_idl_row *stage_hint, bool >> >> is_discard_route) >> >> { >> bool is_ipv4 = strchr(network_s, '.') ? true : false; >> struct ds match = DS_EMPTY_INITIALIZER; >> @@ -9068,8 +9146,8 @@ add_route(struct hmap *lflows, struct >> >> ovn_datapath *od, >> >> op_inport = op; >> } >> } >> - build_route_match(op_inport, network_s, plen, is_src_route, >> >> is_ipv4, >> >> - &match, &priority); >> + build_route_match(op_inport, rtb_id, network_s, plen, is_src_route, >> + is_ipv4, &match, &priority); >> >> struct ds common_actions = DS_EMPTY_INITIALIZER; >> struct ds actions = DS_EMPTY_INITIALIZER; >> @@ -9132,7 +9210,8 @@ build_static_route_flow(struct hmap *lflows, >> >> struct ovn_datapath *od, >> >> char *prefix_s = build_route_prefix_s(&route_->prefix, >> >> route_->plen); >> >> add_route(lflows, route_->is_discard_route ? od : out_port->od, >> >> out_port, >> >> lrp_addr_s, prefix_s, route_->plen, route->nexthop, >> - route_->is_src_route, &route->header_, >> >> route_->is_discard_route); >> >> + route_->is_src_route, route_->route_table_id, >> >> &route->header_, >> >> + route_->is_discard_route); >> >> free(prefix_s); >> } >> @@ -10584,6 +10663,17 @@ build_ND_RA_flows_for_lrouter(struct >> >> ovn_datapath *od, struct hmap *lflows) >> >> } >> } >> >> +/* Logical router ingress table IP_ROUTING_PRE: >> + * by default goto next. (priority 0). */ >> +static void >> +build_ip_routing_pre_flows_for_lrouter(struct ovn_datapath *od, >> + struct hmap *lflows) >> +{ >> + if (od->nbr) { >> + ovn_lflow_add(lflows, od, S_ROUTER_IN_IP_ROUTING_PRE, 0, "1", >> >> "next;"); >> >> + } >> +} >> + >> /* Logical router ingress table IP_ROUTING : IP Routing. >> * >> * A packet that arrives at this table is an IP packet that should be >> @@ -10609,14 +10699,14 @@ build_ip_routing_flows_for_lrouter_port( >> for (int i = 0; i < op->lrp_networks.n_ipv4_addrs; i++) { >> add_route(lflows, op->od, op, >> >> op->lrp_networks.ipv4_addrs[i].addr_s, >> >> op->lrp_networks.ipv4_addrs[i].network_s, >> - op->lrp_networks.ipv4_addrs[i].plen, NULL, false, >> + op->lrp_networks.ipv4_addrs[i].plen, NULL, >> >> false, 0, >> >> &op->nbrp->header_, false); >> } >> >> for (int i = 0; i < op->lrp_networks.n_ipv6_addrs; i++) { >> add_route(lflows, op->od, op, >> >> op->lrp_networks.ipv6_addrs[i].addr_s, >> >> op->lrp_networks.ipv6_addrs[i].network_s, >> - op->lrp_networks.ipv6_addrs[i].plen, NULL, false, >> + op->lrp_networks.ipv6_addrs[i].plen, NULL, >> >> false, 0, >> >> &op->nbrp->header_, false); >> } >> } else if (lsp_is_router(op->nbsp)) { >> @@ -10639,7 +10729,7 @@ build_ip_routing_flows_for_lrouter_port( >> add_route(lflows, peer->od, peer, >> peer->lrp_networks.ipv4_addrs[0].addr_s, >> laddrs->ipv4_addrs[k].network_s, >> - laddrs->ipv4_addrs[k].plen, NULL, false, >> + laddrs->ipv4_addrs[k].plen, NULL, false, >> >> 0, >> >> &peer->nbrp->header_, false); >> } >> } >> @@ -10659,10 +10749,17 @@ build_static_route_flows_for_lrouter( >> struct hmap ecmp_groups = HMAP_INITIALIZER(&ecmp_groups); >> struct hmap unique_routes = HMAP_INITIALIZER(&unique_routes); >> struct ovs_list parsed_routes = >> >> OVS_LIST_INITIALIZER(&parsed_routes); >> >> + struct simap route_tables = SIMAP_INITIALIZER(&route_tables); >> struct ecmp_groups_node *group; >> + >> + for (int i = 0; i < od->nbr->n_ports; i++) { >> + build_route_table_lflow(od, lflows, od->nbr->ports[i], >> + &route_tables); >> + } >> + >> for (int i = 0; i < od->nbr->n_static_routes; i++) { >> struct parsed_route *route = >> - parsed_routes_add(od, ports, &parsed_routes, >> + parsed_routes_add(od, ports, &parsed_routes, >> >> &route_tables, >> >> od->nbr->static_routes[i], >> >> bfd_connections); >> >> if (!route) { >> continue; >> @@ -10695,6 +10792,7 @@ build_static_route_flows_for_lrouter( >> ecmp_groups_destroy(&ecmp_groups); >> unique_routes_destroy(&unique_routes); >> parsed_routes_destroy(&parsed_routes); >> + simap_destroy(&route_tables); >> } >> } >> >> @@ -12800,6 +12898,7 @@ build_lswitch_and_lrouter_iterate_by_od(struct >> >> ovn_datapath *od, >> >> build_neigh_learning_flows_for_lrouter(od, lsi->lflows, >> >> &lsi->match, >> >> &lsi->actions, >> >> lsi->meter_groups); >> >> build_ND_RA_flows_for_lrouter(od, lsi->lflows); >> + build_ip_routing_pre_flows_for_lrouter(od, lsi->lflows); >> build_static_route_flows_for_lrouter(od, lsi->lflows, lsi->ports, >> lsi->bfd_connections); >> build_mcast_lookup_flows_for_lrouter(od, lsi->lflows, &lsi->match, >> diff --git a/northd/ovn-northd.8.xml b/northd/ovn-northd.8.xml >> index 39f4eaa0c..cc2e25367 100644 >> --- a/northd/ovn-northd.8.xml >> +++ b/northd/ovn-northd.8.xml >> @@ -2899,7 +2899,7 @@ icmp6 { >> >> <p> >> If ECMP routes with symmetric reply are configured in the >> - <code>OVN_Northbound</code> database for a gateway router, a >> >> priority-300 >> >> + <code>OVN_Northbound</code> database for a gateway router, a >> >> priority-400 >> >> flow is added for each router port on which symmetric replies are >> configured. The matching logic for these ports essentially >> >> reverses the >> >> configured logic of the ECMP route. So for instance, a route >> >> with a >> >> @@ -3245,7 +3245,35 @@ output; >> </li> >> </ul> >> >> - <h3>Ingress Table 10: IP Routing</h3> >> + <h3>Ingress Table 10: IP Routing Pre</h3> >> + >> + <p> >> + If a packet arrived at this table from Logical Router Port >> >> <var>P</var> >> >> + which has <code>options:route_table</code> value set, a logical >> >> flow with >> >> + match <code>inport == "<var>P</var>"</code> with priority 100 >> >> and action, >> >> + setting unique-generated per-datapath 32-bit value (non-zero) in >> >> OVS >> >> + register 7. This register is checked in next table. >> + </p> >> + >> + <p> >> + This table contains the following logical flows: >> + </p> >> + >> + <ul> >> + <li> >> + <p> >> + Priority-100 flow with match <code>inport == >> >> "LRP_NAME"</code> value >> >> + and action, which set route table identifier in reg7. >> + </p> >> + >> + <p> >> + A priority-0 logical flow with match <code>1</code> has >> >> actions >> >> + <code>next;</code>. >> + </p> >> + </li> >> + </ul> >> + >> + <h3>Ingress Table 11: IP Routing</h3> >> >> <p> >> A packet that arrives at this table is an IP packet that should >> >> be >> >> @@ -3316,10 +3344,10 @@ output; >> <p> >> IPv4 routing table. For each route to IPv4 network >> >> <var>N</var> with >> >> netmask <var>M</var>, on router port <var>P</var> with IP >> >> address >> >> - <var>A</var> and Ethernet >> - address <var>E</var>, a logical flow with match >> >> <code>ip4.dst == >> >> - <var>N</var>/<var>M</var></code>, whose priority is the >> >> number of >> >> - 1-bits in <var>M</var>, has the following actions: >> + <var>A</var> and Ethernet address <var>E</var>, a logical >> >> flow with >> >> + match <code>ip4.dst == <var>N</var>/<var>M</var></code>, >> >> whose >> >> + priority is 100 + the number of 1-bits in <var>M</var>, has >> >> the >> >> + following actions: >> </p> >> >> <pre> >> @@ -3382,6 +3410,13 @@ next; >> If the address <var>A</var> is in the link-local scope, the >> route will be limited to sending on the ingress port. >> </p> >> + >> + <p> >> + For routes with <code>route_table</code> value set >> + <code>reg7 == id</code> is prefixed in logical flow match >> >> portion. >> >> + Priority for routes with <code>route_table</code> value set >> >> is >> >> + the number of 1-bits in <var>M</var>. >> + </p> >> </li> >> >> <li> >> @@ -3408,7 +3443,7 @@ select(reg8[16..31], <var>MID1</var>, >> >> <var>MID2</var>, ...); >> >> </li> >> </ul> >> >> - <h3>Ingress Table 11: IP_ROUTING_ECMP</h3> >> + <h3>Ingress Table 12: IP_ROUTING_ECMP</h3> >> >> <p> >> This table implements the second part of IP routing for ECMP >> >> routes >> >> @@ -3460,7 +3495,7 @@ outport = <var>P</var>; >> </li> >> </ul> >> >> - <h3>Ingress Table 12: Router policies</h3> >> + <h3>Ingress Table 13: Router policies</h3> >> <p> >> This table adds flows for the logical router policies configured >> on the logical router. Please see the >> @@ -3532,7 +3567,7 @@ next; >> </li> >> </ul> >> >> - <h3>Ingress Table 13: ECMP handling for router policies</h3> >> + <h3>Ingress Table 14: ECMP handling for router policies</h3> >> <p> >> This table handles the ECMP for the router policies configured >> with multiple nexthops. >> @@ -3576,7 +3611,7 @@ outport = <var>P</var> >> </li> >> </ul> >> >> - <h3>Ingress Table 14: ARP/ND Resolution</h3> >> + <h3>Ingress Table 15: ARP/ND Resolution</h3> >> >> <p> >> Any packet that reaches this table is an IP packet whose next-hop >> @@ -3767,7 +3802,7 @@ outport = <var>P</var> >> >> </ul> >> >> - <h3>Ingress Table 15: Check packet length</h3> >> + <h3>Ingress Table 16: Check packet length</h3> >> >> <p> >> For distributed logical routers or gateway routers with gateway >> @@ -3797,7 +3832,7 @@ REGBIT_PKT_LARGER = >> >> check_pkt_larger(<var>L</var>); next; >> >> and advances to the next table. >> </p> >> >> - <h3>Ingress Table 16: Handle larger packets</h3> >> + <h3>Ingress Table 17: Handle larger packets</h3> >> >> <p> >> For distributed logical routers or gateway routers with gateway >> >> port >> >> @@ -3860,7 +3895,7 @@ icmp6 { >> and advances to the next table. >> </p> >> >> - <h3>Ingress Table 17: Gateway Redirect</h3> >> + <h3>Ingress Table 18: Gateway Redirect</h3> >> >> <p> >> For distributed logical routers where one or more of the logical >> >> router >> >> @@ -3907,7 +3942,7 @@ icmp6 { >> </li> >> </ul> >> >> - <h3>Ingress Table 18: ARP Request</h3> >> + <h3>Ingress Table 19: ARP Request</h3> >> >> <p> >> In the common case where the Ethernet destination has been >> >> resolved, this >> >> diff --git a/ovn-nb.ovsschema b/ovn-nb.ovsschema >> index 2ac8ef3ea..a0a171e19 100644 >> --- a/ovn-nb.ovsschema >> +++ b/ovn-nb.ovsschema >> @@ -1,7 +1,7 @@ >> { >> "name": "OVN_Northbound", >> - "version": "5.32.1", >> - "cksum": "2805328215 29734", >> + "version": "5.33.1", >> + "cksum": "3874993350 29785", >> "tables": { >> "NB_Global": { >> "columns": { >> @@ -387,6 +387,7 @@ >> "isRoot": false}, >> "Logical_Router_Static_Route": { >> "columns": { >> + "route_table": {"type": "string"}, >> "ip_prefix": {"type": "string"}, >> "policy": {"type": {"key": {"type": "string", >> "enum": ["set", ["src-ip", >> diff --git a/ovn-nb.xml b/ovn-nb.xml >> index d8266ed4d..b2917c363 100644 >> --- a/ovn-nb.xml >> +++ b/ovn-nb.xml >> @@ -2772,6 +2772,14 @@ >> prefix according to RFC3663 >> </p> >> </column> >> + >> + <column name="options" key="route_table"> >> + Designates lookup Logical_Router_Static_Routes with specified >> + <code>route_table</code> value. Routes to directly connected >> >> networks >> >> + from same Logical Router and routes without >> >> <code>route_table</code> >> >> + option set have higher priority than routes with >> + <code>route_table</code> option set. >> + </column> >> </group> >> >> <group title="Attachment"> >> @@ -2891,6 +2899,28 @@ >> </p> >> </column> >> >> + <column name="route_table"> >> + <p> >> + Any string to place route to separate routing table. If >> >> Logical Router >> >> + Port has configured value in <ref table="Logical_Router_Port" >> + column="options" key="route_table"/> other than empty string, >> >> OVN >> >> + performs route lookup for all packets entering Logical Router >> >> ingress >> >> + pipeline from this port in the following manner: >> + </p> >> + >> + <ul> >> + <li> >> + 1. First lookup among "global" routes: routes without >> + <code>route_table</code> value set and routes to directly >> >> connected >> >> + networks. >> + </li> >> + <li> >> + 2. Next lookup among routes with same >> >> <code>route_table</code> value >> >> + as specified in LRP's options:route_table field. >> + </li> >> + </ul> >> + </column> >> + >> <column name="external_ids" key="ic-learned-route"> >> <code>ovn-ic</code> populates this key if the route is learned >> >> from the >> >> global <ref db="OVN_IC_Southbound"/> database. In this case the >> >> value >> >> diff --git a/tests/ovn-ic.at b/tests/ovn-ic.at >> index 32f4e9d02..3aab54362 100644 >> --- a/tests/ovn-ic.at >> +++ b/tests/ovn-ic.at >> @@ -281,6 +281,7 @@ done >> >> AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr1], [0], [dnl >> IPv4 Routes >> +Route Table global: >> 10.11.1.0/24 169.254.0.1 dst-ip >> 10.11.2.0/24 169.254.100.2 dst-ip (learned) >> 10.22.1.0/24 169.254.0.2 src-ip >> @@ -299,6 +300,7 @@ ovn_as az1 ovn-nbctl set nb_global . >> >> options:ic-route-learn=false >> >> OVS_WAIT_WHILE([ovn_as az1 ovn-nbctl lr-route-list lr1 | grep learned]) >> AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr1], [0], [dnl >> IPv4 Routes >> +Route Table global: >> 10.11.1.0/24 169.254.0.1 dst-ip >> 10.22.1.0/24 169.254.0.2 src-ip >> ]) >> @@ -314,6 +316,7 @@ ovn_as az1 ovn-nbctl set nb_global . >> >> options:ic-route-adv=false >> >> OVS_WAIT_WHILE([ovn_as az2 ovn-nbctl lr-route-list lr2 | grep learned]) >> AT_CHECK([ovn_as az2 ovn-nbctl lr-route-list lr2], [0], [dnl >> IPv4 Routes >> +Route Table global: >> 10.11.2.0/24 169.254.0.1 dst-ip >> 10.22.2.0/24 169.254.0.2 src-ip >> ]) >> @@ -332,6 +335,7 @@ done >> # Default route should NOT get advertised or learned, by default. >> AT_CHECK([ovn_as az2 ovn-nbctl lr-route-list lr2], [0], [dnl >> IPv4 Routes >> +Route Table global: >> 10.11.1.0/24 169.254.100.1 dst-ip (learned) >> 10.11.2.0/24 169.254.0.1 dst-ip >> 10.22.2.0/24 169.254.0.2 src-ip >> diff --git a/tests/ovn-nbctl.at b/tests/ovn-nbctl.at >> index 9b80ae410..ddb5536ce 100644 >> --- a/tests/ovn-nbctl.at >> +++ b/tests/ovn-nbctl.at >> @@ -1520,6 +1520,7 @@ AT_CHECK([ovn-nbctl --ecmp --policy=src-ip >> >> lr-route-add lr0 20.0.0.0/24 11.0.0.1 >> >> >> AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >> IPv4 Routes >> +Route Table global: >> 10.0.0.0/24 11.0.0.1 dst-ip >> 10.0.1.0/24 11.0.1.1 dst-ip lp0 >> 10.0.10.0/24 dst-ip lp0 >> @@ -1534,6 +1535,7 @@ AT_CHECK([ovn-nbctl lrp-add lr0 lp1 >> >> f0:00:00:00:00:02 11.0.0.254/24]) >> >> AT_CHECK([ovn-nbctl --may-exist lr-route-add lr0 10.0.0.111/24 >> >> 11.0.0.1 lp1]) >> >> AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >> IPv4 Routes >> +Route Table global: >> 10.0.0.0/24 11.0.0.1 dst-ip lp1 >> 10.0.1.0/24 11.0.1.1 dst-ip lp0 >> 10.0.10.0/24 dst-ip lp0 >> @@ -1564,6 +1566,7 @@ AT_CHECK([ovn-nbctl --policy=src-ip lr-route-del >> >> lr0 9.16.1.0/24]) >> >> >> AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >> IPv4 Routes >> +Route Table global: >> 10.0.0.0/24 11.0.0.1 dst-ip lp1 >> 10.0.10.0/24 dst-ip lp0 >> 10.0.0.0/24 11.0.0.2 src-ip >> @@ -1575,6 +1578,7 @@ AT_CHECK([ovn-nbctl --policy=dst-ip lr-route-del >> >> lr0 10.0.0.0/24]) >> >> AT_CHECK([ovn-nbctl --policy=src-ip lr-route-del lr0 10.0.0.0/24]) >> AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >> IPv4 Routes >> +Route Table global: >> 10.0.10.0/24 dst-ip lp0 >> 0.0.0.0/0 192.168.0.1 dst-ip >> ]) >> @@ -1585,6 +1589,7 @@ AT_CHECK([ovn-nbctl --policy=src-ip lr-route-add >> >> lr0 10.0.0.0/24 11.0.0.2]) >> >> AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24]) >> AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >> IPv4 Routes >> +Route Table global: >> 10.0.10.0/24 dst-ip lp0 >> 0.0.0.0/0 192.168.0.1 dst-ip >> ]) >> @@ -1601,6 +1606,7 @@ AT_CHECK([ovn-nbctl --ecmp lr-route-add lr0 >> >> 10.0.0.0/24 11.0.0.3]) >> >> AT_CHECK([ovn-nbctl --ecmp lr-route-add lr0 10.0.0.0/24 11.0.0.4 lp0]) >> AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >> IPv4 Routes >> +Route Table global: >> 10.0.0.0/24 11.0.0.1 dst-ip ecmp >> 10.0.0.0/24 11.0.0.2 dst-ip ecmp >> 10.0.0.0/24 11.0.0.3 dst-ip ecmp >> @@ -1615,6 +1621,7 @@ dnl Delete ecmp routes >> AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 11.0.0.1]) >> AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >> IPv4 Routes >> +Route Table global: >> 10.0.0.0/24 11.0.0.2 dst-ip ecmp >> 10.0.0.0/24 11.0.0.3 dst-ip ecmp >> 10.0.0.0/24 11.0.0.4 dst-ip lp0 ecmp >> @@ -1622,12 +1629,14 @@ IPv4 Routes >> AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 11.0.0.2]) >> AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >> IPv4 Routes >> +Route Table global: >> 10.0.0.0/24 11.0.0.3 dst-ip ecmp >> 10.0.0.0/24 11.0.0.4 dst-ip lp0 ecmp >> ]) >> AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 11.0.0.4 lp0]) >> AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >> IPv4 Routes >> +Route Table global: >> 10.0.0.0/24 11.0.0.3 dst-ip >> ]) >> AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 11.0.0.3]) >> @@ -1641,6 +1650,7 @@ AT_CHECK([ovn-nbctl lr-route-add lr0 >> >> 2001:0db8:1::/64 2001:0db8:0:f103::1]) >> >> >> AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >> IPv6 Routes >> +Route Table global: >> 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 >> 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip >> ::/0 2001:db8:0:f101::1 dst-ip >> @@ -1650,6 +1660,7 @@ AT_CHECK([ovn-nbctl lr-route-del lr0 >> >> 2001:0db8:0::/64]) >> >> >> AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >> IPv6 Routes >> +Route Table global: >> 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip >> ::/0 2001:db8:0:f101::1 dst-ip >> ]) >> @@ -1677,11 +1688,13 @@ AT_CHECK([ovn-nbctl --may-exist >> >> --ecmp-symmetric-reply lr-route-add lr0 2003:0db >> >> >> AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >> IPv4 Routes >> +Route Table global: >> 10.0.0.0/24 11.0.0.1 dst-ip >> 10.0.1.0/24 11.0.1.1 dst-ip lp0 >> 0.0.0.0/0 192.168.0.1 dst-ip >> >> IPv6 Routes >> +Route Table global: >> 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 >> 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip ecmp >> 2001:db8:1::/64 2001:db8:0:f103::2 dst-ip ecmp >> @@ -1696,7 +1709,188 @@ AT_CHECK([ovn-nbctl lrp-add lr0 lr0-p0 >> >> 00:00:01:01:02:03 192.168.10.1/24]) >> >> bfd_uuid=$(ovn-nbctl create bfd logical_port=lr0-p0 dst_ip=100.0.0.50 >> >> status=down min_tx=250 min_rx=250 detect_mult=10) >> >> AT_CHECK([ovn-nbctl lr-route-add lr0 100.0.0.0/24 192.168.0.1]) >> route_uuid=$(fetch_column nb:logical_router_static_route _uuid >> >> ip_prefix="100.0.0.0/24") >> >> -AT_CHECK([ovn-nbctl set logical_router_static_route $route_uuid >> >> bfd=$bfd_uuid])]) >> >> +AT_CHECK([ovn-nbctl set logical_router_static_route $route_uuid >> >> bfd=$bfd_uuid]) >> >> + >> +check ovn-nbctl lr-route-del lr0 >> +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >> +]) >> + >> +dnl Check IPv4 routes in route table >> +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 0.0.0.0/0 >> >> 192.168.0.1 >> >> +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 10.0.1.1/24 >> >> 11.0.1.1 lp0 >> >> +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 10.0.0.1/24 >> >> 11.0.0.1 >> >> +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >> +IPv4 Routes >> +Route Table rtb-1: >> + 10.0.0.0/24 11.0.0.1 dst-ip >> + 10.0.1.0/24 11.0.1.1 dst-ip lp0 >> + 0.0.0.0/0 192.168.0.1 dst-ip >> +]) >> + >> +check ovn-nbctl lr-route-del lr0 >> +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >> +]) >> + >> +dnl Check IPv6 routes in route table >> +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 0:0:0:0:0:0:0:0/0 >> >> 2001:0db8:0:f101::1 >> >> +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 2001:0db8:0::/64 >> >> 2001:0db8:0:f102::1 lp0 >> >> +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 2001:0db8:1::/64 >> >> 2001:0db8:0:f103::1 >> >> + >> +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >> +IPv6 Routes >> +Route Table rtb-1: >> + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 >> + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip >> + ::/0 2001:db8:0:f101::1 dst-ip >> +]) >> + >> +dnl Check IPv4 and IPv6 routes in route table >> +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 0.0.0.0/0 >> >> 192.168.0.1 >> >> +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 10.0.1.1/24 >> >> 11.0.1.1 lp0 >> >> +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 10.0.0.1/24 >> >> 11.0.0.1 >> >> + >> +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >> +IPv4 Routes >> +Route Table rtb-1: >> + 10.0.0.0/24 11.0.0.1 dst-ip >> + 10.0.1.0/24 11.0.1.1 dst-ip lp0 >> + 0.0.0.0/0 192.168.0.1 dst-ip >> + >> +IPv6 Routes >> +Route Table rtb-1: >> + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 >> + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip >> + ::/0 2001:db8:0:f101::1 dst-ip >> +]) >> + >> +# Add routes in another route table >> +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 0.0.0.0/0 >> >> 192.168.0.1 >> >> +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 10.0.1.1/24 >> >> 11.0.1.1 lp0 >> >> +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 10.0.0.1/24 >> >> 11.0.0.1 >> >> +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 0:0:0:0:0:0:0:0/0 >> >> 2001:0db8:0:f101::1 >> >> +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 2001:0db8:0::/64 >> >> 2001:0db8:0:f102::1 lp0 >> >> +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 2001:0db8:1::/64 >> >> 2001:0db8:0:f103::1 >> >> + >> +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >> +IPv4 Routes >> +Route Table rtb-1: >> + 10.0.0.0/24 11.0.0.1 dst-ip >> + 10.0.1.0/24 11.0.1.1 dst-ip lp0 >> + 0.0.0.0/0 192.168.0.1 dst-ip >> + >> +Route Table rtb-2: >> + 10.0.0.0/24 11.0.0.1 dst-ip >> + 10.0.1.0/24 11.0.1.1 dst-ip lp0 >> + 0.0.0.0/0 192.168.0.1 dst-ip >> + >> +IPv6 Routes >> +Route Table rtb-1: >> + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 >> + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip >> + ::/0 2001:db8:0:f101::1 dst-ip >> + >> +Route Table rtb-2: >> + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 >> + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip >> + ::/0 2001:db8:0:f101::1 dst-ip >> +]) >> + >> +# Add routes to global route table >> +check ovn-nbctl lr-route-add lr0 0.0.0.0/0 192.168.0.1 >> +check ovn-nbctl lr-route-add lr0 10.0.1.1/24 11.0.1.1 lp0 >> +check ovn-nbctl lr-route-add lr0 10.0.0.1/24 11.0.0.1 >> +check ovn-nbctl lr-route-add lr0 0:0:0:0:0:0:0:0/0 2001:0db8:0:f101::1 >> +check ovn-nbctl lr-route-add lr0 2001:0db8:0::/64 2001:0db8:0:f102::1 >> >> lp0 >> >> +check check ovn-nbctl lr-route-add lr0 2001:0db8:1::/64 >> >> 2001:0db8:0:f103::1 >> >> + >> +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl >> +IPv4 Routes >> +Route Table global: >> + 10.0.0.0/24 11.0.0.1 dst-ip >> + 10.0.1.0/24 11.0.1.1 dst-ip lp0 >> + 0.0.0.0/0 192.168.0.1 dst-ip >> + >> +Route Table rtb-1: >> + 10.0.0.0/24 11.0.0.1 dst-ip >> + 10.0.1.0/24 11.0.1.1 dst-ip lp0 >> + 0.0.0.0/0 192.168.0.1 dst-ip >> + >> +Route Table rtb-2: >> + 10.0.0.0/24 11.0.0.1 dst-ip >> + 10.0.1.0/24 11.0.1.1 dst-ip lp0 >> + 0.0.0.0/0 192.168.0.1 dst-ip >> + >> +IPv6 Routes >> +Route Table global: >> + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 >> + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip >> + ::/0 2001:db8:0:f101::1 dst-ip >> + >> +Route Table rtb-1: >> + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 >> + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip >> + ::/0 2001:db8:0:f101::1 dst-ip >> + >> +Route Table rtb-2: >> + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 >> + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip >> + ::/0 2001:db8:0:f101::1 dst-ip >> +]) >> + >> +# delete IPv4 route from rtb-1 >> +check ovn-nbctl --route-table=rtb-1 lr-route-del lr0 10.0.0.0/24 >> +AT_CHECK([ovn-nbctl --route-table=rtb-1 lr-route-list lr0], [0], [dnl >> +IPv4 Routes >> +Route Table rtb-1: >> + 10.0.1.0/24 11.0.1.1 dst-ip lp0 >> + 0.0.0.0/0 192.168.0.1 dst-ip >> + >> +IPv6 Routes >> +Route Table rtb-1: >> + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 >> + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip >> + ::/0 2001:db8:0:f101::1 dst-ip >> +]) >> + >> +# delete IPv6 route from rtb-2 >> +check ovn-nbctl --route-table=rtb-2 lr-route-del lr0 2001:db8::/64 >> +AT_CHECK([ovn-nbctl --route-table=rtb-2 lr-route-list lr0], [0], [dnl >> +IPv4 Routes >> +Route Table rtb-2: >> + 10.0.0.0/24 11.0.0.1 dst-ip >> + 10.0.1.0/24 11.0.1.1 dst-ip lp0 >> + 0.0.0.0/0 192.168.0.1 dst-ip >> + >> +IPv6 Routes >> +Route Table rtb-2: >> + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip >> + ::/0 2001:db8:0:f101::1 dst-ip >> +]) >> + >> +check ovn-nbctl lr-route-del lr0 >> + >> +# ECMP route in route table >> +check ovn-nbctl --route-table=rtb1 lr-route-add lr0 0.0.0.0/0 >> >> 192.168.0.1 >> >> +check ovn-nbctl --ecmp --route-table=rtb1 lr-route-add lr0 0.0.0.0/0 >> >> 192.168.0.2 >> >> + >> +# Negative route table case: same prefix >> +AT_CHECK([ovn-nbctl --route-table=rtb1 lr-route-add lr0 0.0.0.0/0 >> >> 192.168.0.1], [1], [], [dnl >> >> +ovn-nbctl: duplicate prefix: 0.0.0.0/0 (policy: dst-ip). Use option >> >> --ecmp to allow this for ECMP routing. >> >> +]) >> + >> +# Negative route table case: same prefix & nexthop with ecmp >> +AT_CHECK([ovn-nbctl --ecmp --route-table=rtb1 lr-route-add lr0 >> >> 0.0.0.0/0 192.168.0.2], [1], [], [dnl >> >> +ovn-nbctl: duplicate nexthop for the same ECMP route >> +]) >> + >> +# Add routes to global route table >> +check ovn-nbctl lrp-add lr0 lrp0 00:00:00:00:00:01 1.1.1.1/24 >> +check ovn-nbctl lrp-set-options lrp0 route_table=rtb1 >> +AT_CHECK([ovn-nbctl get logical-router-port lrp0 options:route_table], >> >> [0], [dnl >> >> +rtb1 >> +]) >> +check `ovn-nbctl show lr0 | grep lrp0 -A3 | grep route_table=rtb1` >> +]) >> >> dnl >> >> --------------------------------------------------------------------- >> >> >> diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at >> index 3eebb55b6..e71e65bcc 100644 >> --- a/tests/ovn-northd.at >> +++ b/tests/ovn-northd.at >> @@ -5111,7 +5111,7 @@ check ovn-nbctl --wait=sb --ecmp-symmetric-reply >> >> lr-route-add lr0 1.0.0.1 192.16 >> >> >> ovn-sbctl dump-flows lr0 > lr0flows >> AT_CHECK([grep -e "lr_in_ip_routing.*select" lr0flows | sed >> >> 's/table=../table=??/' | sort], [0], [dnl >> >> - table=??(lr_in_ip_routing ), priority=65 , match=(ip4.dst == >> >> 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = 1; >> reg8[[16..31]] = select(1, 2);) >> >> + table=??(lr_in_ip_routing ), priority=165 , match=(ip4.dst == >> >> 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = 1; >> reg8[[16..31]] = select(1, 2);) >> >> ]) >> AT_CHECK([grep -e "lr_in_ip_routing_ecmp" lr0flows | sed >> >> 's/192\.168\.0\..0/192.168.0.??/' | sed 's/table=../table=??/' | sort], >> [0], [dnl >> >> table=??(lr_in_ip_routing_ecmp), priority=100 , >> >> match=(reg8[[0..15]] == 1 && reg8[[16..31]] == 1), action=(reg0 = >> 192.168.0.??; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = >> "lr0-public"; next;) >> >> @@ -5124,7 +5124,7 @@ check ovn-nbctl --wait=sb --ecmp-symmetric-reply >> >> lr-route-add lr0 1.0.0.1 192.16 >> >> >> ovn-sbctl dump-flows lr0 > lr0flows >> AT_CHECK([grep -e "lr_in_ip_routing.*select" lr0flows | sed >> >> 's/table=../table=??/' | sort], [0], [dnl >> >> - table=??(lr_in_ip_routing ), priority=65 , match=(ip4.dst == >> >> 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = 1; >> reg8[[16..31]] = select(1, 2);) >> >> + table=??(lr_in_ip_routing ), priority=165 , match=(ip4.dst == >> >> 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = 1; >> reg8[[16..31]] = select(1, 2);) >> >> ]) >> AT_CHECK([grep -e "lr_in_ip_routing_ecmp" lr0flows | sed >> >> 's/192\.168\.0\..0/192.168.0.??/' | sed 's/table=../table=??/' | sort], >> [0], [dnl >> >> table=??(lr_in_ip_routing_ecmp), priority=100 , >> >> match=(reg8[[0..15]] == 1 && reg8[[16..31]] == 1), action=(reg0 = >> 192.168.0.??; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = >> "lr0-public"; next;) >> >> @@ -5139,14 +5139,14 @@ check ovn-nbctl --wait=sb lr-route-add lr0 >> >> 1.0.0.0/24 192.168.0.10 >> >> ovn-sbctl dump-flows lr0 > lr0flows >> >> AT_CHECK([grep -e "lr_in_ip_routing.*192.168.0.10" lr0flows | sed >> >> 's/table=../table=??/' | sort], [0], [dnl >> >> - table=??(lr_in_ip_routing ), priority=49 , match=(ip4.dst == >> >> 1.0.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.10; >> reg1 >> = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; >> flags.loopback = 1; next;) >> >> + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == >> >> 1.0.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.10; >> reg1 >> = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; >> flags.loopback = 1; next;) >> >> ]) >> >> check ovn-nbctl --wait=sb lr-route-add lr0 2.0.0.0/24 lr0-public >> >> ovn-sbctl dump-flows lr0 > lr0flows >> AT_CHECK([grep -e "lr_in_ip_routing.*2.0.0.0" lr0flows | sed >> >> 's/table=../table=??/' | sort], [0], [dnl >> >> - table=??(lr_in_ip_routing ), priority=49 , match=(ip4.dst == >> >> 2.0.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 = >> 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; >> flags.loopback = 1; next;) >> >> + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == >> >> 2.0.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 = >> 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; >> flags.loopback = 1; next;) >> >> ]) >> >> AT_CLEANUP >> @@ -5232,3 +5232,71 @@ AT_CHECK([grep lr_in_gw_redirect lrflows | grep >> >> cr-DR | sed 's/table=../table=?? >> >> >> AT_CLEANUP >> ]) >> + >> + >> +OVN_FOR_EACH_NORTHD([ >> +AT_SETUP([route tables -- flows]) >> +AT_KEYWORDS([route-tables-flows]) >> +ovn_start >> + >> +check ovn-nbctl lr-add lr0 >> +check ovn-nbctl lrp-add lr0 lrp0 00:00:00:00:00:01 192.168.0.1/24 >> +check ovn-nbctl lrp-add lr0 lrp1 00:00:00:00:01:01 192.168.1.1/24 >> +check ovn-nbctl lrp-add lr0 lrp2 00:00:00:00:02:01 192.168.2.1/24 >> +check ovn-nbctl lrp-set-options lrp1 route_table=rtb-1 >> +check ovn-nbctl lrp-set-options lrp2 route_table=rtb-2 >> + >> +check ovn-nbctl lr-route-add lr0 0.0.0.0/0 192.168.0.10 >> +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 192.168.0.0/24 >> >> 192.168.1.10 >> >> +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 0.0.0.0/0 >> >> 192.168.0.10 >> >> +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 1.1.1.1/32 >> >> 192.168.0.20 >> >> +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 2.2.2.2/32 >> >> 192.168.0.30 >> >> +check ovn-nbctl --route-table=rtb-2 --ecmp lr-route-add lr0 2.2.2.2/32 >> >> 192.168.0.31 >> >> +check ovn-nbctl --wait=sb sync >> + >> +ovn-sbctl dump-flows lr0 > lr0flows >> +AT_CAPTURE_FILE([lr0flows]) >> + >> +AT_CHECK([grep -e "lr_in_ip_routing_pre.*match=(1)" lr0flows | sed >> >> 's/table=../table=??/'], [0], [dnl >> >> + table=??(lr_in_ip_routing_pre), priority=0 , match=(1), >> >> action=(next;) >> >> +]) >> + >> +p1_reg=$(grep -oP "lr_in_ip_routing_pre.*lrp1.*action=\(reg7 = \K." >> >> lr0flows) >> >> +p2_reg=$(grep -oP "lr_in_ip_routing_pre.*lrp2.*action=\(reg7 = \K." >> >> lr0flows) >> >> +echo $p1_reg >> +echo $p2_reg >> + >> +# exact register values are not predictable >> +if [[ $p1_reg -eq 2 ] && [ $p2_reg -eq 1 ]]; then >> + echo "swap reg values in dump" >> + sed -i -r s'/^(.*lrp2.*action=\(reg7 = )(1)(.*)/\12\3/g' lr0flows # >> >> "reg7 = 1" -> "reg7 = 2" >> >> + sed -i -r s'/^(.*lrp1.*action=\(reg7 = )(2)(.*)/\11\3/g' lr0flows # >> >> "reg7 = 2" -> "reg7 = 1" >> >> + sed -i -r s'/^(.*match=\(reg7 == )(2)( &&.*lrp1.*)/\11\3/g' lr0flows >> >> # "reg7 == 2" -> "reg7 == 1" >> >> + sed -i -r s'/^(.*match=\(reg7 == )(1)( &&.*lrp0.*)/\12\3/g' lr0flows >> >> # "reg7 == 1" -> "reg7 == 2" >> >> +fi >> + >> +check test "$p1_reg" != "$p2_reg" -a $((p1_reg * p2_reg)) -eq 2 >> + >> +AT_CHECK([grep "lr_in_ip_routing_pre" lr0flows | sed >> >> 's/table=../table=??/' | sort], [0], [dnl >> >> + table=??(lr_in_ip_routing_pre), priority=0 , match=(1), >> >> action=(next;) >> >> + table=??(lr_in_ip_routing_pre), priority=100 , match=(inport == >> >> "lrp1"), action=(reg7 = 1; next;) >> >> + table=??(lr_in_ip_routing_pre), priority=100 , match=(inport == >> >> "lrp2"), action=(reg7 = 2; next;) >> >> +]) >> + >> +grep -e "(lr_in_ip_routing ).*outport" lr0flows >> + >> +AT_CHECK([grep -e "(lr_in_ip_routing ).*outport" lr0flows | sed >> >> 's/table=../table=??/' | sort], [0], [dnl >> >> + table=??(lr_in_ip_routing ), priority=1 , match=(reg7 == 2 && >> >> ip4.dst == 0.0.0.0/0), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = >> 192.168.0.10; reg1 = 192.168.0.1; eth.src = 00:00:00:00:00:01; outport = >> "lrp0"; flags.loopback = 1; next;) >> >> + table=??(lr_in_ip_routing ), priority=101 , match=(ip4.dst == >> >> 0.0.0.0/0), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.10; reg1 >> = 192.168.0.1; eth.src = 00:00:00:00:00:01; outport = "lrp0"; >> flags.loopback = 1; next;) >> >> + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == >> >> 192.168.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 >> = 192.168.0.1; eth.src = 00:00:00:00:00:01; outport = "lrp0"; >> flags.loopback = 1; next;) >> >> + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == >> >> 192.168.1.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 >> = 192.168.1.1; eth.src = 00:00:00:00:01:01; outport = "lrp1"; >> flags.loopback = 1; next;) >> >> + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == >> >> 192.168.2.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 >> = 192.168.2.1; eth.src = 00:00:00:00:02:01; outport = "lrp2"; >> flags.loopback = 1; next;) >> >> + table=??(lr_in_ip_routing ), priority=229 , match=(inport == >> >> "lrp0" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; >> xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe00:1; eth.src = >> 00:00:00:00:00:01; outport = "lrp0"; flags.loopback = 1; next;) >> >> + table=??(lr_in_ip_routing ), priority=229 , match=(inport == >> >> "lrp1" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; >> xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe00:101; eth.src = >> 00:00:00:00:01:01; outport = "lrp1"; flags.loopback = 1; next;) >> >> + table=??(lr_in_ip_routing ), priority=229 , match=(inport == >> >> "lrp2" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; >> xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe00:201; eth.src = >> 00:00:00:00:02:01; outport = "lrp2"; flags.loopback = 1; next;) >> >> + table=??(lr_in_ip_routing ), priority=49 , match=(reg7 == 1 && >> >> ip4.dst == 192.168.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = >> 192.168.1.10; reg1 = 192.168.1.1; eth.src = 00:00:00:00:01:01; outport = >> "lrp1"; flags.loopback = 1; next;) >> >> + table=??(lr_in_ip_routing ), priority=65 , match=(reg7 == 2 && >> >> ip4.dst == 1.1.1.1/32), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = >> 192.168.0.20; reg1 = 192.168.0.1; eth.src = 00:00:00:00:00:01; outport = >> "lrp0"; flags.loopback = 1; next;) >> >> +]) >> + >> +AT_CLEANUP >> +]) >> diff --git a/tests/ovn.at b/tests/ovn.at >> index 49ece8735..60783a14b 100644 >> --- a/tests/ovn.at >> +++ b/tests/ovn.at >> @@ -18145,7 +18145,7 @@ eth_dst=00000000ff01 >> ip_src=$(ip_to_hex 10 0 0 10) >> ip_dst=$(ip_to_hex 172 168 0 101) >> send_icmp_packet 1 1 $eth_src $eth_dst $ip_src $ip_dst c4c9 >> >> 0000000000000000000000 >> >> -AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | awk '/table=25, >> >> n_packets=1, n_bytes=45/{print $7" "$8}'],[0],[dnl >> >> +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | awk '/table=26, >> >> n_packets=1, n_bytes=45/{print $7" "$8}'],[0],[dnl >> >> priority=80,ip,reg15=0x3,metadata=0x3,nw_src=10.0.0.10 actions=drop >> ]) >> >> @@ -22577,6 +22577,433 @@ OVN_CLEANUP([hv1]) >> AT_CLEANUP >> ]) >> >> + >> +OVN_FOR_EACH_NORTHD([ >> +AT_SETUP([route tables -- global routes]) >> +ovn_start >> + >> +# Logical network: >> +# ls1 (192.168.1.0/24) - lrp-lr1-ls1 - lr1 - lrp-lr1-ls2 - ls2 ( >> >> 192.168.2.0/24) >> >> +# >> +# ls1 has lsp11 (192.168.1.11) and ls2 has lsp21 (192.168.2.21) and >> >> lsp22 >> >> +# (192.168.2.22) >> +# >> +# lrp-lr1-ls1 set options:route_table=rtb-1 >> +# >> +# Static routes on lr1: >> +# 0.0.0.0/0 nexthop 192.168.2.21 >> +# 1.1.1.1/32 nexthop 192.168.2.22 route_table=rtb-1 >> +# >> +# Test 1: >> +# lsp11 send packet to 2.2.2.2 >> +# >> +# Expected result: >> +# lsp21 should receive traffic, lsp22 should not receive any traffic >> +# >> +# Test 2: >> +# lsp11 send packet to 1.1.1.1 >> +# >> +# Expected result: >> +# lsp21 should receive traffic, lsp22 should not receive any traffic >> + >> +ovn-nbctl lr-add lr1 >> + >> +for i in 1 2; do >> + ovn-nbctl ls-add ls${i} >> + ovn-nbctl lrp-add lr1 lrp-lr1-ls${i} 00:00:00:01:0${i}:01 >> >> 192.168.${i}.1/24 >> >> + ovn-nbctl lsp-add ls${i} lsp-ls${i}-lr1 -- lsp-set-type >> >> lsp-ls${i}-lr1 router \ >> >> + -- lsp-set-options lsp-ls${i}-lr1 router-port=lrp-lr1-ls${i} \ >> + -- lsp-set-addresses lsp-ls${i}-lr1 router >> +done >> + >> +# install static routes >> +ovn-nbctl lr-route-add lr1 0.0.0.0/0 192.168.2.21 >> +ovn-nbctl --route-table=rtb-1 lr-route-add lr1 1.1.1.1/32 192.168.2.22 >> + >> +# set lrp-lr1-ls1 route table >> +ovn-nbctl lrp-set-options lrp-lr1-ls1 route_table=rtb-1 >> + >> +# Create logical ports >> +ovn-nbctl lsp-add ls1 lsp11 -- \ >> + lsp-set-addresses lsp11 "f0:00:00:00:01:11 192.168.1.11" >> +ovn-nbctl lsp-add ls2 lsp21 -- \ >> + lsp-set-addresses lsp21 "f0:00:00:00:02:21 192.168.2.21" >> +ovn-nbctl lsp-add ls2 lsp22 -- \ >> + lsp-set-addresses lsp22 "f0:00:00:00:02:22 192.168.2.22" >> + >> +net_add n1 >> +sim_add hv1 >> +as hv1 >> +ovs-vsctl add-br br-phys >> +ovn_attach n1 br-phys 192.168.0.1 >> +ovs-vsctl -- add-port br-int hv1-vif1 -- \ >> + set interface hv1-vif1 external-ids:iface-id=lsp11 \ >> + options:tx_pcap=hv1/vif1-tx.pcap \ >> + options:rxq_pcap=hv1/vif1-rx.pcap \ >> + ofport-request=1 >> + >> +ovs-vsctl -- add-port br-int hv1-vif2 -- \ >> + set interface hv1-vif2 external-ids:iface-id=lsp21 \ >> + options:tx_pcap=hv1/vif2-tx.pcap \ >> + options:rxq_pcap=hv1/vif2-rx.pcap \ >> + ofport-request=2 >> + >> +ovs-vsctl -- add-port br-int hv1-vif3 -- \ >> + set interface hv1-vif3 external-ids:iface-id=lsp22 \ >> + options:tx_pcap=hv1/vif3-tx.pcap \ >> + options:rxq_pcap=hv1/vif3-rx.pcap \ >> + ofport-request=3 >> + >> +# wait for earlier changes to take effect >> +check ovn-nbctl --wait=hv sync >> +wait_for_ports_up >> + >> +for i in 1 2; do >> + packet="inport==\"lsp11\" && eth.src==f0:00:00:00:01:11 && >> >> eth.dst==00:00:00:01:01:01 && >> >> + ip4 && ip.ttl==64 && ip4.src==192.168.1.11 && >> >> ip4.dst==$i.$i.$i.$i && icmp" >> >> + AT_CHECK([as hv1 ovs-appctl -t ovn-controller inject-pkt >> >> "$packet"]) >> >> + >> + # Assume all packets go to lsp21. >> + exp_packet="eth.src==00:00:00:01:02:01 && >> >> eth.dst==f0:00:00:00:02:21 && >> >> + ip4 && ip.ttl==63 && ip4.src==192.168.1.11 && >> >> ip4.dst==$i.$i.$i.$i && icmp" >> >> + echo $exp_packet | ovstest test-ovn expr-to-packets >> >> >> expected_lsp21 >> >> +done >> +> expected_lsp22 >> + >> +# lsp21 should recieve 2 packets and lsp22 should recieve no packets >> +OVS_WAIT_UNTIL([ >> + rcv_n1=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" >> >> hv1/vif2-tx.pcap > lsp21.packets && cat lsp21.packets | wc -l` >> >> + rcv_n2=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" >> >> hv1/vif3-tx.pcap > lsp22.packets && cat lsp22.packets | wc -l` >> >> + echo $rcv_n1 $rcv_n2 >> + test $rcv_n1 -eq 2 -a $rcv_n2 -eq 0]) >> + >> +for i in 1 2; do >> + sort expected_lsp2$i > expout >> + AT_CHECK([cat lsp2${i}.packets | sort], [0], [expout]) >> +done >> + >> +OVN_CLEANUP([hv1]) >> +AT_CLEANUP >> +]) >> + >> + >> +OVN_FOR_EACH_NORTHD([ >> +AT_SETUP([route tables -- directly connected routes]) >> +ovn_start >> + >> +# Logical network: >> +# ls1 (192.168.1.0/24) - lrp-lr1-ls1 - lr1 - lrp-lr1-ls2 - ls2 ( >> >> 192.168.2.0/24) >> >> +# >> +# ls1 has lsp11 (192.168.1.11) and ls2 has lsp21 (192.168.2.21) >> +# >> +# lrp-lr1-ls1 set options:route_table=rtb-1 >> +# >> +# Static routes on lr1: >> +# 192.168.2.0/25 nexthop 192.168.1.11 route_table=rtb-1 >> +# >> +# Test 1: >> +# lsp11 send packet to 192.168.2.21 >> +# >> +# Expected result: >> +# lsp21 should receive traffic, lsp11 should not receive any traffic >> + >> +ovn-nbctl lr-add lr1 >> + >> +for i in 1 2; do >> + ovn-nbctl ls-add ls${i} >> + ovn-nbctl lrp-add lr1 lrp-lr1-ls${i} 00:00:00:01:0${i}:01 >> >> 192.168.${i}.1/24 >> >> + ovn-nbctl lsp-add ls${i} lsp-ls${i}-lr1 -- lsp-set-type >> >> lsp-ls${i}-lr1 router \ >> >> + -- lsp-set-options lsp-ls${i}-lr1 router-port=lrp-lr1-ls${i} \ >> + -- lsp-set-addresses lsp-ls${i}-lr1 router >> +done >> + >> +# install static route, which overrides directly-connected routes >> +ovn-nbctl --route-table=rtb-1 lr-route-add lr1 192.168.2.0/25 >> >> 192.168.1.11 >> >> + >> +# set lrp-lr1-ls1 route table >> +ovn-nbctl lrp-set-options lrp-lr1-ls1 route_table=rtb-1 >> + >> +# Create logical ports >> +ovn-nbctl lsp-add ls1 lsp11 -- \ >> + lsp-set-addresses lsp11 "f0:00:00:00:01:11 192.168.1.11" >> +ovn-nbctl lsp-add ls2 lsp21 -- \ >> + lsp-set-addresses lsp21 "f0:00:00:00:02:21 192.168.2.21" >> + >> +net_add n1 >> +sim_add hv1 >> +as hv1 >> +ovs-vsctl add-br br-phys >> +ovn_attach n1 br-phys 192.168.0.1 >> +ovs-vsctl -- add-port br-int hv1-vif1 -- \ >> + set interface hv1-vif1 external-ids:iface-id=lsp11 \ >> + options:tx_pcap=hv1/vif1-tx.pcap \ >> + options:rxq_pcap=hv1/vif1-rx.pcap \ >> + ofport-request=1 >> + >> +ovs-vsctl -- add-port br-int hv1-vif2 -- \ >> + set interface hv1-vif2 external-ids:iface-id=lsp21 \ >> + options:tx_pcap=hv1/vif2-tx.pcap \ >> + options:rxq_pcap=hv1/vif2-rx.pcap \ >> + ofport-request=2 >> + >> +# wait for earlier changes to take effect >> +check ovn-nbctl --wait=hv sync >> +wait_for_ports_up >> + >> +packet="inport==\"lsp11\" && eth.src==f0:00:00:00:01:11 && >> >> eth.dst==00:00:00:01:01:01 && >> >> + ip4 && ip.ttl==64 && ip4.src==192.168.1.11 && >> >> ip4.dst==192.168.2.21 && icmp" >> >> +AT_CHECK([as hv1 ovs-appctl -t ovn-controller inject-pkt "$packet"]) >> + >> +# Assume all packets go to lsp21. >> +exp_packet="eth.src==00:00:00:01:02:01 && eth.dst==f0:00:00:00:02:21 && >> + ip4 && ip.ttl==63 && ip4.src==192.168.1.11 && >> >> ip4.dst==192.168.2.21 && icmp" >> >> +echo $exp_packet | ovstest test-ovn expr-to-packets >> expected_lsp21 >> +> expected_lsp11 >> + >> +# lsp21 should recieve 1 icmp packet and lsp11 should recieve no >> >> packets >> >> +OVS_WAIT_UNTIL([ >> + rcv_n11=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" >> >> hv1/vif1-tx.pcap > lsp11.packets && cat lsp11.packets | wc -l` >> >> + rcv_n21=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" >> >> hv1/vif2-tx.pcap > lsp21.packets && cat lsp21.packets | wc -l` >> >> + echo $rcv_n11 $rcv_n21 >> + test $rcv_n11 -eq 0 -a $rcv_n21 -eq 1]) >> + >> +for i in 11 21; do >> + sort expected_lsp$i > expout >> + AT_CHECK([cat lsp${i}.packets | sort], [0], [expout]) >> +done >> + >> +OVN_CLEANUP([hv1]) >> +AT_CLEANUP >> +]) >> + >> + >> +OVN_FOR_EACH_NORTHD([ >> +AT_SETUP([route tables -- overlapping subnets]) >> +ovn_start >> + >> +# Logical network: >> +# >> +# ls1 (192.168.1.0/24) - lrp-lr1-ls1 -\ /- lrp-lr1-ls2 - ls2 ( >> >> 192.168.2.0/24) >> >> +# lr1 >> +# ls3 (192.168.3.0/24) - lrp-lr1-ls3 -/ \- lrp-lr1-ls4 - ls4 ( >> >> 192.168.4.0/24) >> >> +# >> +# ls1 has lsp11 (192.168.1.11) >> +# ls2 has lsp21 (192.168.2.21) >> +# ls3 has lsp31 (192.168.3.31) >> +# ls4 has lsp41 (192.168.4.41) >> +# >> +# lrp-lr1-ls1 set options:route_table=rtb-1 >> +# lrp-lr1-ls2 set options:route_table=rtb-2 >> +# >> +# Static routes on lr1: >> +# 10.0.0.0/24 nexthop 192.168.3.31 route_table=rtb-1 >> +# 10.0.0.0/24 nexthop 192.168.4.41 route_table=rtb-2 >> +# >> +# Test 1: >> +# lsp11 send packet to 10.0.0.1 >> +# >> +# Expected result: >> +# lsp31 should receive traffic, lsp41 should not receive any traffic >> +# >> +# Test 2: >> +# lsp21 send packet to 10.0.0.1 >> +# >> +# Expected result: >> +# lsp41 should receive traffic, lsp31 should not receive any traffic >> + >> +ovn-nbctl lr-add lr1 >> + >> +# Create logical topology >> +for i in $(seq 1 4); do >> + ovn-nbctl ls-add ls${i} >> + ovn-nbctl lrp-add lr1 lrp-lr1-ls${i} 00:00:00:01:0${i}:01 >> >> 192.168.${i}.1/24 >> >> + ovn-nbctl lsp-add ls${i} lsp-ls${i}-lr1 -- lsp-set-type >> >> lsp-ls${i}-lr1 router \ >> >> + -- lsp-set-options lsp-ls${i}-lr1 router-port=lrp-lr1-ls${i} \ >> + -- lsp-set-addresses lsp-ls${i}-lr1 router >> + ovn-nbctl lsp-add ls$i lsp${i}1 -- \ >> + lsp-set-addresses lsp${i}1 "f0:00:00:00:0${i}:1${i} >> >> 192.168.${i}.${i}1" >> >> +done >> + >> +# install static routes >> +ovn-nbctl --route-table=rtb-1 lr-route-add lr1 10.0.0.0/24 192.168.3.31 >> +ovn-nbctl --route-table=rtb-2 lr-route-add lr1 10.0.0.0/24 192.168.4.41 >> + >> +# set lrp-lr1-ls{1,2} route tables >> +ovn-nbctl lrp-set-options lrp-lr1-ls1 route_table=rtb-1 >> +ovn-nbctl lrp-set-options lrp-lr1-ls2 route_table=rtb-2 >> + >> +net_add n1 >> +sim_add hv1 >> +as hv1 >> +ovs-vsctl add-br br-phys >> +ovn_attach n1 br-phys 192.168.0.1 >> + >> +for i in $(seq 1 4); do >> + ovs-vsctl -- add-port br-int hv1-vif${i} -- \ >> + set interface hv1-vif${i} external-ids:iface-id=lsp${i}1 \ >> + options:tx_pcap=hv1/vif${i}-tx.pcap \ >> + options:rxq_pcap=hv1/vif${i}-rx.pcap \ >> + ofport-request=${i} >> +done >> + >> +# wait for earlier changes to take effect >> +check ovn-nbctl --wait=hv sync >> +wait_for_ports_up >> + >> +# lsp31 should recieve packet coming from lsp11 >> +# lsp41 should recieve packet coming from lsp21 >> +for i in $(seq 1 2); do >> + di=$(( i + 2)) # dst index >> + ri=$(( 5 - i)) # reverse index >> + packet="inport==\"lsp${i}1\" && eth.src==f0:00:00:00:0${i}:1${i} && >> + eth.dst==00:00:00:01:0${i}:01 && ip4 && ip.ttl==64 && >> + ip4.src==192.168.${i}.${i}1 && ip4.dst==10.0.0.1 && icmp" >> + AT_CHECK([as hv1 ovs-appctl -t ovn-controller inject-pkt >> >> "$packet"]) >> >> + >> + # Assume all packets go to lsp${di}1. >> + exp_packet="eth.src==00:00:00:01:0${di}:01 && >> >> eth.dst==f0:00:00:00:0${di}:1${di} && >> >> + ip4 && ip.ttl==63 && ip4.src==192.168.${i}.${i}1 && >> >> ip4.dst==10.0.0.1 && icmp" >> >> + echo $exp_packet | ovstest test-ovn expr-to-packets >> >> >> expected_lsp${di}1 >> >> + > expected_lsp${ri}1 >> + >> + OVS_WAIT_UNTIL([ >> + rcv_n1=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" >> >> hv1/vif${di}-tx.pcap > lsp${di}1.packets && cat lsp${di}1.packets | wc -l` >> >> + rcv_n2=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" >> >> hv1/vif${ri}-tx.pcap > lsp${ri}1.packets && cat lsp${ri}1.packets | wc -l` >> >> + echo $rcv_n1 $rcv_n2 >> + test $rcv_n1 -eq 1 -a $rcv_n2 -eq 0]) >> + >> + for j in "${di}1" "${ri}1"; do >> + sort expected_lsp${j} > expout >> + AT_CHECK([cat lsp${j}.packets | sort], [0], [expout]) >> + done >> + >> + # cleanup tx pcap files >> + for j in "${di}1" "${ri}1"; do >> + ovs-vsctl -- remove interface hv1-vif${di} options tx_pcap >> + > hv1/vif${di}-tx.pcap >> + ovs-vsctl -- set interface hv1-vif${di} >> >> external-ids:iface-id=lsp${di}1 \ >> >> + options:tx_pcap=hv1/vif${di}-tx.pcap >> + done >> +done >> + >> +OVN_CLEANUP([hv1]) >> +AT_CLEANUP >> +]) >> + >> + >> +OVN_FOR_EACH_NORTHD([ >> +AT_SETUP([route tables IPv6 -- overlapping subnets]) >> +ovn_start >> + >> +# Logical network: >> +# >> +# ls1 (2001:db8:1::/64) - lrp-lr1-ls1 -\ /- lrp-lr1-ls2 - ls2 >> >> (2001:db8:2::/64) >> >> +# lr1 >> +# ls3 (2001:db8:3::/64) - lrp-lr1-ls3 -/ \- lrp-lr1-ls4 - ls4 >> >> (2001:db8:4::/64) >> >> +# >> +# ls1 has lsp11 (2001:db8:1::11) >> +# ls2 has lsp21 (2001:db8:2::21) >> +# ls3 has lsp31 (2001:db8:3::31) >> +# ls4 has lsp41 (2001:db8:4::41) >> +# >> +# lrp-lr1-ls1 set options:route_table=rtb-1 >> +# lrp-lr1-ls2 set options:route_table=rtb-2 >> +# >> +# Static routes on lr1: >> +# 2001:db8:2000::/64 nexthop 2001:db8:3::31 route_table=rtb-1 >> +# 2001:db8:2000::/64 nexthop 2001:db8:3::41 route_table=rtb-2 >> +# >> +# Test 1: >> +# lsp11 send packet to 2001:db8:2000::1 >> +# >> +# Expected result: >> +# lsp31 should receive traffic, lsp41 should not receive any traffic >> +# >> +# Test 2: >> +# lsp21 send packet to 2001:db8:2000::1 >> +# >> +# Expected result: >> +# lsp41 should receive traffic, lsp31 should not receive any traffic >> + >> +ovn-nbctl lr-add lr1 >> + >> +# Create logical topology >> +for i in $(seq 1 4); do >> + ovn-nbctl ls-add ls${i} >> + ovn-nbctl lrp-add lr1 lrp-lr1-ls${i} 00:00:00:01:0${i}:01 >> >> 2001:db8:${i}::1/64 >> >> + ovn-nbctl lsp-add ls${i} lsp-ls${i}-lr1 -- lsp-set-type >> >> lsp-ls${i}-lr1 router \ >> >> + -- lsp-set-options lsp-ls${i}-lr1 router-port=lrp-lr1-ls${i} \ >> + -- lsp-set-addresses lsp-ls${i}-lr1 router >> + ovn-nbctl lsp-add ls$i lsp${i}1 -- \ >> + lsp-set-addresses lsp${i}1 "f0:00:00:00:0${i}:1${i} >> >> 2001:db8:${i}::${i}1" >> >> +done >> + >> +# install static routes >> +ovn-nbctl --route-table=rtb-1 lr-route-add lr1 2001:db8:2000::/64 >> >> 2001:db8:3::31 >> >> +ovn-nbctl --route-table=rtb-2 lr-route-add lr1 2001:db8:2000::/64 >> >> 2001:db8:4::41 >> >> + >> +# set lrp-lr1-ls{1,2} route tables >> +ovn-nbctl lrp-set-options lrp-lr1-ls1 route_table=rtb-1 >> +ovn-nbctl lrp-set-options lrp-lr1-ls2 route_table=rtb-2 >> + >> +net_add n1 >> +sim_add hv1 >> +as hv1 >> +ovs-vsctl add-br br-phys >> +ovn_attach n1 br-phys 192.168.0.1 >> + >> +for i in $(seq 1 4); do >> + ovs-vsctl -- add-port br-int hv1-vif${i} -- \ >> + set interface hv1-vif${i} external-ids:iface-id=lsp${i}1 \ >> + options:tx_pcap=hv1/vif${i}-tx.pcap \ >> + options:rxq_pcap=hv1/vif${i}-rx.pcap \ >> + ofport-request=${i} >> +done >> + >> +# wait for earlier changes to take effect >> +AT_CHECK([ovn-nbctl --timeout=3 --wait=hv sync], [0], [ignore]) >> + >> +# lsp31 should recieve packet coming from lsp11 >> +# lsp41 should recieve packet coming from lsp21 >> +for i in $(seq 1 2); do >> + di=$(( i + 2)) # dst index >> + ri=$(( 5 - i)) # reverse index >> + packet="inport==\"lsp${i}1\" && eth.src==f0:00:00:00:0${i}:1${i} && >> + eth.dst==00:00:00:01:0${i}:01 && ip6 && ip.ttl==64 && >> + ip6.src==2001:db8:${i}::${i}1 && ip6.dst==2001:db8:2000::1 >> >> && icmp6" >> >> + AT_CHECK([as hv1 ovs-appctl -t ovn-controller inject-pkt >> >> "$packet"]) >> >> + >> + # Assume all packets go to lsp${di}1. >> + exp_packet="eth.src==00:00:00:01:0${di}:01 && >> >> eth.dst==f0:00:00:00:0${di}:1${di} && ip6 && >> >> + ip.ttl==63 && ip6.src==2001:db8:${i}::${i}1 && >> >> ip6.dst==2001:db8:2000::1 && icmp6" >> >> + echo $exp_packet | ovstest test-ovn expr-to-packets >> >> >> expected_lsp${di}1 >> >> + > expected_lsp${ri}1 >> + >> + OVS_WAIT_UNTIL([ >> + rcv_n1=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" >> >> hv1/vif${di}-tx.pcap > lsp${di}1.packets && cat lsp${di}1.packets | wc -l` >> >> + rcv_n2=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" >> >> hv1/vif${ri}-tx.pcap > lsp${ri}1.packets && cat lsp${ri}1.packets | wc -l` >> >> + echo $rcv_n1 $rcv_n2 >> + test $rcv_n1 -eq 1 -a $rcv_n2 -eq 0]) >> + >> + for j in "${di}1" "${ri}1"; do >> + sort expected_lsp${j} > expout >> + AT_CHECK([cat lsp${j}.packets | sort], [0], [expout]) >> + done >> + >> + # cleanup tx pcap files >> + for j in "${di}1" "${ri}1"; do >> + ovs-vsctl -- remove interface hv1-vif${di} options tx_pcap >> + > hv1/vif${di}-tx.pcap >> + ovs-vsctl -- set interface hv1-vif${di} >> >> external-ids:iface-id=lsp${di}1 \ >> >> + options:tx_pcap=hv1/vif${di}-tx.pcap >> + done >> +done >> + >> +OVN_CLEANUP([hv1]) >> +AT_CLEANUP >> +]) >> + >> + >> OVN_FOR_EACH_NORTHD([ >> AT_SETUP([forwarding group: 3 HVs, 1 LR, 2 LS]) >> AT_KEYWORDS([forwarding-group]) >> @@ -23332,7 +23759,7 @@ ovn-sbctl dump-flows > sbflows >> AT_CAPTURE_FILE([sbflows]) >> AT_CAPTURE_FILE([offlows]) >> OVS_WAIT_UNTIL([ >> - as hv1 ovs-ofctl dump-flows br-int table=20 > offlows >> + as hv1 ovs-ofctl dump-flows br-int table=21 > offlows >> test $(grep -c "load:0x64->NXM_NX_PKT_MARK" offlows) = 1 && \ >> test $(grep -c "load:0x3->NXM_NX_PKT_MARK" offlows) = 1 && \ >> test $(grep -c "load:0x4->NXM_NX_PKT_MARK" offlows) = 1 && \ >> @@ -23425,12 +23852,12 @@ send_ipv4_pkt hv1 hv1-vif1 505400000003 >> >> 00000000ff01 \ >> >> $(ip_to_hex 10 0 0 3) $(ip_to_hex 172 168 0 120) >> >> OVS_WAIT_UNTIL([ >> - test 1 -eq $(as hv1 ovs-ofctl dump-flows br-int table=20 | \ >> + test 1 -eq $(as hv1 ovs-ofctl dump-flows br-int table=21 | \ >> grep "load:0x2->NXM_NX_PKT_MARK" -c) >> ]) >> >> AT_CHECK([ >> - test 0 -eq $(as hv1 ovs-ofctl dump-flows br-int table=20 | \ >> + test 0 -eq $(as hv1 ovs-ofctl dump-flows br-int table=21 | \ >> grep "load:0x64->NXM_NX_PKT_MARK" -c) >> ]) >> >> @@ -24133,7 +24560,7 @@ AT_CHECK([ >> grep "priority=100" | \ >> grep -c >> >> >> "ct(commit,zone=NXM_NX_REG11\\[[0..15\\]],.*exec(move:NXM_OF_ETH_SRC\\[[\\]]->NXM_NX_CT_LABEL\\[[32..79\\]],load:0x[[0-9]]->NXM_NX_CT_LABEL\\[[80..95\\]]))" >> >> >> - grep table=22 hv${hv}flows | \ >> + grep table=23 hv${hv}flows | \ >> grep "priority=200" | \ >> grep -c >> >> "actions=move:NXM_NX_CT_LABEL\\[[32..79\\]]->NXM_OF_ETH_DST\\[[\\]]" >> >> done; :], [0], [dnl >> @@ -24258,7 +24685,7 @@ AT_CHECK([ >> grep "priority=100" | \ >> grep -c >> >> >> "ct(commit,zone=NXM_NX_REG11\\[[0..15\\]],.*exec(move:NXM_OF_ETH_SRC\\[[\\]]->NXM_NX_CT_LABEL\\[[32..79\\]],load:0x[[0-9]]->NXM_NX_CT_LABEL\\[[80..95\\]]))" >> >> >> - grep table=22 hv${hv}flows | \ >> + grep table=23 hv${hv}flows | \ >> grep "priority=200" | \ >> grep -c >> >> "actions=move:NXM_NX_CT_LABEL\\[[32..79\\]]->NXM_OF_ETH_DST\\[[\\]]" >> >> done; :], [0], [dnl >> @@ -24880,7 +25307,7 @@ AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | >> >> grep "actions=controller" | grep >> >> ]) >> >> # The packet should've been dropped in the lr_in_arp_resolve stage. >> -AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | grep -E "table=22, >> >> n_packets=1,.* priority=1,ip,metadata=0x${sw_key},nw_dst=10.0.1.1 >> actions=drop" -c], [0], [dnl >> >> +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | grep -E "table=23, >> >> n_packets=1,.* priority=1,ip,metadata=0x${sw_key},nw_dst=10.0.1.1 >> actions=drop" -c], [0], [dnl >> >> 1 >> ]) >> >> diff --git a/utilities/ovn-nbctl.c b/utilities/ovn-nbctl.c >> index e34bb65f7..0ff10618b 100644 >> --- a/utilities/ovn-nbctl.c >> +++ b/utilities/ovn-nbctl.c >> @@ -329,6 +329,8 @@ Logical router port commands:\n\ >> add logical port PORT on ROUTER\n\ >> lrp-set-gateway-chassis PORT CHASSIS [PRIORITY]\n\ >> set gateway chassis for port PORT\n\ >> + lrp-set-options PORT KEY=VALUE [KEY=VALUE]...\n\ >> + set router port options\n\ >> lrp-del-gateway-chassis PORT CHASSIS\n\ >> delete gateway chassis from port PORT\n\ >> lrp-get-gateway-chassis PORT\n\ >> @@ -351,11 +353,17 @@ Logical router port commands:\n\ >> ('overlay' or 'bridged')\n\ >> \n\ >> Route commands:\n\ >> - [--policy=POLICY] [--ecmp] [--ecmp-symmetric-reply] lr-route-add >> >> ROUTER \n\ >> >> - PREFIX NEXTHOP [PORT]\n\ >> + [--policy=POLICY]\n\ >> + [--ecmp]\n\ >> + [--ecmp-symmetric-reply]\n\ >> + [--route-table=ROUTE_TABLE]\n\ >> + lr-route-add ROUTER PREFIX NEXTHOP [PORT]\n\ >> add a route to ROUTER\n\ >> - [--policy=POLICY] lr-route-del ROUTER [PREFIX [NEXTHOP [PORT]]]\n\ >> + [--policy=POLICY]\n\ >> + [--route-table=ROUTE_TABLE]\n\ >> + lr-route-del ROUTER [PREFIX [NEXTHOP [PORT]]]\n\ >> remove routes from ROUTER\n\ >> + [--route-table=ROUTE_TABLE]\n\ >> lr-route-list ROUTER print routes for ROUTER\n\ >> \n\ >> Policy commands:\n\ >> @@ -743,6 +751,11 @@ print_lr(const struct nbrec_logical_router *lr, >> >> struct ds *s) >> >> ds_put_cstr(s, "]\n"); >> } >> >> + const char *route_table = smap_get(&lrp->options, >> >> "route_table"); >> >> + if (route_table) { >> + ds_put_format(s, " route-table: %s\n", route_table); >> + } >> + >> if (lrp->n_gateway_chassis) { >> const struct nbrec_gateway_chassis **gcs; >> >> @@ -862,6 +875,7 @@ nbctl_pre_show(struct ctl_context *ctx) >> ovsdb_idl_add_column(ctx->idl, >> >> &nbrec_logical_router_port_col_name); >> >> ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_mac); >> ovsdb_idl_add_column(ctx->idl, >> >> &nbrec_logical_router_port_col_networks); >> >> + ovsdb_idl_add_column(ctx->idl, >> >> &nbrec_logical_router_port_col_options); >> >> ovsdb_idl_add_column(ctx->idl, >> >> &nbrec_logical_router_port_col_gateway_chassis); >> >> >> ovsdb_idl_add_column(ctx->idl, >> >> &nbrec_gateway_chassis_col_chassis_name); >> >> @@ -4000,11 +4014,19 @@ nbctl_lr_policy_list(struct ctl_context *ctx) >> >> static struct nbrec_logical_router_static_route * >> nbctl_lr_get_route(const struct nbrec_logical_router *lr, char *prefix, >> - char *next_hop, bool is_src_route, bool ecmp) >> + char *next_hop, bool is_src_route, bool ecmp, >> + char *route_table) >> { >> for (int i = 0; i < lr->n_static_routes; i++) { >> struct nbrec_logical_router_static_route *route = >> >> lr->static_routes[i]; >> >> >> + /* Strict compare for route_table. >> + * If route_table was not specified, >> + * lookup for routes with empty route_table value. */ >> + if (strcmp(route->route_table, route_table ? route_table : >> >> "")) { >> >> + continue; >> + } >> + >> /* Compare route policy. */ >> char *nb_policy = route->policy; >> bool nb_is_src_route = false; >> @@ -4060,6 +4082,8 @@ nbctl_pre_lr_route_add(struct ctl_context *ctx) >> &nbrec_logical_router_static_route_col_bfd); >> ovsdb_idl_add_column(ctx->idl, >> >> &nbrec_logical_router_static_route_col_options); >> >> + ovsdb_idl_add_column(ctx->idl, >> + >> >> &nbrec_logical_router_static_route_col_route_table); >> >> } >> >> static char * OVS_WARN_UNUSED_RESULT >> @@ -4090,6 +4114,7 @@ nbctl_lr_route_add(struct ctl_context *ctx) >> } >> } >> >> + char *route_table = shash_find_data(&ctx->options, >> >> "--route-table"); >> >> bool v6_prefix = false; >> prefix = normalize_ipv4_prefix_str(ctx->argv[2]); >> if (!prefix) { >> @@ -4166,7 +4191,8 @@ nbctl_lr_route_add(struct ctl_context *ctx) >> bool ecmp = shash_find(&ctx->options, "--ecmp") != NULL || >> ecmp_symmetric_reply; >> struct nbrec_logical_router_static_route *route = >> - nbctl_lr_get_route(lr, prefix, next_hop, is_src_route, ecmp); >> + nbctl_lr_get_route(lr, prefix, next_hop, is_src_route, ecmp, >> + route_table); >> >> /* Validations for nexthop = "discard" */ >> if (is_discard_route) { >> @@ -4230,7 +4256,8 @@ nbctl_lr_route_add(struct ctl_context *ctx) >> } >> >> struct nbrec_logical_router_static_route *discard_route = >> - nbctl_lr_get_route(lr, prefix, "discard", is_src_route, true); >> + nbctl_lr_get_route(lr, prefix, "discard", is_src_route, true, >> + route_table); >> if (discard_route) { >> ctl_error(ctx, "discard nexthop for the same ECMP route >> >> exists."); >> >> goto cleanup; >> @@ -4246,6 +4273,9 @@ nbctl_lr_route_add(struct ctl_context *ctx) >> if (policy) { >> nbrec_logical_router_static_route_set_policy(route, policy); >> } >> + if (route_table) { >> + nbrec_logical_router_static_route_set_route_table(route, >> >> route_table); >> >> + } >> >> if (ecmp_symmetric_reply) { >> const struct smap options = SMAP_CONST1(&options, >> @@ -4289,6 +4319,8 @@ nbctl_pre_lr_route_del(struct ctl_context *ctx) >> >> &nbrec_logical_router_static_route_col_nexthop); >> >> ovsdb_idl_add_column(ctx->idl, >> >> &nbrec_logical_router_static_route_col_output_port); >> >> + ovsdb_idl_add_column(ctx->idl, >> + >> >> &nbrec_logical_router_static_route_col_route_table); >> >> >> } >> >> @@ -4302,6 +4334,7 @@ nbctl_lr_route_del(struct ctl_context *ctx) >> return; >> } >> >> + const char *route_table = shash_find_data(&ctx->options, >> >> "--route-table"); >> >> const char *policy = shash_find_data(&ctx->options, "--policy"); >> bool is_src_route = false; >> if (policy) { >> @@ -4392,6 +4425,14 @@ nbctl_lr_route_del(struct ctl_context *ctx) >> } >> } >> >> + /* Strict compare for route_table. >> + * If route_table was not specified, >> + * lookup for routes with empty route_table value. */ >> + if (strcmp(lr->static_routes[i]->route_table, >> + route_table ? route_table : "")) { >> + continue; >> + } >> + >> /* Compare output_port, if specified. */ >> if (output_port) { >> char *rt_output_port = lr->static_routes[i]->output_port; >> @@ -5115,6 +5156,41 @@ nbctl_pre_lrp_del_gateway_chassis(struct >> >> ctl_context *ctx) >> >> ovsdb_idl_add_column(ctx->idl, >> >> &nbrec_gateway_chassis_col_chassis_name); >> >> } >> >> +static void >> +nbctl_pre_lrp_options(struct ctl_context *ctx) >> +{ >> + ovsdb_idl_add_column(ctx->idl, >> >> &nbrec_logical_router_port_col_name); >> >> + ovsdb_idl_add_column(ctx->idl, >> >> &nbrec_logical_router_port_col_options); >> >> +} >> + >> +static void >> +nbctl_lrp_set_options(struct ctl_context *ctx) >> +{ >> + const char *id = ctx->argv[1]; >> + const struct nbrec_logical_router_port *lrp = NULL; >> + size_t i; >> + struct smap options = SMAP_INITIALIZER(&options); >> + >> + char *error = lrp_by_name_or_uuid(ctx, id, true, &lrp); >> + if (error) { >> + ctx->error = error; >> + return; >> + } >> + for (i = 2; i < ctx->argc; i++) { >> + char *key, *value; >> + value = xstrdup(ctx->argv[i]); >> + key = strsep(&value, "="); >> + if (value) { >> + smap_add(&options, key, value); >> + } >> + free(key); >> + } >> + >> + nbrec_logical_router_port_set_options(lrp, &options); >> + >> + smap_destroy(&options); >> +} >> + >> /* Removes logical router port 'lrp->gateway_chassis[idx]'. */ >> static void >> remove_gc(const struct nbrec_logical_router_port *lrp, size_t idx) >> @@ -5891,6 +5967,7 @@ route_cmp_details(const struct >> >> nbrec_logical_router_static_route *r1, >> >> } >> return r1->output_port ? 1 : -1; >> } >> + >> struct ipv4_route { >> int priority; >> ovs_be32 addr; >> @@ -5900,6 +5977,11 @@ struct ipv4_route { >> static int >> __ipv4_route_cmp(const struct ipv4_route *r1, const struct ipv4_route >> >> *r2) >> >> { >> + int rtb_cmp = strcmp(r1->route->route_table, >> + r2->route->route_table); >> + if (rtb_cmp) { >> + return rtb_cmp; >> + } >> if (r1->priority != r2->priority) { >> return r1->priority > r2->priority ? -1 : 1; >> } >> @@ -5931,6 +6013,11 @@ struct ipv6_route { >> static int >> __ipv6_route_cmp(const struct ipv6_route *r1, const struct ipv6_route >> >> *r2) >> >> { >> + int rtb_cmp = strcmp(r1->route->route_table, >> + r2->route->route_table); >> + if (rtb_cmp) { >> + return rtb_cmp; >> + } >> if (r1->priority != r2->priority) { >> return r1->priority > r2->priority ? -1 : 1; >> } >> @@ -6018,6 +6105,8 @@ nbctl_pre_lr_route_list(struct ctl_context *ctx) >> >> &nbrec_logical_router_static_route_col_options); >> >> ovsdb_idl_add_column(ctx->idl, >> &nbrec_logical_router_static_route_col_bfd); >> + ovsdb_idl_add_column(ctx->idl, >> + >> >> &nbrec_logical_router_static_route_col_route_table); >> >> } >> >> static void >> @@ -6035,12 +6124,17 @@ nbctl_lr_route_list(struct ctl_context *ctx) >> return; >> } >> >> + char *route_table = shash_find_data(&ctx->options, >> >> "--route-table"); >> >> + >> ipv4_routes = xmalloc(sizeof *ipv4_routes * lr->n_static_routes); >> ipv6_routes = xmalloc(sizeof *ipv6_routes * lr->n_static_routes); >> >> for (int i = 0; i < lr->n_static_routes; i++) { >> const struct nbrec_logical_router_static_route *route >> = lr->static_routes[i]; >> + if (route_table && strcmp(route->route_table, route_table)) { >> + continue; >> + } >> unsigned int plen; >> ovs_be32 ipv4; >> const char *policy = route->policy ? route->policy : "dst-ip"; >> @@ -6081,6 +6175,7 @@ nbctl_lr_route_list(struct ctl_context *ctx) >> if (n_ipv4_routes) { >> ds_put_cstr(&ctx->output, "IPv4 Routes\n"); >> } >> + const struct nbrec_logical_router_static_route *route; >> for (int i = 0; i < n_ipv4_routes; i++) { >> bool ecmp = false; >> if (i < n_ipv4_routes - 1 && >> @@ -6091,6 +6186,15 @@ nbctl_lr_route_list(struct ctl_context *ctx) >> &ipv4_routes[i - 1])) { >> ecmp = true; >> } >> + >> + route = ipv4_routes[i].route; >> + if (!i || (i > 0 && strcmp(route->route_table, >> + ipv4_routes[i - >> >> 1].route->route_table))) { >> >> + ds_put_format(&ctx->output, "%sRoute Table %s:\n", i ? >> >> "\n" : "", >> >> + strlen(route->route_table) ? >> >> route->route_table >> >> + : "global"); >> + } >> + >> print_route(ipv4_routes[i].route, &ctx->output, ecmp); >> } >> >> @@ -6108,6 +6212,15 @@ nbctl_lr_route_list(struct ctl_context *ctx) >> &ipv6_routes[i - 1])) { >> ecmp = true; >> } >> + >> + route = ipv6_routes[i].route; >> + if (!i || (i > 0 && strcmp(route->route_table, >> + ipv6_routes[i - >> >> 1].route->route_table))) { >> >> + ds_put_format(&ctx->output, "%sRoute Table %s:\n", i ? >> >> "\n" : "", >> >> + strlen(route->route_table) ? >> >> route->route_table >> >> + : "global"); >> + } >> + >> print_route(ipv6_routes[i].route, &ctx->output, ecmp); >> } >> >> @@ -6926,6 +7039,8 @@ static const struct ctl_command_syntax >> >> nbctl_commands[] = { >> >> "PORT CHASSIS [PRIORITY]", >> nbctl_pre_lrp_set_gateway_chassis, nbctl_lrp_set_gateway_chassis, >> NULL, "--may-exist", RW }, >> + { "lrp-set-options", 1, INT_MAX, "PORT KEY=VALUE [KEY=VALUE]...", >> + nbctl_pre_lrp_options, nbctl_lrp_set_options, NULL, "", RW }, >> { "lrp-del-gateway-chassis", 2, 2, "PORT CHASSIS", >> nbctl_pre_lrp_del_gateway_chassis, nbctl_lrp_del_gateway_chassis, >> NULL, "", RW }, >> @@ -6949,12 +7064,13 @@ static const struct ctl_command_syntax >> >> nbctl_commands[] = { >> >> /* logical router route commands. */ >> { "lr-route-add", 3, 4, "ROUTER PREFIX NEXTHOP [PORT]", >> nbctl_pre_lr_route_add, nbctl_lr_route_add, NULL, >> - "--may-exist,--ecmp,--ecmp-symmetric-reply,--policy=,--bfd?", RW >> >> }, >> >> + >> >> >> "--may-exist,--ecmp,--ecmp-symmetric-reply,--policy=,--route-table=,--bfd?", >> >> + RW }, >> { "lr-route-del", 1, 4, "ROUTER [PREFIX [NEXTHOP [PORT]]]", >> nbctl_pre_lr_route_del, nbctl_lr_route_del, NULL, >> - "--if-exists,--policy=", RW }, >> + "--if-exists,--policy=,--route-table=", RW }, >> { "lr-route-list", 1, 1, "ROUTER", nbctl_pre_lr_route_list, >> - nbctl_lr_route_list, NULL, "", RO }, >> + nbctl_lr_route_list, NULL, "--route-table=", RO }, >> >> /* Policy commands */ >> { "lr-policy-add", 4, INT_MAX, >> -- >> 2.30.0 >> >> _______________________________________________ >> dev mailing list >> dev@openvswitch.org >> https://mail.openvswitch.org/mailman/listinfo/ovs-dev >> >> >> >> _______________________________________________ >> dev mailing list >> dev@openvswitch.org >> https://mail.openvswitch.org/mailman/listinfo/ovs-dev >> >> >> > _______________________________________________ > dev mailing list > dev@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
On Mon, Oct 18, 2021 at 6:35 AM Odintsov Vladislav <VlOdintsov@croc.ru> wrote: > > > > regards, > Vladislav Odintsov > > > On 16 Oct 2021, at 03:20, Han Zhou <hzhou@ovn.org> wrote: > > > > On Fri, Oct 15, 2021 at 2:36 AM Vladislav Odintsov <odivlad@gmail.com> > > wrote: > > > >> > >> > >> Regards, > >> Vladislav Odintsov > >> > >> On 15 Oct 2021, at 08:42, Han Zhou <hzhou@ovn.org> wrote: > >> > >> On Thu, Oct 14, 2021 at 12:58 AM Vladislav Odintsov <odivlad@gmail.com> > >> wrote: > >> > >> > >> Hi Han, > >> > >> Thanks for the review. > >> > >> Regards, > >> Vladislav Odintsov > >> > >> On 14 Oct 2021, at 08:13, Han Zhou <hzhou@ovn.org> wrote: > >> > >> > >> > >> On Tue, Oct 5, 2021 at 1:26 PM Vladislav Odintsov <odivlad@gmail.com> > >> > >> wrote: > >> > >> > >> This patch extends Logical Router's routing functionality. > >> Now user may create multiple routing tables within a Logical Router > >> and assign them to Logical Router Ports. > >> > >> Traffic coming from Logical Router Port with assigned route_table > >> is checked against global routes if any (Logical_Router_Static_Routes > >> whith empty route_table field), next against directly connected routes > >> > >> > >> This is not accurate. The "directly connected routes" is NOT after the > >> > >> global routes. Their priority only depends on the prefix length. > >> > >> > >> and then Logical_Router_Static_Routes with same route_table value as > >> in Logical_Router_Port options:route_table field. > >> > >> A new Logical Router ingress table #10 is added - IN_IP_ROUTING_PRE. > >> In this table packets which come from LRPs with configured > >> options:route_table field are checked against inport and in OVS > >> register 7 unique non-zero value identifying route table is written. > >> > >> Then in 11th table IN_IP_ROUTING routes which have non-empty > >> `route_table` field are added with additional match on reg7 value > >> associated with appropriate route_table. > >> > >> > >> Hi Vladislav, > >> > >> First of all, sorry for the delayed review, and thanks for implementing > >> > >> this new feature. > >> > >> > >> I have some questions regarding the feature itself. I remember that we > >> > >> had some discussion earlier for this feature, but it seems I misunderstood > >> the feature you are implementing here. I thought by multiple routing tables > >> you were trying to support something like VRF in physical routers, but this > >> seems to be something different. According to your implementation, instead > >> of assigning LRPs to different routing tables, a LRP can actually > >> participate to both the global routing table and the table with a specific > >> ID. For ingress, global routes are prefered over other routes; for egress > >> (i.e. forwarding to the next hop), it doesn't even enforce any table-id > >> check, so packet routed by a entry of table-X can go out of a LRP with > >> table-id Y. Is my understanding correct about the desired behavior of this > >> feature? > >> > >> > >> > >> Yes, your understanding is correct. > >> This is not VRF. At first glance VRF can be done just by using LR-per-VRF > >> > >> without any code modifications (maybe there are corner cases, but it’s not > >> something I’m currently working on). > >> > >> I agree VRF can be achieved by just separate LRs. I am trying to understand > >> why multiple LRs wouldn't satisfy your use case here. > >> > >> LRP can be optionally assigned to specific routing table name. This means > >> > >> that for LR ingress pipeline packets coming from this specific LRP would be > >> checked against routes with same route_table value within appropriate LR. > >> This is some kind of PBR, analog of "ip rule add iif <interface name> > >> lookup <id>". > >> > >> There is one specific use-case, which requires special handling: > >> > >> directly-connected routes (subnet CIDRs from connected to this LR LRPs). > >> These routes can’t be added manually by user, though routing between LRPs > >> belonging to different routing tables is still needed. So, these routes > >> should be added to global routing table to override routing for LRPs with > >> configured routing tables. If for some reason user wants to prohibit IP > >> connectivity to any LRP (honestly, I can’t imagine, why), it can be done by > >> utilising OVN PBR with drop action or a route with "discard" nexthop. But > >> I’d think in this case whether this LRP is needed in this LR. > >> > >> Also, if user wants to create routes, which apply for all LRPs from all > >> > >> routing tables, it can be done by adding new entries to global routing > >> table. > >> > >> And the third reason to have global routing table - a fully > >> > >> backward-compatible solution. All users, who don’t need multiple routing > >> tables support just continue using static routing in the same manner > >> without any changes. > >> > >> > >> > >> If this is correct, it doesn't seem to be a common/standard requirement > >> > >> (or please educate me if I am wrong). Could you explain a little more about > >> the actual use cases: what kind of real world problems need to be solved by > >> this feature or how are you going to use this. For example, why would a > >> port need to participate in both routing tables? It looks like what you > >> really need is policy routing instead of multiple isolated routing tables. > >> I understand that you already use policy routing to implement ACLs, so it > >> is not convenient to combine other policies (e.g. inport based routing) > >> into the policy routing stage. If that's the case, would it be more generic > >> to support multiple policy routing stages? My concern to the current > >> approach is that it is implemented for a very special use case. It makes > >> the code more complex but when there is a slightly different requirement in > >> the future it becomes insufficient. I am thinking that policy routing seems > >> more flexible and has more potential to be made more generic. Maybe I will > >> have a better understanding when I hear more detailed use cases and > >> considerations from you. > >> > >> > >> > >> I can't agree here in it’s uncommon requirement. > >> This implementation was inspired by AWS Route Tables feature [1]: within > >> > >> a VPC (LR in terms of OVN) user may create multiple routing tables and > >> assign them to different subnets (LRPs) in multiple availability zones. > >> Auto-generated directly-connected routes from LRPs CIDRs are working in the > >> same manner as they do in AWS - apply to all subnets regardless of their > >> association to routing table. GCP has similar behaviour: [2], Azure, I > >> guess, too. > >> > >> Our public cloud (CROC Cloud Platform) supports AWS behaviour [3], so I > >> > >> primarily was oriented on it. Internally we already use this feature and > >> currently it fits our use-case, but again I can't say it is specific. > >> > >> If it is for AWS/GCP alike routing features, then from what I understand > >> what's implemented in this patch is a little different. To implement a VPC > >> model in OVN, I think it is ok to have multiple LRs instead of only one LR > >> for each VPC. So naturally I would use a LR to map to AWS's routing table. > >> In AWS's document (the link you provided), it says: > >> > >> "A subnet can only be associated with one route table at a time" > >> > >> So basically in AWS a subnet is either attached to the default/main route > >> table or a custom table, but not both, right? However, in your use case, a > >> LRP (maps to a subnet) attaches to both "main" and a custom table, which > >> seems not common to me. Or did I miss something? > >> > >> > >> That’s true about AWS, but there is still a bit not accurate about OVN. > >> Global routing table in OVN terms is not that AWS main route table is. > >> Main route table is just a configuration hint for users for implicit route > >> tables association with subnets. > >> Implicitly-associated via main routing table subnets routing functions the > >> same manner as a normal explicit route_table-subnet association. > >> > >> Global routing table in OVN is just a list of routes with higher priority > >> than routes with configured "route_table". > >> > >> I do not offer to configure both tables at the same time. But it is > >> _possible_ to do if required for some reason (for instance to configure > >> some service chaining or just internal VPC services like metadata/another > >> internal APIs, access to another services). > >> Normally, if we talk about AWS Route Table to OVN, it is mostly one-to-one > >> mapping, except "Local route": > >> > > > >> Example: > >> AWS Route Table rtb-xxx: > >> 172.31.0.0/16: local route (VPC CIDR for subnets) > >> 0.0.0.0/0: igw-XXX (internet gateway) > >> > >> AWS Route Table rtb-yyy: > >> 172.31.0.0/16: local route (VPC CIDR for subnets) > >> 5.5.5.5/32: instance-xxx > >> > >> This maps to OVN configuration (for one LR with one subnet 172.31.0.0/24): > >> > >> implicit route (not present in logical router static routes table): > >> 172.31.0.0/24: LRP-subnet-xxx - has highest priority over other route > >> table-oriented routes and can be threat as placed in global routing table > >> > > > > What do you mean by highest priority here? It all depends on the prefix > > length, right? If you have a static route entry that says: 172.31.0.0./25 > > -> xyz, then this route will have higher priority than the directly > > connected subnet. > > > > Yes, it is true, but only for routes within “global” routing table. I left this to save previous behaviour. > It is impossible to create a static route within any non-global routing table which overrides directly connected routes, > because priority for “global” routes is added by 100 (though, it should add >2*128 to support same behavior for IPv6 as you already pointed). > > > > >> > >> Normal static routes: > >> ip_prefix: 0.0.0.0/0, nexthop: <IP for edge LR’s LRP>, route_table: > >> rtb-xxx > >> ip_prefix: 5.5.5.5/32, nexthop: <IP of some LSP, which belongs to > >> VM/container via which route is built>, route_table: rtb-yyy > >> > >> I guess, I understood the reason for misunderstanding: the global routing > >> table, which I referred earlier is a routing table which has no value in > >> "route_table" field and directly-connected routes at the same time. Latter > >> have no records in logical router static routes, but I still referred them > >> as a routes from "global routing table". I can think about terminology > >> here, if it’s a root cause for misunderstanding. What do you think? > >> > > > > Thanks for explaining. In fact I understand what you mean about "global > > routing table", but I think you are implementing something different from > > what AWS provides, primarily because you use a single LR instead of LR per > > routing table. I understand there are reasons why you want to do it this > > way, but here I can think of some challenges of your approach. For example, > > in AWS we could do: > > > > route table main: > > subnet S0: > > 172.31.0.0/24 > > routes: > > 172.31.0.0/16: local > > > > route table R1: > > subnet S1: > > 172.31.1.0/24 > > routes: > > 172.31.0.0/16: local > > 172.31.0.0/24: <some FW/GW> > > Wow. It’s a brand-new behaviour for AWS, about which I was not aware, as it appeared 1.5 months ago [1]. > > Previously, when I was writing this patch series in AWS such static route was impossible to add. > You couldn’t add routes, which are the same or more specific than VPC CIDR block. > > Though in GCP you can’t create same or more specific than subnet route [2]. > I think I can try to update my patch to support AWS approach here. Would it be simpler just don't make the global routes higher priority? We could make it clear to the user that if they add a route in any route table that is overlapping with any directly connected subnets, the longer prefix would take precedence. Would it be more flexible to user? > > But all this is relevant to single LR per VPC (not per route table because of described earlier blocker). > Do you think it is reasonable with such inputs? > > > > > Packet from S1 to S0 will go to FW/GW, where it may be dropped/forwarded to > > S0/or redirected to something outside of AWS ... > > > > While in your approach with OVN, both subnets will be under the same > > logical router, and with different table IDs assigned to the LRPs and > > routes. But when a VM under S1 sends a packet to S0, it will go to the > > destination directly because you are prioritizing the direct-connection > > routes and they are under the same global route table, and there is no way > > to force it to go through some FW/GW from the custom route table. You can > > add a route to achieve this in the global table, because in your design the > > global routes are still applicable to all LRPs and has higher priority, but > > it would impact all the LRPs/subnets, while in the AWS design above it is > > supposed to affect subnets under R1 only. > > > > In addition, for my understanding, S0 and S1 in AWS cannot communicate > > directly, unless there are some specific routes on both route tables to > > route them to some gateway. In other words, there are some isolations > > between route tables. However, in your design with OVN, it is more of > > S0 and S1 can communicate with each other being in different route tables without any specific setup, > the default unremovable “Local route” ensures this. > User only can override "local" route with only existing subnet’s (more specific than "local route") CIDR > to send traffic from S0 to S1 through some specific appliance(s) in S2. > But basic IP connectivity always remains. Regardless of route tables subnet associations. > So I’m still sure that one LR per VPC suites this scenario better, > as I can’t imagine why to place subnets, which don’t need connectivity in one VPC. > Maybe I was wrong. I am not really familiar with AWS and I wanted to try this myself but didn't get the time :( If default connectivity for subnets under different route tables is a requirement (instead of the contrary), then I'd agree that one LR per VPC seems better, because it is more like policy routing instead of for isolation. > > policy routing without isolation at all. I am not saying that your design > > is wrong, but just saying it implements very different behavior than AWS, > > and I am trying to understand your real use cases to have a better > > understanding of the feature you implemented. (You explained that you just > > wanted the AWS behavior, but it looks not exactly the case). > > > > Will fix places where the behaviour has difference with AWS. > And thanks you for pointing such places. > Well, I think it doesn't have to be all the same as AWS, but yes if AWS design makes more sense. > > > >> > >> > >> In my opinion having this feature to be implemented using PBR is less > >> > >> convenient and native for users, who are familiar with behaviour for > >> mentioned above public cloud platforms, because configuring routes should > >> be done in routes section. And adding route table property seems native in > >> this route, not in PBR. Moreover, I _think_ using > >> Logical_Router_Static_Route to extend this feature for OVN-Interconnection > >> becomes quite easy comparing to PBR (though, I didn’t try the latter). > >> > >> I agree if it is just AWS -like requirement, PBR is less convenient. > >> > >> I am trying to understand if it can be achieved with separate LRs. If not, > >> what's special about the requirement, and is the current approach providing > >> a solution common enough so that more use cases can also benefit from? > >> Could you clarify a little more? Thanks again. > >> > >> That was our initial approach - to use separate LRs for each Route Table. > >> We rejected that solution because met some difficulties and blocker. See > >> below: > >> > >> Brief topology description if using LRs per Route Table: > >> Imagine 2 subnets in VPC in 1st AZ, one in another AZ. Each subnet in it’s > >> own Route Table (LR). > >> All subnets must have IP connectivity, so we have to somehow interconnect > >> these Route Table LRs. > >> > > > > That's exactly the same case for AWS, right? If you want direct > > connectivity, you would just put them under the same route table in AWS (or > > same LR in OVN), right? > > > > I’m sorry, my example was not good enough. > Let’s remove AZ’s from that case as they’re not relevant to discussion. > So, we’ve got 2 subnets S0 and S1, and suppose, they’ve got different route tables assigned to them. > If we place them in a single LR, assign different route tables, they would have connectivity "out-of-box". > Same in AWS: because "local route" is added automatically, can’t be removed or modified and ensures IP connectivity between all VPC subnets. > Ok, thanks for explaining. > > > >> > >> [BLOCKER] It is impossible to place route in route table 1 via VM from > >> subnet assiciated to route table 2 if using per-RTB LR. Because in RTB-2 LR > >> we have to add route from RTB 1 and this breaks route table isolation. > >> Route Table 2 LR will start looking up routes and there could be routes > >> from another route tables. This breaks the idea of having LR per Route > >> Table completely. Here we rejected this solution and moved to adding > >> support for route tables in OVN. > >> > >> > > Would it be just the same problem in AWS? Why would a user route a subnet > > of RTB-1 to a VM under RTB-2, and at the same time want route table > > isolation? > > With the single LR solution in OVN, you would not get the isolation between > > the tables, because 1) all LRPs still share the global routing table, 2) > > for output to nexthop there is no distinction between the output interfaces. > > > > No, such problem does not exist. > By route tables "isolation" I meant that routing rules configured in one routing table must not affect routing in another one. > In this scenario user may want to route traffic to internet from S0 (RTB-0) through instance in S1 (RTB-1). > For instance it can be a gateway service (FW, WAF, IPS, etc) in S1, which provides secure access for clients from VPC to the internet with NAT. > So, configuration should be: > - 0.0.0.0/0 via GW-INSTANCE in RTB-0 > - 0.0.0.0/0 via edge LR in RTB-1 > So, instances from all subnets with rtb-0 assigned route table will be routed to internet through GW-INSTANCE, which can analyse/drop/do whatever things with traffic. > This is a small example to show how different route tables can be used in a single VPC. > With multiple OVN LRs per VPC in AZ such topology can’t be created, see: > In LR for RTB-0 we can add route 0.0.0.0/0 via <LR0-to-LR1 link IP>; > bun in LR for RTB-1 we can only add route to internet through either edge-LR or GW-INSTANCE. Not both. So, this case can’t be implemented currently in OVN. > Thanks for the example. It helps understanding your requirements. Why I believe this can be achieved by current OVN with the policy routing feature, I understand as you mentioned earlier you need policy routing for ACL purpose. I also agree that multi-stage policy routing would seem less user-friendly (and maybe also harder to implement), so I support the multi-table feature for the purpose of providing extra policy routing capability. While I still need to take a closer look at the code, I believe it would be helpful to add such example use cases in the documentation for users to better understand the feature. > Hope this helps to understand my problem. > > > > >> But some more cons: > >> > >> 1. complex network topology. Interconnecting all LRs even with some > >> transit switch is harder than having one LR and all VPC-related > >> configuration is done in one place (in AZ). > >> 2. Directly-connected routes. In case we have multiple Route Table LRs, we > >> have to add route for each subnet from another LR. In case one LR per VPC > >> all such routes are installed automatically and learnt via ovn-ic. > >> 3. PBR, LBs. It’s much easier to implement PBR and configuring Load > >> Balancers in one LR, than in multiple. > >> 4. There could be very many LRs, LRPs, LSPs (datapaths in SB) - excess > >> database records, huge database growth. > >> 5. Extra client code complexity (updating routes required configuring > >> routes in many LRs on different availibility zones); > >> 5. We couldn’t use ovn-ic routes learning, because VPC requires out-of-box > >> IP connectivity between subnets, and if connect Route Table LRs between > >> AZs, because connecting multiple LRs from different Route Tables would > >> learn routes without binding to route table. This requires additional > >> isolation at the transit switches level. > >> 6. There were some more problems. Here are listed some, which I could > >> refresh in my mind. > >> > >> From our half-of-a-year experience using LR per VPC is very comfortable > >> and it looks quite extendable it terms of network features. > >> > >> Let me know if this makes sense. > >> > >> > > Thanks for the list of the cons! Most of the cons you listed above seem to > > apply to the AWS model, too, right? It is ok to have different requirements > > Honestly, I can’t imagine these bullets to "AWS model". Maybe only ovn-ic, which > requires modifications in ovn-ic codebase to support route_tables in routes. > But this work is already done. I think the disagreement was mainly from the understanding of the behavior of directly connected subnets between different routing tables. I assume my understanding was wrong for now :) > > > than AWS, but we'd better define it clearly and it would be helpful with > > some typical use cases that solve specific problems. I have more > > information now after your explanation, but still not really clear under > > what scenarios would this feature be used. > > > > @Numan Siddique <numans@ovn.org> @Mark Michelson <mmichels@redhat.com> > > could you let me know your thoughts on this, too? Would you see similar use > > cases in Redhat customers? Or does this design match well with your > > potential use cases? I'd like to be careful when implementing something > > like this and make sure we are doing it the right way. > > One comment here from my side: > by "this design" we should talk about "behaviour similar to AWS/other public platforms route tables support". > I think that it is a good feature for opensource SDN, which can be futher used by opensource cloud platforms like OpenStack to extend routing capabilities. > Agree. > > > > Thanks, > > Han > > 1: https://aws.amazon.com/blogs/aws/inspect-subnet-to-subnet-traffic-with-amazon-vpc-more-specific-routing/ > 2: https://cloud.google.com/vpc/docs/routes#subnet-routes > > > > >> Han > >> > >> > >> > >> I haven't finished reviewing the code yet, but I have one comment > >> > >> regarding adding 100 to the priority of the global routes. For IPv6, the > >> priority range from 0 to 120x2=240, so adding 100 is not enough. It would > >> create overlapping priority ranges, and some table-id specific route > >> entries may override the global routes. > >> > >> > >> > >> Thanks, I’ll dig into this when you finish review. > >> > >> > >> Let me know if I answered your questions or if you have new ones. > >> Again many thanks for your time and digging into this patch series. > >> > >> 1: https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Route_Tables.html > >> 2: https://cloud.google.com/vpc/docs/routes#subnet-routes > >> 3: https://docs.cloud.croc.ru/en/services/networks/routetables.html > >> > >> Thanks, > >> Han > >>
On Tue, Oct 19, 2021 at 3:28 AM Han Zhou <hzhou@ovn.org> wrote: > > On Mon, Oct 18, 2021 at 6:35 AM Odintsov Vladislav <VlOdintsov@croc.ru> > wrote: > > > > > > > > regards, > > Vladislav Odintsov > > > > > On 16 Oct 2021, at 03:20, Han Zhou <hzhou@ovn.org> wrote: > > > > > > On Fri, Oct 15, 2021 at 2:36 AM Vladislav Odintsov <odivlad@gmail.com> > > > wrote: > > > > > >> > > >> > > >> Regards, > > >> Vladislav Odintsov > > >> > > >> On 15 Oct 2021, at 08:42, Han Zhou <hzhou@ovn.org> wrote: > > >> > > >> On Thu, Oct 14, 2021 at 12:58 AM Vladislav Odintsov <odivlad@gmail.com> > > >> wrote: > > >> > > >> > > >> Hi Han, > > >> > > >> Thanks for the review. > > >> > > >> Regards, > > >> Vladislav Odintsov > > >> > > >> On 14 Oct 2021, at 08:13, Han Zhou <hzhou@ovn.org> wrote: > > >> > > >> > > >> > > >> On Tue, Oct 5, 2021 at 1:26 PM Vladislav Odintsov <odivlad@gmail.com> > > >> > > >> wrote: > > >> > > >> > > >> This patch extends Logical Router's routing functionality. > > >> Now user may create multiple routing tables within a Logical Router > > >> and assign them to Logical Router Ports. > > >> > > >> Traffic coming from Logical Router Port with assigned route_table > > >> is checked against global routes if any (Logical_Router_Static_Routes > > >> whith empty route_table field), next against directly connected routes > > >> > > >> > > >> This is not accurate. The "directly connected routes" is NOT after the > > >> > > >> global routes. Their priority only depends on the prefix length. > > >> > > >> > > >> and then Logical_Router_Static_Routes with same route_table value as > > >> in Logical_Router_Port options:route_table field. > > >> > > >> A new Logical Router ingress table #10 is added - IN_IP_ROUTING_PRE. > > >> In this table packets which come from LRPs with configured > > >> options:route_table field are checked against inport and in OVS > > >> register 7 unique non-zero value identifying route table is written. > > >> > > >> Then in 11th table IN_IP_ROUTING routes which have non-empty > > >> `route_table` field are added with additional match on reg7 value > > >> associated with appropriate route_table. > > >> > > >> > > >> Hi Vladislav, > > >> > > >> First of all, sorry for the delayed review, and thanks for implementing > > >> > > >> this new feature. > > >> > > >> > > >> I have some questions regarding the feature itself. I remember that we > > >> > > >> had some discussion earlier for this feature, but it seems I > misunderstood > > >> the feature you are implementing here. I thought by multiple routing > tables > > >> you were trying to support something like VRF in physical routers, but > this > > >> seems to be something different. According to your implementation, > instead > > >> of assigning LRPs to different routing tables, a LRP can actually > > >> participate to both the global routing table and the table with a > specific > > >> ID. For ingress, global routes are prefered over other routes; for > egress > > >> (i.e. forwarding to the next hop), it doesn't even enforce any table-id > > >> check, so packet routed by a entry of table-X can go out of a LRP with > > >> table-id Y. Is my understanding correct about the desired behavior of > this > > >> feature? > > >> > > >> > > >> > > >> Yes, your understanding is correct. > > >> This is not VRF. At first glance VRF can be done just by using > LR-per-VRF > > >> > > >> without any code modifications (maybe there are corner cases, but it’s > not > > >> something I’m currently working on). > > >> > > >> I agree VRF can be achieved by just separate LRs. I am trying to > understand > > >> why multiple LRs wouldn't satisfy your use case here. > > >> > > >> LRP can be optionally assigned to specific routing table name. This > means > > >> > > >> that for LR ingress pipeline packets coming from this specific LRP > would be > > >> checked against routes with same route_table value within appropriate > LR. > > >> This is some kind of PBR, analog of "ip rule add iif <interface name> > > >> lookup <id>". > > >> > > >> There is one specific use-case, which requires special handling: > > >> > > >> directly-connected routes (subnet CIDRs from connected to this LR > LRPs). > > >> These routes can’t be added manually by user, though routing between > LRPs > > >> belonging to different routing tables is still needed. So, these routes > > >> should be added to global routing table to override routing for LRPs > with > > >> configured routing tables. If for some reason user wants to prohibit IP > > >> connectivity to any LRP (honestly, I can’t imagine, why), it can be > done by > > >> utilising OVN PBR with drop action or a route with "discard" nexthop. > But > > >> I’d think in this case whether this LRP is needed in this LR. > > >> > > >> Also, if user wants to create routes, which apply for all LRPs from all > > >> > > >> routing tables, it can be done by adding new entries to global routing > > >> table. > > >> > > >> And the third reason to have global routing table - a fully > > >> > > >> backward-compatible solution. All users, who don’t need multiple > routing > > >> tables support just continue using static routing in the same manner > > >> without any changes. > > >> > > >> > > >> > > >> If this is correct, it doesn't seem to be a common/standard requirement > > >> > > >> (or please educate me if I am wrong). Could you explain a little more > about > > >> the actual use cases: what kind of real world problems need to be > solved by > > >> this feature or how are you going to use this. For example, why would a > > >> port need to participate in both routing tables? It looks like what you > > >> really need is policy routing instead of multiple isolated routing > tables. > > >> I understand that you already use policy routing to implement ACLs, so > it > > >> is not convenient to combine other policies (e.g. inport based routing) > > >> into the policy routing stage. If that's the case, would it be more > generic > > >> to support multiple policy routing stages? My concern to the current > > >> approach is that it is implemented for a very special use case. It > makes > > >> the code more complex but when there is a slightly different > requirement in > > >> the future it becomes insufficient. I am thinking that policy routing > seems > > >> more flexible and has more potential to be made more generic. Maybe I > will > > >> have a better understanding when I hear more detailed use cases and > > >> considerations from you. > > >> > > >> > > >> > > >> I can't agree here in it’s uncommon requirement. > > >> This implementation was inspired by AWS Route Tables feature [1]: > within > > >> > > >> a VPC (LR in terms of OVN) user may create multiple routing tables and > > >> assign them to different subnets (LRPs) in multiple availability zones. > > >> Auto-generated directly-connected routes from LRPs CIDRs are working > in the > > >> same manner as they do in AWS - apply to all subnets regardless of > their > > >> association to routing table. GCP has similar behaviour: [2], Azure, I > > >> guess, too. > > >> > > >> Our public cloud (CROC Cloud Platform) supports AWS behaviour [3], so I > > >> > > >> primarily was oriented on it. Internally we already use this feature > and > > >> currently it fits our use-case, but again I can't say it is specific. > > >> > > >> If it is for AWS/GCP alike routing features, then from what I > understand > > >> what's implemented in this patch is a little different. To implement a > VPC > > >> model in OVN, I think it is ok to have multiple LRs instead of only > one LR > > >> for each VPC. So naturally I would use a LR to map to AWS's routing > table. > > >> In AWS's document (the link you provided), it says: > > >> > > >> "A subnet can only be associated with one route table at a time" > > >> > > >> So basically in AWS a subnet is either attached to the default/main > route > > >> table or a custom table, but not both, right? However, in your use > case, a > > >> LRP (maps to a subnet) attaches to both "main" and a custom table, > which > > >> seems not common to me. Or did I miss something? > > >> > > >> > > >> That’s true about AWS, but there is still a bit not accurate about OVN. > > >> Global routing table in OVN terms is not that AWS main route table is. > > >> Main route table is just a configuration hint for users for implicit > route > > >> tables association with subnets. > > >> Implicitly-associated via main routing table subnets routing functions > the > > >> same manner as a normal explicit route_table-subnet association. > > >> > > >> Global routing table in OVN is just a list of routes with higher > priority > > >> than routes with configured "route_table". > > >> > > >> I do not offer to configure both tables at the same time. But it is > > >> _possible_ to do if required for some reason (for instance to configure > > >> some service chaining or just internal VPC services like > metadata/another > > >> internal APIs, access to another services). > > >> Normally, if we talk about AWS Route Table to OVN, it is mostly > one-to-one > > >> mapping, except "Local route": > > >> > > > > > >> Example: > > >> AWS Route Table rtb-xxx: > > >> 172.31.0.0/16: local route (VPC CIDR for subnets) > > >> 0.0.0.0/0: igw-XXX (internet gateway) > > >> > > >> AWS Route Table rtb-yyy: > > >> 172.31.0.0/16: local route (VPC CIDR for subnets) > > >> 5.5.5.5/32: instance-xxx > > >> > > >> This maps to OVN configuration (for one LR with one subnet > 172.31.0.0/24): > > >> > > >> implicit route (not present in logical router static routes table): > > >> 172.31.0.0/24: LRP-subnet-xxx - has highest priority over other route > > >> table-oriented routes and can be threat as placed in global routing > table > > >> > > > > > > What do you mean by highest priority here? It all depends on the prefix > > > length, right? If you have a static route entry that says: > 172.31.0.0./25 > > > -> xyz, then this route will have higher priority than the directly > > > connected subnet. > > > > > > > Yes, it is true, but only for routes within “global” routing table. I > left this to save previous behaviour. > > It is impossible to create a static route within any non-global routing > table which overrides directly connected routes, > > because priority for “global” routes is added by 100 (though, it should > add >2*128 to support same behavior for IPv6 as you already pointed). > > > > > > > >> > > >> Normal static routes: > > >> ip_prefix: 0.0.0.0/0, nexthop: <IP for edge LR’s LRP>, route_table: > > >> rtb-xxx > > >> ip_prefix: 5.5.5.5/32, nexthop: <IP of some LSP, which belongs to > > >> VM/container via which route is built>, route_table: rtb-yyy > > >> > > >> I guess, I understood the reason for misunderstanding: the global > routing > > >> table, which I referred earlier is a routing table which has no value > in > > >> "route_table" field and directly-connected routes at the same time. > Latter > > >> have no records in logical router static routes, but I still referred > them > > >> as a routes from "global routing table". I can think about terminology > > >> here, if it’s a root cause for misunderstanding. What do you think? > > >> > > > > > > Thanks for explaining. In fact I understand what you mean about "global > > > routing table", but I think you are implementing something different > from > > > what AWS provides, primarily because you use a single LR instead of LR > per > > > routing table. I understand there are reasons why you want to do it this > > > way, but here I can think of some challenges of your approach. For > example, > > > in AWS we could do: > > > > > > route table main: > > > subnet S0: > > > 172.31.0.0/24 > > > routes: > > > 172.31.0.0/16: local > > > > > > route table R1: > > > subnet S1: > > > 172.31.1.0/24 > > > routes: > > > 172.31.0.0/16: local > > > 172.31.0.0/24: <some FW/GW> > > > > Wow. It’s a brand-new behaviour for AWS, about which I was not aware, as > it appeared 1.5 months ago [1]. > > > > Previously, when I was writing this patch series in AWS such static route > was impossible to add. > > You couldn’t add routes, which are the same or more specific than VPC > CIDR block. > > > > Though in GCP you can’t create same or more specific than subnet route > [2]. > > I think I can try to update my patch to support AWS approach here. > > Would it be simpler just don't make the global routes higher priority? We > could make it clear to the user that if they add a route in any route table > that is overlapping with any directly connected subnets, the longer prefix > would take precedence. Would it be more flexible to user? > > > > > But all this is relevant to single LR per VPC (not per route table > because of described earlier blocker). > > Do you think it is reasonable with such inputs? > > > > > > > > Packet from S1 to S0 will go to FW/GW, where it may be > dropped/forwarded to > > > S0/or redirected to something outside of AWS ... > > > > > > While in your approach with OVN, both subnets will be under the same > > > logical router, and with different table IDs assigned to the LRPs and > > > routes. But when a VM under S1 sends a packet to S0, it will go to the > > > destination directly because you are prioritizing the direct-connection > > > routes and they are under the same global route table, and there is no > way > > > to force it to go through some FW/GW from the custom route table. You > can > > > add a route to achieve this in the global table, because in your design > the > > > global routes are still applicable to all LRPs and has higher priority, > but > > > it would impact all the LRPs/subnets, while in the AWS design above it > is > > > supposed to affect subnets under R1 only. > > > > > > In addition, for my understanding, S0 and S1 in AWS cannot communicate > > > directly, unless there are some specific routes on both route tables to > > > route them to some gateway. In other words, there are some isolations > > > between route tables. However, in your design with OVN, it is more of > > > > S0 and S1 can communicate with each other being in different route tables > without any specific setup, > > the default unremovable “Local route” ensures this. > > User only can override "local" route with only existing subnet’s (more > specific than "local route") CIDR > > to send traffic from S0 to S1 through some specific appliance(s) in S2. > > But basic IP connectivity always remains. Regardless of route tables > subnet associations. > > So I’m still sure that one LR per VPC suites this scenario better, > > as I can’t imagine why to place subnets, which don’t need connectivity in > one VPC. > > > Maybe I was wrong. I am not really familiar with AWS and I wanted to try > this myself but didn't get the time :( > If default connectivity for subnets under different route tables is a > requirement (instead of the contrary), then I'd agree that one LR per VPC > seems better, because it is more like policy routing instead of for > isolation. > > > > policy routing without isolation at all. I am not saying that your > design > > > is wrong, but just saying it implements very different behavior than > AWS, > > > and I am trying to understand your real use cases to have a better > > > understanding of the feature you implemented. (You explained that you > just > > > wanted the AWS behavior, but it looks not exactly the case). > > > > > > > Will fix places where the behaviour has difference with AWS. > > And thanks you for pointing such places. > > > Well, I think it doesn't have to be all the same as AWS, but yes if AWS > design makes more sense. > > > > > > >> > > >> > > >> In my opinion having this feature to be implemented using PBR is less > > >> > > >> convenient and native for users, who are familiar with behaviour for > > >> mentioned above public cloud platforms, because configuring routes > should > > >> be done in routes section. And adding route table property seems > native in > > >> this route, not in PBR. Moreover, I _think_ using > > >> Logical_Router_Static_Route to extend this feature for > OVN-Interconnection > > >> becomes quite easy comparing to PBR (though, I didn’t try the latter). > > >> > > >> I agree if it is just AWS -like requirement, PBR is less convenient. > > >> > > >> I am trying to understand if it can be achieved with separate LRs. If > not, > > >> what's special about the requirement, and is the current approach > providing > > >> a solution common enough so that more use cases can also benefit from? > > >> Could you clarify a little more? Thanks again. > > >> > > >> That was our initial approach - to use separate LRs for each Route > Table. > > >> We rejected that solution because met some difficulties and blocker. > See > > >> below: > > >> > > >> Brief topology description if using LRs per Route Table: > > >> Imagine 2 subnets in VPC in 1st AZ, one in another AZ. Each subnet in > it’s > > >> own Route Table (LR). > > >> All subnets must have IP connectivity, so we have to somehow > interconnect > > >> these Route Table LRs. > > >> > > > > > > That's exactly the same case for AWS, right? If you want direct > > > connectivity, you would just put them under the same route table in AWS > (or > > > same LR in OVN), right? > > > > > > > I’m sorry, my example was not good enough. > > Let’s remove AZ’s from that case as they’re not relevant to discussion. > > So, we’ve got 2 subnets S0 and S1, and suppose, they’ve got different > route tables assigned to them. > > If we place them in a single LR, assign different route tables, they > would have connectivity "out-of-box". > > Same in AWS: because "local route" is added automatically, can’t be > removed or modified and ensures IP connectivity between all VPC subnets. > > > Ok, thanks for explaining. > > > > > > >> > > >> [BLOCKER] It is impossible to place route in route table 1 via VM from > > >> subnet assiciated to route table 2 if using per-RTB LR. Because in > RTB-2 LR > > >> we have to add route from RTB 1 and this breaks route table isolation. > > >> Route Table 2 LR will start looking up routes and there could be routes > > >> from another route tables. This breaks the idea of having LR per Route > > >> Table completely. Here we rejected this solution and moved to adding > > >> support for route tables in OVN. > > >> > > >> > > > Would it be just the same problem in AWS? Why would a user route a > subnet > > > of RTB-1 to a VM under RTB-2, and at the same time want route table > > > isolation? > > > With the single LR solution in OVN, you would not get the isolation > between > > > the tables, because 1) all LRPs still share the global routing table, 2) > > > for output to nexthop there is no distinction between the output > interfaces. > > > > > > > No, such problem does not exist. > > By route tables "isolation" I meant that routing rules configured in one > routing table must not affect routing in another one. > > In this scenario user may want to route traffic to internet from S0 > (RTB-0) through instance in S1 (RTB-1). > > For instance it can be a gateway service (FW, WAF, IPS, etc) in S1, which > provides secure access for clients from VPC to the internet with NAT. > > So, configuration should be: > > - 0.0.0.0/0 via GW-INSTANCE in RTB-0 > > - 0.0.0.0/0 via edge LR in RTB-1 > > So, instances from all subnets with rtb-0 assigned route table will be > routed to internet through GW-INSTANCE, which can analyse/drop/do whatever > things with traffic. > > This is a small example to show how different route tables can be used in > a single VPC. > > With multiple OVN LRs per VPC in AZ such topology can’t be created, see: > > In LR for RTB-0 we can add route 0.0.0.0/0 via <LR0-to-LR1 link IP>; > > bun in LR for RTB-1 we can only add route to internet through either > edge-LR or GW-INSTANCE. Not both. So, this case can’t be implemented > currently in OVN. > > > Thanks for the example. It helps understanding your requirements. Why I > believe this can be achieved by current OVN with the policy routing > feature, I understand as you mentioned earlier you need policy routing for > ACL purpose. I also agree that multi-stage policy routing would seem less > user-friendly (and maybe also harder to implement), so I support the > multi-table feature for the purpose of providing extra policy routing > capability. > > While I still need to take a closer look at the code, I believe it would be > helpful to add such example use cases in the documentation for users to > better understand the feature. > > > Hope this helps to understand my problem. > > > > > > > >> But some more cons: > > >> > > >> 1. complex network topology. Interconnecting all LRs even with some > > >> transit switch is harder than having one LR and all VPC-related > > >> configuration is done in one place (in AZ). > > >> 2. Directly-connected routes. In case we have multiple Route Table > LRs, we > > >> have to add route for each subnet from another LR. In case one LR per > VPC > > >> all such routes are installed automatically and learnt via ovn-ic. > > >> 3. PBR, LBs. It’s much easier to implement PBR and configuring Load > > >> Balancers in one LR, than in multiple. > > >> 4. There could be very many LRs, LRPs, LSPs (datapaths in SB) - excess > > >> database records, huge database growth. > > >> 5. Extra client code complexity (updating routes required configuring > > >> routes in many LRs on different availibility zones); > > >> 5. We couldn’t use ovn-ic routes learning, because VPC requires > out-of-box > > >> IP connectivity between subnets, and if connect Route Table LRs between > > >> AZs, because connecting multiple LRs from different Route Tables would > > >> learn routes without binding to route table. This requires additional > > >> isolation at the transit switches level. > > >> 6. There were some more problems. Here are listed some, which I could > > >> refresh in my mind. > > >> > > >> From our half-of-a-year experience using LR per VPC is very comfortable > > >> and it looks quite extendable it terms of network features. > > >> > > >> Let me know if this makes sense. > > >> > > >> > > > Thanks for the list of the cons! Most of the cons you listed above seem > to > > > apply to the AWS model, too, right? It is ok to have different > requirements > > > > Honestly, I can’t imagine these bullets to "AWS model". Maybe only > ovn-ic, which > > requires modifications in ovn-ic codebase to support route_tables in > routes. > > But this work is already done. > > I think the disagreement was mainly from the understanding of the behavior > of directly connected subnets between different routing tables. I assume my > understanding was wrong for now :) > > > > > than AWS, but we'd better define it clearly and it would be helpful with > > > some typical use cases that solve specific problems. I have more > > > information now after your explanation, but still not really clear under > > > what scenarios would this feature be used. > > > > > > @Numan Siddique <numans@ovn.org> @Mark Michelson <mmichels@redhat.com> > > > could you let me know your thoughts on this, too? Would you see similar > use > > > cases in Redhat customers? Or does this design match well with your > > > potential use cases? I'd like to be careful when implementing something > > > like this and make sure we are doing it the right way. I don't think there is any use case in ovn-k8s or openstack neutron which would use this feature. Sorry. I've not followed the discussion here closely. What is the tl;dr. @Han Zhou You're fine with this feature and will we be accepting this in OVN ? Thanks Numan > > > > One comment here from my side: > > by "this design" we should talk about "behaviour similar to AWS/other > public platforms route tables support". > > I think that it is a good feature for opensource SDN, which can be futher > used by opensource cloud platforms like OpenStack to extend routing > capabilities. > > > Agree. > > > > > > > Thanks, > > > Han > > > > 1: > https://aws.amazon.com/blogs/aws/inspect-subnet-to-subnet-traffic-with-amazon-vpc-more-specific-routing/ > > 2: https://cloud.google.com/vpc/docs/routes#subnet-routes > > > > > > > >> Han > > >> > > >> > > >> > > >> I haven't finished reviewing the code yet, but I have one comment > > >> > > >> regarding adding 100 to the priority of the global routes. For IPv6, > the > > >> priority range from 0 to 120x2=240, so adding 100 is not enough. It > would > > >> create overlapping priority ranges, and some table-id specific route > > >> entries may override the global routes. > > >> > > >> > > >> > > >> Thanks, I’ll dig into this when you finish review. > > >> > > >> > > >> Let me know if I answered your questions or if you have new ones. > > >> Again many thanks for your time and digging into this patch series. > > >> > > >> 1: > https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Route_Tables.html > > >> 2: https://cloud.google.com/vpc/docs/routes#subnet-routes > > >> 3: https://docs.cloud.croc.ru/en/services/networks/routetables.html > > >> > > >> Thanks, > > >> Han > > >> > _______________________________________________ > dev mailing list > dev@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
On Tue, Oct 19, 2021 at 3:21 PM Numan Siddique <numans@ovn.org> wrote: > > On Tue, Oct 19, 2021 at 3:28 AM Han Zhou <hzhou@ovn.org> wrote: > > > > On Mon, Oct 18, 2021 at 6:35 AM Odintsov Vladislav <VlOdintsov@croc.ru> > > wrote: > > > > > > > > > > > > regards, > > > Vladislav Odintsov > > > > > > > On 16 Oct 2021, at 03:20, Han Zhou <hzhou@ovn.org> wrote: > > > > > > > > On Fri, Oct 15, 2021 at 2:36 AM Vladislav Odintsov < odivlad@gmail.com> > > > > wrote: > > > > > > > >> > > > >> > > > >> Regards, > > > >> Vladislav Odintsov > > > >> > > > >> On 15 Oct 2021, at 08:42, Han Zhou <hzhou@ovn.org> wrote: > > > >> > > > >> On Thu, Oct 14, 2021 at 12:58 AM Vladislav Odintsov < odivlad@gmail.com> > > > >> wrote: > > > >> > > > >> > > > >> Hi Han, > > > >> > > > >> Thanks for the review. > > > >> > > > >> Regards, > > > >> Vladislav Odintsov > > > >> > > > >> On 14 Oct 2021, at 08:13, Han Zhou <hzhou@ovn.org> wrote: > > > >> > > > >> > > > >> > > > >> On Tue, Oct 5, 2021 at 1:26 PM Vladislav Odintsov < odivlad@gmail.com> > > > >> > > > >> wrote: > > > >> > > > >> > > > >> This patch extends Logical Router's routing functionality. > > > >> Now user may create multiple routing tables within a Logical Router > > > >> and assign them to Logical Router Ports. > > > >> > > > >> Traffic coming from Logical Router Port with assigned route_table > > > >> is checked against global routes if any (Logical_Router_Static_Routes > > > >> whith empty route_table field), next against directly connected routes > > > >> > > > >> > > > >> This is not accurate. The "directly connected routes" is NOT after the > > > >> > > > >> global routes. Their priority only depends on the prefix length. > > > >> > > > >> > > > >> and then Logical_Router_Static_Routes with same route_table value as > > > >> in Logical_Router_Port options:route_table field. > > > >> > > > >> A new Logical Router ingress table #10 is added - IN_IP_ROUTING_PRE. > > > >> In this table packets which come from LRPs with configured > > > >> options:route_table field are checked against inport and in OVS > > > >> register 7 unique non-zero value identifying route table is written. > > > >> > > > >> Then in 11th table IN_IP_ROUTING routes which have non-empty > > > >> `route_table` field are added with additional match on reg7 value > > > >> associated with appropriate route_table. > > > >> > > > >> > > > >> Hi Vladislav, > > > >> > > > >> First of all, sorry for the delayed review, and thanks for implementing > > > >> > > > >> this new feature. > > > >> > > > >> > > > >> I have some questions regarding the feature itself. I remember that we > > > >> > > > >> had some discussion earlier for this feature, but it seems I > > misunderstood > > > >> the feature you are implementing here. I thought by multiple routing > > tables > > > >> you were trying to support something like VRF in physical routers, but > > this > > > >> seems to be something different. According to your implementation, > > instead > > > >> of assigning LRPs to different routing tables, a LRP can actually > > > >> participate to both the global routing table and the table with a > > specific > > > >> ID. For ingress, global routes are prefered over other routes; for > > egress > > > >> (i.e. forwarding to the next hop), it doesn't even enforce any table-id > > > >> check, so packet routed by a entry of table-X can go out of a LRP with > > > >> table-id Y. Is my understanding correct about the desired behavior of > > this > > > >> feature? > > > >> > > > >> > > > >> > > > >> Yes, your understanding is correct. > > > >> This is not VRF. At first glance VRF can be done just by using > > LR-per-VRF > > > >> > > > >> without any code modifications (maybe there are corner cases, but it’s > > not > > > >> something I’m currently working on). > > > >> > > > >> I agree VRF can be achieved by just separate LRs. I am trying to > > understand > > > >> why multiple LRs wouldn't satisfy your use case here. > > > >> > > > >> LRP can be optionally assigned to specific routing table name. This > > means > > > >> > > > >> that for LR ingress pipeline packets coming from this specific LRP > > would be > > > >> checked against routes with same route_table value within appropriate > > LR. > > > >> This is some kind of PBR, analog of "ip rule add iif <interface name> > > > >> lookup <id>". > > > >> > > > >> There is one specific use-case, which requires special handling: > > > >> > > > >> directly-connected routes (subnet CIDRs from connected to this LR > > LRPs). > > > >> These routes can’t be added manually by user, though routing between > > LRPs > > > >> belonging to different routing tables is still needed. So, these routes > > > >> should be added to global routing table to override routing for LRPs > > with > > > >> configured routing tables. If for some reason user wants to prohibit IP > > > >> connectivity to any LRP (honestly, I can’t imagine, why), it can be > > done by > > > >> utilising OVN PBR with drop action or a route with "discard" nexthop. > > But > > > >> I’d think in this case whether this LRP is needed in this LR. > > > >> > > > >> Also, if user wants to create routes, which apply for all LRPs from all > > > >> > > > >> routing tables, it can be done by adding new entries to global routing > > > >> table. > > > >> > > > >> And the third reason to have global routing table - a fully > > > >> > > > >> backward-compatible solution. All users, who don’t need multiple > > routing > > > >> tables support just continue using static routing in the same manner > > > >> without any changes. > > > >> > > > >> > > > >> > > > >> If this is correct, it doesn't seem to be a common/standard requirement > > > >> > > > >> (or please educate me if I am wrong). Could you explain a little more > > about > > > >> the actual use cases: what kind of real world problems need to be > > solved by > > > >> this feature or how are you going to use this. For example, why would a > > > >> port need to participate in both routing tables? It looks like what you > > > >> really need is policy routing instead of multiple isolated routing > > tables. > > > >> I understand that you already use policy routing to implement ACLs, so > > it > > > >> is not convenient to combine other policies (e.g. inport based routing) > > > >> into the policy routing stage. If that's the case, would it be more > > generic > > > >> to support multiple policy routing stages? My concern to the current > > > >> approach is that it is implemented for a very special use case. It > > makes > > > >> the code more complex but when there is a slightly different > > requirement in > > > >> the future it becomes insufficient. I am thinking that policy routing > > seems > > > >> more flexible and has more potential to be made more generic. Maybe I > > will > > > >> have a better understanding when I hear more detailed use cases and > > > >> considerations from you. > > > >> > > > >> > > > >> > > > >> I can't agree here in it’s uncommon requirement. > > > >> This implementation was inspired by AWS Route Tables feature [1]: > > within > > > >> > > > >> a VPC (LR in terms of OVN) user may create multiple routing tables and > > > >> assign them to different subnets (LRPs) in multiple availability zones. > > > >> Auto-generated directly-connected routes from LRPs CIDRs are working > > in the > > > >> same manner as they do in AWS - apply to all subnets regardless of > > their > > > >> association to routing table. GCP has similar behaviour: [2], Azure, I > > > >> guess, too. > > > >> > > > >> Our public cloud (CROC Cloud Platform) supports AWS behaviour [3], so I > > > >> > > > >> primarily was oriented on it. Internally we already use this feature > > and > > > >> currently it fits our use-case, but again I can't say it is specific. > > > >> > > > >> If it is for AWS/GCP alike routing features, then from what I > > understand > > > >> what's implemented in this patch is a little different. To implement a > > VPC > > > >> model in OVN, I think it is ok to have multiple LRs instead of only > > one LR > > > >> for each VPC. So naturally I would use a LR to map to AWS's routing > > table. > > > >> In AWS's document (the link you provided), it says: > > > >> > > > >> "A subnet can only be associated with one route table at a time" > > > >> > > > >> So basically in AWS a subnet is either attached to the default/main > > route > > > >> table or a custom table, but not both, right? However, in your use > > case, a > > > >> LRP (maps to a subnet) attaches to both "main" and a custom table, > > which > > > >> seems not common to me. Or did I miss something? > > > >> > > > >> > > > >> That’s true about AWS, but there is still a bit not accurate about OVN. > > > >> Global routing table in OVN terms is not that AWS main route table is. > > > >> Main route table is just a configuration hint for users for implicit > > route > > > >> tables association with subnets. > > > >> Implicitly-associated via main routing table subnets routing functions > > the > > > >> same manner as a normal explicit route_table-subnet association. > > > >> > > > >> Global routing table in OVN is just a list of routes with higher > > priority > > > >> than routes with configured "route_table". > > > >> > > > >> I do not offer to configure both tables at the same time. But it is > > > >> _possible_ to do if required for some reason (for instance to configure > > > >> some service chaining or just internal VPC services like > > metadata/another > > > >> internal APIs, access to another services). > > > >> Normally, if we talk about AWS Route Table to OVN, it is mostly > > one-to-one > > > >> mapping, except "Local route": > > > >> > > > > > > > >> Example: > > > >> AWS Route Table rtb-xxx: > > > >> 172.31.0.0/16: local route (VPC CIDR for subnets) > > > >> 0.0.0.0/0: igw-XXX (internet gateway) > > > >> > > > >> AWS Route Table rtb-yyy: > > > >> 172.31.0.0/16: local route (VPC CIDR for subnets) > > > >> 5.5.5.5/32: instance-xxx > > > >> > > > >> This maps to OVN configuration (for one LR with one subnet > > 172.31.0.0/24): > > > >> > > > >> implicit route (not present in logical router static routes table): > > > >> 172.31.0.0/24: LRP-subnet-xxx - has highest priority over other route > > > >> table-oriented routes and can be threat as placed in global routing > > table > > > >> > > > > > > > > What do you mean by highest priority here? It all depends on the prefix > > > > length, right? If you have a static route entry that says: > > 172.31.0.0./25 > > > > -> xyz, then this route will have higher priority than the directly > > > > connected subnet. > > > > > > > > > > Yes, it is true, but only for routes within “global” routing table. I > > left this to save previous behaviour. > > > It is impossible to create a static route within any non-global routing > > table which overrides directly connected routes, > > > because priority for “global” routes is added by 100 (though, it should > > add >2*128 to support same behavior for IPv6 as you already pointed). > > > > > > > > > > >> > > > >> Normal static routes: > > > >> ip_prefix: 0.0.0.0/0, nexthop: <IP for edge LR’s LRP>, route_table: > > > >> rtb-xxx > > > >> ip_prefix: 5.5.5.5/32, nexthop: <IP of some LSP, which belongs to > > > >> VM/container via which route is built>, route_table: rtb-yyy > > > >> > > > >> I guess, I understood the reason for misunderstanding: the global > > routing > > > >> table, which I referred earlier is a routing table which has no value > > in > > > >> "route_table" field and directly-connected routes at the same time. > > Latter > > > >> have no records in logical router static routes, but I still referred > > them > > > >> as a routes from "global routing table". I can think about terminology > > > >> here, if it’s a root cause for misunderstanding. What do you think? > > > >> > > > > > > > > Thanks for explaining. In fact I understand what you mean about "global > > > > routing table", but I think you are implementing something different > > from > > > > what AWS provides, primarily because you use a single LR instead of LR > > per > > > > routing table. I understand there are reasons why you want to do it this > > > > way, but here I can think of some challenges of your approach. For > > example, > > > > in AWS we could do: > > > > > > > > route table main: > > > > subnet S0: > > > > 172.31.0.0/24 > > > > routes: > > > > 172.31.0.0/16: local > > > > > > > > route table R1: > > > > subnet S1: > > > > 172.31.1.0/24 > > > > routes: > > > > 172.31.0.0/16: local > > > > 172.31.0.0/24: <some FW/GW> > > > > > > Wow. It’s a brand-new behaviour for AWS, about which I was not aware, as > > it appeared 1.5 months ago [1]. > > > > > > Previously, when I was writing this patch series in AWS such static route > > was impossible to add. > > > You couldn’t add routes, which are the same or more specific than VPC > > CIDR block. > > > > > > Though in GCP you can’t create same or more specific than subnet route > > [2]. > > > I think I can try to update my patch to support AWS approach here. > > > > Would it be simpler just don't make the global routes higher priority? We > > could make it clear to the user that if they add a route in any route table > > that is overlapping with any directly connected subnets, the longer prefix > > would take precedence. Would it be more flexible to user? > > > > > > > > But all this is relevant to single LR per VPC (not per route table > > because of described earlier blocker). > > > Do you think it is reasonable with such inputs? > > > > > > > > > > > Packet from S1 to S0 will go to FW/GW, where it may be > > dropped/forwarded to > > > > S0/or redirected to something outside of AWS ... > > > > > > > > While in your approach with OVN, both subnets will be under the same > > > > logical router, and with different table IDs assigned to the LRPs and > > > > routes. But when a VM under S1 sends a packet to S0, it will go to the > > > > destination directly because you are prioritizing the direct-connection > > > > routes and they are under the same global route table, and there is no > > way > > > > to force it to go through some FW/GW from the custom route table. You > > can > > > > add a route to achieve this in the global table, because in your design > > the > > > > global routes are still applicable to all LRPs and has higher priority, > > but > > > > it would impact all the LRPs/subnets, while in the AWS design above it > > is > > > > supposed to affect subnets under R1 only. > > > > > > > > In addition, for my understanding, S0 and S1 in AWS cannot communicate > > > > directly, unless there are some specific routes on both route tables to > > > > route them to some gateway. In other words, there are some isolations > > > > between route tables. However, in your design with OVN, it is more of > > > > > > S0 and S1 can communicate with each other being in different route tables > > without any specific setup, > > > the default unremovable “Local route” ensures this. > > > User only can override "local" route with only existing subnet’s (more > > specific than "local route") CIDR > > > to send traffic from S0 to S1 through some specific appliance(s) in S2. > > > But basic IP connectivity always remains. Regardless of route tables > > subnet associations. > > > So I’m still sure that one LR per VPC suites this scenario better, > > > as I can’t imagine why to place subnets, which don’t need connectivity in > > one VPC. > > > > > Maybe I was wrong. I am not really familiar with AWS and I wanted to try > > this myself but didn't get the time :( > > If default connectivity for subnets under different route tables is a > > requirement (instead of the contrary), then I'd agree that one LR per VPC > > seems better, because it is more like policy routing instead of for > > isolation. > > > > > > policy routing without isolation at all. I am not saying that your > > design > > > > is wrong, but just saying it implements very different behavior than > > AWS, > > > > and I am trying to understand your real use cases to have a better > > > > understanding of the feature you implemented. (You explained that you > > just > > > > wanted the AWS behavior, but it looks not exactly the case). > > > > > > > > > > Will fix places where the behaviour has difference with AWS. > > > And thanks you for pointing such places. > > > > > Well, I think it doesn't have to be all the same as AWS, but yes if AWS > > design makes more sense. > > > > > > > > > >> > > > >> > > > >> In my opinion having this feature to be implemented using PBR is less > > > >> > > > >> convenient and native for users, who are familiar with behaviour for > > > >> mentioned above public cloud platforms, because configuring routes > > should > > > >> be done in routes section. And adding route table property seems > > native in > > > >> this route, not in PBR. Moreover, I _think_ using > > > >> Logical_Router_Static_Route to extend this feature for > > OVN-Interconnection > > > >> becomes quite easy comparing to PBR (though, I didn’t try the latter). > > > >> > > > >> I agree if it is just AWS -like requirement, PBR is less convenient. > > > >> > > > >> I am trying to understand if it can be achieved with separate LRs. If > > not, > > > >> what's special about the requirement, and is the current approach > > providing > > > >> a solution common enough so that more use cases can also benefit from? > > > >> Could you clarify a little more? Thanks again. > > > >> > > > >> That was our initial approach - to use separate LRs for each Route > > Table. > > > >> We rejected that solution because met some difficulties and blocker. > > See > > > >> below: > > > >> > > > >> Brief topology description if using LRs per Route Table: > > > >> Imagine 2 subnets in VPC in 1st AZ, one in another AZ. Each subnet in > > it’s > > > >> own Route Table (LR). > > > >> All subnets must have IP connectivity, so we have to somehow > > interconnect > > > >> these Route Table LRs. > > > >> > > > > > > > > That's exactly the same case for AWS, right? If you want direct > > > > connectivity, you would just put them under the same route table in AWS > > (or > > > > same LR in OVN), right? > > > > > > > > > > I’m sorry, my example was not good enough. > > > Let’s remove AZ’s from that case as they’re not relevant to discussion. > > > So, we’ve got 2 subnets S0 and S1, and suppose, they’ve got different > > route tables assigned to them. > > > If we place them in a single LR, assign different route tables, they > > would have connectivity "out-of-box". > > > Same in AWS: because "local route" is added automatically, can’t be > > removed or modified and ensures IP connectivity between all VPC subnets. > > > > > Ok, thanks for explaining. > > > > > > > > > >> > > > >> [BLOCKER] It is impossible to place route in route table 1 via VM from > > > >> subnet assiciated to route table 2 if using per-RTB LR. Because in > > RTB-2 LR > > > >> we have to add route from RTB 1 and this breaks route table isolation. > > > >> Route Table 2 LR will start looking up routes and there could be routes > > > >> from another route tables. This breaks the idea of having LR per Route > > > >> Table completely. Here we rejected this solution and moved to adding > > > >> support for route tables in OVN. > > > >> > > > >> > > > > Would it be just the same problem in AWS? Why would a user route a > > subnet > > > > of RTB-1 to a VM under RTB-2, and at the same time want route table > > > > isolation? > > > > With the single LR solution in OVN, you would not get the isolation > > between > > > > the tables, because 1) all LRPs still share the global routing table, 2) > > > > for output to nexthop there is no distinction between the output > > interfaces. > > > > > > > > > > No, such problem does not exist. > > > By route tables "isolation" I meant that routing rules configured in one > > routing table must not affect routing in another one. > > > In this scenario user may want to route traffic to internet from S0 > > (RTB-0) through instance in S1 (RTB-1). > > > For instance it can be a gateway service (FW, WAF, IPS, etc) in S1, which > > provides secure access for clients from VPC to the internet with NAT. > > > So, configuration should be: > > > - 0.0.0.0/0 via GW-INSTANCE in RTB-0 > > > - 0.0.0.0/0 via edge LR in RTB-1 > > > So, instances from all subnets with rtb-0 assigned route table will be > > routed to internet through GW-INSTANCE, which can analyse/drop/do whatever > > things with traffic. > > > This is a small example to show how different route tables can be used in > > a single VPC. > > > With multiple OVN LRs per VPC in AZ such topology can’t be created, see: > > > In LR for RTB-0 we can add route 0.0.0.0/0 via <LR0-to-LR1 link IP>; > > > bun in LR for RTB-1 we can only add route to internet through either > > edge-LR or GW-INSTANCE. Not both. So, this case can’t be implemented > > currently in OVN. > > > > > Thanks for the example. It helps understanding your requirements. Why I > > believe this can be achieved by current OVN with the policy routing > > feature, I understand as you mentioned earlier you need policy routing for > > ACL purpose. I also agree that multi-stage policy routing would seem less > > user-friendly (and maybe also harder to implement), so I support the > > multi-table feature for the purpose of providing extra policy routing > > capability. > > > > While I still need to take a closer look at the code, I believe it would be > > helpful to add such example use cases in the documentation for users to > > better understand the feature. > > > > > Hope this helps to understand my problem. > > > > > > > > > > >> But some more cons: > > > >> > > > >> 1. complex network topology. Interconnecting all LRs even with some > > > >> transit switch is harder than having one LR and all VPC-related > > > >> configuration is done in one place (in AZ). > > > >> 2. Directly-connected routes. In case we have multiple Route Table > > LRs, we > > > >> have to add route for each subnet from another LR. In case one LR per > > VPC > > > >> all such routes are installed automatically and learnt via ovn-ic. > > > >> 3. PBR, LBs. It’s much easier to implement PBR and configuring Load > > > >> Balancers in one LR, than in multiple. > > > >> 4. There could be very many LRs, LRPs, LSPs (datapaths in SB) - excess > > > >> database records, huge database growth. > > > >> 5. Extra client code complexity (updating routes required configuring > > > >> routes in many LRs on different availibility zones); > > > >> 5. We couldn’t use ovn-ic routes learning, because VPC requires > > out-of-box > > > >> IP connectivity between subnets, and if connect Route Table LRs between > > > >> AZs, because connecting multiple LRs from different Route Tables would > > > >> learn routes without binding to route table. This requires additional > > > >> isolation at the transit switches level. > > > >> 6. There were some more problems. Here are listed some, which I could > > > >> refresh in my mind. > > > >> > > > >> From our half-of-a-year experience using LR per VPC is very comfortable > > > >> and it looks quite extendable it terms of network features. > > > >> > > > >> Let me know if this makes sense. > > > >> > > > >> > > > > Thanks for the list of the cons! Most of the cons you listed above seem > > to > > > > apply to the AWS model, too, right? It is ok to have different > > requirements > > > > > > Honestly, I can’t imagine these bullets to "AWS model". Maybe only > > ovn-ic, which > > > requires modifications in ovn-ic codebase to support route_tables in > > routes. > > > But this work is already done. > > > > I think the disagreement was mainly from the understanding of the behavior > > of directly connected subnets between different routing tables. I assume my > > understanding was wrong for now :) > > > > > > > than AWS, but we'd better define it clearly and it would be helpful with > > > > some typical use cases that solve specific problems. I have more > > > > information now after your explanation, but still not really clear under > > > > what scenarios would this feature be used. > > > > > > > > @Numan Siddique <numans@ovn.org> @Mark Michelson < mmichels@redhat.com> > > > > could you let me know your thoughts on this, too? Would you see similar > > use > > > > cases in Redhat customers? Or does this design match well with your > > > > potential use cases? I'd like to be careful when implementing something > > > > like this and make sure we are doing it the right way. > > I don't think there is any use case in ovn-k8s or openstack neutron which would > use this feature. > > Sorry. I've not followed the discussion here closely. > > What is the tl;dr. > > @Han Zhou You're fine with this feature and will we be accepting this in OVN ? Yes, after the discussions, I think the requirement is valid and the approach addresses the requirement properly without increasing complexity too much. I had a comment regarding the priority handling. Vladislav mentioned he will update the patch to support similar behavior of AWS that allows users to add routes that overrides directly connected subnet routes. My recommendation is just don't add extra priority to the global routing table, and just let the longer prefix route take precedence. Thanks, Han > > Thanks > Numan > > > > > > > One comment here from my side: > > > by "this design" we should talk about "behaviour similar to AWS/other > > public platforms route tables support". > > > I think that it is a good feature for opensource SDN, which can be futher > > used by opensource cloud platforms like OpenStack to extend routing > > capabilities. > > > > > Agree. > > > > > > > > > > Thanks, > > > > Han > > > > > > 1: > > https://aws.amazon.com/blogs/aws/inspect-subnet-to-subnet-traffic-with-amazon-vpc-more-specific-routing/ > > > 2: https://cloud.google.com/vpc/docs/routes#subnet-routes > > > > > > > > > > >> Han > > > >> > > > >> > > > >> > > > >> I haven't finished reviewing the code yet, but I have one comment > > > >> > > > >> regarding adding 100 to the priority of the global routes. For IPv6, > > the > > > >> priority range from 0 to 120x2=240, so adding 100 is not enough. It > > would > > > >> create overlapping priority ranges, and some table-id specific route > > > >> entries may override the global routes. > > > >> > > > >> > > > >> > > > >> Thanks, I’ll dig into this when you finish review. > > > >> > > > >> > > > >> Let me know if I answered your questions or if you have new ones. > > > >> Again many thanks for your time and digging into this patch series. > > > >> > > > >> 1: > > https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Route_Tables.html > > > >> 2: https://cloud.google.com/vpc/docs/routes#subnet-routes > > > >> 3: https://docs.cloud.croc.ru/en/services/networks/routetables.html > > > >> > > > >> Thanks, > > > >> Han > > > >> > > _______________________________________________ > > dev mailing list > > dev@openvswitch.org > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Hi Han, Numan, I’ve posted a new version of this series [1] and addressed your comments and suggestions. I’ll appreciate if you can take a look on this please and I’d be happy if this can be included in next OVN release. Thanks. 1: https://patchwork.ozlabs.org/project/ovn/cover/20211111191306.6369-1-odivlad@gmail.com/ Regards, Vladislav Odintsov On 20 Oct 2021, at 10:09, Han Zhou <hzhou@ovn.org<mailto:hzhou@ovn.org>> wrote: On Tue, Oct 19, 2021 at 3:21 PM Numan Siddique <numans@ovn.org<mailto:numans@ovn.org>> wrote: > > On Tue, Oct 19, 2021 at 3:28 AM Han Zhou <hzhou@ovn.org<mailto:hzhou@ovn.org>> wrote: > > > > On Mon, Oct 18, 2021 at 6:35 AM Odintsov Vladislav <VlOdintsov@croc.ru<mailto:VlOdintsov@croc.ru>> > > wrote: > > > > > > > > > > > > regards, > > > Vladislav Odintsov > > > > > > > On 16 Oct 2021, at 03:20, Han Zhou <hzhou@ovn.org<mailto:hzhou@ovn.org>> wrote: > > > > > > > > On Fri, Oct 15, 2021 at 2:36 AM Vladislav Odintsov <odivlad@gmail.com<mailto:odivlad@gmail.com>> > > > > wrote: > > > > > > > >> > > > >> > > > >> Regards, > > > >> Vladislav Odintsov > > > >> > > > >> On 15 Oct 2021, at 08:42, Han Zhou <hzhou@ovn.org<mailto:hzhou@ovn.org>> wrote: > > > >> > > > >> On Thu, Oct 14, 2021 at 12:58 AM Vladislav Odintsov <odivlad@gmail.com<mailto:odivlad@gmail.com>> > > > >> wrote: > > > >> > > > >> > > > >> Hi Han, > > > >> > > > >> Thanks for the review. > > > >> > > > >> Regards, > > > >> Vladislav Odintsov > > > >> > > > >> On 14 Oct 2021, at 08:13, Han Zhou <hzhou@ovn.org<mailto:hzhou@ovn.org>> wrote: > > > >> > > > >> > > > >> > > > >> On Tue, Oct 5, 2021 at 1:26 PM Vladislav Odintsov <odivlad@gmail.com<mailto:odivlad@gmail.com>> > > > >> > > > >> wrote: > > > >> > > > >> > > > >> This patch extends Logical Router's routing functionality. > > > >> Now user may create multiple routing tables within a Logical Router > > > >> and assign them to Logical Router Ports. > > > >> > > > >> Traffic coming from Logical Router Port with assigned route_table > > > >> is checked against global routes if any (Logical_Router_Static_Routes > > > >> whith empty route_table field), next against directly connected routes > > > >> > > > >> > > > >> This is not accurate. The "directly connected routes" is NOT after the > > > >> > > > >> global routes. Their priority only depends on the prefix length. > > > >> > > > >> > > > >> and then Logical_Router_Static_Routes with same route_table value as > > > >> in Logical_Router_Port options:route_table field. > > > >> > > > >> A new Logical Router ingress table #10 is added - IN_IP_ROUTING_PRE. > > > >> In this table packets which come from LRPs with configured > > > >> options:route_table field are checked against inport and in OVS > > > >> register 7 unique non-zero value identifying route table is written. > > > >> > > > >> Then in 11th table IN_IP_ROUTING routes which have non-empty > > > >> `route_table` field are added with additional match on reg7 value > > > >> associated with appropriate route_table. > > > >> > > > >> > > > >> Hi Vladislav, > > > >> > > > >> First of all, sorry for the delayed review, and thanks for implementing > > > >> > > > >> this new feature. > > > >> > > > >> > > > >> I have some questions regarding the feature itself. I remember that we > > > >> > > > >> had some discussion earlier for this feature, but it seems I > > misunderstood > > > >> the feature you are implementing here. I thought by multiple routing > > tables > > > >> you were trying to support something like VRF in physical routers, but > > this > > > >> seems to be something different. According to your implementation, > > instead > > > >> of assigning LRPs to different routing tables, a LRP can actually > > > >> participate to both the global routing table and the table with a > > specific > > > >> ID. For ingress, global routes are prefered over other routes; for > > egress > > > >> (i.e. forwarding to the next hop), it doesn't even enforce any table-id > > > >> check, so packet routed by a entry of table-X can go out of a LRP with > > > >> table-id Y. Is my understanding correct about the desired behavior of > > this > > > >> feature? > > > >> > > > >> > > > >> > > > >> Yes, your understanding is correct. > > > >> This is not VRF. At first glance VRF can be done just by using > > LR-per-VRF > > > >> > > > >> without any code modifications (maybe there are corner cases, but it’s > > not > > > >> something I’m currently working on). > > > >> > > > >> I agree VRF can be achieved by just separate LRs. I am trying to > > understand > > > >> why multiple LRs wouldn't satisfy your use case here. > > > >> > > > >> LRP can be optionally assigned to specific routing table name. This > > means > > > >> > > > >> that for LR ingress pipeline packets coming from this specific LRP > > would be > > > >> checked against routes with same route_table value within appropriate > > LR. > > > >> This is some kind of PBR, analog of "ip rule add iif <interface name> > > > >> lookup <id>". > > > >> > > > >> There is one specific use-case, which requires special handling: > > > >> > > > >> directly-connected routes (subnet CIDRs from connected to this LR > > LRPs). > > > >> These routes can’t be added manually by user, though routing between > > LRPs > > > >> belonging to different routing tables is still needed. So, these routes > > > >> should be added to global routing table to override routing for LRPs > > with > > > >> configured routing tables. If for some reason user wants to prohibit IP > > > >> connectivity to any LRP (honestly, I can’t imagine, why), it can be > > done by > > > >> utilising OVN PBR with drop action or a route with "discard" nexthop. > > But > > > >> I’d think in this case whether this LRP is needed in this LR. > > > >> > > > >> Also, if user wants to create routes, which apply for all LRPs from all > > > >> > > > >> routing tables, it can be done by adding new entries to global routing > > > >> table. > > > >> > > > >> And the third reason to have global routing table - a fully > > > >> > > > >> backward-compatible solution. All users, who don’t need multiple > > routing > > > >> tables support just continue using static routing in the same manner > > > >> without any changes. > > > >> > > > >> > > > >> > > > >> If this is correct, it doesn't seem to be a common/standard requirement > > > >> > > > >> (or please educate me if I am wrong). Could you explain a little more > > about > > > >> the actual use cases: what kind of real world problems need to be > > solved by > > > >> this feature or how are you going to use this. For example, why would a > > > >> port need to participate in both routing tables? It looks like what you > > > >> really need is policy routing instead of multiple isolated routing > > tables. > > > >> I understand that you already use policy routing to implement ACLs, so > > it > > > >> is not convenient to combine other policies (e.g. inport based routing) > > > >> into the policy routing stage. If that's the case, would it be more > > generic > > > >> to support multiple policy routing stages? My concern to the current > > > >> approach is that it is implemented for a very special use case. It > > makes > > > >> the code more complex but when there is a slightly different > > requirement in > > > >> the future it becomes insufficient. I am thinking that policy routing > > seems > > > >> more flexible and has more potential to be made more generic. Maybe I > > will > > > >> have a better understanding when I hear more detailed use cases and > > > >> considerations from you. > > > >> > > > >> > > > >> > > > >> I can't agree here in it’s uncommon requirement. > > > >> This implementation was inspired by AWS Route Tables feature [1]: > > within > > > >> > > > >> a VPC (LR in terms of OVN) user may create multiple routing tables and > > > >> assign them to different subnets (LRPs) in multiple availability zones. > > > >> Auto-generated directly-connected routes from LRPs CIDRs are working > > in the > > > >> same manner as they do in AWS - apply to all subnets regardless of > > their > > > >> association to routing table. GCP has similar behaviour: [2], Azure, I > > > >> guess, too. > > > >> > > > >> Our public cloud (CROC Cloud Platform) supports AWS behaviour [3], so I > > > >> > > > >> primarily was oriented on it. Internally we already use this feature > > and > > > >> currently it fits our use-case, but again I can't say it is specific. > > > >> > > > >> If it is for AWS/GCP alike routing features, then from what I > > understand > > > >> what's implemented in this patch is a little different. To implement a > > VPC > > > >> model in OVN, I think it is ok to have multiple LRs instead of only > > one LR > > > >> for each VPC. So naturally I would use a LR to map to AWS's routing > > table. > > > >> In AWS's document (the link you provided), it says: > > > >> > > > >> "A subnet can only be associated with one route table at a time" > > > >> > > > >> So basically in AWS a subnet is either attached to the default/main > > route > > > >> table or a custom table, but not both, right? However, in your use > > case, a > > > >> LRP (maps to a subnet) attaches to both "main" and a custom table, > > which > > > >> seems not common to me. Or did I miss something? > > > >> > > > >> > > > >> That’s true about AWS, but there is still a bit not accurate about OVN. > > > >> Global routing table in OVN terms is not that AWS main route table is. > > > >> Main route table is just a configuration hint for users for implicit > > route > > > >> tables association with subnets. > > > >> Implicitly-associated via main routing table subnets routing functions > > the > > > >> same manner as a normal explicit route_table-subnet association. > > > >> > > > >> Global routing table in OVN is just a list of routes with higher > > priority > > > >> than routes with configured "route_table". > > > >> > > > >> I do not offer to configure both tables at the same time. But it is > > > >> _possible_ to do if required for some reason (for instance to configure > > > >> some service chaining or just internal VPC services like > > metadata/another > > > >> internal APIs, access to another services). > > > >> Normally, if we talk about AWS Route Table to OVN, it is mostly > > one-to-one > > > >> mapping, except "Local route": > > > >> > > > > > > > >> Example: > > > >> AWS Route Table rtb-xxx: > > > >> 172.31.0.0/16<http://172.31.0.0/16>: local route (VPC CIDR for subnets) > > > >> 0.0.0.0/0<http://0.0.0.0/0>: igw-XXX (internet gateway) > > > >> > > > >> AWS Route Table rtb-yyy: > > > >> 172.31.0.0/16<http://172.31.0.0/16>: local route (VPC CIDR for subnets) > > > >> 5.5.5.5/32<http://5.5.5.5/32>: instance-xxx > > > >> > > > >> This maps to OVN configuration (for one LR with one subnet > > 172.31.0.0/24<http://172.31.0.0/24>): > > > >> > > > >> implicit route (not present in logical router static routes table): > > > >> 172.31.0.0/24<http://172.31.0.0/24>: LRP-subnet-xxx - has highest priority over other route > > > >> table-oriented routes and can be threat as placed in global routing > > table > > > >> > > > > > > > > What do you mean by highest priority here? It all depends on the prefix > > > > length, right? If you have a static route entry that says: > > 172.31.0.0./25 > > > > -> xyz, then this route will have higher priority than the directly > > > > connected subnet. > > > > > > > > > > Yes, it is true, but only for routes within “global” routing table. I > > left this to save previous behaviour. > > > It is impossible to create a static route within any non-global routing > > table which overrides directly connected routes, > > > because priority for “global” routes is added by 100 (though, it should > > add >2*128 to support same behavior for IPv6 as you already pointed). > > > > > > > > > > >> > > > >> Normal static routes: > > > >> ip_prefix: 0.0.0.0/0<http://0.0.0.0/0>, nexthop: <IP for edge LR’s LRP>, route_table: > > > >> rtb-xxx > > > >> ip_prefix: 5.5.5.5/32<http://5.5.5.5/32>, nexthop: <IP of some LSP, which belongs to > > > >> VM/container via which route is built>, route_table: rtb-yyy > > > >> > > > >> I guess, I understood the reason for misunderstanding: the global > > routing > > > >> table, which I referred earlier is a routing table which has no value > > in > > > >> "route_table" field and directly-connected routes at the same time. > > Latter > > > >> have no records in logical router static routes, but I still referred > > them > > > >> as a routes from "global routing table". I can think about terminology > > > >> here, if it’s a root cause for misunderstanding. What do you think? > > > >> > > > > > > > > Thanks for explaining. In fact I understand what you mean about "global > > > > routing table", but I think you are implementing something different > > from > > > > what AWS provides, primarily because you use a single LR instead of LR > > per > > > > routing table. I understand there are reasons why you want to do it this > > > > way, but here I can think of some challenges of your approach. For > > example, > > > > in AWS we could do: > > > > > > > > route table main: > > > > subnet S0: > > > > 172.31.0.0/24<http://172.31.0.0/24> > > > > routes: > > > > 172.31.0.0/16<http://172.31.0.0/16>: local > > > > > > > > route table R1: > > > > subnet S1: > > > > 172.31.1.0/24<http://172.31.1.0/24> > > > > routes: > > > > 172.31.0.0/16<http://172.31.0.0/16>: local > > > > 172.31.0.0/24<http://172.31.0.0/24>: <some FW/GW> > > > > > > Wow. It’s a brand-new behaviour for AWS, about which I was not aware, as > > it appeared 1.5 months ago [1]. > > > > > > Previously, when I was writing this patch series in AWS such static route > > was impossible to add. > > > You couldn’t add routes, which are the same or more specific than VPC > > CIDR block. > > > > > > Though in GCP you can’t create same or more specific than subnet route > > [2]. > > > I think I can try to update my patch to support AWS approach here. > > > > Would it be simpler just don't make the global routes higher priority? We > > could make it clear to the user that if they add a route in any route table > > that is overlapping with any directly connected subnets, the longer prefix > > would take precedence. Would it be more flexible to user? > > > > > > > > But all this is relevant to single LR per VPC (not per route table > > because of described earlier blocker). > > > Do you think it is reasonable with such inputs? > > > > > > > > > > > Packet from S1 to S0 will go to FW/GW, where it may be > > dropped/forwarded to > > > > S0/or redirected to something outside of AWS ... > > > > > > > > While in your approach with OVN, both subnets will be under the same > > > > logical router, and with different table IDs assigned to the LRPs and > > > > routes. But when a VM under S1 sends a packet to S0, it will go to the > > > > destination directly because you are prioritizing the direct-connection > > > > routes and they are under the same global route table, and there is no > > way > > > > to force it to go through some FW/GW from the custom route table. You > > can > > > > add a route to achieve this in the global table, because in your design > > the > > > > global routes are still applicable to all LRPs and has higher priority, > > but > > > > it would impact all the LRPs/subnets, while in the AWS design above it > > is > > > > supposed to affect subnets under R1 only. > > > > > > > > In addition, for my understanding, S0 and S1 in AWS cannot communicate > > > > directly, unless there are some specific routes on both route tables to > > > > route them to some gateway. In other words, there are some isolations > > > > between route tables. However, in your design with OVN, it is more of > > > > > > S0 and S1 can communicate with each other being in different route tables > > without any specific setup, > > > the default unremovable “Local route” ensures this. > > > User only can override "local" route with only existing subnet’s (more > > specific than "local route") CIDR > > > to send traffic from S0 to S1 through some specific appliance(s) in S2. > > > But basic IP connectivity always remains. Regardless of route tables > > subnet associations. > > > So I’m still sure that one LR per VPC suites this scenario better, > > > as I can’t imagine why to place subnets, which don’t need connectivity in > > one VPC. > > > > > Maybe I was wrong. I am not really familiar with AWS and I wanted to try > > this myself but didn't get the time :( > > If default connectivity for subnets under different route tables is a > > requirement (instead of the contrary), then I'd agree that one LR per VPC > > seems better, because it is more like policy routing instead of for > > isolation. > > > > > > policy routing without isolation at all. I am not saying that your > > design > > > > is wrong, but just saying it implements very different behavior than > > AWS, > > > > and I am trying to understand your real use cases to have a better > > > > understanding of the feature you implemented. (You explained that you > > just > > > > wanted the AWS behavior, but it looks not exactly the case). > > > > > > > > > > Will fix places where the behaviour has difference with AWS. > > > And thanks you for pointing such places. > > > > > Well, I think it doesn't have to be all the same as AWS, but yes if AWS > > design makes more sense. > > > > > > > > > >> > > > >> > > > >> In my opinion having this feature to be implemented using PBR is less > > > >> > > > >> convenient and native for users, who are familiar with behaviour for > > > >> mentioned above public cloud platforms, because configuring routes > > should > > > >> be done in routes section. And adding route table property seems > > native in > > > >> this route, not in PBR. Moreover, I _think_ using > > > >> Logical_Router_Static_Route to extend this feature for > > OVN-Interconnection > > > >> becomes quite easy comparing to PBR (though, I didn’t try the latter). > > > >> > > > >> I agree if it is just AWS -like requirement, PBR is less convenient. > > > >> > > > >> I am trying to understand if it can be achieved with separate LRs. If > > not, > > > >> what's special about the requirement, and is the current approach > > providing > > > >> a solution common enough so that more use cases can also benefit from? > > > >> Could you clarify a little more? Thanks again. > > > >> > > > >> That was our initial approach - to use separate LRs for each Route > > Table. > > > >> We rejected that solution because met some difficulties and blocker. > > See > > > >> below: > > > >> > > > >> Brief topology description if using LRs per Route Table: > > > >> Imagine 2 subnets in VPC in 1st AZ, one in another AZ. Each subnet in > > it’s > > > >> own Route Table (LR). > > > >> All subnets must have IP connectivity, so we have to somehow > > interconnect > > > >> these Route Table LRs. > > > >> > > > > > > > > That's exactly the same case for AWS, right? If you want direct > > > > connectivity, you would just put them under the same route table in AWS > > (or > > > > same LR in OVN), right? > > > > > > > > > > I’m sorry, my example was not good enough. > > > Let’s remove AZ’s from that case as they’re not relevant to discussion. > > > So, we’ve got 2 subnets S0 and S1, and suppose, they’ve got different > > route tables assigned to them. > > > If we place them in a single LR, assign different route tables, they > > would have connectivity "out-of-box". > > > Same in AWS: because "local route" is added automatically, can’t be > > removed or modified and ensures IP connectivity between all VPC subnets. > > > > > Ok, thanks for explaining. > > > > > > > > > >> > > > >> [BLOCKER] It is impossible to place route in route table 1 via VM from > > > >> subnet assiciated to route table 2 if using per-RTB LR. Because in > > RTB-2 LR > > > >> we have to add route from RTB 1 and this breaks route table isolation. > > > >> Route Table 2 LR will start looking up routes and there could be routes > > > >> from another route tables. This breaks the idea of having LR per Route > > > >> Table completely. Here we rejected this solution and moved to adding > > > >> support for route tables in OVN. > > > >> > > > >> > > > > Would it be just the same problem in AWS? Why would a user route a > > subnet > > > > of RTB-1 to a VM under RTB-2, and at the same time want route table > > > > isolation? > > > > With the single LR solution in OVN, you would not get the isolation > > between > > > > the tables, because 1) all LRPs still share the global routing table, 2) > > > > for output to nexthop there is no distinction between the output > > interfaces. > > > > > > > > > > No, such problem does not exist. > > > By route tables "isolation" I meant that routing rules configured in one > > routing table must not affect routing in another one. > > > In this scenario user may want to route traffic to internet from S0 > > (RTB-0) through instance in S1 (RTB-1). > > > For instance it can be a gateway service (FW, WAF, IPS, etc) in S1, which > > provides secure access for clients from VPC to the internet with NAT. > > > So, configuration should be: > > > - 0.0.0.0/0<http://0.0.0.0/0> via GW-INSTANCE in RTB-0 > > > - 0.0.0.0/0<http://0.0.0.0/0> via edge LR in RTB-1 > > > So, instances from all subnets with rtb-0 assigned route table will be > > routed to internet through GW-INSTANCE, which can analyse/drop/do whatever > > things with traffic. > > > This is a small example to show how different route tables can be used in > > a single VPC. > > > With multiple OVN LRs per VPC in AZ such topology can’t be created, see: > > > In LR for RTB-0 we can add route 0.0.0.0/0<http://0.0.0.0/0> via <LR0-to-LR1 link IP>; > > > bun in LR for RTB-1 we can only add route to internet through either > > edge-LR or GW-INSTANCE. Not both. So, this case can’t be implemented > > currently in OVN. > > > > > Thanks for the example. It helps understanding your requirements. Why I > > believe this can be achieved by current OVN with the policy routing > > feature, I understand as you mentioned earlier you need policy routing for > > ACL purpose. I also agree that multi-stage policy routing would seem less > > user-friendly (and maybe also harder to implement), so I support the > > multi-table feature for the purpose of providing extra policy routing > > capability. > > > > While I still need to take a closer look at the code, I believe it would be > > helpful to add such example use cases in the documentation for users to > > better understand the feature. > > > > > Hope this helps to understand my problem. > > > > > > > > > > >> But some more cons: > > > >> > > > >> 1. complex network topology. Interconnecting all LRs even with some > > > >> transit switch is harder than having one LR and all VPC-related > > > >> configuration is done in one place (in AZ). > > > >> 2. Directly-connected routes. In case we have multiple Route Table > > LRs, we > > > >> have to add route for each subnet from another LR. In case one LR per > > VPC > > > >> all such routes are installed automatically and learnt via ovn-ic. > > > >> 3. PBR, LBs. It’s much easier to implement PBR and configuring Load > > > >> Balancers in one LR, than in multiple. > > > >> 4. There could be very many LRs, LRPs, LSPs (datapaths in SB) - excess > > > >> database records, huge database growth. > > > >> 5. Extra client code complexity (updating routes required configuring > > > >> routes in many LRs on different availibility zones); > > > >> 5. We couldn’t use ovn-ic routes learning, because VPC requires > > out-of-box > > > >> IP connectivity between subnets, and if connect Route Table LRs between > > > >> AZs, because connecting multiple LRs from different Route Tables would > > > >> learn routes without binding to route table. This requires additional > > > >> isolation at the transit switches level. > > > >> 6. There were some more problems. Here are listed some, which I could > > > >> refresh in my mind. > > > >> > > > >> From our half-of-a-year experience using LR per VPC is very comfortable > > > >> and it looks quite extendable it terms of network features. > > > >> > > > >> Let me know if this makes sense. > > > >> > > > >> > > > > Thanks for the list of the cons! Most of the cons you listed above seem > > to > > > > apply to the AWS model, too, right? It is ok to have different > > requirements > > > > > > Honestly, I can’t imagine these bullets to "AWS model". Maybe only > > ovn-ic, which > > > requires modifications in ovn-ic codebase to support route_tables in > > routes. > > > But this work is already done. > > > > I think the disagreement was mainly from the understanding of the behavior > > of directly connected subnets between different routing tables. I assume my > > understanding was wrong for now :) > > > > > > > than AWS, but we'd better define it clearly and it would be helpful with > > > > some typical use cases that solve specific problems. I have more > > > > information now after your explanation, but still not really clear under > > > > what scenarios would this feature be used. > > > > > > > > @Numan Siddique <numans@ovn.org<mailto:numans@ovn.org>> @Mark Michelson <mmichels@redhat.com<mailto:mmichels@redhat.com>> > > > > could you let me know your thoughts on this, too? Would you see similar > > use > > > > cases in Redhat customers? Or does this design match well with your > > > > potential use cases? I'd like to be careful when implementing something > > > > like this and make sure we are doing it the right way. > > I don't think there is any use case in ovn-k8s or openstack neutron which would > use this feature. > > Sorry. I've not followed the discussion here closely. > > What is the tl;dr. > > @Han Zhou You're fine with this feature and will we be accepting this in OVN ? Yes, after the discussions, I think the requirement is valid and the approach addresses the requirement properly without increasing complexity too much. I had a comment regarding the priority handling. Vladislav mentioned he will update the patch to support similar behavior of AWS that allows users to add routes that overrides directly connected subnet routes. My recommendation is just don't add extra priority to the global routing table, and just let the longer prefix route take precedence. Thanks, Han > > Thanks > Numan > > > > > > > One comment here from my side: > > > by "this design" we should talk about "behaviour similar to AWS/other > > public platforms route tables support". > > > I think that it is a good feature for opensource SDN, which can be futher > > used by opensource cloud platforms like OpenStack to extend routing > > capabilities. > > > > > Agree. > > > > > > > > > > Thanks, > > > > Han > > > > > > 1: > > https://aws.amazon.com/blogs/aws/inspect-subnet-to-subnet-traffic-with-amazon-vpc-more-specific-routing/ > > > 2: https://cloud.google.com/vpc/docs/routes#subnet-routes > > > > > > > > > > >> Han > > > >> > > > >> > > > >> > > > >> I haven't finished reviewing the code yet, but I have one comment > > > >> > > > >> regarding adding 100 to the priority of the global routes. For IPv6, > > the > > > >> priority range from 0 to 120x2=240, so adding 100 is not enough. It > > would > > > >> create overlapping priority ranges, and some table-id specific route > > > >> entries may override the global routes. > > > >> > > > >> > > > >> > > > >> Thanks, I’ll dig into this when you finish review. > > > >> > > > >> > > > >> Let me know if I answered your questions or if you have new ones. > > > >> Again many thanks for your time and digging into this patch series. > > > >> > > > >> 1: > > https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Route_Tables.html > > > >> 2: https://cloud.google.com/vpc/docs/routes#subnet-routes > > > >> 3: https://docs.cloud.croc.ru/en/services/networks/routetables.html > > > >> > > > >> Thanks, > > > >> Han > > > >> > > _______________________________________________ > > dev mailing list > > dev@openvswitch.org<mailto:dev@openvswitch.org> > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
On Thu, Nov 11, 2021 at 2:18 PM Odintsov Vladislav <VlOdintsov@croc.ru> wrote: > > Hi Han, Numan, > > I’ve posted a new version of this series [1] and addressed your comments and suggestions. > I’ll appreciate if you can take a look on this please and I’d be happy if this can be included in next OVN release. > > Thanks. Thanks Vladislav, I'll take a look at the patches next week. Numan > > 1: https://patchwork.ozlabs.org/project/ovn/cover/20211111191306.6369-1-odivlad@gmail.com/ > > Regards, > Vladislav Odintsov > > On 20 Oct 2021, at 10:09, Han Zhou <hzhou@ovn.org<mailto:hzhou@ovn.org>> wrote: > > > > > On Tue, Oct 19, 2021 at 3:21 PM Numan Siddique <numans@ovn.org<mailto:numans@ovn.org>> wrote: > > > > On Tue, Oct 19, 2021 at 3:28 AM Han Zhou <hzhou@ovn.org<mailto:hzhou@ovn.org>> wrote: > > > > > > On Mon, Oct 18, 2021 at 6:35 AM Odintsov Vladislav <VlOdintsov@croc.ru<mailto:VlOdintsov@croc.ru>> > > > wrote: > > > > > > > > > > > > > > > > regards, > > > > Vladislav Odintsov > > > > > > > > > On 16 Oct 2021, at 03:20, Han Zhou <hzhou@ovn.org<mailto:hzhou@ovn.org>> wrote: > > > > > > > > > > On Fri, Oct 15, 2021 at 2:36 AM Vladislav Odintsov <odivlad@gmail.com<mailto:odivlad@gmail.com>> > > > > > wrote: > > > > > > > > > >> > > > > >> > > > > >> Regards, > > > > >> Vladislav Odintsov > > > > >> > > > > >> On 15 Oct 2021, at 08:42, Han Zhou <hzhou@ovn.org<mailto:hzhou@ovn.org>> wrote: > > > > >> > > > > >> On Thu, Oct 14, 2021 at 12:58 AM Vladislav Odintsov <odivlad@gmail.com<mailto:odivlad@gmail.com>> > > > > >> wrote: > > > > >> > > > > >> > > > > >> Hi Han, > > > > >> > > > > >> Thanks for the review. > > > > >> > > > > >> Regards, > > > > >> Vladislav Odintsov > > > > >> > > > > >> On 14 Oct 2021, at 08:13, Han Zhou <hzhou@ovn.org<mailto:hzhou@ovn.org>> wrote: > > > > >> > > > > >> > > > > >> > > > > >> On Tue, Oct 5, 2021 at 1:26 PM Vladislav Odintsov <odivlad@gmail.com<mailto:odivlad@gmail.com>> > > > > >> > > > > >> wrote: > > > > >> > > > > >> > > > > >> This patch extends Logical Router's routing functionality. > > > > >> Now user may create multiple routing tables within a Logical Router > > > > >> and assign them to Logical Router Ports. > > > > >> > > > > >> Traffic coming from Logical Router Port with assigned route_table > > > > >> is checked against global routes if any (Logical_Router_Static_Routes > > > > >> whith empty route_table field), next against directly connected routes > > > > >> > > > > >> > > > > >> This is not accurate. The "directly connected routes" is NOT after the > > > > >> > > > > >> global routes. Their priority only depends on the prefix length. > > > > >> > > > > >> > > > > >> and then Logical_Router_Static_Routes with same route_table value as > > > > >> in Logical_Router_Port options:route_table field. > > > > >> > > > > >> A new Logical Router ingress table #10 is added - IN_IP_ROUTING_PRE. > > > > >> In this table packets which come from LRPs with configured > > > > >> options:route_table field are checked against inport and in OVS > > > > >> register 7 unique non-zero value identifying route table is written. > > > > >> > > > > >> Then in 11th table IN_IP_ROUTING routes which have non-empty > > > > >> `route_table` field are added with additional match on reg7 value > > > > >> associated with appropriate route_table. > > > > >> > > > > >> > > > > >> Hi Vladislav, > > > > >> > > > > >> First of all, sorry for the delayed review, and thanks for implementing > > > > >> > > > > >> this new feature. > > > > >> > > > > >> > > > > >> I have some questions regarding the feature itself. I remember that we > > > > >> > > > > >> had some discussion earlier for this feature, but it seems I > > > misunderstood > > > > >> the feature you are implementing here. I thought by multiple routing > > > tables > > > > >> you were trying to support something like VRF in physical routers, but > > > this > > > > >> seems to be something different. According to your implementation, > > > instead > > > > >> of assigning LRPs to different routing tables, a LRP can actually > > > > >> participate to both the global routing table and the table with a > > > specific > > > > >> ID. For ingress, global routes are prefered over other routes; for > > > egress > > > > >> (i.e. forwarding to the next hop), it doesn't even enforce any table-id > > > > >> check, so packet routed by a entry of table-X can go out of a LRP with > > > > >> table-id Y. Is my understanding correct about the desired behavior of > > > this > > > > >> feature? > > > > >> > > > > >> > > > > >> > > > > >> Yes, your understanding is correct. > > > > >> This is not VRF. At first glance VRF can be done just by using > > > LR-per-VRF > > > > >> > > > > >> without any code modifications (maybe there are corner cases, but it’s > > > not > > > > >> something I’m currently working on). > > > > >> > > > > >> I agree VRF can be achieved by just separate LRs. I am trying to > > > understand > > > > >> why multiple LRs wouldn't satisfy your use case here. > > > > >> > > > > >> LRP can be optionally assigned to specific routing table name. This > > > means > > > > >> > > > > >> that for LR ingress pipeline packets coming from this specific LRP > > > would be > > > > >> checked against routes with same route_table value within appropriate > > > LR. > > > > >> This is some kind of PBR, analog of "ip rule add iif <interface name> > > > > >> lookup <id>". > > > > >> > > > > >> There is one specific use-case, which requires special handling: > > > > >> > > > > >> directly-connected routes (subnet CIDRs from connected to this LR > > > LRPs). > > > > >> These routes can’t be added manually by user, though routing between > > > LRPs > > > > >> belonging to different routing tables is still needed. So, these routes > > > > >> should be added to global routing table to override routing for LRPs > > > with > > > > >> configured routing tables. If for some reason user wants to prohibit IP > > > > >> connectivity to any LRP (honestly, I can’t imagine, why), it can be > > > done by > > > > >> utilising OVN PBR with drop action or a route with "discard" nexthop. > > > But > > > > >> I’d think in this case whether this LRP is needed in this LR. > > > > >> > > > > >> Also, if user wants to create routes, which apply for all LRPs from all > > > > >> > > > > >> routing tables, it can be done by adding new entries to global routing > > > > >> table. > > > > >> > > > > >> And the third reason to have global routing table - a fully > > > > >> > > > > >> backward-compatible solution. All users, who don’t need multiple > > > routing > > > > >> tables support just continue using static routing in the same manner > > > > >> without any changes. > > > > >> > > > > >> > > > > >> > > > > >> If this is correct, it doesn't seem to be a common/standard requirement > > > > >> > > > > >> (or please educate me if I am wrong). Could you explain a little more > > > about > > > > >> the actual use cases: what kind of real world problems need to be > > > solved by > > > > >> this feature or how are you going to use this. For example, why would a > > > > >> port need to participate in both routing tables? It looks like what you > > > > >> really need is policy routing instead of multiple isolated routing > > > tables. > > > > >> I understand that you already use policy routing to implement ACLs, so > > > it > > > > >> is not convenient to combine other policies (e.g. inport based routing) > > > > >> into the policy routing stage. If that's the case, would it be more > > > generic > > > > >> to support multiple policy routing stages? My concern to the current > > > > >> approach is that it is implemented for a very special use case. It > > > makes > > > > >> the code more complex but when there is a slightly different > > > requirement in > > > > >> the future it becomes insufficient. I am thinking that policy routing > > > seems > > > > >> more flexible and has more potential to be made more generic. Maybe I > > > will > > > > >> have a better understanding when I hear more detailed use cases and > > > > >> considerations from you. > > > > >> > > > > >> > > > > >> > > > > >> I can't agree here in it’s uncommon requirement. > > > > >> This implementation was inspired by AWS Route Tables feature [1]: > > > within > > > > >> > > > > >> a VPC (LR in terms of OVN) user may create multiple routing tables and > > > > >> assign them to different subnets (LRPs) in multiple availability zones. > > > > >> Auto-generated directly-connected routes from LRPs CIDRs are working > > > in the > > > > >> same manner as they do in AWS - apply to all subnets regardless of > > > their > > > > >> association to routing table. GCP has similar behaviour: [2], Azure, I > > > > >> guess, too. > > > > >> > > > > >> Our public cloud (CROC Cloud Platform) supports AWS behaviour [3], so I > > > > >> > > > > >> primarily was oriented on it. Internally we already use this feature > > > and > > > > >> currently it fits our use-case, but again I can't say it is specific. > > > > >> > > > > >> If it is for AWS/GCP alike routing features, then from what I > > > understand > > > > >> what's implemented in this patch is a little different. To implement a > > > VPC > > > > >> model in OVN, I think it is ok to have multiple LRs instead of only > > > one LR > > > > >> for each VPC. So naturally I would use a LR to map to AWS's routing > > > table. > > > > >> In AWS's document (the link you provided), it says: > > > > >> > > > > >> "A subnet can only be associated with one route table at a time" > > > > >> > > > > >> So basically in AWS a subnet is either attached to the default/main > > > route > > > > >> table or a custom table, but not both, right? However, in your use > > > case, a > > > > >> LRP (maps to a subnet) attaches to both "main" and a custom table, > > > which > > > > >> seems not common to me. Or did I miss something? > > > > >> > > > > >> > > > > >> That’s true about AWS, but there is still a bit not accurate about OVN. > > > > >> Global routing table in OVN terms is not that AWS main route table is. > > > > >> Main route table is just a configuration hint for users for implicit > > > route > > > > >> tables association with subnets. > > > > >> Implicitly-associated via main routing table subnets routing functions > > > the > > > > >> same manner as a normal explicit route_table-subnet association. > > > > >> > > > > >> Global routing table in OVN is just a list of routes with higher > > > priority > > > > >> than routes with configured "route_table". > > > > >> > > > > >> I do not offer to configure both tables at the same time. But it is > > > > >> _possible_ to do if required for some reason (for instance to configure > > > > >> some service chaining or just internal VPC services like > > > metadata/another > > > > >> internal APIs, access to another services). > > > > >> Normally, if we talk about AWS Route Table to OVN, it is mostly > > > one-to-one > > > > >> mapping, except "Local route": > > > > >> > > > > > > > > > >> Example: > > > > >> AWS Route Table rtb-xxx: > > > > >> 172.31.0.0/16<http://172.31.0.0/16>: local route (VPC CIDR for subnets) > > > > >> 0.0.0.0/0<http://0.0.0.0/0>: igw-XXX (internet gateway) > > > > >> > > > > >> AWS Route Table rtb-yyy: > > > > >> 172.31.0.0/16<http://172.31.0.0/16>: local route (VPC CIDR for subnets) > > > > >> 5.5.5.5/32<http://5.5.5.5/32>: instance-xxx > > > > >> > > > > >> This maps to OVN configuration (for one LR with one subnet > > > 172.31.0.0/24<http://172.31.0.0/24>): > > > > >> > > > > >> implicit route (not present in logical router static routes table): > > > > >> 172.31.0.0/24<http://172.31.0.0/24>: LRP-subnet-xxx - has highest priority over other route > > > > >> table-oriented routes and can be threat as placed in global routing > > > table > > > > >> > > > > > > > > > > What do you mean by highest priority here? It all depends on the prefix > > > > > length, right? If you have a static route entry that says: > > > 172.31.0.0./25 > > > > > -> xyz, then this route will have higher priority than the directly > > > > > connected subnet. > > > > > > > > > > > > > Yes, it is true, but only for routes within “global” routing table. I > > > left this to save previous behaviour. > > > > It is impossible to create a static route within any non-global routing > > > table which overrides directly connected routes, > > > > because priority for “global” routes is added by 100 (though, it should > > > add >2*128 to support same behavior for IPv6 as you already pointed). > > > > > > > > > > > > > >> > > > > >> Normal static routes: > > > > >> ip_prefix: 0.0.0.0/0<http://0.0.0.0/0>, nexthop: <IP for edge LR’s LRP>, route_table: > > > > >> rtb-xxx > > > > >> ip_prefix: 5.5.5.5/32<http://5.5.5.5/32>, nexthop: <IP of some LSP, which belongs to > > > > >> VM/container via which route is built>, route_table: rtb-yyy > > > > >> > > > > >> I guess, I understood the reason for misunderstanding: the global > > > routing > > > > >> table, which I referred earlier is a routing table which has no value > > > in > > > > >> "route_table" field and directly-connected routes at the same time. > > > Latter > > > > >> have no records in logical router static routes, but I still referred > > > them > > > > >> as a routes from "global routing table". I can think about terminology > > > > >> here, if it’s a root cause for misunderstanding. What do you think? > > > > >> > > > > > > > > > > Thanks for explaining. In fact I understand what you mean about "global > > > > > routing table", but I think you are implementing something different > > > from > > > > > what AWS provides, primarily because you use a single LR instead of LR > > > per > > > > > routing table. I understand there are reasons why you want to do it this > > > > > way, but here I can think of some challenges of your approach. For > > > example, > > > > > in AWS we could do: > > > > > > > > > > route table main: > > > > > subnet S0: > > > > > 172.31.0.0/24<http://172.31.0.0/24> > > > > > routes: > > > > > 172.31.0.0/16<http://172.31.0.0/16>: local > > > > > > > > > > route table R1: > > > > > subnet S1: > > > > > 172.31.1.0/24<http://172.31.1.0/24> > > > > > routes: > > > > > 172.31.0.0/16<http://172.31.0.0/16>: local > > > > > 172.31.0.0/24<http://172.31.0.0/24>: <some FW/GW> > > > > > > > > Wow. It’s a brand-new behaviour for AWS, about which I was not aware, as > > > it appeared 1.5 months ago [1]. > > > > > > > > Previously, when I was writing this patch series in AWS such static route > > > was impossible to add. > > > > You couldn’t add routes, which are the same or more specific than VPC > > > CIDR block. > > > > > > > > Though in GCP you can’t create same or more specific than subnet route > > > [2]. > > > > I think I can try to update my patch to support AWS approach here. > > > > > > Would it be simpler just don't make the global routes higher priority? We > > > could make it clear to the user that if they add a route in any route table > > > that is overlapping with any directly connected subnets, the longer prefix > > > would take precedence. Would it be more flexible to user? > > > > > > > > > > > But all this is relevant to single LR per VPC (not per route table > > > because of described earlier blocker). > > > > Do you think it is reasonable with such inputs? > > > > > > > > > > > > > > Packet from S1 to S0 will go to FW/GW, where it may be > > > dropped/forwarded to > > > > > S0/or redirected to something outside of AWS ... > > > > > > > > > > While in your approach with OVN, both subnets will be under the same > > > > > logical router, and with different table IDs assigned to the LRPs and > > > > > routes. But when a VM under S1 sends a packet to S0, it will go to the > > > > > destination directly because you are prioritizing the direct-connection > > > > > routes and they are under the same global route table, and there is no > > > way > > > > > to force it to go through some FW/GW from the custom route table. You > > > can > > > > > add a route to achieve this in the global table, because in your design > > > the > > > > > global routes are still applicable to all LRPs and has higher priority, > > > but > > > > > it would impact all the LRPs/subnets, while in the AWS design above it > > > is > > > > > supposed to affect subnets under R1 only. > > > > > > > > > > In addition, for my understanding, S0 and S1 in AWS cannot communicate > > > > > directly, unless there are some specific routes on both route tables to > > > > > route them to some gateway. In other words, there are some isolations > > > > > between route tables. However, in your design with OVN, it is more of > > > > > > > > S0 and S1 can communicate with each other being in different route tables > > > without any specific setup, > > > > the default unremovable “Local route” ensures this. > > > > User only can override "local" route with only existing subnet’s (more > > > specific than "local route") CIDR > > > > to send traffic from S0 to S1 through some specific appliance(s) in S2. > > > > But basic IP connectivity always remains. Regardless of route tables > > > subnet associations. > > > > So I’m still sure that one LR per VPC suites this scenario better, > > > > as I can’t imagine why to place subnets, which don’t need connectivity in > > > one VPC. > > > > > > > Maybe I was wrong. I am not really familiar with AWS and I wanted to try > > > this myself but didn't get the time :( > > > If default connectivity for subnets under different route tables is a > > > requirement (instead of the contrary), then I'd agree that one LR per VPC > > > seems better, because it is more like policy routing instead of for > > > isolation. > > > > > > > > policy routing without isolation at all. I am not saying that your > > > design > > > > > is wrong, but just saying it implements very different behavior than > > > AWS, > > > > > and I am trying to understand your real use cases to have a better > > > > > understanding of the feature you implemented. (You explained that you > > > just > > > > > wanted the AWS behavior, but it looks not exactly the case). > > > > > > > > > > > > > Will fix places where the behaviour has difference with AWS. > > > > And thanks you for pointing such places. > > > > > > > Well, I think it doesn't have to be all the same as AWS, but yes if AWS > > > design makes more sense. > > > > > > > > > > > > >> > > > > >> > > > > >> In my opinion having this feature to be implemented using PBR is less > > > > >> > > > > >> convenient and native for users, who are familiar with behaviour for > > > > >> mentioned above public cloud platforms, because configuring routes > > > should > > > > >> be done in routes section. And adding route table property seems > > > native in > > > > >> this route, not in PBR. Moreover, I _think_ using > > > > >> Logical_Router_Static_Route to extend this feature for > > > OVN-Interconnection > > > > >> becomes quite easy comparing to PBR (though, I didn’t try the latter). > > > > >> > > > > >> I agree if it is just AWS -like requirement, PBR is less convenient. > > > > >> > > > > >> I am trying to understand if it can be achieved with separate LRs. If > > > not, > > > > >> what's special about the requirement, and is the current approach > > > providing > > > > >> a solution common enough so that more use cases can also benefit from? > > > > >> Could you clarify a little more? Thanks again. > > > > >> > > > > >> That was our initial approach - to use separate LRs for each Route > > > Table. > > > > >> We rejected that solution because met some difficulties and blocker. > > > See > > > > >> below: > > > > >> > > > > >> Brief topology description if using LRs per Route Table: > > > > >> Imagine 2 subnets in VPC in 1st AZ, one in another AZ. Each subnet in > > > it’s > > > > >> own Route Table (LR). > > > > >> All subnets must have IP connectivity, so we have to somehow > > > interconnect > > > > >> these Route Table LRs. > > > > >> > > > > > > > > > > That's exactly the same case for AWS, right? If you want direct > > > > > connectivity, you would just put them under the same route table in AWS > > > (or > > > > > same LR in OVN), right? > > > > > > > > > > > > > I’m sorry, my example was not good enough. > > > > Let’s remove AZ’s from that case as they’re not relevant to discussion. > > > > So, we’ve got 2 subnets S0 and S1, and suppose, they’ve got different > > > route tables assigned to them. > > > > If we place them in a single LR, assign different route tables, they > > > would have connectivity "out-of-box". > > > > Same in AWS: because "local route" is added automatically, can’t be > > > removed or modified and ensures IP connectivity between all VPC subnets. > > > > > > > Ok, thanks for explaining. > > > > > > > > > > > > >> > > > > >> [BLOCKER] It is impossible to place route in route table 1 via VM from > > > > >> subnet assiciated to route table 2 if using per-RTB LR. Because in > > > RTB-2 LR > > > > >> we have to add route from RTB 1 and this breaks route table isolation. > > > > >> Route Table 2 LR will start looking up routes and there could be routes > > > > >> from another route tables. This breaks the idea of having LR per Route > > > > >> Table completely. Here we rejected this solution and moved to adding > > > > >> support for route tables in OVN. > > > > >> > > > > >> > > > > > Would it be just the same problem in AWS? Why would a user route a > > > subnet > > > > > of RTB-1 to a VM under RTB-2, and at the same time want route table > > > > > isolation? > > > > > With the single LR solution in OVN, you would not get the isolation > > > between > > > > > the tables, because 1) all LRPs still share the global routing table, 2) > > > > > for output to nexthop there is no distinction between the output > > > interfaces. > > > > > > > > > > > > > No, such problem does not exist. > > > > By route tables "isolation" I meant that routing rules configured in one > > > routing table must not affect routing in another one. > > > > In this scenario user may want to route traffic to internet from S0 > > > (RTB-0) through instance in S1 (RTB-1). > > > > For instance it can be a gateway service (FW, WAF, IPS, etc) in S1, which > > > provides secure access for clients from VPC to the internet with NAT. > > > > So, configuration should be: > > > > - 0.0.0.0/0<http://0.0.0.0/0> via GW-INSTANCE in RTB-0 > > > > - 0.0.0.0/0<http://0.0.0.0/0> via edge LR in RTB-1 > > > > So, instances from all subnets with rtb-0 assigned route table will be > > > routed to internet through GW-INSTANCE, which can analyse/drop/do whatever > > > things with traffic. > > > > This is a small example to show how different route tables can be used in > > > a single VPC. > > > > With multiple OVN LRs per VPC in AZ such topology can’t be created, see: > > > > In LR for RTB-0 we can add route 0.0.0.0/0<http://0.0.0.0/0> via <LR0-to-LR1 link IP>; > > > > bun in LR for RTB-1 we can only add route to internet through either > > > edge-LR or GW-INSTANCE. Not both. So, this case can’t be implemented > > > currently in OVN. > > > > > > > Thanks for the example. It helps understanding your requirements. Why I > > > believe this can be achieved by current OVN with the policy routing > > > feature, I understand as you mentioned earlier you need policy routing for > > > ACL purpose. I also agree that multi-stage policy routing would seem less > > > user-friendly (and maybe also harder to implement), so I support the > > > multi-table feature for the purpose of providing extra policy routing > > > capability. > > > > > > While I still need to take a closer look at the code, I believe it would be > > > helpful to add such example use cases in the documentation for users to > > > better understand the feature. > > > > > > > Hope this helps to understand my problem. > > > > > > > > > > > > > >> But some more cons: > > > > >> > > > > >> 1. complex network topology. Interconnecting all LRs even with some > > > > >> transit switch is harder than having one LR and all VPC-related > > > > >> configuration is done in one place (in AZ). > > > > >> 2. Directly-connected routes. In case we have multiple Route Table > > > LRs, we > > > > >> have to add route for each subnet from another LR. In case one LR per > > > VPC > > > > >> all such routes are installed automatically and learnt via ovn-ic. > > > > >> 3. PBR, LBs. It’s much easier to implement PBR and configuring Load > > > > >> Balancers in one LR, than in multiple. > > > > >> 4. There could be very many LRs, LRPs, LSPs (datapaths in SB) - excess > > > > >> database records, huge database growth. > > > > >> 5. Extra client code complexity (updating routes required configuring > > > > >> routes in many LRs on different availibility zones); > > > > >> 5. We couldn’t use ovn-ic routes learning, because VPC requires > > > out-of-box > > > > >> IP connectivity between subnets, and if connect Route Table LRs between > > > > >> AZs, because connecting multiple LRs from different Route Tables would > > > > >> learn routes without binding to route table. This requires additional > > > > >> isolation at the transit switches level. > > > > >> 6. There were some more problems. Here are listed some, which I could > > > > >> refresh in my mind. > > > > >> > > > > >> From our half-of-a-year experience using LR per VPC is very comfortable > > > > >> and it looks quite extendable it terms of network features. > > > > >> > > > > >> Let me know if this makes sense. > > > > >> > > > > >> > > > > > Thanks for the list of the cons! Most of the cons you listed above seem > > > to > > > > > apply to the AWS model, too, right? It is ok to have different > > > requirements > > > > > > > > Honestly, I can’t imagine these bullets to "AWS model". Maybe only > > > ovn-ic, which > > > > requires modifications in ovn-ic codebase to support route_tables in > > > routes. > > > > But this work is already done. > > > > > > I think the disagreement was mainly from the understanding of the behavior > > > of directly connected subnets between different routing tables. I assume my > > > understanding was wrong for now :) > > > > > > > > > than AWS, but we'd better define it clearly and it would be helpful with > > > > > some typical use cases that solve specific problems. I have more > > > > > information now after your explanation, but still not really clear under > > > > > what scenarios would this feature be used. > > > > > > > > > > @Numan Siddique <numans@ovn.org<mailto:numans@ovn.org>> @Mark Michelson <mmichels@redhat.com<mailto:mmichels@redhat.com>> > > > > > could you let me know your thoughts on this, too? Would you see similar > > > use > > > > > cases in Redhat customers? Or does this design match well with your > > > > > potential use cases? I'd like to be careful when implementing something > > > > > like this and make sure we are doing it the right way. > > > > I don't think there is any use case in ovn-k8s or openstack neutron which would > > use this feature. > > > > Sorry. I've not followed the discussion here closely. > > > > What is the tl;dr. > > > > @Han Zhou You're fine with this feature and will we be accepting this in OVN ? > > Yes, after the discussions, I think the requirement is valid and the approach addresses the requirement properly without increasing complexity too much. > I had a comment regarding the priority handling. Vladislav mentioned he will update the patch to support similar behavior of AWS that allows users to add routes that overrides directly connected subnet routes. My recommendation is just don't add extra priority to the global routing table, and just let the longer prefix route take precedence. > > Thanks, > Han > > > > > Thanks > > Numan > > > > > > > > > > One comment here from my side: > > > > by "this design" we should talk about "behaviour similar to AWS/other > > > public platforms route tables support". > > > > I think that it is a good feature for opensource SDN, which can be futher > > > used by opensource cloud platforms like OpenStack to extend routing > > > capabilities. > > > > > > > Agree. > > > > > > > > > > > > > Thanks, > > > > > Han > > > > > > > > 1: > > > https://aws.amazon.com/blogs/aws/inspect-subnet-to-subnet-traffic-with-amazon-vpc-more-specific-routing/ > > > > 2: https://cloud.google.com/vpc/docs/routes#subnet-routes > > > > > > > > > > > > > >> Han > > > > >> > > > > >> > > > > >> > > > > >> I haven't finished reviewing the code yet, but I have one comment > > > > >> > > > > >> regarding adding 100 to the priority of the global routes. For IPv6, > > > the > > > > >> priority range from 0 to 120x2=240, so adding 100 is not enough. It > > > would > > > > >> create overlapping priority ranges, and some table-id specific route > > > > >> entries may override the global routes. > > > > >> > > > > >> > > > > >> > > > > >> Thanks, I’ll dig into this when you finish review. > > > > >> > > > > >> > > > > >> Let me know if I answered your questions or if you have new ones. > > > > >> Again many thanks for your time and digging into this patch series. > > > > >> > > > > >> 1: > > > https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Route_Tables.html > > > > >> 2: https://cloud.google.com/vpc/docs/routes#subnet-routes > > > > >> 3: https://docs.cloud.croc.ru/en/services/networks/routetables.html > > > > >> > > > > >> Thanks, > > > > >> Han > > > > >> > > > _______________________________________________ > > > dev mailing list > > > dev@openvswitch.org<mailto:dev@openvswitch.org> > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > _______________________________________________ > dev mailing list > dev@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
Thanks Numan, I’ve forgotten to update description according to new changes. A new patch version v8 was submitted: https://patchwork.ozlabs.org/project/ovn/cover/20211113094353.17690-1-odivlad@gmail.com/ Please check it out. Regards, Vladislav Odintsov On 12 Nov 2021, at 23:07, Numan Siddique <numans@ovn.org<mailto:numans@ovn.org>> wrote: On Thu, Nov 11, 2021 at 2:18 PM Odintsov Vladislav <VlOdintsov@croc.ru<mailto:VlOdintsov@croc.ru>> wrote: Hi Han, Numan, I’ve posted a new version of this series [1] and addressed your comments and suggestions. I’ll appreciate if you can take a look on this please and I’d be happy if this can be included in next OVN release. Thanks. Thanks Vladislav, I'll take a look at the patches next week. Numan 1: https://patchwork.ozlabs.org/project/ovn/cover/20211111191306.6369-1-odivlad@gmail.com/ Regards, Vladislav Odintsov On 20 Oct 2021, at 10:09, Han Zhou <hzhou@ovn.org<mailto:hzhou@ovn.org><mailto:hzhou@ovn.org>> wrote: On Tue, Oct 19, 2021 at 3:21 PM Numan Siddique <numans@ovn.org<mailto:numans@ovn.org><mailto:numans@ovn.org>> wrote: On Tue, Oct 19, 2021 at 3:28 AM Han Zhou <hzhou@ovn.org<mailto:hzhou@ovn.org><mailto:hzhou@ovn.org>> wrote: On Mon, Oct 18, 2021 at 6:35 AM Odintsov Vladislav <VlOdintsov@croc.ru<mailto:VlOdintsov@croc.ru><mailto:VlOdintsov@croc.ru>> wrote: regards, Vladislav Odintsov On 16 Oct 2021, at 03:20, Han Zhou <hzhou@ovn.org<mailto:hzhou@ovn.org><mailto:hzhou@ovn.org>> wrote: On Fri, Oct 15, 2021 at 2:36 AM Vladislav Odintsov <odivlad@gmail.com<mailto:odivlad@gmail.com><mailto:odivlad@gmail.com>> wrote: Regards, Vladislav Odintsov On 15 Oct 2021, at 08:42, Han Zhou <hzhou@ovn.org<mailto:hzhou@ovn.org><mailto:hzhou@ovn.org>> wrote: On Thu, Oct 14, 2021 at 12:58 AM Vladislav Odintsov <odivlad@gmail.com<mailto:odivlad@gmail.com><mailto:odivlad@gmail.com>> wrote: Hi Han, Thanks for the review. Regards, Vladislav Odintsov On 14 Oct 2021, at 08:13, Han Zhou <hzhou@ovn.org<mailto:hzhou@ovn.org><mailto:hzhou@ovn.org>> wrote: On Tue, Oct 5, 2021 at 1:26 PM Vladislav Odintsov <odivlad@gmail.com<mailto:odivlad@gmail.com><mailto:odivlad@gmail.com>> wrote: This patch extends Logical Router's routing functionality. Now user may create multiple routing tables within a Logical Router and assign them to Logical Router Ports. Traffic coming from Logical Router Port with assigned route_table is checked against global routes if any (Logical_Router_Static_Routes whith empty route_table field), next against directly connected routes This is not accurate. The "directly connected routes" is NOT after the global routes. Their priority only depends on the prefix length. and then Logical_Router_Static_Routes with same route_table value as in Logical_Router_Port options:route_table field. A new Logical Router ingress table #10 is added - IN_IP_ROUTING_PRE. In this table packets which come from LRPs with configured options:route_table field are checked against inport and in OVS register 7 unique non-zero value identifying route table is written. Then in 11th table IN_IP_ROUTING routes which have non-empty `route_table` field are added with additional match on reg7 value associated with appropriate route_table. Hi Vladislav, First of all, sorry for the delayed review, and thanks for implementing this new feature. I have some questions regarding the feature itself. I remember that we had some discussion earlier for this feature, but it seems I misunderstood the feature you are implementing here. I thought by multiple routing tables you were trying to support something like VRF in physical routers, but this seems to be something different. According to your implementation, instead of assigning LRPs to different routing tables, a LRP can actually participate to both the global routing table and the table with a specific ID. For ingress, global routes are prefered over other routes; for egress (i.e. forwarding to the next hop), it doesn't even enforce any table-id check, so packet routed by a entry of table-X can go out of a LRP with table-id Y. Is my understanding correct about the desired behavior of this feature? Yes, your understanding is correct. This is not VRF. At first glance VRF can be done just by using LR-per-VRF without any code modifications (maybe there are corner cases, but it’s not something I’m currently working on). I agree VRF can be achieved by just separate LRs. I am trying to understand why multiple LRs wouldn't satisfy your use case here. LRP can be optionally assigned to specific routing table name. This means that for LR ingress pipeline packets coming from this specific LRP would be checked against routes with same route_table value within appropriate LR. This is some kind of PBR, analog of "ip rule add iif <interface name> lookup <id>". There is one specific use-case, which requires special handling: directly-connected routes (subnet CIDRs from connected to this LR LRPs). These routes can’t be added manually by user, though routing between LRPs belonging to different routing tables is still needed. So, these routes should be added to global routing table to override routing for LRPs with configured routing tables. If for some reason user wants to prohibit IP connectivity to any LRP (honestly, I can’t imagine, why), it can be done by utilising OVN PBR with drop action or a route with "discard" nexthop. But I’d think in this case whether this LRP is needed in this LR. Also, if user wants to create routes, which apply for all LRPs from all routing tables, it can be done by adding new entries to global routing table. And the third reason to have global routing table - a fully backward-compatible solution. All users, who don’t need multiple routing tables support just continue using static routing in the same manner without any changes. If this is correct, it doesn't seem to be a common/standard requirement (or please educate me if I am wrong). Could you explain a little more about the actual use cases: what kind of real world problems need to be solved by this feature or how are you going to use this. For example, why would a port need to participate in both routing tables? It looks like what you really need is policy routing instead of multiple isolated routing tables. I understand that you already use policy routing to implement ACLs, so it is not convenient to combine other policies (e.g. inport based routing) into the policy routing stage. If that's the case, would it be more generic to support multiple policy routing stages? My concern to the current approach is that it is implemented for a very special use case. It makes the code more complex but when there is a slightly different requirement in the future it becomes insufficient. I am thinking that policy routing seems more flexible and has more potential to be made more generic. Maybe I will have a better understanding when I hear more detailed use cases and considerations from you. I can't agree here in it’s uncommon requirement. This implementation was inspired by AWS Route Tables feature [1]: within a VPC (LR in terms of OVN) user may create multiple routing tables and assign them to different subnets (LRPs) in multiple availability zones. Auto-generated directly-connected routes from LRPs CIDRs are working in the same manner as they do in AWS - apply to all subnets regardless of their association to routing table. GCP has similar behaviour: [2], Azure, I guess, too. Our public cloud (CROC Cloud Platform) supports AWS behaviour [3], so I primarily was oriented on it. Internally we already use this feature and currently it fits our use-case, but again I can't say it is specific. If it is for AWS/GCP alike routing features, then from what I understand what's implemented in this patch is a little different. To implement a VPC model in OVN, I think it is ok to have multiple LRs instead of only one LR for each VPC. So naturally I would use a LR to map to AWS's routing table. In AWS's document (the link you provided), it says: "A subnet can only be associated with one route table at a time" So basically in AWS a subnet is either attached to the default/main route table or a custom table, but not both, right? However, in your use case, a LRP (maps to a subnet) attaches to both "main" and a custom table, which seems not common to me. Or did I miss something? That’s true about AWS, but there is still a bit not accurate about OVN. Global routing table in OVN terms is not that AWS main route table is. Main route table is just a configuration hint for users for implicit route tables association with subnets. Implicitly-associated via main routing table subnets routing functions the same manner as a normal explicit route_table-subnet association. Global routing table in OVN is just a list of routes with higher priority than routes with configured "route_table". I do not offer to configure both tables at the same time. But it is _possible_ to do if required for some reason (for instance to configure some service chaining or just internal VPC services like metadata/another internal APIs, access to another services). Normally, if we talk about AWS Route Table to OVN, it is mostly one-to-one mapping, except "Local route": Example: AWS Route Table rtb-xxx: 172.31.0.0/16<http://172.31.0.0/16>: local route (VPC CIDR for subnets) 0.0.0.0/0<http://0.0.0.0/0>: igw-XXX (internet gateway) AWS Route Table rtb-yyy: 172.31.0.0/16<http://172.31.0.0/16>: local route (VPC CIDR for subnets) 5.5.5.5/32<http://5.5.5.5/32>: instance-xxx This maps to OVN configuration (for one LR with one subnet 172.31.0.0/24<http://172.31.0.0/24>): implicit route (not present in logical router static routes table): 172.31.0.0/24<http://172.31.0.0/24>: LRP-subnet-xxx - has highest priority over other route table-oriented routes and can be threat as placed in global routing table What do you mean by highest priority here? It all depends on the prefix length, right? If you have a static route entry that says: 172.31.0.0./25 -> xyz, then this route will have higher priority than the directly connected subnet. Yes, it is true, but only for routes within “global” routing table. I left this to save previous behaviour. It is impossible to create a static route within any non-global routing table which overrides directly connected routes, because priority for “global” routes is added by 100 (though, it should add >2*128 to support same behavior for IPv6 as you already pointed). Normal static routes: ip_prefix: 0.0.0.0/0<http://0.0.0.0/0>, nexthop: <IP for edge LR’s LRP>, route_table: rtb-xxx ip_prefix: 5.5.5.5/32<http://5.5.5.5/32>, nexthop: <IP of some LSP, which belongs to VM/container via which route is built>, route_table: rtb-yyy I guess, I understood the reason for misunderstanding: the global routing table, which I referred earlier is a routing table which has no value in "route_table" field and directly-connected routes at the same time. Latter have no records in logical router static routes, but I still referred them as a routes from "global routing table". I can think about terminology here, if it’s a root cause for misunderstanding. What do you think? Thanks for explaining. In fact I understand what you mean about "global routing table", but I think you are implementing something different from what AWS provides, primarily because you use a single LR instead of LR per routing table. I understand there are reasons why you want to do it this way, but here I can think of some challenges of your approach. For example, in AWS we could do: route table main: subnet S0: 172.31.0.0/24<http://172.31.0.0/24> routes: 172.31.0.0/16<http://172.31.0.0/16>: local route table R1: subnet S1: 172.31.1.0/24<http://172.31.1.0/24> routes: 172.31.0.0/16<http://172.31.0.0/16>: local 172.31.0.0/24<http://172.31.0.0/24>: <some FW/GW> Wow. It’s a brand-new behaviour for AWS, about which I was not aware, as it appeared 1.5 months ago [1]. Previously, when I was writing this patch series in AWS such static route was impossible to add. You couldn’t add routes, which are the same or more specific than VPC CIDR block. Though in GCP you can’t create same or more specific than subnet route [2]. I think I can try to update my patch to support AWS approach here. Would it be simpler just don't make the global routes higher priority? We could make it clear to the user that if they add a route in any route table that is overlapping with any directly connected subnets, the longer prefix would take precedence. Would it be more flexible to user? But all this is relevant to single LR per VPC (not per route table because of described earlier blocker). Do you think it is reasonable with such inputs? Packet from S1 to S0 will go to FW/GW, where it may be dropped/forwarded to S0/or redirected to something outside of AWS ... While in your approach with OVN, both subnets will be under the same logical router, and with different table IDs assigned to the LRPs and routes. But when a VM under S1 sends a packet to S0, it will go to the destination directly because you are prioritizing the direct-connection routes and they are under the same global route table, and there is no way to force it to go through some FW/GW from the custom route table. You can add a route to achieve this in the global table, because in your design the global routes are still applicable to all LRPs and has higher priority, but it would impact all the LRPs/subnets, while in the AWS design above it is supposed to affect subnets under R1 only. In addition, for my understanding, S0 and S1 in AWS cannot communicate directly, unless there are some specific routes on both route tables to route them to some gateway. In other words, there are some isolations between route tables. However, in your design with OVN, it is more of S0 and S1 can communicate with each other being in different route tables without any specific setup, the default unremovable “Local route” ensures this. User only can override "local" route with only existing subnet’s (more specific than "local route") CIDR to send traffic from S0 to S1 through some specific appliance(s) in S2. But basic IP connectivity always remains. Regardless of route tables subnet associations. So I’m still sure that one LR per VPC suites this scenario better, as I can’t imagine why to place subnets, which don’t need connectivity in one VPC. Maybe I was wrong. I am not really familiar with AWS and I wanted to try this myself but didn't get the time :( If default connectivity for subnets under different route tables is a requirement (instead of the contrary), then I'd agree that one LR per VPC seems better, because it is more like policy routing instead of for isolation. policy routing without isolation at all. I am not saying that your design is wrong, but just saying it implements very different behavior than AWS, and I am trying to understand your real use cases to have a better understanding of the feature you implemented. (You explained that you just wanted the AWS behavior, but it looks not exactly the case). Will fix places where the behaviour has difference with AWS. And thanks you for pointing such places. Well, I think it doesn't have to be all the same as AWS, but yes if AWS design makes more sense. In my opinion having this feature to be implemented using PBR is less convenient and native for users, who are familiar with behaviour for mentioned above public cloud platforms, because configuring routes should be done in routes section. And adding route table property seems native in this route, not in PBR. Moreover, I _think_ using Logical_Router_Static_Route to extend this feature for OVN-Interconnection becomes quite easy comparing to PBR (though, I didn’t try the latter). I agree if it is just AWS -like requirement, PBR is less convenient. I am trying to understand if it can be achieved with separate LRs. If not, what's special about the requirement, and is the current approach providing a solution common enough so that more use cases can also benefit from? Could you clarify a little more? Thanks again. That was our initial approach - to use separate LRs for each Route Table. We rejected that solution because met some difficulties and blocker. See below: Brief topology description if using LRs per Route Table: Imagine 2 subnets in VPC in 1st AZ, one in another AZ. Each subnet in it’s own Route Table (LR). All subnets must have IP connectivity, so we have to somehow interconnect these Route Table LRs. That's exactly the same case for AWS, right? If you want direct connectivity, you would just put them under the same route table in AWS (or same LR in OVN), right? I’m sorry, my example was not good enough. Let’s remove AZ’s from that case as they’re not relevant to discussion. So, we’ve got 2 subnets S0 and S1, and suppose, they’ve got different route tables assigned to them. If we place them in a single LR, assign different route tables, they would have connectivity "out-of-box". Same in AWS: because "local route" is added automatically, can’t be removed or modified and ensures IP connectivity between all VPC subnets. Ok, thanks for explaining. [BLOCKER] It is impossible to place route in route table 1 via VM from subnet assiciated to route table 2 if using per-RTB LR. Because in RTB-2 LR we have to add route from RTB 1 and this breaks route table isolation. Route Table 2 LR will start looking up routes and there could be routes from another route tables. This breaks the idea of having LR per Route Table completely. Here we rejected this solution and moved to adding support for route tables in OVN. Would it be just the same problem in AWS? Why would a user route a subnet of RTB-1 to a VM under RTB-2, and at the same time want route table isolation? With the single LR solution in OVN, you would not get the isolation between the tables, because 1) all LRPs still share the global routing table, 2) for output to nexthop there is no distinction between the output interfaces. No, such problem does not exist. By route tables "isolation" I meant that routing rules configured in one routing table must not affect routing in another one. In this scenario user may want to route traffic to internet from S0 (RTB-0) through instance in S1 (RTB-1). For instance it can be a gateway service (FW, WAF, IPS, etc) in S1, which provides secure access for clients from VPC to the internet with NAT. So, configuration should be: - 0.0.0.0/0<http://0.0.0.0/0> via GW-INSTANCE in RTB-0 - 0.0.0.0/0<http://0.0.0.0/0> via edge LR in RTB-1 So, instances from all subnets with rtb-0 assigned route table will be routed to internet through GW-INSTANCE, which can analyse/drop/do whatever things with traffic. This is a small example to show how different route tables can be used in a single VPC. With multiple OVN LRs per VPC in AZ such topology can’t be created, see: In LR for RTB-0 we can add route 0.0.0.0/0<http://0.0.0.0/0> via <LR0-to-LR1 link IP>; bun in LR for RTB-1 we can only add route to internet through either edge-LR or GW-INSTANCE. Not both. So, this case can’t be implemented currently in OVN. Thanks for the example. It helps understanding your requirements. Why I believe this can be achieved by current OVN with the policy routing feature, I understand as you mentioned earlier you need policy routing for ACL purpose. I also agree that multi-stage policy routing would seem less user-friendly (and maybe also harder to implement), so I support the multi-table feature for the purpose of providing extra policy routing capability. While I still need to take a closer look at the code, I believe it would be helpful to add such example use cases in the documentation for users to better understand the feature. Hope this helps to understand my problem. But some more cons: 1. complex network topology. Interconnecting all LRs even with some transit switch is harder than having one LR and all VPC-related configuration is done in one place (in AZ). 2. Directly-connected routes. In case we have multiple Route Table LRs, we have to add route for each subnet from another LR. In case one LR per VPC all such routes are installed automatically and learnt via ovn-ic. 3. PBR, LBs. It’s much easier to implement PBR and configuring Load Balancers in one LR, than in multiple. 4. There could be very many LRs, LRPs, LSPs (datapaths in SB) - excess database records, huge database growth. 5. Extra client code complexity (updating routes required configuring routes in many LRs on different availibility zones); 5. We couldn’t use ovn-ic routes learning, because VPC requires out-of-box IP connectivity between subnets, and if connect Route Table LRs between AZs, because connecting multiple LRs from different Route Tables would learn routes without binding to route table. This requires additional isolation at the transit switches level. 6. There were some more problems. Here are listed some, which I could refresh in my mind. From our half-of-a-year experience using LR per VPC is very comfortable and it looks quite extendable it terms of network features. Let me know if this makes sense. Thanks for the list of the cons! Most of the cons you listed above seem to apply to the AWS model, too, right? It is ok to have different requirements Honestly, I can’t imagine these bullets to "AWS model". Maybe only ovn-ic, which requires modifications in ovn-ic codebase to support route_tables in routes. But this work is already done. I think the disagreement was mainly from the understanding of the behavior of directly connected subnets between different routing tables. I assume my understanding was wrong for now :) than AWS, but we'd better define it clearly and it would be helpful with some typical use cases that solve specific problems. I have more information now after your explanation, but still not really clear under what scenarios would this feature be used. @Numan Siddique <numans@ovn.org<mailto:numans@ovn.org><mailto:numans@ovn.org>> @Mark Michelson <mmichels@redhat.com<mailto:mmichels@redhat.com><mailto:mmichels@redhat.com>> could you let me know your thoughts on this, too? Would you see similar use cases in Redhat customers? Or does this design match well with your potential use cases? I'd like to be careful when implementing something like this and make sure we are doing it the right way. I don't think there is any use case in ovn-k8s or openstack neutron which would use this feature. Sorry. I've not followed the discussion here closely. What is the tl;dr. @Han Zhou You're fine with this feature and will we be accepting this in OVN ? Yes, after the discussions, I think the requirement is valid and the approach addresses the requirement properly without increasing complexity too much. I had a comment regarding the priority handling. Vladislav mentioned he will update the patch to support similar behavior of AWS that allows users to add routes that overrides directly connected subnet routes. My recommendation is just don't add extra priority to the global routing table, and just let the longer prefix route take precedence. Thanks, Han Thanks Numan One comment here from my side: by "this design" we should talk about "behaviour similar to AWS/other public platforms route tables support". I think that it is a good feature for opensource SDN, which can be futher used by opensource cloud platforms like OpenStack to extend routing capabilities. Agree. Thanks, Han 1: https://aws.amazon.com/blogs/aws/inspect-subnet-to-subnet-traffic-with-amazon-vpc-more-specific-routing/ 2: https://cloud.google.com/vpc/docs/routes#subnet-routes Han I haven't finished reviewing the code yet, but I have one comment regarding adding 100 to the priority of the global routes. For IPv6, the priority range from 0 to 120x2=240, so adding 100 is not enough. It would create overlapping priority ranges, and some table-id specific route entries may override the global routes. Thanks, I’ll dig into this when you finish review. Let me know if I answered your questions or if you have new ones. Again many thanks for your time and digging into this patch series. 1: https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Route_Tables.html 2: https://cloud.google.com/vpc/docs/routes#subnet-routes 3: https://docs.cloud.croc.ru/en/services/networks/routetables.html Thanks, Han
diff --git a/northd/northd.c b/northd/northd.c index 092eca829..6a020cb2e 100644 --- a/northd/northd.c +++ b/northd/northd.c @@ -148,15 +148,16 @@ enum ovn_stage { PIPELINE_STAGE(ROUTER, IN, ECMP_STATEFUL, 7, "lr_in_ecmp_stateful") \ PIPELINE_STAGE(ROUTER, IN, ND_RA_OPTIONS, 8, "lr_in_nd_ra_options") \ PIPELINE_STAGE(ROUTER, IN, ND_RA_RESPONSE, 9, "lr_in_nd_ra_response") \ - PIPELINE_STAGE(ROUTER, IN, IP_ROUTING, 10, "lr_in_ip_routing") \ - PIPELINE_STAGE(ROUTER, IN, IP_ROUTING_ECMP, 11, "lr_in_ip_routing_ecmp") \ - PIPELINE_STAGE(ROUTER, IN, POLICY, 12, "lr_in_policy") \ - PIPELINE_STAGE(ROUTER, IN, POLICY_ECMP, 13, "lr_in_policy_ecmp") \ - PIPELINE_STAGE(ROUTER, IN, ARP_RESOLVE, 14, "lr_in_arp_resolve") \ - PIPELINE_STAGE(ROUTER, IN, CHK_PKT_LEN , 15, "lr_in_chk_pkt_len") \ - PIPELINE_STAGE(ROUTER, IN, LARGER_PKTS, 16, "lr_in_larger_pkts") \ - PIPELINE_STAGE(ROUTER, IN, GW_REDIRECT, 17, "lr_in_gw_redirect") \ - PIPELINE_STAGE(ROUTER, IN, ARP_REQUEST, 18, "lr_in_arp_request") \ + PIPELINE_STAGE(ROUTER, IN, IP_ROUTING_PRE, 10, "lr_in_ip_routing_pre") \ + PIPELINE_STAGE(ROUTER, IN, IP_ROUTING, 11, "lr_in_ip_routing") \ + PIPELINE_STAGE(ROUTER, IN, IP_ROUTING_ECMP, 12, "lr_in_ip_routing_ecmp") \ + PIPELINE_STAGE(ROUTER, IN, POLICY, 13, "lr_in_policy") \ + PIPELINE_STAGE(ROUTER, IN, POLICY_ECMP, 14, "lr_in_policy_ecmp") \ + PIPELINE_STAGE(ROUTER, IN, ARP_RESOLVE, 15, "lr_in_arp_resolve") \ + PIPELINE_STAGE(ROUTER, IN, CHK_PKT_LEN, 16, "lr_in_chk_pkt_len") \ + PIPELINE_STAGE(ROUTER, IN, LARGER_PKTS, 17, "lr_in_larger_pkts") \ + PIPELINE_STAGE(ROUTER, IN, GW_REDIRECT, 18, "lr_in_gw_redirect") \ + PIPELINE_STAGE(ROUTER, IN, ARP_REQUEST, 19, "lr_in_arp_request") \ \ /* Logical router egress stages. */ \ PIPELINE_STAGE(ROUTER, OUT, UNDNAT, 0, "lr_out_undnat") \ @@ -225,6 +226,7 @@ enum ovn_stage { #define REG_NEXT_HOP_IPV6 "xxreg0" #define REG_SRC_IPV4 "reg1" #define REG_SRC_IPV6 "xxreg1" +#define REG_ROUTE_TABLE_ID "reg7" #define REG_ORIG_TP_DPORT_ROUTER "reg9[16..31]" @@ -287,8 +289,9 @@ enum ovn_stage { * | R6 | UNUSED | X | | G | IN_IP_ROUTING)| * | | | R | | 1 | | * +-----+--------------------------+ E | UNUSED | | | - * | R7 | UNUSED | G | | | | - * | | | 3 | | | | + * | R7 | ROUTE_TABLE_ID | G | | | | + * | | (>= IN_IP_ROUTING_PRE && | 3 | | | | + * | | <= IN_IP_ROUTING) | | | | | * +-----+--------------------------+---+-----------------+---+---------------+ * | R8 | ECMP_GROUP_ID | | | * | | ECMP_MEMBER_ID | X | | @@ -8511,11 +8514,72 @@ cleanup: ds_destroy(&actions); } +static uint32_t +route_table_add(struct simap *route_tables, const char *route_table_name) +{ + /* route table ids start from 1 */ + uint32_t rtb_id = simap_count(route_tables) + 1; + + if (rtb_id == UINT16_MAX) { + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1); + VLOG_WARN_RL(&rl, "too many route tables for Logical Router."); + return 0; + } + + if (!simap_put(route_tables, route_table_name, rtb_id)) { + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1); + VLOG_WARN_RL(&rl, "Route table id unexpectedly appeared"); + } + + return rtb_id; +} + +static uint32_t +get_route_table_id(struct simap *route_tables, const char *route_table_name) +{ + if (!route_table_name || !strlen(route_table_name)) { + return 0; + } + + uint32_t rtb_id = simap_get(route_tables, route_table_name); + if (!rtb_id) { + rtb_id = route_table_add(route_tables, route_table_name); + } + + return rtb_id; +} + +static void +build_route_table_lflow(struct ovn_datapath *od, struct hmap *lflows, + struct nbrec_logical_router_port *lrp, + struct simap *route_tables) +{ + struct ds match = DS_EMPTY_INITIALIZER; + struct ds actions = DS_EMPTY_INITIALIZER; + + const char *route_table_name = smap_get(&lrp->options, "route_table"); + uint32_t rtb_id = get_route_table_id(route_tables, route_table_name); + if (!rtb_id) { + return; + } + + ds_put_format(&match, "inport == \"%s\"", lrp->name); + ds_put_format(&actions, "%s = %d; next;", + REG_ROUTE_TABLE_ID, rtb_id); + + ovn_lflow_add(lflows, od, S_ROUTER_IN_IP_ROUTING_PRE, 100, + ds_cstr(&match), ds_cstr(&actions)); + + ds_destroy(&match); + ds_destroy(&actions); +} + struct parsed_route { struct ovs_list list_node; struct in6_addr prefix; unsigned int plen; bool is_src_route; + uint32_t route_table_id; uint32_t hash; const struct nbrec_logical_router_static_route *route; bool ecmp_symmetric_reply; @@ -8540,7 +8604,7 @@ find_static_route_outport(struct ovn_datapath *od, struct hmap *ports, * Otherwise return NULL. */ static struct parsed_route * parsed_routes_add(struct ovn_datapath *od, struct hmap *ports, - struct ovs_list *routes, + struct ovs_list *routes, struct simap *route_tables, const struct nbrec_logical_router_static_route *route, struct hmap *bfd_connections) { @@ -8622,6 +8686,7 @@ parsed_routes_add(struct ovn_datapath *od, struct hmap *ports, struct parsed_route *pr = xzalloc(sizeof *pr); pr->prefix = prefix; pr->plen = plen; + pr->route_table_id = get_route_table_id(route_tables, route->route_table); pr->is_src_route = (route->policy && !strcmp(route->policy, "src-ip")); pr->hash = route_hash(pr); @@ -8655,6 +8720,7 @@ struct ecmp_groups_node { struct in6_addr prefix; unsigned int plen; bool is_src_route; + uint32_t route_table_id; uint16_t route_count; struct ovs_list route_list; /* Contains ecmp_route_list_node */ }; @@ -8663,7 +8729,7 @@ static void ecmp_groups_add_route(struct ecmp_groups_node *group, const struct parsed_route *route) { - if (group->route_count == UINT16_MAX) { + if (group->route_count == UINT16_MAX) { static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 1); VLOG_WARN_RL(&rl, "too many routes in a single ecmp group."); return; @@ -8692,6 +8758,7 @@ ecmp_groups_add(struct hmap *ecmp_groups, eg->prefix = route->prefix; eg->plen = route->plen; eg->is_src_route = route->is_src_route; + eg->route_table_id = route->route_table_id; ovs_list_init(&eg->route_list); ecmp_groups_add_route(eg, route); @@ -8705,7 +8772,8 @@ ecmp_groups_find(struct hmap *ecmp_groups, struct parsed_route *route) HMAP_FOR_EACH_WITH_HASH (eg, hmap_node, route->hash, ecmp_groups) { if (ipv6_addr_equals(&eg->prefix, &route->prefix) && eg->plen == route->plen && - eg->is_src_route == route->is_src_route) { + eg->is_src_route == route->is_src_route && + eg->route_table_id == route->route_table_id) { return eg; } } @@ -8752,7 +8820,8 @@ unique_routes_remove(struct hmap *unique_routes, HMAP_FOR_EACH_WITH_HASH (ur, hmap_node, route->hash, unique_routes) { if (ipv6_addr_equals(&route->prefix, &ur->route->prefix) && route->plen == ur->route->plen && - route->is_src_route == ur->route->is_src_route) { + route->is_src_route == ur->route->is_src_route && + route->route_table_id == ur->route->route_table_id) { hmap_remove(unique_routes, &ur->hmap_node); const struct parsed_route *existed_route = ur->route; free(ur); @@ -8790,9 +8859,9 @@ build_route_prefix_s(const struct in6_addr *prefix, unsigned int plen) } static void -build_route_match(const struct ovn_port *op_inport, const char *network_s, - int plen, bool is_src_route, bool is_ipv4, struct ds *match, - uint16_t *priority) +build_route_match(const struct ovn_port *op_inport, uint32_t rtb_id, + const char *network_s, int plen, bool is_src_route, + bool is_ipv4, struct ds *match, uint16_t *priority) { const char *dir; /* The priority here is calculated to implement longest-prefix-match @@ -8808,6 +8877,15 @@ build_route_match(const struct ovn_port *op_inport, const char *network_s, if (op_inport) { ds_put_format(match, "inport == %s && ", op_inport->json_key); } + if (rtb_id) { + ds_put_format(match, "%s == %d && ", REG_ROUTE_TABLE_ID, rtb_id); + } else { + /* Route-table assigned LRPs' routes should have lower priority + * in order not to affect directly-connected global routes. + * So, enlarge non-route-table routes priority by 100. + */ + *priority += 100; + } ds_put_format(match, "ip%s.%s == %s/%d", is_ipv4 ? "4" : "6", dir, network_s, plen); } @@ -8946,7 +9024,7 @@ add_ecmp_symmetric_reply_flows(struct hmap *lflows, out_port->lrp_networks.ea_s, IN6_IS_ADDR_V4MAPPED(&route->prefix) ? "" : "xx", port_ip, out_port->json_key); - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_IP_ROUTING, 300, + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_IP_ROUTING, 400, ds_cstr(&match), ds_cstr(&actions), &st_route->header_); @@ -8976,8 +9054,8 @@ build_ecmp_route_flow(struct hmap *lflows, struct ovn_datapath *od, struct ds route_match = DS_EMPTY_INITIALIZER; char *prefix_s = build_route_prefix_s(&eg->prefix, eg->plen); - build_route_match(NULL, prefix_s, eg->plen, eg->is_src_route, is_ipv4, - &route_match, &priority); + build_route_match(NULL, eg->route_table_id, prefix_s, eg->plen, + eg->is_src_route, is_ipv4, &route_match, &priority); free(prefix_s); struct ds actions = DS_EMPTY_INITIALIZER; @@ -9052,8 +9130,8 @@ static void add_route(struct hmap *lflows, struct ovn_datapath *od, const struct ovn_port *op, const char *lrp_addr_s, const char *network_s, int plen, const char *gateway, - bool is_src_route, const struct ovsdb_idl_row *stage_hint, - bool is_discard_route) + bool is_src_route, const uint32_t rtb_id, + const struct ovsdb_idl_row *stage_hint, bool is_discard_route) { bool is_ipv4 = strchr(network_s, '.') ? true : false; struct ds match = DS_EMPTY_INITIALIZER; @@ -9068,8 +9146,8 @@ add_route(struct hmap *lflows, struct ovn_datapath *od, op_inport = op; } } - build_route_match(op_inport, network_s, plen, is_src_route, is_ipv4, - &match, &priority); + build_route_match(op_inport, rtb_id, network_s, plen, is_src_route, + is_ipv4, &match, &priority); struct ds common_actions = DS_EMPTY_INITIALIZER; struct ds actions = DS_EMPTY_INITIALIZER; @@ -9132,7 +9210,8 @@ build_static_route_flow(struct hmap *lflows, struct ovn_datapath *od, char *prefix_s = build_route_prefix_s(&route_->prefix, route_->plen); add_route(lflows, route_->is_discard_route ? od : out_port->od, out_port, lrp_addr_s, prefix_s, route_->plen, route->nexthop, - route_->is_src_route, &route->header_, route_->is_discard_route); + route_->is_src_route, route_->route_table_id, &route->header_, + route_->is_discard_route); free(prefix_s); } @@ -10584,6 +10663,17 @@ build_ND_RA_flows_for_lrouter(struct ovn_datapath *od, struct hmap *lflows) } } +/* Logical router ingress table IP_ROUTING_PRE: + * by default goto next. (priority 0). */ +static void +build_ip_routing_pre_flows_for_lrouter(struct ovn_datapath *od, + struct hmap *lflows) +{ + if (od->nbr) { + ovn_lflow_add(lflows, od, S_ROUTER_IN_IP_ROUTING_PRE, 0, "1", "next;"); + } +} + /* Logical router ingress table IP_ROUTING : IP Routing. * * A packet that arrives at this table is an IP packet that should be @@ -10609,14 +10699,14 @@ build_ip_routing_flows_for_lrouter_port( for (int i = 0; i < op->lrp_networks.n_ipv4_addrs; i++) { add_route(lflows, op->od, op, op->lrp_networks.ipv4_addrs[i].addr_s, op->lrp_networks.ipv4_addrs[i].network_s, - op->lrp_networks.ipv4_addrs[i].plen, NULL, false, + op->lrp_networks.ipv4_addrs[i].plen, NULL, false, 0, &op->nbrp->header_, false); } for (int i = 0; i < op->lrp_networks.n_ipv6_addrs; i++) { add_route(lflows, op->od, op, op->lrp_networks.ipv6_addrs[i].addr_s, op->lrp_networks.ipv6_addrs[i].network_s, - op->lrp_networks.ipv6_addrs[i].plen, NULL, false, + op->lrp_networks.ipv6_addrs[i].plen, NULL, false, 0, &op->nbrp->header_, false); } } else if (lsp_is_router(op->nbsp)) { @@ -10639,7 +10729,7 @@ build_ip_routing_flows_for_lrouter_port( add_route(lflows, peer->od, peer, peer->lrp_networks.ipv4_addrs[0].addr_s, laddrs->ipv4_addrs[k].network_s, - laddrs->ipv4_addrs[k].plen, NULL, false, + laddrs->ipv4_addrs[k].plen, NULL, false, 0, &peer->nbrp->header_, false); } } @@ -10659,10 +10749,17 @@ build_static_route_flows_for_lrouter( struct hmap ecmp_groups = HMAP_INITIALIZER(&ecmp_groups); struct hmap unique_routes = HMAP_INITIALIZER(&unique_routes); struct ovs_list parsed_routes = OVS_LIST_INITIALIZER(&parsed_routes); + struct simap route_tables = SIMAP_INITIALIZER(&route_tables); struct ecmp_groups_node *group; + + for (int i = 0; i < od->nbr->n_ports; i++) { + build_route_table_lflow(od, lflows, od->nbr->ports[i], + &route_tables); + } + for (int i = 0; i < od->nbr->n_static_routes; i++) { struct parsed_route *route = - parsed_routes_add(od, ports, &parsed_routes, + parsed_routes_add(od, ports, &parsed_routes, &route_tables, od->nbr->static_routes[i], bfd_connections); if (!route) { continue; @@ -10695,6 +10792,7 @@ build_static_route_flows_for_lrouter( ecmp_groups_destroy(&ecmp_groups); unique_routes_destroy(&unique_routes); parsed_routes_destroy(&parsed_routes); + simap_destroy(&route_tables); } } @@ -12800,6 +12898,7 @@ build_lswitch_and_lrouter_iterate_by_od(struct ovn_datapath *od, build_neigh_learning_flows_for_lrouter(od, lsi->lflows, &lsi->match, &lsi->actions, lsi->meter_groups); build_ND_RA_flows_for_lrouter(od, lsi->lflows); + build_ip_routing_pre_flows_for_lrouter(od, lsi->lflows); build_static_route_flows_for_lrouter(od, lsi->lflows, lsi->ports, lsi->bfd_connections); build_mcast_lookup_flows_for_lrouter(od, lsi->lflows, &lsi->match, diff --git a/northd/ovn-northd.8.xml b/northd/ovn-northd.8.xml index 39f4eaa0c..cc2e25367 100644 --- a/northd/ovn-northd.8.xml +++ b/northd/ovn-northd.8.xml @@ -2899,7 +2899,7 @@ icmp6 { <p> If ECMP routes with symmetric reply are configured in the - <code>OVN_Northbound</code> database for a gateway router, a priority-300 + <code>OVN_Northbound</code> database for a gateway router, a priority-400 flow is added for each router port on which symmetric replies are configured. The matching logic for these ports essentially reverses the configured logic of the ECMP route. So for instance, a route with a @@ -3245,7 +3245,35 @@ output; </li> </ul> - <h3>Ingress Table 10: IP Routing</h3> + <h3>Ingress Table 10: IP Routing Pre</h3> + + <p> + If a packet arrived at this table from Logical Router Port <var>P</var> + which has <code>options:route_table</code> value set, a logical flow with + match <code>inport == "<var>P</var>"</code> with priority 100 and action, + setting unique-generated per-datapath 32-bit value (non-zero) in OVS + register 7. This register is checked in next table. + </p> + + <p> + This table contains the following logical flows: + </p> + + <ul> + <li> + <p> + Priority-100 flow with match <code>inport == "LRP_NAME"</code> value + and action, which set route table identifier in reg7. + </p> + + <p> + A priority-0 logical flow with match <code>1</code> has actions + <code>next;</code>. + </p> + </li> + </ul> + + <h3>Ingress Table 11: IP Routing</h3> <p> A packet that arrives at this table is an IP packet that should be @@ -3316,10 +3344,10 @@ output; <p> IPv4 routing table. For each route to IPv4 network <var>N</var> with netmask <var>M</var>, on router port <var>P</var> with IP address - <var>A</var> and Ethernet - address <var>E</var>, a logical flow with match <code>ip4.dst == - <var>N</var>/<var>M</var></code>, whose priority is the number of - 1-bits in <var>M</var>, has the following actions: + <var>A</var> and Ethernet address <var>E</var>, a logical flow with + match <code>ip4.dst == <var>N</var>/<var>M</var></code>, whose + priority is 100 + the number of 1-bits in <var>M</var>, has the + following actions: </p> <pre> @@ -3382,6 +3410,13 @@ next; If the address <var>A</var> is in the link-local scope, the route will be limited to sending on the ingress port. </p> + + <p> + For routes with <code>route_table</code> value set + <code>reg7 == id</code> is prefixed in logical flow match portion. + Priority for routes with <code>route_table</code> value set is + the number of 1-bits in <var>M</var>. + </p> </li> <li> @@ -3408,7 +3443,7 @@ select(reg8[16..31], <var>MID1</var>, <var>MID2</var>, ...); </li> </ul> - <h3>Ingress Table 11: IP_ROUTING_ECMP</h3> + <h3>Ingress Table 12: IP_ROUTING_ECMP</h3> <p> This table implements the second part of IP routing for ECMP routes @@ -3460,7 +3495,7 @@ outport = <var>P</var>; </li> </ul> - <h3>Ingress Table 12: Router policies</h3> + <h3>Ingress Table 13: Router policies</h3> <p> This table adds flows for the logical router policies configured on the logical router. Please see the @@ -3532,7 +3567,7 @@ next; </li> </ul> - <h3>Ingress Table 13: ECMP handling for router policies</h3> + <h3>Ingress Table 14: ECMP handling for router policies</h3> <p> This table handles the ECMP for the router policies configured with multiple nexthops. @@ -3576,7 +3611,7 @@ outport = <var>P</var> </li> </ul> - <h3>Ingress Table 14: ARP/ND Resolution</h3> + <h3>Ingress Table 15: ARP/ND Resolution</h3> <p> Any packet that reaches this table is an IP packet whose next-hop @@ -3767,7 +3802,7 @@ outport = <var>P</var> </ul> - <h3>Ingress Table 15: Check packet length</h3> + <h3>Ingress Table 16: Check packet length</h3> <p> For distributed logical routers or gateway routers with gateway @@ -3797,7 +3832,7 @@ REGBIT_PKT_LARGER = check_pkt_larger(<var>L</var>); next; and advances to the next table. </p> - <h3>Ingress Table 16: Handle larger packets</h3> + <h3>Ingress Table 17: Handle larger packets</h3> <p> For distributed logical routers or gateway routers with gateway port @@ -3860,7 +3895,7 @@ icmp6 { and advances to the next table. </p> - <h3>Ingress Table 17: Gateway Redirect</h3> + <h3>Ingress Table 18: Gateway Redirect</h3> <p> For distributed logical routers where one or more of the logical router @@ -3907,7 +3942,7 @@ icmp6 { </li> </ul> - <h3>Ingress Table 18: ARP Request</h3> + <h3>Ingress Table 19: ARP Request</h3> <p> In the common case where the Ethernet destination has been resolved, this diff --git a/ovn-nb.ovsschema b/ovn-nb.ovsschema index 2ac8ef3ea..a0a171e19 100644 --- a/ovn-nb.ovsschema +++ b/ovn-nb.ovsschema @@ -1,7 +1,7 @@ { "name": "OVN_Northbound", - "version": "5.32.1", - "cksum": "2805328215 29734", + "version": "5.33.1", + "cksum": "3874993350 29785", "tables": { "NB_Global": { "columns": { @@ -387,6 +387,7 @@ "isRoot": false}, "Logical_Router_Static_Route": { "columns": { + "route_table": {"type": "string"}, "ip_prefix": {"type": "string"}, "policy": {"type": {"key": {"type": "string", "enum": ["set", ["src-ip", diff --git a/ovn-nb.xml b/ovn-nb.xml index d8266ed4d..b2917c363 100644 --- a/ovn-nb.xml +++ b/ovn-nb.xml @@ -2772,6 +2772,14 @@ prefix according to RFC3663 </p> </column> + + <column name="options" key="route_table"> + Designates lookup Logical_Router_Static_Routes with specified + <code>route_table</code> value. Routes to directly connected networks + from same Logical Router and routes without <code>route_table</code> + option set have higher priority than routes with + <code>route_table</code> option set. + </column> </group> <group title="Attachment"> @@ -2891,6 +2899,28 @@ </p> </column> + <column name="route_table"> + <p> + Any string to place route to separate routing table. If Logical Router + Port has configured value in <ref table="Logical_Router_Port" + column="options" key="route_table"/> other than empty string, OVN + performs route lookup for all packets entering Logical Router ingress + pipeline from this port in the following manner: + </p> + + <ul> + <li> + 1. First lookup among "global" routes: routes without + <code>route_table</code> value set and routes to directly connected + networks. + </li> + <li> + 2. Next lookup among routes with same <code>route_table</code> value + as specified in LRP's options:route_table field. + </li> + </ul> + </column> + <column name="external_ids" key="ic-learned-route"> <code>ovn-ic</code> populates this key if the route is learned from the global <ref db="OVN_IC_Southbound"/> database. In this case the value diff --git a/tests/ovn-ic.at b/tests/ovn-ic.at index 32f4e9d02..3aab54362 100644 --- a/tests/ovn-ic.at +++ b/tests/ovn-ic.at @@ -281,6 +281,7 @@ done AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr1], [0], [dnl IPv4 Routes +Route Table global: 10.11.1.0/24 169.254.0.1 dst-ip 10.11.2.0/24 169.254.100.2 dst-ip (learned) 10.22.1.0/24 169.254.0.2 src-ip @@ -299,6 +300,7 @@ ovn_as az1 ovn-nbctl set nb_global . options:ic-route-learn=false OVS_WAIT_WHILE([ovn_as az1 ovn-nbctl lr-route-list lr1 | grep learned]) AT_CHECK([ovn_as az1 ovn-nbctl lr-route-list lr1], [0], [dnl IPv4 Routes +Route Table global: 10.11.1.0/24 169.254.0.1 dst-ip 10.22.1.0/24 169.254.0.2 src-ip ]) @@ -314,6 +316,7 @@ ovn_as az1 ovn-nbctl set nb_global . options:ic-route-adv=false OVS_WAIT_WHILE([ovn_as az2 ovn-nbctl lr-route-list lr2 | grep learned]) AT_CHECK([ovn_as az2 ovn-nbctl lr-route-list lr2], [0], [dnl IPv4 Routes +Route Table global: 10.11.2.0/24 169.254.0.1 dst-ip 10.22.2.0/24 169.254.0.2 src-ip ]) @@ -332,6 +335,7 @@ done # Default route should NOT get advertised or learned, by default. AT_CHECK([ovn_as az2 ovn-nbctl lr-route-list lr2], [0], [dnl IPv4 Routes +Route Table global: 10.11.1.0/24 169.254.100.1 dst-ip (learned) 10.11.2.0/24 169.254.0.1 dst-ip 10.22.2.0/24 169.254.0.2 src-ip diff --git a/tests/ovn-nbctl.at b/tests/ovn-nbctl.at index 9b80ae410..ddb5536ce 100644 --- a/tests/ovn-nbctl.at +++ b/tests/ovn-nbctl.at @@ -1520,6 +1520,7 @@ AT_CHECK([ovn-nbctl --ecmp --policy=src-ip lr-route-add lr0 20.0.0.0/24 11.0.0.1 AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl IPv4 Routes +Route Table global: 10.0.0.0/24 11.0.0.1 dst-ip 10.0.1.0/24 11.0.1.1 dst-ip lp0 10.0.10.0/24 dst-ip lp0 @@ -1534,6 +1535,7 @@ AT_CHECK([ovn-nbctl lrp-add lr0 lp1 f0:00:00:00:00:02 11.0.0.254/24]) AT_CHECK([ovn-nbctl --may-exist lr-route-add lr0 10.0.0.111/24 11.0.0.1 lp1]) AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl IPv4 Routes +Route Table global: 10.0.0.0/24 11.0.0.1 dst-ip lp1 10.0.1.0/24 11.0.1.1 dst-ip lp0 10.0.10.0/24 dst-ip lp0 @@ -1564,6 +1566,7 @@ AT_CHECK([ovn-nbctl --policy=src-ip lr-route-del lr0 9.16.1.0/24]) AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl IPv4 Routes +Route Table global: 10.0.0.0/24 11.0.0.1 dst-ip lp1 10.0.10.0/24 dst-ip lp0 10.0.0.0/24 11.0.0.2 src-ip @@ -1575,6 +1578,7 @@ AT_CHECK([ovn-nbctl --policy=dst-ip lr-route-del lr0 10.0.0.0/24]) AT_CHECK([ovn-nbctl --policy=src-ip lr-route-del lr0 10.0.0.0/24]) AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl IPv4 Routes +Route Table global: 10.0.10.0/24 dst-ip lp0 0.0.0.0/0 192.168.0.1 dst-ip ]) @@ -1585,6 +1589,7 @@ AT_CHECK([ovn-nbctl --policy=src-ip lr-route-add lr0 10.0.0.0/24 11.0.0.2]) AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24]) AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl IPv4 Routes +Route Table global: 10.0.10.0/24 dst-ip lp0 0.0.0.0/0 192.168.0.1 dst-ip ]) @@ -1601,6 +1606,7 @@ AT_CHECK([ovn-nbctl --ecmp lr-route-add lr0 10.0.0.0/24 11.0.0.3]) AT_CHECK([ovn-nbctl --ecmp lr-route-add lr0 10.0.0.0/24 11.0.0.4 lp0]) AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl IPv4 Routes +Route Table global: 10.0.0.0/24 11.0.0.1 dst-ip ecmp 10.0.0.0/24 11.0.0.2 dst-ip ecmp 10.0.0.0/24 11.0.0.3 dst-ip ecmp @@ -1615,6 +1621,7 @@ dnl Delete ecmp routes AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 11.0.0.1]) AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl IPv4 Routes +Route Table global: 10.0.0.0/24 11.0.0.2 dst-ip ecmp 10.0.0.0/24 11.0.0.3 dst-ip ecmp 10.0.0.0/24 11.0.0.4 dst-ip lp0 ecmp @@ -1622,12 +1629,14 @@ IPv4 Routes AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 11.0.0.2]) AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl IPv4 Routes +Route Table global: 10.0.0.0/24 11.0.0.3 dst-ip ecmp 10.0.0.0/24 11.0.0.4 dst-ip lp0 ecmp ]) AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 11.0.0.4 lp0]) AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl IPv4 Routes +Route Table global: 10.0.0.0/24 11.0.0.3 dst-ip ]) AT_CHECK([ovn-nbctl lr-route-del lr0 10.0.0.0/24 11.0.0.3]) @@ -1641,6 +1650,7 @@ AT_CHECK([ovn-nbctl lr-route-add lr0 2001:0db8:1::/64 2001:0db8:0:f103::1]) AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl IPv6 Routes +Route Table global: 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip ::/0 2001:db8:0:f101::1 dst-ip @@ -1650,6 +1660,7 @@ AT_CHECK([ovn-nbctl lr-route-del lr0 2001:0db8:0::/64]) AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl IPv6 Routes +Route Table global: 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip ::/0 2001:db8:0:f101::1 dst-ip ]) @@ -1677,11 +1688,13 @@ AT_CHECK([ovn-nbctl --may-exist --ecmp-symmetric-reply lr-route-add lr0 2003:0db AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl IPv4 Routes +Route Table global: 10.0.0.0/24 11.0.0.1 dst-ip 10.0.1.0/24 11.0.1.1 dst-ip lp0 0.0.0.0/0 192.168.0.1 dst-ip IPv6 Routes +Route Table global: 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip ecmp 2001:db8:1::/64 2001:db8:0:f103::2 dst-ip ecmp @@ -1696,7 +1709,188 @@ AT_CHECK([ovn-nbctl lrp-add lr0 lr0-p0 00:00:01:01:02:03 192.168.10.1/24]) bfd_uuid=$(ovn-nbctl create bfd logical_port=lr0-p0 dst_ip=100.0.0.50 status=down min_tx=250 min_rx=250 detect_mult=10) AT_CHECK([ovn-nbctl lr-route-add lr0 100.0.0.0/24 192.168.0.1]) route_uuid=$(fetch_column nb:logical_router_static_route _uuid ip_prefix="100.0.0.0/24") -AT_CHECK([ovn-nbctl set logical_router_static_route $route_uuid bfd=$bfd_uuid])]) +AT_CHECK([ovn-nbctl set logical_router_static_route $route_uuid bfd=$bfd_uuid]) + +check ovn-nbctl lr-route-del lr0 +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl +]) + +dnl Check IPv4 routes in route table +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 0.0.0.0/0 192.168.0.1 +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 10.0.1.1/24 11.0.1.1 lp0 +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 10.0.0.1/24 11.0.0.1 +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl +IPv4 Routes +Route Table rtb-1: + 10.0.0.0/24 11.0.0.1 dst-ip + 10.0.1.0/24 11.0.1.1 dst-ip lp0 + 0.0.0.0/0 192.168.0.1 dst-ip +]) + +check ovn-nbctl lr-route-del lr0 +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl +]) + +dnl Check IPv6 routes in route table +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 0:0:0:0:0:0:0:0/0 2001:0db8:0:f101::1 +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 2001:0db8:0::/64 2001:0db8:0:f102::1 lp0 +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 2001:0db8:1::/64 2001:0db8:0:f103::1 + +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl +IPv6 Routes +Route Table rtb-1: + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip + ::/0 2001:db8:0:f101::1 dst-ip +]) + +dnl Check IPv4 and IPv6 routes in route table +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 0.0.0.0/0 192.168.0.1 +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 10.0.1.1/24 11.0.1.1 lp0 +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 10.0.0.1/24 11.0.0.1 + +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl +IPv4 Routes +Route Table rtb-1: + 10.0.0.0/24 11.0.0.1 dst-ip + 10.0.1.0/24 11.0.1.1 dst-ip lp0 + 0.0.0.0/0 192.168.0.1 dst-ip + +IPv6 Routes +Route Table rtb-1: + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip + ::/0 2001:db8:0:f101::1 dst-ip +]) + +# Add routes in another route table +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 0.0.0.0/0 192.168.0.1 +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 10.0.1.1/24 11.0.1.1 lp0 +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 10.0.0.1/24 11.0.0.1 +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 0:0:0:0:0:0:0:0/0 2001:0db8:0:f101::1 +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 2001:0db8:0::/64 2001:0db8:0:f102::1 lp0 +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 2001:0db8:1::/64 2001:0db8:0:f103::1 + +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl +IPv4 Routes +Route Table rtb-1: + 10.0.0.0/24 11.0.0.1 dst-ip + 10.0.1.0/24 11.0.1.1 dst-ip lp0 + 0.0.0.0/0 192.168.0.1 dst-ip + +Route Table rtb-2: + 10.0.0.0/24 11.0.0.1 dst-ip + 10.0.1.0/24 11.0.1.1 dst-ip lp0 + 0.0.0.0/0 192.168.0.1 dst-ip + +IPv6 Routes +Route Table rtb-1: + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip + ::/0 2001:db8:0:f101::1 dst-ip + +Route Table rtb-2: + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip + ::/0 2001:db8:0:f101::1 dst-ip +]) + +# Add routes to global route table +check ovn-nbctl lr-route-add lr0 0.0.0.0/0 192.168.0.1 +check ovn-nbctl lr-route-add lr0 10.0.1.1/24 11.0.1.1 lp0 +check ovn-nbctl lr-route-add lr0 10.0.0.1/24 11.0.0.1 +check ovn-nbctl lr-route-add lr0 0:0:0:0:0:0:0:0/0 2001:0db8:0:f101::1 +check ovn-nbctl lr-route-add lr0 2001:0db8:0::/64 2001:0db8:0:f102::1 lp0 +check check ovn-nbctl lr-route-add lr0 2001:0db8:1::/64 2001:0db8:0:f103::1 + +AT_CHECK([ovn-nbctl lr-route-list lr0], [0], [dnl +IPv4 Routes +Route Table global: + 10.0.0.0/24 11.0.0.1 dst-ip + 10.0.1.0/24 11.0.1.1 dst-ip lp0 + 0.0.0.0/0 192.168.0.1 dst-ip + +Route Table rtb-1: + 10.0.0.0/24 11.0.0.1 dst-ip + 10.0.1.0/24 11.0.1.1 dst-ip lp0 + 0.0.0.0/0 192.168.0.1 dst-ip + +Route Table rtb-2: + 10.0.0.0/24 11.0.0.1 dst-ip + 10.0.1.0/24 11.0.1.1 dst-ip lp0 + 0.0.0.0/0 192.168.0.1 dst-ip + +IPv6 Routes +Route Table global: + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip + ::/0 2001:db8:0:f101::1 dst-ip + +Route Table rtb-1: + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip + ::/0 2001:db8:0:f101::1 dst-ip + +Route Table rtb-2: + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip + ::/0 2001:db8:0:f101::1 dst-ip +]) + +# delete IPv4 route from rtb-1 +check ovn-nbctl --route-table=rtb-1 lr-route-del lr0 10.0.0.0/24 +AT_CHECK([ovn-nbctl --route-table=rtb-1 lr-route-list lr0], [0], [dnl +IPv4 Routes +Route Table rtb-1: + 10.0.1.0/24 11.0.1.1 dst-ip lp0 + 0.0.0.0/0 192.168.0.1 dst-ip + +IPv6 Routes +Route Table rtb-1: + 2001:db8::/64 2001:db8:0:f102::1 dst-ip lp0 + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip + ::/0 2001:db8:0:f101::1 dst-ip +]) + +# delete IPv6 route from rtb-2 +check ovn-nbctl --route-table=rtb-2 lr-route-del lr0 2001:db8::/64 +AT_CHECK([ovn-nbctl --route-table=rtb-2 lr-route-list lr0], [0], [dnl +IPv4 Routes +Route Table rtb-2: + 10.0.0.0/24 11.0.0.1 dst-ip + 10.0.1.0/24 11.0.1.1 dst-ip lp0 + 0.0.0.0/0 192.168.0.1 dst-ip + +IPv6 Routes +Route Table rtb-2: + 2001:db8:1::/64 2001:db8:0:f103::1 dst-ip + ::/0 2001:db8:0:f101::1 dst-ip +]) + +check ovn-nbctl lr-route-del lr0 + +# ECMP route in route table +check ovn-nbctl --route-table=rtb1 lr-route-add lr0 0.0.0.0/0 192.168.0.1 +check ovn-nbctl --ecmp --route-table=rtb1 lr-route-add lr0 0.0.0.0/0 192.168.0.2 + +# Negative route table case: same prefix +AT_CHECK([ovn-nbctl --route-table=rtb1 lr-route-add lr0 0.0.0.0/0 192.168.0.1], [1], [], [dnl +ovn-nbctl: duplicate prefix: 0.0.0.0/0 (policy: dst-ip). Use option --ecmp to allow this for ECMP routing. +]) + +# Negative route table case: same prefix & nexthop with ecmp +AT_CHECK([ovn-nbctl --ecmp --route-table=rtb1 lr-route-add lr0 0.0.0.0/0 192.168.0.2], [1], [], [dnl +ovn-nbctl: duplicate nexthop for the same ECMP route +]) + +# Add routes to global route table +check ovn-nbctl lrp-add lr0 lrp0 00:00:00:00:00:01 1.1.1.1/24 +check ovn-nbctl lrp-set-options lrp0 route_table=rtb1 +AT_CHECK([ovn-nbctl get logical-router-port lrp0 options:route_table], [0], [dnl +rtb1 +]) +check `ovn-nbctl show lr0 | grep lrp0 -A3 | grep route_table=rtb1` +]) dnl --------------------------------------------------------------------- diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at index 3eebb55b6..e71e65bcc 100644 --- a/tests/ovn-northd.at +++ b/tests/ovn-northd.at @@ -5111,7 +5111,7 @@ check ovn-nbctl --wait=sb --ecmp-symmetric-reply lr-route-add lr0 1.0.0.1 192.16 ovn-sbctl dump-flows lr0 > lr0flows AT_CHECK([grep -e "lr_in_ip_routing.*select" lr0flows | sed 's/table=../table=??/' | sort], [0], [dnl - table=??(lr_in_ip_routing ), priority=65 , match=(ip4.dst == 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = 1; reg8[[16..31]] = select(1, 2);) + table=??(lr_in_ip_routing ), priority=165 , match=(ip4.dst == 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = 1; reg8[[16..31]] = select(1, 2);) ]) AT_CHECK([grep -e "lr_in_ip_routing_ecmp" lr0flows | sed 's/192\.168\.0\..0/192.168.0.??/' | sed 's/table=../table=??/' | sort], [0], [dnl table=??(lr_in_ip_routing_ecmp), priority=100 , match=(reg8[[0..15]] == 1 && reg8[[16..31]] == 1), action=(reg0 = 192.168.0.??; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; next;) @@ -5124,7 +5124,7 @@ check ovn-nbctl --wait=sb --ecmp-symmetric-reply lr-route-add lr0 1.0.0.1 192.16 ovn-sbctl dump-flows lr0 > lr0flows AT_CHECK([grep -e "lr_in_ip_routing.*select" lr0flows | sed 's/table=../table=??/' | sort], [0], [dnl - table=??(lr_in_ip_routing ), priority=65 , match=(ip4.dst == 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = 1; reg8[[16..31]] = select(1, 2);) + table=??(lr_in_ip_routing ), priority=165 , match=(ip4.dst == 1.0.0.1/32), action=(ip.ttl--; flags.loopback = 1; reg8[[0..15]] = 1; reg8[[16..31]] = select(1, 2);) ]) AT_CHECK([grep -e "lr_in_ip_routing_ecmp" lr0flows | sed 's/192\.168\.0\..0/192.168.0.??/' | sed 's/table=../table=??/' | sort], [0], [dnl table=??(lr_in_ip_routing_ecmp), priority=100 , match=(reg8[[0..15]] == 1 && reg8[[16..31]] == 1), action=(reg0 = 192.168.0.??; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; next;) @@ -5139,14 +5139,14 @@ check ovn-nbctl --wait=sb lr-route-add lr0 1.0.0.0/24 192.168.0.10 ovn-sbctl dump-flows lr0 > lr0flows AT_CHECK([grep -e "lr_in_ip_routing.*192.168.0.10" lr0flows | sed 's/table=../table=??/' | sort], [0], [dnl - table=??(lr_in_ip_routing ), priority=49 , match=(ip4.dst == 1.0.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.10; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; flags.loopback = 1; next;) + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == 1.0.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.10; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; flags.loopback = 1; next;) ]) check ovn-nbctl --wait=sb lr-route-add lr0 2.0.0.0/24 lr0-public ovn-sbctl dump-flows lr0 > lr0flows AT_CHECK([grep -e "lr_in_ip_routing.*2.0.0.0" lr0flows | sed 's/table=../table=??/' | sort], [0], [dnl - table=??(lr_in_ip_routing ), priority=49 , match=(ip4.dst == 2.0.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; flags.loopback = 1; next;) + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == 2.0.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 = 192.168.0.1; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; flags.loopback = 1; next;) ]) AT_CLEANUP @@ -5232,3 +5232,71 @@ AT_CHECK([grep lr_in_gw_redirect lrflows | grep cr-DR | sed 's/table=../table=?? AT_CLEANUP ]) + + +OVN_FOR_EACH_NORTHD([ +AT_SETUP([route tables -- flows]) +AT_KEYWORDS([route-tables-flows]) +ovn_start + +check ovn-nbctl lr-add lr0 +check ovn-nbctl lrp-add lr0 lrp0 00:00:00:00:00:01 192.168.0.1/24 +check ovn-nbctl lrp-add lr0 lrp1 00:00:00:00:01:01 192.168.1.1/24 +check ovn-nbctl lrp-add lr0 lrp2 00:00:00:00:02:01 192.168.2.1/24 +check ovn-nbctl lrp-set-options lrp1 route_table=rtb-1 +check ovn-nbctl lrp-set-options lrp2 route_table=rtb-2 + +check ovn-nbctl lr-route-add lr0 0.0.0.0/0 192.168.0.10 +check ovn-nbctl --route-table=rtb-1 lr-route-add lr0 192.168.0.0/24 192.168.1.10 +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 0.0.0.0/0 192.168.0.10 +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 1.1.1.1/32 192.168.0.20 +check ovn-nbctl --route-table=rtb-2 lr-route-add lr0 2.2.2.2/32 192.168.0.30 +check ovn-nbctl --route-table=rtb-2 --ecmp lr-route-add lr0 2.2.2.2/32 192.168.0.31 +check ovn-nbctl --wait=sb sync + +ovn-sbctl dump-flows lr0 > lr0flows +AT_CAPTURE_FILE([lr0flows]) + +AT_CHECK([grep -e "lr_in_ip_routing_pre.*match=(1)" lr0flows | sed 's/table=../table=??/'], [0], [dnl + table=??(lr_in_ip_routing_pre), priority=0 , match=(1), action=(next;) +]) + +p1_reg=$(grep -oP "lr_in_ip_routing_pre.*lrp1.*action=\(reg7 = \K." lr0flows) +p2_reg=$(grep -oP "lr_in_ip_routing_pre.*lrp2.*action=\(reg7 = \K." lr0flows) +echo $p1_reg +echo $p2_reg + +# exact register values are not predictable +if [[ $p1_reg -eq 2 ] && [ $p2_reg -eq 1 ]]; then + echo "swap reg values in dump" + sed -i -r s'/^(.*lrp2.*action=\(reg7 = )(1)(.*)/\12\3/g' lr0flows # "reg7 = 1" -> "reg7 = 2" + sed -i -r s'/^(.*lrp1.*action=\(reg7 = )(2)(.*)/\11\3/g' lr0flows # "reg7 = 2" -> "reg7 = 1" + sed -i -r s'/^(.*match=\(reg7 == )(2)( &&.*lrp1.*)/\11\3/g' lr0flows # "reg7 == 2" -> "reg7 == 1" + sed -i -r s'/^(.*match=\(reg7 == )(1)( &&.*lrp0.*)/\12\3/g' lr0flows # "reg7 == 1" -> "reg7 == 2" +fi + +check test "$p1_reg" != "$p2_reg" -a $((p1_reg * p2_reg)) -eq 2 + +AT_CHECK([grep "lr_in_ip_routing_pre" lr0flows | sed 's/table=../table=??/' | sort], [0], [dnl + table=??(lr_in_ip_routing_pre), priority=0 , match=(1), action=(next;) + table=??(lr_in_ip_routing_pre), priority=100 , match=(inport == "lrp1"), action=(reg7 = 1; next;) + table=??(lr_in_ip_routing_pre), priority=100 , match=(inport == "lrp2"), action=(reg7 = 2; next;) +]) + +grep -e "(lr_in_ip_routing ).*outport" lr0flows + +AT_CHECK([grep -e "(lr_in_ip_routing ).*outport" lr0flows | sed 's/table=../table=??/' | sort], [0], [dnl + table=??(lr_in_ip_routing ), priority=1 , match=(reg7 == 2 && ip4.dst == 0.0.0.0/0), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.10; reg1 = 192.168.0.1; eth.src = 00:00:00:00:00:01; outport = "lrp0"; flags.loopback = 1; next;) + table=??(lr_in_ip_routing ), priority=101 , match=(ip4.dst == 0.0.0.0/0), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.10; reg1 = 192.168.0.1; eth.src = 00:00:00:00:00:01; outport = "lrp0"; flags.loopback = 1; next;) + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == 192.168.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 = 192.168.0.1; eth.src = 00:00:00:00:00:01; outport = "lrp0"; flags.loopback = 1; next;) + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == 192.168.1.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 = 192.168.1.1; eth.src = 00:00:00:00:01:01; outport = "lrp1"; flags.loopback = 1; next;) + table=??(lr_in_ip_routing ), priority=149 , match=(ip4.dst == 192.168.2.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = ip4.dst; reg1 = 192.168.2.1; eth.src = 00:00:00:00:02:01; outport = "lrp2"; flags.loopback = 1; next;) + table=??(lr_in_ip_routing ), priority=229 , match=(inport == "lrp0" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe00:1; eth.src = 00:00:00:00:00:01; outport = "lrp0"; flags.loopback = 1; next;) + table=??(lr_in_ip_routing ), priority=229 , match=(inport == "lrp1" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe00:101; eth.src = 00:00:00:00:01:01; outport = "lrp1"; flags.loopback = 1; next;) + table=??(lr_in_ip_routing ), priority=229 , match=(inport == "lrp2" && ip6.dst == fe80::/64), action=(ip.ttl--; reg8[[0..15]] = 0; xxreg0 = ip6.dst; xxreg1 = fe80::200:ff:fe00:201; eth.src = 00:00:00:00:02:01; outport = "lrp2"; flags.loopback = 1; next;) + table=??(lr_in_ip_routing ), priority=49 , match=(reg7 == 1 && ip4.dst == 192.168.0.0/24), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.1.10; reg1 = 192.168.1.1; eth.src = 00:00:00:00:01:01; outport = "lrp1"; flags.loopback = 1; next;) + table=??(lr_in_ip_routing ), priority=65 , match=(reg7 == 2 && ip4.dst == 1.1.1.1/32), action=(ip.ttl--; reg8[[0..15]] = 0; reg0 = 192.168.0.20; reg1 = 192.168.0.1; eth.src = 00:00:00:00:00:01; outport = "lrp0"; flags.loopback = 1; next;) +]) + +AT_CLEANUP +]) diff --git a/tests/ovn.at b/tests/ovn.at index 49ece8735..60783a14b 100644 --- a/tests/ovn.at +++ b/tests/ovn.at @@ -18145,7 +18145,7 @@ eth_dst=00000000ff01 ip_src=$(ip_to_hex 10 0 0 10) ip_dst=$(ip_to_hex 172 168 0 101) send_icmp_packet 1 1 $eth_src $eth_dst $ip_src $ip_dst c4c9 0000000000000000000000 -AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | awk '/table=25, n_packets=1, n_bytes=45/{print $7" "$8}'],[0],[dnl +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | awk '/table=26, n_packets=1, n_bytes=45/{print $7" "$8}'],[0],[dnl priority=80,ip,reg15=0x3,metadata=0x3,nw_src=10.0.0.10 actions=drop ]) @@ -22577,6 +22577,433 @@ OVN_CLEANUP([hv1]) AT_CLEANUP ]) + +OVN_FOR_EACH_NORTHD([ +AT_SETUP([route tables -- global routes]) +ovn_start + +# Logical network: +# ls1 (192.168.1.0/24) - lrp-lr1-ls1 - lr1 - lrp-lr1-ls2 - ls2 (192.168.2.0/24) +# +# ls1 has lsp11 (192.168.1.11) and ls2 has lsp21 (192.168.2.21) and lsp22 +# (192.168.2.22) +# +# lrp-lr1-ls1 set options:route_table=rtb-1 +# +# Static routes on lr1: +# 0.0.0.0/0 nexthop 192.168.2.21 +# 1.1.1.1/32 nexthop 192.168.2.22 route_table=rtb-1 +# +# Test 1: +# lsp11 send packet to 2.2.2.2 +# +# Expected result: +# lsp21 should receive traffic, lsp22 should not receive any traffic +# +# Test 2: +# lsp11 send packet to 1.1.1.1 +# +# Expected result: +# lsp21 should receive traffic, lsp22 should not receive any traffic + +ovn-nbctl lr-add lr1 + +for i in 1 2; do + ovn-nbctl ls-add ls${i} + ovn-nbctl lrp-add lr1 lrp-lr1-ls${i} 00:00:00:01:0${i}:01 192.168.${i}.1/24 + ovn-nbctl lsp-add ls${i} lsp-ls${i}-lr1 -- lsp-set-type lsp-ls${i}-lr1 router \ + -- lsp-set-options lsp-ls${i}-lr1 router-port=lrp-lr1-ls${i} \ + -- lsp-set-addresses lsp-ls${i}-lr1 router +done + +# install static routes +ovn-nbctl lr-route-add lr1 0.0.0.0/0 192.168.2.21 +ovn-nbctl --route-table=rtb-1 lr-route-add lr1 1.1.1.1/32 192.168.2.22 + +# set lrp-lr1-ls1 route table +ovn-nbctl lrp-set-options lrp-lr1-ls1 route_table=rtb-1 + +# Create logical ports +ovn-nbctl lsp-add ls1 lsp11 -- \ + lsp-set-addresses lsp11 "f0:00:00:00:01:11 192.168.1.11" +ovn-nbctl lsp-add ls2 lsp21 -- \ + lsp-set-addresses lsp21 "f0:00:00:00:02:21 192.168.2.21" +ovn-nbctl lsp-add ls2 lsp22 -- \ + lsp-set-addresses lsp22 "f0:00:00:00:02:22 192.168.2.22" + +net_add n1 +sim_add hv1 +as hv1 +ovs-vsctl add-br br-phys +ovn_attach n1 br-phys 192.168.0.1 +ovs-vsctl -- add-port br-int hv1-vif1 -- \ + set interface hv1-vif1 external-ids:iface-id=lsp11 \ + options:tx_pcap=hv1/vif1-tx.pcap \ + options:rxq_pcap=hv1/vif1-rx.pcap \ + ofport-request=1 + +ovs-vsctl -- add-port br-int hv1-vif2 -- \ + set interface hv1-vif2 external-ids:iface-id=lsp21 \ + options:tx_pcap=hv1/vif2-tx.pcap \ + options:rxq_pcap=hv1/vif2-rx.pcap \ + ofport-request=2 + +ovs-vsctl -- add-port br-int hv1-vif3 -- \ + set interface hv1-vif3 external-ids:iface-id=lsp22 \ + options:tx_pcap=hv1/vif3-tx.pcap \ + options:rxq_pcap=hv1/vif3-rx.pcap \ + ofport-request=3 + +# wait for earlier changes to take effect +check ovn-nbctl --wait=hv sync +wait_for_ports_up + +for i in 1 2; do + packet="inport==\"lsp11\" && eth.src==f0:00:00:00:01:11 && eth.dst==00:00:00:01:01:01 && + ip4 && ip.ttl==64 && ip4.src==192.168.1.11 && ip4.dst==$i.$i.$i.$i && icmp" + AT_CHECK([as hv1 ovs-appctl -t ovn-controller inject-pkt "$packet"]) + + # Assume all packets go to lsp21. + exp_packet="eth.src==00:00:00:01:02:01 && eth.dst==f0:00:00:00:02:21 && + ip4 && ip.ttl==63 && ip4.src==192.168.1.11 && ip4.dst==$i.$i.$i.$i && icmp" + echo $exp_packet | ovstest test-ovn expr-to-packets >> expected_lsp21 +done +> expected_lsp22 + +# lsp21 should recieve 2 packets and lsp22 should recieve no packets +OVS_WAIT_UNTIL([ + rcv_n1=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" hv1/vif2-tx.pcap > lsp21.packets && cat lsp21.packets | wc -l` + rcv_n2=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" hv1/vif3-tx.pcap > lsp22.packets && cat lsp22.packets | wc -l` + echo $rcv_n1 $rcv_n2 + test $rcv_n1 -eq 2 -a $rcv_n2 -eq 0]) + +for i in 1 2; do + sort expected_lsp2$i > expout + AT_CHECK([cat lsp2${i}.packets | sort], [0], [expout]) +done + +OVN_CLEANUP([hv1]) +AT_CLEANUP +]) + + +OVN_FOR_EACH_NORTHD([ +AT_SETUP([route tables -- directly connected routes]) +ovn_start + +# Logical network: +# ls1 (192.168.1.0/24) - lrp-lr1-ls1 - lr1 - lrp-lr1-ls2 - ls2 (192.168.2.0/24) +# +# ls1 has lsp11 (192.168.1.11) and ls2 has lsp21 (192.168.2.21) +# +# lrp-lr1-ls1 set options:route_table=rtb-1 +# +# Static routes on lr1: +# 192.168.2.0/25 nexthop 192.168.1.11 route_table=rtb-1 +# +# Test 1: +# lsp11 send packet to 192.168.2.21 +# +# Expected result: +# lsp21 should receive traffic, lsp11 should not receive any traffic + +ovn-nbctl lr-add lr1 + +for i in 1 2; do + ovn-nbctl ls-add ls${i} + ovn-nbctl lrp-add lr1 lrp-lr1-ls${i} 00:00:00:01:0${i}:01 192.168.${i}.1/24 + ovn-nbctl lsp-add ls${i} lsp-ls${i}-lr1 -- lsp-set-type lsp-ls${i}-lr1 router \ + -- lsp-set-options lsp-ls${i}-lr1 router-port=lrp-lr1-ls${i} \ + -- lsp-set-addresses lsp-ls${i}-lr1 router +done + +# install static route, which overrides directly-connected routes +ovn-nbctl --route-table=rtb-1 lr-route-add lr1 192.168.2.0/25 192.168.1.11 + +# set lrp-lr1-ls1 route table +ovn-nbctl lrp-set-options lrp-lr1-ls1 route_table=rtb-1 + +# Create logical ports +ovn-nbctl lsp-add ls1 lsp11 -- \ + lsp-set-addresses lsp11 "f0:00:00:00:01:11 192.168.1.11" +ovn-nbctl lsp-add ls2 lsp21 -- \ + lsp-set-addresses lsp21 "f0:00:00:00:02:21 192.168.2.21" + +net_add n1 +sim_add hv1 +as hv1 +ovs-vsctl add-br br-phys +ovn_attach n1 br-phys 192.168.0.1 +ovs-vsctl -- add-port br-int hv1-vif1 -- \ + set interface hv1-vif1 external-ids:iface-id=lsp11 \ + options:tx_pcap=hv1/vif1-tx.pcap \ + options:rxq_pcap=hv1/vif1-rx.pcap \ + ofport-request=1 + +ovs-vsctl -- add-port br-int hv1-vif2 -- \ + set interface hv1-vif2 external-ids:iface-id=lsp21 \ + options:tx_pcap=hv1/vif2-tx.pcap \ + options:rxq_pcap=hv1/vif2-rx.pcap \ + ofport-request=2 + +# wait for earlier changes to take effect +check ovn-nbctl --wait=hv sync +wait_for_ports_up + +packet="inport==\"lsp11\" && eth.src==f0:00:00:00:01:11 && eth.dst==00:00:00:01:01:01 && + ip4 && ip.ttl==64 && ip4.src==192.168.1.11 && ip4.dst==192.168.2.21 && icmp" +AT_CHECK([as hv1 ovs-appctl -t ovn-controller inject-pkt "$packet"]) + +# Assume all packets go to lsp21. +exp_packet="eth.src==00:00:00:01:02:01 && eth.dst==f0:00:00:00:02:21 && + ip4 && ip.ttl==63 && ip4.src==192.168.1.11 && ip4.dst==192.168.2.21 && icmp" +echo $exp_packet | ovstest test-ovn expr-to-packets >> expected_lsp21 +> expected_lsp11 + +# lsp21 should recieve 1 icmp packet and lsp11 should recieve no packets +OVS_WAIT_UNTIL([ + rcv_n11=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" hv1/vif1-tx.pcap > lsp11.packets && cat lsp11.packets | wc -l` + rcv_n21=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" hv1/vif2-tx.pcap > lsp21.packets && cat lsp21.packets | wc -l` + echo $rcv_n11 $rcv_n21 + test $rcv_n11 -eq 0 -a $rcv_n21 -eq 1]) + +for i in 11 21; do + sort expected_lsp$i > expout + AT_CHECK([cat lsp${i}.packets | sort], [0], [expout]) +done + +OVN_CLEANUP([hv1]) +AT_CLEANUP +]) + + +OVN_FOR_EACH_NORTHD([ +AT_SETUP([route tables -- overlapping subnets]) +ovn_start + +# Logical network: +# +# ls1 (192.168.1.0/24) - lrp-lr1-ls1 -\ /- lrp-lr1-ls2 - ls2 (192.168.2.0/24) +# lr1 +# ls3 (192.168.3.0/24) - lrp-lr1-ls3 -/ \- lrp-lr1-ls4 - ls4 (192.168.4.0/24) +# +# ls1 has lsp11 (192.168.1.11) +# ls2 has lsp21 (192.168.2.21) +# ls3 has lsp31 (192.168.3.31) +# ls4 has lsp41 (192.168.4.41) +# +# lrp-lr1-ls1 set options:route_table=rtb-1 +# lrp-lr1-ls2 set options:route_table=rtb-2 +# +# Static routes on lr1: +# 10.0.0.0/24 nexthop 192.168.3.31 route_table=rtb-1 +# 10.0.0.0/24 nexthop 192.168.4.41 route_table=rtb-2 +# +# Test 1: +# lsp11 send packet to 10.0.0.1 +# +# Expected result: +# lsp31 should receive traffic, lsp41 should not receive any traffic +# +# Test 2: +# lsp21 send packet to 10.0.0.1 +# +# Expected result: +# lsp41 should receive traffic, lsp31 should not receive any traffic + +ovn-nbctl lr-add lr1 + +# Create logical topology +for i in $(seq 1 4); do + ovn-nbctl ls-add ls${i} + ovn-nbctl lrp-add lr1 lrp-lr1-ls${i} 00:00:00:01:0${i}:01 192.168.${i}.1/24 + ovn-nbctl lsp-add ls${i} lsp-ls${i}-lr1 -- lsp-set-type lsp-ls${i}-lr1 router \ + -- lsp-set-options lsp-ls${i}-lr1 router-port=lrp-lr1-ls${i} \ + -- lsp-set-addresses lsp-ls${i}-lr1 router + ovn-nbctl lsp-add ls$i lsp${i}1 -- \ + lsp-set-addresses lsp${i}1 "f0:00:00:00:0${i}:1${i} 192.168.${i}.${i}1" +done + +# install static routes +ovn-nbctl --route-table=rtb-1 lr-route-add lr1 10.0.0.0/24 192.168.3.31 +ovn-nbctl --route-table=rtb-2 lr-route-add lr1 10.0.0.0/24 192.168.4.41 + +# set lrp-lr1-ls{1,2} route tables +ovn-nbctl lrp-set-options lrp-lr1-ls1 route_table=rtb-1 +ovn-nbctl lrp-set-options lrp-lr1-ls2 route_table=rtb-2 + +net_add n1 +sim_add hv1 +as hv1 +ovs-vsctl add-br br-phys +ovn_attach n1 br-phys 192.168.0.1 + +for i in $(seq 1 4); do + ovs-vsctl -- add-port br-int hv1-vif${i} -- \ + set interface hv1-vif${i} external-ids:iface-id=lsp${i}1 \ + options:tx_pcap=hv1/vif${i}-tx.pcap \ + options:rxq_pcap=hv1/vif${i}-rx.pcap \ + ofport-request=${i} +done + +# wait for earlier changes to take effect +check ovn-nbctl --wait=hv sync +wait_for_ports_up + +# lsp31 should recieve packet coming from lsp11 +# lsp41 should recieve packet coming from lsp21 +for i in $(seq 1 2); do + di=$(( i + 2)) # dst index + ri=$(( 5 - i)) # reverse index + packet="inport==\"lsp${i}1\" && eth.src==f0:00:00:00:0${i}:1${i} && + eth.dst==00:00:00:01:0${i}:01 && ip4 && ip.ttl==64 && + ip4.src==192.168.${i}.${i}1 && ip4.dst==10.0.0.1 && icmp" + AT_CHECK([as hv1 ovs-appctl -t ovn-controller inject-pkt "$packet"]) + + # Assume all packets go to lsp${di}1. + exp_packet="eth.src==00:00:00:01:0${di}:01 && eth.dst==f0:00:00:00:0${di}:1${di} && + ip4 && ip.ttl==63 && ip4.src==192.168.${i}.${i}1 && ip4.dst==10.0.0.1 && icmp" + echo $exp_packet | ovstest test-ovn expr-to-packets >> expected_lsp${di}1 + > expected_lsp${ri}1 + + OVS_WAIT_UNTIL([ + rcv_n1=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" hv1/vif${di}-tx.pcap > lsp${di}1.packets && cat lsp${di}1.packets | wc -l` + rcv_n2=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" hv1/vif${ri}-tx.pcap > lsp${ri}1.packets && cat lsp${ri}1.packets | wc -l` + echo $rcv_n1 $rcv_n2 + test $rcv_n1 -eq 1 -a $rcv_n2 -eq 0]) + + for j in "${di}1" "${ri}1"; do + sort expected_lsp${j} > expout + AT_CHECK([cat lsp${j}.packets | sort], [0], [expout]) + done + + # cleanup tx pcap files + for j in "${di}1" "${ri}1"; do + ovs-vsctl -- remove interface hv1-vif${di} options tx_pcap + > hv1/vif${di}-tx.pcap + ovs-vsctl -- set interface hv1-vif${di} external-ids:iface-id=lsp${di}1 \ + options:tx_pcap=hv1/vif${di}-tx.pcap + done +done + +OVN_CLEANUP([hv1]) +AT_CLEANUP +]) + + +OVN_FOR_EACH_NORTHD([ +AT_SETUP([route tables IPv6 -- overlapping subnets]) +ovn_start + +# Logical network: +# +# ls1 (2001:db8:1::/64) - lrp-lr1-ls1 -\ /- lrp-lr1-ls2 - ls2 (2001:db8:2::/64) +# lr1 +# ls3 (2001:db8:3::/64) - lrp-lr1-ls3 -/ \- lrp-lr1-ls4 - ls4 (2001:db8:4::/64) +# +# ls1 has lsp11 (2001:db8:1::11) +# ls2 has lsp21 (2001:db8:2::21) +# ls3 has lsp31 (2001:db8:3::31) +# ls4 has lsp41 (2001:db8:4::41) +# +# lrp-lr1-ls1 set options:route_table=rtb-1 +# lrp-lr1-ls2 set options:route_table=rtb-2 +# +# Static routes on lr1: +# 2001:db8:2000::/64 nexthop 2001:db8:3::31 route_table=rtb-1 +# 2001:db8:2000::/64 nexthop 2001:db8:3::41 route_table=rtb-2 +# +# Test 1: +# lsp11 send packet to 2001:db8:2000::1 +# +# Expected result: +# lsp31 should receive traffic, lsp41 should not receive any traffic +# +# Test 2: +# lsp21 send packet to 2001:db8:2000::1 +# +# Expected result: +# lsp41 should receive traffic, lsp31 should not receive any traffic + +ovn-nbctl lr-add lr1 + +# Create logical topology +for i in $(seq 1 4); do + ovn-nbctl ls-add ls${i} + ovn-nbctl lrp-add lr1 lrp-lr1-ls${i} 00:00:00:01:0${i}:01 2001:db8:${i}::1/64 + ovn-nbctl lsp-add ls${i} lsp-ls${i}-lr1 -- lsp-set-type lsp-ls${i}-lr1 router \ + -- lsp-set-options lsp-ls${i}-lr1 router-port=lrp-lr1-ls${i} \ + -- lsp-set-addresses lsp-ls${i}-lr1 router + ovn-nbctl lsp-add ls$i lsp${i}1 -- \ + lsp-set-addresses lsp${i}1 "f0:00:00:00:0${i}:1${i} 2001:db8:${i}::${i}1" +done + +# install static routes +ovn-nbctl --route-table=rtb-1 lr-route-add lr1 2001:db8:2000::/64 2001:db8:3::31 +ovn-nbctl --route-table=rtb-2 lr-route-add lr1 2001:db8:2000::/64 2001:db8:4::41 + +# set lrp-lr1-ls{1,2} route tables +ovn-nbctl lrp-set-options lrp-lr1-ls1 route_table=rtb-1 +ovn-nbctl lrp-set-options lrp-lr1-ls2 route_table=rtb-2 + +net_add n1 +sim_add hv1 +as hv1 +ovs-vsctl add-br br-phys +ovn_attach n1 br-phys 192.168.0.1 + +for i in $(seq 1 4); do + ovs-vsctl -- add-port br-int hv1-vif${i} -- \ + set interface hv1-vif${i} external-ids:iface-id=lsp${i}1 \ + options:tx_pcap=hv1/vif${i}-tx.pcap \ + options:rxq_pcap=hv1/vif${i}-rx.pcap \ + ofport-request=${i} +done + +# wait for earlier changes to take effect +AT_CHECK([ovn-nbctl --timeout=3 --wait=hv sync], [0], [ignore]) + +# lsp31 should recieve packet coming from lsp11 +# lsp41 should recieve packet coming from lsp21 +for i in $(seq 1 2); do + di=$(( i + 2)) # dst index + ri=$(( 5 - i)) # reverse index + packet="inport==\"lsp${i}1\" && eth.src==f0:00:00:00:0${i}:1${i} && + eth.dst==00:00:00:01:0${i}:01 && ip6 && ip.ttl==64 && + ip6.src==2001:db8:${i}::${i}1 && ip6.dst==2001:db8:2000::1 && icmp6" + AT_CHECK([as hv1 ovs-appctl -t ovn-controller inject-pkt "$packet"]) + + # Assume all packets go to lsp${di}1. + exp_packet="eth.src==00:00:00:01:0${di}:01 && eth.dst==f0:00:00:00:0${di}:1${di} && ip6 && + ip.ttl==63 && ip6.src==2001:db8:${i}::${i}1 && ip6.dst==2001:db8:2000::1 && icmp6" + echo $exp_packet | ovstest test-ovn expr-to-packets >> expected_lsp${di}1 + > expected_lsp${ri}1 + + OVS_WAIT_UNTIL([ + rcv_n1=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" hv1/vif${di}-tx.pcap > lsp${di}1.packets && cat lsp${di}1.packets | wc -l` + rcv_n2=`$PYTHON "$ovs_srcdir/utilities/ovs-pcap.in" hv1/vif${ri}-tx.pcap > lsp${ri}1.packets && cat lsp${ri}1.packets | wc -l` + echo $rcv_n1 $rcv_n2 + test $rcv_n1 -eq 1 -a $rcv_n2 -eq 0]) + + for j in "${di}1" "${ri}1"; do + sort expected_lsp${j} > expout + AT_CHECK([cat lsp${j}.packets | sort], [0], [expout]) + done + + # cleanup tx pcap files + for j in "${di}1" "${ri}1"; do + ovs-vsctl -- remove interface hv1-vif${di} options tx_pcap + > hv1/vif${di}-tx.pcap + ovs-vsctl -- set interface hv1-vif${di} external-ids:iface-id=lsp${di}1 \ + options:tx_pcap=hv1/vif${di}-tx.pcap + done +done + +OVN_CLEANUP([hv1]) +AT_CLEANUP +]) + + OVN_FOR_EACH_NORTHD([ AT_SETUP([forwarding group: 3 HVs, 1 LR, 2 LS]) AT_KEYWORDS([forwarding-group]) @@ -23332,7 +23759,7 @@ ovn-sbctl dump-flows > sbflows AT_CAPTURE_FILE([sbflows]) AT_CAPTURE_FILE([offlows]) OVS_WAIT_UNTIL([ - as hv1 ovs-ofctl dump-flows br-int table=20 > offlows + as hv1 ovs-ofctl dump-flows br-int table=21 > offlows test $(grep -c "load:0x64->NXM_NX_PKT_MARK" offlows) = 1 && \ test $(grep -c "load:0x3->NXM_NX_PKT_MARK" offlows) = 1 && \ test $(grep -c "load:0x4->NXM_NX_PKT_MARK" offlows) = 1 && \ @@ -23425,12 +23852,12 @@ send_ipv4_pkt hv1 hv1-vif1 505400000003 00000000ff01 \ $(ip_to_hex 10 0 0 3) $(ip_to_hex 172 168 0 120) OVS_WAIT_UNTIL([ - test 1 -eq $(as hv1 ovs-ofctl dump-flows br-int table=20 | \ + test 1 -eq $(as hv1 ovs-ofctl dump-flows br-int table=21 | \ grep "load:0x2->NXM_NX_PKT_MARK" -c) ]) AT_CHECK([ - test 0 -eq $(as hv1 ovs-ofctl dump-flows br-int table=20 | \ + test 0 -eq $(as hv1 ovs-ofctl dump-flows br-int table=21 | \ grep "load:0x64->NXM_NX_PKT_MARK" -c) ]) @@ -24133,7 +24560,7 @@ AT_CHECK([ grep "priority=100" | \ grep -c "ct(commit,zone=NXM_NX_REG11\\[[0..15\\]],.*exec(move:NXM_OF_ETH_SRC\\[[\\]]->NXM_NX_CT_LABEL\\[[32..79\\]],load:0x[[0-9]]->NXM_NX_CT_LABEL\\[[80..95\\]]))" - grep table=22 hv${hv}flows | \ + grep table=23 hv${hv}flows | \ grep "priority=200" | \ grep -c "actions=move:NXM_NX_CT_LABEL\\[[32..79\\]]->NXM_OF_ETH_DST\\[[\\]]" done; :], [0], [dnl @@ -24258,7 +24685,7 @@ AT_CHECK([ grep "priority=100" | \ grep -c "ct(commit,zone=NXM_NX_REG11\\[[0..15\\]],.*exec(move:NXM_OF_ETH_SRC\\[[\\]]->NXM_NX_CT_LABEL\\[[32..79\\]],load:0x[[0-9]]->NXM_NX_CT_LABEL\\[[80..95\\]]))" - grep table=22 hv${hv}flows | \ + grep table=23 hv${hv}flows | \ grep "priority=200" | \ grep -c "actions=move:NXM_NX_CT_LABEL\\[[32..79\\]]->NXM_OF_ETH_DST\\[[\\]]" done; :], [0], [dnl @@ -24880,7 +25307,7 @@ AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | grep "actions=controller" | grep ]) # The packet should've been dropped in the lr_in_arp_resolve stage. -AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | grep -E "table=22, n_packets=1,.* priority=1,ip,metadata=0x${sw_key},nw_dst=10.0.1.1 actions=drop" -c], [0], [dnl +AT_CHECK([as hv1 ovs-ofctl dump-flows br-int | grep -E "table=23, n_packets=1,.* priority=1,ip,metadata=0x${sw_key},nw_dst=10.0.1.1 actions=drop" -c], [0], [dnl 1 ]) diff --git a/utilities/ovn-nbctl.c b/utilities/ovn-nbctl.c index e34bb65f7..0ff10618b 100644 --- a/utilities/ovn-nbctl.c +++ b/utilities/ovn-nbctl.c @@ -329,6 +329,8 @@ Logical router port commands:\n\ add logical port PORT on ROUTER\n\ lrp-set-gateway-chassis PORT CHASSIS [PRIORITY]\n\ set gateway chassis for port PORT\n\ + lrp-set-options PORT KEY=VALUE [KEY=VALUE]...\n\ + set router port options\n\ lrp-del-gateway-chassis PORT CHASSIS\n\ delete gateway chassis from port PORT\n\ lrp-get-gateway-chassis PORT\n\ @@ -351,11 +353,17 @@ Logical router port commands:\n\ ('overlay' or 'bridged')\n\ \n\ Route commands:\n\ - [--policy=POLICY] [--ecmp] [--ecmp-symmetric-reply] lr-route-add ROUTER \n\ - PREFIX NEXTHOP [PORT]\n\ + [--policy=POLICY]\n\ + [--ecmp]\n\ + [--ecmp-symmetric-reply]\n\ + [--route-table=ROUTE_TABLE]\n\ + lr-route-add ROUTER PREFIX NEXTHOP [PORT]\n\ add a route to ROUTER\n\ - [--policy=POLICY] lr-route-del ROUTER [PREFIX [NEXTHOP [PORT]]]\n\ + [--policy=POLICY]\n\ + [--route-table=ROUTE_TABLE]\n\ + lr-route-del ROUTER [PREFIX [NEXTHOP [PORT]]]\n\ remove routes from ROUTER\n\ + [--route-table=ROUTE_TABLE]\n\ lr-route-list ROUTER print routes for ROUTER\n\ \n\ Policy commands:\n\ @@ -743,6 +751,11 @@ print_lr(const struct nbrec_logical_router *lr, struct ds *s) ds_put_cstr(s, "]\n"); } + const char *route_table = smap_get(&lrp->options, "route_table"); + if (route_table) { + ds_put_format(s, " route-table: %s\n", route_table); + } + if (lrp->n_gateway_chassis) { const struct nbrec_gateway_chassis **gcs; @@ -862,6 +875,7 @@ nbctl_pre_show(struct ctl_context *ctx) ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_name); ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_mac); ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_networks); + ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_options); ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_gateway_chassis); ovsdb_idl_add_column(ctx->idl, &nbrec_gateway_chassis_col_chassis_name); @@ -4000,11 +4014,19 @@ nbctl_lr_policy_list(struct ctl_context *ctx) static struct nbrec_logical_router_static_route * nbctl_lr_get_route(const struct nbrec_logical_router *lr, char *prefix, - char *next_hop, bool is_src_route, bool ecmp) + char *next_hop, bool is_src_route, bool ecmp, + char *route_table) { for (int i = 0; i < lr->n_static_routes; i++) { struct nbrec_logical_router_static_route *route = lr->static_routes[i]; + /* Strict compare for route_table. + * If route_table was not specified, + * lookup for routes with empty route_table value. */ + if (strcmp(route->route_table, route_table ? route_table : "")) { + continue; + } + /* Compare route policy. */ char *nb_policy = route->policy; bool nb_is_src_route = false; @@ -4060,6 +4082,8 @@ nbctl_pre_lr_route_add(struct ctl_context *ctx) &nbrec_logical_router_static_route_col_bfd); ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_static_route_col_options); + ovsdb_idl_add_column(ctx->idl, + &nbrec_logical_router_static_route_col_route_table); } static char * OVS_WARN_UNUSED_RESULT @@ -4090,6 +4114,7 @@ nbctl_lr_route_add(struct ctl_context *ctx) } } + char *route_table = shash_find_data(&ctx->options, "--route-table"); bool v6_prefix = false; prefix = normalize_ipv4_prefix_str(ctx->argv[2]); if (!prefix) { @@ -4166,7 +4191,8 @@ nbctl_lr_route_add(struct ctl_context *ctx) bool ecmp = shash_find(&ctx->options, "--ecmp") != NULL || ecmp_symmetric_reply; struct nbrec_logical_router_static_route *route = - nbctl_lr_get_route(lr, prefix, next_hop, is_src_route, ecmp); + nbctl_lr_get_route(lr, prefix, next_hop, is_src_route, ecmp, + route_table); /* Validations for nexthop = "discard" */ if (is_discard_route) { @@ -4230,7 +4256,8 @@ nbctl_lr_route_add(struct ctl_context *ctx) } struct nbrec_logical_router_static_route *discard_route = - nbctl_lr_get_route(lr, prefix, "discard", is_src_route, true); + nbctl_lr_get_route(lr, prefix, "discard", is_src_route, true, + route_table); if (discard_route) { ctl_error(ctx, "discard nexthop for the same ECMP route exists."); goto cleanup; @@ -4246,6 +4273,9 @@ nbctl_lr_route_add(struct ctl_context *ctx) if (policy) { nbrec_logical_router_static_route_set_policy(route, policy); } + if (route_table) { + nbrec_logical_router_static_route_set_route_table(route, route_table); + } if (ecmp_symmetric_reply) { const struct smap options = SMAP_CONST1(&options, @@ -4289,6 +4319,8 @@ nbctl_pre_lr_route_del(struct ctl_context *ctx) &nbrec_logical_router_static_route_col_nexthop); ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_static_route_col_output_port); + ovsdb_idl_add_column(ctx->idl, + &nbrec_logical_router_static_route_col_route_table); } @@ -4302,6 +4334,7 @@ nbctl_lr_route_del(struct ctl_context *ctx) return; } + const char *route_table = shash_find_data(&ctx->options, "--route-table"); const char *policy = shash_find_data(&ctx->options, "--policy"); bool is_src_route = false; if (policy) { @@ -4392,6 +4425,14 @@ nbctl_lr_route_del(struct ctl_context *ctx) } } + /* Strict compare for route_table. + * If route_table was not specified, + * lookup for routes with empty route_table value. */ + if (strcmp(lr->static_routes[i]->route_table, + route_table ? route_table : "")) { + continue; + } + /* Compare output_port, if specified. */ if (output_port) { char *rt_output_port = lr->static_routes[i]->output_port; @@ -5115,6 +5156,41 @@ nbctl_pre_lrp_del_gateway_chassis(struct ctl_context *ctx) ovsdb_idl_add_column(ctx->idl, &nbrec_gateway_chassis_col_chassis_name); } +static void +nbctl_pre_lrp_options(struct ctl_context *ctx) +{ + ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_name); + ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_port_col_options); +} + +static void +nbctl_lrp_set_options(struct ctl_context *ctx) +{ + const char *id = ctx->argv[1]; + const struct nbrec_logical_router_port *lrp = NULL; + size_t i; + struct smap options = SMAP_INITIALIZER(&options); + + char *error = lrp_by_name_or_uuid(ctx, id, true, &lrp); + if (error) { + ctx->error = error; + return; + } + for (i = 2; i < ctx->argc; i++) { + char *key, *value; + value = xstrdup(ctx->argv[i]); + key = strsep(&value, "="); + if (value) { + smap_add(&options, key, value); + } + free(key); + } + + nbrec_logical_router_port_set_options(lrp, &options); + + smap_destroy(&options); +} + /* Removes logical router port 'lrp->gateway_chassis[idx]'. */ static void remove_gc(const struct nbrec_logical_router_port *lrp, size_t idx) @@ -5891,6 +5967,7 @@ route_cmp_details(const struct nbrec_logical_router_static_route *r1, } return r1->output_port ? 1 : -1; } + struct ipv4_route { int priority; ovs_be32 addr; @@ -5900,6 +5977,11 @@ struct ipv4_route { static int __ipv4_route_cmp(const struct ipv4_route *r1, const struct ipv4_route *r2) { + int rtb_cmp = strcmp(r1->route->route_table, + r2->route->route_table); + if (rtb_cmp) { + return rtb_cmp; + } if (r1->priority != r2->priority) { return r1->priority > r2->priority ? -1 : 1; } @@ -5931,6 +6013,11 @@ struct ipv6_route { static int __ipv6_route_cmp(const struct ipv6_route *r1, const struct ipv6_route *r2) { + int rtb_cmp = strcmp(r1->route->route_table, + r2->route->route_table); + if (rtb_cmp) { + return rtb_cmp; + } if (r1->priority != r2->priority) { return r1->priority > r2->priority ? -1 : 1; } @@ -6018,6 +6105,8 @@ nbctl_pre_lr_route_list(struct ctl_context *ctx) &nbrec_logical_router_static_route_col_options); ovsdb_idl_add_column(ctx->idl, &nbrec_logical_router_static_route_col_bfd); + ovsdb_idl_add_column(ctx->idl, + &nbrec_logical_router_static_route_col_route_table); } static void @@ -6035,12 +6124,17 @@ nbctl_lr_route_list(struct ctl_context *ctx) return; } + char *route_table = shash_find_data(&ctx->options, "--route-table"); + ipv4_routes = xmalloc(sizeof *ipv4_routes * lr->n_static_routes); ipv6_routes = xmalloc(sizeof *ipv6_routes * lr->n_static_routes); for (int i = 0; i < lr->n_static_routes; i++) { const struct nbrec_logical_router_static_route *route = lr->static_routes[i]; + if (route_table && strcmp(route->route_table, route_table)) { + continue; + } unsigned int plen; ovs_be32 ipv4; const char *policy = route->policy ? route->policy : "dst-ip"; @@ -6081,6 +6175,7 @@ nbctl_lr_route_list(struct ctl_context *ctx) if (n_ipv4_routes) { ds_put_cstr(&ctx->output, "IPv4 Routes\n"); } + const struct nbrec_logical_router_static_route *route; for (int i = 0; i < n_ipv4_routes; i++) { bool ecmp = false; if (i < n_ipv4_routes - 1 && @@ -6091,6 +6186,15 @@ nbctl_lr_route_list(struct ctl_context *ctx) &ipv4_routes[i - 1])) { ecmp = true; } + + route = ipv4_routes[i].route; + if (!i || (i > 0 && strcmp(route->route_table, + ipv4_routes[i - 1].route->route_table))) { + ds_put_format(&ctx->output, "%sRoute Table %s:\n", i ? "\n" : "", + strlen(route->route_table) ? route->route_table + : "global"); + } + print_route(ipv4_routes[i].route, &ctx->output, ecmp); } @@ -6108,6 +6212,15 @@ nbctl_lr_route_list(struct ctl_context *ctx) &ipv6_routes[i - 1])) { ecmp = true; } + + route = ipv6_routes[i].route; + if (!i || (i > 0 && strcmp(route->route_table, + ipv6_routes[i - 1].route->route_table))) { + ds_put_format(&ctx->output, "%sRoute Table %s:\n", i ? "\n" : "", + strlen(route->route_table) ? route->route_table + : "global"); + } + print_route(ipv6_routes[i].route, &ctx->output, ecmp); } @@ -6926,6 +7039,8 @@ static const struct ctl_command_syntax nbctl_commands[] = { "PORT CHASSIS [PRIORITY]", nbctl_pre_lrp_set_gateway_chassis, nbctl_lrp_set_gateway_chassis, NULL, "--may-exist", RW }, + { "lrp-set-options", 1, INT_MAX, "PORT KEY=VALUE [KEY=VALUE]...", + nbctl_pre_lrp_options, nbctl_lrp_set_options, NULL, "", RW }, { "lrp-del-gateway-chassis", 2, 2, "PORT CHASSIS", nbctl_pre_lrp_del_gateway_chassis, nbctl_lrp_del_gateway_chassis, NULL, "", RW }, @@ -6949,12 +7064,13 @@ static const struct ctl_command_syntax nbctl_commands[] = { /* logical router route commands. */ { "lr-route-add", 3, 4, "ROUTER PREFIX NEXTHOP [PORT]", nbctl_pre_lr_route_add, nbctl_lr_route_add, NULL, - "--may-exist,--ecmp,--ecmp-symmetric-reply,--policy=,--bfd?", RW }, + "--may-exist,--ecmp,--ecmp-symmetric-reply,--policy=,--route-table=,--bfd?", + RW }, { "lr-route-del", 1, 4, "ROUTER [PREFIX [NEXTHOP [PORT]]]", nbctl_pre_lr_route_del, nbctl_lr_route_del, NULL, - "--if-exists,--policy=", RW }, + "--if-exists,--policy=,--route-table=", RW }, { "lr-route-list", 1, 1, "ROUTER", nbctl_pre_lr_route_list, - nbctl_lr_route_list, NULL, "", RO }, + nbctl_lr_route_list, NULL, "--route-table=", RO }, /* Policy commands */ { "lr-policy-add", 4, INT_MAX,