From patchwork Wed Jul 31 10:41:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mohammad Heib X-Patchwork-Id: 1967028 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=UGK3m3ck; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WYpYd0szmz1ybb for ; Wed, 31 Jul 2024 20:41:41 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id B3F0881491; Wed, 31 Jul 2024 10:41:36 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id 8YX0ElOoPw_s; Wed, 31 Jul 2024 10:41:35 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=2605:bc80:3010:104::8cd3:938; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 7A7278164B Authentication-Results: smtp1.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=UGK3m3ck Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTPS id 7A7278164B; Wed, 31 Jul 2024 10:41:34 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 28DF1C0035; Wed, 31 Jul 2024 10:41:34 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 8FFEEC002A for ; Wed, 31 Jul 2024 10:41:32 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 7224D60B07 for ; Wed, 31 Jul 2024 10:41:32 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id 679ZiPBDVKB5 for ; Wed, 31 Jul 2024 10:41:31 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=170.10.129.124; helo=us-smtp-delivery-124.mimecast.com; envelope-from=mheib@redhat.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp3.osuosl.org 8850C6070C Authentication-Results: smtp3.osuosl.org; dmarc=pass (p=none dis=none) header.from=redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 8850C6070C Authentication-Results: smtp3.osuosl.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=UGK3m3ck Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id 8850C6070C for ; Wed, 31 Jul 2024 10:41:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1722422490; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jgwrWbYXaOkddhF4/T8Rk2ZTQM7X+4YGIWrbpi+vxX8=; b=UGK3m3ckFKiXnj8qeqj+fudO/Jxf9j1PM/q/kwwsI4MO5k6FwT7Fi3nl1VscdqFH6uoIzT Q2aDEcWUq6DXY+qmK0pcxON+j7y6Y2pMK/N0faDy/Eg8W1SRC8qX/EBvKL7Uk18m+82KyQ sSFPNH8excQ6KlMBhVyf8PYkZcr84R0= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-665-nr_b7r93O9iw0NJEgCHB7w-1; Wed, 31 Jul 2024 06:41:28 -0400 X-MC-Unique: nr_b7r93O9iw0NJEgCHB7w-1 Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 264AD1954B37 for ; Wed, 31 Jul 2024 10:41:28 +0000 (UTC) Received: from mheiblap.localdomain.com (unknown [10.47.238.131]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 9E9E51955F3B; Wed, 31 Jul 2024 10:41:26 +0000 (UTC) From: Mohammad Heib To: dev@openvswitch.org Date: Wed, 31 Jul 2024 13:41:12 +0300 Message-Id: <20240731104114.636699-2-mheib@redhat.com> In-Reply-To: <20240731104114.636699-1-mheib@redhat.com> References: <20240731104114.636699-1-mheib@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH ovn 1/3] Northd: Start tracking virtual port binding requests. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Northd handles virtual port binding requests received by ovn-controllers without tracking those requests or saving any info about the last binding requests and the number of requests received for an individual virtual port. This patch adds a basic tracking mechanism for each virtual port that future patches will use to limit/pause the controller from sending binding requests for a specific virtual port if this port overflows the system by such requests. Signed-off-by: Mohammad Heib --- northd/northd.c | 88 +++++++++++++++++++++++++++++++++++++++++++++ northd/northd.h | 2 ++ northd/ovn-northd.c | 3 ++ 3 files changed, 93 insertions(+) diff --git a/northd/northd.c b/northd/northd.c index a8a0b6f94..7437b1dfe 100644 --- a/northd/northd.c +++ b/northd/northd.c @@ -3757,6 +3757,79 @@ build_lb_port_related_data( build_lswitch_lbs_from_lrouter(lr_datapaths, lb_dps_map, lb_group_dps_map); } +/* + * These functions implements the binding request tracking for a virtual + * port which can be used to limit virtual port binding requests + * and avoid system overflow. + * + * Virtual port binding requests must not exceed + * VPORT_MAX_BINDING_REQUEST_TRESHOLD within a VPORT_BINDING_TIMEFRAME, + * otherwise, this vport must be defined as overflowed and should limit + * the binding request in this port for a certain time. + */ +#define VPORT_BINDING_TIMEFRAME 10000 +#define VPORT_MAX_BINDING_REQUEST_TRESHOLD 15 + +struct hmap tracked_virtual_ports; + +struct tracked_virtual_port { + struct hmap_node node; + /* + * Use port name instaed of ovn_port refrence to make + * sure that virtual port tracking data will be permanent accross + * northd loops and we can keep track the target ports. + */ + char *name; + long long int First_bind_in_tframe; + size_t Bind_request_cnt; +}; + +static struct tracked_virtual_port * +find_tracked_virtual_port(const char *name) { + struct tracked_virtual_port *vport; + HMAP_FOR_EACH (vport, node, &tracked_virtual_ports) { + if (!strcmp(name, vport->name)) { + return vport; + } + } + return NULL; +} + +static void +add_to_tracked_virtual_ports(const char *name) { + struct tracked_virtual_port *vport = find_tracked_virtual_port(name); + if (!vport) { + vport = xmalloc(sizeof *vport); + vport->name = xstrdup(name); + vport->First_bind_in_tframe = 0; + vport->Bind_request_cnt = 0; + hmap_insert(&tracked_virtual_ports, &vport->node, + hash_string(name, 0)); + } +} + +static void +remove_from_tracked_virtual_ports(const char *name) { + struct tracked_virtual_port *vport = find_tracked_virtual_port(name); + if (vport) { + free(vport->name); + hmap_remove(&tracked_virtual_ports, &vport->node); + free(vport); + } +} + +void init_tracked_virtual_ports(void) { + hmap_init(&tracked_virtual_ports); +} + +void destroy_tracked_virtual_ports(void) { + struct tracked_virtual_port *vport; + HMAP_FOR_EACH_SAFE (vport, node, &tracked_virtual_ports) { + remove_from_tracked_virtual_ports(vport->name); + } + hmap_destroy(&tracked_virtual_ports); +} + /* Syncs the SB port binding for the ovn_port 'op' of a logical switch port. * Caller should make sure that the OVN SB IDL txn is not NULL. Presently it * only syncs the nat column of port binding corresponding to the 'op->nbsp' */ @@ -4163,6 +4236,9 @@ build_ports(struct ovsdb_idl_txn *ovnsb_txn, op, queue_id_bitmap, &active_ha_chassis_grps); sbrec_port_binding_set_logical_port(op->sb, op->key); + if (!strcmp(op->sb->type, "virtual")) { + add_to_tracked_virtual_ports(op->sb->logical_port); + } ovs_list_remove(&op->list); } @@ -4170,6 +4246,9 @@ build_ports(struct ovsdb_idl_txn *ovnsb_txn, if (!ovs_list_is_empty(&sb_only)) { LIST_FOR_EACH_SAFE (op, list, &sb_only) { ovs_list_remove(&op->list); + if (!strcmp(op->sb->type, "virtual")) { + remove_from_tracked_virtual_ports(op->sb->logical_port); + } sbrec_port_binding_delete(op->sb); ovn_port_destroy(ports, op); } @@ -4554,6 +4633,12 @@ ls_handle_lsp_changes(struct ovsdb_idl_txn *ovnsb_idl_txn, if (!op) { goto fail; } + + if (!strcmp(new_nbsp->type, "virtual")) { + /* Add to virtual port tracking map */ + add_to_tracked_virtual_ports(op->nbsp->name); + } + add_op_to_northd_tracked_ports(&trk_lsps->created, op); } else if (ls_port_has_changed(new_nbsp)) { /* Existing port updated */ @@ -4614,6 +4699,9 @@ ls_handle_lsp_changes(struct ovsdb_idl_txn *ovnsb_idl_txn, add_op_to_northd_tracked_ports(&trk_lsps->deleted, op); hmap_remove(&nd->ls_ports, &op->key_node); hmap_remove(&od->ports, &op->dp_node); + if (!strcmp(op->sb->type, "virtual")) { + remove_from_tracked_virtual_ports(op->sb->logical_port); + } sbrec_port_binding_delete(op->sb); delete_fdb_entry(ni->sbrec_fdb_by_dp_and_port, od->tunnel_key, op->tunnel_key); diff --git a/northd/northd.h b/northd/northd.h index d4a8d75ab..5129f4afe 100644 --- a/northd/northd.h +++ b/northd/northd.h @@ -789,4 +789,6 @@ is_vxlan_mode(const struct smap *nb_options, uint32_t get_ovn_max_dp_key_local(bool _vxlan_mode); +void init_tracked_virtual_ports(void); +void destroy_tracked_virtual_ports(void); #endif /* NORTHD_H */ diff --git a/northd/ovn-northd.c b/northd/ovn-northd.c index d71114f35..e58abdbcd 100644 --- a/northd/ovn-northd.c +++ b/northd/ovn-northd.c @@ -891,6 +891,8 @@ main(int argc, char *argv[]) /* Initialize incremental processing engine for ovn-northd */ inc_proc_northd_init(&ovnnb_idl_loop, &ovnsb_idl_loop); + init_tracked_virtual_ports(); + unsigned int ovnnb_cond_seqno = UINT_MAX; unsigned int ovnsb_cond_seqno = UINT_MAX; @@ -1079,6 +1081,7 @@ main(int argc, char *argv[]) stopwatch_start(NORTHD_LOOP_STOPWATCH_NAME, time_msec()); } inc_proc_northd_cleanup(); + destroy_tracked_virtual_ports(); ovsdb_idl_loop_destroy(&ovnnb_idl_loop); ovsdb_idl_loop_destroy(&ovnsb_idl_loop); From patchwork Wed Jul 31 10:41:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mohammad Heib X-Patchwork-Id: 1967029 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=f46+WY+j; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WYpYd2YZ4z1yfG for ; Wed, 31 Jul 2024 20:41:41 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 54CA54092F; Wed, 31 Jul 2024 10:41:38 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id H5JTO-xGml1W; Wed, 31 Jul 2024 10:41:37 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=2605:bc80:3010:104::8cd3:938; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org DA7BE408EC Authentication-Results: smtp4.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=f46+WY+j Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id DA7BE408EC; Wed, 31 Jul 2024 10:41:36 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 9E8F1C002A; Wed, 31 Jul 2024 10:41:36 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 91E51C0078 for ; Wed, 31 Jul 2024 10:41:34 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 55705401A2 for ; Wed, 31 Jul 2024 10:41:34 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id Dun98pme8fSP for ; Wed, 31 Jul 2024 10:41:33 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=170.10.129.124; helo=us-smtp-delivery-124.mimecast.com; envelope-from=mheib@redhat.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp4.osuosl.org 2F4D440150 Authentication-Results: smtp4.osuosl.org; dmarc=pass (p=none dis=none) header.from=redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 2F4D440150 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp4.osuosl.org (Postfix) with ESMTPS id 2F4D440150 for ; Wed, 31 Jul 2024 10:41:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1722422492; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1O5QYGM7/AYrqc75NVeCvA1TB6Qvvo+BWFEhJ0B8zA4=; b=f46+WY+jQ21yPir2OWg1MQM411WI4OyWd+92RCHbuEwWWMsBPueHH/jbRJOSUlXdDNDCRp y2Bm5kiCh0ju4mAX/Mr42THI4IoykcXHq96Q/6I+W17GrnRKudhI9SNxj77ysuswlBx3pV ZAOexcg2EsTq/k9DHyBuspqSx9qsM14= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-683-SohEl8S8Ocuwq_OsEYOQkQ-1; Wed, 31 Jul 2024 06:41:30 -0400 X-MC-Unique: SohEl8S8Ocuwq_OsEYOQkQ-1 Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id F2D551954224 for ; Wed, 31 Jul 2024 10:41:29 +0000 (UTC) Received: from mheiblap.localdomain.com (unknown [10.47.238.131]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 7442A1955E80; Wed, 31 Jul 2024 10:41:28 +0000 (UTC) From: Mohammad Heib To: dev@openvswitch.org Date: Wed, 31 Jul 2024 13:41:13 +0300 Message-Id: <20240731104114.636699-3-mheib@redhat.com> In-Reply-To: <20240731104114.636699-2-mheib@redhat.com> References: <20240731104114.636699-1-mheib@redhat.com> <20240731104114.636699-2-mheib@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH ovn 2/3] Northd: Pause virtual port binding requests for crowded ports. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" ovn-controller sends binding requests to update the virtual parent of a virtual port to northd, in some cases those requests are not handled immediately and ovn-controller keeps sending requests over and over which can lead to flooding northd with these requests. This patch add the ability to pause virtual ports that send so many binding requests to northd. Signed-off-by: Mohammad Heib --- northd/northd.c | 98 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 98 insertions(+) diff --git a/northd/northd.c b/northd/northd.c index 7437b1dfe..49df463eb 100644 --- a/northd/northd.c +++ b/northd/northd.c @@ -3093,6 +3093,15 @@ ovn_port_update_sbrec(struct ovsdb_idl_txn *ovnsb_txn, "qdisc_queue_id", "%d", queue_id); } + if (smap_get_bool(&op->sb->options, + "binding_request_pause", false)) { + long long int p_time = smap_get_ullong(&op->sb->options, + "binding_request_pause_ts", 0); + smap_add_format(&options, "binding_request_pause_ts", + "%lld", p_time); + smap_add(&options, "binding_request_pause", "true"); + } + if (smap_get_bool(&op->od->nbs->other_config, "vlan-passthru", false)) { smap_add(&options, "vlan-passthru", "true"); } @@ -3830,6 +3839,90 @@ void destroy_tracked_virtual_ports(void) { hmap_destroy(&tracked_virtual_ports); } +/* + * For every virtual port that send request to update thier virtual_parent + * This function will update the following Port_binding options if needed: + * + * 1. tracked_virtual_port record belongs to this virtual port was created + * when this port created. This tracked struct have two main Fields: + * + * a. First_bind_in_tframe: this field will be set to the time that + * binding request were reicved for this vport for the first time + * within a timeframe. + * + * b. Bind_request_cnt: this filed will be incresses every time a binding + * request recived for that virtual port. + * + * + * 2. For each binding request received for a specific virtual port + * check if the time diff between now and the first time that a + * binding request were recived for this port within a pre-define + * timeframe is less than that timeframe. + * + * 3. If the previous condition true increase Bind_request_cnt and + * check if the total recived binding request recived for this port + * within a time fram exceeded the VPORT_MAX_BINDING_REQUEST_TRESHOLD + * set the Port_binding options: + * + * PB:OPTIONS:binding_request_pause=true + * PB:OPTIONS:binding_request_pause_ts=time_now + * + * + * 4. When ovn-controller recived a new GARP for this virtual port + * before sending a binding request update to northd it will check + * if the port have binding_request_pause=true, ovn-controller will do + * the following: + * + * If the PB:OPTIONS:binding_request_pause_ts + 10 seconds greater + * than the time now (GARP processing time), drop the GARP packet. + * + * Otherwise, set the PB:OPTIONS:binding_request_pause=false and resume + * binding request handling on this virtual port. + * + * + */ +static void +vport_binding_request_exceed_threshold(struct ovn_port *op) +{ + struct tracked_virtual_port * vport = + find_tracked_virtual_port(op->key); + if (op->sb != NULL) { + /* This port already paused or not found ignore it */ + if ((smap_get_bool(&op->sb->options, "binding_request_pause", + false) == true) || !vport) { + return; + } + } + + long long int cur_time = time_msec(); + + /* Still in the range of the time frame. */ + if ((vport->First_bind_in_tframe + VPORT_BINDING_TIMEFRAME) > cur_time) { + if (++vport->Bind_request_cnt > VPORT_MAX_BINDING_REQUEST_TRESHOLD) { + if (op->sb != NULL) { + static struct vlog_rate_limit rl = + VLOG_RATE_LIMIT_INIT(1, 1); + VLOG_WARN_RL(&rl, "Pausing virtual port %s from sending" + " binding requests for few seconds. " + " This port was paused in order to reduce the load on the" + " network.\n" , vport->name); + struct smap options; + smap_clone(&options, &op->sb->options); + smap_add(&options, "binding_request_pause", "true"); + smap_add_format(&options, "binding_request_pause_ts", "%lld", + cur_time); + sbrec_port_binding_set_options(op->sb, &options); + } + } + } else { + /* New Timeframe, that mean we had less than max binding + * request for this vport with it the past time frame. + */ + vport->First_bind_in_tframe = cur_time; + vport->Bind_request_cnt = 0; + } +} + /* Syncs the SB port binding for the ovn_port 'op' of a logical switch port. * Caller should make sure that the OVN SB IDL txn is not NULL. Presently it * only syncs the nat column of port binding corresponding to the 'op->nbsp' */ @@ -5019,6 +5112,11 @@ northd_handle_sb_port_binding_changes( "IDL row, which is unusual.", pb->logical_port); return false; } + + if (sbrec_port_binding_is_updated(pb, + SBREC_PORT_BINDING_COL_VIRTUAL_PARENT)) { + vport_binding_request_exceed_threshold(op); + } } } return true; From patchwork Wed Jul 31 10:41:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mohammad Heib X-Patchwork-Id: 1967030 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=QrbEqVwG; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WYpYg463jz1ybb for ; Wed, 31 Jul 2024 20:41:43 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 44C094023B; Wed, 31 Jul 2024 10:41:41 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id La46yZTGL1CQ; Wed, 31 Jul 2024 10:41:40 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=2605:bc80:3010:104::8cd3:938; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org EEEC540924 Authentication-Results: smtp4.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=QrbEqVwG Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp4.osuosl.org (Postfix) with ESMTPS id EEEC540924; Wed, 31 Jul 2024 10:41:39 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id BC50AC002B; Wed, 31 Jul 2024 10:41:39 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id BE7E0C0033 for ; Wed, 31 Jul 2024 10:41:37 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id B50E58175F for ; Wed, 31 Jul 2024 10:41:37 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id JOJu9OvUW0Sg for ; Wed, 31 Jul 2024 10:41:36 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=170.10.129.124; helo=us-smtp-delivery-124.mimecast.com; envelope-from=mheib@redhat.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp1.osuosl.org 1CA168174F Authentication-Results: smtp1.osuosl.org; dmarc=pass (p=none dis=none) header.from=redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 1CA168174F Authentication-Results: smtp1.osuosl.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=QrbEqVwG Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id 1CA168174F for ; Wed, 31 Jul 2024 10:41:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1722422495; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=y0rlcvOx2KwgQfnwNv7qG7eYx4AG6W3GdcEk4KK8iXs=; b=QrbEqVwGcnkrtafyReAjR/3PYo9BvI8/ulgPwVex9DveNnfPy9uEmH+uPIUOWTHGTY+9dw UtbI7pe7aOo7fa5R39zH+6n+EGLZSURRj4AZmTfqmPqV9Wwc4LC4hB/MY47zNVKF1O4PoV 2Z5ByxIWajpxwO2dZMOo5kNC5iMhIP8= Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-691-2B1J1Uc7M_aKz4VfZ3c5nw-1; Wed, 31 Jul 2024 06:41:33 -0400 X-MC-Unique: 2B1J1Uc7M_aKz4VfZ3c5nw-1 Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 7F2E91955D42 for ; Wed, 31 Jul 2024 10:41:32 +0000 (UTC) Received: from mheiblap.localdomain.com (unknown [10.47.238.131]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 9F4AB1955E80; Wed, 31 Jul 2024 10:41:30 +0000 (UTC) From: Mohammad Heib To: dev@openvswitch.org Date: Wed, 31 Jul 2024 13:41:14 +0300 Message-Id: <20240731104114.636699-4-mheib@redhat.com> In-Reply-To: <20240731104114.636699-3-mheib@redhat.com> References: <20240731104114.636699-1-mheib@redhat.com> <20240731104114.636699-2-mheib@redhat.com> <20240731104114.636699-3-mheib@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH ovn 3/3] controller: Drop binding requests for paused virtual port. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Drop the binding requests for a virtual port if that port set to pause in northd. Signed-off-by: Mohammad Heib --- controller/pinctrl.c | 39 ++++++++++++++++++- tests/ovn.at | 91 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 128 insertions(+), 2 deletions(-) diff --git a/controller/pinctrl.c b/controller/pinctrl.c index 7cbb0cf81..7420f2009 100644 --- a/controller/pinctrl.c +++ b/controller/pinctrl.c @@ -7057,11 +7057,16 @@ struct put_vport_binding { /* This vport record Only relevant if "new_record" is true. */ bool new_record; + /* The creation time in pinctrl thread */ + long long int creation_time; }; /* Contains "struct put_vport_binding"s. */ static struct hmap put_vport_bindings; +/* pause duration for port that set puased in northd. */ +#define PAUSE_DURATION 10000 + /* * Validate if the vport_binding record that was added * by the pinctrl thread is still relevant and needs @@ -7145,7 +7150,7 @@ run_put_vport_binding(struct ovsdb_idl_txn *ovnsb_idl_txn OVS_UNUSED, struct ovsdb_idl_index *sbrec_datapath_binding_by_key, struct ovsdb_idl_index *sbrec_port_binding_by_key, const struct sbrec_chassis *chassis, - const struct put_vport_binding *vpb) + struct put_vport_binding *vpb) { /* Convert logical datapath and logical port key into lport. */ const struct sbrec_port_binding *pb = lport_lookup_by_key( @@ -7159,6 +7164,35 @@ run_put_vport_binding(struct ovsdb_idl_txn *ovnsb_idl_txn OVS_UNUSED, return; } + if (smap_get(&pb->options, "binding_request_pause")) { + long long int p_time = smap_get_ullong(&pb->options, + "binding_request_pause_ts", 0); + /* Pause duration for this port still relevant, drop this + * binding request, and set vpb->new_record=false to make sure + * that it will be deleted from the list when flushing the list. + */ + if ((p_time + PAUSE_DURATION) > vpb->creation_time) { + vpb->new_record = false; + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 5); + VLOG_DBG_RL(&rl, + "Virtual lport %s drop binding request port " + "in pause state\n", pb->logical_port); + + return; + } else { + VLOG_INFO("Virtual lport %s binding requests paused " + "for 10 seconds, resume binding requests handling.", + pb->logical_port); + struct smap options; + smap_clone(&options, &pb->options); + smap_remove(&options, "binding_request_pause"); + smap_remove(&options, "binding_request_pause_ts"); + sbrec_port_binding_set_options(pb, &options); + smap_destroy(&options); + + } + } + /* pinctrl module updates the port binding only for type 'virtual'. */ if (!strcmp(pb->type, "virtual")) { const struct sbrec_port_binding *parent = lport_lookup_by_key( @@ -7187,7 +7221,7 @@ run_put_vport_bindings(struct ovsdb_idl_txn *ovnsb_idl_txn, return; } - const struct put_vport_binding *vpb; + struct put_vport_binding *vpb; HMAP_FOR_EACH (vpb, hmap_node, &put_vport_bindings) { run_put_vport_binding(ovnsb_idl_txn, sbrec_datapath_binding_by_key, sbrec_port_binding_by_key, chassis, vpb); @@ -7232,6 +7266,7 @@ pinctrl_handle_bind_vport( vpb->vport_key = vport_key; vpb->vport_parent_key = vport_parent_key; vpb->new_record = true; + vpb->creation_time = time_msec(); notify_pinctrl_main(); } diff --git a/tests/ovn.at b/tests/ovn.at index b31afbfb3..408505ee9 100644 --- a/tests/ovn.at +++ b/tests/ovn.at @@ -22442,6 +22442,97 @@ OVN_CLEANUP([hv1], [hv2]) AT_CLEANUP ]) +# Create 10 HV's each have 3 VIF ports that all +# sends Garp at the same time to bind vport sw0-vir +# this will create high pressure on SB/North and will +# lead to a transaction dropping by SB. +# +# Northd must be able to detect such cases and pause +# binding requests for this specific port for a certain +# amount of time. +# +OVN_FOR_EACH_NORTHD([ +AT_SETUP([virtual ports - binding requests storm]) +AT_KEYWORDS([virtual ports]) +ovn_start + +send_garp() { + local hv=$1 inport=$2 eth_src=$3 eth_dst=$4 spa=$5 tpa=$6 + local request=${eth_dst}${eth_src}08060001080006040001${eth_src}${spa}${eth_dst}${tpa} + as hv$hv ovs-appctl netdev-dummy/receive vif$hv$inport $request +} + +net_add n1 +check ovn-nbctl ls-add sw0 +check ovn-nbctl ls-add sw1 +parents="" +for i in {1..9}; do + sim_add hv$i + as hv$i + ovs-vsctl add-br br-phys + ovn_attach n1 br-phys 192.168.0.$i + ovs-appctl -t ovn-controller vlog/set dbg + + for j in {1..3}; do + check ovn-nbctl lsp-add sw0 sw0-h$i-p$j + check ovn-nbctl lsp-set-addresses sw0-h$i-p$j "50:54:00:00:00:$i$j 10.0.0.$i$j 1000::$i$j" + check ovn-nbctl lsp-set-port-security sw0-h$i-p$j "50:54:00:00:00:$i$j 10.0.0.$i$j 10.0.0.120 1000::$i$j" + parents+=$"sw0-h$i-p$j," + ovs-vsctl -- add-port br-int vif$i$j -- \ + set interface vif$i$j \ + external-ids:iface-id=sw0-h$i-p$j \ + options:tx_pcap=hv$i/vif$i$j-tx.pcap \ + options:rxq_pcap=hv$i/vif$i$j-rx.pcap \ + ofport-request=$i$j + done +done + +ovs-vsctl -- add-port br-int vifsw1 -- \ + set interface vifsw1 \ + external-ids:iface-id=sw1-p1 \ + options:tx_pcap=hv$i/vifsw1-tx.pcap \ + options:rxq_pcap=hv$i/vifsw1-rx.pcap \ + ofport-request=122 + +check ovn-nbctl lsp-add sw0 sw0-vir +check ovn-nbctl lsp-set-addresses sw0-vir "50:54:00:00:10:10 10.0.0.120" +check ovn-nbctl lsp-set-port-security sw0-vir "50:54:00:00:10:10 10.0.0.120" +check ovn-nbctl lsp-set-type sw0-vir virtual +check ovn-nbctl set logical_switch_port sw0-vir options:virtual-ip=10.0.0.120 +check ovn-nbctl set logical_switch_port sw0-vir options:virtual-parents=$parents + +# Add an ACL that matches on sw0-vir being bound locally. +check ovn-nbctl acl-add sw0 to-lport 1000 'is_chassis_resident("sw0-vir") && ip4' allow +OVN_POPULATE_ARP +wait_for_ports_up + +# Start sending many Garp requests on randomly selected ports and chassis +# to pressure the SB/Northd +eth_dst=ffffffffffff +spa=$(ip_to_hex 10 0 0 120) +tpa=$(ip_to_hex 10 0 0 120) +while : ; do + random_hv=$(shuf -i 1-9 -n 1) + random_port=$(shuf -i 1-3 -n 1) + eth_src=5054000000$random_hv$random_port + send_garp $random_hv $random_port $eth_src $eth_dst $spa $tpa + sleep 0.2 +done & +pid1=$! + +OVS_WAIT_UNTIL([test 0 != `grep -c "Pausing virtual port sw0-vir from sending binding requests for few seconds." northd/ovn-northd.log`]) +# Kill the Loop before exiting otherwise the subshell will keeps +# try to send pkts on HVs ovs interfaces which been cleaned and deleted +# and that will casue test to failed. +kill -9 $pid1 +wait $pid1 + +for i in {1..9}; do + OVN_CLEANUP_SBOX([hv$i]) +done +AT_CLEANUP +]) + OVN_FOR_EACH_NORTHD([ # Run ovn-nbctl in daemon mode, change to a backup database and verify that # an insert operation is not allowed.