From patchwork Thu Aug 1 10:44:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mohammad Heib X-Patchwork-Id: 1967612 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=P/BLVo7i; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WZQZg6H7sz1ybV for ; Thu, 1 Aug 2024 20:44:43 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id DF23040EC3; Thu, 1 Aug 2024 10:44:39 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id 9IOlHxdVS5dx; Thu, 1 Aug 2024 10:44:38 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.9.56; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org B4EEC40496 Authentication-Results: smtp2.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key, unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=P/BLVo7i Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp2.osuosl.org (Postfix) with ESMTPS id B4EEC40496; Thu, 1 Aug 2024 10:44:38 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 87F77C002B; Thu, 1 Aug 2024 10:44:38 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id E5708C002A for ; Thu, 1 Aug 2024 10:44:37 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id C36F240E8D for ; Thu, 1 Aug 2024 10:44:37 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id vlKJCuzb_Ia9 for ; Thu, 1 Aug 2024 10:44:36 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=170.10.129.124; helo=us-smtp-delivery-124.mimecast.com; envelope-from=mheib@redhat.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp2.osuosl.org 2BE2D403D2 Authentication-Results: smtp2.osuosl.org; dmarc=pass (p=none dis=none) header.from=redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 2BE2D403D2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id 2BE2D403D2 for ; Thu, 1 Aug 2024 10:44:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1722509074; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=l/R7nnhFcT/+VkF+UrKb9wgg8tqNlqgyVZp1mWxxO/Q=; b=P/BLVo7ivb5NUHJ0zNKg7jnRKyMTu3IwjOGTdYnHXEJmm623WDM+j701jpqlgTboDuMSGa 4sJLVGee2h2m6Z+V3subfJzoY7e7/PYn1ADo2KNsepLNkne3Xbaj0nh/h6Z2cSbKQffQQA k8GIwBT33zfnZVMtDUR+WvecyFmQ9To= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-627-rSt-qYEGPxGFlGFy-27qcg-1; Thu, 01 Aug 2024 06:44:33 -0400 X-MC-Unique: rSt-qYEGPxGFlGFy-27qcg-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 05DD41955D4E for ; Thu, 1 Aug 2024 10:44:33 +0000 (UTC) Received: from mheiblap.localdomain.com (unknown [10.47.238.182]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 852EE19560AE; Thu, 1 Aug 2024 10:44:30 +0000 (UTC) From: Mohammad Heib To: dev@openvswitch.org Date: Thu, 1 Aug 2024 13:44:20 +0300 Message-Id: <20240801104422.124876-2-mheib@redhat.com> In-Reply-To: <20240801104422.124876-1-mheib@redhat.com> References: <20240801104422.124876-1-mheib@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH ovn v2 1/3] Northd: Start tracking virtual port binding requests. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Northd handles virtual port binding requests received by ovn-controllers without tracking those requests or saving any info about the last binding requests and the number of requests received for an individual virtual port. This patch adds a basic tracking mechanism for each virtual port that future patches will use to limit/pause the controller from sending binding requests for a specific virtual port if this port overflows the system by such requests. Signed-off-by: Mohammad Heib --- northd/northd.c | 88 +++++++++++++++++++++++++++++++++++++++++++++ northd/northd.h | 2 ++ northd/ovn-northd.c | 3 ++ 3 files changed, 93 insertions(+) diff --git a/northd/northd.c b/northd/northd.c index a8a0b6f94..89d5b2936 100644 --- a/northd/northd.c +++ b/northd/northd.c @@ -3757,6 +3757,79 @@ build_lb_port_related_data( build_lswitch_lbs_from_lrouter(lr_datapaths, lb_dps_map, lb_group_dps_map); } +/* + * These functions implements the binding request tracking for a virtual + * port which can be used to limit virtual port binding requests + * and avoid system overflow. + * + * Virtual port binding requests must not exceed + * VPORT_MAX_BINDING_REQUEST_TRESHOLD within a VPORT_BINDING_TIMEFRAME, + * otherwise, this vport must be defined as overflowed and should limit + * the binding request in this port for a certain time. + */ +#define VPORT_BINDING_TIMEFRAME 10000 +#define VPORT_MAX_BINDING_REQUEST_TRESHOLD 15 + +static struct hmap tracked_virtual_ports; + +struct tracked_virtual_port { + struct hmap_node node; + /* + * Use port name instaed of ovn_port refrence to make + * sure that virtual port tracking data will be permanent accross + * northd loops and we can keep track the target ports. + */ + char *name; + long long int First_bind_in_tframe; + size_t Bind_request_cnt; +}; + +static struct tracked_virtual_port * +find_tracked_virtual_port(const char *name) { + struct tracked_virtual_port *vport; + HMAP_FOR_EACH (vport, node, &tracked_virtual_ports) { + if (!strcmp(name, vport->name)) { + return vport; + } + } + return NULL; +} + +static void +add_to_tracked_virtual_ports(const char *name) { + struct tracked_virtual_port *vport = find_tracked_virtual_port(name); + if (!vport) { + vport = xmalloc(sizeof *vport); + vport->name = xstrdup(name); + vport->First_bind_in_tframe = 0; + vport->Bind_request_cnt = 0; + hmap_insert(&tracked_virtual_ports, &vport->node, + hash_string(name, 0)); + } +} + +static void +remove_from_tracked_virtual_ports(const char *name) { + struct tracked_virtual_port *vport = find_tracked_virtual_port(name); + if (vport) { + free(vport->name); + hmap_remove(&tracked_virtual_ports, &vport->node); + free(vport); + } +} + +void init_tracked_virtual_ports(void) { + hmap_init(&tracked_virtual_ports); +} + +void destroy_tracked_virtual_ports(void) { + struct tracked_virtual_port *vport; + HMAP_FOR_EACH_SAFE (vport, node, &tracked_virtual_ports) { + remove_from_tracked_virtual_ports(vport->name); + } + hmap_destroy(&tracked_virtual_ports); +} + /* Syncs the SB port binding for the ovn_port 'op' of a logical switch port. * Caller should make sure that the OVN SB IDL txn is not NULL. Presently it * only syncs the nat column of port binding corresponding to the 'op->nbsp' */ @@ -4163,6 +4236,9 @@ build_ports(struct ovsdb_idl_txn *ovnsb_txn, op, queue_id_bitmap, &active_ha_chassis_grps); sbrec_port_binding_set_logical_port(op->sb, op->key); + if (!strcmp(op->sb->type, "virtual")) { + add_to_tracked_virtual_ports(op->sb->logical_port); + } ovs_list_remove(&op->list); } @@ -4170,6 +4246,9 @@ build_ports(struct ovsdb_idl_txn *ovnsb_txn, if (!ovs_list_is_empty(&sb_only)) { LIST_FOR_EACH_SAFE (op, list, &sb_only) { ovs_list_remove(&op->list); + if (!strcmp(op->sb->type, "virtual")) { + remove_from_tracked_virtual_ports(op->sb->logical_port); + } sbrec_port_binding_delete(op->sb); ovn_port_destroy(ports, op); } @@ -4554,6 +4633,12 @@ ls_handle_lsp_changes(struct ovsdb_idl_txn *ovnsb_idl_txn, if (!op) { goto fail; } + + if (!strcmp(new_nbsp->type, "virtual")) { + /* Add to virtual port tracking map */ + add_to_tracked_virtual_ports(op->nbsp->name); + } + add_op_to_northd_tracked_ports(&trk_lsps->created, op); } else if (ls_port_has_changed(new_nbsp)) { /* Existing port updated */ @@ -4614,6 +4699,9 @@ ls_handle_lsp_changes(struct ovsdb_idl_txn *ovnsb_idl_txn, add_op_to_northd_tracked_ports(&trk_lsps->deleted, op); hmap_remove(&nd->ls_ports, &op->key_node); hmap_remove(&od->ports, &op->dp_node); + if (!strcmp(op->sb->type, "virtual")) { + remove_from_tracked_virtual_ports(op->sb->logical_port); + } sbrec_port_binding_delete(op->sb); delete_fdb_entry(ni->sbrec_fdb_by_dp_and_port, od->tunnel_key, op->tunnel_key); diff --git a/northd/northd.h b/northd/northd.h index d4a8d75ab..5129f4afe 100644 --- a/northd/northd.h +++ b/northd/northd.h @@ -789,4 +789,6 @@ is_vxlan_mode(const struct smap *nb_options, uint32_t get_ovn_max_dp_key_local(bool _vxlan_mode); +void init_tracked_virtual_ports(void); +void destroy_tracked_virtual_ports(void); #endif /* NORTHD_H */ diff --git a/northd/ovn-northd.c b/northd/ovn-northd.c index d71114f35..e58abdbcd 100644 --- a/northd/ovn-northd.c +++ b/northd/ovn-northd.c @@ -891,6 +891,8 @@ main(int argc, char *argv[]) /* Initialize incremental processing engine for ovn-northd */ inc_proc_northd_init(&ovnnb_idl_loop, &ovnsb_idl_loop); + init_tracked_virtual_ports(); + unsigned int ovnnb_cond_seqno = UINT_MAX; unsigned int ovnsb_cond_seqno = UINT_MAX; @@ -1079,6 +1081,7 @@ main(int argc, char *argv[]) stopwatch_start(NORTHD_LOOP_STOPWATCH_NAME, time_msec()); } inc_proc_northd_cleanup(); + destroy_tracked_virtual_ports(); ovsdb_idl_loop_destroy(&ovnnb_idl_loop); ovsdb_idl_loop_destroy(&ovnsb_idl_loop); From patchwork Thu Aug 1 10:44:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mohammad Heib X-Patchwork-Id: 1967613 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=ZnPwPhLb; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WZQZj65dGz1ybV for ; Thu, 1 Aug 2024 20:44:45 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 1314680F42; Thu, 1 Aug 2024 10:44:44 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id cuuyLadnMJyt; Thu, 1 Aug 2024 10:44:40 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.9.56; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org A20BA80D1B Authentication-Results: smtp1.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=ZnPwPhLb Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTPS id A20BA80D1B; Thu, 1 Aug 2024 10:44:40 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 77C63C0035; Thu, 1 Aug 2024 10:44:40 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) by lists.linuxfoundation.org (Postfix) with ESMTP id DD29DC002A for ; Thu, 1 Aug 2024 10:44:38 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id CC97A40E46 for ; Thu, 1 Aug 2024 10:44:38 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id 4c5zxRxjOaGd for ; Thu, 1 Aug 2024 10:44:38 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=170.10.133.124; helo=us-smtp-delivery-124.mimecast.com; envelope-from=mheib@redhat.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp4.osuosl.org EA6A1409A4 Authentication-Results: smtp4.osuosl.org; dmarc=pass (p=none dis=none) header.from=redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org EA6A1409A4 Authentication-Results: smtp4.osuosl.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=ZnPwPhLb Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp4.osuosl.org (Postfix) with ESMTPS id EA6A1409A4 for ; Thu, 1 Aug 2024 10:44:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1722509076; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ARnsgZHUrL5Gqeh5HG+4UoTGIp2gmfqDamGEmPiN7X4=; b=ZnPwPhLbx8ccS3GBRrgZ6SavG9+41AB1Odc1F/QM2awdbaaCeNCNxpRfNNVPN64B1zstbK ZOsQjzvP1NpszQO0YL3xs3Rvsi+BVYNXv39g/GICbWdQ7anvglp2o1iFaijat4es8hIegM w0RUbDHYsn8wdJ6zSop4Iq4FS0jV7SU= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-652-FuG_GFEdOvek4W09odU8JQ-1; Thu, 01 Aug 2024 06:44:35 -0400 X-MC-Unique: FuG_GFEdOvek4W09odU8JQ-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id E442E1955D47 for ; Thu, 1 Aug 2024 10:44:34 +0000 (UTC) Received: from mheiblap.localdomain.com (unknown [10.47.238.182]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 75A5119560B2; Thu, 1 Aug 2024 10:44:33 +0000 (UTC) From: Mohammad Heib To: dev@openvswitch.org Date: Thu, 1 Aug 2024 13:44:21 +0300 Message-Id: <20240801104422.124876-3-mheib@redhat.com> In-Reply-To: <20240801104422.124876-2-mheib@redhat.com> References: <20240801104422.124876-1-mheib@redhat.com> <20240801104422.124876-2-mheib@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH ovn v2 2/3] Northd: Pause virtual port binding requests for crowded ports. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" ovn-controller sends binding requests to update the virtual parent of a virtual port to northd, in some cases those requests are not handled immediately and ovn-controller keeps sending requests over and over which can lead to flooding northd with these requests. This patch add the ability to pause virtual ports that send so many binding requests to northd. Signed-off-by: Mohammad Heib --- northd/northd.c | 98 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 98 insertions(+) diff --git a/northd/northd.c b/northd/northd.c index 89d5b2936..a2782031d 100644 --- a/northd/northd.c +++ b/northd/northd.c @@ -3093,6 +3093,15 @@ ovn_port_update_sbrec(struct ovsdb_idl_txn *ovnsb_txn, "qdisc_queue_id", "%d", queue_id); } + if (smap_get_bool(&op->sb->options, + "binding_request_pause", false)) { + long long int p_time = smap_get_ullong(&op->sb->options, + "binding_request_pause_ts", 0); + smap_add_format(&options, "binding_request_pause_ts", + "%lld", p_time); + smap_add(&options, "binding_request_pause", "true"); + } + if (smap_get_bool(&op->od->nbs->other_config, "vlan-passthru", false)) { smap_add(&options, "vlan-passthru", "true"); } @@ -3830,6 +3839,90 @@ void destroy_tracked_virtual_ports(void) { hmap_destroy(&tracked_virtual_ports); } +/* + * For every virtual port that send request to update thier virtual_parent + * This function will update the following Port_binding options if needed: + * + * 1. tracked_virtual_port record belongs to this virtual port was created + * when this port created. This tracked struct have two main Fields: + * + * a. First_bind_in_tframe: this field will be set to the time that + * binding request were reicved for this vport for the first time + * within a timeframe. + * + * b. Bind_request_cnt: this filed will be incresses every time a binding + * request recived for that virtual port. + * + * + * 2. For each binding request received for a specific virtual port + * check if the time diff between now and the first time that a + * binding request were recived for this port within a pre-define + * timeframe is less than that timeframe. + * + * 3. If the previous condition true increase Bind_request_cnt and + * check if the total recived binding request recived for this port + * within a time fram exceeded the VPORT_MAX_BINDING_REQUEST_TRESHOLD + * set the Port_binding options: + * + * PB:OPTIONS:binding_request_pause=true + * PB:OPTIONS:binding_request_pause_ts=time_now + * + * + * 4. When ovn-controller recived a new GARP for this virtual port + * before sending a binding request update to northd it will check + * if the port have binding_request_pause=true, ovn-controller will do + * the following: + * + * If the PB:OPTIONS:binding_request_pause_ts + 10 seconds greater + * than the time now (GARP processing time), drop the GARP packet. + * + * Otherwise, set the PB:OPTIONS:binding_request_pause=false and resume + * binding request handling on this virtual port. + * + * + */ +static void +vport_binding_request_exceed_threshold(struct ovn_port *op) +{ + struct tracked_virtual_port * vport = + find_tracked_virtual_port(op->key); + if (op->sb != NULL) { + /* This port already paused or not found ignore it */ + if ((smap_get_bool(&op->sb->options, "binding_request_pause", + false) == true) || !vport) { + return; + } + } + + long long int cur_time = time_msec(); + + /* Still in the range of the time frame. */ + if ((vport->First_bind_in_tframe + VPORT_BINDING_TIMEFRAME) > cur_time) { + if (++vport->Bind_request_cnt > VPORT_MAX_BINDING_REQUEST_TRESHOLD) { + if (op->sb != NULL) { + static struct vlog_rate_limit rl = + VLOG_RATE_LIMIT_INIT(1, 1); + VLOG_WARN_RL(&rl, "Pausing virtual port %s from sending" + " binding requests for few seconds. " + " This port was paused in order to reduce the load on the" + " network.\n" , vport->name); + struct smap options; + smap_clone(&options, &op->sb->options); + smap_add(&options, "binding_request_pause", "true"); + smap_add_format(&options, "binding_request_pause_ts", "%lld", + cur_time); + sbrec_port_binding_set_options(op->sb, &options); + } + } + } else { + /* New Timeframe, that mean we had less than max binding + * request for this vport with it the past time frame. + */ + vport->First_bind_in_tframe = cur_time; + vport->Bind_request_cnt = 0; + } +} + /* Syncs the SB port binding for the ovn_port 'op' of a logical switch port. * Caller should make sure that the OVN SB IDL txn is not NULL. Presently it * only syncs the nat column of port binding corresponding to the 'op->nbsp' */ @@ -5019,6 +5112,11 @@ northd_handle_sb_port_binding_changes( "IDL row, which is unusual.", pb->logical_port); return false; } + + if (sbrec_port_binding_is_updated(pb, + SBREC_PORT_BINDING_COL_VIRTUAL_PARENT)) { + vport_binding_request_exceed_threshold(op); + } } } return true; From patchwork Thu Aug 1 10:44:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mohammad Heib X-Patchwork-Id: 1967614 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=MuTy2d4I; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4WZQZm3vDlz1ybV for ; Thu, 1 Aug 2024 20:44:48 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 09CC881926; Thu, 1 Aug 2024 10:44:44 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id 6v48c3phma3g; Thu, 1 Aug 2024 10:44:43 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=2605:bc80:3010:104::8cd3:938; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 9B38480BCA Authentication-Results: smtp1.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=MuTy2d4I Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTPS id 9B38480BCA; Thu, 1 Aug 2024 10:44:43 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 6C7ADC002B; Thu, 1 Aug 2024 10:44:43 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 268EAC002A for ; Thu, 1 Aug 2024 10:44:42 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id A04C980DA4 for ; Thu, 1 Aug 2024 10:44:41 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id dVqjQBCPTQVi for ; Thu, 1 Aug 2024 10:44:40 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=170.10.133.124; helo=us-smtp-delivery-124.mimecast.com; envelope-from=mheib@redhat.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp1.osuosl.org DFDD280DB6 Authentication-Results: smtp1.osuosl.org; dmarc=pass (p=none dis=none) header.from=redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org DFDD280DB6 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id DFDD280DB6 for ; Thu, 1 Aug 2024 10:44:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1722509078; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=y0rlcvOx2KwgQfnwNv7qG7eYx4AG6W3GdcEk4KK8iXs=; b=MuTy2d4IYBmyh1A4W1kKGc1gWDBUW8HzyitvP2PIFLqfYxGa3/xGkPn/QK++XDJlUSRoiY nsG9CzfGL0wP0Xo/dw4IZTyCtLuwR8UgdIzHGbi3pNBDsMb4SZ4e1GVctGGC1OJ1td14Zj kn7I+Osxg32tTxT2ivwxIYhuiusP9+A= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-680-GspocqwfMmWup8McFIdSlQ-1; Thu, 01 Aug 2024 06:44:37 -0400 X-MC-Unique: GspocqwfMmWup8McFIdSlQ-1 Received: from mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id C7CF81955D57 for ; Thu, 1 Aug 2024 10:44:36 +0000 (UTC) Received: from mheiblap.localdomain.com (unknown [10.47.238.182]) by mx-prod-int-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 5E65819560AE; Thu, 1 Aug 2024 10:44:35 +0000 (UTC) From: Mohammad Heib To: dev@openvswitch.org Date: Thu, 1 Aug 2024 13:44:22 +0300 Message-Id: <20240801104422.124876-4-mheib@redhat.com> In-Reply-To: <20240801104422.124876-3-mheib@redhat.com> References: <20240801104422.124876-1-mheib@redhat.com> <20240801104422.124876-2-mheib@redhat.com> <20240801104422.124876-3-mheib@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.12 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH ovn v2 3/3] controller: Drop binding requests for paused virtual port. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Drop the binding requests for a virtual port if that port set to pause in northd. Signed-off-by: Mohammad Heib --- controller/pinctrl.c | 39 ++++++++++++++++++- tests/ovn.at | 91 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 128 insertions(+), 2 deletions(-) diff --git a/controller/pinctrl.c b/controller/pinctrl.c index 7cbb0cf81..7420f2009 100644 --- a/controller/pinctrl.c +++ b/controller/pinctrl.c @@ -7057,11 +7057,16 @@ struct put_vport_binding { /* This vport record Only relevant if "new_record" is true. */ bool new_record; + /* The creation time in pinctrl thread */ + long long int creation_time; }; /* Contains "struct put_vport_binding"s. */ static struct hmap put_vport_bindings; +/* pause duration for port that set puased in northd. */ +#define PAUSE_DURATION 10000 + /* * Validate if the vport_binding record that was added * by the pinctrl thread is still relevant and needs @@ -7145,7 +7150,7 @@ run_put_vport_binding(struct ovsdb_idl_txn *ovnsb_idl_txn OVS_UNUSED, struct ovsdb_idl_index *sbrec_datapath_binding_by_key, struct ovsdb_idl_index *sbrec_port_binding_by_key, const struct sbrec_chassis *chassis, - const struct put_vport_binding *vpb) + struct put_vport_binding *vpb) { /* Convert logical datapath and logical port key into lport. */ const struct sbrec_port_binding *pb = lport_lookup_by_key( @@ -7159,6 +7164,35 @@ run_put_vport_binding(struct ovsdb_idl_txn *ovnsb_idl_txn OVS_UNUSED, return; } + if (smap_get(&pb->options, "binding_request_pause")) { + long long int p_time = smap_get_ullong(&pb->options, + "binding_request_pause_ts", 0); + /* Pause duration for this port still relevant, drop this + * binding request, and set vpb->new_record=false to make sure + * that it will be deleted from the list when flushing the list. + */ + if ((p_time + PAUSE_DURATION) > vpb->creation_time) { + vpb->new_record = false; + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 5); + VLOG_DBG_RL(&rl, + "Virtual lport %s drop binding request port " + "in pause state\n", pb->logical_port); + + return; + } else { + VLOG_INFO("Virtual lport %s binding requests paused " + "for 10 seconds, resume binding requests handling.", + pb->logical_port); + struct smap options; + smap_clone(&options, &pb->options); + smap_remove(&options, "binding_request_pause"); + smap_remove(&options, "binding_request_pause_ts"); + sbrec_port_binding_set_options(pb, &options); + smap_destroy(&options); + + } + } + /* pinctrl module updates the port binding only for type 'virtual'. */ if (!strcmp(pb->type, "virtual")) { const struct sbrec_port_binding *parent = lport_lookup_by_key( @@ -7187,7 +7221,7 @@ run_put_vport_bindings(struct ovsdb_idl_txn *ovnsb_idl_txn, return; } - const struct put_vport_binding *vpb; + struct put_vport_binding *vpb; HMAP_FOR_EACH (vpb, hmap_node, &put_vport_bindings) { run_put_vport_binding(ovnsb_idl_txn, sbrec_datapath_binding_by_key, sbrec_port_binding_by_key, chassis, vpb); @@ -7232,6 +7266,7 @@ pinctrl_handle_bind_vport( vpb->vport_key = vport_key; vpb->vport_parent_key = vport_parent_key; vpb->new_record = true; + vpb->creation_time = time_msec(); notify_pinctrl_main(); } diff --git a/tests/ovn.at b/tests/ovn.at index b31afbfb3..408505ee9 100644 --- a/tests/ovn.at +++ b/tests/ovn.at @@ -22442,6 +22442,97 @@ OVN_CLEANUP([hv1], [hv2]) AT_CLEANUP ]) +# Create 10 HV's each have 3 VIF ports that all +# sends Garp at the same time to bind vport sw0-vir +# this will create high pressure on SB/North and will +# lead to a transaction dropping by SB. +# +# Northd must be able to detect such cases and pause +# binding requests for this specific port for a certain +# amount of time. +# +OVN_FOR_EACH_NORTHD([ +AT_SETUP([virtual ports - binding requests storm]) +AT_KEYWORDS([virtual ports]) +ovn_start + +send_garp() { + local hv=$1 inport=$2 eth_src=$3 eth_dst=$4 spa=$5 tpa=$6 + local request=${eth_dst}${eth_src}08060001080006040001${eth_src}${spa}${eth_dst}${tpa} + as hv$hv ovs-appctl netdev-dummy/receive vif$hv$inport $request +} + +net_add n1 +check ovn-nbctl ls-add sw0 +check ovn-nbctl ls-add sw1 +parents="" +for i in {1..9}; do + sim_add hv$i + as hv$i + ovs-vsctl add-br br-phys + ovn_attach n1 br-phys 192.168.0.$i + ovs-appctl -t ovn-controller vlog/set dbg + + for j in {1..3}; do + check ovn-nbctl lsp-add sw0 sw0-h$i-p$j + check ovn-nbctl lsp-set-addresses sw0-h$i-p$j "50:54:00:00:00:$i$j 10.0.0.$i$j 1000::$i$j" + check ovn-nbctl lsp-set-port-security sw0-h$i-p$j "50:54:00:00:00:$i$j 10.0.0.$i$j 10.0.0.120 1000::$i$j" + parents+=$"sw0-h$i-p$j," + ovs-vsctl -- add-port br-int vif$i$j -- \ + set interface vif$i$j \ + external-ids:iface-id=sw0-h$i-p$j \ + options:tx_pcap=hv$i/vif$i$j-tx.pcap \ + options:rxq_pcap=hv$i/vif$i$j-rx.pcap \ + ofport-request=$i$j + done +done + +ovs-vsctl -- add-port br-int vifsw1 -- \ + set interface vifsw1 \ + external-ids:iface-id=sw1-p1 \ + options:tx_pcap=hv$i/vifsw1-tx.pcap \ + options:rxq_pcap=hv$i/vifsw1-rx.pcap \ + ofport-request=122 + +check ovn-nbctl lsp-add sw0 sw0-vir +check ovn-nbctl lsp-set-addresses sw0-vir "50:54:00:00:10:10 10.0.0.120" +check ovn-nbctl lsp-set-port-security sw0-vir "50:54:00:00:10:10 10.0.0.120" +check ovn-nbctl lsp-set-type sw0-vir virtual +check ovn-nbctl set logical_switch_port sw0-vir options:virtual-ip=10.0.0.120 +check ovn-nbctl set logical_switch_port sw0-vir options:virtual-parents=$parents + +# Add an ACL that matches on sw0-vir being bound locally. +check ovn-nbctl acl-add sw0 to-lport 1000 'is_chassis_resident("sw0-vir") && ip4' allow +OVN_POPULATE_ARP +wait_for_ports_up + +# Start sending many Garp requests on randomly selected ports and chassis +# to pressure the SB/Northd +eth_dst=ffffffffffff +spa=$(ip_to_hex 10 0 0 120) +tpa=$(ip_to_hex 10 0 0 120) +while : ; do + random_hv=$(shuf -i 1-9 -n 1) + random_port=$(shuf -i 1-3 -n 1) + eth_src=5054000000$random_hv$random_port + send_garp $random_hv $random_port $eth_src $eth_dst $spa $tpa + sleep 0.2 +done & +pid1=$! + +OVS_WAIT_UNTIL([test 0 != `grep -c "Pausing virtual port sw0-vir from sending binding requests for few seconds." northd/ovn-northd.log`]) +# Kill the Loop before exiting otherwise the subshell will keeps +# try to send pkts on HVs ovs interfaces which been cleaned and deleted +# and that will casue test to failed. +kill -9 $pid1 +wait $pid1 + +for i in {1..9}; do + OVN_CLEANUP_SBOX([hv$i]) +done +AT_CLEANUP +]) + OVN_FOR_EACH_NORTHD([ # Run ovn-nbctl in daemon mode, change to a backup database and verify that # an insert operation is not allowed.