From patchwork Wed Dec 20 14:31:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mohammad Heib X-Patchwork-Id: 1878554 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=c8LMG8Hv; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::137; helo=smtp4.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp4.osuosl.org (smtp4.osuosl.org [IPv6:2605:bc80:3010::137]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4SwGG64t2qz23dF for ; Thu, 21 Dec 2023 01:31:26 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id 415C2420E8; Wed, 20 Dec 2023 14:31:24 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 415C2420E8 Authentication-Results: smtp4.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=c8LMG8Hv X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Q7cD3ct7xfJ1; Wed, 20 Dec 2023 14:31:23 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTPS id 0FD3142079; Wed, 20 Dec 2023 14:31:22 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp4.osuosl.org 0FD3142079 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 0C0D8C0DD2; Wed, 20 Dec 2023 14:31:20 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 5F6EDC0037 for ; Wed, 20 Dec 2023 14:31:18 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 37A78436E4 for ; Wed, 20 Dec 2023 14:31:18 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 37A78436E4 Authentication-Results: smtp2.osuosl.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=c8LMG8Hv X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4Y9gXIKRmLgB for ; Wed, 20 Dec 2023 14:31:13 +0000 (UTC) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id 8FFDE436E3 for ; Wed, 20 Dec 2023 14:31:13 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 8FFDE436E3 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1703082672; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zxJe8sYQdakIZaHUiAxMQvSS3Uhoq4Jm0hTt2YF9S8I=; b=c8LMG8Hvhp6Y3tv6R8AWTtvpfmzMyYbuFN0S+eYO6O4ID2WYmrX8NJIVWV91y3dJuX4EBB sYwB4T4KOKL27kQ44ciVdP3RzLtpEX01+CwgsziB4nIe9HBjs+ZmDTGTie+DsmLGeeFQYb MYIKIsBNppW95ddak5z++0HIEOBVhH0= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-46-NLT63PFhO06Mx3ILVIA6aw-1; Wed, 20 Dec 2023 09:31:11 -0500 X-MC-Unique: NLT63PFhO06Mx3ILVIA6aw-1 Received: from smtp.corp.redhat.com (int-mx10.intmail.prod.int.rdu2.redhat.com [10.11.54.10]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E0F771C37FA0 for ; Wed, 20 Dec 2023 14:31:10 +0000 (UTC) Received: from mheiblap.localdomain (unknown [10.47.238.119]) by smtp.corp.redhat.com (Postfix) with ESMTP id 40E4C492BE6; Wed, 20 Dec 2023 14:31:10 +0000 (UTC) From: Mohammad Heib To: dev@openvswitch.org Date: Wed, 20 Dec 2023 16:31:03 +0200 Message-Id: <20231220143105.916908-2-mheib@redhat.com> In-Reply-To: <20231220143105.916908-1-mheib@redhat.com> References: <20231220143105.916908-1-mheib@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.10 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH ovn 2/4] ovn-ic: implement basic INB change handling status X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" This patch implements a basic sequence number protocol that can be used by CMS to determine if the changes applied to INB are successfully propagated to ISB. The implementation of this patch relies on OVN-ICs instances to update the ISB by adding a per AZ a nb_ic_cfg counter that will be updated by the OVN-IC once it is done and commit all needed changes to the ISB, and according to this AZ:nb_ic_cfg the ISB and INB will be updating about the status of the changes. Signed-off-by: Mohammad Heib --- ic/ovn-ic.c | 98 +++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 92 insertions(+), 6 deletions(-) diff --git a/ic/ovn-ic.c b/ic/ovn-ic.c index 12e2729ce..b08b0e2d4 100644 --- a/ic/ovn-ic.c +++ b/ic/ovn-ic.c @@ -1786,16 +1786,96 @@ route_run(struct ic_context *ctx, hmap_destroy(&ic_lrs); } +/* + * This function implements a sequence number protocol that can be used by + * the INB end user to verify that ISB is synced with all the changes that + * are done be the user/AZs-controllers: + * + * Since we have multiple IC instances running in different regions + * we can't rely on one of them to update the ISB and sync that update + * to INB since other ICs can make changes in parallel. + * So to have a sequence number protocol working properly we must + * make sure that all the IC instances are synced with the ISB first + * and then update the INB. + * + * To guarantee that all instances are synced with ISB first, each IC + * will do the following steps: + * + * 1. when local ovn-ic sees that INB:nb_ic_cfg has updated we will set + * the ic_sb_loop->next_cfg to match the INB:nb_ic_cfg and increment + * the value of AZ:nb_ic_cfg and wait until we get confirmation from + * the server. + * + * 2. once this IC instance changes for ISB are committed successfully + * (next loop), the value of cur_cfg will be updated to match + * the INB:nb_ic_cfg that indicate that our local instance is up to date + * and no more changes need to be done for ISB. + * + * 3. validate that the AZ:nb_ic_cfg to match the INB:nb_ic_cfg. + * + * 4. Go through all the AZs and check if all have the same value of + * AZ:nb_ic_cfg that means all the AZs are done with ISB changes and ISB are + * up to date with INB, so we can set the values of ISB:nb_ic_cfg to + * INB:nb_ic_cfg and INB:sb_ic_cfg to INB:nb_ic_cfg. + */ static void -ovn_db_run(struct ic_context *ctx) +update_sequence_numbers(const struct icsbrec_availability_zone *az, + struct ic_context *ctx, + struct ovsdb_idl_loop *ic_sb_loop) { - const struct icsbrec_availability_zone *az = az_run(ctx); - VLOG_DBG("Availability zone: %s", az ? az->name : "not created yet."); + bool azs_cfg_equals = true; - if (!az) { - return; + if (ctx->ovnisb_txn && ctx->ovninb_txn) { + const struct icnbrec_ic_nb_global *ic_nb = icnbrec_ic_nb_global_first( + ctx->ovninb_idl); + if (!ic_nb) { + ic_nb = icnbrec_ic_nb_global_insert(ctx->ovninb_txn); + } + const struct icsbrec_ic_sb_global *ic_sb = icsbrec_ic_sb_global_first( + ctx->ovnisb_idl); + if (!ic_sb) { + ic_sb = icsbrec_ic_sb_global_insert(ctx->ovnisb_txn); + } + + if ((ic_nb->nb_ic_cfg != ic_sb->nb_ic_cfg) && + (ic_nb->nb_ic_cfg != az->nb_ic_cfg)) { + /* Deal with potential overflows. */ + if (az->nb_ic_cfg == LLONG_MAX) { + icsbrec_availability_zone_set_nb_ic_cfg(az, 0); + } + ic_sb_loop->next_cfg = ic_nb->nb_ic_cfg; + ovsdb_idl_txn_increment(ctx->ovnisb_txn, &az->header_, + &icsbrec_availability_zone_col_nb_ic_cfg, true); + goto done; + } + + /* handle cases where accidentally AZ:ic_nb_cfg exceeds + * the INB:ic_nb_cfg. + */ + if (az->nb_ic_cfg != ic_sb_loop->cur_cfg) { + icsbrec_availability_zone_set_nb_ic_cfg(az, ic_sb_loop->cur_cfg); + goto done; + } + + const struct icsbrec_availability_zone *other_az; + ICSBREC_AVAILABILITY_ZONE_FOR_EACH (other_az, ctx->ovnisb_idl) { + if (other_az->nb_ic_cfg != az->nb_ic_cfg) { + azs_cfg_equals = false; + } + } + + if (azs_cfg_equals && (ic_nb->nb_ic_cfg != ic_sb->nb_ic_cfg)) { + icsbrec_ic_sb_global_set_nb_ic_cfg(ic_sb, az->nb_ic_cfg); + icnbrec_ic_nb_global_set_sb_ic_cfg(ic_nb, az->nb_ic_cfg); + } } + done: ; +} +static void +ovn_db_run(struct ic_context *ctx, + const struct icsbrec_availability_zone *az) +{ ts_run(ctx); gateway_run(ctx, az); port_binding_run(ctx, az); @@ -2222,7 +2302,13 @@ main(int argc, char *argv[]) ovsdb_idl_has_ever_connected(ctx.ovnsb_idl) && ovsdb_idl_has_ever_connected(ctx.ovninb_idl) && ovsdb_idl_has_ever_connected(ctx.ovnisb_idl)) { - ovn_db_run(&ctx); + const struct icsbrec_availability_zone *az = az_run(&ctx); + VLOG_DBG("Availability zone: %s", az ? az->name : + "not created yet."); + if (az) { + ovn_db_run(&ctx, az); + update_sequence_numbers(az, &ctx, &ovnisb_idl_loop); + } } int rc1 = ovsdb_idl_loop_commit_and_wait(&ovnnb_idl_loop);