From patchwork Tue Dec 6 15:34:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frode Nordahl X-Patchwork-Id: 1712867 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=Pq5/Fbv2; dkim-atps=neutral Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-384) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4NRPcD0ggdz23yq for ; Wed, 7 Dec 2022 02:34:51 +1100 (AEDT) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 2758961015; Tue, 6 Dec 2022 15:34:50 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 2758961015 Authentication-Results: smtp3.osuosl.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=canonical.com header.i=@canonical.com header.a=rsa-sha256 header.s=20210705 header.b=Pq5/Fbv2 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WJ_da1Dv8N5T; Tue, 6 Dec 2022 15:34:49 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id EA49160F86; Tue, 6 Dec 2022 15:34:47 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org EA49160F86 Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id BF193C0032; Tue, 6 Dec 2022 15:34:47 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 94BC6C002D for ; Tue, 6 Dec 2022 15:34:45 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 62A86608B7 for ; Tue, 6 Dec 2022 15:34:45 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org 62A86608B7 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3OozG4sPq6K7 for ; Tue, 6 Dec 2022 15:34:44 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp3.osuosl.org CB46760640 Received: from smtp-relay-canonical-0.canonical.com (smtp-relay-canonical-0.canonical.com [185.125.188.120]) by smtp3.osuosl.org (Postfix) with ESMTPS id CB46760640 for ; Tue, 6 Dec 2022 15:34:43 +0000 (UTC) Received: from frode-threadripper.. (2.general.frode.uk.vpn [10.172.193.251]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by smtp-relay-canonical-0.canonical.com (Postfix) with ESMTPSA id 7C622423DE; Tue, 6 Dec 2022 15:34:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=canonical.com; s=20210705; t=1670340877; bh=hbv69f4w7F8i2S8z22JM+M5KbbLQ8CHAe83P0m2VZIE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Pq5/Fbv2206nY/5tZIboYMjOkuWuSlkIICsWfbBZrKSuM9RTcflmNIpIHihzX5UFD yd/xXswQ6g1SqlIkpU8mqabjZppDiqDe1MJV5J6n9iif/h5+yUhYRWexdsxFF8v592 TIYSgxfVztpwmnrQ3fk3WkyYX7eQ1E4TDXNZ7dVVjAicbcfEoRjdt6uLU/ygTD0GRN 5Yy1zji2cqxdqfK2cyWK1NnQzwEN7PfyMnt6y/WE8fN50GwysAhrLdsi7ApEyAVY+z 6f/JS4B//X5imm7g8e+fGRsFlf00R8pVZo9PJkEVO9BWUrclgSFe6GeoQYgW9gY8ff 7czKbQ9AgsdgA== From: Frode Nordahl To: dev@openvswitch.org Date: Tue, 6 Dec 2022 16:34:35 +0100 Message-Id: <20221206153435.4140159-1-frode.nordahl@canonical.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <53bde97a-c7dd-739e-c6f4-53fbd8fed335@redhat.com> References: <53bde97a-c7dd-739e-c6f4-53fbd8fed335@redhat.com> MIME-Version: 1.0 Cc: Dumitru Ceara Subject: [ovs-dev] [PATCH ovn v2] docs: Extend upgrade documentation. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" As uncovered during the OVSCON'22 open discussion forum on upgrades, there are some challenges in upgrading from older versions of OVN. Document version requirements for performing a controller first rolling upgrade. Add a section about how to perform a fail-safe upgrade for deployments that want to upgrade beyond a supported version span. Reported-at: https://bugs.launchpad.net/bugs/1940043 Signed-off-by: Frode Nordahl Acked-by: Dumitru Ceara Acked-by: Mark Michelson Acked-by: Han Zhou --- Documentation/intro/install/ovn-upgrades.rst | 92 ++++++++++++++++++-- 1 file changed, 84 insertions(+), 8 deletions(-) diff --git a/Documentation/intro/install/ovn-upgrades.rst b/Documentation/intro/install/ovn-upgrades.rst index 4c131987e..ed84bb5f0 100644 --- a/Documentation/intro/install/ovn-upgrades.rst +++ b/Documentation/intro/install/ovn-upgrades.rst @@ -27,7 +27,12 @@ OVN Upgrades Since OVN is a distributed system, special consideration must be given to the process used to upgrade OVN across a deployment. This document discusses -the recommended upgrade process. +the two recommended `Upgrade procedures`_, `Rolling upgrade`_ and `Fail-safe +upgrade`_. + +Which one to choose depends on whether you are running a version of OVN that is +within range of upstream support for upgrades to the version of OVN you want to +upgrade to. Release Notes ------------- @@ -43,21 +48,84 @@ upgraded together, partly for convenience. OVN is included in OVS releases so it's easiest to upgrade them together. OVN may also make use of new features of OVS only available in that release. +Upgrade procedures +------------------ + +Rolling upgrade +~~~~~~~~~~~~~~~ + +1. `Upgrade ovn-controller`_ + +2. `Upgrade OVN Databases and ovn-northd`_ + +3. `Upgrade OVN Integration`_ + +In order to successfully perform a rolling upgrade, the ovn-controller process +needs to understand the structure of the database for the version you are +upgrading from and to simultaneously. + +To avoid buildup of complexity and technical debt we limit the span of versions +supported for a rolling upgrade on `Long-term Support Releases`_ (LTS), and it +should always be possible to upgrade from the previous LTS version to the next. + +The first LTS version of OVN was 22.03. If you want to upgrade between other +versions, you can use the `Fail-safe upgrade`_ procedure. + +Fail-safe upgrade +~~~~~~~~~~~~~~~~~ + +1. Upgrade to the most recent point release or package version available for + the major version of OVN you are upgrading from. + +2. Enable the version pinning feature in the ovn-controller by setting the + ``external_ids:ovn-match-northd-version`` flag to 'true' as documented in + the `ovn-controller man page`_. + +3. If the version of OVN you are upgrading from does not have the `version + pinning check in the incremental processing engine`_, you must stop + ovn-northd and manually change the northd_internal_version to ensure the + controllers go into fail-safe mode before processing changes induced by the + upgrade. + + $ sudo /usr/share/ovn/scripts/ovn-ctl stop_northd --ovn-manage-ovsdb=no + $ sudo ovn-sbctl set sb-global . options:northd_internal_version="foo" + +4. `Upgrade OVN Databases and ovn-northd`_ + +5. `Upgrade ovn-controller`_ + +6. `Upgrade OVN Integration`_ + +When upgrading between a span of versions that is not supported, you may be at +risk for the new ovn-controller process not understanding the structure of the +old database, which may lead to data plane downtime for running instances. + +To avoid this there is a fail safe approach, which involves making the +ovn-controller process refrain from making changes to the local flow state when +a version mismatch between the ovn-controller and ovn-northd is detected. + +Steps +----- + +This section documents individual steps in a upgrade procedure in no particular +order. For information on ordering of the steps, please refer to the `Upgrade +procedures`_ section. + Upgrade ovn-controller ----------------------- +~~~~~~~~~~~~~~~~~~~~~~ You should start by upgrading ovn-controller on each host it's running on. First, you upgrade the OVS and OVN packages. Then, restart the ovn-controller service. You can restart with ovn-ctl:: - $ sudo /usr/share/openvswitch/scripts/ovn-ctl restart_controller + $ sudo /usr/share/ovn/scripts/ovn-ctl restart_controller or with systemd:: $ sudo systemd restart ovn-controller Upgrade OVN Databases and ovn-northd ------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The OVN databases and ovn-northd should be upgraded next. Since ovn-controller has already been upgraded, it will be ready to operate on any new functionality @@ -69,14 +137,14 @@ automatically restarts the databases and upgrades the database schema, as well. You may perform this restart using the ovn-ctl script:: - $ sudo /usr/share/openvswitch/scripts/ovn-ctl restart_northd + $ sudo /usr/share/ovn/scripts/ovn-ctl restart_northd or if you're using a Linux distribution with systemd:: $ sudo systemctl restart ovn-northd Schema Change -^^^^^^^^^^^^^ ++++++++++++++ During database upgrading, if there is schema change, the DB file will be converted to the new schema automatically, if the schema change is backward @@ -103,8 +171,8 @@ of known impactible schema changes and how to fix when error encountered. $ ovn-sbctl chassis-del -Upgrading OVN Integration -------------------------- +Upgrade OVN Integration +~~~~~~~~~~~~~~~~~~~~~~~ Lastly, you may also want to upgrade integration with OVN that you may be using. For example, this could be the OpenStack Neutron driver or @@ -113,3 +181,11 @@ ovn-kubernetes. OVN's northbound database schema is a backwards compatible interface, so you should be able to safely complete an OVN upgrade before upgrading any integration in use. + +.. LINKS +.. _Long-term Support Releases: + ../../internals/release-process.html#long-term-support-releases +.. _ovn-controller man page: + https://www.ovn.org/support/dist-docs/ovn-controller.8.html +.. _version pinning check in the incremental processing engine: + https://github.com/ovn-org/ovn/commit/c2eeb2c98ea8