[ovs-dev,v3,12/12] documentation: Document ovs-flowviz.

Add a man page for ovs-flowviz as well as a topic page with some more
detailed examples.

Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
---
 Documentation/automake.mk                   |   4 +-
 Documentation/conf.py                       |   2 +
 Documentation/ref/index.rst                 |   1 +
 Documentation/ref/ovs-flowviz.8.rst         | 531 ++++++++++++++++++++
 Documentation/topics/flow-visualization.rst | 271 ++++++++++
 Documentation/topics/index.rst              |   1 +
 6 files changed, 809 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/ref/ovs-flowviz.8.rst
 create mode 100644 Documentation/topics/flow-visualization.rst

On 4/9/24 09:06, Adrian Moreno wrote:
> Add a man page for ovs-flowviz as well as a topic page with some more
> detailed examples.
> 
> Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
> ---
>   Documentation/automake.mk                   |   4 +-
>   Documentation/conf.py                       |   2 +
>   Documentation/ref/index.rst                 |   1 +
>   Documentation/ref/ovs-flowviz.8.rst         | 531 ++++++++++++++++++++
>   Documentation/topics/flow-visualization.rst | 271 ++++++++++
>   Documentation/topics/index.rst              |   1 +
>   6 files changed, 809 insertions(+), 1 deletion(-)
>   create mode 100644 Documentation/ref/ovs-flowviz.8.rst
>   create mode 100644 Documentation/topics/flow-visualization.rst
> 


Checkpatch raises some warnings on this patch. Most of them are about line 
length which I think can be ignored because it's a rST file and they are command 
line outputs which, if wrapped, would be way more unreadable.

However, there is also this:

WARNING: New doc ovs-flowviz.8.rst not listed in Documentation/automake.mk
Lines checked: 896, Warnings: 50, Errors: 0

I've identified this as a bug in checkpatch.py, for which a patch is on its way.

--
Adrián

> diff --git a/Documentation/automake.mk b/Documentation/automake.mk
> index 47d2e336a..539870aa2 100644
> --- a/Documentation/automake.mk
> +++ b/Documentation/automake.mk
> @@ -45,7 +45,7 @@ DOC_SOURCE = \
>   	Documentation/topics/fuzzing/ovs-fuzzing-infrastructure.rst \
>   	Documentation/topics/fuzzing/ovs-fuzzers.rst \
>   	Documentation/topics/fuzzing/security-analysis-of-ovs-fuzzers.rst \
> -	Documentation/topics/testing.rst \
> +	Documentation/topics/flow-visualization.rst \
>   	Documentation/topics/integration.rst \
>   	Documentation/topics/language-bindings.rst \
>   	Documentation/topics/networking-namespaces.rst \
> @@ -55,6 +55,7 @@ DOC_SOURCE = \
>   	Documentation/topics/ovsdb-replication.rst \
>   	Documentation/topics/porting.rst \
>   	Documentation/topics/record-replay.rst \
> +	Documentation/topics/testing.rst \
>   	Documentation/topics/tracing.rst \
>   	Documentation/topics/usdt-probes.rst \
>   	Documentation/topics/userspace-checksum-offloading.rst \
> @@ -162,6 +163,7 @@ RST_MANPAGES = \
>   	ovs-actions.7.rst \
>   	ovs-appctl.8.rst \
>   	ovs-ctl.8.rst \
> +	ovs-flowviz.8.rst \
>   	ovs-l3ping.8.rst \
>   	ovs-parse-backtrace.8.rst \
>   	ovs-pki.8.rst \
> diff --git a/Documentation/conf.py b/Documentation/conf.py
> index 15785605a..3a82f23a7 100644
> --- a/Documentation/conf.py
> +++ b/Documentation/conf.py
> @@ -120,6 +120,8 @@ _man_pages = [
>        u'utility for configuring running Open vSwitch daemons'),
>       ('ovs-ctl.8',
>        u'OVS startup helper script'),
> +    ('ovs-flowviz.8',
> +     u'utility for visualizing OpenFlow and datapath flows'),
>       ('ovs-l3ping.8',
>        u'check network deployment for L3 tunneling problems'),
>       ('ovs-parse-backtrace.8',
> diff --git a/Documentation/ref/index.rst b/Documentation/ref/index.rst
> index 03ada932f..7f2fe6177 100644
> --- a/Documentation/ref/index.rst
> +++ b/Documentation/ref/index.rst
> @@ -42,6 +42,7 @@ time:
>      ovs-actions.7
>      ovs-appctl.8
>      ovs-ctl.8
> +   ovs-flowviz.8
>      ovs-l3ping.8
>      ovs-pki.8
>      ovs-sim.1
> diff --git a/Documentation/ref/ovs-flowviz.8.rst b/Documentation/ref/ovs-flowviz.8.rst
> new file mode 100644
> index 000000000..b5b1befb0
> --- /dev/null
> +++ b/Documentation/ref/ovs-flowviz.8.rst
> @@ -0,0 +1,531 @@
> +..
> +      Licensed under the Apache License, Version 2.0 (the "License"); you may
> +      not use this file except in compliance with the License. You may obtain
> +      a copy of the License at
> +
> +          http://www.apache.org/licenses/LICENSE-2.0
> +
> +      Unless required by applicable law or agreed to in writing, software
> +      distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
> +      WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
> +      License for the specific language governing permissions and limitations
> +      under the License.
> +
> +      Convention for heading levels in Open vSwitch documentation:
> +
> +      =======  Heading 0 (reserved for the title in a document)
> +      -------  Heading 1
> +      ~~~~~~~  Heading 2
> +      +++++++  Heading 3
> +      '''''''  Heading 4
> +
> +      Avoid deeper levels because they do not render well.
> +
> +===========
> +ovs-flowviz
> +===========
> +
> +Synopsis
> +========
> +
> +``ovs-flowviz``
> +[``[-i | --input] <[alias,]file>``]
> +[``[-c | --config] <file>``]
> +[``[-f | --filter] <filter>``]
> +[``[-h | --highlight] <filter>``]
> +[``--style <style>``]
> +*<flow_type>* *<format>* [<arg>...]
> +
> +``ovs-flowviz --help``
> +
> +Description
> +===========
> +
> +The ``ovs-flowviz`` program helps visualize OpenFlow and datapath flow dumps
> +in different formats in order to make them more easily understood.
> +
> +The program works by reading flows from ``stdin`` or from a file specified
> +in the ``--input`` option, filtering them, highlighting them, and finally
> +outputting them in one of the predefined formats.
> +
> +
> +Options
> +=======
> +
> +.. program: ovs-flowviz
> +
> +.. option:: -h, --help
> +
> +    Prints a brief help message to the console.
> +
> +.. option:: -i <[alias,]file>, --input <[alias,]file>
> +
> +    Specifies the file to read flows from. If not provided, ``ovs-flowviz``
> +    will read flows from stdin.
> +
> +    This option can be specified multiple times.
> +    The file path can prepended by an alias that will be shown in the output.
> +    For example: ``--input node1,/path/to/file1 --input node2,/path/to/file2``
> +
> +.. option:: -c <file>, --config <file>
> +
> +    Specifies the style configuration file to use. ``ovs-flowviz`` ships with
> +    a default configuration file but it can be overridden using this option.
> +    Styles defined in the style configuration file will be select-able using
> +    the ``--style`` option.
> +
> +    For more details on the style configuration file, see
> +    `Style Configuration File`_ section below.
> +
> +.. option:: -f <filter>, --filter <filter>
> +
> +   Tells ``ovs-flowviz`` to filter the flows and only show the ones that match
> +   the expression (although some formats implement filtering differently,
> +   see `Datapath tree format`_ below).
> +
> +   The filtering syntax is detailed in `Filtering Syntax`_.
> +
> +.. option:: -h <filter>, --highlight <filter>
> +
> +   Tells ``ovs-flowviz`` to highlight the flows that match the provided filter
> +
> +   The filtering syntax is detailed in `Filtering Syntax`_.
> +
> +.. option:: --style <style>
> +
> +   Specifies the style to use. The style must have been defined in the
> +   style configuration file.
> +
> +.. option:: <flow_type>
> +
> +   "openflow" or "datapath".
> +
> +.. option:: <format>
> +
> +   See `Supported formats`_ section.
> +
> +
> +Supported formats
> +=================
> +
> +``ovs-flowviz`` supports several visualization formats for both OpenFlow and
> +datapath flows that are summarized in the following table:
> +
> +.. list-table::
> +   :widths: 20 10 70
> +   :align: center
> +   :header-rows: 1
> +
> +   * - Flow Type
> +     - Format
> +     - Description
> +   * - Both
> +     - console
> +     - Prints the flows in a configurable, colorful style in the console.
> +   * - Both
> +     - json
> +     - Prints the flows in JSON format.
> +   * - Both
> +     - html
> +     - Prints the flows in an HTML list.
> +   * - Openflow
> +     - cookie
> +     - Prints the flows in the console sorted by cookie.
> +   * - Openflow
> +     - logic
> +     - Prints the logical structure of flows in the console.
> +   * - Datapath
> +     - tree
> +     - Prints the flows a tree structure arranged by `recirc_id`.
> +   * - Datapath
> +     - graph
> +     - Prints a graphviz graph of the flows arranged by `recirc_id`.
> +
> +
> +Console format
> +~~~~~~~~~~~~~~
> +
> +The ``console`` works for both OpenFlow and datapath flow types and prints
> +flows in the terminal with the style determined by the ``--style`` option.
> +
> +Additionally, it accepts the following arguments:
> +
> +.. option:: -h, --heat-map
> +
> +   This option changes the color of the packet and byte counters to reflect
> +   their relative size. The color gradient goes through the following colors:
> +   blue (coldest, lowest), cyan, green, yellow, red (hottest, highest)
> +
> +   Note filtering is applied before the range is calculated.
> +
> +
> +JSON format
> +~~~~~~~~~~~
> +
> +The ``json`` format works for both OpenFlow and datapath flow types and prints
> +flows in JSON format. See `JSON Syntax`_ for more details.
> +
> +
> +HTML format
> +~~~~~~~~~~~
> +
> +The ``html`` format works for both OpenFlow and datapath flows and prints
> +flows in an HTML table that offers some basic interactivity. OpenFlow flows
> +are sorted in tables and datapath flows are arranged in flow trees
> +(see `Datapath tree format`_ for more details).
> +
> +Styles defined via Style Configuration File and selected via ``--style`` option
> +also apply to ``html`` format.
> +
> +
> +OpenFlow cookie format
> +~~~~~~~~~~~~~~~~~~~~~~
> +
> +The OpenFlow ``cookie`` format is similar to the ``console`` format but
> +instead of arranging the flows per table, it arranges the flows per cookie.
> +
> +
> +Openflow logic format
> +~~~~~~~~~~~~~~~~~~~~~
> +
> +The OpenFlow ``logic`` format helps visualize the logic structure of OpenFlow
> +pipelines by arranging flows into *logical blocks*.
> +A logical block is a set of flows that have:
> +
> +* Same ``priority``.
> +* Match on the same fields (regardless of the match value and mask).
> +* Execute the same actions (regardless of the actions' arguments,
> +  except for resubmit and output).
> +* Optionally, the ``cookie`` can be counted as part of the logical flow.
> +
> +This format supports the following extra arguments:
> +
> +.. option:: -s, --show-flows
> +
> +    Show all the flows under each logical block.
> +
> +.. option:: -d, --ovn-detrace
> +
> +    Use ovn-detrace.py script to extract cookie information (implies '-c').
> +
> +.. option:: -c, --cookie
> +
> +    Consider the cookie in the logical block.
> +
> +.. option:: --ovn-detrace-path <path>
> +
> +    Use an alternative path to look for ovn_detrace.py script.
> +
> +.. option:: --ovnnb-db text
> +
> +   Specify the OVN NB database string (implies '-d').
> +   Default value is "unix:/var/run/ovn/ovnnb_db.sock".
> +
> +.. option:: --ovnsb-db text
> +
> +   Specify the OVN SB database string (implies '-d').
> +   Default value is "unix:/var/run/ovn/ovnsb_db.sock".
> +
> +.. option:: --o <text>, --ovn-filter <text>
> +
> +   Specify the a filter to be run on the ovn-detrace information.
> +   Syntax: python regular expression
> +   (See https://docs.python.org/3/library/re.html).
> +
> +.. option:: -h, --heat-map
> +
> +   This option changes the color of the packet and byte counters to reflect
> +   their relative size. The color gradient goes through the following colors:
> +   blue (coldest, lowest), cyan, green, yellow, red (hottest, highest)
> +
> +   Note filtering is applied before the range is calculated.
> +
> +
> +Datapath tree format
> +~~~~~~~~~~~~~~~~~~~~
> +
> +The datapath ``tree`` format arranges datapath flows in a hierarchical tree
> +based on `recirc_id`. At the first level, flows with `recirc_id(0)` are
> +listed. If a flow contains a `recirc()` action with a specific `recirc_id`,
> +flows matching on that `recirc_id` are listed below. This is done recursively
> +for all actions.
> +
> +The result is a hierarchical representation that helps understand how actions
> +are related to each other via recirculation. Note flows with a specific
> +non-zero `recirc_id` are listed below each flow that has a corresponding
> +`recirc()` action. Therefore, they would be duplicated leading to a longer
> +output.
> +
> +Also, filtering works in a slightly different way for datapath flow trees.
> +Unlike other formats where a filter simply removes non-matching flows,
> +the output of a filtered datapath flow tree will show full sub-trees
> +that contain at least one flow that satisfies the filter.
> +
> +The ``html`` format prints this same tree in an interactive HTML table.
> +
> +
> +Datapath graph format
> +~~~~~~~~~~~~~~~~~~~~~
> +
> +The datapath ``graph`` generates a graphviz visual representation of the
> +same tree-like flow hierarchy that the ``tree`` format prints.
> +
> +It supports the following extra argument:
> +
> +.. option:: -h, --html
> +
> +    Prints the graphviz format in an svg image alongside the interactive HTML
> +    table of flows (that 'html' format would print).
> +
> +
> +JSON Syntax
> +===========
> +
> +Both OpenFlow and datapath `json` formats print a JSON list of JSON
> +objects each of one representing an individual flow.S
> +
> +Each flow object contains the following keys:
> +
> +**orig**
> +    Contains the original flow string.
> +
> +
> +**info**
> +   Contains an object with the flow information
> +   such as: cookie, duration, table, n_packets, n_bytes, etc.
> +
> +
> +**match**
> +   Contains an object with the flow match.
> +   For each match, the object contains a key-value where the key is the name
> +   of the match as defined in ovs-fields and ovs-ofctl and the value
> +   represents the match value. The way each value is represented depends on its
> +   type. See `Value representation`_.
> +
> +
> +**actions**
> +   Contains a list of action objects.
> +   Each action is represented by an JSON object that has one key and one value.
> +   The key corresponds to the action name. The value represents the arguments
> +   of such key. See `Action representation`_.
> +
> +
> +**ufid**
> +   (datapath flows only) Contains the ufid.
> +
> +
> +Value representation
> +~~~~~~~~~~~~~~~~~~~~
> +
> +Values are represented differently depending on their type:
> +
> +* Flags: Fields that represent flags (e.g: tcp) are represented by boolean
> +  "true"
> +
> +* Decimal / Hexadecimal: They are represented by their integer value.
> +  If they support masking, they are represented by a dictionary with two keys:
> +  value contains the field value and mask contains the mask. Both are integers.
> +
> +* Ethernet: They are represented by a string: {address}[/{mask}]
> +
> +* IPv4 / IPv6: They are represented by a string {address}[/mask]
> +
> +* Registers: They are represented by a dictionary with three keys:
> +  field contains the field value (string), start and end that represent the
> +  first and last bit of the register.
> +
> +For example, the register
> +::
> +
> +
> +   NXM_NX_REG10[0..15]
> +
> +
> +is represented as
> +::
> +
> +
> +   {
> +       "field": "NXM_NX_REG10",
> +       "start": 0,
> +       "end": 15
> +   },
> +
> +
> +Action representation
> +~~~~~~~~~~~~~~~~~~~~~
> +
> +Actions are generally represented by an object that has a single key and a
> +value. The key is the action name as defined ovs-actions.
> +
> +The value of actions that have no arguments (such as ``drop``) is
> +(boolean) ``true``.
> +
> +The value of actions that have a list of arguments (e.g:
> +``resubmit([port],[table],[ct])``) is an object that has the name of the
> +argument as key. The argument names for each action is defined in
> +ovs-actions. For example, the action
> +::
> +
> +   resubmit(,10)
> +
> +is represented as
> +::
> +
> +   {
> +       "redirect": {
> +           "port": "",
> +           "table": 10
> +       }
> +   }
> +
> +The value of actions that have a key-word list as arguments
> +(e.g: ``ct([argument])``) is an object whose keys correspond to the keys
> +defined in ``ovs-actions(7)``. The way values are represented depends
> +on the type of the argument.
> +For example, the action
> +::
> +
> +   ct(table=14,zone=NXM_NX_REG12[0..15],nat)
> +
> +is represented as
> +::
> +
> +   {
> +       "ct": {
> +           "table": 14,
> +           "zone": {
> +               "field": "NXM_NX_REG12",
> +               "start": 0,
> +               "end": 15
> +           },
> +           "nat": true
> +       }
> +   }
> +
> +
> +Style Configuration File
> +========================
> +
> +The style configuration file that can be selected via the ``--config`` option
> +has INI syntax and can define any number of styles to be used by both
> +``console`` and ``html`` formats. Once defined in the configuration file
> +they can be selected using the ``--style`` option.
> +
> +INI sections are used to define styles, ``[styles.mystyle]`` defines a style
> +called `mystle`. Within a section styles can be defined as:
> +
> +::
> +
> +     [FORMAT].[PORTION].[SELECTOR].[ELEMENT] = [VALUE]
> +
> +
> +**FORMAT**
> +   Either ``console`` or ``html``
> +
> +**PORTION**
> +   The part of the a key-value the style applies to. It can be:
> +   ``key`` (to indicate the key part of a key-value), ``value`` (to indicate
> +   the value part of a key-value), ``flag`` (to indicate a single flag)
> +   or ``delim`` (to indicate delimiters such as parentheses, brackets, etc).
> +
> +**SELECTOR**
> +   Is used to select what key-value the style applies to. It can be:
> +   ``highlighted`` (to indicate highlighted key-values), ``type.<type>``
> +   to indicate certain types such as `IPAddress` or `EthMask` or `<keyname>`
> +   to select a particular key name.
> +
> +**ELEMENT**
> +   Is used to select what style element to modify. It can be one
> +   of: **color** or **underline** (only for **console** format).
> +
> +**VALUE**
> +   Is either a color hex, other color names defined in the rich python
> +   library (https://rich.readthedocs.io/en/stable/appendix/colors.html) or
> +   "true" if the element is ``underline``.
> +
> +A default configuration file is shipped with the tool and it's path is printed
> +in the ``--help`` output. A detailed description of the syntax alongside
> +some examples is available there.
> +
> +
> +Filtering syntax
> +================
> +
> +``ovs-flowviz`` provides rich highlighting and filtering. The special command
> +``ovs-flowviz filter`` dumps the filtering syntax:
> +
> +::
> +
> +    $ ovs-flowviz filter
> +    Filter Syntax
> +    *************
> +
> +       [! | not ] {key}[[.subkey]...] [OPERATOR] {value})] [LOGICAL OPERATOR] ...
> +
> +      Comparison operators are:
> +          =   equality
> +          <   less than
> +          >   more than
> +          ~=  masking (valid for IP and Ethernet fields)
> +
> +      Logical operators are:
> +          !{expr}:  NOT
> +          {expr} && {expr}: AND
> +          {expr} || {expr}: OR
> +
> +      Matches and flow metadata:
> +          To compare against a match or info field, use the field directly, e.g:
> +              priority=100
> +              n_bytes>10
> +          Use simple keywords for flags:
> +              tcp and ip_src=192.168.1.1
> +
> +      Actions:
> +          Actions values might be dictionaries, use subkeys to access individual
> +          values, e.g:
> +              output.port=3
> +          Use simple keywords for flags
> +              drop
> +
> +      Examples of valid filters.
> +          nw_addr~=192.168.1.1 && (tcp_dst=80 || tcp_dst=443)
> +          arp=true && !arp_tsa=192.168.1.1
> +          n_bytes>0 && drop=true
> +
> +
> +Example expressions:
> +::
> +
> +   n_bytes > 0 and drop
> +   nw_src~=192.168.1.1 or arp.tsa=192.168.1.1
> +   ! tcp && output.port=2
> +
> +
> +Examples
> +========
> +
> +Print OpenFlow flows sorted by cookie adding OVN data to each one:
> +::
> +
> +    $ ovs-flowviz -i flows.txt openflow cookie --ovn-detrace
> +
> +Print OpenFlow logical structure, showing the flows and heat-map:
> +::
> +
> +    $ ovs-flowviz -i flows.txt openflow logic --show-flows --heat-map
> +
> +Display OpenFlow flows in HTML format with "light" style and highlight drops:
> +::
> +
> +    $ ovs-flowviz -i flows.txt --style "light" --highlight "n_packets > 0 and drop" openflow html > flows.html
> +
> +Display the datapath flows in an interactive graphviz + HTML view:
> +::
> +
> +    $ ovs-flowviz -i flows.txt datapath graph --html > flows.html
> +
> +Display the datapath flow trees that lead to packets being sent to port 10:
> +::
> +
> +    $ ovs-flowviz -i flows.txt --filter "output.port=10" datapath tree
> diff --git a/Documentation/topics/flow-visualization.rst b/Documentation/topics/flow-visualization.rst
> new file mode 100644
> index 000000000..62d0d6bd8
> --- /dev/null
> +++ b/Documentation/topics/flow-visualization.rst
> @@ -0,0 +1,271 @@
> +..
> +      Licensed under the Apache License, Version 2.0 (the "License"); you may
> +      not use this file except in compliance with the License. You may obtain
> +      a copy of the License at
> +
> +          http://www.apache.org/licenses/LICENSE-2.0
> +
> +      Unless required by applicable law or agreed to in writing, software
> +      distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
> +      WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
> +      License for the specific language governing permissions and limitations
> +      under the License.
> +
> +      Convention for heading levels in Open vSwitch documentation:
> +
> +      =======  Heading 0 (reserved for the title in a document)
> +      -------  Heading 1
> +      ~~~~~~~  Heading 2
> +      +++++++  Heading 3
> +      '''''''  Heading 4
> +
> +      Avoid deeper levels because they do not render well.
> +
> +==================================
> +Visualizing flows with ovs-flowviz
> +==================================
> +
> +When troubleshooting networking issues with OVS, we typically end up looking
> +at OpenFlow or datapath flow dumps. These dumps tend to be quite dense and
> +difficult to reason about.
> +
> +``ovs-flowviz`` is a utility script that helps visualizing OpenFlow and
> +datapath flows to make it easier to understand what is going on.
> +
> +The `ovs-flowviz(8)`_ manpage describes its basic usage. In this document a few
> +of its advanced visualization formats will be expanded.
> +
> +
> +Installing ovs-flowviz
> +----------------------
> +
> +``ovs-flowviz`` is part of the openvswitch python package but its
> +extra dependencies have to be installed explicitly by running:
> +::
> +
> +    $ pip install openvswitch[flowviz]
> +
> +Or, if you are working with the OVS tree:
> +::
> +
> +    $ cd python && pip install .[flowviz]
> +
> +Visualizing OpenFlow logical block
> +----------------------------------
> +
> +When controllers such as OVN write OpenFlow flows, they typically organize
> +flows in functional blocks. These blocks can expand to multiple flows that
> +"look similar", in the sense that they match on the same fields and have
> +similar actions.
> +
> +However, when we look at a flow dump the number of flows can make it difficult
> +to perceive this logical functionality that the controller is trying to
> +implement using OpenFlow.
> +
> +In this example, we are going to use ``ovs-flowviz openflow logic``
> +visualization to understand an OVN flow dump a bit better.
> +
> +On a particular flow dump we have 23 flows in table 0:
> +::
> +
> +   $ grep -c "table=0" flows.txt
> +   23
> +
> +If we look at the first few lines, the amount of information can be
> +overwhelming and difficult our analysis:
> +
> +::
> +
> +    $ head flows.txt
> +      cookie=0xf76b4b20, duration=765.107s, table=0, n_packets=0, n_bytes=0, priority=180,vlan_tci=0x0000/0x1000 actions=conjunction(100,2/2)
> +      cookie=0xf76b4b20, duration=765.107s, table=0, n_packets=0, n_bytes=0, priority=180,conj_id=100,in_port="patch-br-int-to",vlan_tci=0x0000/0x1000 actions=load:0xa->NXM_NX_REG13[],load:0xc->NXM_NX_REG11[],load:0xb->NXM_NX_REG12[],load:0xb->OXM_OF_METADATA[],load:0x1->NXM_NX_REG14[],mod_dl_src:02:42:ac:12:00:03,resubmit(,8)
> +      cookie=0x0, duration=765.388s, table=0, n_packets=0, n_bytes=0, priority=100,in_port="ovn-6bb3b3-0" actions=move:NXM_NX_TUN_ID[0..23]->OXM_OF_METADATA[0..23],move:NXM_NX_TUN_METADATA0[16..30]->NXM_NX_REG14[0..14],move:NXM_NX_TUN_METADATA0[0..15]->NXM_NX_REG15[0..15],resubmit(,40)
> +      cookie=0x0, duration=765.388s, table=0, n_packets=0, n_bytes=0, priority=100,in_port="ovn-a6ff98-0" actions=move:NXM_NX_TUN_ID[0..23]->OXM_OF_METADATA[0..23],move:NXM_NX_TUN_METADATA0[16..30]->NXM_NX_REG14[0..14],move:NXM_NX_TUN_METADATA0[0..15]->NXM_NX_REG15[0..15],resubmit(,40)
> +      cookie=0xf2ca6195, duration=765.107s, table=0, n_packets=6, n_bytes=636, priority=100,in_port="ovn-k8s-mp0" actions=load:0x1->NXM_NX_REG13[],load:0x2->NXM_NX_REG11[],load:0x7->NXM_NX_REG12[],load:0x4->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],resubmit(,8)
> +      cookie=0x236e941d, duration=408.874s, table=0, n_packets=11, n_bytes=846, priority=100,in_port=aceac9829941d11 actions=load:0x11->NXM_NX_REG13[],load:0x2->NXM_NX_REG11[],load:0x7->NXM_NX_REG12[],load:0x4->OXM_OF_METADATA[],load:0x3->NXM_NX_REG14[],resubmit(,8)
> +      cookie=0x3facf689, duration=405.581s, table=0, n_packets=11, n_bytes=846, priority=100,in_port="363ba22029cd92b" actions=load:0x12->NXM_NX_REG13[],load:0x2->NXM_NX_REG11[],load:0x7->NXM_NX_REG12[],load:0x4->OXM_OF_METADATA[],load:0x4->NXM_NX_REG14[],resubmit(,8)
> +      cookie=0xe7c8c4bb, duration=405.570s, table=0, n_packets=11, n_bytes=846, priority=100,in_port="6a62cde0d50ef44" actions=load:0x13->NXM_NX_REG13[],load:0x2->NXM_NX_REG11[],load:0x7->NXM_NX_REG12[],load:0x4->OXM_OF_METADATA[],load:0x5->NXM_NX_REG14[],resubmit(,8)
> +      cookie=0x99a0ffc1, duration=59.391s, table=0, n_packets=8, n_bytes=636, priority=100,in_port="5ff3bfaaa4eb622" actions=load:0x14->NXM_NX_REG13[],load:0x2->NXM_NX_REG11[],load:0x7->NXM_NX_REG12[],load:0x4->OXM_OF_METADATA[],load:0x6->NXM_NX_REG14[],resubmit(,8)
> +      cookie=0xe1b5c263, duration=59.365s, table=0, n_packets=8, n_bytes=636, priority=100,in_port="8d9e0bc76347e59" actions=load:0x15->NXM_NX_REG13[],load:0x2->NXM_NX_REG11[],load:0x7->NXM_NX_REG12[],load:0x4->OXM_OF_METADATA[],load:0x7->NXM_NX_REG14[],resubmit(,8)
> +
> +
> +However, we can better understand what table 0 does by looking at its
> +logical representation.
> +::
> +
> +   $ ovs-flowviz -i flows.txt -f "table=0" openflow logic
> +    Ofproto Flows (logical)
> +    └── ** TABLE 0 **
> +        ├── priority=180 priority,vlan_tci  --->  conjunction ( x 1 )
> +        ├── priority=180 priority,conj_id,in_port,vlan_tci  --->  load,load,load,load,load,mod_dl_src resubmit(,8), ( x 1 )
> +        ├── priority=100 priority,in_port  --->  move,move,move resubmit(,40), ( x 2 )
> +        ├── priority=100 priority,in_port  --->  load,load,load,load,load resubmit(,8), ( x 16 )
> +        ├── priority=100 priority,in_port,vlan_tci  --->  load,load,load,load,load resubmit(,8), ( x 1 )
> +        ├── priority=100 priority,in_port,dl_vlan  --->  strip_vlan,load,load,load,load,load resubmit(,8), ( x 1 )
> +        └── priority=0 priority  --->   drop, ( x 1 )
> +
> +
> +In only a few logical blocks, we have a good overview of what this table is
> +doing. It looks like it's adding metadata based on input ports and vlan
> +IDs and mainly sending traffic to table 8.
> +
> +Let's look at table 8, an in this case, let's filter out the flows that have
> +not been hit by actual traffic. This is quite easy to do with the arithmetic
> +filtering expressions:
> +::
> +
> +   $ ovs-flowviz -i flows.txt -f "table=8 and n_packets>0" openflow logic
> +
> +    Ofproto Flows (logical)
> +    └── ** TABLE 8 **
> +        ├── priority=50 priority,reg14,metadata,dl_dst  --->  load resubmit(,9), ( x 3 )
> +        └── priority=50 priority,metadata  --->  load,move resubmit(,73),resubmit(,9), ( x 2 )
> +
> +At this point, we might find ourselves a bit lost since we may not remember
> +what metadata OVN stored in the previous table. Here is where
> +``ovs-flowviz``'s OVN integration could come useful. Let's connect to the
> +running OVN instance and ask it about the flows we're looking at.
> +
> +::
> +
> +    $ export OVN_NB_DB=tcp:172.18.0.4:6641
> +    $ export OVN_SB_DB=tcp:172.18.0.4:6642
> +    $ ovs-flowviz -i flows.txt -f "table=8 and n_packets>0" openflow logic --ovn-detrace
> +    Ofproto Flows (logical)
> +    └── ** TABLE 8 **
> +        ├── cookie=0xe10c34ee priority=50 priority,reg14,metadata,dl_dst  --->  load resubmit(,9), ( x 1 )
> +        │   └── OVN Info
> +        │       ├── *  Logical datapaths:
> +        │       ├── *      "ovn_cluster_router" (366e1c41-0f3d-4420-b796-10692b64e3e4)
> +        │       ├── *  Logical flow: table=0 (lr_in_admission), priority=50, match=(eth.mcast && inport == "rtos-ovn-worker2), actions=(xreg0[0..47] = 0a:58:0a:f4:01:01; next;)
> +        │       └── *  Logical Router Port: rtos-ovn-worker2 mac 0a:58:0a:f4:01:01 networks ['10.244.1.1/24'] ipv6_ra_configs {}
> +        ├── cookie=0x11e1adbc priority=50 priority,reg14,metadata,dl_dst  --->  load resubmit(,9), ( x 1 )
> +        │   └── OVN Info
> +        │       ├── *  Logical datapaths:
> +        │       ├── *      "GR_ovn-worker2" (c07f8387-6479-4e81-9304-9f8e54f81c56)
> +        │       ├── *  Logical flow: table=0 (lr_in_admission), priority=50, match=(eth.mcast && inport == "rtoe-GR_ovn-worker2), actions=(xreg0[0..47] = 02:42:ac:12:00:03; next;)
> +        │       └── *  Logical Router Port: rtoe-GR_ovn-worker2 mac 02:42:ac:12:00:03 networks ['172.18.0.3/16'] ipv6_ra_configs {}
> +        ├── cookie=0xf42133f  priority=50 priority,reg14,metadata,dl_dst  --->  load resubmit(,9), ( x 1 )
> +        │   └── OVN Info
> +        │       ├── *  Logical datapaths:
> +        │       ├── *      "GR_ovn-worker2" (c07f8387-6479-4e81-9304-9f8e54f81c56)
> +        │       ├── *  Logical flow: table=0 (lr_in_admission), priority=50, match=(eth.dst == 02:42:ac:12:00:03 && inport == "rtoe-GR_ovn-worker2), actions=(xreg0[0..47] = 02:42:ac:12:00:03; next;)
> +        │       └── *  Logical Router Port: rtoe-GR_ovn-worker2 mac 02:42:ac:12:00:03 networks ['172.18.0.3/16'] ipv6_ra_configs {}
> +        └── cookie=0x43a0327  priority=50 priority,metadata  --->  load,move resubmit(,73),resubmit(,9), ( x 2 )
> +            └── OVN Info
> +                ├── *  Logical datapaths:
> +                ├── *      "ovn-worker" (24280d0b-fee0-4f8e-ba4f-036a9b9af921)
> +                ├── *      "ovn-control-plane" (3262a782-8961-416b-805e-08233e8fda72)
> +                ├── *      "ext_ovn-worker2" (3f88dcd2-c56d-478f-a3b1-c7aee2efe967)
> +                ├── *      "ext_ovn-worker" (5facbaf0-485d-4cf5-8940-eff9678ef7bb)
> +                ├── *      "ext_ovn-control-plane" (8b0aecb6-b05a-48a7-ad09-72524bb91d40)
> +                ├── *      "join" (e2dc230e-2f2a-4b93-93fa-0fe495163514)
> +                ├── *      "ovn-worker2" (f7709fbf-d728-4cff-9b9b-150461cc75d2)
> +                └── *  Logical flow: table=0 (ls_in_check_port_sec), priority=50, match=(1), actions=(reg0[15] = check_in_port_sec(); next;)
> +
> +That's way better. ``ovs-flowviz`` has automatically added the `cookie` to the
> +logical block key so have more blocks but in exchange, it has looked up each
> +cookie on the running OVN databases and inserted the known information on each
> +block. So now we see what OVN is trying to do, the logical flow that generated
> +each OpenFlow flow and the logical datapath each flow belongs to.
> +
> +Visualizing datapath flow trees
> +-------------------------------
> +
> +Now, let's see another typical usecase that can lead to eyestrain:
> +understanding datapath conntrack recirculations.
> +
> +OVS makes heavy use of connection tracking and the ``recirc()`` action
> +to build complex datapaths. Typically, OVS will insert a flow that,
> +when matched, will send the packet through conntrack (using the ``ct`` action)
> +and recirculate it with a particular recirculation id (``recirc_id``). Then, a
> +flow matching on that ``recirc_id`` will be matched and further process the
> +packet. This can happen more than once for a given packet.
> +
> +This sequential set of events is, however, difficult to visualize when you
> +look at a datapath flow dump. Flows are unordered recirculations that need to
> +be followed manually (typically, with heavy use of "grep").
> +
> +For this use-case, ``ovs-flowviz datapath tree`` format can be extremely
> +useful. It builds a hierarchical tree based on the ``recirc_id`` matches and
> +``recirc()`` actions and indents flows based on it.
> +
> +Here is an example.
> +::
> +
> +    ── recirc_id(0),in_port(3),eth(...),ipv4(...),tcp(dst=8181), actions:ct(zone=2,nat),recirc(0x19348)
> +    │   ├── recirc_id(0x19348),in_port(3),ct_state(-new+est-rel-rpl-inv+trk),ct_label(0/0x3),eth(...),eth_type,ipv4(), actions:ct(zone=27,nat),recirc(0x10)
> +    │   │   ├── recirc_id(0x10),in_port(3),ct_state(-new+est-rel-rpl-inv+trk),eth(...),ipv4(...), actions:9
> +    │   │   ├── recirc_id(0x10),in_port(3),ct_state(-new+est-rel+rpl-inv+trk),eth(...),ipv4(...), actions:9
> +    │   │   └── recirc_id(0x10),in_port(3),ct_state(+new-est-rel-rpl-inv+trk),eth(...),ipv4(...), actions:ct(commit,zone=27,label=0/0x1),9
> +    │   └── recirc_id(0x19348),in_port(3),ct_state(+new-est-rel-rpl-inv+trk),eth(...),ipv4(...),  actions:ct(commit,zone=2,label=0/0x1),ct(zone=27,nat),recirc(0x10)
> +    │       ├── recirc_id(0x10),in_port(3),ct_state(-new+est-rel-rpl-inv+trk),eth(...),ipv4(...), actions:9
> +    │       ├── recirc_id(0x10),in_port(3),ct_state(-new+est-rel+rpl-inv+trk),eth(...),ipv4(...), actions:9
> +    │       └── recirc_id(0x10),in_port(3),ct_state(+new-est-rel-rpl-inv+trk),eth(...),ipv4(...), actions:ct(commit,zone=27,label=0/0x1),9
> +
> +The above shows a typical conntrack recirculation flow.
> +The first flow (with ``recir_id(0)``) sends the packet through conntrack
> +system and recirculates with ``recirc_id(0x19348)``.
> +Then, based on the ``ct_state`` the packet processing branches out into two
> +flows. Each flow resends the packet through conntrack and recirculates the
> +packet one more time. Finally, the packet is processed by 3 flows
> +on ``recirc_id(10)``.
> +
> +This 3-stage processing is now very clear.
> +
> +Note that this format can yield longer outputs since some flows (in this
> +example those with ``recirc_id(10)`` can be repeated. However, the result
> +is a clear representation of an otherwise difficult to see conntrack
> +interaction.
> +
> +This example shows only a single "subtree". If we use this command to display
> +a big flow dump, the output can be lengthy. Here are two (combinable) ways to
> +help out.
> +
> +Plotting datapath trees
> +~~~~~~~~~~~~~~~~~~~~~~~
> +
> +By using the ``ovs-flowviz datapath html`` format, long datapath trees can
> +be displayed in an interactive HTML table. The resulting web allows you to
> +collapse and expand subtrees so you can focus on what you're looking for.
> +
> +In addition, the ``ovs-flowviz datapath graph`` format generates a graphviz
> +graph definition where each block of flows with the same ``recirc_id`` match
> +are arranged together and edges are created to represent recirculations.
> +Also, this format comes with further goodies such as displaying the conntrack
> +zones which are key to understanding what the datapath is really doing with a
> +packet.
> +
> +These two formats (``html`` and ``graph``) can even be combined. By using the
> +``ovs-flowviz datapath graph --html`` command, you'll get an interactive
> +HTML table alongside a `svg` graphical representation of the flows. Click on
> +a flow on the svg and it'll take you to the corresponding entry in the
> +flow table.
> +
> +
> +Filtering
> +~~~~~~~~~
> +
> +Apart from being able to expand and collapse subtrees, we can use filtering.
> +
> +However, filtering works in a slightly different way compared with OpenFlow
> +flows. Instead of just removing non-matching flows, the output
> +of a filtered datapath flow tree will show full sub-trees that contain at
> +least one flow that satisfies the filter.
> +
> +For example, let's take the flows in the above example, and let's imagine we
> +want to understand what traffic is going out on port ``9``. We could run
> +the tool as:
> +::
> +
> +   $ ovs-appctl dpctl/dump-flows | ovs-flowviz -f "output.port=9" datapath tree
> +
> +The resulting flow tree will contain all of the flows above, even those
> +with ``recirc_id(0)`` and ``recirc_id(19348)`` that don't actually output
> +traffic to port ``9``. Why? because they are all part of a subtree that
> +contains flows that do output packets on port ``9``
> +
> +That way, we see the "full picture" of how traffic on port ``9`` is being
> +processed.
> +
> +.. _ovs-flowviz(8): https://docs.openvswitch.org/en/latest/ref/ovs-flowviz.8
> diff --git a/Documentation/topics/index.rst b/Documentation/topics/index.rst
> index f239fcf83..9ddb145dd 100644
> --- a/Documentation/topics/index.rst
> +++ b/Documentation/topics/index.rst
> @@ -58,3 +58,4 @@ OVS
>      userspace-checksum-offloading
>      userspace-tx-steering
>      usdt-probes
> +   flow-visualization

On 4/9/24 09:06, Adrian Moreno wrote:
> Add a man page for ovs-flowviz as well as a topic page with some more
> detailed examples.
> 
> Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
> ---
>  Documentation/automake.mk                   |   4 +-
>  Documentation/conf.py                       |   2 +
>  Documentation/ref/index.rst                 |   1 +
>  Documentation/ref/ovs-flowviz.8.rst         | 531 ++++++++++++++++++++
>  Documentation/topics/flow-visualization.rst | 271 ++++++++++
>  Documentation/topics/index.rst              |   1 +
>  6 files changed, 809 insertions(+), 1 deletion(-)
>  create mode 100644 Documentation/ref/ovs-flowviz.8.rst
>  create mode 100644 Documentation/topics/flow-visualization.rst

Hi, Adrian.

This patch still breaks RPM build since new files are not
mentioned in the fedora spec file.  See the CI failure.

But I'd suggest to wait a bit for review on this version
before re-posting just for this change.

Best regards, Ilya Maximets.

On 9 Apr 2024, at 9:06, Adrian Moreno wrote:

> Add a man page for ovs-flowviz as well as a topic page with some more
> detailed examples.
>
> Signed-off-by: Adrian Moreno <amorenoz@redhat.com>

In addition to Ilya’s comments, find 3 small comments below. The rest looks good.

Cheers,

Eelco

> ---
>  Documentation/automake.mk                   |   4 +-
>  Documentation/conf.py                       |   2 +
>  Documentation/ref/index.rst                 |   1 +
>  Documentation/ref/ovs-flowviz.8.rst         | 531 ++++++++++++++++++++
>  Documentation/topics/flow-visualization.rst | 271 ++++++++++
>  Documentation/topics/index.rst              |   1 +
>  6 files changed, 809 insertions(+), 1 deletion(-)
>  create mode 100644 Documentation/ref/ovs-flowviz.8.rst
>  create mode 100644 Documentation/topics/flow-visualization.rst
>
> diff --git a/Documentation/automake.mk b/Documentation/automake.mk
> index 47d2e336a..539870aa2 100644
> --- a/Documentation/automake.mk
> +++ b/Documentation/automake.mk
> @@ -45,7 +45,7 @@ DOC_SOURCE = \
>  	Documentation/topics/fuzzing/ovs-fuzzing-infrastructure.rst \
>  	Documentation/topics/fuzzing/ovs-fuzzers.rst \
>  	Documentation/topics/fuzzing/security-analysis-of-ovs-fuzzers.rst \
> -	Documentation/topics/testing.rst \
> +	Documentation/topics/flow-visualization.rst \
>  	Documentation/topics/integration.rst \
>  	Documentation/topics/language-bindings.rst \
>  	Documentation/topics/networking-namespaces.rst \
> @@ -55,6 +55,7 @@ DOC_SOURCE = \
>  	Documentation/topics/ovsdb-replication.rst \
>  	Documentation/topics/porting.rst \
>  	Documentation/topics/record-replay.rst \
> +	Documentation/topics/testing.rst \
>  	Documentation/topics/tracing.rst \
>  	Documentation/topics/usdt-probes.rst \
>  	Documentation/topics/userspace-checksum-offloading.rst \
> @@ -162,6 +163,7 @@ RST_MANPAGES = \
>  	ovs-actions.7.rst \
>  	ovs-appctl.8.rst \
>  	ovs-ctl.8.rst \
> +	ovs-flowviz.8.rst \
>  	ovs-l3ping.8.rst \
>  	ovs-parse-backtrace.8.rst \
>  	ovs-pki.8.rst \
> diff --git a/Documentation/conf.py b/Documentation/conf.py
> index 15785605a..3a82f23a7 100644
> --- a/Documentation/conf.py
> +++ b/Documentation/conf.py
> @@ -120,6 +120,8 @@ _man_pages = [
>       u'utility for configuring running Open vSwitch daemons'),
>      ('ovs-ctl.8',
>       u'OVS startup helper script'),
> +    ('ovs-flowviz.8',
> +     u'utility for visualizing OpenFlow and datapath flows'),
>      ('ovs-l3ping.8',
>       u'check network deployment for L3 tunneling problems'),
>      ('ovs-parse-backtrace.8',
> diff --git a/Documentation/ref/index.rst b/Documentation/ref/index.rst
> index 03ada932f..7f2fe6177 100644
> --- a/Documentation/ref/index.rst
> +++ b/Documentation/ref/index.rst
> @@ -42,6 +42,7 @@ time:
>     ovs-actions.7
>     ovs-appctl.8
>     ovs-ctl.8
> +   ovs-flowviz.8
>     ovs-l3ping.8
>     ovs-pki.8
>     ovs-sim.1
> diff --git a/Documentation/ref/ovs-flowviz.8.rst b/Documentation/ref/ovs-flowviz.8.rst
> new file mode 100644
> index 000000000..b5b1befb0
> --- /dev/null
> +++ b/Documentation/ref/ovs-flowviz.8.rst
> @@ -0,0 +1,531 @@
> +..
> +      Licensed under the Apache License, Version 2.0 (the "License"); you may
> +      not use this file except in compliance with the License. You may obtain
> +      a copy of the License at
> +
> +          http://www.apache.org/licenses/LICENSE-2.0
> +
> +      Unless required by applicable law or agreed to in writing, software
> +      distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
> +      WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
> +      License for the specific language governing permissions and limitations
> +      under the License.
> +
> +      Convention for heading levels in Open vSwitch documentation:
> +
> +      =======  Heading 0 (reserved for the title in a document)
> +      -------  Heading 1
> +      ~~~~~~~  Heading 2
> +      +++++++  Heading 3
> +      '''''''  Heading 4
> +
> +      Avoid deeper levels because they do not render well.
> +
> +===========
> +ovs-flowviz
> +===========
> +
> +Synopsis
> +========
> +
> +``ovs-flowviz``
> +[``[-i | --input] <[alias,]file>``]
> +[``[-c | --config] <file>``]
> +[``[-f | --filter] <filter>``]
> +[``[-h | --highlight] <filter>``]
> +[``--style <style>``]
> +*<flow_type>* *<format>* [<arg>...]
> +
> +``ovs-flowviz --help``
> +
> +Description
> +===========
> +
> +The ``ovs-flowviz`` program helps visualize OpenFlow and datapath flow dumps
> +in different formats in order to make them more easily understood.
> +
> +The program works by reading flows from ``stdin`` or from a file specified
> +in the ``--input`` option, filtering them, highlighting them, and finally
> +outputting them in one of the predefined formats.
> +
> +
> +Options
> +=======
> +
> +.. program: ovs-flowviz
> +
> +.. option:: -h, --help
> +
> +    Prints a brief help message to the console.
> +
> +.. option:: -i <[alias,]file>, --input <[alias,]file>
> +
> +    Specifies the file to read flows from. If not provided, ``ovs-flowviz``
> +    will read flows from stdin.
> +
> +    This option can be specified multiple times.
> +    The file path can prepended by an alias that will be shown in the output.
> +    For example: ``--input node1,/path/to/file1 --input node2,/path/to/file2``
> +
> +.. option:: -c <file>, --config <file>
> +
> +    Specifies the style configuration file to use. ``ovs-flowviz`` ships with
> +    a default configuration file but it can be overridden using this option.
> +    Styles defined in the style configuration file will be select-able using
> +    the ``--style`` option.
> +
> +    For more details on the style configuration file, see
> +    `Style Configuration File`_ section below.
> +
> +.. option:: -f <filter>, --filter <filter>
> +
> +   Tells ``ovs-flowviz`` to filter the flows and only show the ones that match
> +   the expression (although some formats implement filtering differently,
> +   see `Datapath tree format`_ below).
> +
> +   The filtering syntax is detailed in `Filtering Syntax`_.
> +
> +.. option:: -h <filter>, --highlight <filter>
> +
> +   Tells ``ovs-flowviz`` to highlight the flows that match the provided filter
> +
> +   The filtering syntax is detailed in `Filtering Syntax`_.
> +
> +.. option:: --style <style>
> +
> +   Specifies the style to use. The style must have been defined in the
> +   style configuration file.
> +
> +.. option:: <flow_type>
> +
> +   "openflow" or "datapath".
> +
> +.. option:: <format>
> +
> +   See `Supported formats`_ section.
> +
> +
> +Supported formats
> +=================
> +
> +``ovs-flowviz`` supports several visualization formats for both OpenFlow and
> +datapath flows that are summarized in the following table:
> +
> +.. list-table::
> +   :widths: 20 10 70
> +   :align: center
> +   :header-rows: 1
> +
> +   * - Flow Type
> +     - Format
> +     - Description
> +   * - Both
> +     - console
> +     - Prints the flows in a configurable, colorful style in the console.
> +   * - Both
> +     - json
> +     - Prints the flows in JSON format.
> +   * - Both
> +     - html
> +     - Prints the flows in an HTML list.
> +   * - Openflow
> +     - cookie
> +     - Prints the flows in the console sorted by cookie.
> +   * - Openflow
> +     - logic
> +     - Prints the logical structure of flows in the console.
> +   * - Datapath
> +     - tree
> +     - Prints the flows a tree structure arranged by `recirc_id`.
> +   * - Datapath
> +     - graph
> +     - Prints a graphviz graph of the flows arranged by `recirc_id`.
> +
> +
> +Console format
> +~~~~~~~~~~~~~~
> +
> +The ``console`` works for both OpenFlow and datapath flow types and prints
> +flows in the terminal with the style determined by the ``--style`` option.
> +
> +Additionally, it accepts the following arguments:
> +
> +.. option:: -h, --heat-map
> +
> +   This option changes the color of the packet and byte counters to reflect
> +   their relative size. The color gradient goes through the following colors:
> +   blue (coldest, lowest), cyan, green, yellow, red (hottest, highest)
> +
> +   Note filtering is applied before the range is calculated.
> +
> +
> +JSON format
> +~~~~~~~~~~~
> +
> +The ``json`` format works for both OpenFlow and datapath flow types and prints
> +flows in JSON format. See `JSON Syntax`_ for more details.
> +
> +
> +HTML format
> +~~~~~~~~~~~
> +
> +The ``html`` format works for both OpenFlow and datapath flows and prints
> +flows in an HTML table that offers some basic interactivity. OpenFlow flows
> +are sorted in tables and datapath flows are arranged in flow trees
> +(see `Datapath tree format`_ for more details).
> +
> +Styles defined via Style Configuration File and selected via ``--style`` option
> +also apply to ``html`` format.
> +
> +
> +OpenFlow cookie format
> +~~~~~~~~~~~~~~~~~~~~~~
> +
> +The OpenFlow ``cookie`` format is similar to the ``console`` format but
> +instead of arranging the flows per table, it arranges the flows per cookie.
> +
> +
> +Openflow logic format
> +~~~~~~~~~~~~~~~~~~~~~
> +
> +The OpenFlow ``logic`` format helps visualize the logic structure of OpenFlow
> +pipelines by arranging flows into *logical blocks*.
> +A logical block is a set of flows that have:
> +
> +* Same ``priority``.
> +* Match on the same fields (regardless of the match value and mask).
> +* Execute the same actions (regardless of the actions' arguments,
> +  except for resubmit and output).
> +* Optionally, the ``cookie`` can be counted as part of the logical flow.
> +
> +This format supports the following extra arguments:
> +
> +.. option:: -s, --show-flows
> +
> +    Show all the flows under each logical block.
> +
> +.. option:: -d, --ovn-detrace
> +
> +    Use ovn-detrace.py script to extract cookie information (implies '-c').
> +
> +.. option:: -c, --cookie
> +
> +    Consider the cookie in the logical block.
> +
> +.. option:: --ovn-detrace-path <path>
> +
> +    Use an alternative path to look for ovn_detrace.py script.
> +
> +.. option:: --ovnnb-db text
> +
> +   Specify the OVN NB database string (implies '-d').
> +   Default value is "unix:/var/run/ovn/ovnnb_db.sock".
> +
> +.. option:: --ovnsb-db text
> +
> +   Specify the OVN SB database string (implies '-d').
> +   Default value is "unix:/var/run/ovn/ovnsb_db.sock".
> +
> +.. option:: --o <text>, --ovn-filter <text>
> +
> +   Specify the a filter to be run on the ovn-detrace information.
> +   Syntax: python regular expression
> +   (See https://docs.python.org/3/library/re.html).
> +
> +.. option:: -h, --heat-map
> +
> +   This option changes the color of the packet and byte counters to reflect
> +   their relative size. The color gradient goes through the following colors:
> +   blue (coldest, lowest), cyan, green, yellow, red (hottest, highest)
> +
> +   Note filtering is applied before the range is calculated.
> +
> +
> +Datapath tree format
> +~~~~~~~~~~~~~~~~~~~~
> +
> +The datapath ``tree`` format arranges datapath flows in a hierarchical tree
> +based on `recirc_id`. At the first level, flows with `recirc_id(0)` are
> +listed. If a flow contains a `recirc()` action with a specific `recirc_id`,
> +flows matching on that `recirc_id` are listed below. This is done recursively
> +for all actions.
> +
> +The result is a hierarchical representation that helps understand how actions
> +are related to each other via recirculation. Note flows with a specific
> +non-zero `recirc_id` are listed below each flow that has a corresponding
> +`recirc()` action. Therefore, they would be duplicated leading to a longer
> +output.
> +
> +Also, filtering works in a slightly different way for datapath flow trees.
> +Unlike other formats where a filter simply removes non-matching flows,
> +the output of a filtered datapath flow tree will show full sub-trees
> +that contain at least one flow that satisfies the filter.
> +
> +The ``html`` format prints this same tree in an interactive HTML table.
> +
> +
> +Datapath graph format
> +~~~~~~~~~~~~~~~~~~~~~
> +
> +The datapath ``graph`` generates a graphviz visual representation of the
> +same tree-like flow hierarchy that the ``tree`` format prints.
> +
> +It supports the following extra argument:
> +
> +.. option:: -h, --html
> +
> +    Prints the graphviz format in an svg image alongside the interactive HTML
> +    table of flows (that 'html' format would print).
> +
> +
> +JSON Syntax
> +===========
> +
> +Both OpenFlow and datapath `json` formats print a JSON list of JSON
> +objects each of one representing an individual flow.S

There is a trailing S.

> +
> +Each flow object contains the following keys:
> +
> +**orig**
> +    Contains the original flow string.
> +
> +
> +**info**
> +   Contains an object with the flow information
> +   such as: cookie, duration, table, n_packets, n_bytes, etc.
> +
> +
> +**match**
> +   Contains an object with the flow match.
> +   For each match, the object contains a key-value where the key is the name
> +   of the match as defined in ovs-fields and ovs-ofctl and the value
> +   represents the match value. The way each value is represented depends on its
> +   type. See `Value representation`_.
> +
> +
> +**actions**
> +   Contains a list of action objects.
> +   Each action is represented by an JSON object that has one key and one value.
> +   The key corresponds to the action name. The value represents the arguments
> +   of such key. See `Action representation`_.
> +
> +
> +**ufid**
> +   (datapath flows only) Contains the ufid.
> +
> +
> +Value representation
> +~~~~~~~~~~~~~~~~~~~~
> +
> +Values are represented differently depending on their type:
> +
> +* Flags: Fields that represent flags (e.g: tcp) are represented by boolean
> +  "true"
> +
> +* Decimal / Hexadecimal: They are represented by their integer value.
> +  If they support masking, they are represented by a dictionary with two keys:
> +  value contains the field value and mask contains the mask. Both are integers.
> +
> +* Ethernet: They are represented by a string: {address}[/{mask}]
> +
> +* IPv4 / IPv6: They are represented by a string {address}[/mask]
> +
> +* Registers: They are represented by a dictionary with three keys:
> +  field contains the field value (string), start and end that represent the
> +  first and last bit of the register.
> +
> +For example, the register
> +::
> +
> +
> +   NXM_NX_REG10[0..15]
> +
> +
> +is represented as
> +::
> +
> +
> +   {
> +       "field": "NXM_NX_REG10",
> +       "start": 0,
> +       "end": 15
> +   },
> +
> +
> +Action representation
> +~~~~~~~~~~~~~~~~~~~~~
> +
> +Actions are generally represented by an object that has a single key and a
> +value. The key is the action name as defined ovs-actions.
> +
> +The value of actions that have no arguments (such as ``drop``) is
> +(boolean) ``true``.
> +
> +The value of actions that have a list of arguments (e.g:
> +``resubmit([port],[table],[ct])``) is an object that has the name of the
> +argument as key. The argument names for each action is defined in
> +ovs-actions. For example, the action
> +::
> +
> +   resubmit(,10)
> +
> +is represented as
> +::
> +
> +   {
> +       "redirect": {
> +           "port": "",
> +           "table": 10
> +       }
> +   }
> +
> +The value of actions that have a key-word list as arguments
> +(e.g: ``ct([argument])``) is an object whose keys correspond to the keys
> +defined in ``ovs-actions(7)``. The way values are represented depends
> +on the type of the argument.
> +For example, the action
> +::
> +
> +   ct(table=14,zone=NXM_NX_REG12[0..15],nat)
> +
> +is represented as
> +::
> +
> +   {
> +       "ct": {
> +           "table": 14,
> +           "zone": {
> +               "field": "NXM_NX_REG12",
> +               "start": 0,
> +               "end": 15
> +           },
> +           "nat": true
> +       }
> +   }
> +
> +
> +Style Configuration File
> +========================
> +
> +The style configuration file that can be selected via the ``--config`` option
> +has INI syntax and can define any number of styles to be used by both
> +``console`` and ``html`` formats. Once defined in the configuration file
> +they can be selected using the ``--style`` option.
> +
> +INI sections are used to define styles, ``[styles.mystyle]`` defines a style
> +called `mystle`. Within a section styles can be defined as:
> +
> +::
> +
> +     [FORMAT].[PORTION].[SELECTOR].[ELEMENT] = [VALUE]
> +
> +
> +**FORMAT**
> +   Either ``console`` or ``html``
> +
> +**PORTION**
> +   The part of the a key-value the style applies to. It can be:
> +   ``key`` (to indicate the key part of a key-value), ``value`` (to indicate
> +   the value part of a key-value), ``flag`` (to indicate a single flag)
> +   or ``delim`` (to indicate delimiters such as parentheses, brackets, etc).
> +
> +**SELECTOR**
> +   Is used to select what key-value the style applies to. It can be:
> +   ``highlighted`` (to indicate highlighted key-values), ``type.<type>``
> +   to indicate certain types such as `IPAddress` or `EthMask` or `<keyname>`
> +   to select a particular key name.
> +
> +**ELEMENT**
> +   Is used to select what style element to modify. It can be one
> +   of: **color** or **underline** (only for **console** format).
> +
> +**VALUE**
> +   Is either a color hex, other color names defined in the rich python
> +   library (https://rich.readthedocs.io/en/stable/appendix/colors.html) or
> +   "true" if the element is ``underline``.
> +
> +A default configuration file is shipped with the tool and it's path is printed

it’s -> its

> +in the ``--help`` output. A detailed description of the syntax alongside
> +some examples is available there.

is available -> are available

> +
> +
> +Filtering syntax
> +================
> +
> +``ovs-flowviz`` provides rich highlighting and filtering. The special command
> +``ovs-flowviz filter`` dumps the filtering syntax:
> +
> +::
> +
> +    $ ovs-flowviz filter
> +    Filter Syntax
> +    *************
> +
> +       [! | not ] {key}[[.subkey]...] [OPERATOR] {value})] [LOGICAL OPERATOR] ...
> +
> +      Comparison operators are:
> +          =   equality
> +          <   less than
> +          >   more than
> +          ~=  masking (valid for IP and Ethernet fields)
> +
> +      Logical operators are:
> +          !{expr}:  NOT
> +          {expr} && {expr}: AND
> +          {expr} || {expr}: OR
> +
> +      Matches and flow metadata:
> +          To compare against a match or info field, use the field directly, e.g:
> +              priority=100
> +              n_bytes>10
> +          Use simple keywords for flags:
> +              tcp and ip_src=192.168.1.1
> +
> +      Actions:
> +          Actions values might be dictionaries, use subkeys to access individual
> +          values, e.g:
> +              output.port=3
> +          Use simple keywords for flags
> +              drop
> +
> +      Examples of valid filters.
> +          nw_addr~=192.168.1.1 && (tcp_dst=80 || tcp_dst=443)
> +          arp=true && !arp_tsa=192.168.1.1
> +          n_bytes>0 && drop=true
> +
> +
> +Example expressions:
> +::
> +
> +   n_bytes > 0 and drop
> +   nw_src~=192.168.1.1 or arp.tsa=192.168.1.1
> +   ! tcp && output.port=2
> +
> +
> +Examples
> +========
> +
> +Print OpenFlow flows sorted by cookie adding OVN data to each one:
> +::
> +
> +    $ ovs-flowviz -i flows.txt openflow cookie --ovn-detrace
> +
> +Print OpenFlow logical structure, showing the flows and heat-map:
> +::
> +
> +    $ ovs-flowviz -i flows.txt openflow logic --show-flows --heat-map
> +
> +Display OpenFlow flows in HTML format with "light" style and highlight drops:
> +::
> +
> +    $ ovs-flowviz -i flows.txt --style "light" --highlight "n_packets > 0 and drop" openflow html > flows.html
> +
> +Display the datapath flows in an interactive graphviz + HTML view:
> +::
> +
> +    $ ovs-flowviz -i flows.txt datapath graph --html > flows.html
> +
> +Display the datapath flow trees that lead to packets being sent to port 10:
> +::
> +
> +    $ ovs-flowviz -i flows.txt --filter "output.port=10" datapath tree
> diff --git a/Documentation/topics/flow-visualization.rst b/Documentation/topics/flow-visualization.rst
> new file mode 100644
> index 000000000..62d0d6bd8
> --- /dev/null
> +++ b/Documentation/topics/flow-visualization.rst
> @@ -0,0 +1,271 @@
> +..
> +      Licensed under the Apache License, Version 2.0 (the "License"); you may
> +      not use this file except in compliance with the License. You may obtain
> +      a copy of the License at
> +
> +          http://www.apache.org/licenses/LICENSE-2.0
> +
> +      Unless required by applicable law or agreed to in writing, software
> +      distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
> +      WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
> +      License for the specific language governing permissions and limitations
> +      under the License.
> +
> +      Convention for heading levels in Open vSwitch documentation:
> +
> +      =======  Heading 0 (reserved for the title in a document)
> +      -------  Heading 1
> +      ~~~~~~~  Heading 2
> +      +++++++  Heading 3
> +      '''''''  Heading 4
> +
> +      Avoid deeper levels because they do not render well.
> +
> +==================================
> +Visualizing flows with ovs-flowviz
> +==================================
> +
> +When troubleshooting networking issues with OVS, we typically end up looking
> +at OpenFlow or datapath flow dumps. These dumps tend to be quite dense and
> +difficult to reason about.
> +
> +``ovs-flowviz`` is a utility script that helps visualizing OpenFlow and
> +datapath flows to make it easier to understand what is going on.
> +
> +The `ovs-flowviz(8)`_ manpage describes its basic usage. In this document a few
> +of its advanced visualization formats will be expanded.
> +
> +
> +Installing ovs-flowviz
> +----------------------
> +
> +``ovs-flowviz`` is part of the openvswitch python package but its
> +extra dependencies have to be installed explicitly by running:
> +::
> +
> +    $ pip install openvswitch[flowviz]
> +
> +Or, if you are working with the OVS tree:
> +::
> +
> +    $ cd python && pip install .[flowviz]
> +
> +Visualizing OpenFlow logical block
> +----------------------------------
> +
> +When controllers such as OVN write OpenFlow flows, they typically organize
> +flows in functional blocks. These blocks can expand to multiple flows that
> +"look similar", in the sense that they match on the same fields and have
> +similar actions.
> +
> +However, when we look at a flow dump the number of flows can make it difficult
> +to perceive this logical functionality that the controller is trying to
> +implement using OpenFlow.
> +
> +In this example, we are going to use ``ovs-flowviz openflow logic``
> +visualization to understand an OVN flow dump a bit better.
> +
> +On a particular flow dump we have 23 flows in table 0:
> +::
> +
> +   $ grep -c "table=0" flows.txt
> +   23
> +
> +If we look at the first few lines, the amount of information can be
> +overwhelming and difficult our analysis:
> +
> +::
> +
> +    $ head flows.txt
> +      cookie=0xf76b4b20, duration=765.107s, table=0, n_packets=0, n_bytes=0, priority=180,vlan_tci=0x0000/0x1000 actions=conjunction(100,2/2)
> +      cookie=0xf76b4b20, duration=765.107s, table=0, n_packets=0, n_bytes=0, priority=180,conj_id=100,in_port="patch-br-int-to",vlan_tci=0x0000/0x1000 actions=load:0xa->NXM_NX_REG13[],load:0xc->NXM_NX_REG11[],load:0xb->NXM_NX_REG12[],load:0xb->OXM_OF_METADATA[],load:0x1->NXM_NX_REG14[],mod_dl_src:02:42:ac:12:00:03,resubmit(,8)
> +      cookie=0x0, duration=765.388s, table=0, n_packets=0, n_bytes=0, priority=100,in_port="ovn-6bb3b3-0" actions=move:NXM_NX_TUN_ID[0..23]->OXM_OF_METADATA[0..23],move:NXM_NX_TUN_METADATA0[16..30]->NXM_NX_REG14[0..14],move:NXM_NX_TUN_METADATA0[0..15]->NXM_NX_REG15[0..15],resubmit(,40)
> +      cookie=0x0, duration=765.388s, table=0, n_packets=0, n_bytes=0, priority=100,in_port="ovn-a6ff98-0" actions=move:NXM_NX_TUN_ID[0..23]->OXM_OF_METADATA[0..23],move:NXM_NX_TUN_METADATA0[16..30]->NXM_NX_REG14[0..14],move:NXM_NX_TUN_METADATA0[0..15]->NXM_NX_REG15[0..15],resubmit(,40)
> +      cookie=0xf2ca6195, duration=765.107s, table=0, n_packets=6, n_bytes=636, priority=100,in_port="ovn-k8s-mp0" actions=load:0x1->NXM_NX_REG13[],load:0x2->NXM_NX_REG11[],load:0x7->NXM_NX_REG12[],load:0x4->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],resubmit(,8)
> +      cookie=0x236e941d, duration=408.874s, table=0, n_packets=11, n_bytes=846, priority=100,in_port=aceac9829941d11 actions=load:0x11->NXM_NX_REG13[],load:0x2->NXM_NX_REG11[],load:0x7->NXM_NX_REG12[],load:0x4->OXM_OF_METADATA[],load:0x3->NXM_NX_REG14[],resubmit(,8)
> +      cookie=0x3facf689, duration=405.581s, table=0, n_packets=11, n_bytes=846, priority=100,in_port="363ba22029cd92b" actions=load:0x12->NXM_NX_REG13[],load:0x2->NXM_NX_REG11[],load:0x7->NXM_NX_REG12[],load:0x4->OXM_OF_METADATA[],load:0x4->NXM_NX_REG14[],resubmit(,8)
> +      cookie=0xe7c8c4bb, duration=405.570s, table=0, n_packets=11, n_bytes=846, priority=100,in_port="6a62cde0d50ef44" actions=load:0x13->NXM_NX_REG13[],load:0x2->NXM_NX_REG11[],load:0x7->NXM_NX_REG12[],load:0x4->OXM_OF_METADATA[],load:0x5->NXM_NX_REG14[],resubmit(,8)
> +      cookie=0x99a0ffc1, duration=59.391s, table=0, n_packets=8, n_bytes=636, priority=100,in_port="5ff3bfaaa4eb622" actions=load:0x14->NXM_NX_REG13[],load:0x2->NXM_NX_REG11[],load:0x7->NXM_NX_REG12[],load:0x4->OXM_OF_METADATA[],load:0x6->NXM_NX_REG14[],resubmit(,8)
> +      cookie=0xe1b5c263, duration=59.365s, table=0, n_packets=8, n_bytes=636, priority=100,in_port="8d9e0bc76347e59" actions=load:0x15->NXM_NX_REG13[],load:0x2->NXM_NX_REG11[],load:0x7->NXM_NX_REG12[],load:0x4->OXM_OF_METADATA[],load:0x7->NXM_NX_REG14[],resubmit(,8)
> +
> +
> +However, we can better understand what table 0 does by looking at its
> +logical representation.
> +::
> +
> +   $ ovs-flowviz -i flows.txt -f "table=0" openflow logic
> +    Ofproto Flows (logical)
> +    └── ** TABLE 0 **
> +        ├── priority=180 priority,vlan_tci  --->  conjunction ( x 1 )
> +        ├── priority=180 priority,conj_id,in_port,vlan_tci  --->  load,load,load,load,load,mod_dl_src resubmit(,8), ( x 1 )
> +        ├── priority=100 priority,in_port  --->  move,move,move resubmit(,40), ( x 2 )
> +        ├── priority=100 priority,in_port  --->  load,load,load,load,load resubmit(,8), ( x 16 )
> +        ├── priority=100 priority,in_port,vlan_tci  --->  load,load,load,load,load resubmit(,8), ( x 1 )
> +        ├── priority=100 priority,in_port,dl_vlan  --->  strip_vlan,load,load,load,load,load resubmit(,8), ( x 1 )
> +        └── priority=0 priority  --->   drop, ( x 1 )
> +
> +
> +In only a few logical blocks, we have a good overview of what this table is
> +doing. It looks like it's adding metadata based on input ports and vlan
> +IDs and mainly sending traffic to table 8.
> +
> +Let's look at table 8, an in this case, let's filter out the flows that have
> +not been hit by actual traffic. This is quite easy to do with the arithmetic
> +filtering expressions:
> +::
> +
> +   $ ovs-flowviz -i flows.txt -f "table=8 and n_packets>0" openflow logic
> +
> +    Ofproto Flows (logical)
> +    └── ** TABLE 8 **
> +        ├── priority=50 priority,reg14,metadata,dl_dst  --->  load resubmit(,9), ( x 3 )
> +        └── priority=50 priority,metadata  --->  load,move resubmit(,73),resubmit(,9), ( x 2 )
> +
> +At this point, we might find ourselves a bit lost since we may not remember
> +what metadata OVN stored in the previous table. Here is where
> +``ovs-flowviz``'s OVN integration could come useful. Let's connect to the
> +running OVN instance and ask it about the flows we're looking at.
> +
> +::
> +
> +    $ export OVN_NB_DB=tcp:172.18.0.4:6641
> +    $ export OVN_SB_DB=tcp:172.18.0.4:6642
> +    $ ovs-flowviz -i flows.txt -f "table=8 and n_packets>0" openflow logic --ovn-detrace
> +    Ofproto Flows (logical)
> +    └── ** TABLE 8 **
> +        ├── cookie=0xe10c34ee priority=50 priority,reg14,metadata,dl_dst  --->  load resubmit(,9), ( x 1 )
> +        │   └── OVN Info
> +        │       ├── *  Logical datapaths:
> +        │       ├── *      "ovn_cluster_router" (366e1c41-0f3d-4420-b796-10692b64e3e4)
> +        │       ├── *  Logical flow: table=0 (lr_in_admission), priority=50, match=(eth.mcast && inport == "rtos-ovn-worker2), actions=(xreg0[0..47] = 0a:58:0a:f4:01:01; next;)
> +        │       └── *  Logical Router Port: rtos-ovn-worker2 mac 0a:58:0a:f4:01:01 networks ['10.244.1.1/24'] ipv6_ra_configs {}
> +        ├── cookie=0x11e1adbc priority=50 priority,reg14,metadata,dl_dst  --->  load resubmit(,9), ( x 1 )
> +        │   └── OVN Info
> +        │       ├── *  Logical datapaths:
> +        │       ├── *      "GR_ovn-worker2" (c07f8387-6479-4e81-9304-9f8e54f81c56)
> +        │       ├── *  Logical flow: table=0 (lr_in_admission), priority=50, match=(eth.mcast && inport == "rtoe-GR_ovn-worker2), actions=(xreg0[0..47] = 02:42:ac:12:00:03; next;)
> +        │       └── *  Logical Router Port: rtoe-GR_ovn-worker2 mac 02:42:ac:12:00:03 networks ['172.18.0.3/16'] ipv6_ra_configs {}
> +        ├── cookie=0xf42133f  priority=50 priority,reg14,metadata,dl_dst  --->  load resubmit(,9), ( x 1 )
> +        │   └── OVN Info
> +        │       ├── *  Logical datapaths:
> +        │       ├── *      "GR_ovn-worker2" (c07f8387-6479-4e81-9304-9f8e54f81c56)
> +        │       ├── *  Logical flow: table=0 (lr_in_admission), priority=50, match=(eth.dst == 02:42:ac:12:00:03 && inport == "rtoe-GR_ovn-worker2), actions=(xreg0[0..47] = 02:42:ac:12:00:03; next;)
> +        │       └── *  Logical Router Port: rtoe-GR_ovn-worker2 mac 02:42:ac:12:00:03 networks ['172.18.0.3/16'] ipv6_ra_configs {}
> +        └── cookie=0x43a0327  priority=50 priority,metadata  --->  load,move resubmit(,73),resubmit(,9), ( x 2 )
> +            └── OVN Info
> +                ├── *  Logical datapaths:
> +                ├── *      "ovn-worker" (24280d0b-fee0-4f8e-ba4f-036a9b9af921)
> +                ├── *      "ovn-control-plane" (3262a782-8961-416b-805e-08233e8fda72)
> +                ├── *      "ext_ovn-worker2" (3f88dcd2-c56d-478f-a3b1-c7aee2efe967)
> +                ├── *      "ext_ovn-worker" (5facbaf0-485d-4cf5-8940-eff9678ef7bb)
> +                ├── *      "ext_ovn-control-plane" (8b0aecb6-b05a-48a7-ad09-72524bb91d40)
> +                ├── *      "join" (e2dc230e-2f2a-4b93-93fa-0fe495163514)
> +                ├── *      "ovn-worker2" (f7709fbf-d728-4cff-9b9b-150461cc75d2)
> +                └── *  Logical flow: table=0 (ls_in_check_port_sec), priority=50, match=(1), actions=(reg0[15] = check_in_port_sec(); next;)
> +
> +That's way better. ``ovs-flowviz`` has automatically added the `cookie` to the
> +logical block key so have more blocks but in exchange, it has looked up each
> +cookie on the running OVN databases and inserted the known information on each
> +block. So now we see what OVN is trying to do, the logical flow that generated
> +each OpenFlow flow and the logical datapath each flow belongs to.
> +
> +Visualizing datapath flow trees
> +-------------------------------
> +
> +Now, let's see another typical usecase that can lead to eyestrain:
> +understanding datapath conntrack recirculations.
> +
> +OVS makes heavy use of connection tracking and the ``recirc()`` action
> +to build complex datapaths. Typically, OVS will insert a flow that,
> +when matched, will send the packet through conntrack (using the ``ct`` action)
> +and recirculate it with a particular recirculation id (``recirc_id``). Then, a
> +flow matching on that ``recirc_id`` will be matched and further process the
> +packet. This can happen more than once for a given packet.
> +
> +This sequential set of events is, however, difficult to visualize when you
> +look at a datapath flow dump. Flows are unordered recirculations that need to
> +be followed manually (typically, with heavy use of "grep").
> +
> +For this use-case, ``ovs-flowviz datapath tree`` format can be extremely
> +useful. It builds a hierarchical tree based on the ``recirc_id`` matches and
> +``recirc()`` actions and indents flows based on it.
> +
> +Here is an example.
> +::
> +
> +    ── recirc_id(0),in_port(3),eth(...),ipv4(...),tcp(dst=8181), actions:ct(zone=2,nat),recirc(0x19348)
> +    │   ├── recirc_id(0x19348),in_port(3),ct_state(-new+est-rel-rpl-inv+trk),ct_label(0/0x3),eth(...),eth_type,ipv4(), actions:ct(zone=27,nat),recirc(0x10)
> +    │   │   ├── recirc_id(0x10),in_port(3),ct_state(-new+est-rel-rpl-inv+trk),eth(...),ipv4(...), actions:9
> +    │   │   ├── recirc_id(0x10),in_port(3),ct_state(-new+est-rel+rpl-inv+trk),eth(...),ipv4(...), actions:9
> +    │   │   └── recirc_id(0x10),in_port(3),ct_state(+new-est-rel-rpl-inv+trk),eth(...),ipv4(...), actions:ct(commit,zone=27,label=0/0x1),9
> +    │   └── recirc_id(0x19348),in_port(3),ct_state(+new-est-rel-rpl-inv+trk),eth(...),ipv4(...),  actions:ct(commit,zone=2,label=0/0x1),ct(zone=27,nat),recirc(0x10)
> +    │       ├── recirc_id(0x10),in_port(3),ct_state(-new+est-rel-rpl-inv+trk),eth(...),ipv4(...), actions:9
> +    │       ├── recirc_id(0x10),in_port(3),ct_state(-new+est-rel+rpl-inv+trk),eth(...),ipv4(...), actions:9
> +    │       └── recirc_id(0x10),in_port(3),ct_state(+new-est-rel-rpl-inv+trk),eth(...),ipv4(...), actions:ct(commit,zone=27,label=0/0x1),9
> +
> +The above shows a typical conntrack recirculation flow.
> +The first flow (with ``recir_id(0)``) sends the packet through conntrack
> +system and recirculates with ``recirc_id(0x19348)``.
> +Then, based on the ``ct_state`` the packet processing branches out into two
> +flows. Each flow resends the packet through conntrack and recirculates the
> +packet one more time. Finally, the packet is processed by 3 flows
> +on ``recirc_id(10)``.
> +
> +This 3-stage processing is now very clear.
> +
> +Note that this format can yield longer outputs since some flows (in this
> +example those with ``recirc_id(10)`` can be repeated. However, the result
> +is a clear representation of an otherwise difficult to see conntrack
> +interaction.
> +
> +This example shows only a single "subtree". If we use this command to display
> +a big flow dump, the output can be lengthy. Here are two (combinable) ways to
> +help out.
> +
> +Plotting datapath trees
> +~~~~~~~~~~~~~~~~~~~~~~~
> +
> +By using the ``ovs-flowviz datapath html`` format, long datapath trees can
> +be displayed in an interactive HTML table. The resulting web allows you to
> +collapse and expand subtrees so you can focus on what you're looking for.
> +
> +In addition, the ``ovs-flowviz datapath graph`` format generates a graphviz
> +graph definition where each block of flows with the same ``recirc_id`` match
> +are arranged together and edges are created to represent recirculations.
> +Also, this format comes with further goodies such as displaying the conntrack
> +zones which are key to understanding what the datapath is really doing with a
> +packet.
> +
> +These two formats (``html`` and ``graph``) can even be combined. By using the
> +``ovs-flowviz datapath graph --html`` command, you'll get an interactive
> +HTML table alongside a `svg` graphical representation of the flows. Click on
> +a flow on the svg and it'll take you to the corresponding entry in the
> +flow table.
> +
> +
> +Filtering
> +~~~~~~~~~
> +
> +Apart from being able to expand and collapse subtrees, we can use filtering.
> +
> +However, filtering works in a slightly different way compared with OpenFlow
> +flows. Instead of just removing non-matching flows, the output
> +of a filtered datapath flow tree will show full sub-trees that contain at
> +least one flow that satisfies the filter.
> +
> +For example, let's take the flows in the above example, and let's imagine we
> +want to understand what traffic is going out on port ``9``. We could run
> +the tool as:
> +::
> +
> +   $ ovs-appctl dpctl/dump-flows | ovs-flowviz -f "output.port=9" datapath tree
> +
> +The resulting flow tree will contain all of the flows above, even those
> +with ``recirc_id(0)`` and ``recirc_id(19348)`` that don't actually output
> +traffic to port ``9``. Why? because they are all part of a subtree that
> +contains flows that do output packets on port ``9``
> +
> +That way, we see the "full picture" of how traffic on port ``9`` is being
> +processed.
> +
> +.. _ovs-flowviz(8): https://docs.openvswitch.org/en/latest/ref/ovs-flowviz.8
> diff --git a/Documentation/topics/index.rst b/Documentation/topics/index.rst
> index f239fcf83..9ddb145dd 100644
> --- a/Documentation/topics/index.rst
> +++ b/Documentation/topics/index.rst
> @@ -58,3 +58,4 @@ OVS
>     userspace-checksum-offloading
>     userspace-tx-steering
>     usdt-probes
> +   flow-visualization
> -- 
> 2.44.0
>
> _______________________________________________
> dev mailing list
> dev@openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev

On 4/12/24 11:10, Eelco Chaudron wrote:
> 
> 
> On 9 Apr 2024, at 9:06, Adrian Moreno wrote:
> 
>> Add a man page for ovs-flowviz as well as a topic page with some more
>> detailed examples.
>>
>> Signed-off-by: Adrian Moreno <amorenoz@redhat.com>
> 
> In addition to Ilya’s comments, find 3 small comments below. The rest looks good.
> 

Thanks. Will send another version.

> Cheers,
> 
> Eelco
> 
>> ---
>>   Documentation/automake.mk                   |   4 +-
>>   Documentation/conf.py                       |   2 +
>>   Documentation/ref/index.rst                 |   1 +
>>   Documentation/ref/ovs-flowviz.8.rst         | 531 ++++++++++++++++++++
>>   Documentation/topics/flow-visualization.rst | 271 ++++++++++
>>   Documentation/topics/index.rst              |   1 +
>>   6 files changed, 809 insertions(+), 1 deletion(-)
>>   create mode 100644 Documentation/ref/ovs-flowviz.8.rst
>>   create mode 100644 Documentation/topics/flow-visualization.rst
>>
>> diff --git a/Documentation/automake.mk b/Documentation/automake.mk
>> index 47d2e336a..539870aa2 100644
>> --- a/Documentation/automake.mk
>> +++ b/Documentation/automake.mk
>> @@ -45,7 +45,7 @@ DOC_SOURCE = \
>>   	Documentation/topics/fuzzing/ovs-fuzzing-infrastructure.rst \
>>   	Documentation/topics/fuzzing/ovs-fuzzers.rst \
>>   	Documentation/topics/fuzzing/security-analysis-of-ovs-fuzzers.rst \
>> -	Documentation/topics/testing.rst \
>> +	Documentation/topics/flow-visualization.rst \
>>   	Documentation/topics/integration.rst \
>>   	Documentation/topics/language-bindings.rst \
>>   	Documentation/topics/networking-namespaces.rst \
>> @@ -55,6 +55,7 @@ DOC_SOURCE = \
>>   	Documentation/topics/ovsdb-replication.rst \
>>   	Documentation/topics/porting.rst \
>>   	Documentation/topics/record-replay.rst \
>> +	Documentation/topics/testing.rst \
>>   	Documentation/topics/tracing.rst \
>>   	Documentation/topics/usdt-probes.rst \
>>   	Documentation/topics/userspace-checksum-offloading.rst \
>> @@ -162,6 +163,7 @@ RST_MANPAGES = \
>>   	ovs-actions.7.rst \
>>   	ovs-appctl.8.rst \
>>   	ovs-ctl.8.rst \
>> +	ovs-flowviz.8.rst \
>>   	ovs-l3ping.8.rst \
>>   	ovs-parse-backtrace.8.rst \
>>   	ovs-pki.8.rst \
>> diff --git a/Documentation/conf.py b/Documentation/conf.py
>> index 15785605a..3a82f23a7 100644
>> --- a/Documentation/conf.py
>> +++ b/Documentation/conf.py
>> @@ -120,6 +120,8 @@ _man_pages = [
>>        u'utility for configuring running Open vSwitch daemons'),
>>       ('ovs-ctl.8',
>>        u'OVS startup helper script'),
>> +    ('ovs-flowviz.8',
>> +     u'utility for visualizing OpenFlow and datapath flows'),
>>       ('ovs-l3ping.8',
>>        u'check network deployment for L3 tunneling problems'),
>>       ('ovs-parse-backtrace.8',
>> diff --git a/Documentation/ref/index.rst b/Documentation/ref/index.rst
>> index 03ada932f..7f2fe6177 100644
>> --- a/Documentation/ref/index.rst
>> +++ b/Documentation/ref/index.rst
>> @@ -42,6 +42,7 @@ time:
>>      ovs-actions.7
>>      ovs-appctl.8
>>      ovs-ctl.8
>> +   ovs-flowviz.8
>>      ovs-l3ping.8
>>      ovs-pki.8
>>      ovs-sim.1
>> diff --git a/Documentation/ref/ovs-flowviz.8.rst b/Documentation/ref/ovs-flowviz.8.rst
>> new file mode 100644
>> index 000000000..b5b1befb0
>> --- /dev/null
>> +++ b/Documentation/ref/ovs-flowviz.8.rst
>> @@ -0,0 +1,531 @@
>> +..
>> +      Licensed under the Apache License, Version 2.0 (the "License"); you may
>> +      not use this file except in compliance with the License. You may obtain
>> +      a copy of the License at
>> +
>> +          http://www.apache.org/licenses/LICENSE-2.0
>> +
>> +      Unless required by applicable law or agreed to in writing, software
>> +      distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
>> +      WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
>> +      License for the specific language governing permissions and limitations
>> +      under the License.
>> +
>> +      Convention for heading levels in Open vSwitch documentation:
>> +
>> +      =======  Heading 0 (reserved for the title in a document)
>> +      -------  Heading 1
>> +      ~~~~~~~  Heading 2
>> +      +++++++  Heading 3
>> +      '''''''  Heading 4
>> +
>> +      Avoid deeper levels because they do not render well.
>> +
>> +===========
>> +ovs-flowviz
>> +===========
>> +
>> +Synopsis
>> +========
>> +
>> +``ovs-flowviz``
>> +[``[-i | --input] <[alias,]file>``]
>> +[``[-c | --config] <file>``]
>> +[``[-f | --filter] <filter>``]
>> +[``[-h | --highlight] <filter>``]
>> +[``--style <style>``]
>> +*<flow_type>* *<format>* [<arg>...]
>> +
>> +``ovs-flowviz --help``
>> +
>> +Description
>> +===========
>> +
>> +The ``ovs-flowviz`` program helps visualize OpenFlow and datapath flow dumps
>> +in different formats in order to make them more easily understood.
>> +
>> +The program works by reading flows from ``stdin`` or from a file specified
>> +in the ``--input`` option, filtering them, highlighting them, and finally
>> +outputting them in one of the predefined formats.
>> +
>> +
>> +Options
>> +=======
>> +
>> +.. program: ovs-flowviz
>> +
>> +.. option:: -h, --help
>> +
>> +    Prints a brief help message to the console.
>> +
>> +.. option:: -i <[alias,]file>, --input <[alias,]file>
>> +
>> +    Specifies the file to read flows from. If not provided, ``ovs-flowviz``
>> +    will read flows from stdin.
>> +
>> +    This option can be specified multiple times.
>> +    The file path can prepended by an alias that will be shown in the output.
>> +    For example: ``--input node1,/path/to/file1 --input node2,/path/to/file2``
>> +
>> +.. option:: -c <file>, --config <file>
>> +
>> +    Specifies the style configuration file to use. ``ovs-flowviz`` ships with
>> +    a default configuration file but it can be overridden using this option.
>> +    Styles defined in the style configuration file will be select-able using
>> +    the ``--style`` option.
>> +
>> +    For more details on the style configuration file, see
>> +    `Style Configuration File`_ section below.
>> +
>> +.. option:: -f <filter>, --filter <filter>
>> +
>> +   Tells ``ovs-flowviz`` to filter the flows and only show the ones that match
>> +   the expression (although some formats implement filtering differently,
>> +   see `Datapath tree format`_ below).
>> +
>> +   The filtering syntax is detailed in `Filtering Syntax`_.
>> +
>> +.. option:: -h <filter>, --highlight <filter>
>> +
>> +   Tells ``ovs-flowviz`` to highlight the flows that match the provided filter
>> +
>> +   The filtering syntax is detailed in `Filtering Syntax`_.
>> +
>> +.. option:: --style <style>
>> +
>> +   Specifies the style to use. The style must have been defined in the
>> +   style configuration file.
>> +
>> +.. option:: <flow_type>
>> +
>> +   "openflow" or "datapath".
>> +
>> +.. option:: <format>
>> +
>> +   See `Supported formats`_ section.
>> +
>> +
>> +Supported formats
>> +=================
>> +
>> +``ovs-flowviz`` supports several visualization formats for both OpenFlow and
>> +datapath flows that are summarized in the following table:
>> +
>> +.. list-table::
>> +   :widths: 20 10 70
>> +   :align: center
>> +   :header-rows: 1
>> +
>> +   * - Flow Type
>> +     - Format
>> +     - Description
>> +   * - Both
>> +     - console
>> +     - Prints the flows in a configurable, colorful style in the console.
>> +   * - Both
>> +     - json
>> +     - Prints the flows in JSON format.
>> +   * - Both
>> +     - html
>> +     - Prints the flows in an HTML list.
>> +   * - Openflow
>> +     - cookie
>> +     - Prints the flows in the console sorted by cookie.
>> +   * - Openflow
>> +     - logic
>> +     - Prints the logical structure of flows in the console.
>> +   * - Datapath
>> +     - tree
>> +     - Prints the flows a tree structure arranged by `recirc_id`.
>> +   * - Datapath
>> +     - graph
>> +     - Prints a graphviz graph of the flows arranged by `recirc_id`.
>> +
>> +
>> +Console format
>> +~~~~~~~~~~~~~~
>> +
>> +The ``console`` works for both OpenFlow and datapath flow types and prints
>> +flows in the terminal with the style determined by the ``--style`` option.
>> +
>> +Additionally, it accepts the following arguments:
>> +
>> +.. option:: -h, --heat-map
>> +
>> +   This option changes the color of the packet and byte counters to reflect
>> +   their relative size. The color gradient goes through the following colors:
>> +   blue (coldest, lowest), cyan, green, yellow, red (hottest, highest)
>> +
>> +   Note filtering is applied before the range is calculated.
>> +
>> +
>> +JSON format
>> +~~~~~~~~~~~
>> +
>> +The ``json`` format works for both OpenFlow and datapath flow types and prints
>> +flows in JSON format. See `JSON Syntax`_ for more details.
>> +
>> +
>> +HTML format
>> +~~~~~~~~~~~
>> +
>> +The ``html`` format works for both OpenFlow and datapath flows and prints
>> +flows in an HTML table that offers some basic interactivity. OpenFlow flows
>> +are sorted in tables and datapath flows are arranged in flow trees
>> +(see `Datapath tree format`_ for more details).
>> +
>> +Styles defined via Style Configuration File and selected via ``--style`` option
>> +also apply to ``html`` format.
>> +
>> +
>> +OpenFlow cookie format
>> +~~~~~~~~~~~~~~~~~~~~~~
>> +
>> +The OpenFlow ``cookie`` format is similar to the ``console`` format but
>> +instead of arranging the flows per table, it arranges the flows per cookie.
>> +
>> +
>> +Openflow logic format
>> +~~~~~~~~~~~~~~~~~~~~~
>> +
>> +The OpenFlow ``logic`` format helps visualize the logic structure of OpenFlow
>> +pipelines by arranging flows into *logical blocks*.
>> +A logical block is a set of flows that have:
>> +
>> +* Same ``priority``.
>> +* Match on the same fields (regardless of the match value and mask).
>> +* Execute the same actions (regardless of the actions' arguments,
>> +  except for resubmit and output).
>> +* Optionally, the ``cookie`` can be counted as part of the logical flow.
>> +
>> +This format supports the following extra arguments:
>> +
>> +.. option:: -s, --show-flows
>> +
>> +    Show all the flows under each logical block.
>> +
>> +.. option:: -d, --ovn-detrace
>> +
>> +    Use ovn-detrace.py script to extract cookie information (implies '-c').
>> +
>> +.. option:: -c, --cookie
>> +
>> +    Consider the cookie in the logical block.
>> +
>> +.. option:: --ovn-detrace-path <path>
>> +
>> +    Use an alternative path to look for ovn_detrace.py script.
>> +
>> +.. option:: --ovnnb-db text
>> +
>> +   Specify the OVN NB database string (implies '-d').
>> +   Default value is "unix:/var/run/ovn/ovnnb_db.sock".
>> +
>> +.. option:: --ovnsb-db text
>> +
>> +   Specify the OVN SB database string (implies '-d').
>> +   Default value is "unix:/var/run/ovn/ovnsb_db.sock".
>> +
>> +.. option:: --o <text>, --ovn-filter <text>
>> +
>> +   Specify the a filter to be run on the ovn-detrace information.
>> +   Syntax: python regular expression
>> +   (See https://docs.python.org/3/library/re.html).
>> +
>> +.. option:: -h, --heat-map
>> +
>> +   This option changes the color of the packet and byte counters to reflect
>> +   their relative size. The color gradient goes through the following colors:
>> +   blue (coldest, lowest), cyan, green, yellow, red (hottest, highest)
>> +
>> +   Note filtering is applied before the range is calculated.
>> +
>> +
>> +Datapath tree format
>> +~~~~~~~~~~~~~~~~~~~~
>> +
>> +The datapath ``tree`` format arranges datapath flows in a hierarchical tree
>> +based on `recirc_id`. At the first level, flows with `recirc_id(0)` are
>> +listed. If a flow contains a `recirc()` action with a specific `recirc_id`,
>> +flows matching on that `recirc_id` are listed below. This is done recursively
>> +for all actions.
>> +
>> +The result is a hierarchical representation that helps understand how actions
>> +are related to each other via recirculation. Note flows with a specific
>> +non-zero `recirc_id` are listed below each flow that has a corresponding
>> +`recirc()` action. Therefore, they would be duplicated leading to a longer
>> +output.
>> +
>> +Also, filtering works in a slightly different way for datapath flow trees.
>> +Unlike other formats where a filter simply removes non-matching flows,
>> +the output of a filtered datapath flow tree will show full sub-trees
>> +that contain at least one flow that satisfies the filter.
>> +
>> +The ``html`` format prints this same tree in an interactive HTML table.
>> +
>> +
>> +Datapath graph format
>> +~~~~~~~~~~~~~~~~~~~~~
>> +
>> +The datapath ``graph`` generates a graphviz visual representation of the
>> +same tree-like flow hierarchy that the ``tree`` format prints.
>> +
>> +It supports the following extra argument:
>> +
>> +.. option:: -h, --html
>> +
>> +    Prints the graphviz format in an svg image alongside the interactive HTML
>> +    table of flows (that 'html' format would print).
>> +
>> +
>> +JSON Syntax
>> +===========
>> +
>> +Both OpenFlow and datapath `json` formats print a JSON list of JSON
>> +objects each of one representing an individual flow.S
> 
> There is a trailing S.
> 
>> +
>> +Each flow object contains the following keys:
>> +
>> +**orig**
>> +    Contains the original flow string.
>> +
>> +
>> +**info**
>> +   Contains an object with the flow information
>> +   such as: cookie, duration, table, n_packets, n_bytes, etc.
>> +
>> +
>> +**match**
>> +   Contains an object with the flow match.
>> +   For each match, the object contains a key-value where the key is the name
>> +   of the match as defined in ovs-fields and ovs-ofctl and the value
>> +   represents the match value. The way each value is represented depends on its
>> +   type. See `Value representation`_.
>> +
>> +
>> +**actions**
>> +   Contains a list of action objects.
>> +   Each action is represented by an JSON object that has one key and one value.
>> +   The key corresponds to the action name. The value represents the arguments
>> +   of such key. See `Action representation`_.
>> +
>> +
>> +**ufid**
>> +   (datapath flows only) Contains the ufid.
>> +
>> +
>> +Value representation
>> +~~~~~~~~~~~~~~~~~~~~
>> +
>> +Values are represented differently depending on their type:
>> +
>> +* Flags: Fields that represent flags (e.g: tcp) are represented by boolean
>> +  "true"
>> +
>> +* Decimal / Hexadecimal: They are represented by their integer value.
>> +  If they support masking, they are represented by a dictionary with two keys:
>> +  value contains the field value and mask contains the mask. Both are integers.
>> +
>> +* Ethernet: They are represented by a string: {address}[/{mask}]
>> +
>> +* IPv4 / IPv6: They are represented by a string {address}[/mask]
>> +
>> +* Registers: They are represented by a dictionary with three keys:
>> +  field contains the field value (string), start and end that represent the
>> +  first and last bit of the register.
>> +
>> +For example, the register
>> +::
>> +
>> +
>> +   NXM_NX_REG10[0..15]
>> +
>> +
>> +is represented as
>> +::
>> +
>> +
>> +   {
>> +       "field": "NXM_NX_REG10",
>> +       "start": 0,
>> +       "end": 15
>> +   },
>> +
>> +
>> +Action representation
>> +~~~~~~~~~~~~~~~~~~~~~
>> +
>> +Actions are generally represented by an object that has a single key and a
>> +value. The key is the action name as defined ovs-actions.
>> +
>> +The value of actions that have no arguments (such as ``drop``) is
>> +(boolean) ``true``.
>> +
>> +The value of actions that have a list of arguments (e.g:
>> +``resubmit([port],[table],[ct])``) is an object that has the name of the
>> +argument as key. The argument names for each action is defined in
>> +ovs-actions. For example, the action
>> +::
>> +
>> +   resubmit(,10)
>> +
>> +is represented as
>> +::
>> +
>> +   {
>> +       "redirect": {
>> +           "port": "",
>> +           "table": 10
>> +       }
>> +   }
>> +
>> +The value of actions that have a key-word list as arguments
>> +(e.g: ``ct([argument])``) is an object whose keys correspond to the keys
>> +defined in ``ovs-actions(7)``. The way values are represented depends
>> +on the type of the argument.
>> +For example, the action
>> +::
>> +
>> +   ct(table=14,zone=NXM_NX_REG12[0..15],nat)
>> +
>> +is represented as
>> +::
>> +
>> +   {
>> +       "ct": {
>> +           "table": 14,
>> +           "zone": {
>> +               "field": "NXM_NX_REG12",
>> +               "start": 0,
>> +               "end": 15
>> +           },
>> +           "nat": true
>> +       }
>> +   }
>> +
>> +
>> +Style Configuration File
>> +========================
>> +
>> +The style configuration file that can be selected via the ``--config`` option
>> +has INI syntax and can define any number of styles to be used by both
>> +``console`` and ``html`` formats. Once defined in the configuration file
>> +they can be selected using the ``--style`` option.
>> +
>> +INI sections are used to define styles, ``[styles.mystyle]`` defines a style
>> +called `mystle`. Within a section styles can be defined as:
>> +
>> +::
>> +
>> +     [FORMAT].[PORTION].[SELECTOR].[ELEMENT] = [VALUE]
>> +
>> +
>> +**FORMAT**
>> +   Either ``console`` or ``html``
>> +
>> +**PORTION**
>> +   The part of the a key-value the style applies to. It can be:
>> +   ``key`` (to indicate the key part of a key-value), ``value`` (to indicate
>> +   the value part of a key-value), ``flag`` (to indicate a single flag)
>> +   or ``delim`` (to indicate delimiters such as parentheses, brackets, etc).
>> +
>> +**SELECTOR**
>> +   Is used to select what key-value the style applies to. It can be:
>> +   ``highlighted`` (to indicate highlighted key-values), ``type.<type>``
>> +   to indicate certain types such as `IPAddress` or `EthMask` or `<keyname>`
>> +   to select a particular key name.
>> +
>> +**ELEMENT**
>> +   Is used to select what style element to modify. It can be one
>> +   of: **color** or **underline** (only for **console** format).
>> +
>> +**VALUE**
>> +   Is either a color hex, other color names defined in the rich python
>> +   library (https://rich.readthedocs.io/en/stable/appendix/colors.html) or
>> +   "true" if the element is ``underline``.
>> +
>> +A default configuration file is shipped with the tool and it's path is printed
> 
> it’s -> its
> 
>> +in the ``--help`` output. A detailed description of the syntax alongside
>> +some examples is available there.
> 
> is available -> are available
> 
>> +
>> +
>> +Filtering syntax
>> +================
>> +
>> +``ovs-flowviz`` provides rich highlighting and filtering. The special command
>> +``ovs-flowviz filter`` dumps the filtering syntax:
>> +
>> +::
>> +
>> +    $ ovs-flowviz filter
>> +    Filter Syntax
>> +    *************
>> +
>> +       [! | not ] {key}[[.subkey]...] [OPERATOR] {value})] [LOGICAL OPERATOR] ...
>> +
>> +      Comparison operators are:
>> +          =   equality
>> +          <   less than
>> +          >   more than
>> +          ~=  masking (valid for IP and Ethernet fields)
>> +
>> +      Logical operators are:
>> +          !{expr}:  NOT
>> +          {expr} && {expr}: AND
>> +          {expr} || {expr}: OR
>> +
>> +      Matches and flow metadata:
>> +          To compare against a match or info field, use the field directly, e.g:
>> +              priority=100
>> +              n_bytes>10
>> +          Use simple keywords for flags:
>> +              tcp and ip_src=192.168.1.1
>> +
>> +      Actions:
>> +          Actions values might be dictionaries, use subkeys to access individual
>> +          values, e.g:
>> +              output.port=3
>> +          Use simple keywords for flags
>> +              drop
>> +
>> +      Examples of valid filters.
>> +          nw_addr~=192.168.1.1 && (tcp_dst=80 || tcp_dst=443)
>> +          arp=true && !arp_tsa=192.168.1.1
>> +          n_bytes>0 && drop=true
>> +
>> +
>> +Example expressions:
>> +::
>> +
>> +   n_bytes > 0 and drop
>> +   nw_src~=192.168.1.1 or arp.tsa=192.168.1.1
>> +   ! tcp && output.port=2
>> +
>> +
>> +Examples
>> +========
>> +
>> +Print OpenFlow flows sorted by cookie adding OVN data to each one:
>> +::
>> +
>> +    $ ovs-flowviz -i flows.txt openflow cookie --ovn-detrace
>> +
>> +Print OpenFlow logical structure, showing the flows and heat-map:
>> +::
>> +
>> +    $ ovs-flowviz -i flows.txt openflow logic --show-flows --heat-map
>> +
>> +Display OpenFlow flows in HTML format with "light" style and highlight drops:
>> +::
>> +
>> +    $ ovs-flowviz -i flows.txt --style "light" --highlight "n_packets > 0 and drop" openflow html > flows.html
>> +
>> +Display the datapath flows in an interactive graphviz + HTML view:
>> +::
>> +
>> +    $ ovs-flowviz -i flows.txt datapath graph --html > flows.html
>> +
>> +Display the datapath flow trees that lead to packets being sent to port 10:
>> +::
>> +
>> +    $ ovs-flowviz -i flows.txt --filter "output.port=10" datapath tree
>> diff --git a/Documentation/topics/flow-visualization.rst b/Documentation/topics/flow-visualization.rst
>> new file mode 100644
>> index 000000000..62d0d6bd8
>> --- /dev/null
>> +++ b/Documentation/topics/flow-visualization.rst
>> @@ -0,0 +1,271 @@
>> +..
>> +      Licensed under the Apache License, Version 2.0 (the "License"); you may
>> +      not use this file except in compliance with the License. You may obtain
>> +      a copy of the License at
>> +
>> +          http://www.apache.org/licenses/LICENSE-2.0
>> +
>> +      Unless required by applicable law or agreed to in writing, software
>> +      distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
>> +      WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
>> +      License for the specific language governing permissions and limitations
>> +      under the License.
>> +
>> +      Convention for heading levels in Open vSwitch documentation:
>> +
>> +      =======  Heading 0 (reserved for the title in a document)
>> +      -------  Heading 1
>> +      ~~~~~~~  Heading 2
>> +      +++++++  Heading 3
>> +      '''''''  Heading 4
>> +
>> +      Avoid deeper levels because they do not render well.
>> +
>> +==================================
>> +Visualizing flows with ovs-flowviz
>> +==================================
>> +
>> +When troubleshooting networking issues with OVS, we typically end up looking
>> +at OpenFlow or datapath flow dumps. These dumps tend to be quite dense and
>> +difficult to reason about.
>> +
>> +``ovs-flowviz`` is a utility script that helps visualizing OpenFlow and
>> +datapath flows to make it easier to understand what is going on.
>> +
>> +The `ovs-flowviz(8)`_ manpage describes its basic usage. In this document a few
>> +of its advanced visualization formats will be expanded.
>> +
>> +
>> +Installing ovs-flowviz
>> +----------------------
>> +
>> +``ovs-flowviz`` is part of the openvswitch python package but its
>> +extra dependencies have to be installed explicitly by running:
>> +::
>> +
>> +    $ pip install openvswitch[flowviz]
>> +
>> +Or, if you are working with the OVS tree:
>> +::
>> +
>> +    $ cd python && pip install .[flowviz]
>> +
>> +Visualizing OpenFlow logical block
>> +----------------------------------
>> +
>> +When controllers such as OVN write OpenFlow flows, they typically organize
>> +flows in functional blocks. These blocks can expand to multiple flows that
>> +"look similar", in the sense that they match on the same fields and have
>> +similar actions.
>> +
>> +However, when we look at a flow dump the number of flows can make it difficult
>> +to perceive this logical functionality that the controller is trying to
>> +implement using OpenFlow.
>> +
>> +In this example, we are going to use ``ovs-flowviz openflow logic``
>> +visualization to understand an OVN flow dump a bit better.
>> +
>> +On a particular flow dump we have 23 flows in table 0:
>> +::
>> +
>> +   $ grep -c "table=0" flows.txt
>> +   23
>> +
>> +If we look at the first few lines, the amount of information can be
>> +overwhelming and difficult our analysis:
>> +
>> +::
>> +
>> +    $ head flows.txt
>> +      cookie=0xf76b4b20, duration=765.107s, table=0, n_packets=0, n_bytes=0, priority=180,vlan_tci=0x0000/0x1000 actions=conjunction(100,2/2)
>> +      cookie=0xf76b4b20, duration=765.107s, table=0, n_packets=0, n_bytes=0, priority=180,conj_id=100,in_port="patch-br-int-to",vlan_tci=0x0000/0x1000 actions=load:0xa->NXM_NX_REG13[],load:0xc->NXM_NX_REG11[],load:0xb->NXM_NX_REG12[],load:0xb->OXM_OF_METADATA[],load:0x1->NXM_NX_REG14[],mod_dl_src:02:42:ac:12:00:03,resubmit(,8)
>> +      cookie=0x0, duration=765.388s, table=0, n_packets=0, n_bytes=0, priority=100,in_port="ovn-6bb3b3-0" actions=move:NXM_NX_TUN_ID[0..23]->OXM_OF_METADATA[0..23],move:NXM_NX_TUN_METADATA0[16..30]->NXM_NX_REG14[0..14],move:NXM_NX_TUN_METADATA0[0..15]->NXM_NX_REG15[0..15],resubmit(,40)
>> +      cookie=0x0, duration=765.388s, table=0, n_packets=0, n_bytes=0, priority=100,in_port="ovn-a6ff98-0" actions=move:NXM_NX_TUN_ID[0..23]->OXM_OF_METADATA[0..23],move:NXM_NX_TUN_METADATA0[16..30]->NXM_NX_REG14[0..14],move:NXM_NX_TUN_METADATA0[0..15]->NXM_NX_REG15[0..15],resubmit(,40)
>> +      cookie=0xf2ca6195, duration=765.107s, table=0, n_packets=6, n_bytes=636, priority=100,in_port="ovn-k8s-mp0" actions=load:0x1->NXM_NX_REG13[],load:0x2->NXM_NX_REG11[],load:0x7->NXM_NX_REG12[],load:0x4->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],resubmit(,8)
>> +      cookie=0x236e941d, duration=408.874s, table=0, n_packets=11, n_bytes=846, priority=100,in_port=aceac9829941d11 actions=load:0x11->NXM_NX_REG13[],load:0x2->NXM_NX_REG11[],load:0x7->NXM_NX_REG12[],load:0x4->OXM_OF_METADATA[],load:0x3->NXM_NX_REG14[],resubmit(,8)
>> +      cookie=0x3facf689, duration=405.581s, table=0, n_packets=11, n_bytes=846, priority=100,in_port="363ba22029cd92b" actions=load:0x12->NXM_NX_REG13[],load:0x2->NXM_NX_REG11[],load:0x7->NXM_NX_REG12[],load:0x4->OXM_OF_METADATA[],load:0x4->NXM_NX_REG14[],resubmit(,8)
>> +      cookie=0xe7c8c4bb, duration=405.570s, table=0, n_packets=11, n_bytes=846, priority=100,in_port="6a62cde0d50ef44" actions=load:0x13->NXM_NX_REG13[],load:0x2->NXM_NX_REG11[],load:0x7->NXM_NX_REG12[],load:0x4->OXM_OF_METADATA[],load:0x5->NXM_NX_REG14[],resubmit(,8)
>> +      cookie=0x99a0ffc1, duration=59.391s, table=0, n_packets=8, n_bytes=636, priority=100,in_port="5ff3bfaaa4eb622" actions=load:0x14->NXM_NX_REG13[],load:0x2->NXM_NX_REG11[],load:0x7->NXM_NX_REG12[],load:0x4->OXM_OF_METADATA[],load:0x6->NXM_NX_REG14[],resubmit(,8)
>> +      cookie=0xe1b5c263, duration=59.365s, table=0, n_packets=8, n_bytes=636, priority=100,in_port="8d9e0bc76347e59" actions=load:0x15->NXM_NX_REG13[],load:0x2->NXM_NX_REG11[],load:0x7->NXM_NX_REG12[],load:0x4->OXM_OF_METADATA[],load:0x7->NXM_NX_REG14[],resubmit(,8)
>> +
>> +
>> +However, we can better understand what table 0 does by looking at its
>> +logical representation.
>> +::
>> +
>> +   $ ovs-flowviz -i flows.txt -f "table=0" openflow logic
>> +    Ofproto Flows (logical)
>> +    └── ** TABLE 0 **
>> +        ├── priority=180 priority,vlan_tci  --->  conjunction ( x 1 )
>> +        ├── priority=180 priority,conj_id,in_port,vlan_tci  --->  load,load,load,load,load,mod_dl_src resubmit(,8), ( x 1 )
>> +        ├── priority=100 priority,in_port  --->  move,move,move resubmit(,40), ( x 2 )
>> +        ├── priority=100 priority,in_port  --->  load,load,load,load,load resubmit(,8), ( x 16 )
>> +        ├── priority=100 priority,in_port,vlan_tci  --->  load,load,load,load,load resubmit(,8), ( x 1 )
>> +        ├── priority=100 priority,in_port,dl_vlan  --->  strip_vlan,load,load,load,load,load resubmit(,8), ( x 1 )
>> +        └── priority=0 priority  --->   drop, ( x 1 )
>> +
>> +
>> +In only a few logical blocks, we have a good overview of what this table is
>> +doing. It looks like it's adding metadata based on input ports and vlan
>> +IDs and mainly sending traffic to table 8.
>> +
>> +Let's look at table 8, an in this case, let's filter out the flows that have
>> +not been hit by actual traffic. This is quite easy to do with the arithmetic
>> +filtering expressions:
>> +::
>> +
>> +   $ ovs-flowviz -i flows.txt -f "table=8 and n_packets>0" openflow logic
>> +
>> +    Ofproto Flows (logical)
>> +    └── ** TABLE 8 **
>> +        ├── priority=50 priority,reg14,metadata,dl_dst  --->  load resubmit(,9), ( x 3 )
>> +        └── priority=50 priority,metadata  --->  load,move resubmit(,73),resubmit(,9), ( x 2 )
>> +
>> +At this point, we might find ourselves a bit lost since we may not remember
>> +what metadata OVN stored in the previous table. Here is where
>> +``ovs-flowviz``'s OVN integration could come useful. Let's connect to the
>> +running OVN instance and ask it about the flows we're looking at.
>> +
>> +::
>> +
>> +    $ export OVN_NB_DB=tcp:172.18.0.4:6641
>> +    $ export OVN_SB_DB=tcp:172.18.0.4:6642
>> +    $ ovs-flowviz -i flows.txt -f "table=8 and n_packets>0" openflow logic --ovn-detrace
>> +    Ofproto Flows (logical)
>> +    └── ** TABLE 8 **
>> +        ├── cookie=0xe10c34ee priority=50 priority,reg14,metadata,dl_dst  --->  load resubmit(,9), ( x 1 )
>> +        │   └── OVN Info
>> +        │       ├── *  Logical datapaths:
>> +        │       ├── *      "ovn_cluster_router" (366e1c41-0f3d-4420-b796-10692b64e3e4)
>> +        │       ├── *  Logical flow: table=0 (lr_in_admission), priority=50, match=(eth.mcast && inport == "rtos-ovn-worker2), actions=(xreg0[0..47] = 0a:58:0a:f4:01:01; next;)
>> +        │       └── *  Logical Router Port: rtos-ovn-worker2 mac 0a:58:0a:f4:01:01 networks ['10.244.1.1/24'] ipv6_ra_configs {}
>> +        ├── cookie=0x11e1adbc priority=50 priority,reg14,metadata,dl_dst  --->  load resubmit(,9), ( x 1 )
>> +        │   └── OVN Info
>> +        │       ├── *  Logical datapaths:
>> +        │       ├── *      "GR_ovn-worker2" (c07f8387-6479-4e81-9304-9f8e54f81c56)
>> +        │       ├── *  Logical flow: table=0 (lr_in_admission), priority=50, match=(eth.mcast && inport == "rtoe-GR_ovn-worker2), actions=(xreg0[0..47] = 02:42:ac:12:00:03; next;)
>> +        │       └── *  Logical Router Port: rtoe-GR_ovn-worker2 mac 02:42:ac:12:00:03 networks ['172.18.0.3/16'] ipv6_ra_configs {}
>> +        ├── cookie=0xf42133f  priority=50 priority,reg14,metadata,dl_dst  --->  load resubmit(,9), ( x 1 )
>> +        │   └── OVN Info
>> +        │       ├── *  Logical datapaths:
>> +        │       ├── *      "GR_ovn-worker2" (c07f8387-6479-4e81-9304-9f8e54f81c56)
>> +        │       ├── *  Logical flow: table=0 (lr_in_admission), priority=50, match=(eth.dst == 02:42:ac:12:00:03 && inport == "rtoe-GR_ovn-worker2), actions=(xreg0[0..47] = 02:42:ac:12:00:03; next;)
>> +        │       └── *  Logical Router Port: rtoe-GR_ovn-worker2 mac 02:42:ac:12:00:03 networks ['172.18.0.3/16'] ipv6_ra_configs {}
>> +        └── cookie=0x43a0327  priority=50 priority,metadata  --->  load,move resubmit(,73),resubmit(,9), ( x 2 )
>> +            └── OVN Info
>> +                ├── *  Logical datapaths:
>> +                ├── *      "ovn-worker" (24280d0b-fee0-4f8e-ba4f-036a9b9af921)
>> +                ├── *      "ovn-control-plane" (3262a782-8961-416b-805e-08233e8fda72)
>> +                ├── *      "ext_ovn-worker2" (3f88dcd2-c56d-478f-a3b1-c7aee2efe967)
>> +                ├── *      "ext_ovn-worker" (5facbaf0-485d-4cf5-8940-eff9678ef7bb)
>> +                ├── *      "ext_ovn-control-plane" (8b0aecb6-b05a-48a7-ad09-72524bb91d40)
>> +                ├── *      "join" (e2dc230e-2f2a-4b93-93fa-0fe495163514)
>> +                ├── *      "ovn-worker2" (f7709fbf-d728-4cff-9b9b-150461cc75d2)
>> +                └── *  Logical flow: table=0 (ls_in_check_port_sec), priority=50, match=(1), actions=(reg0[15] = check_in_port_sec(); next;)
>> +
>> +That's way better. ``ovs-flowviz`` has automatically added the `cookie` to the
>> +logical block key so have more blocks but in exchange, it has looked up each
>> +cookie on the running OVN databases and inserted the known information on each
>> +block. So now we see what OVN is trying to do, the logical flow that generated
>> +each OpenFlow flow and the logical datapath each flow belongs to.
>> +
>> +Visualizing datapath flow trees
>> +-------------------------------
>> +
>> +Now, let's see another typical usecase that can lead to eyestrain:
>> +understanding datapath conntrack recirculations.
>> +
>> +OVS makes heavy use of connection tracking and the ``recirc()`` action
>> +to build complex datapaths. Typically, OVS will insert a flow that,
>> +when matched, will send the packet through conntrack (using the ``ct`` action)
>> +and recirculate it with a particular recirculation id (``recirc_id``). Then, a
>> +flow matching on that ``recirc_id`` will be matched and further process the
>> +packet. This can happen more than once for a given packet.
>> +
>> +This sequential set of events is, however, difficult to visualize when you
>> +look at a datapath flow dump. Flows are unordered recirculations that need to
>> +be followed manually (typically, with heavy use of "grep").
>> +
>> +For this use-case, ``ovs-flowviz datapath tree`` format can be extremely
>> +useful. It builds a hierarchical tree based on the ``recirc_id`` matches and
>> +``recirc()`` actions and indents flows based on it.
>> +
>> +Here is an example.
>> +::
>> +
>> +    ── recirc_id(0),in_port(3),eth(...),ipv4(...),tcp(dst=8181), actions:ct(zone=2,nat),recirc(0x19348)
>> +    │   ├── recirc_id(0x19348),in_port(3),ct_state(-new+est-rel-rpl-inv+trk),ct_label(0/0x3),eth(...),eth_type,ipv4(), actions:ct(zone=27,nat),recirc(0x10)
>> +    │   │   ├── recirc_id(0x10),in_port(3),ct_state(-new+est-rel-rpl-inv+trk),eth(...),ipv4(...), actions:9
>> +    │   │   ├── recirc_id(0x10),in_port(3),ct_state(-new+est-rel+rpl-inv+trk),eth(...),ipv4(...), actions:9
>> +    │   │   └── recirc_id(0x10),in_port(3),ct_state(+new-est-rel-rpl-inv+trk),eth(...),ipv4(...), actions:ct(commit,zone=27,label=0/0x1),9
>> +    │   └── recirc_id(0x19348),in_port(3),ct_state(+new-est-rel-rpl-inv+trk),eth(...),ipv4(...),  actions:ct(commit,zone=2,label=0/0x1),ct(zone=27,nat),recirc(0x10)
>> +    │       ��── recirc_id(0x10),in_port(3),ct_state(-new+est-rel-rpl-inv+trk),eth(...),ipv4(...), actions:9
>> +    │       ├── recirc_id(0x10),in_port(3),ct_state(-new+est-rel+rpl-inv+trk),eth(...),ipv4(...), actions:9
>> +    │       └── recirc_id(0x10),in_port(3),ct_state(+new-est-rel-rpl-inv+trk),eth(...),ipv4(...), actions:ct(commit,zone=27,label=0/0x1),9
>> +
>> +The above shows a typical conntrack recirculation flow.
>> +The first flow (with ``recir_id(0)``) sends the packet through conntrack
>> +system and recirculates with ``recirc_id(0x19348)``.
>> +Then, based on the ``ct_state`` the packet processing branches out into two
>> +flows. Each flow resends the packet through conntrack and recirculates the
>> +packet one more time. Finally, the packet is processed by 3 flows
>> +on ``recirc_id(10)``.
>> +
>> +This 3-stage processing is now very clear.
>> +
>> +Note that this format can yield longer outputs since some flows (in this
>> +example those with ``recirc_id(10)`` can be repeated. However, the result
>> +is a clear representation of an otherwise difficult to see conntrack
>> +interaction.
>> +
>> +This example shows only a single "subtree". If we use this command to display
>> +a big flow dump, the output can be lengthy. Here are two (combinable) ways to
>> +help out.
>> +
>> +Plotting datapath trees
>> +~~~~~~~~~~~~~~~~~~~~~~~
>> +
>> +By using the ``ovs-flowviz datapath html`` format, long datapath trees can
>> +be displayed in an interactive HTML table. The resulting web allows you to
>> +collapse and expand subtrees so you can focus on what you're looking for.
>> +
>> +In addition, the ``ovs-flowviz datapath graph`` format generates a graphviz
>> +graph definition where each block of flows with the same ``recirc_id`` match
>> +are arranged together and edges are created to represent recirculations.
>> +Also, this format comes with further goodies such as displaying the conntrack
>> +zones which are key to understanding what the datapath is really doing with a
>> +packet.
>> +
>> +These two formats (``html`` and ``graph``) can even be combined. By using the
>> +``ovs-flowviz datapath graph --html`` command, you'll get an interactive
>> +HTML table alongside a `svg` graphical representation of the flows. Click on
>> +a flow on the svg and it'll take you to the corresponding entry in the
>> +flow table.
>> +
>> +
>> +Filtering
>> +~~~~~~~~~
>> +
>> +Apart from being able to expand and collapse subtrees, we can use filtering.
>> +
>> +However, filtering works in a slightly different way compared with OpenFlow
>> +flows. Instead of just removing non-matching flows, the output
>> +of a filtered datapath flow tree will show full sub-trees that contain at
>> +least one flow that satisfies the filter.
>> +
>> +For example, let's take the flows in the above example, and let's imagine we
>> +want to understand what traffic is going out on port ``9``. We could run
>> +the tool as:
>> +::
>> +
>> +   $ ovs-appctl dpctl/dump-flows | ovs-flowviz -f "output.port=9" datapath tree
>> +
>> +The resulting flow tree will contain all of the flows above, even those
>> +with ``recirc_id(0)`` and ``recirc_id(19348)`` that don't actually output
>> +traffic to port ``9``. Why? because they are all part of a subtree that
>> +contains flows that do output packets on port ``9``
>> +
>> +That way, we see the "full picture" of how traffic on port ``9`` is being
>> +processed.
>> +
>> +.. _ovs-flowviz(8): https://docs.openvswitch.org/en/latest/ref/ovs-flowviz.8
>> diff --git a/Documentation/topics/index.rst b/Documentation/topics/index.rst
>> index f239fcf83..9ddb145dd 100644
>> --- a/Documentation/topics/index.rst
>> +++ b/Documentation/topics/index.rst
>> @@ -58,3 +58,4 @@ OVS
>>      userspace-checksum-offloading
>>      userspace-tx-steering
>>      usdt-probes
>> +   flow-visualization
>> -- 
>> 2.44.0
>>
>> _______________________________________________
>> dev mailing list
>> dev@openvswitch.org
>> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>