From patchwork Tue May 7 14:30:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Adri=C3=A1n_Moreno?= X-Patchwork-Id: 1932502 X-Patchwork-Delegate: echaudro@redhat.com Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=XR8btFBr; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=patchwork.ozlabs.org) Received: from smtp1.osuosl.org (smtp1.osuosl.org [IPv6:2605:bc80:3010::138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4VYggr4QBgz1xnT for ; Wed, 8 May 2024 00:31:20 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id BE4CB821CE; Tue, 7 May 2024 14:31:13 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id bj5yVpax0KKM; Tue, 7 May 2024 14:31:09 +0000 (UTC) X-Comment: SPF check N/A for local connections - client-ip=140.211.9.56; helo=lists.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver= DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org AADAA8217F Authentication-Results: smtp1.osuosl.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=XR8btFBr Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp1.osuosl.org (Postfix) with ESMTPS id AADAA8217F; Tue, 7 May 2024 14:31:05 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 7EE01C0DDA; Tue, 7 May 2024 14:31:05 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id E6544C0037 for ; Tue, 7 May 2024 14:31:03 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id C3427409A6 for ; Tue, 7 May 2024 14:31:03 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id yoT3UM_tBYLj for ; Tue, 7 May 2024 14:31:02 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=170.10.133.124; helo=us-smtp-delivery-124.mimecast.com; envelope-from=amorenoz@redhat.com; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp2.osuosl.org F07F64045D Authentication-Results: smtp2.osuosl.org; dmarc=pass (p=none dis=none) header.from=redhat.com DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org F07F64045D Authentication-Results: smtp2.osuosl.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=XR8btFBr Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id F07F64045D for ; Tue, 7 May 2024 14:31:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1715092260; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sfkJR2/4YLDculi6AT+6joL2n+hBMGqhodK2zykrM3I=; b=XR8btFBrCCzUymW5J+8QajZ0zxv48ox/8d6/Vt/Yr4smfGu8i15eP8dadpALLUb0PAk2TM Q1lG4tsxcDX9KclBVBUU9EQIGt7WWQSeDxPubdBY6vRKlci5HoC9ZulPZ9qb3sj3wXH5Ry 4PofG2JzkusZTircJydRAOHMv2Nze3U= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-390-wxyt_4GrNxqe8R5wyBnVNw-1; Tue, 07 May 2024 10:30:59 -0400 X-MC-Unique: wxyt_4GrNxqe8R5wyBnVNw-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id F0CE585A58C for ; Tue, 7 May 2024 14:30:58 +0000 (UTC) Received: from antares.redhat.com (unknown [10.39.192.88]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1F1853C25; Tue, 7 May 2024 14:30:57 +0000 (UTC) From: Adrian Moreno To: dev@openvswitch.org Date: Tue, 7 May 2024 16:30:28 +0200 Message-ID: <20240507143044.3056232-3-amorenoz@redhat.com> In-Reply-To: <20240507143044.3056232-1-amorenoz@redhat.com> References: <20240507143044.3056232-1-amorenoz@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.1 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH v4 02/12] python: ovs: flowviz: Add file processing infra. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" process.py contains a useful base class that processes files odp.py and ofp.py: contain datapath and openflow subcommand definitions as well as the first formatting option: json. Also, this patch adds basic filtering support. Examples: $ ovs-ofctl dump-flows br-int | ovs-flowviz openflow json $ ovs-ofctl dump-flows br-int > flows.txt && ovs-flowviz -i flows.txt openflow json $ ovs-ofctl appctl dpctl/dump-flows | ovs-flowviz -f 'ct' datapath json $ ovs-ofctl appctl dpctl/dump-flows > flows.txt && ovs-flowviz -i flows.txt -f 'drop' datapath json Signed-off-by: Adrian Moreno Acked-by: Eelco Chaudron --- python/automake.mk | 5 +- python/ovs/flowviz/__init__.py | 2 + python/ovs/flowviz/main.py | 102 +++++++++++++++++- python/ovs/flowviz/odp/cli.py | 42 ++++++++ python/ovs/flowviz/ofp/cli.py | 42 ++++++++ python/ovs/flowviz/process.py | 192 +++++++++++++++++++++++++++++++++ 6 files changed, 383 insertions(+), 2 deletions(-) create mode 100644 python/ovs/flowviz/odp/cli.py create mode 100644 python/ovs/flowviz/ofp/cli.py create mode 100644 python/ovs/flowviz/process.py diff --git a/python/automake.mk b/python/automake.mk index 124032c92..fd5e74081 100644 --- a/python/automake.mk +++ b/python/automake.mk @@ -67,8 +67,11 @@ ovs_flowviz = \ python/ovs/flowviz/__init__.py \ python/ovs/flowviz/main.py \ python/ovs/flowviz/odp/__init__.py \ + python/ovs/flowviz/odp/cli.py \ python/ovs/flowviz/ofp/__init__.py \ - python/ovs/flowviz/ovs-flowviz + python/ovs/flowviz/ofp/cli.py \ + python/ovs/flowviz/ovs-flowviz \ + python/ovs/flowviz/process.py # These python files are used at build time but not runtime, diff --git a/python/ovs/flowviz/__init__.py b/python/ovs/flowviz/__init__.py index e69de29bb..898dba522 100644 --- a/python/ovs/flowviz/__init__.py +++ b/python/ovs/flowviz/__init__.py @@ -0,0 +1,2 @@ +import ovs.flowviz.ofp.cli # noqa: F401 +import ovs.flowviz.odp.cli # noqa: F401 diff --git a/python/ovs/flowviz/main.py b/python/ovs/flowviz/main.py index f5bf142be..64b0e8a0a 100644 --- a/python/ovs/flowviz/main.py +++ b/python/ovs/flowviz/main.py @@ -13,17 +13,64 @@ # limitations under the License. import click +import os + +from ovs.flow.filter import OFFilter class Options(dict): """Options dictionary""" +def validate_input(ctx, param, value): + """Validate the "-i" option""" + result = list() + for input_str in value: + parts = input_str.strip().split(",") + if len(parts) == 2: + file_parts = tuple(parts) + elif len(parts) == 1: + file_parts = tuple(["Filename: " + parts[0], parts[0]]) + else: + raise click.BadParameter( + "input filename should have the following format: " + "[alias,]FILENAME" + ) + + if not os.path.isfile(file_parts[1]): + raise click.BadParameter( + "input filename %s does not exist" % file_parts[1] + ) + result.append(file_parts) + return result + + @click.group( context_settings=dict(help_option_names=["-h", "--help"]), ) +@click.option( + "-i", + "--input", + "filename", + help="Read flows from specified filepath. If not provided, flows will be" + " read from stdin. This option can be specified multiple times." + " Format [alias,]FILENAME. Where alias is a name that shall be used to" + " refer to this FILENAME", + multiple=True, + type=click.Path(), + callback=validate_input, +) +@click.option( + "-f", + "--filter", + help="Filter flows that match the filter expression." + "Run 'ovs-flowviz filter' for a detailed description of the filtering " + "syntax", + type=str, + show_default=False, +) @click.pass_context -def maincli(ctx): +def maincli(ctx, filename, filter): """ OpenvSwitch flow visualization utility. @@ -31,6 +78,59 @@ def maincli(ctx): (such as the output of ovs-ofctl dump-flows or ovs-appctl dpctl/dump-flows) and prints them in different formats. """ + ctx.obj = Options() + ctx.obj["filename"] = filename or None + if filter: + try: + ctx.obj["filter"] = OFFilter(filter) + except Exception as e: + raise click.BadParameter("Wrong filter syntax: {}".format(e)) + + +@maincli.command(hidden=True) +@click.pass_context +def filter(ctx): + """ + \b + Filter Syntax + ************* + + [! | not ] {key}[[.subkey]...] [OPERATOR] {value})] [LOGICAL OPERATOR] ... + + \b + Comparison operators are: + = equality + < less than + > more than + ~= masking (valid for IP and Ethernet fields) + + \b + Logical operators are: + !{expr}: NOT + {expr} && {expr}: AND + {expr} || {expr}: OR + + \b + Matches and flow metadata: + To compare against a match or info field, use the field directly, e.g: + priority=100 + n_bytes>10 + Use simple keywords for flags: + tcp and ip_src=192.168.1.1 + \b + Actions: + Actions values might be dictionaries, use subkeys to access individual + values, e.g: + output.port=3 + Use simple keywords for flags + drop + + \b + Examples of valid filters. + nw_addr~=192.168.1.1 && (tcp_dst=80 || tcp_dst=443) + arp=true && !arp_tsa=192.168.1.1 + n_bytes>0 && drop=true""" + click.echo(ctx.command.get_help(ctx)) def main(): diff --git a/python/ovs/flowviz/odp/cli.py b/python/ovs/flowviz/odp/cli.py new file mode 100644 index 000000000..ed2f82065 --- /dev/null +++ b/python/ovs/flowviz/odp/cli.py @@ -0,0 +1,42 @@ +# Copyright (c) 2023 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at: +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import click + +from ovs.flowviz.main import maincli +from ovs.flowviz.process import ( + DatapathFactory, + JSONProcessor, +) + + +@maincli.group(subcommand_metavar="FORMAT") +@click.pass_obj +def datapath(opts): + """Process Datapath Flows.""" + pass + + +class JSONPrint(DatapathFactory, JSONProcessor): + def __init__(self, opts): + super().__init__(opts) + + +@datapath.command() +@click.pass_obj +def json(opts): + """Print the flows in JSON format.""" + proc = JSONPrint(opts) + proc.process() + print(proc.json_string()) diff --git a/python/ovs/flowviz/ofp/cli.py b/python/ovs/flowviz/ofp/cli.py new file mode 100644 index 000000000..b9a2a8aad --- /dev/null +++ b/python/ovs/flowviz/ofp/cli.py @@ -0,0 +1,42 @@ +# Copyright (c) 2023 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at: +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import click + +from ovs.flowviz.main import maincli +from ovs.flowviz.process import ( + OpenFlowFactory, + JSONProcessor, +) + + +@maincli.group(subcommand_metavar="FORMAT") +@click.pass_obj +def openflow(opts): + """Process OpenFlow Flows.""" + pass + + +class JSONPrint(OpenFlowFactory, JSONProcessor): + def __init__(self, opts): + super().__init__(opts) + + +@openflow.command() +@click.pass_obj +def json(opts): + """Print the flows in JSON format.""" + proc = JSONPrint(opts) + proc.process() + print(proc.json_string()) diff --git a/python/ovs/flowviz/process.py b/python/ovs/flowviz/process.py new file mode 100644 index 000000000..3e520e431 --- /dev/null +++ b/python/ovs/flowviz/process.py @@ -0,0 +1,192 @@ +# Copyright (c) 2023 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at: +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys +import json +import click + +from ovs.flow.decoders import FlowEncoder +from ovs.flow.odp import ODPFlow +from ovs.flow.ofp import OFPFlow + + +class FileProcessor(object): + """Base class for file-based Flow processing. It is able to create flows + from strings found in a file (or stdin). + + The process of parsing the flows is extendable in many ways by deriving + this class. + + When process() is called, the base class will: + - call self.start_file() for each new file that get's processed + - call self.create_flow() for each flow line + - apply the filter defined in opts if provided (can be optionally + disabled) + - call self.process_flow() for after the flow has been filtered + - call self.stop_file() after the file has been processed entirely + + In the case of stdin, the filename and file alias is 'stdin'. + + Child classes must at least implement create_flow() and process_flow() + functions. + + Args: + opts (dict): Options dictionary + """ + + def __init__(self, opts): + self.opts = opts + + # Methods that must be implemented by derived classes. + def init(self): + """Called before the flow processing begins.""" + pass + + def start_file(self, alias, filename): + """Called before the processing of a file begins. + Args: + alias(str): The alias name of the filename + filename(str): The filename string + """ + pass + + def create_flow(self, line, idx): + """Called for each line in the file. + Args: + line(str): The flow line + idx(int): The line index + + Returns a Flow. + Must be implemented by child classes. + """ + raise NotImplementedError + + def process_flow(self, flow, name): + """Called for built flow (after filtering). + Args: + flow(Flow): The flow created by create_flow + name(str): The name of the file from which the flow comes + """ + raise NotImplementedError + + def stop_file(self, alias, filename): + """Called after the processing of a file ends. + Args: + alias(str): The alias name of the filename + filename(str): The filename string + """ + pass + + def end(self): + """Called after the processing ends.""" + pass + + def process(self, do_filter=True): + idx = 0 + filenames = self.opts.get("filename") + filt = self.opts.get("filter") if do_filter else None + self.init() + if filenames: + for alias, filename in filenames: + try: + with open(filename) as f: + self.start_file(alias, filename) + for line in f: + flow = self.create_flow(line, idx) + idx += 1 + if not flow or (filt and not filt.evaluate(flow)): + continue + self.process_flow(flow, alias) + self.stop_file(alias, filename) + except IOError as e: + raise click.BadParameter( + "Failed to read from file {} ({}): {}".format( + filename, e.errno, e.strerror + ) + ) + else: + data = sys.stdin.read() + self.start_file("stdin", "stdin") + for line in data.split("\n"): + line = line.strip() + if line: + flow = self.create_flow(line, idx) + idx += 1 + if ( + not flow + or not getattr(flow, "_sections", None) + or (filt and not filt.evaluate(flow)) + ): + continue + self.process_flow(flow, "stdin") + self.stop_file("stdin", "stdin") + self.end() + + +class DatapathFactory(): + """A mixin class that creates Datapath flows.""" + + def create_flow(self, line, idx): + # Skip strings commonly found in Datapath flow dumps. + if any(s in line for s in [ + "flow-dump from the main thread", + "flow-dump from pmd on core", + ]): + return None + + return ODPFlow(line, idx) + + +class OpenFlowFactory(): + """A mixin class that creates OpenFlow flows.""" + + def create_flow(self, line, idx): + # Skip strings commonly found in OpenFlow flow dumps. + if " reply " in line: + return None + + return OFPFlow(line, idx) + + +class JSONProcessor(FileProcessor): + """A FileProcessor that prints flows in JSON format.""" + + def __init__(self, opts): + super().__init__(opts) + self.flows = dict() + + def start_file(self, name, filename): + self.flows_list = list() + + def stop_file(self, name, filename): + self.flows[name] = self.flows_list + + def process_flow(self, flow, name): + self.flows_list.append(flow) + + def json_string(self): + if len(self.flows.keys()) > 1: + return json.dumps( + [ + {"name": name, "flows": [flow.dict() for flow in flows]} + for name, flows in self.flows.items() + ], + indent=4, + cls=FlowEncoder, + ) + return json.dumps( + [flow.dict() for flow in self.flows_list], + indent=4, + cls=FlowEncoder, + )