From patchwork Fri Apr 1 22:40:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Toso X-Patchwork-Id: 1612433 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=PBXLxbA7; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KVZt43xDLz9sG4 for ; Sat, 2 Apr 2022 09:42:02 +1100 (AEDT) Received: from localhost ([::1]:45214 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1naPxu-0001wm-Ey for incoming@patchwork.ozlabs.org; Fri, 01 Apr 2022 18:41:58 -0400 Received: from eggs.gnu.org ([209.51.188.92]:39146) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1naPxD-0001t3-L6 for qemu-devel@nongnu.org; Fri, 01 Apr 2022 18:41:15 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:44008) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1naPxA-0005PF-9l for qemu-devel@nongnu.org; Fri, 01 Apr 2022 18:41:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648852870; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=I6xH/5BPizyYOOs1YymptuSXhHP4izaoxIpDOPlhO/g=; b=PBXLxbA7pZPxjSbi2nCnkFnFFUGvWh77yJ1RuJW6gh4BSBOzjG5WY6EKCDmHq9Nm72wWt4 3WMOi6xkT2Bf8n1hnwsm5Ee3so7Dl5ZkzYxkeFA4pSwWQOv4kdkgxlIXcS+ETGcpGEYOIW Wllh/4sUQOaFH0B/2GGWtPy7mFR2sQ8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-376-rB3AQGIhNnCNt_a197Kffg-1; Fri, 01 Apr 2022 18:41:09 -0400 X-MC-Unique: rB3AQGIhNnCNt_a197Kffg-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2D99D185A7BA for ; Fri, 1 Apr 2022 22:41:09 +0000 (UTC) Received: from tapioca.redhat.com (unknown [10.40.193.147]) by smtp.corp.redhat.com (Postfix) with ESMTP id B3B579D70; Fri, 1 Apr 2022 22:41:07 +0000 (UTC) From: Victor Toso To: qemu-devel@nongnu.org Subject: [RFC PATCH v1 1/8] qapi: golang: Generate qapi's enum types in Go Date: Sat, 2 Apr 2022 00:40:57 +0200 Message-Id: <20220401224104.145961-2-victortoso@redhat.com> In-Reply-To: <20220401224104.145961-1-victortoso@redhat.com> References: <20220401224104.145961-1-victortoso@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=victortoso@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: John Snow , Eric Blake , Markus Armbruster Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This patch handles QAPI enum types and generates its equivalent in Go. The highlights of this implementation are: 1. For each QAPI enum, we will define an int32 type in Go to be the assigned type of this specific enum 2. While in the Go codebase we can use the generated enum values, the specification requires that, on the wire, the enumeration type's value to be represented by its string name. For this reason, each Go type that represent's a QAPI enum will be implementing the Marshaler[0] and Unmarshaler[1] interfaces to seamless handle QMP's string to Go int32 value and vice-versa. 3. Naming: CamelCase will be used in any identifier that we want to export [2], which is everything in this patch. [0] https://pkg.go.dev/encoding/json#Marshaler [1] https://pkg.go.dev/encoding/json#Unmarshaler [2] https://go.dev/ref/spec#Exported_identifiers Signed-off-by: Victor Toso --- qapi/meson.build | 1 + scripts/qapi/golang.py | 225 +++++++++++++++++++++++++++++++++++++++++ scripts/qapi/main.py | 2 + 3 files changed, 228 insertions(+) create mode 100644 scripts/qapi/golang.py diff --git a/qapi/meson.build b/qapi/meson.build index 656ef0e039..0951692332 100644 --- a/qapi/meson.build +++ b/qapi/meson.build @@ -90,6 +90,7 @@ qapi_nonmodule_outputs = [ 'qapi-init-commands.h', 'qapi-init-commands.c', 'qapi-events.h', 'qapi-events.c', 'qapi-emit-events.c', 'qapi-emit-events.h', + 'qapibara.go', ] # First build all sources diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py new file mode 100644 index 0000000000..070d4cbbae --- /dev/null +++ b/scripts/qapi/golang.py @@ -0,0 +1,225 @@ +""" +Golang QAPI generator +""" +# Copyright (c) 2021 Red Hat Inc. +# +# Authors: +# Victor Toso +# +# This work is licensed under the terms of the GNU GPL, version 2. +# See the COPYING file in the top-level directory. + +# Just for type hint on self +from __future__ import annotations + +import os +from typing import Dict, List, Optional + +from .schema import ( + QAPISchema, + QAPISchemaVisitor, + QAPISchemaEnumMember, + QAPISchemaFeature, + QAPISchemaIfCond, + QAPISchemaObjectType, + QAPISchemaObjectTypeMember, + QAPISchemaVariants, +) +from .source import QAPISourceInfo + +class QAPISchemaGenGolangVisitor(QAPISchemaVisitor): + + def __init__(self, prefix: str): + super().__init__() + self.target = {name: "" for name in ["enum"]} + self.schema = None + self._docmap = {} + self.golang_package_name = "qapi" + + def visit_begin(self, schema): + self.schema = schema + + # Every Go file needs to reference its package name + for target in self.target: + self.target[target] = f"package {self.golang_package_name}\n" + + # Iterate once in schema.docs to map doc objects to its name + for doc in schema.docs: + if doc.symbol is None: + continue + self._docmap[doc.symbol] = doc + + def visit_end(self): + self.schema = None + + def visit_object_type(self: QAPISchemaGenGolangVisitor, + name: str, + info: Optional[QAPISourceInfo], + ifcond: QAPISchemaIfCond, + features: List[QAPISchemaFeature], + base: Optional[QAPISchemaObjectType], + members: List[QAPISchemaObjectTypeMember], + variants: Optional[QAPISchemaVariants] + ) -> None: + pass + + def visit_alternate_type(self: QAPISchemaGenGolangVisitor, + name: str, + info: Optional[QAPISourceInfo], + ifcond: QAPISchemaIfCond, + features: List[QAPISchemaFeature], + variants: QAPISchemaVariants + ) -> None: + pass + + def visit_enum_type(self: QAPISchemaGenGolangVisitor, + name: str, + info: Optional[QAPISourceInfo], + ifcond: QAPISchemaIfCond, + features: List[QAPISchemaFeature], + members: List[QAPISchemaEnumMember], + prefix: Optional[str] + ) -> None: + doc = self._docmap.get(name, None) + doc_struct, doc_fields = qapi_to_golang_struct_docs(doc) + + value = qapi_to_field_name_enum(members[0].name) + fields = f"\t{name}{value} {name} = iota\n" + for member in members[1:]: + field_doc = " " + doc_fields.get(member.name, "") if doc_fields else "" + value = qapi_to_field_name_enum(member.name) + fields += f"\t{name}{value}{field_doc}\n" + + self.target["enum"] += f''' +{doc_struct} +type {name} int32 +const ( +{fields[:-1]} +) +{generate_marshal_methods_enum(members)} +''' + + def visit_array_type(self, name, info, ifcond, element_type): + pass + + def visit_command(self, + name: str, + info: Optional[QAPISourceInfo], + ifcond: QAPISchemaIfCond, + features: List[QAPISchemaFeature], + arg_type: Optional[QAPISchemaObjectType], + ret_type: Optional[QAPISchemaType], + gen: bool, + success_response: bool, + boxed: bool, + allow_oob: bool, + allow_preconfig: bool, + coroutine: bool) -> None: + pass + + def visit_event(self, name, info, ifcond, features, arg_type, boxed): + pass + + def write(self, output_dir: str) -> None: + for module_name, content in self.target.items(): + go_module = module_name + "s.go" + go_dir = "go" + pathname = os.path.join(output_dir, go_dir, go_module) + odir = os.path.dirname(pathname) + os.makedirs(odir, exist_ok=True) + + with open(pathname, "w") as outfile: + outfile.write(content) + + +def gen_golang(schema: QAPISchema, + output_dir: str, + prefix: str) -> None: + vis = QAPISchemaGenGolangVisitor(prefix) + schema.visit(vis) + vis.write(output_dir) + +def generate_marshal_methods_enum(members: List[QAPISchemaEnumMember]) -> str: + type = qapi_to_go_type_name(members[0].defined_in, "enum") + + marshal_switch_cases = "" + unmarshal_switch_cases = "" + for i in range(len(members)): + go_type = type + qapi_to_field_name_enum(members[i].name) + name = members[i].name + + marshal_switch_cases += f''' + case {go_type}: + return []byte(`"{name}"`), nil''' + + unmarshal_switch_cases += f''' + case "{name}": + (*s) = {go_type}''' + + return f''' +func (s {type}) MarshalJSON() ([]byte, error) {{ + switch s {{ +{marshal_switch_cases[1:]} + default: + fmt.Println("Failed to decode {type}", s) + }} + return nil, errors.New("Failed") +}} + +func (s *{type}) UnmarshalJSON(data []byte) error {{ + var name string + + if err := json.Unmarshal(data, &name); err != nil {{ + return err + }} + + switch name {{ +{unmarshal_switch_cases[1:]} + default: + fmt.Println("Failed to decode {type}", *s) + }} + return nil +}} +''' + +# Takes the documentation object of a specific type and returns +# that type's documentation followed by its member's docs. +def qapi_to_golang_struct_docs(doc: QAPIDoc) -> (str, Dict[str, str]): + if doc is None: + return "// No documentation available", None + + main = "" + if len(doc.body.text) > 0: + main = f"// {doc.body.text}".replace("\n", "\n// ") + + for section in doc.sections: + # Skip sections that are not useful for Golang consumers + if section.name and "TODO" in section.name: + continue + + # Small hack to only add // when doc.body.text was empty + prefix = "// " if len(main) == 0 else "\n\n" + main += f"{prefix}{section.name}: {section.text}".replace("\n", "\n// ") + + fields = {} + for key, value in doc.args.items(): + if len(value.text) > 0: + fields[key] = " // " + ' '.join(value.text.replace("\n", " ").split()) + + return main, fields + +def qapi_to_field_name_enum(name: str) -> str: + return name.title().replace("-", "") + +def qapi_to_go_type_name(name: str, meta: str) -> str: + if name.startswith("q_obj_"): + name = name[6:] + + # We want to keep CamelCase for Golang types. We want to avoid removing + # already set CameCase names while fixing uppercase ones, eg: + # 1) q_obj_SocketAddress_base -> SocketAddressBase + # 2) q_obj_WATCHDOG-arg -> WatchdogArg + words = [word for word in name.replace("_", "-").split("-")] + name = words[0].title() if words[0].islower() or words[0].isupper() else words[0] + name += ''.join(word.title() for word in words[1:]) + return name diff --git a/scripts/qapi/main.py b/scripts/qapi/main.py index fc216a53d3..661fb1e091 100644 --- a/scripts/qapi/main.py +++ b/scripts/qapi/main.py @@ -15,6 +15,7 @@ from .common import must_match from .error import QAPIError from .events import gen_events +from .golang import gen_golang from .introspect import gen_introspect from .schema import QAPISchema from .types import gen_types @@ -54,6 +55,7 @@ def generate(schema_file: str, gen_events(schema, output_dir, prefix) gen_introspect(schema, output_dir, prefix, unmask) + gen_golang(schema, output_dir, prefix) def main() -> int: """ From patchwork Fri Apr 1 22:40:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Victor Toso X-Patchwork-Id: 1612436 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=d2v1Vnis; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KVZwD3lpRz9sG4 for ; Sat, 2 Apr 2022 09:43:56 +1100 (AEDT) Received: from localhost ([::1]:51596 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1naPzm-0006PT-8I for incoming@patchwork.ozlabs.org; Fri, 01 Apr 2022 18:43:54 -0400 Received: from eggs.gnu.org ([209.51.188.92]:39144) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1naPxE-0001sr-8l for qemu-devel@nongnu.org; Fri, 01 Apr 2022 18:41:16 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:33687) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1naPxA-0005PX-EQ for qemu-devel@nongnu.org; Fri, 01 Apr 2022 18:41:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648852871; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EZ/25VGpLmuwPak1J7FfgNG28lCp3PDDIoaiEuZVhsk=; b=d2v1Vnis1gJSTKuC9rJh+lk6BEPQAQcB0BlrHRsxqKi11iwVvjwu+m//PaR8+GI4NytjFq /y42X1G9rJvvftWcb52UG1tZW+DQ5hoynhkRoCRMUCbzNyBQyHLu5ZbKHtEsR9GG+snX+3 ZjX5p1cMCAG3+sjdAs0onFYpKlFnDkQ= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-315-GbB08wyQOsqgOZqzZ1E9Yw-1; Fri, 01 Apr 2022 18:41:11 -0400 X-MC-Unique: GbB08wyQOsqgOZqzZ1E9Yw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AFFFC2A2AD40 for ; Fri, 1 Apr 2022 22:41:10 +0000 (UTC) Received: from tapioca.redhat.com (unknown [10.40.193.147]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8CEF99E6E; Fri, 1 Apr 2022 22:41:09 +0000 (UTC) From: Victor Toso To: qemu-devel@nongnu.org Subject: [RFC PATCH v1 2/8] qapi: golang: Generate qapi's alternate types in Go Date: Sat, 2 Apr 2022 00:40:58 +0200 Message-Id: <20220401224104.145961-3-victortoso@redhat.com> In-Reply-To: <20220401224104.145961-1-victortoso@redhat.com> References: <20220401224104.145961-1-victortoso@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=victortoso@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.129.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: John Snow , Eric Blake , Markus Armbruster Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This patch handles QAPI alternate types and generates data structures in Go that handles it. At this moment, there are 5 alternates in qemu/qapi, they are: * BlockDirtyBitmapMergeSource * Qcow2OverlapChecks * BlockdevRef * BlockdevRefOrNull * StrOrNull Alternate types are similar to Union but without a discriminator that can be used to identify the underlying value on the wire. It is needed to infer it. That can't be easily mapped in Go. For each Alternate type, we will be using a Any type to hold the value. 'Any' is an alias type for interface{} (similar to void* in C). Similarly to the Enum types (see previous commit), we will implement Marshaler and Unmarshaler interfaces for the Alternate types and in those MarshalJSON() and UnmarshalJSON() methods is where we are going to put the logic to read/set alternate's value. Note that on UnmarshalJSON(), a helper function called StrictDecode() will be used. This function is the main logic to infer if a given JSON object fits in a given Go struct. Because we only have 5 alternate types, it is not hard to validate the unmarshaling logic but we might need to improve it in the future if Alternate with branches that have similar fields appear. Examples: * BlockdevRef ```go // Data to set in BlockdevOptions qcow2 := BlockdevOptionsQcow2{} // BlockdevRef using a string qcow2.File = BlockdevRef{Value: "/some/place/my-image"} opt := BlockdevOptions{} opt.Driver = BlockdevDriverQcow2 opt.Value = qcow2 b, _ := json.Marshal(data.s) // string(b) == `{"driver":"qcow2","file":"/some/place/my-image"}` ``` Signed-off-by: Victor Toso --- scripts/qapi/golang.py | 157 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 155 insertions(+), 2 deletions(-) diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py index 070d4cbbae..8be31bd902 100644 --- a/scripts/qapi/golang.py +++ b/scripts/qapi/golang.py @@ -31,7 +31,8 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor): def __init__(self, prefix: str): super().__init__() - self.target = {name: "" for name in ["enum"]} + self.target = {name: "" for name in ["alternate", "enum", "helper"]} + self.objects_seen = {} self.schema = None self._docmap = {} self.golang_package_name = "qapi" @@ -43,6 +44,10 @@ def visit_begin(self, schema): for target in self.target: self.target[target] = f"package {self.golang_package_name}\n" + self.target["helper"] += f''' + // Alias for go version lower than 1.18 + type Any = interface{{}}''' + # Iterate once in schema.docs to map doc objects to its name for doc in schema.docs: if doc.symbol is None: @@ -52,6 +57,22 @@ def visit_begin(self, schema): def visit_end(self): self.schema = None + self.target["helper"] += ''' +// Creates a decoder that errors on unknown Fields +// Returns true if successfully decoded @from string @into type +// Returns false without error is failed with "unknown field" +// Returns false with error is a different error was found +func StrictDecode(into interface{}, from []byte) error { + dec := json.NewDecoder(strings.NewReader(string(from))) + dec.DisallowUnknownFields() + + if err := dec.Decode(into); err != nil { + return err + } + return nil +} +''' + def visit_object_type(self: QAPISchemaGenGolangVisitor, name: str, info: Optional[QAPISourceInfo], @@ -70,7 +91,123 @@ def visit_alternate_type(self: QAPISchemaGenGolangVisitor, features: List[QAPISchemaFeature], variants: QAPISchemaVariants ) -> None: - pass + assert name not in self.objects_seen + self.objects_seen[name] = True + + # Alternate marshal logic + # + # To avoid programming errors by users of this generated Go module, + # we add a runtime check to error out in case the underlying Go type + # doesn't not match any of supported types of the Alternate type. + # + # Also, Golang's json Marshal will include as JSON's object, the + # wrapper we use to hold the Go struct (Value Any -> `Value: {...}`) + # This would not be an valid QMP message so we workaround it by + # calling RemoveValueObject function. + doc = self._docmap.get(name, None) + doc_struct, doc_fields = qapi_to_golang_struct_docs(doc) + + members_doc = '''// Options are:''' + if_supported_types = "" + for var in variants.variants: + field_doc = doc_fields.get(var.name, "") + field_go_type = qapi_schema_type_to_go_type(var.type.name) + members_doc += f'''\n// * {var.name} ({field_go_type}):{field_doc[3:]}''' + + if field_go_type == "nil": + field_go_type = "*string" + + if_supported_types += f'''typestr != "{field_go_type}" &&\n\t\t''' + + # Alternate unmarshal logic + # + # With Alternate types, we have to check the JSON data in order to + # identify what is the target Go type. So, this is different than an + # union which has an identifier that we can check first. + # StrictDecode function tries to match the given JSON data to a given + # Go type and it'll error in case it doesn´t fit, for instance, when + # there were members in the JSON data that had no equivalent in the + # target Go type. + # + # For this reason, the order is important. + # + # At this moment, the only field that must be checked first is JSON + # NULL, which is relevant to a few alternate types. In the future, we + # might need to improve the logic to be foolproof between target Go + # types that might have a common base (non existing Today). + check_type_str = ''' + // Check for {name} + {{ + var value {go_type} + if err := StrictDecode(&value, data); {error_check} {{ + s.Value = {set_value} + return nil + }} + }}''' + reference_checks = "" + for var in variants.variants: + if var.type.name == "null": + # We use a pointer (by referece) to check for JSON's NULL + reference_checks += check_type_str.format( + name = var.type.name, + go_type = "*string", + error_check = "err == nil && value == nil", + set_value = "nil") + break; + + value_checks = "" + for var in variants.variants: + if var.type.name != "null": + go_type = qapi_schema_type_to_go_type(var.type.name) + value_checks += check_type_str.format( + name = var.type.name, + go_type = go_type, + error_check = "err == nil", + set_value = "value") + + unmarshal_checks = "" + if len(reference_checks) > 0 and len(value_checks) > 0: + unmarshal_checks = reference_checks[1:] + value_checks + else: + unmarshal_checks = reference_checks[1:] if len(reference_checks) > 0 else value_checks[1:] + + self.target["alternate"] += f''' +{doc_struct} +type {name} struct {{ +{members_doc} + Value Any +}} + +func (s {name}) MarshalJSON() ([]byte, error) {{ + typestr := fmt.Sprintf("%T", s.Value) + typestr = typestr[strings.LastIndex(typestr, ".")+1:] + + // Runtime check for supported types + if typestr != "" && +{if_supported_types[:-6]} {{ + return nil, errors.New(fmt.Sprintf("Type is not supported: %s", typestr)) + }} + + b, err := json.Marshal(s.Value); + if err != nil {{ + return nil, err + }} + + return b, nil +}} + +func (s *{name}) UnmarshalJSON(data []byte) error {{ +{unmarshal_checks} + // Check type to error out nicely + {{ + var value Any + if err := json.Unmarshal(data, &value); err != nil {{ + return err + }} + return errors.New(fmt.Sprintf("Unsupported type %T (value: %v)", value, value)) + }} +}} +''' def visit_enum_type(self: QAPISchemaGenGolangVisitor, name: str, @@ -208,6 +345,22 @@ def qapi_to_golang_struct_docs(doc: QAPIDoc) -> (str, Dict[str, str]): return main, fields +def qapi_schema_type_to_go_type(type: str) -> str: + schema_types_to_go = {'str': 'string', 'null': 'nil', 'bool': 'bool', + 'number': 'float64', 'size': 'uint64', 'int': 'int64', 'int8': 'int8', + 'int16': 'int16', 'int32': 'int32', 'int64': 'int64', 'uint8': 'uint8', + 'uint16': 'uint16', 'uint32': 'uint32', 'uint64': 'uint64', + 'any': 'Any', 'QType': 'QType', + } + + prefix = "" + if type.endswith("List"): + prefix = "[]" + type = type[:-4] + + type = schema_types_to_go.get(type, type) + return prefix + type + def qapi_to_field_name_enum(name: str) -> str: return name.title().replace("-", "") From patchwork Fri Apr 1 22:40:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Toso X-Patchwork-Id: 1612438 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=ebAHZ7g0; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KVZyf2Gxjz9sG4 for ; Sat, 2 Apr 2022 09:46:02 +1100 (AEDT) Received: from localhost ([::1]:55908 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1naQ1o-0000qm-3a for incoming@patchwork.ozlabs.org; Fri, 01 Apr 2022 18:46:00 -0400 Received: from eggs.gnu.org ([209.51.188.92]:39178) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1naPxG-0001zz-6W for qemu-devel@nongnu.org; Fri, 01 Apr 2022 18:41:18 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:28125) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1naPxE-0005Pz-ER for qemu-devel@nongnu.org; Fri, 01 Apr 2022 18:41:17 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648852875; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xGk99wOZ5GBeeMkIGm0UFq2RzAWQRX/6iGYVGqcKP04=; b=ebAHZ7g0ALeWBY71lCOtJ/SqLPi9Qh9uf+zo9cWURPgFEHO1PNlIeu6Tn7xd9afMoFbCa+ 7sQ/i2FbYjolBWZYkPQmboGy2U/uSWfyYNSAMJrxOOWBFLgRrZMFJieNSHX+zOQZ4790oh ucmB3sxPanQyQeoHQnFKBnh7sPrfQ/4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-393-LPPDVImwPIu6HBh-4IJmIA-1; Fri, 01 Apr 2022 18:41:12 -0400 X-MC-Unique: LPPDVImwPIu6HBh-4IJmIA-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 25A5C811E76 for ; Fri, 1 Apr 2022 22:41:12 +0000 (UTC) Received: from tapioca.redhat.com (unknown [10.40.193.147]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1BD849E6E; Fri, 1 Apr 2022 22:41:10 +0000 (UTC) From: Victor Toso To: qemu-devel@nongnu.org Subject: [RFC PATCH v1 3/8] qapi: golang: Generate qapi's struct types in Go Date: Sat, 2 Apr 2022 00:40:59 +0200 Message-Id: <20220401224104.145961-4-victortoso@redhat.com> In-Reply-To: <20220401224104.145961-1-victortoso@redhat.com> References: <20220401224104.145961-1-victortoso@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=victortoso@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.129.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: John Snow , Eric Blake , Markus Armbruster Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This patch handles QAPI struct types and generates the equivalent types in Go. At the time of this writing, it generates 375 structures. The highlights of this implementation are: 1. Generating an Go struct that requires a @base type, the @base type is embedded in this Go structure. Example: See InetSocketAddress with it's base InetSocketAddressBase 2. Differently from previous two types ('enum' and 'alternate'), the generated QAPI's struct type do not need to implement Marshaler and Unmarshaler interfaces. This generated structures will naturally match with JSON Objects. 3. About the Go struct's fields: i) They can be either by Value or Reference. ii) Every field that is marked as optional in the QAPI specification are translated to Reference fields in its Go structure. This design decision is the most straightforward way to check if a given field was set or not. iii) Mandatory fields are always by Value with the exception of QAPI arrays, which are handled by Reference (to a block of memory) by Go. iv) All the fields are named with Uppercase due Golang's export convention. v) In order to avoid any kind of issues when encoding ordecoding, to or from JSON, we mark all fields with its @name and, when it is optional, member, with @omitempty Signed-off-by: Victor Toso --- scripts/qapi/golang.py | 79 ++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 77 insertions(+), 2 deletions(-) diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py index 8be31bd902..50e39f8925 100644 --- a/scripts/qapi/golang.py +++ b/scripts/qapi/golang.py @@ -31,7 +31,7 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor): def __init__(self, prefix: str): super().__init__() - self.target = {name: "" for name in ["alternate", "enum", "helper"]} + self.target = {name: "" for name in ["alternate", "enum", "helper", "struct"]} self.objects_seen = {} self.schema = None self._docmap = {} @@ -82,7 +82,31 @@ def visit_object_type(self: QAPISchemaGenGolangVisitor, members: List[QAPISchemaObjectTypeMember], variants: Optional[QAPISchemaVariants] ) -> None: - pass + # Do not handle anything besides structs + if (name == self.schema.the_empty_object_type.name or + not isinstance(name, str) or + info.defn_meta not in ["struct"]): + return + + assert name not in self.objects_seen + self.objects_seen[name] = True + + # visit all inner objects as well, they are not going to be + # called by python's generator. + if variants: + for var in variants.variants: + assert isinstance(var.type, QAPISchemaObjectType) + self.visit_object_type(self, + var.type.name, + var.type.info, + var.type.ifcond, + var.type.base, + var.type.local_members, + var.type.variants) + + doc = self._docmap.get(info.defn_name, None) + self.target[info.defn_meta] += qapi_to_golang_struct(name, doc, info, + ifcond, features, base, members, variants) def visit_alternate_type(self: QAPISchemaGenGolangVisitor, name: str, @@ -276,6 +300,14 @@ def gen_golang(schema: QAPISchema, schema.visit(vis) vis.write(output_dir) +# Helper function for boxed or self contained structures. +def generate_struct_type(type_name, args="", doc_struct="") -> str: + args = args if len(args) == 0 else f"\n{args}\n" + return f''' +{doc_struct} +type {type_name} struct {{{args}}} +''' + def generate_marshal_methods_enum(members: List[QAPISchemaEnumMember]) -> str: type = qapi_to_go_type_name(members[0].defined_in, "enum") @@ -345,6 +377,46 @@ def qapi_to_golang_struct_docs(doc: QAPIDoc) -> (str, Dict[str, str]): return main, fields +# Helper function that is used for most of QAPI types +def qapi_to_golang_struct(name: str, + doc: QAPIDoc, + info: Optional[QAPISourceInfo], + ifcond: QAPISchemaIfCond, + features: List[QAPISchemaFeature], + base: Optional[QAPISchemaObjectType], + members: List[QAPISchemaObjectTypeMember], + variants: Optional[QAPISchemaVariants]) -> str: + + type_name = qapi_to_go_type_name(name, info.defn_meta) + doc_struct, doc_fields = qapi_to_golang_struct_docs(doc) + + base_fields = "" + if base: + base_type_name = qapi_to_go_type_name(base.name, base.meta) + base_fields = f"\t// Base type for this struct\n\t{base_type_name}\n" + + own_fields = "" + for memb in members: + field = qapi_to_field_name(memb.name) + member_type = qapi_schema_type_to_go_type(memb.type.name) + + # In Golang, we are using "encoding/json" library to Marshal and Unmarshal between + # over-the-wire QMP and Golang struct. The field tag 'omitempty' does not behave as + # expected for some types with default values and they only way to "ignore by default" + # unset fields is to have them as reference in the Struct. + # This way, a *bool and *int can be ignored where a bool or int might have been set. + isptr = "*" if memb.optional and member_type[0] not in "*[" else "" + optional = ",omitempty" if memb.optional else "" + fieldtag = '`json:"{name}{optional}"`'.format(name=memb.name,optional=optional) + + field_doc = doc_fields.get(memb.name, "") + own_fields += f"\t{field} {isptr}{member_type}{fieldtag}{field_doc}\n" + + all_fields = base_fields if len(base_fields) > 0 else "" + all_fields += own_fields[:-1] if len(own_fields) > 0 else "" + + return generate_struct_type(type_name, all_fields, doc_struct) + def qapi_schema_type_to_go_type(type: str) -> str: schema_types_to_go = {'str': 'string', 'null': 'nil', 'bool': 'bool', 'number': 'float64', 'size': 'uint64', 'int': 'int64', 'int8': 'int8', @@ -364,6 +436,9 @@ def qapi_schema_type_to_go_type(type: str) -> str: def qapi_to_field_name_enum(name: str) -> str: return name.title().replace("-", "") +def qapi_to_field_name(name: str) -> str: + return name.title().replace("_", "").replace("-", "") + def qapi_to_go_type_name(name: str, meta: str) -> str: if name.startswith("q_obj_"): name = name[6:] From patchwork Fri Apr 1 22:41:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Toso X-Patchwork-Id: 1612437 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=QRwY0L3I; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KVZwj70QJz9sG4 for ; Sat, 2 Apr 2022 09:44:21 +1100 (AEDT) Received: from localhost ([::1]:52872 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1naQ0B-0007GN-NE for incoming@patchwork.ozlabs.org; Fri, 01 Apr 2022 18:44:19 -0400 Received: from eggs.gnu.org ([209.51.188.92]:39222) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1naPxI-000250-0y for qemu-devel@nongnu.org; Fri, 01 Apr 2022 18:41:20 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:28502) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1naPxG-0005QL-57 for qemu-devel@nongnu.org; Fri, 01 Apr 2022 18:41:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648852877; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xSFfrvWh1Xt40OQDhsyiB4ZhWoJF6DziWwoJBjKGkdo=; b=QRwY0L3IqMZSyZDhVd6sLsc7D103hhxcCwwJSuMwnMh9LHXoFWiAu2VXFjhcpeK8ILQr5e C7EP54yFrWo7rlFs4GAAtgLHYHqWQWwHWmdewIKkvvg+XBcG9N6KnaOLbBGafQPaGB+0Jw Y21Pvr0hJ1DGxLNZIC4lK2KftYKeWRg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-567-wJxMd-U9OUCjuSqYIGXh5Q-1; Fri, 01 Apr 2022 18:41:14 -0400 X-MC-Unique: wJxMd-U9OUCjuSqYIGXh5Q-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 23883802803 for ; Fri, 1 Apr 2022 22:41:14 +0000 (UTC) Received: from tapioca.redhat.com (unknown [10.40.193.147]) by smtp.corp.redhat.com (Postfix) with ESMTP id 85AA49E6A; Fri, 1 Apr 2022 22:41:12 +0000 (UTC) From: Victor Toso To: qemu-devel@nongnu.org Subject: [RFC PATCH v1 4/8] qapi: golang: Generate qapi's union types in Go Date: Sat, 2 Apr 2022 00:41:00 +0200 Message-Id: <20220401224104.145961-5-victortoso@redhat.com> In-Reply-To: <20220401224104.145961-1-victortoso@redhat.com> References: <20220401224104.145961-1-victortoso@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=victortoso@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.129.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: John Snow , Eric Blake , Markus Armbruster Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This patch handles QAPI union types and generates the equivalent data structures and methods in Go to handle it. At the moment of this writing, it generates 67 structures. The QAPI union type can be summarized by its common members that are defined in a @base struct and a @value. The @value type can vary and depends on @base's field that we call @discriminator. The @discriminator is always a Enum type. Golang does not have Unions. The generation of QAPI union type in Go with this patch, follows similar approach to what is done for QAPI struct types and QAPI alternate types. Similarly to Go implementation of QAPI alternate types, we will implement the Marshaler and Unmarshaler interfaces to seamless decode from JSON objects to Golang structs and vice versa. Similarly to Go implementation of QAPI struct types, we will need to tag @base fields accordingly. The embedded documentation in Golang's structures and fields are particularly important here, to help developers know what Types to use for @value. Runtime checks too. Signed-off-by: Victor Toso --- scripts/qapi/golang.py | 124 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 119 insertions(+), 5 deletions(-) diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py index 50e39f8925..0a1bf430ba 100644 --- a/scripts/qapi/golang.py +++ b/scripts/qapi/golang.py @@ -31,7 +31,7 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor): def __init__(self, prefix: str): super().__init__() - self.target = {name: "" for name in ["alternate", "enum", "helper", "struct"]} + self.target = {name: "" for name in ["alternate", "enum", "helper", "struct", "union"]} self.objects_seen = {} self.schema = None self._docmap = {} @@ -82,10 +82,10 @@ def visit_object_type(self: QAPISchemaGenGolangVisitor, members: List[QAPISchemaObjectTypeMember], variants: Optional[QAPISchemaVariants] ) -> None: - # Do not handle anything besides structs + # Do not handle anything besides struct and unions. if (name == self.schema.the_empty_object_type.name or not isinstance(name, str) or - info.defn_meta not in ["struct"]): + info.defn_meta not in ["struct", "union"]): return assert name not in self.objects_seen @@ -351,6 +351,93 @@ def generate_marshal_methods_enum(members: List[QAPISchemaEnumMember]) -> str: }} ''' +# Marshal methods for Union types +def generate_marshal_methods(type: str, + type_dict: Dict[str, str], + discriminator: str = "", + base: str = "") -> str: + assert base != "" + discriminator = "base." + discriminator + + switch_case_format = ''' + case {name}: + value := {case_type}{{}} + if err := json.Unmarshal(data, &value); err != nil {{ + return err + }} + s.Value = value''' + + if_supported_types = "" + added = {} + switch_cases = "" + for name in sorted(type_dict): + case_type = type_dict[name] + isptr = "*" if case_type[0] not in "*[" else "" + switch_cases += switch_case_format.format(name = name, + case_type = case_type) + if case_type not in added: + if_supported_types += f'''typestr != "{case_type}" &&\n\t\t''' + added[case_type] = True + + marshalfn = f''' +func (s {type}) MarshalJSON() ([]byte, error) {{ + base, err := json.Marshal(s.{base}) + if err != nil {{ + return nil, err + }} + + typestr := fmt.Sprintf("%T", s.Value) + typestr = typestr[strings.LastIndex(typestr, ".")+1:] + + // "The branches need not cover all possible enum values" + // This means that on Marshal, we can safely ignore empty values + if typestr == "" {{ + return []byte(base), nil + }} + + // Runtime check for supported value types + if {if_supported_types[:-6]} {{ + return nil, errors.New(fmt.Sprintf("Type is not supported: %s", typestr)) + }} + value, err := json.Marshal(s.Value) + if err != nil {{ + return nil, err + }} + + // Workaround to avoid checking s.Value being empty + if string(value) == "{{}}" {{ + return []byte(base), nil + }} + + // Removes the last '}}' from base and the first '{{' from value, in order to + // return a single JSON object. + result := fmt.Sprintf("%s,%s", base[:len(base)-1], value[1:]) + return []byte(result), nil +}} +''' + unmarshal_base = f''' + var base {base} + if err := json.Unmarshal(data, &base); err != nil {{ + return err + }} + s.{base} = base +''' + unmarshal_default_warn = f''' + default: + fmt.Println("Failed to decode {type}", {discriminator})''' + + return f'''{marshalfn} +func (s *{type}) UnmarshalJSON(data []byte) error {{ + {unmarshal_base} + switch {discriminator} {{ +{switch_cases[1:]} + {unmarshal_default_warn} + }} + + return nil +}} +''' + # Takes the documentation object of a specific type and returns # that type's documentation followed by its member's docs. def qapi_to_golang_struct_docs(doc: QAPIDoc) -> (str, Dict[str, str]): @@ -412,10 +499,37 @@ def qapi_to_golang_struct(name: str, field_doc = doc_fields.get(memb.name, "") own_fields += f"\t{field} {isptr}{member_type}{fieldtag}{field_doc}\n" + union_types = {} + variant_fields = "" + if variants: + variant_fields = f"// Value based on @{variants.tag_member.name}, possible types:" + for var in variants.variants: + if var.type.is_implicit(): + continue + + name = variants.tag_member._type_name + var.name.title().replace("-", "") + union_types[name] = var.type.name + variant_fields += f"\n\t// * {var.type.c_unboxed_type()}" + + variant_fields += f"\n\tValue Any" + all_fields = base_fields if len(base_fields) > 0 else "" all_fields += own_fields[:-1] if len(own_fields) > 0 else "" - - return generate_struct_type(type_name, all_fields, doc_struct) + all_fields += variant_fields if len(variant_fields) > 0 else "" + + unmarshal_fn = "" + if info.defn_meta == "union" and variants is not None: + # Union's without variants are the Union's base data structure. + # e.g: SchemaInfo's base is SchemainfoBase. + discriminator = qapi_to_field_name(variants.tag_member.name) + base = qapi_to_go_type_name(variants.tag_member.defined_in, + variants.tag_member.info.defn_meta) + unmarshal_fn = generate_marshal_methods(type_name, + union_types, + discriminator = discriminator, + base = base_type_name) + + return generate_struct_type(type_name, all_fields, doc_struct) + unmarshal_fn def qapi_schema_type_to_go_type(type: str) -> str: schema_types_to_go = {'str': 'string', 'null': 'nil', 'bool': 'bool', From patchwork Fri Apr 1 22:41:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Toso X-Patchwork-Id: 1612435 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=LgRKSZmz; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KVZtj36JRz9sG4 for ; Sat, 2 Apr 2022 09:42:37 +1100 (AEDT) Received: from localhost ([::1]:47020 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1naPyV-0003Cf-Fr for incoming@patchwork.ozlabs.org; Fri, 01 Apr 2022 18:42:35 -0400 Received: from eggs.gnu.org ([209.51.188.92]:39214) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1naPxH-00024h-Vy for qemu-devel@nongnu.org; Fri, 01 Apr 2022 18:41:20 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:54868) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1naPxG-0005QI-0P for qemu-devel@nongnu.org; Fri, 01 Apr 2022 18:41:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648852877; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=InuXhU4iwrbe0GkBIVkp7McvHI2BOEzzRIPXtveGZxA=; b=LgRKSZmzimY4NI7y7yF51e47nUjlJJEnS3q5XjsXuPEds3p/q1k7MpDAcNK/Uu86zEqmqX 2Q3WBPbcy1XrfUv59nO9UxZOY4DoGivDWnhSzLT/Sgg4ZKtXtkImspJGK6d2dQkkLlHXfs 6a1sRHnxNMF59qFVtwcq5NltBLi2Owo= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-597-yln5NY0JM8-ywpOYX8FNag-1; Fri, 01 Apr 2022 18:41:16 -0400 X-MC-Unique: yln5NY0JM8-ywpOYX8FNag-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E543E185A7A4 for ; Fri, 1 Apr 2022 22:41:15 +0000 (UTC) Received: from tapioca.redhat.com (unknown [10.40.193.147]) by smtp.corp.redhat.com (Postfix) with ESMTP id C3AF59E61; Fri, 1 Apr 2022 22:41:14 +0000 (UTC) From: Victor Toso To: qemu-devel@nongnu.org Subject: [RFC PATCH v1 5/8] qapi: golang: Generate qapi's event types in Go Date: Sat, 2 Apr 2022 00:41:01 +0200 Message-Id: <20220401224104.145961-6-victortoso@redhat.com> In-Reply-To: <20220401224104.145961-1-victortoso@redhat.com> References: <20220401224104.145961-1-victortoso@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=victortoso@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.129.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: John Snow , Eric Blake , Markus Armbruster Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This patch handles QAPI event types and generates data structures in Go that handles it. At the moment of this writing, it generates 51 structures (49 events) In Golang, each event is handled as a Go structure and there is no big difference, in the Go generated code, between what is a QAPI event type and what is a QAPI struct. Each QAPI event has the suffix 'Event' in its Golang data structure and contains the fields, mandatory and optional, that can be sent or received. In addition, there are two structures added to handle QAPI specification for event types: 'Event' and 'EventBase'. 'EventBase' contains @Name and @Timestamp fields and then 'Event' extends 'EventBase' with an @Arg field of type 'Any'. The 'Event' type implements the Unmarshaler to decode the QMP JSON Object into the correct Golang (event) struct. The goal here is to facilitate receiving Events. A TODO for this type is to implement Marshaler for 'Event'. It'll containt runtime checks to validate before transforming the struct into a JSON Object. Example: ```go qmpMsg := `{ "event" : "MIGRATION", "timestamp":{ "seconds": 1432121972, "microseconds": 744001 }, "data":{ "status": "completed" } }` e := Event{} _ = json.Unmarshal([]byte(qmpMsg), &e) // e.Name == "MIGRATION" // e.Arg.(MigrationEvent).Status == MigrationStatusCompleted ``` Signed-off-by: Victor Toso --- scripts/qapi/golang.py | 92 ++++++++++++++++++++++++++++++++++++++---- 1 file changed, 84 insertions(+), 8 deletions(-) diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py index 0a1bf430ba..3bb66d07c7 100644 --- a/scripts/qapi/golang.py +++ b/scripts/qapi/golang.py @@ -31,9 +31,10 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor): def __init__(self, prefix: str): super().__init__() - self.target = {name: "" for name in ["alternate", "enum", "helper", "struct", "union"]} + self.target = {name: "" for name in ["alternate", "enum", "event", "helper", "struct", "union"]} self.objects_seen = {} self.schema = None + self.events = {} self._docmap = {} self.golang_package_name = "qapi" @@ -57,6 +58,24 @@ def visit_begin(self, schema): def visit_end(self): self.schema = None + # EventBase and Event are not specified in the QAPI schema, + # so we need to generate it ourselves. + self.target["event"] += ''' +type EventBase struct { + Name string `json:"event"` + Timestamp struct { + Seconds int64 `json:"seconds"` + Microseconds int64 `json:"microseconds"` + } `json:"timestamp"` +} + +type Event struct { + EventBase + Arg Any `json:"data,omitempty"` +} +''' + self.target["event"] += generate_marshal_methods('Event', self.events) + self.target["helper"] += ''' // Creates a decoder that errors on unknown Fields // Returns true if successfully decoded @from string @into type @@ -279,7 +298,28 @@ def visit_command(self, pass def visit_event(self, name, info, ifcond, features, arg_type, boxed): - pass + assert name == info.defn_name + type_name = qapi_to_go_type_name(name, info.defn_meta) + self.events[name] = type_name + + doc = self._docmap.get(name, None) + self_contained = True if not arg_type or not arg_type.name.startswith("q_obj") else False + content = "" + if self_contained: + doc_struct, _ = qapi_to_golang_struct_docs(doc) + content = generate_struct_type(type_name, "", doc_struct) + else: + assert isinstance(arg_type, QAPISchemaObjectType) + content = qapi_to_golang_struct(name, + doc, + arg_type.info, + arg_type.ifcond, + arg_type.features, + arg_type.base, + arg_type.members, + arg_type.variants) + + self.target["event"] += content def write(self, output_dir: str) -> None: for module_name, content in self.target.items(): @@ -351,15 +391,41 @@ def generate_marshal_methods_enum(members: List[QAPISchemaEnumMember]) -> str: }} ''' -# Marshal methods for Union types +# Marshal methods for Event and Union types def generate_marshal_methods(type: str, type_dict: Dict[str, str], discriminator: str = "", base: str = "") -> str: - assert base != "" - discriminator = "base." + discriminator - - switch_case_format = ''' + type_is_union = False + json_field = "" + struct_field = "" + if type == "Event": + base = type + "Base" + discriminator = "base.Name" + struct_field = "Arg" + json_field = "data" + else: + assert base != "" + discriminator = "base." + discriminator + type_is_union = True + + switch_case_format = "" + if not type_is_union: + switch_case_format = ''' + case "{name}": + tmp := struct {{ + Value {isptr}{case_type} `json:"{json_field},omitempty"` + }}{{}} + if err := json.Unmarshal(data, &tmp); err != nil {{ + return err + }} + if tmp.Value == nil {{ + s.{struct_field} = nil + }} else {{ + s.{struct_field} = {isptr}tmp.Value + }}''' + else: + switch_case_format = ''' case {name}: value := {case_type}{{}} if err := json.Unmarshal(data, &value); err != nil {{ @@ -374,12 +440,17 @@ def generate_marshal_methods(type: str, case_type = type_dict[name] isptr = "*" if case_type[0] not in "*[" else "" switch_cases += switch_case_format.format(name = name, + struct_field = struct_field, + json_field = json_field, + isptr = isptr, case_type = case_type) if case_type not in added: if_supported_types += f'''typestr != "{case_type}" &&\n\t\t''' added[case_type] = True - marshalfn = f''' + marshalfn = "" + if type_is_union: + marshalfn = f''' func (s {type}) MarshalJSON() ([]byte, error) {{ base, err := json.Marshal(s.{base}) if err != nil {{ @@ -564,4 +635,9 @@ def qapi_to_go_type_name(name: str, meta: str) -> str: words = [word for word in name.replace("_", "-").split("-")] name = words[0].title() if words[0].islower() or words[0].isupper() else words[0] name += ''.join(word.title() for word in words[1:]) + + if meta == "event": + name = name[:-3] if name.endswith("Arg") else name + name += meta.title() + return name From patchwork Fri Apr 1 22:41:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Toso X-Patchwork-Id: 1612440 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=C0kxCPPC; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KVb1Y30Qcz9sG4 for ; Sat, 2 Apr 2022 09:48:31 +1100 (AEDT) Received: from localhost ([::1]:60246 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1naQ4C-00049g-Uc for incoming@patchwork.ozlabs.org; Fri, 01 Apr 2022 18:48:28 -0400 Received: from eggs.gnu.org ([209.51.188.92]:39254) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1naPxJ-0002A0-JH for qemu-devel@nongnu.org; Fri, 01 Apr 2022 18:41:21 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:47163) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1naPxH-0005Qk-HF for qemu-devel@nongnu.org; Fri, 01 Apr 2022 18:41:21 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648852879; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wZXMBWOTEdvWfBJyI1mCujsRhcs36+ZrJHiK6scddBs=; b=C0kxCPPC2IaqhlAQSvZw++o/Gy1VZ0RxyhLAprCZ8/C3wqP4+pCEU4V3jdNjHWYf/Yq80p P/sscOh4Lj22kN/pnsUqExJlnwh1Vpzgj2Ly6eRvcX7NYKYW8moSlr9SK6fmwFg9AbMto0 p2ZtMqDE4slfHywhi4I9wh6qxH2M8bw= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-654-HbaSe5VOOJ2mLtGE1SWIdA-1; Fri, 01 Apr 2022 18:41:18 -0400 X-MC-Unique: HbaSe5VOOJ2mLtGE1SWIdA-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C031C811E75 for ; Fri, 1 Apr 2022 22:41:17 +0000 (UTC) Received: from tapioca.redhat.com (unknown [10.40.193.147]) by smtp.corp.redhat.com (Postfix) with ESMTP id 521579E61; Fri, 1 Apr 2022 22:41:16 +0000 (UTC) From: Victor Toso To: qemu-devel@nongnu.org Subject: [RFC PATCH v1 6/8] qapi: golang: Generate qapi's command types in Go Date: Sat, 2 Apr 2022 00:41:02 +0200 Message-Id: <20220401224104.145961-7-victortoso@redhat.com> In-Reply-To: <20220401224104.145961-1-victortoso@redhat.com> References: <20220401224104.145961-1-victortoso@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=victortoso@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.133.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: John Snow , Eric Blake , Markus Armbruster Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This patch handles QAPI command types and generates data structures in Go that decodes from QMP JSON Object to Go data structure and vice versa. At the time of this writing, it generates 210 structures (208 commands) This is very similar to previous commit, that handles QAPI union types in Go. Each QAPI command will generate a Go struct with the suffix 'Command'. Its fields, if any, are the mandatory or optional arguments defined in the QAPI command. Simlar to Event, this patch adds two structures to handle QAPI specification for command types: 'Command' and 'CommandBase'. 'CommandBase' contains @Id and @Name. 'Command' extends 'CommandBase' with an @Arg field of type 'Any'. The 'Command' type implements the Unmarshaler to decode QMP JSON Object into the correct Golang (command) struct. Marshal example: ```go cmdArg := SetPasswordCommand{} cmdArg.Protocol = DisplayProtocolVnc cmdArg.Password = "secret" cmd := Command{} cmd.Name = "set_password" cmd.Arg = cmdArg b, _ := json.Marshal(&cmd) // string(b) == `{"execute":"set_password","arguments":{"protocol":"vnc","password":"secret"}}` ``` Unmarshal example: ```go qmpCommand := ` { "execute": "set_password", "arguments":{ "protocol": "vnc", "password": "secret" } }` cmd := Command{} _ = json.Unmarshal([]byte(qmpCommand), &cmd) // cmd.Name == "set_password" // cmd1.Arg.(SetPasswordCommand).Protocol == DisplayProtocolVnc ``` Signed-off-by: Victor Toso --- scripts/qapi/golang.py | 49 ++++++++++++++++++++++++++++++++++++++---- 1 file changed, 45 insertions(+), 4 deletions(-) diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py index 3bb66d07c7..0b9c19babb 100644 --- a/scripts/qapi/golang.py +++ b/scripts/qapi/golang.py @@ -31,10 +31,11 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor): def __init__(self, prefix: str): super().__init__() - self.target = {name: "" for name in ["alternate", "enum", "event", "helper", "struct", "union"]} + self.target = {name: "" for name in ["alternate", "command", "enum", "event", "helper", "struct", "union"]} self.objects_seen = {} self.schema = None self.events = {} + self.commands = {} self._docmap = {} self.golang_package_name = "qapi" @@ -76,6 +77,19 @@ def visit_end(self): ''' self.target["event"] += generate_marshal_methods('Event', self.events) + self.target["command"] += ''' +type CommandBase struct { + Id string `json:"id,omitempty"` + Name string `json:"execute"` +} + +type Command struct { + CommandBase + Arg Any `json:"arguments,omitempty"` +} +''' + self.target["command"] += generate_marshal_methods('Command', self.commands) + self.target["helper"] += ''' // Creates a decoder that errors on unknown Fields // Returns true if successfully decoded @from string @into type @@ -295,7 +309,29 @@ def visit_command(self, allow_oob: bool, allow_preconfig: bool, coroutine: bool) -> None: - pass + assert name == info.defn_name + type_name = qapi_to_go_type_name(name, info.defn_meta) + self.commands[name] = type_name + + doc = self._docmap.get(name, None) + self_contained = True if not arg_type or not arg_type.name.startswith("q_obj") else False + content = "" + if boxed or self_contained: + args = "" if not arg_type else "\n" + arg_type.name + doc_struct, _ = qapi_to_golang_struct_docs(doc) + content = generate_struct_type(type_name, args, doc_struct) + else: + assert isinstance(arg_type, QAPISchemaObjectType) + content = qapi_to_golang_struct(name, + doc, + arg_type.info, + arg_type.ifcond, + arg_type.features, + arg_type.base, + arg_type.members, + arg_type.variants) + + self.target["command"] += content def visit_event(self, name, info, ifcond, features, arg_type, boxed): assert name == info.defn_name @@ -391,7 +427,7 @@ def generate_marshal_methods_enum(members: List[QAPISchemaEnumMember]) -> str: }} ''' -# Marshal methods for Event and Union types +# Marshal methods for Event, Commad and Union types def generate_marshal_methods(type: str, type_dict: Dict[str, str], discriminator: str = "", @@ -404,6 +440,11 @@ def generate_marshal_methods(type: str, discriminator = "base.Name" struct_field = "Arg" json_field = "data" + elif type == "Command": + base = type + "Base" + discriminator = "base.Name" + struct_field = "Arg" + json_field = "arguments" else: assert base != "" discriminator = "base." + discriminator @@ -636,7 +677,7 @@ def qapi_to_go_type_name(name: str, meta: str) -> str: name = words[0].title() if words[0].islower() or words[0].isupper() else words[0] name += ''.join(word.title() for word in words[1:]) - if meta == "event": + if meta == "event" or meta == "command": name = name[:-3] if name.endswith("Arg") else name name += meta.title() From patchwork Fri Apr 1 22:41:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Toso X-Patchwork-Id: 1612439 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=UA67JrTs; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KVZz45Nc8z9s0r for ; Sat, 2 Apr 2022 09:46:24 +1100 (AEDT) Received: from localhost ([::1]:57170 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1naQ2A-00027q-QS for incoming@patchwork.ozlabs.org; Fri, 01 Apr 2022 18:46:22 -0400 Received: from eggs.gnu.org ([209.51.188.92]:39276) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1naPxL-0002FK-4q for qemu-devel@nongnu.org; Fri, 01 Apr 2022 18:41:23 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:27175) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1naPxJ-0005RT-Ev for qemu-devel@nongnu.org; Fri, 01 Apr 2022 18:41:22 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648852880; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4pJQD6VkrgHuQFDZ9pmVsufEgTW8aQpUaRR1B2d0MUg=; b=UA67JrTsxCJJ6ChpqfHWpaVOUcBHppYH+1pM7Am+wZ0rijxsiYA54+lFHSUYT8x9KsffN7 tj8mFlvARGrpbsONoCR3wrrLdqvK0MzZrBGWddUNG4qs7bTy+m/YiHiuhMLuFkdap8Imyk 4GgjK1XJ/RYtsi8z2pASY0LiBCzIYoc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-447-hrjpn0wgPd6zCyOPCKYdrw-1; Fri, 01 Apr 2022 18:41:19 -0400 X-MC-Unique: hrjpn0wgPd6zCyOPCKYdrw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6818783395E for ; Fri, 1 Apr 2022 22:41:19 +0000 (UTC) Received: from tapioca.redhat.com (unknown [10.40.193.147]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2BEAD9E69; Fri, 1 Apr 2022 22:41:17 +0000 (UTC) From: Victor Toso To: qemu-devel@nongnu.org Subject: [RFC PATCH v1 7/8] qapi: golang: Add CommandResult type to Go Date: Sat, 2 Apr 2022 00:41:03 +0200 Message-Id: <20220401224104.145961-8-victortoso@redhat.com> In-Reply-To: <20220401224104.145961-1-victortoso@redhat.com> References: <20220401224104.145961-1-victortoso@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=victortoso@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.129.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: John Snow , Eric Blake , Markus Armbruster Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" This patch adds a struct type in Go that will handle return values for QAPI's command types. The return value of a Command is, encouraged to be, QAPI's complext types or an Array of those. The 'CommandResult' type acts in similar fashion to 'Event' and 'Command', in order to map the right return data structure based on the issued 'Command'. This patch also adds a syntax sugar method to 'Command' to return the 'CommandResult' struct to use when receiving the return data. Example: ```go cmd := Command{} cmd.Name = `query-tpm-models` // bytes, _ := json.Marshal(&cmd) // send bytes ... received := `{"return":["tpm-tis","tpm-crb","tpm-spapr"]}` cmdRet := cmd.GetReturnType() _ = json.Unmarshal([]byte(received), &cmdRet) // cmdRet.Value.([]TpmModel)[2] == TpmModelTpmSpapr ``` Signed-off-by: Victor Toso --- scripts/qapi/golang.py | 42 ++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 40 insertions(+), 2 deletions(-) diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py index 0b9c19babb..5d3395514d 100644 --- a/scripts/qapi/golang.py +++ b/scripts/qapi/golang.py @@ -36,6 +36,7 @@ def __init__(self, prefix: str): self.schema = None self.events = {} self.commands = {} + self.command_results = {} self._docmap = {} self.golang_package_name = "qapi" @@ -90,6 +91,32 @@ def visit_end(self): ''' self.target["command"] += generate_marshal_methods('Command', self.commands) + self.target["command"] += ''' +type CommandResult struct { + CommandBase + Value Any `json:"return,omitempty"` +} + +func (s Command) GetReturnType() CommandResult { + return CommandResult{ + CommandBase: s.CommandBase, + } +} + +// In order to evaluate nil value to empty JSON object +func (s *CommandResult) MarshalJSON() ([]byte, error) { + if s.Value == nil { + return []byte(`{"return":{}}`), nil + } + tmp := struct { + Value Any `json:"return"` + }{Value: s.Value} + + return json.Marshal(&tmp) +} +''' + self.target["command"] += generate_marshal_methods('CommandResult', self.command_results) + self.target["helper"] += ''' // Creates a decoder that errors on unknown Fields // Returns true if successfully decoded @from string @into type @@ -312,6 +339,9 @@ def visit_command(self, assert name == info.defn_name type_name = qapi_to_go_type_name(name, info.defn_meta) self.commands[name] = type_name + if ret_type: + ret_type_name = qapi_schema_type_to_go_type(ret_type.name) + self.command_results[name] = ret_type_name doc = self._docmap.get(name, None) self_contained = True if not arg_type or not arg_type.name.startswith("q_obj") else False @@ -445,6 +475,11 @@ def generate_marshal_methods(type: str, discriminator = "base.Name" struct_field = "Arg" json_field = "arguments" + elif type == "CommandResult": + base = "CommandBase" + discriminator = "s.Name" + struct_field = "Value" + json_field = "return" else: assert base != "" discriminator = "base." + discriminator @@ -527,14 +562,17 @@ def generate_marshal_methods(type: str, return []byte(result), nil }} ''' - unmarshal_base = f''' + unmarshal_base = "" + unmarshal_default_warn = "" + if type != "CommandResult": + unmarshal_base = f''' var base {base} if err := json.Unmarshal(data, &base); err != nil {{ return err }} s.{base} = base ''' - unmarshal_default_warn = f''' + unmarshal_default_warn = f''' default: fmt.Println("Failed to decode {type}", {discriminator})''' From patchwork Fri Apr 1 22:41:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Toso X-Patchwork-Id: 1612441 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: bilbo.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=ARhL15IL; dkim-atps=neutral Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by bilbo.ozlabs.org (Postfix) with ESMTPS id 4KVb1q4Dp6z9sG4 for ; Sat, 2 Apr 2022 09:48:47 +1100 (AEDT) Received: from localhost ([::1]:33144 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1naQ4T-00051k-Hu for incoming@patchwork.ozlabs.org; Fri, 01 Apr 2022 18:48:45 -0400 Received: from eggs.gnu.org ([209.51.188.92]:39286) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1naPxM-0002HN-7T for qemu-devel@nongnu.org; Fri, 01 Apr 2022 18:41:24 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:41461) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1naPxK-0005Rq-QM for qemu-devel@nongnu.org; Fri, 01 Apr 2022 18:41:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648852882; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=m2y6noU8i5iQs9foSEVD7vizxisFTiJM6wNoZnk8KtI=; b=ARhL15ILTST1WAJLhK6/9ZFCowEJlseRK1DQU5LD1/6SM4VbYQbJC8z3FcS+1X1KFooo5+ rjfiI4rCCrQNpbdUC16PK3TXAAbXNer5Gy5HaiyJswPkz14I/qqA8uecJLcn4dndy/JYFo SKMSPw07ydTPsPsFu0zQTY3sNr2M708= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-383-cGNWNmuUMK2Nkj2IJceooQ-1; Fri, 01 Apr 2022 18:41:21 -0400 X-MC-Unique: cGNWNmuUMK2Nkj2IJceooQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D210685A5A8 for ; Fri, 1 Apr 2022 22:41:20 +0000 (UTC) Received: from tapioca.redhat.com (unknown [10.40.193.147]) by smtp.corp.redhat.com (Postfix) with ESMTP id C7D8D9E61; Fri, 1 Apr 2022 22:41:19 +0000 (UTC) From: Victor Toso To: qemu-devel@nongnu.org Subject: [RFC PATCH v1 8/8] qapi: golang: document skip function visit_array_types Date: Sat, 2 Apr 2022 00:41:04 +0200 Message-Id: <20220401224104.145961-9-victortoso@redhat.com> In-Reply-To: <20220401224104.145961-1-victortoso@redhat.com> References: <20220401224104.145961-1-victortoso@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.11.54.5 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=victortoso@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Received-SPF: pass client-ip=170.10.129.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -28 X-Spam_score: -2.9 X-Spam_bar: -- X-Spam_report: (-2.9 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: John Snow , Eric Blake , Markus Armbruster Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Victor Toso --- scripts/qapi/golang.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py index 5d3395514d..9a775d0691 100644 --- a/scripts/qapi/golang.py +++ b/scripts/qapi/golang.py @@ -321,7 +321,12 @@ def visit_enum_type(self: QAPISchemaGenGolangVisitor, ''' def visit_array_type(self, name, info, ifcond, element_type): - pass + # TLDR: We don't need to any extra boilerplate in Go to handle Arrays. + # + # This function is implemented just to be sure that: + # 1. Every array type ends with List + # 2. Every array type's element is the array type without 'List' + assert name.endswith("List") and name[:-4] == element_type.name def visit_command(self, name: str,