From patchwork Wed Sep 27 11:25:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Toso X-Patchwork-Id: 1840171 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=L2eDBakV; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4RwZ8s4XF0z1ypJ for ; Wed, 27 Sep 2023 21:27:41 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qlSg3-0002PE-Rm; Wed, 27 Sep 2023 07:25:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qlSfy-0002Od-BO for qemu-devel@nongnu.org; Wed, 27 Sep 2023 07:25:54 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qlSfw-0001Yv-D0 for qemu-devel@nongnu.org; Wed, 27 Sep 2023 07:25:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1695813950; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=00JI1RwlBBmXYEiPlKOJV34H262RZWf9bT/B+xvDdOk=; b=L2eDBakVUL2TjnwooN+Eg/3rFJ1b9kjvUPA+b6n04n0uNlq7ldkO4a6Ef4mr1xihchj3El r/6Rccmb24GMXgf8i8xG1p2902krtjKy0X0RLWnRc9sk+YVpsxaOcmZmWVQl6dpOMND77Z XJFq8SJrcSshn+bwga+iB7SA3JWWoPM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-537-FK9fyarUOV-QclxpqbwM1w-1; Wed, 27 Sep 2023 07:25:48 -0400 X-MC-Unique: FK9fyarUOV-QclxpqbwM1w-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4816C8039CF for ; Wed, 27 Sep 2023 11:25:48 +0000 (UTC) Received: from tapioca.lan (unknown [10.45.224.4]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3305F1054FC2; Wed, 27 Sep 2023 11:25:47 +0000 (UTC) From: Victor Toso To: qemu-devel@nongnu.org Cc: Markus Armbruster , John Snow , =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= Subject: [PATCH v1 1/9] qapi: golang: Generate qapi's enum types in Go Date: Wed, 27 Sep 2023 13:25:36 +0200 Message-ID: <20230927112544.85011-2-victortoso@redhat.com> In-Reply-To: <20230927112544.85011-1-victortoso@redhat.com> References: <20230927112544.85011-1-victortoso@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.129.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org This patch handles QAPI enum types and generates its equivalent in Go. Basically, Enums are being handled as strings in Golang. 1. For each QAPI enum, we will define a string type in Go to be the assigned type of this specific enum. 2. Naming: CamelCase will be used in any identifier that we want to export [0], which is everything. [0] https://go.dev/ref/spec#Exported_identifiers Example: qapi: | { 'enum': 'DisplayProtocol', | 'data': [ 'vnc', 'spice' ] } go: | type DisplayProtocol string | | const ( | DisplayProtocolVnc DisplayProtocol = "vnc" | DisplayProtocolSpice DisplayProtocol = "spice" | ) Signed-off-by: Victor Toso --- scripts/qapi/golang.py | 140 +++++++++++++++++++++++++++++++++++++++++ scripts/qapi/main.py | 2 + 2 files changed, 142 insertions(+) create mode 100644 scripts/qapi/golang.py diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py new file mode 100644 index 0000000000..87081cdd05 --- /dev/null +++ b/scripts/qapi/golang.py @@ -0,0 +1,140 @@ +""" +Golang QAPI generator +""" +# Copyright (c) 2023 Red Hat Inc. +# +# Authors: +# Victor Toso +# +# This work is licensed under the terms of the GNU GPL, version 2. +# See the COPYING file in the top-level directory. + +# due QAPISchemaVisitor interface +# pylint: disable=too-many-arguments + +# Just for type hint on self +from __future__ import annotations + +import os +from typing import List, Optional + +from .schema import ( + QAPISchema, + QAPISchemaType, + QAPISchemaVisitor, + QAPISchemaEnumMember, + QAPISchemaFeature, + QAPISchemaIfCond, + QAPISchemaObjectType, + QAPISchemaObjectTypeMember, + QAPISchemaVariants, +) +from .source import QAPISourceInfo + +TEMPLATE_ENUM = ''' +type {name} string +const ( +{fields} +) +''' + + +def gen_golang(schema: QAPISchema, + output_dir: str, + prefix: str) -> None: + vis = QAPISchemaGenGolangVisitor(prefix) + schema.visit(vis) + vis.write(output_dir) + + +def qapi_to_field_name_enum(name: str) -> str: + return name.title().replace("-", "") + + +class QAPISchemaGenGolangVisitor(QAPISchemaVisitor): + + def __init__(self, _: str): + super().__init__() + types = ["enum"] + self.target = {name: "" for name in types} + self.schema = None + self.golang_package_name = "qapi" + + def visit_begin(self, schema): + self.schema = schema + + # Every Go file needs to reference its package name + for target in self.target: + self.target[target] = f"package {self.golang_package_name}\n" + + def visit_end(self): + self.schema = None + + def visit_object_type(self: QAPISchemaGenGolangVisitor, + name: str, + info: Optional[QAPISourceInfo], + ifcond: QAPISchemaIfCond, + features: List[QAPISchemaFeature], + base: Optional[QAPISchemaObjectType], + members: List[QAPISchemaObjectTypeMember], + variants: Optional[QAPISchemaVariants] + ) -> None: + pass + + def visit_alternate_type(self: QAPISchemaGenGolangVisitor, + name: str, + info: Optional[QAPISourceInfo], + ifcond: QAPISchemaIfCond, + features: List[QAPISchemaFeature], + variants: QAPISchemaVariants + ) -> None: + pass + + def visit_enum_type(self: QAPISchemaGenGolangVisitor, + name: str, + info: Optional[QAPISourceInfo], + ifcond: QAPISchemaIfCond, + features: List[QAPISchemaFeature], + members: List[QAPISchemaEnumMember], + prefix: Optional[str] + ) -> None: + + value = qapi_to_field_name_enum(members[0].name) + fields = "" + for member in members: + value = qapi_to_field_name_enum(member.name) + fields += f'''\t{name}{value} {name} = "{member.name}"\n''' + + self.target["enum"] += TEMPLATE_ENUM.format(name=name, fields=fields[:-1]) + + def visit_array_type(self, name, info, ifcond, element_type): + pass + + def visit_command(self, + name: str, + info: Optional[QAPISourceInfo], + ifcond: QAPISchemaIfCond, + features: List[QAPISchemaFeature], + arg_type: Optional[QAPISchemaObjectType], + ret_type: Optional[QAPISchemaType], + gen: bool, + success_response: bool, + boxed: bool, + allow_oob: bool, + allow_preconfig: bool, + coroutine: bool) -> None: + pass + + def visit_event(self, name, info, ifcond, features, arg_type, boxed): + pass + + def write(self, output_dir: str) -> None: + for module_name, content in self.target.items(): + go_module = module_name + "s.go" + go_dir = "go" + pathname = os.path.join(output_dir, go_dir, go_module) + odir = os.path.dirname(pathname) + os.makedirs(odir, exist_ok=True) + + with open(pathname, "w", encoding="ascii") as outfile: + outfile.write(content) diff --git a/scripts/qapi/main.py b/scripts/qapi/main.py index 316736b6a2..cdbb3690fd 100644 --- a/scripts/qapi/main.py +++ b/scripts/qapi/main.py @@ -15,6 +15,7 @@ from .common import must_match from .error import QAPIError from .events import gen_events +from .golang import gen_golang from .introspect import gen_introspect from .schema import QAPISchema from .types import gen_types @@ -54,6 +55,7 @@ def generate(schema_file: str, gen_events(schema, output_dir, prefix) gen_introspect(schema, output_dir, prefix, unmask) + gen_golang(schema, output_dir, prefix) def main() -> int: """ From patchwork Wed Sep 27 11:25:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Toso X-Patchwork-Id: 1840168 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=gFhZYnnr; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4RwZ8S0fh7z1ypJ for ; Wed, 27 Sep 2023 21:27:20 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qlSgA-0002Qb-0U; Wed, 27 Sep 2023 07:26:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qlSfz-0002Ow-Uf for qemu-devel@nongnu.org; Wed, 27 Sep 2023 07:25:56 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qlSfw-0001Z1-Cs for qemu-devel@nongnu.org; Wed, 27 Sep 2023 07:25:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1695813951; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=sAUPdE5jD/YH8iyHvAfYBP9FgZYvjuflEAbYCGCnSJY=; b=gFhZYnnrDLiA9RwYLne86ssDAyOJduIU70V1bH6zOLm07EIkwudRS8AZfjNBlyDMggtnFG qJkhziCv6VYlO7DQJ+KMKIXgRU732+PG1ui2sVTKk4SvHREvUjVqwG1X+NmLc2I9x/LfQ0 PJlyT8/bPoa0rzybFMgf0n41ZTvJJU4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-410-XdABruarPyaYtewfwFxPEQ-1; Wed, 27 Sep 2023 07:25:50 -0400 X-MC-Unique: XdABruarPyaYtewfwFxPEQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B5096801FA9 for ; Wed, 27 Sep 2023 11:25:49 +0000 (UTC) Received: from tapioca.lan (unknown [10.45.224.4]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9CA0710F1BE9; Wed, 27 Sep 2023 11:25:48 +0000 (UTC) From: Victor Toso To: qemu-devel@nongnu.org Cc: Markus Armbruster , John Snow , =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= Subject: [PATCH v1 2/9] qapi: golang: Generate qapi's alternate types in Go Date: Wed, 27 Sep 2023 13:25:37 +0200 Message-ID: <20230927112544.85011-3-victortoso@redhat.com> In-Reply-To: <20230927112544.85011-1-victortoso@redhat.com> References: <20230927112544.85011-1-victortoso@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.133.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org This patch handles QAPI alternate types and generates data structures in Go that handles it. Alternate types are similar to Union but without a discriminator that can be used to identify the underlying value on the wire. It is needed to infer it. In Go, most of the types [*] are mapped as optional fields and Marshal and Unmarshal methods will be handling the data checks. Example: qapi: | { 'alternate': 'BlockdevRef', | 'data': { 'definition': 'BlockdevOptions', | 'reference': 'str' } } go: | type BlockdevRef struct { | Definition *BlockdevOptions | Reference *string | } usage: | input := `{"driver":"qcow2","data-file":"/some/place/my-image"}` | k := BlockdevRef{} | err := json.Unmarshal([]byte(input), &k) | if err != nil { | panic(err) | } | // *k.Definition.Qcow2.DataFile.Reference == "/some/place/my-image" [*] The exception for optional fields as default is to Types that can accept JSON Null as a value like StrOrNull and BlockdevRefOrNull. For this case, we translate Null with a boolean value in a field called IsNull. This will be explained better in the documentation patch of this series but the main rationale is around Marshaling to and from JSON and Go data structures. Example: qapi: | { 'alternate': 'StrOrNull', | 'data': { 's': 'str', | 'n': 'null' } } go: | type StrOrNull struct { | S *string | IsNull bool | } Signed-off-by: Victor Toso --- scripts/qapi/golang.py | 188 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 185 insertions(+), 3 deletions(-) diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py index 87081cdd05..43dbdde14c 100644 --- a/scripts/qapi/golang.py +++ b/scripts/qapi/golang.py @@ -16,10 +16,11 @@ from __future__ import annotations import os -from typing import List, Optional +from typing import Tuple, List, Optional from .schema import ( QAPISchema, + QAPISchemaAlternateType, QAPISchemaType, QAPISchemaVisitor, QAPISchemaEnumMember, @@ -38,6 +39,76 @@ ) ''' +TEMPLATE_HELPER = ''' +// Creates a decoder that errors on unknown Fields +// Returns nil if successfully decoded @from payload to @into type +// Returns error if failed to decode @from payload to @into type +func StrictDecode(into interface{}, from []byte) error { + dec := json.NewDecoder(strings.NewReader(string(from))) + dec.DisallowUnknownFields() + + if err := dec.Decode(into); err != nil { + return err + } + return nil +} +''' + +TEMPLATE_ALTERNATE = ''' +// Only implemented on Alternate types that can take JSON NULL as value. +// +// This is a helper for the marshalling code. It should return true only when +// the Alternate is empty (no members are set), otherwise it returns false and +// the member set to be Marshalled. +type AbsentAlternate interface { + ToAnyOrAbsent() (any, bool) +} +''' + +TEMPLATE_ALTERNATE_NULLABLE_CHECK = ''' + }} else if s.{var_name} != nil {{ + return *s.{var_name}, false''' + +TEMPLATE_ALTERNATE_MARSHAL_CHECK = ''' + if s.{var_name} != nil {{ + return json.Marshal(s.{var_name}) + }} else ''' + +TEMPLATE_ALTERNATE_UNMARSHAL_CHECK = ''' + // Check for {var_type} + {{ + s.{var_name} = new({var_type}) + if err := StrictDecode(s.{var_name}, data); err == nil {{ + return nil + }} + s.{var_name} = nil + }} +''' + +TEMPLATE_ALTERNATE_NULLABLE = ''' +func (s *{name}) ToAnyOrAbsent() (any, bool) {{ + if s != nil {{ + if s.IsNull {{ + return nil, false +{absent_check_fields} + }} + }} + + return nil, true +}} +''' + +TEMPLATE_ALTERNATE_METHODS = ''' +func (s {name}) MarshalJSON() ([]byte, error) {{ + {marshal_check_fields} + return {marshal_return_default} +}} + +func (s *{name}) UnmarshalJSON(data []byte) error {{ + {unmarshal_check_fields} + return fmt.Errorf("Can't convert to {name}: %s", string(data)) +}} +''' def gen_golang(schema: QAPISchema, output_dir: str, @@ -46,27 +117,135 @@ def gen_golang(schema: QAPISchema, schema.visit(vis) vis.write(output_dir) +def qapi_to_field_name(name: str) -> str: + return name.title().replace("_", "").replace("-", "") def qapi_to_field_name_enum(name: str) -> str: return name.title().replace("-", "") +def qapi_schema_type_to_go_type(qapitype: str) -> str: + schema_types_to_go = { + 'str': 'string', 'null': 'nil', 'bool': 'bool', 'number': + 'float64', 'size': 'uint64', 'int': 'int64', 'int8': 'int8', + 'int16': 'int16', 'int32': 'int32', 'int64': 'int64', 'uint8': + 'uint8', 'uint16': 'uint16', 'uint32': 'uint32', 'uint64': + 'uint64', 'any': 'any', 'QType': 'QType', + } + + prefix = "" + if qapitype.endswith("List"): + prefix = "[]" + qapitype = qapitype[:-4] + + qapitype = schema_types_to_go.get(qapitype, qapitype) + return prefix + qapitype + +def qapi_field_to_go_field(member_name: str, type_name: str) -> Tuple[str, str, str]: + # Nothing to generate on null types. We update some + # variables to handle json-null on marshalling methods. + if type_name == "null": + return "IsNull", "bool", "" + + # This function is called on Alternate, so fields should be ptrs + return qapi_to_field_name(member_name), qapi_schema_type_to_go_type(type_name), "*" + +# Helper function for boxed or self contained structures. +def generate_struct_type(type_name, args="") -> str: + args = args if len(args) == 0 else f"\n{args}\n" + with_type = f"\ntype {type_name}" if len(type_name) > 0 else "" + return f'''{with_type} struct {{{args}}} +''' + +def generate_template_alternate(self: QAPISchemaGenGolangVisitor, + name: str, + variants: Optional[QAPISchemaVariants]) -> str: + absent_check_fields = "" + variant_fields = "" + # to avoid having to check accept_null_types + nullable = False + if name in self.accept_null_types: + # In QEMU QAPI schema, only StrOrNull and BlockdevRefOrNull. + nullable = True + marshal_return_default = '''[]byte("{}"), nil''' + marshal_check_fields = ''' + if s.IsNull { + return []byte("null"), nil + } else ''' + unmarshal_check_fields = ''' + // Check for json-null first + if string(data) == "null" { + s.IsNull = true + return nil + } + ''' + else: + marshal_return_default = f'nil, errors.New("{name} has empty fields")' + marshal_check_fields = "" + unmarshal_check_fields = f''' + // Check for json-null first + if string(data) == "null" {{ + return errors.New(`null not supported for {name}`) + }} + ''' + + for var in variants.variants: + var_name, var_type, isptr = qapi_field_to_go_field(var.name, var.type.name) + variant_fields += f"\t{var_name} {isptr}{var_type}\n" + + # Null is special, handled first + if var.type.name == "null": + assert nullable + continue + + if nullable: + absent_check_fields += TEMPLATE_ALTERNATE_NULLABLE_CHECK.format(var_name=var_name)[1:] + marshal_check_fields += TEMPLATE_ALTERNATE_MARSHAL_CHECK.format(var_name=var_name) + unmarshal_check_fields += TEMPLATE_ALTERNATE_UNMARSHAL_CHECK.format(var_name=var_name, + var_type=var_type)[1:] + + content = generate_struct_type(name, variant_fields) + if nullable: + content += TEMPLATE_ALTERNATE_NULLABLE.format(name=name, + absent_check_fields=absent_check_fields) + content += TEMPLATE_ALTERNATE_METHODS.format(name=name, + marshal_check_fields=marshal_check_fields[1:-5], + marshal_return_default=marshal_return_default, + unmarshal_check_fields=unmarshal_check_fields[1:]) + return content + class QAPISchemaGenGolangVisitor(QAPISchemaVisitor): def __init__(self, _: str): super().__init__() - types = ["enum"] + types = ["alternate", "enum", "helper"] self.target = {name: "" for name in types} + self.objects_seen = {} self.schema = None self.golang_package_name = "qapi" + self.accept_null_types = [] def visit_begin(self, schema): self.schema = schema + # We need to be aware of any types that accept JSON NULL + for name, entity in self.schema._entity_dict.items(): + if not isinstance(entity, QAPISchemaAlternateType): + # Assume that only Alternate types accept JSON NULL + continue + + for var in entity.variants.variants: + if var.type.name == 'null': + self.accept_null_types.append(name) + break + # Every Go file needs to reference its package name for target in self.target: self.target[target] = f"package {self.golang_package_name}\n" + self.target["helper"] += TEMPLATE_HELPER + self.target["alternate"] += TEMPLATE_ALTERNATE + def visit_end(self): self.schema = None @@ -88,7 +267,10 @@ def visit_alternate_type(self: QAPISchemaGenGolangVisitor, features: List[QAPISchemaFeature], variants: QAPISchemaVariants ) -> None: - pass + assert name not in self.objects_seen + self.objects_seen[name] = True + + self.target["alternate"] += generate_template_alternate(self, name, variants) def visit_enum_type(self: QAPISchemaGenGolangVisitor, name: str, From patchwork Wed Sep 27 11:25:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Toso X-Patchwork-Id: 1840169 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=hLi5kFz3; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4RwZ8Z6k3Rz1ypJ for ; Wed, 27 Sep 2023 21:27:26 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qlSgC-0002R3-Pu; Wed, 27 Sep 2023 07:26:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qlSg0-0002Ox-0P for qemu-devel@nongnu.org; Wed, 27 Sep 2023 07:25:56 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qlSfy-0001eQ-62 for qemu-devel@nongnu.org; Wed, 27 Sep 2023 07:25:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1695813953; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tdMKxD6s/M2setfWmq7XekuO/g+jsw3BqbxY6Sx7EuM=; b=hLi5kFz3iPK8kC/X39EvxT9K5hsoiUUbpQRoM9WL0WYgTA7nG5QnSN82btfXd+VXIogSMi SS5il0EWn4pqZTlZsOyGYM08sQ8QFLi0Xr65AHNTsiyeLj7yM/KJY0NEFLNBmUM3aGfxOR ZwuFsiEF0fIOPKSajXV4jszFBVA2ZjM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-649-4l1YD71JPT2MXjbBSNKN3A-1; Wed, 27 Sep 2023 07:25:51 -0400 X-MC-Unique: 4l1YD71JPT2MXjbBSNKN3A-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 2330E85A5BA for ; Wed, 27 Sep 2023 11:25:51 +0000 (UTC) Received: from tapioca.lan (unknown [10.45.224.4]) by smtp.corp.redhat.com (Postfix) with ESMTP id 12B851005B96; Wed, 27 Sep 2023 11:25:49 +0000 (UTC) From: Victor Toso To: qemu-devel@nongnu.org Cc: Markus Armbruster , John Snow , =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= Subject: [PATCH v1 3/9] qapi: golang: Generate qapi's struct types in Go Date: Wed, 27 Sep 2023 13:25:38 +0200 Message-ID: <20230927112544.85011-4-victortoso@redhat.com> In-Reply-To: <20230927112544.85011-1-victortoso@redhat.com> References: <20230927112544.85011-1-victortoso@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.129.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org This patch handles QAPI struct types and generates the equivalent types in Go. The following patch adds extra logic when a member of the struct has a Type that can take JSON Null value (e.g: StrOrNull in QEMU) The highlights of this implementation are: 1. Generating an Go struct that requires a @base type, the @base type fields are copied over to the Go struct. The advantage of this approach is to not have embed structs in any of the QAPI types. Note that embedding a @base type is recursive, that is, if the @base type has a @base, all of those fields will be copied over. 2. About the Go struct's fields: i) They can be either by Value or Reference. ii) Every field that is marked as optional in the QAPI specification are translated to Reference fields in its Go structure. This design decision is the most straightforward way to check if a given field was set or not. Exception only for types that can take JSON Null value. iii) Mandatory fields are always by Value with the exception of QAPI arrays, which are handled by Reference (to a block of memory) by Go. iv) All the fields are named with Uppercase due Golang's export convention. v) In order to avoid any kind of issues when encoding or decoding, to or from JSON, we mark all fields with its @name and, when it is optional, member, with @omitempty Example: qapi: | { 'struct': 'BlockdevCreateOptionsFile', | 'data': { 'filename': 'str', | 'size': 'size', | '*preallocation': 'PreallocMode', | '*nocow': 'bool', | '*extent-size-hint': 'size'} } go: | type BlockdevCreateOptionsFile struct { | Filename string `json:"filename"` | Size uint64 `json:"size"` | Preallocation *PreallocMode `json:"preallocation,omitempty"` | Nocow *bool `json:"nocow,omitempty"` | ExtentSizeHint *uint64 `json:"extent-size-hint,omitempty"` | } Signed-off-by: Victor Toso --- scripts/qapi/golang.py | 138 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 136 insertions(+), 2 deletions(-) diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py index 43dbdde14c..1b19e4b232 100644 --- a/scripts/qapi/golang.py +++ b/scripts/qapi/golang.py @@ -117,12 +117,35 @@ def gen_golang(schema: QAPISchema, schema.visit(vis) vis.write(output_dir) +def qapi_name_is_base(name: str) -> bool: + return qapi_name_is_object(name) and name.endswith("-base") + +def qapi_name_is_object(name: str) -> bool: + return name.startswith("q_obj_") + def qapi_to_field_name(name: str) -> str: return name.title().replace("_", "").replace("-", "") def qapi_to_field_name_enum(name: str) -> str: return name.title().replace("-", "") +def qapi_to_go_type_name(name: str) -> str: + if qapi_name_is_object(name): + name = name[6:] + + # We want to keep CamelCase for Golang types. We want to avoid removing + # already set CameCase names while fixing uppercase ones, eg: + # 1) q_obj_SocketAddress_base -> SocketAddressBase + # 2) q_obj_WATCHDOG-arg -> WatchdogArg + words = list(name.replace("_", "-").split("-")) + name = words[0] + if name.islower() or name.isupper(): + name = name.title() + + name += ''.join(word.title() for word in words[1:]) + + return name + def qapi_schema_type_to_go_type(qapitype: str) -> str: schema_types_to_go = { 'str': 'string', 'null': 'nil', 'bool': 'bool', 'number': @@ -156,6 +179,82 @@ def generate_struct_type(type_name, args="") -> str: return f'''{with_type} struct {{{args}}} ''' +def get_struct_field(self: QAPISchemaGenGolangVisitor, + qapi_name: str, + qapi_type_name: str, + is_optional: bool, + is_variant: bool) -> str: + + field = qapi_to_field_name(qapi_name) + member_type = qapi_schema_type_to_go_type(qapi_type_name) + + optional = "" + if is_optional: + if member_type not in self.accept_null_types: + optional = ",omitempty" + + # Use pointer to type when field is optional + isptr = "*" if is_optional and member_type[0] not in "*[" else "" + + fieldtag = '`json:"-"`' if is_variant else f'`json:"{qapi_name}{optional}"`' + return f"\t{field} {isptr}{member_type}{fieldtag}\n" + +def recursive_base(self: QAPISchemaGenGolangVisitor, + base: Optional[QAPISchemaObjectType]) -> str: + fields = "" + + if not base: + return fields + + if base.base is not None: + embed_base = self.schema.lookup_entity(base.base.name) + fields = recursive_base(self, embed_base) + + for member in base.local_members: + if base.variants and base.variants.tag_member.name == member.name: + fields += '''// Discriminator\n''' + + field = get_struct_field(self, member.name, member.type.name, member.optional, False) + fields += field + + if len(fields) > 0: + fields += "\n" + + return fields + +# Helper function that is used for most of QAPI types +def qapi_to_golang_struct(self: QAPISchemaGenGolangVisitor, + name: str, + _: Optional[QAPISourceInfo], + __: QAPISchemaIfCond, + ___: List[QAPISchemaFeature], + base: Optional[QAPISchemaObjectType], + members: List[QAPISchemaObjectTypeMember], + variants: Optional[QAPISchemaVariants]) -> str: + + + fields = recursive_base(self, base) + + if members: + for member in members: + field = get_struct_field(self, member.name, member.type.name, member.optional, False) + fields += field + + fields += "\n" + + if variants: + fields += "\t// Variants fields\n" + for variant in variants.variants: + if variant.type.is_implicit(): + continue + + field = get_struct_field(self, variant.name, variant.type.name, True, True) + fields += field + + type_name = qapi_to_go_type_name(name) + content = generate_struct_type(type_name, fields) + return content + def generate_template_alternate(self: QAPISchemaGenGolangVisitor, name: str, variants: Optional[QAPISchemaVariants]) -> str: @@ -218,7 +317,7 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor): def __init__(self, _: str): super().__init__() - types = ["alternate", "enum", "helper"] + types = ["alternate", "enum", "helper", "struct"] self.target = {name: "" for name in types} self.objects_seen = {} self.schema = None @@ -258,7 +357,42 @@ def visit_object_type(self: QAPISchemaGenGolangVisitor, members: List[QAPISchemaObjectTypeMember], variants: Optional[QAPISchemaVariants] ) -> None: - pass + # Do not handle anything besides struct. + if (name == self.schema.the_empty_object_type.name or + not isinstance(name, str) or + info.defn_meta not in ["struct"]): + return + + # Base structs are embed + if qapi_name_is_base(name): + return + + # Safety checks. + assert name not in self.objects_seen + self.objects_seen[name] = True + + # visit all inner objects as well, they are not going to be + # called by python's generator. + if variants: + for var in variants.variants: + assert isinstance(var.type, QAPISchemaObjectType) + self.visit_object_type(self, + var.type.name, + var.type.info, + var.type.ifcond, + var.type.base, + var.type.local_members, + var.type.variants) + + # Save generated Go code to be written later + self.target[info.defn_meta] += qapi_to_golang_struct(self, + name, + info, + ifcond, + features, + base, + members, + variants) def visit_alternate_type(self: QAPISchemaGenGolangVisitor, name: str, From patchwork Wed Sep 27 11:25:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Toso X-Patchwork-Id: 1840166 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=eSuuKbms; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4RwZ844WXZz1ypJ for ; Wed, 27 Sep 2023 21:27:00 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qlSgB-0002Qk-5H; Wed, 27 Sep 2023 07:26:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qlSg1-0002PD-9G for qemu-devel@nongnu.org; Wed, 27 Sep 2023 07:25:59 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qlSfz-0001ew-4w for qemu-devel@nongnu.org; Wed, 27 Sep 2023 07:25:56 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1695813954; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Sbr3+MleNVs6y//3PACnhJFMQPKbSZzJ3TpKSsTd94Y=; b=eSuuKbmspulg/jUJvwLb+TWYO5jyydyifabxlDzjNKxscgFt5zXXFHceVuz/fWofsbNNSN VH+25MFCSMREVl9FesbV5Gy7+mD206yJOYsvnOR1ASGJtpsi7Ct39OWlyZYsbIlg9gtEgn wCl/gAPP8z2zPl54VO5mC4dL+4nbzzs= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-647-rRyHeFcWOsOmlSMQfWbNUQ-1; Wed, 27 Sep 2023 07:25:52 -0400 X-MC-Unique: rRyHeFcWOsOmlSMQfWbNUQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 875C9280FEC8 for ; Wed, 27 Sep 2023 11:25:52 +0000 (UTC) Received: from tapioca.lan (unknown [10.45.224.4]) by smtp.corp.redhat.com (Postfix) with ESMTP id 76B4210EE6C9; Wed, 27 Sep 2023 11:25:51 +0000 (UTC) From: Victor Toso To: qemu-devel@nongnu.org Cc: Markus Armbruster , John Snow , =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= Subject: [PATCH v1 4/9] qapi: golang: structs: Address 'null' members Date: Wed, 27 Sep 2023 13:25:39 +0200 Message-ID: <20230927112544.85011-5-victortoso@redhat.com> In-Reply-To: <20230927112544.85011-1-victortoso@redhat.com> References: <20230927112544.85011-1-victortoso@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.129.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: 12 X-Spam_score: 1.2 X-Spam_bar: + X-Spam_report: (1.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_SBL_CSS=3.335, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Explaining why this is needed needs some context, so taking the example of StrOrNull alternate type and considering a simplified struct that has two fields: qapi: | { 'struct': 'MigrationExample', | 'data': { '*label': 'StrOrNull', | 'target': 'StrOrNull' } } We have a optional member 'label' which can have three JSON values: 1. A string: { "label": "happy" } 2. A null : { "label": null } 3. Absent : {} The member 'target' is not optional, hence it can't be absent. A Go struct that contains a optional type that can be JSON Null like 'label' in the example above, will need extra care when Marshaling and Unmarshaling from JSON. This patch handles this very specific case: - It implements the Marshaler interface for these structs to properly handle these values. - It adds the interface AbsentAlternate() and implement it for any Alternate that can be JSON Null Signed-off-by: Victor Toso --- scripts/qapi/golang.py | 195 ++++++++++++++++++++++++++++++++++++++--- 1 file changed, 184 insertions(+), 11 deletions(-) diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py index 1b19e4b232..8320af99b6 100644 --- a/scripts/qapi/golang.py +++ b/scripts/qapi/golang.py @@ -110,6 +110,27 @@ }} ''' +TEMPLATE_STRUCT_WITH_NULLABLE_MARSHAL = ''' +func (s {type_name}) MarshalJSON() ([]byte, error) {{ + m := make(map[string]any) + {map_members} + {map_special} + return json.Marshal(&m) +}} + +func (s *{type_name}) UnmarshalJSON(data []byte) error {{ + tmp := {struct}{{}} + + if err := json.Unmarshal(data, &tmp); err != nil {{ + return err + }} + + {set_members} + {set_special} + return nil +}} +''' + def gen_golang(schema: QAPISchema, output_dir: str, prefix: str) -> None: @@ -182,45 +203,187 @@ def generate_struct_type(type_name, args="") -> str: def get_struct_field(self: QAPISchemaGenGolangVisitor, qapi_name: str, qapi_type_name: str, + within_nullable_struct: bool, is_optional: bool, - is_variant: bool) -> str: + is_variant: bool) -> Tuple[str, bool]: field = qapi_to_field_name(qapi_name) member_type = qapi_schema_type_to_go_type(qapi_type_name) + is_nullable = False optional = "" if is_optional: - if member_type not in self.accept_null_types: + if member_type in self.accept_null_types: + is_nullable = True + else: optional = ",omitempty" # Use pointer to type when field is optional isptr = "*" if is_optional and member_type[0] not in "*[" else "" + if within_nullable_struct: + # Within a struct which has a field of type that can hold JSON NULL, + # we have to _not_ use a pointer, otherwise the Marshal methods are + # not called. + isptr = "" if member_type in self.accept_null_types else isptr + fieldtag = '`json:"-"`' if is_variant else f'`json:"{qapi_name}{optional}"`' - return f"\t{field} {isptr}{member_type}{fieldtag}\n" + return f"\t{field} {isptr}{member_type}{fieldtag}\n", is_nullable + +# This helper is used whithin a struct that has members that accept JSON NULL. +def map_and_set(is_nullable: bool, + field: str, + field_is_optional: bool, + name: str) -> Tuple[str, str]: + + mapstr = "" + setstr = "" + if is_nullable: + mapstr = f''' + if val, absent := s.{field}.ToAnyOrAbsent(); !absent {{ + m["{name}"] = val + }} +''' + setstr += f''' + if _, absent := (&tmp.{field}).ToAnyOrAbsent(); !absent {{ + s.{field} = &tmp.{field} + }} +''' + elif field_is_optional: + mapstr = f''' + if s.{field} != nil {{ + m["{name}"] = s.{field} + }} +''' + setstr = f'''\ts.{field} = tmp.{field}\n''' + else: + mapstr = f'''\tm["{name}"] = s.{field}\n''' + setstr = f'''\ts.{field} = tmp.{field}\n''' + + return mapstr, setstr + +def recursive_base_nullable(self: QAPISchemaGenGolangVisitor, + base: Optional[QAPISchemaObjectType]) -> Tuple[str, str, str, str, str]: + fields = "" + map_members = "" + set_members = "" + map_special = "" + set_special = "" + + if not base: + return fields, map_members, set_members, map_special, set_special + + if base.base is not None: + embed_base = self.schema.lookup_entity(base.base.name) + fields, map_members, set_members, map_special, set_special = recursive_base_nullable(self, embed_base) + + for member in base.local_members: + field, _ = get_struct_field(self, member.name, member.type.name, + True, member.optional, False) + fields += field + + member_type = qapi_schema_type_to_go_type(member.type.name) + nullable = member_type in self.accept_null_types + field_name = qapi_to_field_name(member.name) + tomap, toset = map_and_set(nullable, field_name, member.optional, member.name) + if nullable: + map_special += tomap + set_special += toset + else: + map_members += tomap + set_members += toset + + return fields, map_members, set_members, map_special, set_special + +# Helper function. This is executed when the QAPI schema has members +# that could accept JSON NULL (e.g: StrOrNull in QEMU"s QAPI schema). +# This struct will need to be extended with Marshal/Unmarshal methods to +# properly handle such atypical members. +# +# Only the Marshallaing methods are generated but we do need to iterate over +# all the members to properly set/check them in those methods. +def struct_with_nullable_generate_marshal(self: QAPISchemaGenGolangVisitor, + name: str, + base: Optional[QAPISchemaObjectType], + members: List[QAPISchemaObjectTypeMember], + variants: Optional[QAPISchemaVariants]) -> str: + + + fields, map_members, set_members, map_special, set_special = recursive_base_nullable(self, base) + + if members: + for member in members: + field, _ = get_struct_field(self, member.name, member.type.name, + True, member.optional, False) + fields += field + + member_type = qapi_schema_type_to_go_type(member.type.name) + nullable = member_type in self.accept_null_types + tomap, toset = map_and_set(nullable, qapi_to_field_name(member.name), + member.optional, member.name) + if nullable: + map_special += tomap + set_special += toset + else: + map_members += tomap + set_members += toset + + fields += "\n" + + if variants: + for variant in variants.variants: + if variant.type.is_implicit(): + continue + + field, _ = get_struct_field(self, variant.name, variant.type.name, + True, variant.optional, True) + fields += field + + member_type = qapi_schema_type_to_go_type(variant.type.name) + nullable = member_type in self.accept_null_types + tomap, toset = map_and_set(nullable, qapi_to_field_name(variant.name), + variant.optional, variant.name) + if nullable: + map_special += tomap + set_special += toset + else: + map_members += tomap + set_members += toset + + type_name = qapi_to_go_type_name(name) + struct = generate_struct_type("", fields)[:-1] + return TEMPLATE_STRUCT_WITH_NULLABLE_MARSHAL.format(struct=struct, + type_name=type_name, + map_members=map_members, + map_special=map_special, + set_members=set_members, + set_special=set_special) def recursive_base(self: QAPISchemaGenGolangVisitor, - base: Optional[QAPISchemaObjectType]) -> str: + base: Optional[QAPISchemaObjectType]) -> Tuple[str, bool]: fields = "" + with_nullable = False if not base: - return fields + return fields, with_nullable if base.base is not None: embed_base = self.schema.lookup_entity(base.base.name) - fields = recursive_base(self, embed_base) + fields, with_nullable = recursive_base(self, embed_base) for member in base.local_members: if base.variants and base.variants.tag_member.name == member.name: fields += '''// Discriminator\n''' - field = get_struct_field(self, member.name, member.type.name, member.optional, False) + field, nullable = get_struct_field(self, member.name, member.type.name, + False, member.optional, False) fields += field + with_nullable = True if nullable else with_nullable if len(fields) > 0: fields += "\n" - return fields + return fields, with_nullable # Helper function that is used for most of QAPI types def qapi_to_golang_struct(self: QAPISchemaGenGolangVisitor, @@ -233,12 +396,14 @@ def qapi_to_golang_struct(self: QAPISchemaGenGolangVisitor, variants: Optional[QAPISchemaVariants]) -> str: - fields = recursive_base(self, base) + fields, with_nullable = recursive_base(self, base) if members: for member in members: - field = get_struct_field(self, member.name, member.type.name, member.optional, False) + field, nullable = get_struct_field(self, member.name, member.type.name, + False, member.optional, False) fields += field + with_nullable = True if nullable else with_nullable fields += "\n" @@ -248,11 +413,19 @@ def qapi_to_golang_struct(self: QAPISchemaGenGolangVisitor, if variant.type.is_implicit(): continue - field = get_struct_field(self, variant.name, variant.type.name, True, True) + field, nullable = get_struct_field(self, variant.name, variant.type.name, + False, True, True) fields += field + with_nullable = True if nullable else with_nullable type_name = qapi_to_go_type_name(name) content = generate_struct_type(type_name, fields) + if with_nullable: + content += struct_with_nullable_generate_marshal(self, + name, + base, + members, + variants) return content def generate_template_alternate(self: QAPISchemaGenGolangVisitor, From patchwork Wed Sep 27 11:25:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Toso X-Patchwork-Id: 1840173 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=ZDN4QiFq; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4RwZ8x1W9Xz1ypJ for ; Wed, 27 Sep 2023 21:27:45 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qlSgD-0002RM-AR; Wed, 27 Sep 2023 07:26:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qlSg3-0002PN-B0 for qemu-devel@nongnu.org; Wed, 27 Sep 2023 07:25:59 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qlSg1-0001f8-1x for qemu-devel@nongnu.org; Wed, 27 Sep 2023 07:25:58 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1695813955; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wEdD3oUQ1Me33qV1xhUdRwIKVbJ+g3b+cRlu+QgXspI=; b=ZDN4QiFqqNXr4vtSY1sHTVXUBMsPu1w7DyQjZeSYvHhzcLqJ9xJujIzCkxW72CJ5kN/t5/ n1//6XA1M2N1gQBBPDJmzhWLpE321djxk2uLMQWymGpaSS57CIgn1K28411Rd2hViiJPVc tuTM2hBiYz0J2EqXJG42/UwneaJ92mc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-605-UANJskxDNe6zhw59FoYTRw-1; Wed, 27 Sep 2023 07:25:54 -0400 X-MC-Unique: UANJskxDNe6zhw59FoYTRw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id EF3648039C1 for ; Wed, 27 Sep 2023 11:25:53 +0000 (UTC) Received: from tapioca.lan (unknown [10.45.224.4]) by smtp.corp.redhat.com (Postfix) with ESMTP id DABBB10F1BE9; Wed, 27 Sep 2023 11:25:52 +0000 (UTC) From: Victor Toso To: qemu-devel@nongnu.org Cc: Markus Armbruster , John Snow , =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= Subject: [PATCH v1 5/9] qapi: golang: Generate qapi's union types in Go Date: Wed, 27 Sep 2023 13:25:40 +0200 Message-ID: <20230927112544.85011-6-victortoso@redhat.com> In-Reply-To: <20230927112544.85011-1-victortoso@redhat.com> References: <20230927112544.85011-1-victortoso@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.129.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org This patch handles QAPI union types and generates the equivalent data structures and methods in Go to handle it. The QAPI union type has two types of fields: The @base and the @Variants members. The @base fields can be considered common members for the union while only one field maximum is set for the @Variants. In the QAPI specification, it defines a @discriminator field, which is an Enum type. The purpose of the @discriminator is to identify which @variant type is being used. Not that @discriminator's enum might have more values than the union's data struct. This is fine. The union does not need to handle all cases of the enum, but it should accept them without error. For this specific case, we keep the @discriminator field in every union type. The union types implement the Marshaler and Unmarshaler interfaces to seamless decode from JSON objects to Golang structs and vice versa. qapi: | { 'union': 'SetPasswordOptions', | 'base': { 'protocol': 'DisplayProtocol', | 'password': 'str', | '*connected': 'SetPasswordAction' }, | 'discriminator': 'protocol', | 'data': { 'vnc': 'SetPasswordOptionsVnc' } } go: | type SetPasswordOptions struct { | Protocol DisplayProtocol `json:"protocol"` | Password string `json:"password"` | Connected *SetPasswordAction `json:"connected,omitempty"` | | // Variants fields | Vnc *SetPasswordOptionsVnc `json:"-"` | } Signed-off-by: Victor Toso --- scripts/qapi/golang.py | 170 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 167 insertions(+), 3 deletions(-) diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py index 8320af99b6..343c9c9b95 100644 --- a/scripts/qapi/golang.py +++ b/scripts/qapi/golang.py @@ -52,6 +52,17 @@ } return nil } + +// This helper is used to move struct's fields into a map. +// This function is useful to merge JSON objects. +func unwrapToMap(m map[string]any, data any) error { + if bytes, err := json.Marshal(&data); err != nil { + return fmt.Errorf("unwrapToMap: %s", err) + } else if err := json.Unmarshal(bytes, &m); err != nil { + return fmt.Errorf("unwrapToMap: %s, data=%s", err, string(bytes)) + } + return nil +} ''' TEMPLATE_ALTERNATE = ''' @@ -131,6 +142,62 @@ }} ''' +TEMPLATE_UNION_CHECK_FIELD = ''' +if s.{field} != nil && err == nil {{ + if len(bytes) != 0 {{ + err = errors.New(`multiple variant fields set`) + }} else if err = unwrapToMap(m, s.{field}); err == nil {{ + s.{discriminator} = {go_enum_value} + m["{member_name}"] = {go_enum_value} + bytes, err = json.Marshal(m) + }} +}} +''' + +TEMPLATE_UNION_DRIVER_CASE = ''' +case {go_enum_value}: + s.{field} = new({member_type}) + if err := json.Unmarshal(data, s.{field}); err != nil {{ + s.{field} = nil + return err + }}''' + +TEMPLATE_UNION_METHODS = ''' +func (s {type_name}) MarshalJSON() ([]byte, error) {{ + var bytes []byte + var err error + type Alias {type_name} + v := Alias(s) + m := make(map[string]any) + unwrapToMap(m, &v) + {check_fields} + {check_non_fields_marshal} + if err != nil {{ + return nil, fmt.Errorf("error: marshal: {type_name}: reason='%s', struct='%+v'", err, s) + }} else if len(bytes) == 0 {{ + return nil, fmt.Errorf("error: marshal: {type_name} unsupported, struct='%+v'", s) + }} + return bytes, nil +}} + +func (s *{type_name}) UnmarshalJSON(data []byte) error {{{base_type_def} + tmp := struct {{ + {base_type_name} + }}{{}} + + if err := json.Unmarshal(data, &tmp); err != nil {{ + return err + }} + {base_type_assign_unmarshal} + switch tmp.{discriminator} {{ + {driver_cases} + {check_non_fields_unmarshal} + }} + return nil +}} +''' + + def gen_golang(schema: QAPISchema, output_dir: str, prefix: str) -> None: @@ -428,6 +495,98 @@ def qapi_to_golang_struct(self: QAPISchemaGenGolangVisitor, variants) return content +def qapi_to_golang_methods_union(self: QAPISchemaGenGolangVisitor, + name: str, + base: Optional[QAPISchemaObjectType], + variants: Optional[QAPISchemaVariants] + ) -> str: + + type_name = qapi_to_go_type_name(name) + + assert base + base_type_assign_unmarshal = "" + base_type_name = qapi_to_go_type_name(base.name) + base_type_def = qapi_to_golang_struct(self, + base.name, + base.info, + base.ifcond, + base.features, + base.base, + base.members, + base.variants) + for member in base.local_members: + field = qapi_to_field_name(member.name) + base_type_assign_unmarshal += f'''s.{field} = tmp.{field} +''' + + driver_cases = "" + check_fields = "" + exists = {} + enum_name = variants.tag_member.type.name + discriminator = qapi_to_field_name(variants.tag_member.name) + if variants: + for var in variants.variants: + if var.type.is_implicit(): + continue + + field = qapi_to_field_name(var.name) + enum_value = qapi_to_field_name_enum(var.name) + member_type = qapi_schema_type_to_go_type(var.type.name) + go_enum_value = f'''{enum_name}{enum_value}''' + exists[go_enum_value] = True + + check_fields += TEMPLATE_UNION_CHECK_FIELD.format(field=field, + discriminator=discriminator, + member_name=variants.tag_member.name, + go_enum_value=go_enum_value) + driver_cases += TEMPLATE_UNION_DRIVER_CASE.format(go_enum_value=go_enum_value, + field=field, + member_type=member_type) + + check_non_fields_marshal = "" + check_non_fields_unmarshal = "" + enum_obj = self.schema.lookup_entity(enum_name) + if len(exists) != len(enum_obj.members): + driver_cases += '''\ndefault:''' + check_non_fields_marshal = ''' + // Check for valid values without field members + if len(bytes) == 0 && err == nil && + (''' + check_non_fields_unmarshal = ''' + // Check for valid values without field members + if ''' + + for member in enum_obj.members: + value = qapi_to_field_name_enum(member.name) + go_enum_value = f'''{enum_name}{value}''' + + if go_enum_value in exists: + continue + + check_non_fields_marshal += f'''s.{discriminator} == {go_enum_value} ||\n''' + check_non_fields_unmarshal += f'''tmp.{discriminator} != {go_enum_value} &&\n''' + + check_non_fields_marshal = f'''{check_non_fields_marshal[:-3]}) {{ + type Alias {type_name} + bytes, err = json.Marshal(Alias(s)) + }} +''' + check_non_fields_unmarshal = f'''{check_non_fields_unmarshal[1:-3]} {{ + return fmt.Errorf("error: unmarshal: {type_name}: received unrecognized value: '%s'", + tmp.{discriminator}) + }} +''' + + return TEMPLATE_UNION_METHODS.format(type_name=type_name, + check_fields=check_fields, + check_non_fields_marshal=check_non_fields_marshal, + base_type_def=base_type_def, + base_type_name=base_type_name, + base_type_assign_unmarshal=base_type_assign_unmarshal, + discriminator=discriminator, + driver_cases=driver_cases[1:], + check_non_fields_unmarshal=check_non_fields_unmarshal) + def generate_template_alternate(self: QAPISchemaGenGolangVisitor, name: str, variants: Optional[QAPISchemaVariants]) -> str: @@ -490,7 +649,7 @@ class QAPISchemaGenGolangVisitor(QAPISchemaVisitor): def __init__(self, _: str): super().__init__() - types = ["alternate", "enum", "helper", "struct"] + types = ["alternate", "enum", "helper", "struct", "union"] self.target = {name: "" for name in types} self.objects_seen = {} self.schema = None @@ -530,10 +689,10 @@ def visit_object_type(self: QAPISchemaGenGolangVisitor, members: List[QAPISchemaObjectTypeMember], variants: Optional[QAPISchemaVariants] ) -> None: - # Do not handle anything besides struct. + # Do not handle anything besides struct and unions. if (name == self.schema.the_empty_object_type.name or not isinstance(name, str) or - info.defn_meta not in ["struct"]): + info.defn_meta not in ["struct", "union"]): return # Base structs are embed @@ -566,6 +725,11 @@ def visit_object_type(self: QAPISchemaGenGolangVisitor, base, members, variants) + if info.defn_meta == "union": + self.target[info.defn_meta] += qapi_to_golang_methods_union(self, + name, + base, + variants) def visit_alternate_type(self: QAPISchemaGenGolangVisitor, name: str, From patchwork Wed Sep 27 11:25:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Toso X-Patchwork-Id: 1840174 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=LSCeHtDr; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4RwZ9B4Gh5z1ypJ for ; Wed, 27 Sep 2023 21:27:58 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qlSgG-0002Rg-0w; Wed, 27 Sep 2023 07:26:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qlSg4-0002QD-TN for qemu-devel@nongnu.org; Wed, 27 Sep 2023 07:26:01 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qlSg3-0001fS-4L for qemu-devel@nongnu.org; Wed, 27 Sep 2023 07:26:00 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1695813957; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ghb3L7rr25oPc3vl9o1Af/irNvvAXlxz5WJK2LcKuoY=; b=LSCeHtDr/V1o0wX0N6P8ejQNQWYDYoaZUJ7EDM5fddLAgQxGhGdvJFJp+frZrtr6K+qqtl S3A+jWbrE7peSeoYMG2xCLo6mMTJB+btEYRY3eZirE0F/373oqlYfqaOBbEMUkgIvEDp1D fc8o9cqtdGyLjA1X/9sVJAl5PcSnoAA= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-639-TdczFD_wO4Ce2dQN4Xi5hQ-1; Wed, 27 Sep 2023 07:25:55 -0400 X-MC-Unique: TdczFD_wO4Ce2dQN4Xi5hQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6003A800B35 for ; Wed, 27 Sep 2023 11:25:55 +0000 (UTC) Received: from tapioca.lan (unknown [10.45.224.4]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4FB8F10D14C7; Wed, 27 Sep 2023 11:25:54 +0000 (UTC) From: Victor Toso To: qemu-devel@nongnu.org Cc: Markus Armbruster , John Snow , =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= Subject: [PATCH v1 6/9] qapi: golang: Generate qapi's event types in Go Date: Wed, 27 Sep 2023 13:25:41 +0200 Message-ID: <20230927112544.85011-7-victortoso@redhat.com> In-Reply-To: <20230927112544.85011-1-victortoso@redhat.com> References: <20230927112544.85011-1-victortoso@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.129.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org This patch handles QAPI event types and generates data structures in Go that handles it. We also define a Event interface and two helper functions MarshalEvent and UnmarshalEvent. Example: qapi: | { 'event': 'MEMORY_DEVICE_SIZE_CHANGE', | 'data': { '*id': 'str', 'size': 'size', 'qom-path' : 'str'} } go: | type MemoryDeviceSizeChangeEvent struct { | MessageTimestamp Timestamp `json:"-"` | Id *string `json:"id,omitempty"` | Size uint64 `json:"size"` | QomPath string `json:"qom-path"` | } usage: | input := `{"event":"MEMORY_DEVICE_SIZE_CHANGE",` + | `"timestamp":{"seconds":1588168529,"microseconds":201316},` + | `"data":{"id":"vm0","size":1073741824,"qom-path":"/machine/unattached/device[2]"}}` | e, err := UnmarshalEvent([]byte(input) | if err != nil { | panic(err) | } | if e.GetName() == `MEMORY_DEVICE_SIZE_CHANGE` { | m := e.(*MemoryDeviceSizeChangeEvent) | // m.QomPath == "/machine/unattached/device[2]" | } Signed-off-by: Victor Toso --- scripts/qapi/golang.py | 105 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 100 insertions(+), 5 deletions(-) diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py index 343c9c9b95..ff3b1dd020 100644 --- a/scripts/qapi/golang.py +++ b/scripts/qapi/golang.py @@ -197,6 +197,55 @@ }} ''' +TEMPLATE_EVENT = ''' +type Timestamp struct {{ + Seconds int64 `json:"seconds"` + Microseconds int64 `json:"microseconds"` +}} + +type Event interface {{ + GetName() string + GetTimestamp() Timestamp +}} + +func MarshalEvent(e Event) ([]byte, error) {{ + m := make(map[string]any) + m["event"] = e.GetName() + m["timestamp"] = e.GetTimestamp() + if bytes, err := json.Marshal(e); err != nil {{ + return []byte{{}}, err + }} else if len(bytes) > 2 {{ + m["data"] = e + }} + return json.Marshal(m) +}} + +func UnmarshalEvent(data []byte) (Event, error) {{ + base := struct {{ + Name string `json:"event"` + MessageTimestamp Timestamp `json:"timestamp"` + }}{{}} + if err := json.Unmarshal(data, &base); err != nil {{ + return nil, fmt.Errorf("Failed to decode event: %s", string(data)) + }} + + switch base.Name {{ + {cases} + }} + return nil, errors.New("Failed to recognize event") +}} +''' + +TEMPLATE_EVENT_METHODS = ''' +func (s *{type_name}) GetName() string {{ + return "{name}" +}} + +func (s *{type_name}) GetTimestamp() Timestamp {{ + return s.MessageTimestamp +}} +''' + def gen_golang(schema: QAPISchema, output_dir: str, @@ -217,7 +266,8 @@ def qapi_to_field_name(name: str) -> str: def qapi_to_field_name_enum(name: str) -> str: return name.title().replace("-", "") -def qapi_to_go_type_name(name: str) -> str: +def qapi_to_go_type_name(name: str, + meta: Optional[str] = None) -> str: if qapi_name_is_object(name): name = name[6:] @@ -232,6 +282,11 @@ def qapi_to_go_type_name(name: str) -> str: name += ''.join(word.title() for word in words[1:]) + types = ["event"] + if meta in types: + name = name[:-3] if name.endswith("Arg") else name + name += meta.title().replace(" ", "") + return name def qapi_schema_type_to_go_type(qapitype: str) -> str: @@ -455,7 +510,7 @@ def recursive_base(self: QAPISchemaGenGolangVisitor, # Helper function that is used for most of QAPI types def qapi_to_golang_struct(self: QAPISchemaGenGolangVisitor, name: str, - _: Optional[QAPISourceInfo], + info: Optional[QAPISourceInfo], __: QAPISchemaIfCond, ___: List[QAPISchemaFeature], base: Optional[QAPISchemaObjectType], @@ -464,6 +519,8 @@ def qapi_to_golang_struct(self: QAPISchemaGenGolangVisitor, fields, with_nullable = recursive_base(self, base) + if info.defn_meta == "event": + fields += f'''\tMessageTimestamp Timestamp `json:"-"`\n{fields}''' if members: for member in members: @@ -485,7 +542,7 @@ def qapi_to_golang_struct(self: QAPISchemaGenGolangVisitor, fields += field with_nullable = True if nullable else with_nullable - type_name = qapi_to_go_type_name(name) + type_name = qapi_to_go_type_name(name, info.defn_meta) content = generate_struct_type(type_name, fields) if with_nullable: content += struct_with_nullable_generate_marshal(self, @@ -644,15 +701,34 @@ def generate_template_alternate(self: QAPISchemaGenGolangVisitor, unmarshal_check_fields=unmarshal_check_fields[1:]) return content +def generate_template_event(events: dict[str, str]) -> str: + cases = "" + for name in sorted(events): + case_type = events[name] + cases += f''' +case "{name}": + event := struct {{ + Data {case_type} `json:"data"` + }}{{}} + + if err := json.Unmarshal(data, &event); err != nil {{ + return nil, fmt.Errorf("Failed to unmarshal: %s", string(data)) + }} + event.Data.MessageTimestamp = base.MessageTimestamp + return &event.Data, nil +''' + return TEMPLATE_EVENT.format(cases=cases) + class QAPISchemaGenGolangVisitor(QAPISchemaVisitor): def __init__(self, _: str): super().__init__() - types = ["alternate", "enum", "helper", "struct", "union"] + types = ["alternate", "enum", "event", "helper", "struct", "union"] self.target = {name: "" for name in types} self.objects_seen = {} self.schema = None + self.events = {} self.golang_package_name = "qapi" self.accept_null_types = [] @@ -679,6 +755,7 @@ def visit_begin(self, schema): def visit_end(self): self.schema = None + self.target["event"] += generate_template_event(self.events) def visit_object_type(self: QAPISchemaGenGolangVisitor, name: str, @@ -779,7 +856,25 @@ def visit_command(self, pass def visit_event(self, name, info, ifcond, features, arg_type, boxed): - pass + assert name == info.defn_name + type_name = qapi_to_go_type_name(name, info.defn_meta) + self.events[name] = type_name + + if isinstance(arg_type, QAPISchemaObjectType): + content = qapi_to_golang_struct(self, + name, + arg_type.info, + arg_type.ifcond, + arg_type.features, + arg_type.base, + arg_type.members, + arg_type.variants) + else: + args = '''MessageTimestamp Timestamp `json:"-"`''' + content = generate_struct_type(type_name, args) + + content += TEMPLATE_EVENT_METHODS.format(name=name, type_name=type_name) + self.target["event"] += content def write(self, output_dir: str) -> None: for module_name, content in self.target.items(): From patchwork Wed Sep 27 11:25:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Toso X-Patchwork-Id: 1840165 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=R3n7FTYV; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4RwZ7t6tbXz1ypJ for ; Wed, 27 Sep 2023 21:26:50 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qlSgF-0002Re-JC; Wed, 27 Sep 2023 07:26:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qlSg6-0002QE-2s for qemu-devel@nongnu.org; Wed, 27 Sep 2023 07:26:03 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qlSg3-0001fZ-PZ for qemu-devel@nongnu.org; Wed, 27 Sep 2023 07:26:01 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1695813958; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=U+4HWefqvo2pq/E4rpUkNcKbx9fyX9eJtstDDX9TmnE=; b=R3n7FTYVKieFFLpBlYi/wSr1uvdmITXUhtiS22aaZxcmRCY0kN1PVVObDdRrpZpCbNzKWJ y9nkqyNpiGHqm27mx7jZcplbA9Q1FYTcGEueyphAIPVyzdmn7L/+jKx1pd92vm5RhzHqaM 9M/vZ/JO4P/bdbOCestN/UDCbEruV8k= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-62-L4yw9WtLNTiltEwaFFgTjw-1; Wed, 27 Sep 2023 07:25:57 -0400 X-MC-Unique: L4yw9WtLNTiltEwaFFgTjw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CAAFC3821347 for ; Wed, 27 Sep 2023 11:25:56 +0000 (UTC) Received: from tapioca.lan (unknown [10.45.224.4]) by smtp.corp.redhat.com (Postfix) with ESMTP id B19271005B96; Wed, 27 Sep 2023 11:25:55 +0000 (UTC) From: Victor Toso To: qemu-devel@nongnu.org Cc: Markus Armbruster , John Snow , =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= Subject: [PATCH v1 7/9] qapi: golang: Generate qapi's command types in Go Date: Wed, 27 Sep 2023 13:25:42 +0200 Message-ID: <20230927112544.85011-8-victortoso@redhat.com> In-Reply-To: <20230927112544.85011-1-victortoso@redhat.com> References: <20230927112544.85011-1-victortoso@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.133.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: 12 X-Spam_score: 1.2 X-Spam_bar: + X-Spam_report: (1.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_SBL_CSS=3.335, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org This patch handles QAPI command types and generates data structures in Go that decodes from QMP JSON Object to Go data structure and vice versa. Similar to Event, this patch adds a Command interface and two helper functions MarshalCommand and UnmarshalCommand. Example: qapi: | { 'command': 'set_password', | 'boxed': true, | 'data': 'SetPasswordOptions' } go: | type SetPasswordCommand struct { | SetPasswordOptions | CommandId string `json:"-"` | } usage: | input := `{"execute":"set_password",` + | `"arguments":{"protocol":"vnc",` + | `"password":"secret"}}` | | c, err := UnmarshalCommand([]byte(input)) | if err != nil { | panic(err) | } | | if c.GetName() == `set_password` { | m := c.(*SetPasswordCommand) | // m.Password == "secret" | } Signed-off-by: Victor Toso --- scripts/qapi/golang.py | 97 ++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 94 insertions(+), 3 deletions(-) diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py index ff3b1dd020..52a9124641 100644 --- a/scripts/qapi/golang.py +++ b/scripts/qapi/golang.py @@ -246,6 +246,51 @@ }} ''' +TEMPLATE_COMMAND_METHODS = ''' +func (c *{type_name}) GetName() string {{ + return "{name}" +}} + +func (s *{type_name}) GetId() string {{ + return s.MessageId +}} +''' + +TEMPLATE_COMMAND = ''' +type Command interface {{ + GetId() string + GetName() string +}} + +func MarshalCommand(c Command) ([]byte, error) {{ + m := make(map[string]any) + m["execute"] = c.GetName() + if id := c.GetId(); len(id) > 0 {{ + m["id"] = id + }} + if bytes, err := json.Marshal(c); err != nil {{ + return []byte{{}}, err + }} else if len(bytes) > 2 {{ + m["arguments"] = c + }} + return json.Marshal(m) +}} + +func UnmarshalCommand(data []byte) (Command, error) {{ + base := struct {{ + MessageId string `json:"id,omitempty"` + Name string `json:"execute"` + }}{{}} + if err := json.Unmarshal(data, &base); err != nil {{ + return nil, fmt.Errorf("Failed to decode command: %s", string(data)) + }} + + switch base.Name {{ + {cases} + }} + return nil, errors.New("Failed to recognize command") +}} +''' def gen_golang(schema: QAPISchema, output_dir: str, @@ -282,7 +327,7 @@ def qapi_to_go_type_name(name: str, name += ''.join(word.title() for word in words[1:]) - types = ["event"] + types = ["event", "command"] if meta in types: name = name[:-3] if name.endswith("Arg") else name name += meta.title().replace(" ", "") @@ -521,6 +566,8 @@ def qapi_to_golang_struct(self: QAPISchemaGenGolangVisitor, fields, with_nullable = recursive_base(self, base) if info.defn_meta == "event": fields += f'''\tMessageTimestamp Timestamp `json:"-"`\n{fields}''' + elif info.defn_meta == "command": + fields += f'''\tMessageId string `json:"-"`\n{fields}''' if members: for member in members: @@ -719,16 +766,36 @@ def generate_template_event(events: dict[str, str]) -> str: ''' return TEMPLATE_EVENT.format(cases=cases) +def generate_template_command(commands: dict[str, str]) -> str: + cases = "" + for name in sorted(commands): + case_type = commands[name] + cases += f''' +case "{name}": + command := struct {{ + Args {case_type} `json:"arguments"` + }}{{}} + + if err := json.Unmarshal(data, &command); err != nil {{ + return nil, fmt.Errorf("Failed to unmarshal: %s", string(data)) + }} + command.Args.MessageId = base.MessageId + return &command.Args, nil +''' + content = TEMPLATE_COMMAND.format(cases=cases) + return content + class QAPISchemaGenGolangVisitor(QAPISchemaVisitor): def __init__(self, _: str): super().__init__() - types = ["alternate", "enum", "event", "helper", "struct", "union"] + types = ["alternate", "command", "enum", "event", "helper", "struct", "union"] self.target = {name: "" for name in types} self.objects_seen = {} self.schema = None self.events = {} + self.commands = {} self.golang_package_name = "qapi" self.accept_null_types = [] @@ -756,6 +823,7 @@ def visit_begin(self, schema): def visit_end(self): self.schema = None self.target["event"] += generate_template_event(self.events) + self.target["command"] += generate_template_command(self.commands) def visit_object_type(self: QAPISchemaGenGolangVisitor, name: str, @@ -853,7 +921,30 @@ def visit_command(self, allow_oob: bool, allow_preconfig: bool, coroutine: bool) -> None: - pass + assert name == info.defn_name + + type_name = qapi_to_go_type_name(name, info.defn_meta) + self.commands[name] = type_name + + content = "" + if boxed or not arg_type or not qapi_name_is_object(arg_type.name): + args = "" if not arg_type else "\n" + arg_type.name + args += '''\n\tMessageId string `json:"-"`''' + content = generate_struct_type(type_name, args) + else: + assert isinstance(arg_type, QAPISchemaObjectType) + content = qapi_to_golang_struct(self, + name, + arg_type.info, + arg_type.ifcond, + arg_type.features, + arg_type.base, + arg_type.members, + arg_type.variants) + + content += TEMPLATE_COMMAND_METHODS.format(name=name, + type_name=type_name) + self.target["command"] += content def visit_event(self, name, info, ifcond, features, arg_type, boxed): assert name == info.defn_name From patchwork Wed Sep 27 11:25:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Toso X-Patchwork-Id: 1840172 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=V0ZCdyDn; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4RwZ8s5qtVz1ypS for ; Wed, 27 Sep 2023 21:27:41 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qlSgH-0002So-1s; Wed, 27 Sep 2023 07:26:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qlSgA-0002Qi-6y for qemu-devel@nongnu.org; Wed, 27 Sep 2023 07:26:07 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qlSg6-0001g8-49 for qemu-devel@nongnu.org; Wed, 27 Sep 2023 07:26:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1695813961; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0fzVBihPpVNV5cVfGMBl+fBxOh7yT7S6KuXegwY+jes=; b=V0ZCdyDnMqVA6UMC6c+9YBvZA9LdRFf4dj8upTCSlOrihzhtjHTNK37wwolPaVTSpuvzQp LKkKXLH7QVIpZCkiwWFCXSw2+5ZkTaBEebTptg45n87q8sW2eF8V+G2sB5Y+gMuBrNT4lo Db3qwZRAWF8DAubOd05AlzPibyhcBwc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-691-4WOMpWIRMjue2PCn6pjKxg-1; Wed, 27 Sep 2023 07:25:58 -0400 X-MC-Unique: 4WOMpWIRMjue2PCn6pjKxg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4141A101A53B for ; Wed, 27 Sep 2023 11:25:58 +0000 (UTC) Received: from tapioca.lan (unknown [10.45.224.4]) by smtp.corp.redhat.com (Postfix) with ESMTP id 28C2110EE402; Wed, 27 Sep 2023 11:25:56 +0000 (UTC) From: Victor Toso To: qemu-devel@nongnu.org Cc: Markus Armbruster , John Snow , =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= Subject: [PATCH v1 8/9] qapi: golang: Add CommandResult type to Go Date: Wed, 27 Sep 2023 13:25:43 +0200 Message-ID: <20230927112544.85011-9-victortoso@redhat.com> In-Reply-To: <20230927112544.85011-1-victortoso@redhat.com> References: <20230927112544.85011-1-victortoso@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.129.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org This patch adds a struct type in Go that will handle return values for QAPI's command types. The return value of a Command is, encouraged to be, QAPI's complex types or an Array of those. Every Command has a underlying CommandResult. The EmptyCommandReturn is for those that don't expect any data e.g: `{ "return": {} }`. All CommandReturn types implement the CommandResult interface. Example: qapi: | { 'command': 'query-sev', 'returns': 'SevInfo', | 'if': 'TARGET_I386' } go: | type QuerySevCommandReturn struct { | CommandId string `json:"id,omitempty"` | Result *SevInfo `json:"return"` | Error *QapiError `json:"error,omitempty"` | } usage: | // One can use QuerySevCommandReturn directly or | // command's interface GetReturnType() instead. | | input := `{ "return": { "enabled": true, "api-major" : 0,` + | `"api-minor" : 0, "build-id" : 0,` + | `"policy" : 0, "state" : "running",` + | `"handle" : 1 } } ` | | ret := QuerySevCommandReturn{} | err := json.Unmarshal([]byte(input), &ret) | if ret.Error != nil { | // Handle command failure {"error": { ...}} | } else if ret.Result != nil { | // ret.Result.Enable == true | } Signed-off-by: Victor Toso --- scripts/qapi/golang.py | 72 ++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 70 insertions(+), 2 deletions(-) diff --git a/scripts/qapi/golang.py b/scripts/qapi/golang.py index 52a9124641..48ca0deab0 100644 --- a/scripts/qapi/golang.py +++ b/scripts/qapi/golang.py @@ -40,6 +40,15 @@ ''' TEMPLATE_HELPER = ''' +type QapiError struct { + Class string `json:"class"` + Description string `json:"desc"` +} + +func (err *QapiError) Error() string { + return fmt.Sprintf("%s: %s", err.Class, err.Description) +} + // Creates a decoder that errors on unknown Fields // Returns nil if successfully decoded @from payload to @into type // Returns error if failed to decode @from payload to @into type @@ -254,12 +263,17 @@ func (s *{type_name}) GetId() string {{ return s.MessageId }} + +func (s *{type_name}) GetReturnType() CommandReturn {{ + return &{cmd_ret_name}{{}} +}} ''' TEMPLATE_COMMAND = ''' type Command interface {{ GetId() string GetName() string + GetReturnType() CommandReturn }} func MarshalCommand(c Command) ([]byte, error) {{ @@ -292,6 +306,45 @@ }} ''' +TEMPLATE_COMMAND_RETURN = ''' +type CommandReturn interface { + GetId() string + GetCommandName() string + GetError() error +} +''' + +TEMPLATE_COMMAND_RETURN_METHODS = ''' +type {cmd_ret_name} struct {{ + MessageId string `json:"id,omitempty"` + Error *QapiError `json:"error,omitempty"` +{result} +}} + +func (r *{cmd_ret_name}) GetCommandName() string {{ + return "{name}" +}} + +func (r *{cmd_ret_name}) GetId() string {{ + return r.MessageId +}} + +func (r *{cmd_ret_name}) GetError() error {{ + return r.Error +}} + +{marshal_empty} +''' + +TEMPLATE_COMMAND_RETURN_MARSHAL_EMPTY = ''' +func (r {cmd_ret_name}) MarshalJSON() ([]byte, error) {{ + if r.Error != nil {{ + type Alias {cmd_ret_name} + return json.Marshal(Alias(r)) + }} + return []byte(`{{"return":{{}}}}`), nil +}}''' + def gen_golang(schema: QAPISchema, output_dir: str, prefix: str) -> None: @@ -327,7 +380,7 @@ def qapi_to_go_type_name(name: str, name += ''.join(word.title() for word in words[1:]) - types = ["event", "command"] + types = ["event", "command", "command return"] if meta in types: name = name[:-3] if name.endswith("Arg") else name name += meta.title().replace(" ", "") @@ -783,6 +836,7 @@ def generate_template_command(commands: dict[str, str]) -> str: return &command.Args, nil ''' content = TEMPLATE_COMMAND.format(cases=cases) + content += TEMPLATE_COMMAND_RETURN return content @@ -926,6 +980,15 @@ def visit_command(self, type_name = qapi_to_go_type_name(name, info.defn_meta) self.commands[name] = type_name + cmd_ret_name = qapi_to_go_type_name(name, "command return") + marshal_empty = TEMPLATE_COMMAND_RETURN_MARSHAL_EMPTY.format(cmd_ret_name=cmd_ret_name) + result = "" + if ret_type: + marshal_empty = "" + ret_type_name = qapi_schema_type_to_go_type(ret_type.name) + isptr = "*" if ret_type_name[0] not in "*[" else "" + result = f'''Result {isptr}{ret_type_name} `json:"return"`''' + content = "" if boxed or not arg_type or not qapi_name_is_object(arg_type.name): args = "" if not arg_type else "\n" + arg_type.name @@ -943,7 +1006,12 @@ def visit_command(self, arg_type.variants) content += TEMPLATE_COMMAND_METHODS.format(name=name, - type_name=type_name) + type_name=type_name, + cmd_ret_name=cmd_ret_name) + content += TEMPLATE_COMMAND_RETURN_METHODS.format(name=name, + cmd_ret_name=cmd_ret_name, + result=result, + marshal_empty=marshal_empty) self.target["command"] += content def visit_event(self, name, info, ifcond, features, arg_type, boxed): From patchwork Wed Sep 27 11:25:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victor Toso X-Patchwork-Id: 1840167 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@legolas.ozlabs.org Authentication-Results: legolas.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=WZbgb6c4; dkim-atps=neutral Authentication-Results: legolas.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=nongnu.org (client-ip=209.51.188.17; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=patchwork.ozlabs.org) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-ECDSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by legolas.ozlabs.org (Postfix) with ESMTPS id 4RwZ895dsXz1ypJ for ; Wed, 27 Sep 2023 21:27:05 +1000 (AEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qlSgH-0002Sr-3V; Wed, 27 Sep 2023 07:26:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qlSgB-0002Qs-A5 for qemu-devel@nongnu.org; Wed, 27 Sep 2023 07:26:07 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qlSg6-0001gE-7o for qemu-devel@nongnu.org; Wed, 27 Sep 2023 07:26:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1695813961; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7GwKpX3stjEyNOpO5fFkOHehLuyvDb76vCGkP1116b0=; b=WZbgb6c4TQwsHLjA0aXkTkLWkZUtoo8zR9VkW1HXbHwa7r3EOi1KvjmQEmzGoCIYCDc3oD npQZhuvGCc4vsl4dskbUlU9mKpUK0zl6TKxG7keQImu2RLSIRASZ1OHKHy6MXiFYm/+ccR NQV1cdWnpk5RwrLYD/C4t5BBgsgoEUI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-33-BS2kd1h7PM2YbmN2j8tSLQ-1; Wed, 27 Sep 2023 07:25:59 -0400 X-MC-Unique: BS2kd1h7PM2YbmN2j8tSLQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A30A9811E7B for ; Wed, 27 Sep 2023 11:25:59 +0000 (UTC) Received: from tapioca.lan (unknown [10.45.224.4]) by smtp.corp.redhat.com (Postfix) with ESMTP id 919841054FC2; Wed, 27 Sep 2023 11:25:58 +0000 (UTC) From: Victor Toso To: qemu-devel@nongnu.org Cc: Markus Armbruster , John Snow , =?utf-8?q?Daniel_P_=2E_Berrang=C3=A9?= Subject: [PATCH v1 9/9] docs: add notes on Golang code generator Date: Wed, 27 Sep 2023 13:25:44 +0200 Message-ID: <20230927112544.85011-10-victortoso@redhat.com> In-Reply-To: <20230927112544.85011-1-victortoso@redhat.com> References: <20230927112544.85011-1-victortoso@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 Received-SPF: pass client-ip=170.10.129.124; envelope-from=victortoso@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org The goal of this patch is converge discussions into a documentation, to make it easy and explicit design decisions, known issues and what else might help a person interested in how the Go module is generated. Signed-off-by: Victor Toso --- docs/devel/qapi-golang-code-gen.rst | 341 ++++++++++++++++++++++++++++ 1 file changed, 341 insertions(+) create mode 100644 docs/devel/qapi-golang-code-gen.rst diff --git a/docs/devel/qapi-golang-code-gen.rst b/docs/devel/qapi-golang-code-gen.rst new file mode 100644 index 0000000000..2a91f8fc60 --- /dev/null +++ b/docs/devel/qapi-golang-code-gen.rst @@ -0,0 +1,341 @@ +========================== +QAPI Golang code generator +========================== + +.. + Copyright (C) 2023 Red Hat, Inc. + + This work is licensed under the terms of the GNU GPL, version 2 or + later. See the COPYING file in the top-level directory. + + +Introduction +============ + +This document provides information of how the generated Go code maps +with the QAPI specification, clarifying design decisions when needed. + + +Scope of the generated Go code +============================== + +The scope is limited to data structures that can interpret and be used +to generate valid QMP messages. These data structures are generated +from a QAPI schema and should be able to handle QMP messages from the +same schema. + +The generated Go code is a Go module with data structs that uses Go +standard library `encoding/json`, implementing its field tags and +Marshal interface whenever needed. + + +QAPI types to Go structs +======================== + +Enum +---- + +Enums are mapped as strings in Go, using a specified string type per +Enum to help with type safety in the Go application. + +.. code-block:: JSON + { 'enum': 'HostMemPolicy', + 'data': [ 'default', 'preferred', 'bind', 'interleave' ] } + +.. code-block:: go + type HostMemPolicy string + + const ( + HostMemPolicyDefault HostMemPolicy = "default" + HostMemPolicyPreferred HostMemPolicy = "preferred" + HostMemPolicyBind HostMemPolicy = "bind" + HostMemPolicyInterleave HostMemPolicy = "interleave" + ) + + +Struct +------ + +The mapping between a QAPI struct in Go struct is very straightforward. + - Each member of the QAPI struct has its own field in a Go struct. + - Optional members are pointers type with 'omitempty' field tag set + +One important design decision was to _not_ embed base struct, copying +the base members to the original struct. This reduces the complexity +for the Go application. + +.. code-block:: JSON + { 'struct': 'BlockExportOptionsNbdBase', + 'data': { '*name': 'str', '*description': 'str' } } + + { 'struct': 'BlockExportOptionsNbd', + 'base': 'BlockExportOptionsNbdBase', + 'data': { '*bitmaps': ['BlockDirtyBitmapOrStr'], + '*allocation-depth': 'bool' } } + +.. code-block:: go + type BlockExportOptionsNbd struct { + Name *string `json:"name,omitempty"` + Description *string `json:"description,omitempty"` + + Bitmaps []BlockDirtyBitmapOrStr `json:"bitmaps,omitempty"` + AllocationDepth *bool `json:"allocation-depth,omitempty"` + } + + +Union +----- + +Unions in QAPI are binded to a Enum type which provides all possible +branches of the union. The most important caveat here is that the Union +does not need to implement all possible branches for the Enum. +Receiving a enum value of a unimplemented branch is valid. For this +reason, we keep a discriminator field in each Union Go struct and also +implement the Marshal interface. + +As each Union Go struct type has both the discriminator field and +optional fields, it is important to note that when converting Go struct +to JSON, we only consider the discriminator field if no optional field +member was set. In practice, the user should use the optional fields if +the QAPI Union type has defined them, otherwise the user can set the +discriminator field for the unbranched enum value. + +.. code-block:: JSON + { 'union': 'ImageInfoSpecificQCow2Encryption', + 'base': 'ImageInfoSpecificQCow2EncryptionBase', + 'discriminator': 'format', + 'data': { 'luks': 'QCryptoBlockInfoLUKS' } } + +.. code-block:: go + type ImageInfoSpecificQCow2Encryption struct { + Format BlockdevQcow2EncryptionFormat `json:"format"` + + // Variants fields + Luks *QCryptoBlockInfoLUKS `json:"-"` + } + + func (s ImageInfoSpecificQCow2Encryption) MarshalJSON() ([]byte, error) { + // Normal logic goes here + // ... + + // Check for valid values without field members + if len(bytes) == 0 && err == nil && + (s.Format == BlockdevQcow2EncryptionFormatAes) { + type Alias ImageInfoSpecificQCow2Encryption + bytes, err = json.Marshal(Alias(s)) + } + // ... + } + + + func (s *ImageInfoSpecificQCow2Encryption) UnmarshalJSON(data []byte) error { + // Normal logic goes here + // ... + + switch tmp.Format { + case BlockdevQcow2EncryptionFormatLuks: + // ... + default: + // Check for valid values without field members + if tmp.Format != BlockdevQcow2EncryptionFormatAes { + return fmt.Errorf(...) + } + } + return nil + } + + +Alternate +--------- + +Like Unions, alternates can have a few branches. Unlike Unions, they +don't have a discriminator field and each branch should be a different +class of Type entirely (e.g: You can't have two branches of type int in +one Alternate). + +While the marshalling is similar to Unions, the unmarshalling uses a +try-and-error approach, trying to fit the data payload in one of the +Alternate fields. + +The biggest caveat is handling Alternates that can take JSON Null as +value. The issue lies on `encoding/json` library limitation where +unmarshalling JSON Null data to a Go struct which has the 'omitempty' +field that, it bypass the Marshal interface. The same happens when +marshalling, if the field tag 'omitempty' is used, a nil pointer would +never be translated to null JSON value. + +The problem being, we use pointer to type plus `omitempty` field to +express a QAPI optional member. + +In order to handle JSON Null, the generator needs to do the following: + - Read the QAPI schema prior to generate any code and cache + all alternate types that can take JSON Null + - For all Go structs that should be considered optional and they type + are one of those alternates, do not set `omitempty` and implement + Marshal interface for this Go struct, to properly handle JSON Null + - In the Alternate, uses a boolean 'IsNull' to express a JSON Null + and implement the AbsentAlternate interface, to help sturcts know + if a given Alternate type should be considered Absent (not set) or + any other possible Value, including JSON Null. + +.. code-block:: JSON + { 'alternate': 'BlockdevRefOrNull', + 'data': { 'definition': 'BlockdevOptions', + 'reference': 'str', + 'null': 'null' } } + +.. code-block:: go + type BlockdevRefOrNull struct { + Definition *BlockdevOptions + Reference *string + IsNull bool + } + + func (s *BlockdevRefOrNull) ToAnyOrAbsent() (any, bool) { + if s != nil { + if s.IsNull { + return nil, false + } else if s.Definition != nil { + return *s.Definition, false + } else if s.Reference != nil { + return *s.Reference, false + } + } + + return nil, true + } + + func (s BlockdevRefOrNull) MarshalJSON() ([]byte, error) { + if s.IsNull { + return []byte("null"), nil + } else if s.Definition != nil { + return json.Marshal(s.Definition) + } else if s.Reference != nil { + return json.Marshal(s.Reference) + } + return []byte("{}"), nil + } + + func (s *BlockdevRefOrNull) UnmarshalJSON(data []byte) error { + // Check for json-null first + if string(data) == "null" { + s.IsNull = true + return nil + } + // Check for BlockdevOptions + { + s.Definition = new(BlockdevOptions) + if err := StrictDecode(s.Definition, data); err == nil { + return nil + } + s.Definition = nil + } + // Check for string + { + s.Reference = new(string) + if err := StrictDecode(s.Reference, data); err == nil { + return nil + } + s.Reference = nil + } + + return fmt.Errorf("Can't convert to BlockdevRefOrNull: %s", string(data)) + } + + +Event +----- + +All events are mapped to its own struct with the additional +MessageTimestamp field, for the over-the-wire 'timestamp' value. + +Marshaling and Unmarshaling happens over the Event interface, so users +should use the MarshalEvent() and UnmarshalEvent() methods. + +.. code-block:: JSON + { 'event': 'SHUTDOWN', + 'data': { 'guest': 'bool', + 'reason': 'ShutdownCause' } } + +.. code-block:: go + type Event interface { + GetName() string + GetTimestamp() Timestamp + } + + type ShutdownEvent struct { + MessageTimestamp Timestamp `json:"-"` + Guest bool `json:"guest"` + Reason ShutdownCause `json:"reason"` + } + + func (s *ShutdownEvent) GetName() string { + return "SHUTDOWN" + } + + func (s *ShutdownEvent) GetTimestamp() Timestamp { + return s.MessageTimestamp + } + + +Command +------- + +All commands are mapped to its own struct with the additional MessageId +field for the optional 'id'. If the command has a boxed data struct, +the option struct will be embed in the command struct. + +As commands do require a return value, every command has its own return +type. The Command interface has a GetReturnType() method that returns a +CommandReturn interface, to help Go application handling the data. + +Marshaling and Unmarshaling happens over the Command interface, so +users should use the MarshalCommand() and UnmarshalCommand() methods. + +.. code-block:: JSON + { 'command': 'set_password', + 'boxed': true, + 'data': 'SetPasswordOptions' } + +.. code-block:: go + type Command interface { + GetId() string + GetName() string + GetReturnType() CommandReturn + } + + // SetPasswordOptions is embed + type SetPasswordCommand struct { + SetPasswordOptions + MessageId string `json:"-"` + } + + // This is an union + type SetPasswordOptions struct { + Protocol DisplayProtocol `json:"protocol"` + Password string `json:"password"` + Connected *SetPasswordAction `json:"connected,omitempty"` + + // Variants fields + Vnc *SetPasswordOptionsVnc `json:"-"` + } + +Now an example of a command without boxed type. + +.. code-block:: JSON + { 'command': 'set_link', + 'data': {'name': 'str', 'up': 'bool'} } + +.. code-block:: go + type SetLinkCommand struct { + MessageId string `json:"-"` + Name string `json:"name"` + Up bool `json:"up"` + } + +Known issues +============ + +- Type names might not follow proper Go convention. Andrea suggested an + annotation to the QAPI schema that could solve it. + https://lists.gnu.org/archive/html/qemu-devel/2022-05/msg00127.html