From patchwork Mon May 4 15:05:36 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eric Blake X-Patchwork-Id: 467656 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 0064C1402AB for ; Tue, 5 May 2015 01:15:42 +1000 (AEST) Received: from localhost ([::1]:34636 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YpI5s-00042X-1A for incoming@patchwork.ozlabs.org; Mon, 04 May 2015 11:15:40 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:49659) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YpHwg-0005gS-Tz for qemu-devel@nongnu.org; Mon, 04 May 2015 11:06:12 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YpHwb-0000R4-PH for qemu-devel@nongnu.org; Mon, 04 May 2015 11:06:10 -0400 Received: from mx1.redhat.com ([209.132.183.28]:52432) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YpHwb-0000Qf-Ei for qemu-devel@nongnu.org; Mon, 04 May 2015 11:06:05 -0400 Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id t44F62pj024390 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Mon, 4 May 2015 11:06:03 -0400 Received: from red.redhat.com (ovpn-113-25.phx2.redhat.com [10.3.113.25]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id t44F5dJ0018431; Mon, 4 May 2015 11:06:02 -0400 From: Eric Blake To: qemu-devel@nongnu.org Date: Mon, 4 May 2015 09:05:36 -0600 Message-Id: <1430751937-17523-40-git-send-email-eblake@redhat.com> In-Reply-To: <1430751937-17523-1-git-send-email-eblake@redhat.com> References: <1430751937-17523-1-git-send-email-eblake@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-MIME-Autoconverted: from 8bit to quoted-printable by mx1.redhat.com id t44F62pj024390 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x X-Received-From: 209.132.183.28 Cc: kwolf@redhat.com, berto@igalia.com, armbru@redhat.com Subject: [Qemu-devel] [PATCH v8 39/40] qapi: Support (subset of) \u escapes in strings X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org The handling of \ inside QAPI strings was less than ideal, and really only worked JSON's \/, \\, \", and our extension of \' (an obvious extension, when you realize we use '' instead of "" for strings). For other things, like '\n', it resulted in a literal 'n' instead of a newline. Of course, at the moment, we really have no use for escaped characters, as QAPI has to map to C identifiers, and we currently support ASCII only for that. But down the road, we may add support for default values for string parameters to a command or struct; if that happens, it would be nice to correctly support all JSON escape sequences, such as \n or \uXXXX. This gets us closer, by supporting Unicode escapes in the ASCII range. Since JSON does not require \OCTAL or \xXX escapes, and our QMP implementation does not understand them either, I intentionally reject it here, but it would be an easy addition if we desired it. Likewise, intentionally refusing the NUL byte means we don't have to worry about C strings being shorter than the qapi input. Signed-off-by: Eric Blake Reviewed-by: Markus Armbruster --- v7: add escape-outside-string, unknown-escape, and tighten parser to reject unknown escapes and \u0000 --- scripts/qapi.py | 36 +++++++++++++++++++++++++++- tests/Makefile | 2 ++ tests/qapi-schema/escape-outside-string.err | 1 + tests/qapi-schema/escape-outside-string.exit | 1 + tests/qapi-schema/escape-outside-string.json | 3 +++ tests/qapi-schema/escape-outside-string.out | 0 tests/qapi-schema/escape-too-big.err | 1 + tests/qapi-schema/escape-too-big.exit | 1 + tests/qapi-schema/escape-too-big.json | 3 +++ tests/qapi-schema/escape-too-big.out | 0 tests/qapi-schema/escape-too-short.err | 1 + tests/qapi-schema/escape-too-short.exit | 1 + tests/qapi-schema/escape-too-short.json | 3 +++ tests/qapi-schema/escape-too-short.out | 0 tests/qapi-schema/ident-with-escape.err | 1 - tests/qapi-schema/ident-with-escape.exit | 2 +- tests/qapi-schema/ident-with-escape.json | 2 +- tests/qapi-schema/ident-with-escape.out | 3 +++ tests/qapi-schema/unicode-str.err | 1 + tests/qapi-schema/unicode-str.exit | 1 + tests/qapi-schema/unicode-str.json | 2 ++ tests/qapi-schema/unicode-str.out | 0 tests/qapi-schema/unknown-escape.err | 1 + tests/qapi-schema/unknown-escape.exit | 1 + tests/qapi-schema/unknown-escape.json | 3 +++ tests/qapi-schema/unknown-escape.out | 0 26 files changed, 66 insertions(+), 4 deletions(-) create mode 100644 tests/qapi-schema/escape-outside-string.err create mode 100644 tests/qapi-schema/escape-outside-string.exit create mode 100644 tests/qapi-schema/escape-outside-string.json create mode 100644 tests/qapi-schema/escape-outside-string.out create mode 100644 tests/qapi-schema/escape-too-big.err create mode 100644 tests/qapi-schema/escape-too-big.exit create mode 100644 tests/qapi-schema/escape-too-big.json create mode 100644 tests/qapi-schema/escape-too-big.out create mode 100644 tests/qapi-schema/escape-too-short.err create mode 100644 tests/qapi-schema/escape-too-short.exit create mode 100644 tests/qapi-schema/escape-too-short.json create mode 100644 tests/qapi-schema/escape-too-short.out create mode 100644 tests/qapi-schema/unicode-str.err create mode 100644 tests/qapi-schema/unicode-str.exit create mode 100644 tests/qapi-schema/unicode-str.json create mode 100644 tests/qapi-schema/unicode-str.out create mode 100644 tests/qapi-schema/unknown-escape.err create mode 100644 tests/qapi-schema/unknown-escape.exit create mode 100644 tests/qapi-schema/unknown-escape.json create mode 100644 tests/qapi-schema/unknown-escape.out diff --git a/tests/qapi-schema/unknown-escape.out b/tests/qapi-schema/unknown-escape.out new file mode 100644 index 0000000..e69de29 diff --git a/scripts/qapi.py b/scripts/qapi.py index 3c8ce45..14468ba 100644 --- a/scripts/qapi.py +++ b/scripts/qapi.py @@ -173,7 +173,41 @@ class QAPISchema: raise QAPISchemaError(self, 'Missing terminating "\'"') if esc: - string += ch + if ch == 'b': + string += '\b' + elif ch == 'f': + string += '\f' + elif ch == 'n': + string += '\n' + elif ch == 'r': + string += '\r' + elif ch == 't': + string += '\t' + elif ch == 'u': + value = 0 + for x in range(0, 4): + ch = self.src[self.cursor] + self.cursor += 1 + if ch not in "0123456789abcdefABCDEF": + raise QAPISchemaError(self, + '\\u escape needs 4 ' + 'hex digits') + value = (value << 4) + int(ch, 16) + # If Python 2 and 3 didn't disagree so much on + # how to handle Unicode, then we could allow + # Unicode string defaults. But most of QAPI is + # ASCII-only, so we aren't losing much for now. + if not value or value > 0x7f: + raise QAPISchemaError(self, + 'For now, \\u escape ' + 'only supports non-zero ' + 'values up to \\u007f') + string += chr(value) + elif ch in "\\/'\"": + string += ch + else: + raise QAPISchemaError(self, + "Unknown escape \\%s" %ch) esc = False elif ch == "\\": esc = True diff --git a/tests/Makefile b/tests/Makefile index e2a3bd3..547a249 100644 --- a/tests/Makefile +++ b/tests/Makefile @@ -212,6 +212,8 @@ check-qapi-schema-y := $(addprefix tests/qapi-schema/, \ enum-clash-member.json enum-max-member.json enum-union-clash.json \ enum-bad-name.json funny-char.json indented-expr.json \ missing-type.json bad-ident.json ident-with-escape.json \ + escape-outside-string.json unknown-escape.json \ + escape-too-short.json escape-too-big.json unicode-str.json \ double-type.json bad-base.json bad-type-bool.json bad-type-int.json \ bad-type-dict.json double-data.json unknown-expr-key.json \ redefined-type.json redefined-command.json redefined-builtin.json \ diff --git a/tests/qapi-schema/escape-outside-string.err b/tests/qapi-schema/escape-outside-string.err new file mode 100644 index 0000000..b9b8837 --- /dev/null +++ b/tests/qapi-schema/escape-outside-string.err @@ -0,0 +1 @@ +tests/qapi-schema/escape-outside-string.json:3:27: Stray "\" diff --git a/tests/qapi-schema/escape-outside-string.exit b/tests/qapi-schema/escape-outside-string.exit new file mode 100644 index 0000000..d00491f --- /dev/null +++ b/tests/qapi-schema/escape-outside-string.exit @@ -0,0 +1 @@ +1 diff --git a/tests/qapi-schema/escape-outside-string.json b/tests/qapi-schema/escape-outside-string.json new file mode 100644 index 0000000..482f795 --- /dev/null +++ b/tests/qapi-schema/escape-outside-string.json @@ -0,0 +1,3 @@ +# escape sequences are permitted only inside strings +# { 'command': 'foo', 'data': {} } +{ 'command': 'foo', 'data'\u003a{} } diff --git a/tests/qapi-schema/escape-outside-string.out b/tests/qapi-schema/escape-outside-string.out new file mode 100644 index 0000000..e69de29 diff --git a/tests/qapi-schema/escape-too-big.err b/tests/qapi-schema/escape-too-big.err new file mode 100644 index 0000000..d9aeb5d --- /dev/null +++ b/tests/qapi-schema/escape-too-big.err @@ -0,0 +1 @@ +tests/qapi-schema/escape-too-big.json:3:14: For now, \u escape only supports non-zero values up to \u007f diff --git a/tests/qapi-schema/escape-too-big.exit b/tests/qapi-schema/escape-too-big.exit new file mode 100644 index 0000000..d00491f --- /dev/null +++ b/tests/qapi-schema/escape-too-big.exit @@ -0,0 +1 @@ +1 diff --git a/tests/qapi-schema/escape-too-big.json b/tests/qapi-schema/escape-too-big.json new file mode 100644 index 0000000..62bcecd --- /dev/null +++ b/tests/qapi-schema/escape-too-big.json @@ -0,0 +1,3 @@ +# we don't support full Unicode strings, yet +# { 'command': 'é' } +{ 'command': '\u00e9' } diff --git a/tests/qapi-schema/escape-too-big.out b/tests/qapi-schema/escape-too-big.out new file mode 100644 index 0000000..e69de29 diff --git a/tests/qapi-schema/escape-too-short.err b/tests/qapi-schema/escape-too-short.err new file mode 100644 index 0000000..934de59 --- /dev/null +++ b/tests/qapi-schema/escape-too-short.err @@ -0,0 +1 @@ +tests/qapi-schema/escape-too-short.json:3:14: \u escape needs 4 hex digits diff --git a/tests/qapi-schema/escape-too-short.exit b/tests/qapi-schema/escape-too-short.exit new file mode 100644 index 0000000..d00491f --- /dev/null +++ b/tests/qapi-schema/escape-too-short.exit @@ -0,0 +1 @@ +1 diff --git a/tests/qapi-schema/escape-too-short.json b/tests/qapi-schema/escape-too-short.json new file mode 100644 index 0000000..6cb1dec --- /dev/null +++ b/tests/qapi-schema/escape-too-short.json @@ -0,0 +1,3 @@ +# the \u escape requires 4 hex digits +# { 'command': 'a' } +{ 'command': '\u61' } diff --git a/tests/qapi-schema/escape-too-short.out b/tests/qapi-schema/escape-too-short.out new file mode 100644 index 0000000..e69de29 diff --git a/tests/qapi-schema/ident-with-escape.err b/tests/qapi-schema/ident-with-escape.err index f7d1c55..e69de29 100644 --- a/tests/qapi-schema/ident-with-escape.err +++ b/tests/qapi-schema/ident-with-escape.err @@ -1 +0,0 @@ -tests/qapi-schema/ident-with-escape.json:3: Expression is missing metatype diff --git a/tests/qapi-schema/ident-with-escape.exit b/tests/qapi-schema/ident-with-escape.exit index d00491f..573541a 100644 --- a/tests/qapi-schema/ident-with-escape.exit +++ b/tests/qapi-schema/ident-with-escape.exit @@ -1 +1 @@ -1 +0 diff --git a/tests/qapi-schema/ident-with-escape.json b/tests/qapi-schema/ident-with-escape.json index cfb2050..5661750 100644 --- a/tests/qapi-schema/ident-with-escape.json +++ b/tests/qapi-schema/ident-with-escape.json @@ -1,4 +1,4 @@ -# FIXME: we should allow escape sequences in strings, if they map back to ASCII +# we allow escape sequences in strings, if they map back to ASCII # { 'command': 'fooA', 'data': { 'bar1': 'str' } } { 'c\u006fmmand': '\u0066\u006f\u006FA', 'd\u0061ta': { '\u0062\u0061\u00721': '\u0073\u0074\u0072' } } diff --git a/tests/qapi-schema/ident-with-escape.out b/tests/qapi-schema/ident-with-escape.out index e69de29..4028430 100644 --- a/tests/qapi-schema/ident-with-escape.out +++ b/tests/qapi-schema/ident-with-escape.out @@ -0,0 +1,3 @@ +[OrderedDict([('command', 'fooA'), ('data', OrderedDict([('bar1', 'str')]))])] +[] +[] diff --git a/tests/qapi-schema/unicode-str.err b/tests/qapi-schema/unicode-str.err new file mode 100644 index 0000000..f621cd6 --- /dev/null +++ b/tests/qapi-schema/unicode-str.err @@ -0,0 +1 @@ +tests/qapi-schema/unicode-str.json:2: 'command' uses invalid name 'é' diff --git a/tests/qapi-schema/unicode-str.exit b/tests/qapi-schema/unicode-str.exit new file mode 100644 index 0000000..d00491f --- /dev/null +++ b/tests/qapi-schema/unicode-str.exit @@ -0,0 +1 @@ +1 diff --git a/tests/qapi-schema/unicode-str.json b/tests/qapi-schema/unicode-str.json new file mode 100644 index 0000000..5253a1b --- /dev/null +++ b/tests/qapi-schema/unicode-str.json @@ -0,0 +1,2 @@ +# we don't support full Unicode strings, yet +{ 'command': 'é' } diff --git a/tests/qapi-schema/unicode-str.out b/tests/qapi-schema/unicode-str.out new file mode 100644 index 0000000..e69de29 diff --git a/tests/qapi-schema/unknown-escape.err b/tests/qapi-schema/unknown-escape.err new file mode 100644 index 0000000..000e30d --- /dev/null +++ b/tests/qapi-schema/unknown-escape.err @@ -0,0 +1 @@ +tests/qapi-schema/unknown-escape.json:3:21: Unknown escape \x diff --git a/tests/qapi-schema/unknown-escape.exit b/tests/qapi-schema/unknown-escape.exit new file mode 100644 index 0000000..d00491f --- /dev/null +++ b/tests/qapi-schema/unknown-escape.exit @@ -0,0 +1 @@ +1 diff --git a/tests/qapi-schema/unknown-escape.json b/tests/qapi-schema/unknown-escape.json new file mode 100644 index 0000000..8e6891e --- /dev/null +++ b/tests/qapi-schema/unknown-escape.json @@ -0,0 +1,3 @@ +# we only recognize JSON escape sequences, plus our \' extension (no \x) +# { 'command': 'foo', 'data': {} } +{ 'command': 'foo', 'dat\x61':{} }