From patchwork Thu Nov 16 18:52:14 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Beier X-Patchwork-Id: 838750 X-Patchwork-Delegate: blogic@openwrt.org Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=lists.infradead.org (client-ip=65.50.211.133; helo=bombadil.infradead.org; envelope-from=lede-dev-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org; receiver=) Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="uV5llK9i"; dkim-atps=neutral Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3yd9Nz04Ntz9s4s for ; Fri, 17 Nov 2017 05:53:06 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:Subject:Message-Id: Date:To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=kGz0Th9fiG9mnn1zKHmQl8bwcBQk86Qt4nMw2g1EVKQ=; b=uV5llK9itr6yA3 ewVmIyBYi/Ec4AJo5GO4BGAQ+OsbnVOmcEA1SxP+14gZxNJ7tQTfAPJzdGsLxp66cXNUOyNcXgtsi xMFf6D+uQPgUHm+FM6CGJowd+5PKThGYuHZ5M8RvX2UyII8hINIHj/GCcLS2JuEeq5jIRKkL2KeR8 RjN6cbogvf0akGKg3/6OeGCwkaqsDPTBkDbVVTJHk8WT06HGlGxIV1WlwU3GXn7vhWa2f5ll+ZbIP R/xZHSuvrLVts+Zr1wv800vmPJxTTFpr+X+Obf4sL+J7ZIAyWMAl7r65UsZKnOOUp5vnqptvWRbwb iQJdjwMxj4tXrpcKIhEg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1eFPHR-00042G-Ew; Thu, 16 Nov 2017 18:52:53 +0000 Received: from mail-wm0-f65.google.com ([74.125.82.65]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1eFPHN-00041Q-3q for lede-dev@lists.infradead.org; Thu, 16 Nov 2017 18:52:51 +0000 Received: by mail-wm0-f65.google.com with SMTP id z3so2159670wme.3 for ; Thu, 16 Nov 2017 10:52:28 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=lu6yIT0iUXyAHv2CQq4YXCKt7VuBfNnrpVjdasRm6/M=; b=SLzr8AighxxSLZQ6cXaXDKIympcDeW1FMqIxMWWxGr5Kf/8yAYJ40h8a3F8osyJBQD zsOwXja9qI7xvqrqnIHLG58ikPwp1UkiK7yZgUhB73P4Q62Hr3VEZr+DnFVlvKLQsfVj Ovox6cyENYjx/pYE5QUBUoDyj3eF+undB48+hrSySldz18U8DpW1X/5Z/bDkBMyyfLUH mMrJS4BjhYtvZa+bSNCZkaDgsPaTVa4LW/KC+jYmfNNlJNrNnfTqjX3DED3EF47fTSGi VUzsZMD6BDMN4xF/n++7eYNPRDI853m0HdW39B1QBcOPsVco8sJZMV0xl0W1Lc3Osl7N 0e7w== X-Gm-Message-State: AJaThX58ViSIx3PjpMtqCoA9g5rB0opXfysskPKWX0tzdtaaktIjhoxj OH7+4nEDQm+pdSAeN+AHf6VxHg== X-Google-Smtp-Source: AGs4zMbcQVcotj9A5Y8Gfn2C4p8kQvzkiHT6eFAdny36h9G/WGDHT0zYt03m01htvzCdeo/2iR9rzA== X-Received: by 10.80.142.201 with SMTP id x9mr4084193edx.285.1510858346370; Thu, 16 Nov 2017 10:52:26 -0800 (PST) Received: from denkpad ([178.19.214.171]) by smtp.gmail.com with ESMTPSA id p7sm1489686edj.5.2017.11.16.10.52.25 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 16 Nov 2017 10:52:25 -0800 (PST) Received: from bk by denkpad with local (Exim 4.89) (envelope-from ) id 1eFPGy-0003cr-O4; Thu, 16 Nov 2017 19:52:24 +0100 From: Christian Beier To: lede-dev@lists.infradead.org Date: Thu, 16 Nov 2017 19:52:14 +0100 Message-Id: <20171116185214.13895-1-dontmind@freeshell.org> X-Mailer: git-send-email 2.11.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20171116_105249_321671_A05DC097 X-CRM114-Status: GOOD ( 11.98 ) X-Spam-Score: -1.7 (-) X-Spam-Report: SpamAssassin version 3.4.1 on bombadil.infradead.org summary: Content analysis details: (-1.7 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no trust [74.125.82.65 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (christian.cb.beier[at]googlemail.com) 0.0 HEADER_FROM_DIFFERENT_DOMAINS From and EnvelopeFrom 2nd level mail domains are different -0.0 RCVD_IN_MSPIKE_H3 RBL: Good reputation (+3) [74.125.82.65 listed in wl.mailspike.net] -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] -0.0 RCVD_IN_MSPIKE_WL Mailspike good senders 0.2 FREEMAIL_FORGED_FROMDOMAIN 2nd level domains in From and EnvelopeFrom freemail headers are different Subject: [LEDE-DEV] [PATCH] jshn: add functionality to read big JSON X-BeenThere: lede-dev@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Christian Beier MIME-Version: 1.0 Sender: "Lede-dev" Errors-To: lede-dev-bounces+incoming=patchwork.ozlabs.org@lists.infradead.org The existing read functionality feeds the complete JSON to jshn as a cmdline argument, leading to `-ash: jshn: Argument list too long` errors for JSONs bigger than ca. 100KB. This commit adds the ability to read the JSON directly from a file if wanted, removing this shell-imposed size limit. Tested on x86-64 and ar71xx. An mmap()-based solution was also evaluated, but found to make no performance difference on either platform. Signed-off-by: Christian Beier Signed-off-by: Christian Beier --- jshn.c | 30 ++++++++++++++++++++++++++++-- sh/jshn.sh | 4 ++++ 2 files changed, 32 insertions(+), 2 deletions(-) diff --git a/jshn.c b/jshn.c index 79136dd..a3866a0 100644 --- a/jshn.c +++ b/jshn.c @@ -25,6 +25,8 @@ #include #include #include +#include +#include #include "list.h" #include "avl.h" @@ -300,7 +302,7 @@ out: static int usage(const char *progname) { - fprintf(stderr, "Usage: %s [-n] [-i] -r |-w\n", progname); + fprintf(stderr, "Usage: %s [-n] [-i] -r |-R |-w\n", progname); return 2; } @@ -333,6 +335,10 @@ int main(int argc, char **argv) struct env_var *vars; int i; int ch; + int fd; + struct stat sb; + char *fbuf; + int ret; avl_init(&env_vars, avl_strcmp_var, false, NULL); for (i = 0; environ[i]; i++); @@ -354,7 +360,7 @@ int main(int argc, char **argv) avl_insert(&env_vars, &vars[i].avl); } - while ((ch = getopt(argc, argv, "p:nir:w")) != -1) { + while ((ch = getopt(argc, argv, "p:nir:R:w")) != -1) { switch(ch) { case 'p': var_prefix = optarg; @@ -362,6 +368,26 @@ int main(int argc, char **argv) break; case 'r': return jshn_parse(optarg); + case 'R': + if((fd = open(optarg, O_RDONLY)) == -1) { + fprintf(stderr, "Error opening %s\n", optarg); + return 3; + } + if (fstat(fd, &sb) == -1) { + fprintf(stderr, "Error getting size of %s\n", optarg); + close(fd); + return 3; + } + if(!(fbuf = malloc(sb.st_size)) || read(fd, fbuf, sb.st_size) != sb.st_size) { + fprintf(stderr, "Error reading %s\n", optarg); + free(fbuf); + close(fd); + return 3; + } + ret = jshn_parse(fbuf); + free(fbuf); + close(fd); + return ret; case 'w': return jshn_format(no_newline, indent); case 'n': diff --git a/sh/jshn.sh b/sh/jshn.sh index bf76edb..0a146e1 100644 --- a/sh/jshn.sh +++ b/sh/jshn.sh @@ -174,6 +174,10 @@ json_load() { eval "`jshn -r "$1"`" } +json_load_file() { + eval "`jshn -R "$1"`" +} + json_dump() { jshn "$@" ${JSON_PREFIX:+-p "$JSON_PREFIX"} -w }