From patchwork Wed Apr 10 19:50:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1083561 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="Dlf+3zdG"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44fZX33Jvcz9s71 for ; Thu, 11 Apr 2019 05:50:43 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726672AbfDJTum (ORCPT ); Wed, 10 Apr 2019 15:50:42 -0400 Received: from mail-wm1-f65.google.com ([209.85.128.65]:36832 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726181AbfDJTul (ORCPT ); Wed, 10 Apr 2019 15:50:41 -0400 Received: by mail-wm1-f65.google.com with SMTP id h18so3907971wml.1 for ; Wed, 10 Apr 2019 12:50:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=O6OlTHii6j9e7eTQ+bnhdP8ya9aIrO2QI22g5wX9eOY=; b=Dlf+3zdGa/xsUucE8VU8V1MkDiasfXBFiRiblGXyUJRhZMqTJEiibbyJKboAoXPfT1 wz6mQMyKynUmfa/PWRD2lSvQtfhGL2gyaGO/SWI0rLUtkSA6A/fsnBhrP4Ra7GvLvBEn 4gy8XTynX9dYzprO9EVK4qD98mEodm6TjBZjIBNwgWZRLQsFrx82Db5EIDBIIc1+gHCu HYj6zOLX9M1YLC5rf2Jmk7Y2wrkjFsq58p9eRvngrR+JXqWze8yDWNK0PuhNRK3jdSgM BG/olKCKDSZo6P49mzCclAajEuom0Ze2m4ir/NRYKhYLXxpeTsV3RiJaop9yXXBwj5xh 7g2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=O6OlTHii6j9e7eTQ+bnhdP8ya9aIrO2QI22g5wX9eOY=; b=cMKnEBYetr7lQEG8ERDzsoT1bUT3P61dcIPjPr85qZ8tfOdc4/APMZDi3M8JXN6QPj FCGxDHZ8t9wbxuoSsCjx4dw0nVE53aLkYhXSU+pP7QU+y7Eii+QOnXKGpilAlxeTW3mw ckb6x64w64z0k6aBuBwEomw5B4L1iQfTR2wNVG9cUYBlxjI4rRqB8iBjn0rAHZd+No85 S/IfzKoKlmv9xmoX71byvGVhEZpizhd3Q8ZL0yBYX9hcz9ynR8K2LXjpXiaujnN+s7Vc qRLjoRdprDdsxnAvEjUIsZ3ffEQz6HFMT9qsuZs4Ur6hDW5qYquP8koXEdp/7BAMr7FB Mb1g== X-Gm-Message-State: APjAAAUK2wU+4/cruY4RRAxxWtvX0gwmmRBUqfyQQX2wNnin7rOj8I86 6HuAZEjeh8FA5iVjwjVNTKW9VQ== X-Google-Smtp-Source: APXvYqxhnZ28KPgl7KHA79WsXkfG54+6TnnyBtfvb5Kp2pcJxMPV4i5mVNiay2/W8iC+5jh+VdF4sQ== X-Received: by 2002:a1c:e0d6:: with SMTP id x205mr4153012wmg.28.1554925840152; Wed, 10 Apr 2019 12:50:40 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id 4sm2288540wmg.12.2019.04.10.12.50.39 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 10 Apr 2019 12:50:39 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang Subject: [PATCH/RFC v2 bpf-next 01/19] bpf: refactor propagate_liveness to eliminate duplicated for loop Date: Wed, 10 Apr 2019 20:50:15 +0100 Message-Id: <1554925833-7333-2-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> References: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Propagation for register and stack slot are finished in separate for loop, while they are perfect to be put into a single loop. This could also let them share some common variables in later patches. Signed-off-by: Jiong Wang --- kernel/bpf/verifier.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index b747434..f6ebe80 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -6223,10 +6223,8 @@ static int propagate_liveness(struct bpf_verifier_env *env, return err; } } - } - /* ... and stack slots */ - for (frame = 0; frame <= vstate->curframe; frame++) { + /* Propagate stack slots. */ state = vstate->frame[frame]; parent = vparent->frame[frame]; for (i = 0; i < state->allocated_stack / BPF_REG_SIZE && From patchwork Wed Apr 10 19:50:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1083562 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="uEZdv+/p"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44fZX35h5pz9s7h for ; Thu, 11 Apr 2019 05:50:43 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726677AbfDJTun (ORCPT ); Wed, 10 Apr 2019 15:50:43 -0400 Received: from mail-wm1-f66.google.com ([209.85.128.66]:51954 "EHLO mail-wm1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726230AbfDJTum (ORCPT ); Wed, 10 Apr 2019 15:50:42 -0400 Received: by mail-wm1-f66.google.com with SMTP id 4so3907145wmf.1 for ; Wed, 10 Apr 2019 12:50:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=j3+vr9FJ4cb7vK24Lm/VGA0FkexNyzFHzTsS6dRPTCU=; b=uEZdv+/pFhLYh67FsjyZugndyUP5IS7K84EsLaLSHsVQyN6V1ByyYSYxcB4XubuZQH TW3e43XYu2TNSnvsF5KvrukruZQ0s77ZdKdBSmtWZ75k6aAOimUnhdpY058zhV8sRgIW PMR36A+CH3AusrWqzz+j1JLL+044WouUhhUdIFl0DCBss8Uw5OA/1pBsqi7qcZMeYJIy Dru2IZZ+UE+2cdw/xEA2SIc+JsTRM3UCZvQVugdoXqIAYLJMtTuJKCHWkTA4ebyb8vAb aLhr7iekiq7oFeMjKoBQKbmevP+38fGkCgHPXSh3n1vxS+R8sPr1EFI4EXR+3JeP6nJY vLcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=j3+vr9FJ4cb7vK24Lm/VGA0FkexNyzFHzTsS6dRPTCU=; b=svo+agFplrTTAWP/lQgUfJFC6PR850Hv5tr33a33+J4vWY9MDE4fhAYV4XFybnZ1wN sf6b+PUBKOWdBacFmdsb9vPZ9bQli98Lcg1RPKUaYU76F8k4OcfEscGlrf8U57apZlD2 DQk/JL7qYJqsDsCdTX8b0ofJFrqXnk+jrLR7H4j+Te68yVq2Kd7Jf6OakWovhg1RJhkM SQ9T1gnEEhJtWugltD7jvdoc5EYK36a7qU0UigYk6xKFLx32z3xgbyKelDO5jFXD3+ZN FwW4mB5hdYqLtmrU2DHbdE+c8IXAz0HGtqLR0+KHbmCdah3N9PjFPz7Xaqp54H6Fp4dE ulRQ== X-Gm-Message-State: APjAAAVSv3axLwlfMOP+Ch0mZpudQeBOnbXOKl/M+6mW/yBCy12K12MV UFdP+n3BGWMmTAB22+QLilx1yA== X-Google-Smtp-Source: APXvYqwt8T0P6+2CX/IKz8AeZ2osBO/CVSd3J2eXF3j54mlL4BEjMuFKJI/sZBc0nLiYbaTP8ezNNA== X-Received: by 2002:a1c:f310:: with SMTP id q16mr3855263wmq.102.1554925841171; Wed, 10 Apr 2019 12:50:41 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id 4sm2288540wmg.12.2019.04.10.12.50.40 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 10 Apr 2019 12:50:40 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang Subject: [PATCH/RFC v2 bpf-next 02/19] bpf: refactor propagate_liveness to eliminate code redundance Date: Wed, 10 Apr 2019 20:50:16 +0100 Message-Id: <1554925833-7333-3-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> References: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> Sender: bpf-owner@vger.kernel.org Precedence: bulk List-Id: netdev.vger.kernel.org Access to reg states were not factored out, the consequence is long code for dereferencing them which made the indentation not good for reading. This patch factor out these code so the core code in the loop could be easier to follow. Signed-off-by: Jiong Wang Reviewed-by: Jakub Kicinski --- kernel/bpf/verifier.c | 34 ++++++++++++++++++++-------------- 1 file changed, 20 insertions(+), 14 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index f6ebe80..d57082a 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -6201,8 +6201,9 @@ static int propagate_liveness(struct bpf_verifier_env *env, const struct bpf_verifier_state *vstate, struct bpf_verifier_state *vparent) { - int i, frame, err = 0; + struct bpf_reg_state *state_reg, *parent_reg; struct bpf_func_state *state, *parent; + int i, frame, err = 0; if (vparent->curframe != vstate->curframe) { WARN(1, "propagate_live: parent frame %d current frame %d\n", @@ -6212,28 +6213,33 @@ static int propagate_liveness(struct bpf_verifier_env *env, /* Propagate read liveness of registers... */ BUILD_BUG_ON(BPF_REG_FP + 1 != MAX_BPF_REG); for (frame = 0; frame <= vstate->curframe; frame++) { + parent = vparent->frame[frame]; + state = vstate->frame[frame]; + parent_reg = parent->regs; + state_reg = state->regs; /* We don't need to worry about FP liveness, it's read-only */ for (i = frame < vstate->curframe ? BPF_REG_6 : 0; i < BPF_REG_FP; i++) { - if (vparent->frame[frame]->regs[i].live & REG_LIVE_READ) + if (parent_reg[i].live & REG_LIVE_READ) continue; - if (vstate->frame[frame]->regs[i].live & REG_LIVE_READ) { - err = mark_reg_read(env, &vstate->frame[frame]->regs[i], - &vparent->frame[frame]->regs[i]); - if (err) - return err; - } + if (!(state_reg[i].live & REG_LIVE_READ)) + continue; + err = mark_reg_read(env, &state_reg[i], &parent_reg[i]); + if (err) + return err; } /* Propagate stack slots. */ - state = vstate->frame[frame]; - parent = vparent->frame[frame]; for (i = 0; i < state->allocated_stack / BPF_REG_SIZE && i < parent->allocated_stack / BPF_REG_SIZE; i++) { - if (parent->stack[i].spilled_ptr.live & REG_LIVE_READ) + parent_reg = &parent->stack[i].spilled_ptr; + state_reg = &state->stack[i].spilled_ptr; + if (parent_reg->live & REG_LIVE_READ) continue; - if (state->stack[i].spilled_ptr.live & REG_LIVE_READ) - mark_reg_read(env, &state->stack[i].spilled_ptr, - &parent->stack[i].spilled_ptr); + if (!(state_reg->live & REG_LIVE_READ)) + continue; + err = mark_reg_read(env, state_reg, parent_reg); + if (err) + return err; } } return err; From patchwork Wed Apr 10 19:50:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1083563 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="eQt0WsT7"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44fZX66kJmz9s5c for ; Thu, 11 Apr 2019 05:50:46 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726694AbfDJTup (ORCPT ); Wed, 10 Apr 2019 15:50:45 -0400 Received: from mail-wm1-f66.google.com ([209.85.128.66]:35686 "EHLO mail-wm1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726676AbfDJTun (ORCPT ); Wed, 10 Apr 2019 15:50:43 -0400 Received: by mail-wm1-f66.google.com with SMTP id y197so3912659wmd.0 for ; Wed, 10 Apr 2019 12:50:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=przrOLCJP/2yXV45rk8KRThSpUEF8IQ+x2IHuQUdOZM=; b=eQt0WsT716NYlw+LCPgntW/NYu6FMWKZ0hsSNqfUL/b6FOMFvYf1NH+tJsyn44UsP3 FYfF6FlQNs2QE4pWh+egQV0q1A2KdXZI/oYlb37e7MD2+d5yCd7TJkfnfrh1FFkpVCtk vohlQKpIZNRB9EZqKaMTX0aW6AFVRscHso4wIbrgOlPUCVHjPzGS9AZO5TYCkIVs04Ws D45I5wy8S1Ay9gKWk526PrVVH/Zz55DE02cqriVstBu5vnaQTEhn/qHJsGXf/+f1oSNh wiW9umuGK0tsqX2U4JB/ANIL/x17HILgpoO9gHo4ObyapU0bXCTSJzlY7vavWeIP6itV 67vQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=przrOLCJP/2yXV45rk8KRThSpUEF8IQ+x2IHuQUdOZM=; b=grhNp8nkHi0IDdfpI/FB1h++ZoBC4Dfxzm8KWJiO7iGbNHPWXcWR21WOrNPS/Uo7Q1 M+O/KM58xw1ucIRmmxaxm/f/8Dm+LQn47MU9bUJqqkEtP05os/9roLP/FGFjktqK5FSr xxlP7vEeqNKSBzxlcLp8SiHHnhjBEWBU4KjJHK+U9nJ7CDNLcPtEgA/egg8yuYJfDe+v fiAFjGJsZfXJmKeMJh8O7NQHdBAiYfn5b8/ZggzbzinhGl6G/r5LyPtQmLMvik76Vngx veIkM8jp8RfjEqrKd/U+mSh9q6bRJCYV2JeIALIwg5jB0TjBhkUjXnNKDbaAZTds/8e/ 3t3A== X-Gm-Message-State: APjAAAVqr9TgpQ0xiHU3z89gXgsCIlbRs8467CwQPKztlEAmagGi1HjK kEQ2GIpqKhrnbeXDTKsw/lOcnw== X-Google-Smtp-Source: APXvYqzGkEflsgCvQxNKiYOINcjHWO0llCWJDmqpVOKKQsgFM+X42Pb0yJl2WA5RufJyDZN1H8XA4g== X-Received: by 2002:a1c:7918:: with SMTP id l24mr4140616wme.29.1554925842355; Wed, 10 Apr 2019 12:50:42 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id 4sm2288540wmg.12.2019.04.10.12.50.41 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 10 Apr 2019 12:50:41 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang Subject: [PATCH/RFC v2 bpf-next 03/19] bpf: factor out reg and stack slot propagation into "propagate_liveness_reg" Date: Wed, 10 Apr 2019 20:50:17 +0100 Message-Id: <1554925833-7333-4-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> References: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org After code refactor in previous patches, the propagation logic inside the for loop in "propagate_liveness" becomes clear that they are good enough to be factored out into a common function "propagate_liveness_reg". Signed-off-by: Jiong Wang Reviewed-by: Jakub Kicinski --- kernel/bpf/verifier.c | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index d57082a..8d60dc7 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -6190,6 +6190,22 @@ static bool states_equal(struct bpf_verifier_env *env, return true; } +static int propagate_liveness_reg(struct bpf_verifier_env *env, + struct bpf_reg_state *reg, + struct bpf_reg_state *parent_reg) +{ + int err; + + if (parent_reg->live & REG_LIVE_READ || !(reg->live & REG_LIVE_READ)) + return 0; + + err = mark_reg_read(env, reg, parent_reg); + if (err) + return err; + + return 0; +} + /* A write screens off any subsequent reads; but write marks come from the * straight-line code between a state and its parent. When we arrive at an * equivalent state (jump target or such) we didn't arrive by the straight-line @@ -6219,11 +6235,8 @@ static int propagate_liveness(struct bpf_verifier_env *env, state_reg = state->regs; /* We don't need to worry about FP liveness, it's read-only */ for (i = frame < vstate->curframe ? BPF_REG_6 : 0; i < BPF_REG_FP; i++) { - if (parent_reg[i].live & REG_LIVE_READ) - continue; - if (!(state_reg[i].live & REG_LIVE_READ)) - continue; - err = mark_reg_read(env, &state_reg[i], &parent_reg[i]); + err = propagate_liveness_reg(env, &state_reg[i], + &parent_reg[i]); if (err) return err; } @@ -6233,11 +6246,8 @@ static int propagate_liveness(struct bpf_verifier_env *env, i < parent->allocated_stack / BPF_REG_SIZE; i++) { parent_reg = &parent->stack[i].spilled_ptr; state_reg = &state->stack[i].spilled_ptr; - if (parent_reg->live & REG_LIVE_READ) - continue; - if (!(state_reg->live & REG_LIVE_READ)) - continue; - err = mark_reg_read(env, state_reg, parent_reg); + err = propagate_liveness_reg(env, state_reg, + parent_reg); if (err) return err; } From patchwork Wed Apr 10 19:50:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1083566 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="N//uFze+"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44fZXB0Zf7z9s5c for ; Thu, 11 Apr 2019 05:50:50 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726716AbfDJTut (ORCPT ); Wed, 10 Apr 2019 15:50:49 -0400 Received: from mail-wm1-f68.google.com ([209.85.128.68]:36886 "EHLO mail-wm1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726678AbfDJTup (ORCPT ); Wed, 10 Apr 2019 15:50:45 -0400 Received: by mail-wm1-f68.google.com with SMTP id v14so3903904wmf.2 for ; Wed, 10 Apr 2019 12:50:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=3iIFlXR2eXoRp0nNYSlUsqCTo2SKZyG3oKJceN0EjE8=; b=N//uFze+I0XvR3/1vrCPzFncgfwiLneQjDHOFRsSsbb71joWPjlgq0XByh+OVR1s7V hMKaOIW2e82JwROyLbAGl4PLOHi8alq2iPN9Ejdi7gV6dBiPLrNzzGTMFq8iXQyYAyyB H+s50kBut+sXJqJ5TiaKzaidG0dvXFohg1iDS9b69949qeZGAuB+7jPpmJJhV9L+vGf3 YX7zNFcWcH/sYm3XCOtxrcbFMR87jOb6+oBZaJCTOBThBUWL4Z9k/MwbsejPBvKlR1KH Rjehje3ybSrCgaU+6EN3NkYWgambN0tacCuHkDgsIALyHzb7MV8S02NCpB0CYWuBchUl yXTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=3iIFlXR2eXoRp0nNYSlUsqCTo2SKZyG3oKJceN0EjE8=; b=TQ2RoXtIj9IQ8mXboWMEmUVUHbdAKvyTQ8fVT7EGtKZ4FThGUw4lMWXjQ8spYh8x3H uwMsIGmBRXUHif/NfthtUONeOidQrLl2CjLT3bYX2+WrGV85z7wXCnwDD15eHaFN+lOn 8tdPLrI01gTxZQqiqSptk4DMaDTJuCcUIEXMxjYdswQ1E63fMfaSiY5gVF+DNt22XnJl f2l29kz+3tsxqaWzFUED99qHPNwSHzWXJk2BlSR/OWRHhlsfkcKjtHJ+XXfnHl7pIiwe zhhjmhJLfjDce6jHPMND4dAuJ8Dpl0777uIABabjB0UeY1NW27D0TTrmZdwYdFuGwgWl +n4A== X-Gm-Message-State: APjAAAUBAYemytKddADTaCN1E0WtyxUmKj9cnbWK8u7rPYY9+SNlxnjN rZtX8uRQAdTd8QWPUkaCwd9UDg== X-Google-Smtp-Source: APXvYqzIWoB9j5FMiW5W1zd3sdLTk1f1s/pUb4/hRKT4b6bhAdOxE0rbQrPLJBCv/gYwgkgVwTJBwQ== X-Received: by 2002:a1c:acc8:: with SMTP id v191mr4256502wme.72.1554925843599; Wed, 10 Apr 2019 12:50:43 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id 4sm2288540wmg.12.2019.04.10.12.50.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 10 Apr 2019 12:50:42 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang Subject: [PATCH/RFC v2 bpf-next 04/19] bpf: refactor "check_reg_arg" to eliminate code redundancy Date: Wed, 10 Apr 2019 20:50:18 +0100 Message-Id: <1554925833-7333-5-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> References: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> Sender: bpf-owner@vger.kernel.org Precedence: bulk List-Id: netdev.vger.kernel.org There are a few "regs[regno]" here are there across "check_reg_arg", this patch factor it out into a simple "reg" pointer. The intention is to simplify code indentation and make the later patches in this set look cleaner. Signed-off-by: Jiong Wang Reviewed-by: Jakub Kicinski --- kernel/bpf/verifier.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 8d60dc7..bd30a65 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -1177,30 +1177,32 @@ static int check_reg_arg(struct bpf_verifier_env *env, u32 regno, { struct bpf_verifier_state *vstate = env->cur_state; struct bpf_func_state *state = vstate->frame[vstate->curframe]; - struct bpf_reg_state *regs = state->regs; + struct bpf_reg_state *reg, *regs = state->regs; if (regno >= MAX_BPF_REG) { verbose(env, "R%d is invalid\n", regno); return -EINVAL; } + reg = ®s[regno]; if (t == SRC_OP) { /* check whether register used as source operand can be read */ - if (regs[regno].type == NOT_INIT) { + if (reg->type == NOT_INIT) { verbose(env, "R%d !read_ok\n", regno); return -EACCES; } /* We don't need to worry about FP liveness because it's read-only */ - if (regno != BPF_REG_FP) - return mark_reg_read(env, ®s[regno], - regs[regno].parent); + if (regno == BPF_REG_FP) + return 0; + + return mark_reg_read(env, reg, reg->parent); } else { /* check whether register used as dest operand can be written to */ if (regno == BPF_REG_FP) { verbose(env, "frame pointer is read only\n"); return -EACCES; } - regs[regno].live |= REG_LIVE_WRITTEN; + reg->live |= REG_LIVE_WRITTEN; if (t == DST_OP) mark_reg_unknown(env, regs, regno); } From patchwork Wed Apr 10 19:50:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1083565 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="KEmF+7/q"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44fZX833C4z9s6w for ; Thu, 11 Apr 2019 05:50:48 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726712AbfDJTur (ORCPT ); Wed, 10 Apr 2019 15:50:47 -0400 Received: from mail-wm1-f66.google.com ([209.85.128.66]:54990 "EHLO mail-wm1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726691AbfDJTuq (ORCPT ); Wed, 10 Apr 2019 15:50:46 -0400 Received: by mail-wm1-f66.google.com with SMTP id c1so3891448wml.4 for ; Wed, 10 Apr 2019 12:50:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=bY31K3qUtUB/VTz310uOs02Po0yxYg6fK7p/+dcUrWM=; b=KEmF+7/q8bNbHj1bf1YkPxppykf3uW7x1xQ6VGwRuIeyQ7Dpqotq7EsTU5tSEbqhrI 5VdYNk1OC9GLG6SwQmimVeux5rHkBynfH/inp4Tx9RvCRlmlxf/MUB+8/v8Ng/b7aJ1S 0Nh/lroNrJXuePXf36h4LdOt4X6rJBydPvte4V+IbDBASts555CMw317pt+YbsRFHEzm Fsxm1jQ+Yw0qaCr8GzQW9VrRi/bfq23r3iG7l010hDlo3jiaKw42K7Y6PrajHYPxXqNR c8VWfoLFzKwdPmG29FEohjyGJXAX6xbRSO7Lh7418+2wiikwFCDko3lKMblxDgQl5jR6 HK4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=bY31K3qUtUB/VTz310uOs02Po0yxYg6fK7p/+dcUrWM=; b=Lh+bHCaO3rVOU9bvpad2x3WzO5j8+P4+tROxuesApLdE7JT+M7RZylvvjB/hncPGVV +kfJd3NrrO4hN4PAzMy39X9N7YxqzgtVlEs5ZWrenRAuTSYa/XEXzWL9LAaxTcjrdCo6 hIyGKgbOFj7tlHzkW21Twkl12PUO2u0jgRtFLmGiAD+Ee9PdgsbgOIFyiKyvPsIaL7dc EiVeB+D/oZI0URe0kRFD5SfAK84WO8a3GtgcKVT5oAb7pjTrX/RSXeVLmXi3OtfTdsOt zMelWQnFaQP61rAxkmzajKxyejbosRzcea7ae3AHp9i+BeG3BOQDJyhUjZ/Xxe7DKF2o Mscg== X-Gm-Message-State: APjAAAVzDfn0IRmEwqYv0YN9vRetTobKUsMKBorE0WGIYR3mfk6Up9j2 XA5Bvmy2lmIDItWxl2F2gYLXCQ== X-Google-Smtp-Source: APXvYqxrifPWXqy4eq4q3mbVcHWN/dcziK5EFlQWW13ukbMgyqJJSZuHPKkB39gTmPXUX5rqhwzVWQ== X-Received: by 2002:a7b:ce84:: with SMTP id q4mr4123518wmj.41.1554925844742; Wed, 10 Apr 2019 12:50:44 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id 4sm2288540wmg.12.2019.04.10.12.50.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 10 Apr 2019 12:50:43 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang Subject: [PATCH/RFC v2 bpf-next 05/19] bpf: split read liveness into REG_LIVE_READ64 and REG_LIVE_READ32 Date: Wed, 10 Apr 2019 20:50:19 +0100 Message-Id: <1554925833-7333-6-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> References: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> Sender: bpf-owner@vger.kernel.org Precedence: bulk List-Id: netdev.vger.kernel.org Register liveness infrastructure doesn't track register read width at the moment, while the width information will be needed for the later 32-bit safety analysis pass. This patch take the first step to split read liveness into REG_LIVE_READ64 and REG_LIVE_READ32. Liveness propagation code are updated accordingly. They are taught to understand how to propagate REG_LIVE_READ64 and REG_LIVE_READ32 at the same propagation iteration. For example, "mark_reg_read" now propagate "flags" which could be multiple read bits instead of the single REG_LIVE_READ64. A write still screen off all width of reads. Signed-off-by: Jiong Wang --- include/linux/bpf_verifier.h | 8 +-- kernel/bpf/verifier.c | 119 +++++++++++++++++++++++++++++++++++++++---- 2 files changed, 113 insertions(+), 14 deletions(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index b3ab61f..fba0ebb 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -36,9 +36,11 @@ */ enum bpf_reg_liveness { REG_LIVE_NONE = 0, /* reg hasn't been read or written this branch */ - REG_LIVE_READ, /* reg was read, so we're sensitive to initial value */ - REG_LIVE_WRITTEN, /* reg was written first, screening off later reads */ - REG_LIVE_DONE = 4, /* liveness won't be updating this register anymore */ + REG_LIVE_READ32 = 0x1, /* reg was read, so we're sensitive to initial value */ + REG_LIVE_READ64 = 0x2, /* likewise, but full 64-bit content matters */ + REG_LIVE_READ = REG_LIVE_READ32 | REG_LIVE_READ64, + REG_LIVE_WRITTEN = 0x4, /* reg was written first, screening off later reads */ + REG_LIVE_DONE = 0x8, /* liveness won't be updating this register anymore */ }; struct bpf_reg_state { diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index bd30a65..44e4278 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -1135,7 +1135,7 @@ static int check_subprogs(struct bpf_verifier_env *env) */ static int mark_reg_read(struct bpf_verifier_env *env, const struct bpf_reg_state *state, - struct bpf_reg_state *parent) + struct bpf_reg_state *parent, u8 flags) { bool writes = parent == state->parent; /* Observe write marks */ int cnt = 0; @@ -1150,17 +1150,17 @@ static int mark_reg_read(struct bpf_verifier_env *env, parent->var_off.value, parent->off); return -EFAULT; } - if (parent->live & REG_LIVE_READ) + if ((parent->live & REG_LIVE_READ) == flags) /* The parentage chain never changes and - * this parent was already marked as LIVE_READ. + * this parent was already marked with all read bits. * There is no need to keep walking the chain again and - * keep re-marking all parents as LIVE_READ. + * keep re-marking all parents with reads bits in flags. * This case happens when the same register is read * multiple times without writes into it in-between. */ break; /* ... then we depend on parent's value */ - parent->live |= REG_LIVE_READ; + parent->live |= flags; state = parent; parent = state->parent; writes = true; @@ -1172,12 +1172,95 @@ static int mark_reg_read(struct bpf_verifier_env *env, return 0; } +/* This function is supposed to be used by the following 32-bit optimization + * code only. It returns TRUE if the source or destination register operates + * on 64-bit, otherwise return FALSE. + */ +static bool is_reg64(struct bpf_insn *insn, u32 regno, + struct bpf_reg_state *reg, enum reg_arg_type t) +{ + u8 code, class, op; + + code = insn->code; + class = BPF_CLASS(code); + op = BPF_OP(code); + if (class == BPF_JMP) { + /* BPF_EXIT will reach here because of return value readability + * test for "main" which has s32 return value. + */ + if (op == BPF_EXIT) + return false; + if (op == BPF_CALL) { + /* BPF to BPF call will reach here because of marking + * caller saved clobber with DST_OP_NO_MARK for which we + * don't care the register def because they are anyway + * marked as NOT_INIT already. + */ + if (insn->src_reg == BPF_PSEUDO_CALL) + return false; + /* Helper call will reach here because of arg type + * check. Conservatively marking all args as 64-bit. + */ + return true; + } + } + + if (class == BPF_ALU64 || class == BPF_JMP || + /* BPF_END always use BPF_ALU class. */ + (class == BPF_ALU && op == BPF_END && insn->imm == 64)) + return true; + + if (class == BPF_ALU || class == BPF_JMP32) + return false; + + if (class == BPF_LDX) { + if (t != SRC_OP) + return BPF_SIZE(code) == BPF_DW; + /* LDX source must be ptr. */ + return true; + } + + if (class == BPF_STX) { + if (reg->type != SCALAR_VALUE) + return true; + return BPF_SIZE(code) == BPF_DW; + } + + if (class == BPF_LD) { + u8 mode = BPF_MODE(code); + + /* LD_IMM64 */ + if (mode == BPF_IMM) + return true; + + /* Both LD_IND and LD_ABS return 32-bit data. */ + if (t != SRC_OP) + return false; + + /* Implicit ctx ptr. */ + if (regno == BPF_REG_6) + return true; + + /* Explicit source could be any width. */ + return true; + } + + if (class == BPF_ST) + /* The only source register for BPF_ST is a ptr. */ + return true; + + /* Conservatively return true at default. */ + return true; +} + static int check_reg_arg(struct bpf_verifier_env *env, u32 regno, enum reg_arg_type t) { struct bpf_verifier_state *vstate = env->cur_state; struct bpf_func_state *state = vstate->frame[vstate->curframe]; + struct bpf_insn *insn = env->prog->insnsi + env->insn_idx; struct bpf_reg_state *reg, *regs = state->regs; + bool rw64; if (regno >= MAX_BPF_REG) { verbose(env, "R%d is invalid\n", regno); @@ -1185,6 +1268,7 @@ static int check_reg_arg(struct bpf_verifier_env *env, u32 regno, } reg = ®s[regno]; + rw64 = is_reg64(insn, regno, reg, t); if (t == SRC_OP) { /* check whether register used as source operand can be read */ if (reg->type == NOT_INIT) { @@ -1195,7 +1279,8 @@ static int check_reg_arg(struct bpf_verifier_env *env, u32 regno, if (regno == BPF_REG_FP) return 0; - return mark_reg_read(env, reg, reg->parent); + return mark_reg_read(env, reg, reg->parent, + rw64 ? REG_LIVE_READ64 : REG_LIVE_READ32); } else { /* check whether register used as dest operand can be written to */ if (regno == BPF_REG_FP) { @@ -1382,7 +1467,8 @@ static int check_stack_read(struct bpf_verifier_env *env, state->regs[value_regno].live |= REG_LIVE_WRITTEN; } mark_reg_read(env, ®_state->stack[spi].spilled_ptr, - reg_state->stack[spi].spilled_ptr.parent); + reg_state->stack[spi].spilled_ptr.parent, + REG_LIVE_READ64); return 0; } else { int zeros = 0; @@ -1399,7 +1485,9 @@ static int check_stack_read(struct bpf_verifier_env *env, return -EACCES; } mark_reg_read(env, ®_state->stack[spi].spilled_ptr, - reg_state->stack[spi].spilled_ptr.parent); + reg_state->stack[spi].spilled_ptr.parent, + size == BPF_REG_SIZE + ? REG_LIVE_READ64 : REG_LIVE_READ32); if (value_regno >= 0) { if (zeros == size) { /* any size read into register is zero extended, @@ -2336,7 +2424,9 @@ static int check_stack_boundary(struct bpf_verifier_env *env, int regno, * the whole slot to be marked as 'read' */ mark_reg_read(env, &state->stack[spi].spilled_ptr, - state->stack[spi].spilled_ptr.parent); + state->stack[spi].spilled_ptr.parent, + access_size == BPF_REG_SIZE + ? REG_LIVE_READ64 : REG_LIVE_READ32); } return update_stack_depth(env, state, min_off); } @@ -6196,12 +6286,19 @@ static int propagate_liveness_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg, struct bpf_reg_state *parent_reg) { + u8 parent_bits = parent_reg->live & REG_LIVE_READ; + u8 bits = reg->live & REG_LIVE_READ; + u8 bits_diff = parent_bits ^ bits; + u8 bits_prop = bits_diff & bits; int err; - if (parent_reg->live & REG_LIVE_READ || !(reg->live & REG_LIVE_READ)) + /* "reg" and "parent_reg" has the same read bits, or the bit doesn't + * belong to "reg". + */ + if (!bits_diff || !bits_prop) return 0; - err = mark_reg_read(env, reg, parent_reg); + err = mark_reg_read(env, reg, parent_reg, bits_prop); if (err) return err; From patchwork Wed Apr 10 19:50:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1083596 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="JC7u2aU0"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44fZY04yw6z9s5c for ; Thu, 11 Apr 2019 05:51:32 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726837AbfDJTvb (ORCPT ); Wed, 10 Apr 2019 15:51:31 -0400 Received: from mail-wm1-f68.google.com ([209.85.128.68]:51963 "EHLO mail-wm1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726702AbfDJTur (ORCPT ); Wed, 10 Apr 2019 15:50:47 -0400 Received: by mail-wm1-f68.google.com with SMTP id 4so3907363wmf.1 for ; Wed, 10 Apr 2019 12:50:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=RzjwDgoZtVWp185BRWOoDHMjI70RVPtNBY3dodliqNo=; b=JC7u2aU05D3a29KS6y3EbuAy9TzFe7mvBXSPzkDpcHC/uU1fqB7x2JWhchGFxR+D6m nHj5JboY8ZRm6mNx7bNT1tqZLgSuaFiVX0V6ganHleteQzdjvk78CibJ+cUi8G1whvbI 0i25rxST+AJ+An1UZSx3qeBBX0JuymgJWY5jyRi8N6lpypWhazAMQLscq1hNRY7SZ5bG ODQfNV12/K8emsXUdSqxXdXvkVFzu76hWXnfeALMihXUUNkwu6NdysDO7/wpc8a9+vcq KgNS39iWZTN/dlpnCRB5wbeo7qJPjXXWlTXSWOMB3rKR/cBDriZh8rt7qJ0bKoFJ7ycG xQ4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=RzjwDgoZtVWp185BRWOoDHMjI70RVPtNBY3dodliqNo=; b=LPP3H62YSS/xdIsrTAKEBo1reaxMrehabgMS2mGW6XNVCxXUZHfCtuhupvT2xZUy1I p79Z68ZgG2QylUJ+3eaqGt7SfnjIWUFbnmsMu/qWEUlRvKekyHTXrCH2+oYH9SzbolBx xiG4xrbj28Ia/sQqnFeb2guVrYoF8DgoYQ8Jt/u6ZCgt8UJGAHY6dNhY7weT2aGEM2G9 6rcjfo3dRr2ZdZd0EmshxAoI8HfBn2UeU9RFIeGCL/A9GKL3P/Ey0KL2piRtW/eMad1I pQs10aLgF79QQYSChZwlRvhGE9Jiqe1LTAFNmfpeAhLmmseOf9HO6L6Z3zbdrkypx2sV ZyrQ== X-Gm-Message-State: APjAAAVyBS1qVUN82HYIkmVezBynPQgLtK3JxCxdpLVIZBOPqbwC81dD gu0CJnEpF+Ki0MKYt5uxTBG+bw== X-Google-Smtp-Source: APXvYqwmxWaCPBhnRbmZa3Hvce/ilwFqgcAi5Ns4DBmZR58QU/oUdSgGcuuqCk3PtHmzLF18gUk2qQ== X-Received: by 2002:a7b:cb04:: with SMTP id u4mr4244832wmj.0.1554925845819; Wed, 10 Apr 2019 12:50:45 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id 4sm2288540wmg.12.2019.04.10.12.50.44 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 10 Apr 2019 12:50:44 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang Subject: [PATCH/RFC v2 bpf-next 06/19] bpf: mark lo32 writes that should be zero extended into hi32 Date: Wed, 10 Apr 2019 20:50:20 +0100 Message-Id: <1554925833-7333-7-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> References: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> Sender: bpf-owner@vger.kernel.org Precedence: bulk List-Id: netdev.vger.kernel.org eBPF ISA specification requires high 32-bit cleared when low 32-bit sub-register is written. This applies to destination register of ALU32 etc. JIT back-ends must guarantee this semantic when doing code-gen. x86-64 and arm64 ISA has the same semantic, so the corresponding JIT back-end doesn't need to do extra work. However, 32-bit arches (arm, nfp etc.) and some other 64-bit arches (powerpc, sparc etc), need explicit zero extension sequence to meet such semantic. This is important, because for code the following: u64_value = (u64) u32_value ... other uses of u64_value compiler could exploit the semantic described above and save those zero extensions for extending u32_value to u64_value. Hardware, runtime, or BPF JIT back-ends, are responsible for guaranteeing this. Some benchmarks show ~40% sub-register writes out of total insns, meaning ~40% extra code-gen ( could go up to more for some arches which requires two shifts for zero extension) because JIT back-end needs to do extra code-gen for all such instructions. However this is not always necessary in case u32_value is never cast into a u64, which is quite normal in real life program. So, it would be really good if we could identify those places where such type cast happened, and only do zero extensions for them, not for the others. This could save a lot of BPF code-gen. Algo: - Record indices of instructions that do sub-register def (write). And these indices need to stay with reg state so path pruning and bpf to bpf function call could be handled properly. These indices are kept up to date while doing insn walk. - A full register read on an active sub-register def marks the def insn as needing zero extension on dst register. - A new sub-register write overrides the old one. A new full register write makes the register free of zero extension on dst register. - When propagating read64 during path pruning, also marks def insns whose defs are hanging active sub-register. Reviewed-by: Jakub Kicinski Signed-off-by: Jiong Wang --- include/linux/bpf_verifier.h | 6 ++++++ kernel/bpf/verifier.c | 45 ++++++++++++++++++++++++++++++++++++++++---- 2 files changed, 47 insertions(+), 4 deletions(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index fba0ebb..c1923a5 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -133,6 +133,11 @@ struct bpf_reg_state { * pointing to bpf_func_state. */ u32 frameno; + /* Tracks subreg definition. The stored value is the insn_idx of the + * writing insn. This is safe because subreg_def is used before any insn + * patching which only happens after main verification finished. + */ + s32 subreg_def; enum bpf_reg_liveness live; }; @@ -234,6 +239,7 @@ struct bpf_insn_aux_data { int ctx_field_size; /* the ctx field size for load insn, maybe 0 */ int sanitize_stack_off; /* stack slot to be cleared */ bool seen; /* this insn was processed by the verifier */ + bool zext_dst; /* this insn zero extend dst reg */ u8 alu_state; /* used in combination with alu_limit */ unsigned int orig_idx; /* original instruction index */ }; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 44e4278..b478b90 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -980,6 +980,7 @@ static void mark_reg_not_init(struct bpf_verifier_env *env, __mark_reg_not_init(regs + regno); } +#define DEF_NOT_SUBREG (-1) static void init_reg_state(struct bpf_verifier_env *env, struct bpf_func_state *state) { @@ -990,6 +991,7 @@ static void init_reg_state(struct bpf_verifier_env *env, mark_reg_not_init(env, regs, i); regs[i].live = REG_LIVE_NONE; regs[i].parent = NULL; + regs[i].subreg_def = DEF_NOT_SUBREG; } /* frame pointer */ @@ -1253,6 +1255,19 @@ static bool is_reg64(struct bpf_insn *insn, u32 regno, return true; } +static void mark_insn_zext(struct bpf_verifier_env *env, + struct bpf_reg_state *reg) +{ + s32 def_idx = reg->subreg_def; + + if (def_idx == DEF_NOT_SUBREG) + return; + + env->insn_aux_data[def_idx].zext_dst = true; + /* The dst will be zero extended, so won't be sub-register anymore. */ + reg->subreg_def = DEF_NOT_SUBREG; +} + static int check_reg_arg(struct bpf_verifier_env *env, u32 regno, enum reg_arg_type t) { @@ -1279,6 +1294,9 @@ static int check_reg_arg(struct bpf_verifier_env *env, u32 regno, if (regno == BPF_REG_FP) return 0; + if (rw64) + mark_insn_zext(env, reg); + return mark_reg_read(env, reg, reg->parent, rw64 ? REG_LIVE_READ64 : REG_LIVE_READ32); } else { @@ -1288,6 +1306,7 @@ static int check_reg_arg(struct bpf_verifier_env *env, u32 regno, return -EACCES; } reg->live |= REG_LIVE_WRITTEN; + reg->subreg_def = rw64 ? DEF_NOT_SUBREG : env->insn_idx; if (t == DST_OP) mark_reg_unknown(env, regs, regno); } @@ -2169,6 +2188,12 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn value_regno); if (reg_type_may_be_null(reg_type)) regs[value_regno].id = ++env->id_gen; + /* A load of ctx field could have different + * actual load size with the one encoded in the + * insn. When the dst is PTR, it is for sure not + * a sub-register. + */ + regs[value_regno].subreg_def = DEF_NOT_SUBREG; } regs[value_regno].type = reg_type; } @@ -3340,6 +3365,9 @@ static int check_helper_call(struct bpf_verifier_env *env, int func_id, int insn check_reg_arg(env, caller_saved[i], DST_OP_NO_MARK); } + /* helper call must return full 64-bit R0. */ + regs[BPF_REG_0].subreg_def = DEF_NOT_SUBREG; + /* update return register (already marked as written above) */ if (fn->ret_type == RET_INTEGER) { /* sets type to SCALAR_VALUE */ @@ -4271,6 +4299,7 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) */ *dst_reg = *src_reg; dst_reg->live |= REG_LIVE_WRITTEN; + dst_reg->subreg_def = DEF_NOT_SUBREG; } else { /* R1 = (u32) R2 */ if (is_pointer_value(env, insn->src_reg)) { @@ -4281,6 +4310,7 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) } else if (src_reg->type == SCALAR_VALUE) { *dst_reg = *src_reg; dst_reg->live |= REG_LIVE_WRITTEN; + dst_reg->subreg_def = env->insn_idx; } else { mark_reg_unknown(env, regs, insn->dst_reg); @@ -5344,6 +5374,8 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn) * Already marked as written above. */ mark_reg_unknown(env, regs, BPF_REG_0); + /* ld_abs load up to 32-bit skb data. */ + regs[BPF_REG_0].subreg_def = env->insn_idx; return 0; } @@ -6282,6 +6314,9 @@ static bool states_equal(struct bpf_verifier_env *env, return true; } +/* Return 0 if no propagation happened. Return negative error code if error + * happened. Otherwise, return the propagated bits. + */ static int propagate_liveness_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg, struct bpf_reg_state *parent_reg) @@ -6302,7 +6337,7 @@ static int propagate_liveness_reg(struct bpf_verifier_env *env, if (err) return err; - return 0; + return bits_prop; } /* A write screens off any subsequent reads; but write marks come from the @@ -6336,8 +6371,10 @@ static int propagate_liveness(struct bpf_verifier_env *env, for (i = frame < vstate->curframe ? BPF_REG_6 : 0; i < BPF_REG_FP; i++) { err = propagate_liveness_reg(env, &state_reg[i], &parent_reg[i]); - if (err) + if (err < 0) return err; + if (err & REG_LIVE_READ64) + mark_insn_zext(env, &parent_reg[i]); } /* Propagate stack slots. */ @@ -6347,11 +6384,11 @@ static int propagate_liveness(struct bpf_verifier_env *env, state_reg = &state->stack[i].spilled_ptr; err = propagate_liveness_reg(env, state_reg, parent_reg); - if (err) + if (err < 0) return err; } } - return err; + return 0; } static int is_state_visited(struct bpf_verifier_env *env, int insn_idx) From patchwork Wed Apr 10 19:50:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1083594 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="UAMKuXC/"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44fZXx3S9Wz9s5c for ; Thu, 11 Apr 2019 05:51:29 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726886AbfDJTv2 (ORCPT ); Wed, 10 Apr 2019 15:51:28 -0400 Received: from mail-wr1-f67.google.com ([209.85.221.67]:37140 "EHLO mail-wr1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726448AbfDJTut (ORCPT ); Wed, 10 Apr 2019 15:50:49 -0400 Received: by mail-wr1-f67.google.com with SMTP id w10so4368670wrm.4 for ; Wed, 10 Apr 2019 12:50:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=NlPoGKLapOA094jH5RaXWl2m1CpdKBwMGl/CNr+Rkc8=; b=UAMKuXC/brrvhsXqVdvgzuv0kznGyJTltWHHwL5pbQhUF/vIIteF1aVA1aecPdTVdI 0wYOh/aiSSsHB3/jdCWP0hQs/UaJuBES81Da97qxeiAFww3+a53/JyoY9ACayggVAanG ZGg1Hkod17cYWh8alVmb51BXoBPMVm5QQ9zcajlqg6qIj3OWb/pKx89pRPAs9574Z/5/ V/rlItavvbYUz+DxI+QlrzGub1GH3isc/NXPANDU33QKVvqyFn3kQQsoDM+cMF+38UuJ gJ01hcJk5Iy9csKQpuRRfu2DiPlueSG+dscm+N+GBPwNdzIwQbU2jbo5s13txLg+U1gA Alpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=NlPoGKLapOA094jH5RaXWl2m1CpdKBwMGl/CNr+Rkc8=; b=mKt1a3Auhcg2OVM+Lawfo3iAZtnNuJFGztHTa+VgxagXWwjDYD//8mINFRIm6V0SuG e2IBh7ZlF1c8DAz3qEIe3j9RpZGt1apeVfVA0SSEEUoFyqY/85GrLU8EZ+pTG+KQuwJp STYiEgFnZqYtUN5cuU/iKVLjB5JvNk2gVps5pgL4T1vOzmLNsxyGl74sgtGWNDix8PQ2 axVFhaR2Xp3qsYtfGkQW1NOekw3pJDtSrwTiMBuNmsxrnQ6eMHIHft68bSfF/o6gGJOD h5aLWzofQfvQR4zLpO+Ni/2N8MZ7Kcjj9okIqFLJwf36+v0TXBBOwtVnXGiRILL3Iq5j cdSA== X-Gm-Message-State: APjAAAVOgE90egE4LMUEiQfr7qnP7ZCrGPennJqpbMxWM032dhdmBCWR 5qpFgyZvrW2LrtD1Vgla/VbJyQ== X-Google-Smtp-Source: APXvYqznRfgJgsqo1f54xhUgbQU3y7miXhQnX1088CH1D4cHIF8dr50R2Pi/76zrmp7nAH7pxfayhw== X-Received: by 2002:a5d:428c:: with SMTP id k12mr13003060wrq.279.1554925846818; Wed, 10 Apr 2019 12:50:46 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id 4sm2288540wmg.12.2019.04.10.12.50.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 10 Apr 2019 12:50:46 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang Subject: [PATCH/RFC v2 bpf-next 07/19] bpf: reduce false alarm by refining helper call arg types Date: Wed, 10 Apr 2019 20:50:21 +0100 Message-Id: <1554925833-7333-8-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> References: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Unlike BPF to BPF function call, BPF helper call is calls to native insns that verifier can't walk. So, verifier needs helper proto type descriptions for data-flow purpose. There is such information already, but it is not differentiate sub-register read with full register read. This patch split "enum bpf_arg_type" for sub-register read, and updated descriptions for several functions that shown frequent usage in one Cilium benchmark. "is_reg64" then taught about these new arg types. Reviewed-by: Jakub Kicinski Signed-off-by: Jiong Wang --- include/linux/bpf.h | 3 +++ kernel/bpf/core.c | 2 +- kernel/bpf/helpers.c | 2 +- kernel/bpf/verifier.c | 61 +++++++++++++++++++++++++++++++++++++++++++-------- net/core/filter.c | 28 +++++++++++------------ 5 files changed, 71 insertions(+), 25 deletions(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 65f7094..42f3345 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -197,9 +197,12 @@ enum bpf_arg_type { ARG_CONST_SIZE, /* number of bytes accessed from memory */ ARG_CONST_SIZE_OR_ZERO, /* number of bytes accessed from memory or 0 */ + ARG_CONST_SIZE32, /* Likewise, but size fits into 32-bit */ + ARG_CONST_SIZE32_OR_ZERO, /* Ditto */ ARG_PTR_TO_CTX, /* pointer to context */ ARG_ANYTHING, /* any (initialized) argument is ok */ + ARG_ANYTHING32, /* Likewise, but it is a 32-bit argument */ ARG_PTR_TO_SPIN_LOCK, /* pointer to bpf_spin_lock */ ARG_PTR_TO_SOCK_COMMON, /* pointer to sock_common */ }; diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index ace8c22..2792eda 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -2067,7 +2067,7 @@ const struct bpf_func_proto bpf_tail_call_proto = { .ret_type = RET_VOID, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_CONST_MAP_PTR, - .arg3_type = ARG_ANYTHING, + .arg3_type = ARG_ANYTHING32, }; /* Stub for JITs that only support cBPF. eBPF programs are interpreted. diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index a411fc1..6b7453e 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -218,7 +218,7 @@ const struct bpf_func_proto bpf_get_current_comm_proto = { .gpl_only = false, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_UNINIT_MEM, - .arg2_type = ARG_CONST_SIZE, + .arg2_type = ARG_CONST_SIZE32, }; #if defined(CONFIG_QUEUED_SPINLOCKS) || defined(CONFIG_BPF_ARCH_SPINLOCK) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index b478b90..431fca9 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -1174,12 +1174,47 @@ static int mark_reg_read(struct bpf_verifier_env *env, return 0; } +static bool helper_call_arg64(struct bpf_verifier_env *env, int func_id, + u32 regno) +{ + /* get_func_proto must succeed, other it should have been rejected + * early inside check_helper_call. + */ + const struct bpf_func_proto *fn = + env->ops->get_func_proto(func_id, env->prog); + enum bpf_arg_type arg_type; + + switch (regno) { + case BPF_REG_1: + arg_type = fn->arg1_type; + break; + case BPF_REG_2: + arg_type = fn->arg2_type; + break; + case BPF_REG_3: + arg_type = fn->arg3_type; + break; + case BPF_REG_4: + arg_type = fn->arg4_type; + break; + case BPF_REG_5: + arg_type = fn->arg5_type; + break; + default: + arg_type = ARG_DONTCARE; + } + + return arg_type != ARG_CONST_SIZE32 && + arg_type != ARG_CONST_SIZE32_OR_ZERO && + arg_type != ARG_ANYTHING32; +} + /* This function is supposed to be used by the following 32-bit optimization * code only. It returns TRUE if the source or destination register operates * on 64-bit, otherwise return FALSE. */ -static bool is_reg64(struct bpf_insn *insn, u32 regno, - struct bpf_reg_state *reg, enum reg_arg_type t) +static bool is_reg64(struct bpf_verifier_env *env, struct bpf_insn *insn, + u32 regno, struct bpf_reg_state *reg, enum reg_arg_type t) { u8 code, class, op; @@ -1201,9 +1236,12 @@ static bool is_reg64(struct bpf_insn *insn, u32 regno, if (insn->src_reg == BPF_PSEUDO_CALL) return false; /* Helper call will reach here because of arg type - * check. Conservatively marking all args as 64-bit. + * check. */ - return true; + if (t == SRC_OP) + return helper_call_arg64(env, insn->imm, regno); + + return false; } } @@ -1283,7 +1321,7 @@ static int check_reg_arg(struct bpf_verifier_env *env, u32 regno, } reg = ®s[regno]; - rw64 = is_reg64(insn, regno, reg, t); + rw64 = is_reg64(env, insn, regno, reg, t); if (t == SRC_OP) { /* check whether register used as source operand can be read */ if (reg->type == NOT_INIT) { @@ -2575,7 +2613,9 @@ static bool arg_type_is_mem_ptr(enum bpf_arg_type type) static bool arg_type_is_mem_size(enum bpf_arg_type type) { return type == ARG_CONST_SIZE || - type == ARG_CONST_SIZE_OR_ZERO; + type == ARG_CONST_SIZE_OR_ZERO || + type == ARG_CONST_SIZE32 || + type == ARG_CONST_SIZE32_OR_ZERO; } static int check_func_arg(struct bpf_verifier_env *env, u32 regno, @@ -2593,7 +2633,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 regno, if (err) return err; - if (arg_type == ARG_ANYTHING) { + if (arg_type == ARG_ANYTHING || arg_type == ARG_ANYTHING32) { if (is_pointer_value(env, regno)) { verbose(env, "R%d leaks addr into helper function\n", regno); @@ -2616,7 +2656,9 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 regno, type != expected_type) goto err_type; } else if (arg_type == ARG_CONST_SIZE || - arg_type == ARG_CONST_SIZE_OR_ZERO) { + arg_type == ARG_CONST_SIZE_OR_ZERO || + arg_type == ARG_CONST_SIZE32 || + arg_type == ARG_CONST_SIZE32_OR_ZERO) { expected_type = SCALAR_VALUE; if (type != expected_type) goto err_type; @@ -2710,7 +2752,8 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 regno, meta->map_ptr->value_size, false, meta); } else if (arg_type_is_mem_size(arg_type)) { - bool zero_size_allowed = (arg_type == ARG_CONST_SIZE_OR_ZERO); + bool zero_size_allowed = (arg_type == ARG_CONST_SIZE_OR_ZERO || + arg_type == ARG_CONST_SIZE32_OR_ZERO); /* remember the mem_size which may be used later * to refine return values. diff --git a/net/core/filter.c b/net/core/filter.c index 22eb2ed..3f6d8af 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -1693,9 +1693,9 @@ static const struct bpf_func_proto bpf_skb_store_bytes_proto = { .gpl_only = false, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, - .arg2_type = ARG_ANYTHING, + .arg2_type = ARG_ANYTHING32, .arg3_type = ARG_PTR_TO_MEM, - .arg4_type = ARG_CONST_SIZE, + .arg4_type = ARG_CONST_SIZE32, .arg5_type = ARG_ANYTHING, }; @@ -1724,9 +1724,9 @@ static const struct bpf_func_proto bpf_skb_load_bytes_proto = { .gpl_only = false, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, - .arg2_type = ARG_ANYTHING, + .arg2_type = ARG_ANYTHING32, .arg3_type = ARG_PTR_TO_UNINIT_MEM, - .arg4_type = ARG_CONST_SIZE, + .arg4_type = ARG_CONST_SIZE32, }; BPF_CALL_5(bpf_skb_load_bytes_relative, const struct sk_buff *, skb, @@ -1875,7 +1875,7 @@ static const struct bpf_func_proto bpf_l3_csum_replace_proto = { .gpl_only = false, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, - .arg2_type = ARG_ANYTHING, + .arg2_type = ARG_ANYTHING32, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_ANYTHING, .arg5_type = ARG_ANYTHING, @@ -1928,7 +1928,7 @@ static const struct bpf_func_proto bpf_l4_csum_replace_proto = { .gpl_only = false, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, - .arg2_type = ARG_ANYTHING, + .arg2_type = ARG_ANYTHING32, .arg3_type = ARG_ANYTHING, .arg4_type = ARG_ANYTHING, .arg5_type = ARG_ANYTHING, @@ -1967,9 +1967,9 @@ static const struct bpf_func_proto bpf_csum_diff_proto = { .pkt_access = true, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_MEM_OR_NULL, - .arg2_type = ARG_CONST_SIZE_OR_ZERO, + .arg2_type = ARG_CONST_SIZE32_OR_ZERO, .arg3_type = ARG_PTR_TO_MEM_OR_NULL, - .arg4_type = ARG_CONST_SIZE_OR_ZERO, + .arg4_type = ARG_CONST_SIZE32_OR_ZERO, .arg5_type = ARG_ANYTHING, }; @@ -2150,7 +2150,7 @@ static const struct bpf_func_proto bpf_redirect_proto = { .func = bpf_redirect, .gpl_only = false, .ret_type = RET_INTEGER, - .arg1_type = ARG_ANYTHING, + .arg1_type = ARG_ANYTHING32, .arg2_type = ARG_ANYTHING, }; @@ -2928,7 +2928,7 @@ static const struct bpf_func_proto bpf_skb_change_proto_proto = { .gpl_only = false, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, - .arg2_type = ARG_ANYTHING, + .arg2_type = ARG_ANYTHING32, .arg3_type = ARG_ANYTHING, }; @@ -2948,7 +2948,7 @@ static const struct bpf_func_proto bpf_skb_change_type_proto = { .gpl_only = false, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, - .arg2_type = ARG_ANYTHING, + .arg2_type = ARG_ANYTHING32, }; static u32 bpf_skb_net_base_len(const struct sk_buff *skb) @@ -3236,7 +3236,7 @@ static const struct bpf_func_proto bpf_skb_change_tail_proto = { .gpl_only = false, .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, - .arg2_type = ARG_ANYTHING, + .arg2_type = ARG_ANYTHING32, .arg3_type = ARG_ANYTHING, }; @@ -3832,7 +3832,7 @@ static const struct bpf_func_proto bpf_skb_get_tunnel_key_proto = { .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_UNINIT_MEM, - .arg3_type = ARG_CONST_SIZE, + .arg3_type = ARG_CONST_SIZE32, .arg4_type = ARG_ANYTHING, }; @@ -3941,7 +3941,7 @@ static const struct bpf_func_proto bpf_skb_set_tunnel_key_proto = { .ret_type = RET_INTEGER, .arg1_type = ARG_PTR_TO_CTX, .arg2_type = ARG_PTR_TO_MEM, - .arg3_type = ARG_CONST_SIZE, + .arg3_type = ARG_CONST_SIZE32, .arg4_type = ARG_ANYTHING, }; From patchwork Wed Apr 10 19:50:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1083588 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="IyP2MXO5"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44fZXm5t08z9s5c for ; Thu, 11 Apr 2019 05:51:20 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726766AbfDJTux (ORCPT ); Wed, 10 Apr 2019 15:50:53 -0400 Received: from mail-wr1-f66.google.com ([209.85.221.66]:39139 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726714AbfDJTut (ORCPT ); Wed, 10 Apr 2019 15:50:49 -0400 Received: by mail-wr1-f66.google.com with SMTP id j9so4363178wrn.6 for ; Wed, 10 Apr 2019 12:50:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=/a7fPjf9Bd9xA5WyVTLfwhcS6MBzUipM2CB7yUs93Co=; b=IyP2MXO5kuFNcI4z533iS0JWFeGkFF9VIc1MsxNWOMN0rU/XbtOXgIv1g84naTveg/ FiMMmd7zD+3e/nXUpQsaFz83wXxdtot15L2QZzx2PWOQrmrbJnHB1r4eO7ikoJjv4VF5 GoDkNt9Lod60pPJHgDphoPL62HoTmuiTlDjqTF6+xTdIah+1RaxhC3pTK30aP5/8UN3P V7PU58L6rFfYCqQoe8PqQHa86DRJeaRIXl3dV6MNJI3f8qMxRXBKc9uThic7FNWCXy7n B65qZHtIyXvYPr1tDd0Mdn80pc4d1XhbMTffheWBe25Tr8go+wGRnN70/Gy97QgSa6Jn 0rtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=/a7fPjf9Bd9xA5WyVTLfwhcS6MBzUipM2CB7yUs93Co=; b=SjP3RoKSNtVAPdaUxS1yHiYUtehyBBb1TlvSqdd396/V/tu1A1i8xaaBrZQuYd9RIH GieSGPiiaxj1zUsgZsdZNUU37z2Ba0a83fBv9omU11r9Omn7ggs3LXXNeQ73YSqivSJJ 5+uX66heAXclvEe1qCUtzIzBhCoylfiZMLyPoDcPti7zfIBLr29wHnidi6pgNZeAcdcv hb0fGMlQ6pzC8rIVdCu/DFrcrcbLHMYaRziSICuCzIhNnfWL+0RXluKJu/ozDMIvExhu 1+//pYerMc5bPzpUtUOGRlrd0l12w3IJHFOIREXc4iRIqKhTcOvKRDj/YAqS2nLkOzJe hFuA== X-Gm-Message-State: APjAAAWGFgPzWPQJ9q3CQdg6H9MvpAiw1ht/x2/DWuyMG1LJ4KT+L9tl lkzIj5NmxJIEtJ/S18Z3CAaoJZKL+FM= X-Google-Smtp-Source: APXvYqyMwu48zqkPdMYn0GcKF0scX8hiZApamvYxKQLLTlTivTblaOJ5Od0ES29j+mBJ5l3LbXRJ4A== X-Received: by 2002:adf:f48f:: with SMTP id l15mr29017125wro.248.1554925847968; Wed, 10 Apr 2019 12:50:47 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id 4sm2288540wmg.12.2019.04.10.12.50.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 10 Apr 2019 12:50:47 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang Subject: [PATCH/RFC v2 bpf-next 08/19] bpf: insert explicit zero extension insn when hardware doesn't do it implicitly Date: Wed, 10 Apr 2019 20:50:22 +0100 Message-Id: <1554925833-7333-9-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> References: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> Sender: bpf-owner@vger.kernel.org Precedence: bulk List-Id: netdev.vger.kernel.org After previous patches, verifier has marked those instructions that really need zero extension on dst_reg. It is then for all back-ends to decide how to use such information to eliminate unnecessary zero extension code-gen during JIT compilation. One approach is: 1. Verifier insert explicit zero extension for those instructions that need zero extension. 2. All JIT back-ends do NOT generate zero extension for sub-register write any more. The good thing for this approach is no major change on JIT back-end interface, all back-ends could get this optimization. However, only those back-ends that do not have hardware zero extension want this optimization. For back-ends like x86_64 and AArch64, there is hardware support, so zext insertion should be disabled. This patch introduces new target hook "bpf_jit_hardware_zext" which is default true, meaning the underlying hardware will do zero extension implicitly, therefore zext insertion by verifier will be disabled. Once a back-end overrides this hook to false, then verifier will insert zext sequence to clear high 32-bit of definitions when necessary. Offload targets do not use this native target hook, instead, they could get the optimization results using bpf_prog_offload_ops.finalize. Reviewed-by: Jakub Kicinski Signed-off-by: Jiong Wang --- include/linux/bpf.h | 1 + include/linux/filter.h | 1 + kernel/bpf/core.c | 8 +++++ kernel/bpf/verifier.c | 87 +++++++++++++++++++++++++++++++++++++++++++++++++- 4 files changed, 96 insertions(+), 1 deletion(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 42f3345..1823cb4 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -366,6 +366,7 @@ struct bpf_prog_aux { u32 id; u32 func_cnt; /* used by non-func prog as the number of func progs */ u32 func_idx; /* 0 for non-func prog, the index in func array for func prog */ + bool no_verifier_zext; /* No zero extension insertion by verifier. */ bool offload_requested; struct bpf_prog **func; void *jit_data; /* JIT specific data. arch dependent */ diff --git a/include/linux/filter.h b/include/linux/filter.h index 6074aa0..a12529a 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -819,6 +819,7 @@ u64 __bpf_call_base(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5); struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog); void bpf_jit_compile(struct bpf_prog *prog); +bool bpf_jit_hardware_zext(void); bool bpf_helper_changes_pkt_data(void *func); static inline bool bpf_dump_raw_ok(void) diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 2792eda..1c54274 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -2091,6 +2091,14 @@ bool __weak bpf_helper_changes_pkt_data(void *func) return false; } +/* Return TRUE is the target hardware of JIT will do zero extension to high bits + * when writing to low 32-bit of one register. Otherwise, return FALSE. + */ +bool __weak bpf_jit_hardware_zext(void) +{ + return true; +} + /* To execute LD_ABS/LD_IND instructions __bpf_prog_run() may call * skb_copy_bits(), so provide a weak definition of it for NET-less config. */ diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 431fca9..9141a9a 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -7520,6 +7520,80 @@ static int opt_remove_nops(struct bpf_verifier_env *env) return 0; } +static int opt_subreg_zext_lo32(struct bpf_verifier_env *env) +{ + struct bpf_insn_aux_data orig_aux, *aux = env->insn_aux_data; + struct bpf_insn *insns = env->prog->insnsi; + int i, delta = 0, len = env->prog->len; + struct bpf_insn zext_patch[3]; + struct bpf_prog *new_prog; + + zext_patch[1] = BPF_ALU64_IMM(BPF_LSH, 0, 32); + zext_patch[2] = BPF_ALU64_IMM(BPF_RSH, 0, 32); + for (i = 0; i < len; i++) { + int adj_idx = i + delta; + struct bpf_insn insn; + + if (!aux[adj_idx].zext_dst) + continue; + + insn = insns[adj_idx]; + /* "adjust_insn_aux_data" only retains the original insn aux + * data if insn at patched offset is at the end of the patch + * buffer. That is to say, given the following insn sequence: + * + * insn 1 + * insn 2 + * insn 3 + * + * if the patch offset is at insn 2, then the patch buffer must + * be the following that original insn aux data can be retained. + * + * {lshift, rshift, insn2} + * + * However, zero extension needs to be inserted after insn2, so + * insn patch buffer needs to be the following: + * + * {insn2, lshift, rshift} + * + * which would cause insn aux data of insn2 lost and that data + * is critical for ctx field load instruction transformed + * correctly later inside "convert_ctx_accesses". + * + * The simplest way to fix this to build the following patch + * buffer: + * + * {lshift, rshift, insn-next-to-insn2} + * + * Given insn2 defines a value, it can't be a JMP, hence there + * must be a next insn for it otherwise CFG check should have + * rejected this program. However, insn-next-to-insn2 could + * be a JMP and verifier insn patch infrastructure doesn't + * support adjust offset for JMP inside patch buffer. We would + * end up with a few insn check and offset adj code outside of + * the generic insn patch helpers if we go with this approach. + * + * Therefore, we still use {insn2, lshift, rshift} as the patch + * buffer, we copy and restore insn aux data for insn2 + * explicitly. The change looks simpler and smaller. + */ + zext_patch[0] = insns[adj_idx]; + zext_patch[1].dst_reg = insn.dst_reg; + zext_patch[2].dst_reg = insn.dst_reg; + memcpy(&orig_aux, &aux[adj_idx], sizeof(orig_aux)); + new_prog = bpf_patch_insn_data(env, adj_idx, zext_patch, 3); + if (!new_prog) + return -ENOMEM; + env->prog = new_prog; + insns = new_prog->insnsi; + aux = env->insn_aux_data; + memcpy(&aux[adj_idx], &orig_aux, sizeof(orig_aux)); + delta += 2; + } + + return 0; +} + /* convert load instructions that access fields of a context type into a * sequence of instructions that access fields of the underlying structure: * struct __sk_buff -> struct sk_buff @@ -8351,7 +8425,18 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, if (ret == 0) ret = check_max_stack_depth(env); - /* instruction rewrites happen after this point */ + /* Instruction rewrites happen after this point. + * For offload target, finalize hook has all aux insn info, do any + * customized work there. + */ + if (ret == 0 && !bpf_jit_hardware_zext() && + !bpf_prog_is_dev_bound(env->prog->aux)) { + ret = opt_subreg_zext_lo32(env); + env->prog->aux->no_verifier_zext = !!ret; + } else { + env->prog->aux->no_verifier_zext = true; + } + if (is_priv) { if (ret == 0) opt_hard_wire_dead_code_branches(env); From patchwork Wed Apr 10 19:50:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1083586 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="gUZFEknY"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44fZXl4Ln7z9s5c for ; Thu, 11 Apr 2019 05:51:19 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726763AbfDJTux (ORCPT ); Wed, 10 Apr 2019 15:50:53 -0400 Received: from mail-wr1-f66.google.com ([209.85.221.66]:41252 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726723AbfDJTuw (ORCPT ); Wed, 10 Apr 2019 15:50:52 -0400 Received: by mail-wr1-f66.google.com with SMTP id r4so4344507wrq.8 for ; Wed, 10 Apr 2019 12:50:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=nd5AQSgpxrFUG35Pr+nHitfEvY8VkzebUPhOXd6teig=; b=gUZFEknY3MItC7K9kj8CFSBReHYGGcB7unrRewYMSdofmO0j9/N+ntkmbyN9KhKMqs Jfwpdf1dA7Xij5YNtgNUy5vDdL5TeoPMevFoW8VVCTqNjkozFMuxEZ+P0Ew6lKKZRpv4 igJGj+GKpLP4uG5g+ii6SVVDvk07c8BpCN3HW0QoZgivTyHcAsT3yBk3IP0bE02obPW9 Ol4PKyfmOBryL+FCQ0d0v8kMaKkaIC/FIcgIdW+ienmzFPXnHyNk7VvR2YdkvW0TmB/k IWRxIBHlbliJNCAheaBsJg+1JO3Yvjjo64v+m1/6WpjFPAtgmIT3rWx4/W0sMiSkvqCt IZBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=nd5AQSgpxrFUG35Pr+nHitfEvY8VkzebUPhOXd6teig=; b=klfoL+MNxvM9fvFUvF0Oghmn8XyVNz4Y9Y/D7xCXaAhV/2Gkf6Xs2hNPUMzjyHYRDb pThMEAo/nxrZ89kurG3wHrj2kiS+5IHAiqSQp6dApZGik5A7VgQiUZj7o03UgS6qX8nq Qwno9gx0ylH4qkXXiuFA11iJBKwZXHyFxP03xmAf5Db+HXiYY9Gsxsl0Led+zCeFhqNU UC4s58uD1+a48Mb1vEL11Rh4Uv+sTt1cIXzSlcCh7lPp/PsssjNz+rn7sWjMlccUNjVc 7HGEwtS0y6mjb8FlrlA4QWm2mqHIV4G6XRdLAQeNMhqXZoOuLflaDVPVzAOJziTxY7Yg +V6Q== X-Gm-Message-State: APjAAAVax+++yFTFsQ2AuOzlxgMq2lgUYi4PotRtQsje2CE38p64qCUC A4ayuau/BxVgVyaofsCXJv+i3Q== X-Google-Smtp-Source: APXvYqxofTBIjbpiccymicJb+TFhXC/Wg25mZw79v/mbL4+JIgE3MBVB1QvMEUvncZmiJjuqRgPKUA== X-Received: by 2002:adf:eac9:: with SMTP id o9mr27287657wrn.85.1554925849521; Wed, 10 Apr 2019 12:50:49 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id 4sm2288540wmg.12.2019.04.10.12.50.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 10 Apr 2019 12:50:48 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang Subject: [PATCH/RFC v2 bpf-next 09/19] bpf: introduce new bpf prog load flags "BPF_F_TEST_RND_HI32" Date: Wed, 10 Apr 2019 20:50:23 +0100 Message-Id: <1554925833-7333-10-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> References: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org x86_64 and AArch64 perhaps are two arches that running bpf testsuite frequently, however the zero extension insertion pass is not enabled for them because of their hardware support. It is critical to guarantee the pass correction as it is supposed to be enabled at default for a couple of other arches, for example PowerPC, SPARC, arm, NFP etc. Therefore, it would be very useful if there is a way to test this pass on for example x86_64. The test methodology employed by this set is "poisoning" useless bits. High 32-bit of a definition is randomized if it is identified as not used by any later instructions. Such randomization is only enabled under testing mode which is gated by the new bpf prog load flags "BPF_F_TEST_RND_HI32". Suggested-by: Alexei Starovoitov Signed-off-by: Jiong Wang --- include/uapi/linux/bpf.h | 18 ++++++++++++++++++ kernel/bpf/syscall.c | 4 +++- tools/include/uapi/linux/bpf.h | 18 ++++++++++++++++++ 3 files changed, 39 insertions(+), 1 deletion(-) diff --git a/include/uapi/linux/bpf.h b/include/uapi/linux/bpf.h index af1cbd9..f4214f8 100644 --- a/include/uapi/linux/bpf.h +++ b/include/uapi/linux/bpf.h @@ -256,6 +256,24 @@ enum bpf_attach_type { */ #define BPF_F_ANY_ALIGNMENT (1U << 1) +/* BPF_F_TEST_RND_HI32 is used in BPF_PROG_LOAD command for testing purpose. + * Verifier does sub-register def/use analysis and identifies instructions whose + * def only matters for low 32-bit, high 32-bit is never referenced later + * through implicit zero extension. Therefore verifier notifies JIT back-ends + * that it is safe to ignore clearing high 32-bit for these instructions. This + * saves some back-ends a lot of code-gen. However such optimization is not + * necessary on some arches, for example x86_64, arm64 etc, whose JIT back-ends + * hence hasn't used verifier's analysis result. But, we really want to have a + * way to be able to verify the correctness of the described optimization on + * x86_64 on which testsuites are frequently exercised. + * + * So, this flag is introduced. Once it is set, verifier will randomize high + * 32-bit for those instructions who has been identified as safe to ignore them. + * Then, if verifier is not doing correct analysis, such randomization will + * regress tests to expose bugs. + */ +#define BPF_F_TEST_RND_HI32 (1U << 2) + /* When BPF ldimm64's insn[0].src_reg != 0 then this can have * two extensions: * diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 438199e..225867c 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -1600,7 +1600,9 @@ static int bpf_prog_load(union bpf_attr *attr, union bpf_attr __user *uattr) if (CHECK_ATTR(BPF_PROG_LOAD)) return -EINVAL; - if (attr->prog_flags & ~(BPF_F_STRICT_ALIGNMENT | BPF_F_ANY_ALIGNMENT)) + if (attr->prog_flags & ~(BPF_F_STRICT_ALIGNMENT | + BPF_F_ANY_ALIGNMENT | + BPF_F_TEST_RND_HI32)) return -EINVAL; if (!IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) && diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h index af1cbd9..f4214f8 100644 --- a/tools/include/uapi/linux/bpf.h +++ b/tools/include/uapi/linux/bpf.h @@ -256,6 +256,24 @@ enum bpf_attach_type { */ #define BPF_F_ANY_ALIGNMENT (1U << 1) +/* BPF_F_TEST_RND_HI32 is used in BPF_PROG_LOAD command for testing purpose. + * Verifier does sub-register def/use analysis and identifies instructions whose + * def only matters for low 32-bit, high 32-bit is never referenced later + * through implicit zero extension. Therefore verifier notifies JIT back-ends + * that it is safe to ignore clearing high 32-bit for these instructions. This + * saves some back-ends a lot of code-gen. However such optimization is not + * necessary on some arches, for example x86_64, arm64 etc, whose JIT back-ends + * hence hasn't used verifier's analysis result. But, we really want to have a + * way to be able to verify the correctness of the described optimization on + * x86_64 on which testsuites are frequently exercised. + * + * So, this flag is introduced. Once it is set, verifier will randomize high + * 32-bit for those instructions who has been identified as safe to ignore them. + * Then, if verifier is not doing correct analysis, such randomization will + * regress tests to expose bugs. + */ +#define BPF_F_TEST_RND_HI32 (1U << 2) + /* When BPF ldimm64's insn[0].src_reg != 0 then this can have * two extensions: * From patchwork Wed Apr 10 19:50:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1083592 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="UIToFUT3"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44fZXt0Xysz9s5c for ; Thu, 11 Apr 2019 05:51:26 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726878AbfDJTvY (ORCPT ); Wed, 10 Apr 2019 15:51:24 -0400 Received: from mail-wm1-f65.google.com ([209.85.128.65]:54997 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726577AbfDJTux (ORCPT ); Wed, 10 Apr 2019 15:50:53 -0400 Received: by mail-wm1-f65.google.com with SMTP id c1so3891704wml.4 for ; Wed, 10 Apr 2019 12:50:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=VsnTnWMLFlcUEYVlg2aJmqMzdicYkQExuUgQyT6LC7Q=; b=UIToFUT33Cag+d5ieM6egGBLq64mL+OO92WAdi8TXVygnI/REzyE4l/7+OddK+/Ees LuEVWt3iKXzZJgeYDqQipRr2i1JNQ+pz7kCqitn578GjNZhIW3Rgctcqxgmsr/jHIrCM aDGD6sRjNQ81kd2NOequbYc6X5NQ8YdVj+odIh0DozXusa30pJp0dlpZSBHeCkfSgcTm QXa84BZXuuy2InQZd8x0kdQW1WoIzMWa7LEr90beU5+iQ4c14aeu0GIrht3KRdM8LTVJ NvgTfk7Ysm4TvIxRMwTY8t2yLwcJwrBrmecOJ8zMPvLQbZH0JnOcAiM07hTkxP4wTaNV P7Lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=VsnTnWMLFlcUEYVlg2aJmqMzdicYkQExuUgQyT6LC7Q=; b=gv3Nu//MW7VJqjU07ynkiNRfmhElqp7NmyabJi2ZYuZrKEx+imvEn208KH7PbO3XoF oLg+1jMP/0MbT6WZIjNfa5px5r5CL7FtpnOU/9+UBoBZEnOAU/MR0d/yLG/Mr8f5Xl4r E5MFgFE0fBfNoL2lVTTJSVmY0FgEKerQDB055WDKLCbKc2kJiUpPUS1wSCj0jYLPlDyM 0M1Y3UaKf3zREeyiVOZ29JOfdMgbkZVV2GYXiZHoCe8hIvuVjeVhjbB1pOkr8D4WTCYQ eLfIBWsmg1AJ/z5BHSbKaFpvw0GaLn7Mwny8xkF57l12p/PMixuIXlRHpvcSlOftLnNE u6gQ== X-Gm-Message-State: APjAAAXhEhr0ZlDh++r0CblNmxf5ek+8Qau9+z6a+6EMBtQ4FYcoSu0h tISUjFh2r3NC0mAElE5VMz4zVSXEwuw= X-Google-Smtp-Source: APXvYqyKhv/1zQgeNcdpNzY2gXQx+TV9yixGxlB0jCUiWWXrlZ+mjGtMQ/gcfMNdm2oTuVfp3us45A== X-Received: by 2002:a1c:480b:: with SMTP id v11mr3948485wma.25.1554925850973; Wed, 10 Apr 2019 12:50:50 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id 4sm2288540wmg.12.2019.04.10.12.50.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 10 Apr 2019 12:50:50 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang Subject: [PATCH/RFC v2 bpf-next 10/19] bpf: randomize high 32-bit when BPF_F_TEST_RND_HI32 is set Date: Wed, 10 Apr 2019 20:50:24 +0100 Message-Id: <1554925833-7333-11-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> References: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org This patch randomizes high 32-bit of a definition when BPF_F_TEST_RND_HI32 is set. It does this once the flag set no matter there is hardware zero extension support or not. Because this is a test feature and we want to deliver the most stressful test. Suggested-by: Alexei Starovoitov Signed-off-by: Jiong Wang --- kernel/bpf/verifier.c | 85 ++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 68 insertions(+), 17 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 9141a9a..33407c5 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -7520,24 +7520,70 @@ static int opt_remove_nops(struct bpf_verifier_env *env) return 0; } -static int opt_subreg_zext_lo32(struct bpf_verifier_env *env) +static int opt_subreg_zext_lo32_rnd_hi32(struct bpf_verifier_env *env, + const union bpf_attr *attr) { struct bpf_insn_aux_data orig_aux, *aux = env->insn_aux_data; + struct bpf_insn *patch, zext_patch[3], rnd_hi32_patch[4]; + int i, patch_len, delta = 0, len = env->prog->len; struct bpf_insn *insns = env->prog->insnsi; - int i, delta = 0, len = env->prog->len; - struct bpf_insn zext_patch[3]; struct bpf_prog *new_prog; + bool rnd_hi32; + + rnd_hi32 = attr->prog_flags & BPF_F_TEST_RND_HI32; zext_patch[1] = BPF_ALU64_IMM(BPF_LSH, 0, 32); zext_patch[2] = BPF_ALU64_IMM(BPF_RSH, 0, 32); + rnd_hi32_patch[1] = BPF_ALU64_IMM(BPF_MOV, BPF_REG_AX, 0); + rnd_hi32_patch[2] = BPF_ALU64_IMM(BPF_LSH, BPF_REG_AX, 32); + rnd_hi32_patch[3] = BPF_ALU64_REG(BPF_OR, 0, BPF_REG_AX); for (i = 0; i < len; i++) { int adj_idx = i + delta; struct bpf_insn insn; - if (!aux[adj_idx].zext_dst) + insn = insns[adj_idx]; + if (!aux[adj_idx].zext_dst) { + u8 code, class; + u32 imm_rnd; + + if (!rnd_hi32) + continue; + + code = insn.code; + class = BPF_CLASS(code); + /* Insns doesn't define any value. */ + if (class == BPF_JMP || class == BPF_JMP32 || + class == BPF_STX || class == BPF_ST) + continue; + + /* NOTE: arg "reg" is only used for BPF_STX, as it has + * been ruled out in above check, it is safe to + * pass NULL here. + */ + if (is_reg64(env, &insn, insn.dst_reg, NULL, DST_OP)) { + if (class == BPF_LD && + BPF_MODE(code) == BPF_IMM) + i++; + continue; + } + + /* ctx load could be transformed into wider load. */ + if (class == BPF_LDX && + aux[adj_idx].ptr_type == PTR_TO_CTX) + continue; + + imm_rnd = get_random_int(); + rnd_hi32_patch[0] = insns[adj_idx]; + rnd_hi32_patch[1].imm = imm_rnd; + rnd_hi32_patch[3].dst_reg = insn.dst_reg; + patch = rnd_hi32_patch; + patch_len = 4; + goto apply_patch_buffer; + } + + if (bpf_jit_hardware_zext()) continue; - insn = insns[adj_idx]; /* "adjust_insn_aux_data" only retains the original insn aux * data if insn at patched offset is at the end of the patch * buffer. That is to say, given the following insn sequence: @@ -7580,15 +7626,18 @@ static int opt_subreg_zext_lo32(struct bpf_verifier_env *env) zext_patch[0] = insns[adj_idx]; zext_patch[1].dst_reg = insn.dst_reg; zext_patch[2].dst_reg = insn.dst_reg; + patch = zext_patch; + patch_len = 3; +apply_patch_buffer: memcpy(&orig_aux, &aux[adj_idx], sizeof(orig_aux)); - new_prog = bpf_patch_insn_data(env, adj_idx, zext_patch, 3); + new_prog = bpf_patch_insn_data(env, adj_idx, patch, patch_len); if (!new_prog) return -ENOMEM; env->prog = new_prog; insns = new_prog->insnsi; aux = env->insn_aux_data; memcpy(&aux[adj_idx], &orig_aux, sizeof(orig_aux)); - delta += 2; + delta += patch_len - 1; } return 0; @@ -8425,16 +8474,18 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, if (ret == 0) ret = check_max_stack_depth(env); - /* Instruction rewrites happen after this point. - * For offload target, finalize hook has all aux insn info, do any - * customized work there. - */ - if (ret == 0 && !bpf_jit_hardware_zext() && - !bpf_prog_is_dev_bound(env->prog->aux)) { - ret = opt_subreg_zext_lo32(env); - env->prog->aux->no_verifier_zext = !!ret; - } else { - env->prog->aux->no_verifier_zext = true; + /* Instruction rewrites happen after this point. */ + if (ret == 0) { + if (bpf_prog_is_dev_bound(env->prog->aux)) { + /* For offload target, finalize hook has all aux insn + * info, copy the analysis result at there. + */ + env->prog->aux->no_verifier_zext = true; + } else { + ret = opt_subreg_zext_lo32_rnd_hi32(env, attr); + env->prog->aux->no_verifier_zext = + bpf_jit_hardware_zext() ? true : !!ret; + } } if (is_priv) { From patchwork Wed Apr 10 19:50:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1083587 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="NFBDMKjT"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44fZXm23QGz9s71 for ; Thu, 11 Apr 2019 05:51:20 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726106AbfDJTvT (ORCPT ); Wed, 10 Apr 2019 15:51:19 -0400 Received: from mail-wm1-f68.google.com ([209.85.128.68]:40350 "EHLO mail-wm1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726732AbfDJTux (ORCPT ); Wed, 10 Apr 2019 15:50:53 -0400 Received: by mail-wm1-f68.google.com with SMTP id z24so3917539wmi.5 for ; Wed, 10 Apr 2019 12:50:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=IovskK9UbwdYQWplzamPYQOHiU5bdz12oNCeyO+ZAtU=; b=NFBDMKjTmrBU7RXCjrdZC9VqzwdbLmTIssSxwtNKOMjmLgWDleSVeKTaLOr2F79b2T k6dd7dRhwqhlbkFSU3qfd36rVNRXzHueXOyVWaqCTw23dwRjktkNGirMt8YUVTOjht5v yjG4U0I/qUZ1UzBvjOQ4Ia7plVVmxYG0CSj4D8qnwWutDrw/SBLCdIjRMMhN5WJBJKYF h2HSsaYFU4MBQDZELm9APItUt8OvJHnsWjTeYW1KYPOIh6Drii+xGWOjXFB0WU8SBSu9 AMvzm1oMMWCAqe9pFQhAQDu8RZdHajcA5WiTOon7sWQU0pFoIGrQG+B70mrqc/Y6MIfL prpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=IovskK9UbwdYQWplzamPYQOHiU5bdz12oNCeyO+ZAtU=; b=hEN+HQPX2+1VD7s1Q+M9URavDH8c6yRyiPmsyU/+JQEi0N/4RGL/AVygPMlqmnmb7q EOXU3HWIX7vlDaKDni32e3aonGGEZPSaOOyvrSuw+eVRf4khPBPaLRu0HrRZOLtFwidC 7zcYeViyMowUOz9f4aRKFUQvDn2V1Gq/6wPvDzacVRCfTk95Lv3H0EXJLIGAxEmhX1/u yhfaMW4X0vQOI1MZJHfOzR/uk4MEO/tlAol3upk/b1CGA4I7B5BwDJV/t2Hen0Okftb+ fG9hR11VTrrLV20u41VvTVO6475Hr4Z1kofMAU5l1i7akPkrong2MsO4fbmn4D8T5POL WvHA== X-Gm-Message-State: APjAAAWHBdtGmdAiu+0E+0SCJpAb6mWqY8epIKWSRa3eJCa8kFG2OWSy 5Ue5UUoc9F7/K39CkBTVLtObQsG+0ng= X-Google-Smtp-Source: APXvYqyQuF15w1jbY2lUQIAGrXCZRlZ60ryAc6TeR0cgoHZWJWaqu5W+55/l+NCaSRg65w5ITi8oBg== X-Received: by 2002:a1c:9ad6:: with SMTP id c205mr4142916wme.46.1554925852002; Wed, 10 Apr 2019 12:50:52 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id 4sm2288540wmg.12.2019.04.10.12.50.51 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 10 Apr 2019 12:50:51 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang Subject: [PATCH/RFC v2 bpf-next 11/19] libbpf: new global variable "libbpf_test_mode" Date: Wed, 10 Apr 2019 20:50:25 +0100 Message-Id: <1554925833-7333-12-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> References: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> Sender: bpf-owner@vger.kernel.org Precedence: bulk List-Id: netdev.vger.kernel.org Enable BPF_F_TEST_RND_HI32 for all existing bpf selftests or other independent tests could involve quite a few changes to make sure all bpf prog load places has BPF_F_TEST_RND_HI32 set. Given most of the tests are using libbpf, this patch introduces a new global variable "libbpf_test_mode" into libbpf, once which is set, all bpf prog load issued through libbpf will have BPF_F_TEST_RND_HI32 set automatically, this could minimize changes required from testsuite. The other way might be introducing new load function like "bpf_prog_test_load", which will set BPF_F_TEST_RND_HI32. But there are several prog load APIs, and we need minor changes on some parameters. The global variable approach seems to be a proper first step for easy testsuite porting. Signed-off-by: Jiong Wang --- tools/lib/bpf/bpf.c | 4 ++++ tools/lib/bpf/libbpf.c | 2 ++ tools/lib/bpf/libbpf.h | 2 ++ tools/lib/bpf/libbpf.map | 1 + 4 files changed, 9 insertions(+) diff --git a/tools/lib/bpf/bpf.c b/tools/lib/bpf/bpf.c index dababce..0795a85 100644 --- a/tools/lib/bpf/bpf.c +++ b/tools/lib/bpf/bpf.c @@ -254,6 +254,8 @@ int bpf_load_program_xattr(const struct bpf_load_program_attr *load_attr, if (load_attr->name) memcpy(attr.prog_name, load_attr->name, min(strlen(load_attr->name), BPF_OBJ_NAME_LEN - 1)); + if (libbpf_test_mode) + attr.prog_flags |= BPF_F_TEST_RND_HI32; fd = sys_bpf_prog_load(&attr, sizeof(attr)); if (fd >= 0) @@ -350,6 +352,8 @@ int bpf_verify_program(enum bpf_prog_type type, const struct bpf_insn *insns, log_buf[0] = 0; attr.kern_version = kern_version; attr.prog_flags = prog_flags; + if (libbpf_test_mode) + attr.prog_flags |= BPF_F_TEST_RND_HI32; return sys_bpf_prog_load(&attr, sizeof(attr)); } diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index e2a457e..606643f 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -45,6 +45,8 @@ #include "str_error.h" #include "libbpf_util.h" +bool libbpf_test_mode; + #ifndef EM_BPF #define EM_BPF 247 #endif diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h index c5ff005..b40de66 100644 --- a/tools/lib/bpf/libbpf.h +++ b/tools/lib/bpf/libbpf.h @@ -447,6 +447,8 @@ bpf_program__bpil_addr_to_offs(struct bpf_prog_info_linear *info_linear); LIBBPF_API void bpf_program__bpil_offs_to_addr(struct bpf_prog_info_linear *info_linear); +LIBBPF_API extern bool libbpf_test_mode; + #ifdef __cplusplus } /* extern "C" */ #endif diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map index 6730017..5854e0b 100644 --- a/tools/lib/bpf/libbpf.map +++ b/tools/lib/bpf/libbpf.map @@ -156,6 +156,7 @@ LIBBPF_0.0.2 { bpf_program__get_prog_info_linear; bpf_program__bpil_addr_to_offs; bpf_program__bpil_offs_to_addr; + libbpf_test_mode; } LIBBPF_0.0.1; LIBBPF_0.0.3 { From patchwork Wed Apr 10 19:50:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1083570 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="X5pYrw8b"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44fZXJ2QJmz9s5c for ; Thu, 11 Apr 2019 05:50:56 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726600AbfDJTuz (ORCPT ); Wed, 10 Apr 2019 15:50:55 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:44522 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726756AbfDJTuz (ORCPT ); Wed, 10 Apr 2019 15:50:55 -0400 Received: by mail-wr1-f68.google.com with SMTP id y7so4331389wrn.11 for ; Wed, 10 Apr 2019 12:50:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=1CuzWmgTq2MAcAtGX/tvcBY53hTmNvJxVSfaUHWyquA=; b=X5pYrw8bOmRbRtiwaqQKbP3uewUhZW+C9qMqtZcZDcPKBa7ph4LTV2N2sIFpTyfojB ubh/mLd6ioG5muixOJnjV/0qKnsdDPqDcGeAHizToOGEcI1XDQeeuq1cbla9nalN1DVe oUsTd6tz2WOgWHMbUPm84O4XX/p0TIWnm4HXxd/B0oMGCtnUO9/oCahNkPHCnSNjUKcj am63XWbs5PYDymXsGaXgyJu//f+urYG8jfCSAwaiks7snbZS8laIrdpH9rsbm0IHKgYW QrkfJDn6pT3hfL0/AS6lG9FE2j8w92S1BJfTiewA2PWEYVUcd68Hmnz1nvtQRzmWYB4g rZkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=1CuzWmgTq2MAcAtGX/tvcBY53hTmNvJxVSfaUHWyquA=; b=GazQfr1/LlqJFSi/fHv/xxl60uaMNuZfCXr8yq4tGuWnxSqrEohV1wwIkBGZHEtO6C IIh5ILg6OII/mK681LqH+XMHUbdT0JrDhnpjxj/YWEsoTRSJ6nJ8VFfWCLMIsHmSeLbj lrAdxk4g3HsRHJu20c+WPutWSZI1zvxUl+EUqtWsziTZhw/I1GgxLg55pzl6v+cduZ+A jZyT926LoNcWoqI/IufLFjW2f925en7tfM2G1mYUcuUvuUoxza+2Dhe8emts+YDVRVff RqD/1jCOQ7BLSmr8RbzogqTtiGuw8E/eBHbUuJkeKR8IDC9iiszihMj2tzujS/MFC6IT MVsQ== X-Gm-Message-State: APjAAAUfscGpTKN3FMP37bquu9T3V7VceIHPKkL2nsTHB1MgFsXUDwuU sjWT2wiIhZMkkXtsInFgA8bIfw== X-Google-Smtp-Source: APXvYqzeB8oqBzQJrCYtGgUK14hRID04LPcswJ05fNfnJVKBRb5fL+4iIiHub3u0UNOBbVN8OoEC7A== X-Received: by 2002:a5d:698b:: with SMTP id g11mr29304797wru.65.1554925853091; Wed, 10 Apr 2019 12:50:53 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id 4sm2288540wmg.12.2019.04.10.12.50.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 10 Apr 2019 12:50:52 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang Subject: [PATCH/RFC v2 bpf-next 12/19] selftests: enable hi32 randomization for "test_progs" and "test_verifier" Date: Wed, 10 Apr 2019 20:50:26 +0100 Message-Id: <1554925833-7333-13-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> References: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> Sender: bpf-owner@vger.kernel.org Precedence: bulk List-Id: netdev.vger.kernel.org "test_progs" contains a couple of C based tests, so is a very good assurance on the pass correctness for this patch set. After this patch, BPF_F_TEST_RND_HI32 will be set for all of them and high 32-bit are randomized when possible. This could deliver quite stressful tests on x86_64. hi32 randomized is enabled for "test_verifier" as well. But two testcases needs adjustments due to 16-bit jump distance overflow. - "ld_abs: vlan + abs, test1" The function "bpf_fill_ld_abs_vlan_push_pop" inside test_verifier.c needs to use ALU64 to avoid insertion of hi32 randomization sequence that would overflow the sequence. - bpf_fill_jump_around_ld_abs needs to consider hi32 randomization to the load dst of ld_abs. Signed-off-by: Jiong Wang --- tools/testing/selftests/bpf/test_progs.c | 2 ++ tools/testing/selftests/bpf/test_verifier.c | 7 +++++-- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/bpf/test_progs.c b/tools/testing/selftests/bpf/test_progs.c index bf5c909..4249b5b 100644 --- a/tools/testing/selftests/bpf/test_progs.c +++ b/tools/testing/selftests/bpf/test_progs.c @@ -172,6 +172,8 @@ int main(int ac, char **av) if (ac == 2 && strcmp(av[1], "-s") == 0) verifier_stats = true; + libbpf_test_mode = true; + #define CALL #include #undef CALL diff --git a/tools/testing/selftests/bpf/test_verifier.c b/tools/testing/selftests/bpf/test_verifier.c index e2ebcad..7d067e0 100644 --- a/tools/testing/selftests/bpf/test_verifier.c +++ b/tools/testing/selftests/bpf/test_verifier.c @@ -161,7 +161,7 @@ static void bpf_fill_ld_abs_vlan_push_pop(struct bpf_test *self) goto loop; for (; i < len - 1; i++) - insn[i] = BPF_ALU32_IMM(BPF_MOV, BPF_REG_0, 0xbef); + insn[i] = BPF_ALU64_IMM(BPF_MOV, BPF_REG_0, 0xbef); insn[len - 1] = BPF_EXIT_INSN(); self->prog_len = len; } @@ -170,7 +170,7 @@ static void bpf_fill_jump_around_ld_abs(struct bpf_test *self) { struct bpf_insn *insn = self->fill_insns; /* jump range is limited to 16 bit. every ld_abs is replaced by 6 insns */ - unsigned int len = (1 << 15) / 6; + unsigned int len = (1 << 15) / 9; int i = 0; insn[i++] = BPF_MOV64_REG(BPF_REG_6, BPF_REG_1); @@ -1014,5 +1014,8 @@ int main(int argc, char **argv) } bpf_semi_rand_init(); + + libbpf_test_mode = true; + return do_test(unpriv, from, to); } From patchwork Wed Apr 10 19:50:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1083571 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="wSi9pcDZ"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44fZXJ6zXsz9s6w for ; Thu, 11 Apr 2019 05:50:56 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726756AbfDJTu4 (ORCPT ); Wed, 10 Apr 2019 15:50:56 -0400 Received: from mail-wr1-f67.google.com ([209.85.221.67]:43978 "EHLO mail-wr1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726770AbfDJTuz (ORCPT ); Wed, 10 Apr 2019 15:50:55 -0400 Received: by mail-wr1-f67.google.com with SMTP id k17so4340942wrx.10 for ; Wed, 10 Apr 2019 12:50:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=u7fZb3IEqFJRYe0oJUr+G1NSo0sv1IV8cJXsSnOcig0=; b=wSi9pcDZ9ct8tZUBGakzoQDEogC6W0cuDKRQAuhxv0zrh419r8IZ4hWLtsGIGKoNIr fyE/qUF7npgewhxHz3wJxZNRISTEXB+GlhyA1wYU1UGpNONkfRjf8MDEXpS5iiJFEqDm lfvEGknnGMz6ujpmMOdeNSN5Kv7q8bG4zW5VltOQQBUnScCRqiadkcJW7bevB3Huu8b6 FNYhQcI1ZrW+Yvnl7H7h5ojAQWRcN9Mf3ncwtXfSRRIzqBsUGBLIVvmCoA6vHZepDMEq e5Ir+J9wD+g/E/Gq8GWqG3jSuTRxM1AZvaamoF22KzM2cXaY0A/uP5LHMkUwwvOPJa70 SIEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=u7fZb3IEqFJRYe0oJUr+G1NSo0sv1IV8cJXsSnOcig0=; b=fN0isPseJkZTSOA6tw95IrsgnMS2b+oIFuNNP5vtPrE0Kx1ydhhBpq2nc/8F1xs8zB 9VrRO3eans5nYuyeUM6r8Dg+URB9vhBTwUvT+6VBiGZqqls6S+ZB1HjhhbKM9qWKeHSv uy7DBV3/lzAppWF6UKdEjjReDopWBnlG4rJjPH25JRY5NV/fuPPyb2PdgVwzoSRJ9jyA TM44lnz/6ffWmwFXxMluMk8wdqtZ6lAf1bHPG+ZkdA+9sHf6UPgzTMgJvczYgR2FmCsw LO9njMdIhbtKgOPZ8qLvAnLCfpZx+hqj0XMJdfQzzKkczDhG9gMmm2neE1j33dskwaPf 2qUQ== X-Gm-Message-State: APjAAAWr0A50KT58xyKcves6nFM1Aix5AU7sekiPN5re+EHODdYOEQyv +fDBUh7IP3jtoGyQwQ1SuNiZbg== X-Google-Smtp-Source: APXvYqzmZpmZZgoZ+HZAm+o/4Yjr4e9rlpoh1OaqlVxpynFlC/qhaxEJ32OR/EutnnVGgjbYUPvijg== X-Received: by 2002:a05:6000:1152:: with SMTP id d18mr27486531wrx.32.1554925854086; Wed, 10 Apr 2019 12:50:54 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id 4sm2288540wmg.12.2019.04.10.12.50.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 10 Apr 2019 12:50:53 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang , Shubham Bansal Subject: [PATCH/RFC v2 bpf-next 13/19] arm: bpf: eliminate zero extension code-gen Date: Wed, 10 Apr 2019 20:50:27 +0100 Message-Id: <1554925833-7333-14-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> References: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> Sender: bpf-owner@vger.kernel.org Precedence: bulk List-Id: netdev.vger.kernel.org Cc: Shubham Bansal Signed-off-by: Jiong Wang --- arch/arm/net/bpf_jit_32.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/arch/arm/net/bpf_jit_32.c b/arch/arm/net/bpf_jit_32.c index c8bfbbf..8cecd06 100644 --- a/arch/arm/net/bpf_jit_32.c +++ b/arch/arm/net/bpf_jit_32.c @@ -736,7 +736,8 @@ static inline void emit_a32_alu_r64(const bool is64, const s8 dst[], /* ALU operation */ emit_alu_r(rd[1], rs, true, false, op, ctx); - emit_a32_mov_i(rd[0], 0, ctx); + if (ctx->prog->aux->no_verifier_zext) + emit_a32_mov_i(rd[0], 0, ctx); } arm_bpf_put_reg64(dst, rd, ctx); @@ -758,8 +759,9 @@ static inline void emit_a32_mov_r64(const bool is64, const s8 dst[], struct jit_ctx *ctx) { if (!is64) { emit_a32_mov_r(dst_lo, src_lo, ctx); - /* Zero out high 4 bytes */ - emit_a32_mov_i(dst_hi, 0, ctx); + if (ctx->prog->aux->no_verifier_zext) + /* Zero out high 4 bytes */ + emit_a32_mov_i(dst_hi, 0, ctx); } else if (__LINUX_ARM_ARCH__ < 6 && ctx->cpu_architecture < CPU_ARCH_ARMv5TE) { /* complete 8 byte move */ @@ -1438,7 +1440,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx) } emit_udivmod(rd_lo, rd_lo, rt, ctx, BPF_OP(code)); arm_bpf_put_reg32(dst_lo, rd_lo, ctx); - emit_a32_mov_i(dst_hi, 0, ctx); + if (ctx->prog->aux->no_verifier_zext) + emit_a32_mov_i(dst_hi, 0, ctx); break; case BPF_ALU64 | BPF_DIV | BPF_K: case BPF_ALU64 | BPF_DIV | BPF_X: @@ -1453,7 +1456,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx) return -EINVAL; if (imm) emit_a32_alu_i(dst_lo, imm, ctx, BPF_OP(code)); - emit_a32_mov_i(dst_hi, 0, ctx); + if (ctx->prog->aux->no_verifier_zext) + emit_a32_mov_i(dst_hi, 0, ctx); break; /* dst = dst << imm */ case BPF_ALU64 | BPF_LSH | BPF_K: @@ -1488,7 +1492,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx) /* dst = ~dst */ case BPF_ALU | BPF_NEG: emit_a32_alu_i(dst_lo, 0, ctx, BPF_OP(code)); - emit_a32_mov_i(dst_hi, 0, ctx); + if (ctx->prog->aux->no_verifier_zext) + emit_a32_mov_i(dst_hi, 0, ctx); break; /* dst = ~dst (64 bit) */ case BPF_ALU64 | BPF_NEG: @@ -1838,6 +1843,11 @@ void bpf_jit_compile(struct bpf_prog *prog) /* Nothing to do here. We support Internal BPF. */ } +bool bpf_jit_hardware_zext(void) +{ + return false; +} + struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) { struct bpf_prog *tmp, *orig_prog = prog; From patchwork Wed Apr 10 19:50:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1083584 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="HzZ/RMLb"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44fZXh6sjTz9s6w for ; Thu, 11 Apr 2019 05:51:16 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726215AbfDJTvP (ORCPT ); Wed, 10 Apr 2019 15:51:15 -0400 Received: from mail-wr1-f67.google.com ([209.85.221.67]:37923 "EHLO mail-wr1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726630AbfDJTu4 (ORCPT ); Wed, 10 Apr 2019 15:50:56 -0400 Received: by mail-wr1-f67.google.com with SMTP id k11so4366463wro.5 for ; Wed, 10 Apr 2019 12:50:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=FyyxR9Iqk6cAH0oBjCaJb0bcVV0NGMOOW5Oun5gFHdM=; b=HzZ/RMLbRR0LlDWzLnkZuiW8/E4JzLPukqHbMowxvtAH3ngpu8hPjb2RrOLrSv1Zzz w3bnZ9bplhvqqpT2vV+cf4cqUi7D7QkyFZvtxdXXjEGdyzLgymBA/fBTWvoF+zEAY3Ed +tINQH6XYx7+pgTtDEaUNbvphEFWz2tDq8dUaRHu8KrfaYAcL3WqjrzT2odP8eAgOn1L vYPo6kz9EvCe7rasm7djKef9Q0y0nOpB4PYQXV+FzcyCaSq1GOraYf9MNEtpyTu0XFcy j+UwxBRfDt2ggWnJeu0YcCFmwrlsn4lzxGrQXn+iGfxAFL8kwBrR0qDBuaAtRXUPjzmY 1B+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=FyyxR9Iqk6cAH0oBjCaJb0bcVV0NGMOOW5Oun5gFHdM=; b=ts4kS5TdlO8kdaGXq6mI6HSpgG9TNHr6nBzWR0Pw9NCJSIF8zGaWD3mL5cLYc/q6Yo 7sC6r0SyW/aGM4bsvGRb6Yme0IdLp/NcPHPUgPevOAFU27XWj6mg27ZPoMt0TgLPtJaP +ABWWM3zT1O4PFt3L0EVww7VnSonE3EU2NhyE9JxNA2VmoAhi7Rhx50cvk4YwjbkNN1i 0X5w+UkkTIFvPbecaYCfTouSOxWcOfYKBQePcxKbllOnNHhr3e+MZKrPEYSFrD+X+whb 09eY2MlY3+PJXVueEtXmUbDI89DrT5WxMTNaevGuaENOvS+EmgLaBXUjgklQYwsqD7Vu Aozg== X-Gm-Message-State: APjAAAVquDLUVvE4FVZ0Js879W7Sc9m4gF6i3SSN+LAsDMNNDOd3B/IA HBVFStJIBbX43NBLKpXYwgFtuw== X-Google-Smtp-Source: APXvYqzxXf6ZJEMOzZFrLqKmI0tx2yu77e9UcA0swTVWCzqiYYMtnFBeClEJ9Ap7k5/MJnMoJ1qb6A== X-Received: by 2002:adf:f30a:: with SMTP id i10mr27949031wro.111.1554925854993; Wed, 10 Apr 2019 12:50:54 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id 4sm2288540wmg.12.2019.04.10.12.50.54 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 10 Apr 2019 12:50:54 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang , "Naveen N . Rao" , Sandipan Das Subject: [PATCH/RFC v2 bpf-next 14/19] powerpc: bpf: eliminate zero extension code-gen Date: Wed, 10 Apr 2019 20:50:28 +0100 Message-Id: <1554925833-7333-15-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> References: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> Sender: bpf-owner@vger.kernel.org Precedence: bulk List-Id: netdev.vger.kernel.org Cc: Naveen N. Rao Cc: Sandipan Das Signed-off-by: Jiong Wang --- arch/powerpc/net/bpf_jit_comp64.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c index 21a1dcd..d10621b 100644 --- a/arch/powerpc/net/bpf_jit_comp64.c +++ b/arch/powerpc/net/bpf_jit_comp64.c @@ -559,7 +559,7 @@ static int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, bpf_alu32_trunc: /* Truncate to 32-bits */ - if (BPF_CLASS(code) == BPF_ALU) + if (BPF_CLASS(code) == BPF_ALU && fp->aux->no_verifier_zext) PPC_RLWINM(dst_reg, dst_reg, 0, 0, 31); break; @@ -1046,6 +1046,11 @@ struct powerpc64_jit_data { struct codegen_context ctx; }; +bool bpf_jit_hardware_zext(void) +{ + return false; +} + struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) { u32 proglen; From patchwork Wed Apr 10 19:50:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1083582 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="CauEe20b"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44fZXd5L9Sz9s5c for ; Thu, 11 Apr 2019 05:51:13 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726856AbfDJTvN (ORCPT ); Wed, 10 Apr 2019 15:51:13 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:37156 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726784AbfDJTu6 (ORCPT ); Wed, 10 Apr 2019 15:50:58 -0400 Received: by mail-wr1-f68.google.com with SMTP id w10so4369104wrm.4 for ; Wed, 10 Apr 2019 12:50:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=/wylRhv97waN1hLNQRjtw5HFd7aLKXwfbfkeZTfcGns=; b=CauEe20bVPVPu+dhpUHLI0ywZsDC1Ms1bKy0bR2cIZpEtQjjscwA9urHJkNu57aJV+ 3we51tgocA2matiPupOzyt9K1J1YoycRqSSS3vA1nOsftesGBnKeEavX1MGosJvXLeXU oAW9tTd1iEvWpZO1Y1hJx8PO5a1J2x/g9TdpTQ9PTbdohZofPnpbmLODhQrOQFCEAFmK qT7o8eWYf3BXgSb1vccPkacDnmJju+pzcqFaULdLMS+2A1LMrGJAUBDGNbSIbgiqGobS ETS6nwOGj2AhVQaOi/BunTPuXiCkeHgW2JfApF373nhD+pkwuTKskQdXi+bnPHG29By7 61iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=/wylRhv97waN1hLNQRjtw5HFd7aLKXwfbfkeZTfcGns=; b=QjDhVsJzyNlsg5nBLzidKZoONWF5YU2+S/vfsUt/t/Yng5K+OYjWAj1N1bXSOCU8Nr FEKiu+nMHiBB8FUytG6ypva06Icl7ZGBL6JvQk5onLupflAJdxngyBLTBvpHY91gBdZf HqDiPjBk6zvAF9McCTBH7YK23JFdLFJBOdUrsKZDTBqaLgEaOvsp46I70X2c4LO1O5dL 5ZV0nfl90uWXaqz0jwIzqigYT3mueI47Ico4wR9dTnVTiPRyd6G2s+uymWXWfilLJ3AY z67ug5ZMjVnI/XNG1aYYTTqn7aHOoiLnrlfJzHdjlfBsBa+jXjsqp/30Ho86StIz82l9 JWYQ== X-Gm-Message-State: APjAAAVyIqXXCaQlTA58Jbubb2g+yVcu9Yo5X3t54YByVYmdyKGHrApK pK5mBpI9fLY6MA7DtLxmLuaJBw== X-Google-Smtp-Source: APXvYqzJyHpIKxagu00hYmm6kuJ65OpcVvnScKF15xs1KyaLICKK9F+zdy0J9U1OJIr1KAEqw5QEig== X-Received: by 2002:adf:82c9:: with SMTP id 67mr6586210wrc.261.1554925856263; Wed, 10 Apr 2019 12:50:56 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id 4sm2288540wmg.12.2019.04.10.12.50.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 10 Apr 2019 12:50:55 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang , Martin Schwidefsky , Heiko Carstens Subject: [PATCH/RFC v2 bpf-next 15/19] s390: bpf: eliminate zero extension code-gen Date: Wed, 10 Apr 2019 20:50:29 +0100 Message-Id: <1554925833-7333-16-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> References: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> Sender: bpf-owner@vger.kernel.org Precedence: bulk List-Id: netdev.vger.kernel.org Cc: Martin Schwidefsky Cc: Heiko Carstens Signed-off-by: Jiong Wang --- arch/s390/net/bpf_jit_comp.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/arch/s390/net/bpf_jit_comp.c b/arch/s390/net/bpf_jit_comp.c index 51dd026..59592d7 100644 --- a/arch/s390/net/bpf_jit_comp.c +++ b/arch/s390/net/bpf_jit_comp.c @@ -299,9 +299,11 @@ static inline void reg_set_seen(struct bpf_jit *jit, u32 b1) #define EMIT_ZERO(b1) \ ({ \ - /* llgfr %dst,%dst (zero extend to 64 bit) */ \ - EMIT4(0xb9160000, b1, b1); \ - REG_SET_SEEN(b1); \ + if (fp->aux->no_verifier_zext) { \ + /* llgfr %dst,%dst (zero extend to 64 bit) */ \ + EMIT4(0xb9160000, b1, b1); \ + REG_SET_SEEN(b1); \ + } \ }) /* @@ -1282,6 +1284,11 @@ static int bpf_jit_prog(struct bpf_jit *jit, struct bpf_prog *fp) return 0; } +bool bpf_jit_hardware_zext(void) +{ + return false; +} + /* * Compile eBPF program "fp" */ From patchwork Wed Apr 10 19:50:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1083574 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="h+f/MmuT"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44fZXP3yCNz9s6w for ; Thu, 11 Apr 2019 05:51:01 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726825AbfDJTvA (ORCPT ); Wed, 10 Apr 2019 15:51:00 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:32903 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726794AbfDJTu7 (ORCPT ); Wed, 10 Apr 2019 15:50:59 -0400 Received: by mail-wr1-f68.google.com with SMTP id q1so4408095wrp.0 for ; Wed, 10 Apr 2019 12:50:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=Pi+jMPgfzj2fBkAwzCkG0vyOGnD4cZfJGlYpVzkjSMY=; b=h+f/MmuThlGCO+2gwqFUFZCTLfRc1rF9K0PyqVEz48+Ix/qTjIg7rIOkqThqQpvZlc fkmBmNhRP2Bagpwud+TsggBmi+ZkKOLlKN/sDeAxCl5iJMc27nv5NLAWSlO1KTBuu5yi YvoB387W32CtwlUvthLbwSb4hYc2aOzKRTzmSbhTAF9oIRFZr9W4AFD+L8LwwD3gY+mw mhdFKWMtnoaaIubd1Pkf7hRkUe+ypgr0BgQWeVW/0unI96UtZQllfpM7Cuo7XmVKJg9F rztcWiqgxbXxYMMcEnC4XDAmnTJHiQVEV4GlM80ozdhax+2WPNFiI/AmldOANrwOCPa5 iEnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=Pi+jMPgfzj2fBkAwzCkG0vyOGnD4cZfJGlYpVzkjSMY=; b=UqC1VqpFovbsSD4XD+UWc+GKYpYyYhE2aUe0B3PwtlHZf0kWn87sQNrArqYZ6QKfx/ uMhI4mXBjr8ZIgEYXh5zcdCvtPvM5ynokoNCDg8KjqX84si+n1wxAW3d0Uf/YFBGLTUZ 8blScHEiV/UjYcQNi5qyql98Y8/VIfVCkuNuxLCn9tnEAU9F5+dLNNi6UJKA5E6Oshm3 PR/zMK24yt8iTdwZHbfGUILtXlxxOTdgceo1JJ+FUI7dVGrLI188/kam2sHt47qi7HmE 7e8kGDm3yNgAOasP4X1lm/GVtDC3y3BTqj4IEq6u79Y7XllURITa9AV/2gP1EHW8RFt5 y6zg== X-Gm-Message-State: APjAAAWFUlL0GQDmRfCfNGGIye3WZnu5snDQkE0ZGTK6jq7TzVQrmatn NcYWX9YQfeopd511x7F94+3qUNZ4UEM= X-Google-Smtp-Source: APXvYqx7OlAOpGFjhioIJ3r9IhMKxbffKlpC/zAwgt+4zJpmqshyZ6qiyxXEYRmGKwdrkv5aqmTJmg== X-Received: by 2002:adf:dd82:: with SMTP id x2mr18043501wrl.214.1554925857247; Wed, 10 Apr 2019 12:50:57 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id 4sm2288540wmg.12.2019.04.10.12.50.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 10 Apr 2019 12:50:56 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang , "David S . Miller" Subject: [PATCH/RFC v2 bpf-next 16/19] sparc: bpf: eliminate zero extension code-gen Date: Wed, 10 Apr 2019 20:50:30 +0100 Message-Id: <1554925833-7333-17-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> References: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Cc: David S. Miller Signed-off-by: Jiong Wang --- arch/sparc/net/bpf_jit_comp_64.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/arch/sparc/net/bpf_jit_comp_64.c b/arch/sparc/net/bpf_jit_comp_64.c index 65428e7..e58f84e 100644 --- a/arch/sparc/net/bpf_jit_comp_64.c +++ b/arch/sparc/net/bpf_jit_comp_64.c @@ -1144,7 +1144,8 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx) break; do_alu32_trunc: - if (BPF_CLASS(code) == BPF_ALU) + if (BPF_CLASS(code) == BPF_ALU && + ctx->prog->aux->no_verifier_zext) emit_alu_K(SRL, dst, 0, ctx); break; @@ -1432,6 +1433,11 @@ static void jit_fill_hole(void *area, unsigned int size) *ptr++ = 0x91d02005; /* ta 5 */ } +bool bpf_jit_hardware_zext(void) +{ + return false; +} + struct sparc64_jit_data { struct bpf_binary_header *header; u8 *image; From patchwork Wed Apr 10 19:50:31 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1083575 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="LGvY/WNf"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44fZXS2ZYMz9s7h for ; Thu, 11 Apr 2019 05:51:04 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726805AbfDJTvC (ORCPT ); Wed, 10 Apr 2019 15:51:02 -0400 Received: from mail-wm1-f66.google.com ([209.85.128.66]:39621 "EHLO mail-wm1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726771AbfDJTvC (ORCPT ); Wed, 10 Apr 2019 15:51:02 -0400 Received: by mail-wm1-f66.google.com with SMTP id n25so3892747wmk.4 for ; Wed, 10 Apr 2019 12:50:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=esldtJjtJ7LjAyiBdO+8AWliqTz0U0S4jG86huzM9ss=; b=LGvY/WNfIAH9sRfs5hSkFbE1cY9j2REUkmBhLePzIQJoM6K2/B4SIBJopiW0xEpbDm LRfYVvHorGRMnyg25A6UGyV8cHoE9X7oVtSDtlyzg6P2t/aOjOKTH/u/13URaE4JZ5Nt imGiLK5Mlq495TFuRrATG+anw5vZgy9+1vxqapn7XYZAC0ADGKJ4ykl2RWIuvP9s0zL6 HeQQxN2jOE6O4mFfCG2lXC5orHG212GgrbGD4DsGDEDGrcTjv3eJPEzcJ+mdsERB2Vx7 GIQKjf/p1Jktu4N/frv26YDRcFUapU1Whub8mDQJnWhaqFRCjtdo9u80s5iKekxDfxhK Xpng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=esldtJjtJ7LjAyiBdO+8AWliqTz0U0S4jG86huzM9ss=; b=DD96w6wgUN2FMYHOev02KU5Tunr1sIHvXciQbJU8EUfbbj4xd0Endj2oZjTocIBFoh y/YfLxJ3a7VQlYc5IHbMbsgoAwzqnPST1dvikinh0X8YOWby9Pivk4wFABwPEMhUiMYl oBb8dw8M1fR1nUi2/+N6fHAuOcYWIiwNQ7JyGSM1C4oL8+WhyPclsqqnYKMfr9GWRc9E lpK6jfxHnvUK1Mon1EgHBhBGS8xeHvXF4pID9d87JbUko1rFPcSAlG/RhvsH3z6pSn6e lrE2EfmghahUtkY5VFts4cwcAQx9gS239L4PZK1gGSXmRKOmKmjUqa0XJoVcOSLYrAta uMxg== X-Gm-Message-State: APjAAAVOKn4wTGXHl1zoe7XVJXpNYSxrGl22FHjoaqL4KWki8CgY9iz5 r0l0xHJWXIztey71j07qdvAcvg== X-Google-Smtp-Source: APXvYqzOQy3zlytD7JrLCciUOS41aptEdh8Xs8/g7bLzwzTinWCGmGY56aSX2cOfyj/DFqC58izafA== X-Received: by 2002:a1c:ca07:: with SMTP id a7mr3951256wmg.104.1554925858673; Wed, 10 Apr 2019 12:50:58 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id 4sm2288540wmg.12.2019.04.10.12.50.57 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 10 Apr 2019 12:50:57 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang , Wang YanQing Subject: [PATCH/RFC v2 bpf-next 17/19] x32: bpf: eliminate zero extension code-gen Date: Wed, 10 Apr 2019 20:50:31 +0100 Message-Id: <1554925833-7333-18-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> References: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> Sender: bpf-owner@vger.kernel.org Precedence: bulk List-Id: netdev.vger.kernel.org Cc: Wang YanQing Signed-off-by: Jiong Wang --- arch/x86/net/bpf_jit_comp32.c | 32 ++++++++++++++++++++++---------- 1 file changed, 22 insertions(+), 10 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp32.c b/arch/x86/net/bpf_jit_comp32.c index 0d9cdff..8c6cf22 100644 --- a/arch/x86/net/bpf_jit_comp32.c +++ b/arch/x86/net/bpf_jit_comp32.c @@ -567,7 +567,7 @@ static inline void emit_ia32_alu_r(const bool is64, const bool hi, const u8 op, static inline void emit_ia32_alu_r64(const bool is64, const u8 op, const u8 dst[], const u8 src[], bool dstk, bool sstk, - u8 **pprog) + u8 **pprog, const struct bpf_prog_aux *aux) { u8 *prog = *pprog; @@ -575,7 +575,7 @@ static inline void emit_ia32_alu_r64(const bool is64, const u8 op, if (is64) emit_ia32_alu_r(is64, true, op, dst_hi, src_hi, dstk, sstk, &prog); - else + else if (aux->no_verifier_zext) emit_ia32_mov_i(dst_hi, 0, dstk, &prog); *pprog = prog; } @@ -666,7 +666,8 @@ static inline void emit_ia32_alu_i(const bool is64, const bool hi, const u8 op, /* ALU operation (64 bit) */ static inline void emit_ia32_alu_i64(const bool is64, const u8 op, const u8 dst[], const u32 val, - bool dstk, u8 **pprog) + bool dstk, u8 **pprog, + const struct bpf_prog_aux *aux) { u8 *prog = *pprog; u32 hi = 0; @@ -677,7 +678,7 @@ static inline void emit_ia32_alu_i64(const bool is64, const u8 op, emit_ia32_alu_i(is64, false, op, dst_lo, val, dstk, &prog); if (is64) emit_ia32_alu_i(is64, true, op, dst_hi, hi, dstk, &prog); - else + else if (aux->no_verifier_zext) emit_ia32_mov_i(dst_hi, 0, dstk, &prog); *pprog = prog; @@ -1690,11 +1691,13 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, switch (BPF_SRC(code)) { case BPF_X: emit_ia32_alu_r64(is64, BPF_OP(code), dst, - src, dstk, sstk, &prog); + src, dstk, sstk, &prog, + bpf_prog->aux); break; case BPF_K: emit_ia32_alu_i64(is64, BPF_OP(code), dst, - imm32, dstk, &prog); + imm32, dstk, &prog, + bpf_prog->aux); break; } break; @@ -1713,7 +1716,8 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, false, &prog); break; } - emit_ia32_mov_i(dst_hi, 0, dstk, &prog); + if (bpf_prog->aux->no_verifier_zext) + emit_ia32_mov_i(dst_hi, 0, dstk, &prog); break; case BPF_ALU | BPF_LSH | BPF_X: case BPF_ALU | BPF_RSH | BPF_X: @@ -1733,7 +1737,8 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, &prog); break; } - emit_ia32_mov_i(dst_hi, 0, dstk, &prog); + if (bpf_prog->aux->no_verifier_zext) + emit_ia32_mov_i(dst_hi, 0, dstk, &prog); break; /* dst = dst / src(imm) */ /* dst = dst % src(imm) */ @@ -1755,7 +1760,8 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, &prog); break; } - emit_ia32_mov_i(dst_hi, 0, dstk, &prog); + if (bpf_prog->aux->no_verifier_zext) + emit_ia32_mov_i(dst_hi, 0, dstk, &prog); break; case BPF_ALU64 | BPF_DIV | BPF_K: case BPF_ALU64 | BPF_DIV | BPF_X: @@ -1772,7 +1778,8 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, EMIT2_off32(0xC7, add_1reg(0xC0, IA32_ECX), imm32); emit_ia32_shift_r(BPF_OP(code), dst_lo, IA32_ECX, dstk, false, &prog); - emit_ia32_mov_i(dst_hi, 0, dstk, &prog); + if (bpf_prog->aux->no_verifier_zext) + emit_ia32_mov_i(dst_hi, 0, dstk, &prog); break; /* dst = dst << imm */ case BPF_ALU64 | BPF_LSH | BPF_K: @@ -2367,6 +2374,11 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, return proglen; } +bool bpf_jit_hardware_zext(void) +{ + return false; +} + struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) { struct bpf_binary_header *header = NULL; From patchwork Wed Apr 10 19:50:32 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1083577 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="KndacB3E"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44fZXW0FSKz9s5c for ; Thu, 11 Apr 2019 05:51:07 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726838AbfDJTvD (ORCPT ); Wed, 10 Apr 2019 15:51:03 -0400 Received: from mail-wr1-f66.google.com ([209.85.221.66]:34376 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726818AbfDJTvB (ORCPT ); Wed, 10 Apr 2019 15:51:01 -0400 Received: by mail-wr1-f66.google.com with SMTP id p10so4393495wrq.1 for ; Wed, 10 Apr 2019 12:51:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pNHJxDIKLsiKxKEQK4Ngr2CSz1DZ9y91uELS2TzjMyw=; b=KndacB3EVhZdtL4oZL25Q+OFwLxZkt7XMgADHnAsSA6Li9kngdynCsFtworNbmu76e 2V+evGu9fKk3I77SXzfCYqIUpS9qjfHMbKmvNK3NF6Py03iekZiFAoSJJl6jB+LHD4/y RQTI8CbeavWRVTAB40XHjQl56VMLwILbTpE/3o3aTQmP0EvF5VXFnqILfOEs26mNfnkB 78f4icwTzYbnYnyKn1xzKI7PV9NKXafHhCUAsmom/spUyT+Xu2um5d1lnJTPgMg81Svk mh0SlajMPMGzcOn/nCXI3bWHXU73x5bZgWIGfsydSCb+WO/dh/9cE1BzLn5dcYBFBpvI 6rWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pNHJxDIKLsiKxKEQK4Ngr2CSz1DZ9y91uELS2TzjMyw=; b=ZXL6XlXdD/pSbpydRZUD6vnv5S5CVldhoD6NiIVe9n9k/hrY19KZ4I13AUQjOxvmVn 5086vJ1ek1fBEK/Hpdy9WhMBnlgLvP+wDekMCJh2jnI7VSusnFuCbWnAXxXEmEcISehn +6JMom9+t54bI5FWhrnehvQiFCZnfnSIpfJzjClm+mUhGudarFVlz0OIa0u011wWa8cN HXbD5sRs+hV62rpUXhEKLu2YRHNPo1WxGPCo8yT5eDZQaVxzcFddqjHoWC80tlvG7wo9 vc1AQ4nbplxCwPVCFX02ljFQUZ3Z3XivXZJHQccD54z385FmU92wPJngfQ91/j8J0EU2 g5zA== X-Gm-Message-State: APjAAAWcU0iYkeYzbO3iHSPTn4V4UHyK6IcbFIvUG4dGQoYBSBY1FoWA gFU5HIyHJTRNM2irc/dZii0Ygg== X-Google-Smtp-Source: APXvYqzvwBAXCrCCNRmBXEdCKHyl7bHAS1MOIuh25bhUL7rTkL0qD7skeRJ0uK/OPezXFD7dt6Iw3w== X-Received: by 2002:adf:afd7:: with SMTP id y23mr27416995wrd.254.1554925859493; Wed, 10 Apr 2019 12:50:59 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id 4sm2288540wmg.12.2019.04.10.12.50.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 10 Apr 2019 12:50:58 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= Subject: [PATCH/RFC v2 bpf-next 18/19] riscv: bpf: eliminate zero extension code-gen Date: Wed, 10 Apr 2019 20:50:32 +0100 Message-Id: <1554925833-7333-19-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> References: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org CC: Björn Töpel Signed-off-by: Jiong Wang --- arch/riscv/net/bpf_jit_comp.c | 32 +++++++++++++++++++------------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/arch/riscv/net/bpf_jit_comp.c b/arch/riscv/net/bpf_jit_comp.c index 80b12aa..9cba262 100644 --- a/arch/riscv/net/bpf_jit_comp.c +++ b/arch/riscv/net/bpf_jit_comp.c @@ -731,6 +731,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, { bool is64 = BPF_CLASS(insn->code) == BPF_ALU64 || BPF_CLASS(insn->code) == BPF_JMP; + struct bpf_prog_aux *aux = ctx->prog->aux; int rvoff, i = insn - ctx->prog->insnsi; u8 rd = -1, rs = -1, code = insn->code; s16 off = insn->off; @@ -743,7 +744,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, case BPF_ALU | BPF_MOV | BPF_X: case BPF_ALU64 | BPF_MOV | BPF_X: emit(is64 ? rv_addi(rd, rs, 0) : rv_addiw(rd, rs, 0), ctx); - if (!is64) + if (!is64 && aux->no_verifier_zext) emit_zext_32(rd, ctx); break; @@ -771,19 +772,19 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, case BPF_ALU | BPF_MUL | BPF_X: case BPF_ALU64 | BPF_MUL | BPF_X: emit(is64 ? rv_mul(rd, rd, rs) : rv_mulw(rd, rd, rs), ctx); - if (!is64) + if (!is64 && aux->no_verifier_zext) emit_zext_32(rd, ctx); break; case BPF_ALU | BPF_DIV | BPF_X: case BPF_ALU64 | BPF_DIV | BPF_X: emit(is64 ? rv_divu(rd, rd, rs) : rv_divuw(rd, rd, rs), ctx); - if (!is64) + if (!is64 && aux->no_verifier_zext) emit_zext_32(rd, ctx); break; case BPF_ALU | BPF_MOD | BPF_X: case BPF_ALU64 | BPF_MOD | BPF_X: emit(is64 ? rv_remu(rd, rd, rs) : rv_remuw(rd, rd, rs), ctx); - if (!is64) + if (!is64 && aux->no_verifier_zext) emit_zext_32(rd, ctx); break; case BPF_ALU | BPF_LSH | BPF_X: @@ -867,7 +868,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, case BPF_ALU | BPF_MOV | BPF_K: case BPF_ALU64 | BPF_MOV | BPF_K: emit_imm(rd, imm, ctx); - if (!is64) + if (!is64 && aux->no_verifier_zext) emit_zext_32(rd, ctx); break; @@ -882,7 +883,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, emit(is64 ? rv_add(rd, rd, RV_REG_T1) : rv_addw(rd, rd, RV_REG_T1), ctx); } - if (!is64) + if (!is64 && aux->no_verifier_zext) emit_zext_32(rd, ctx); break; case BPF_ALU | BPF_SUB | BPF_K: @@ -895,7 +896,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, emit(is64 ? rv_sub(rd, rd, RV_REG_T1) : rv_subw(rd, rd, RV_REG_T1), ctx); } - if (!is64) + if (!is64 && aux->no_verifier_zext) emit_zext_32(rd, ctx); break; case BPF_ALU | BPF_AND | BPF_K: @@ -906,7 +907,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, emit_imm(RV_REG_T1, imm, ctx); emit(rv_and(rd, rd, RV_REG_T1), ctx); } - if (!is64) + if (!is64 && aux->no_verifier_zext) emit_zext_32(rd, ctx); break; case BPF_ALU | BPF_OR | BPF_K: @@ -917,7 +918,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, emit_imm(RV_REG_T1, imm, ctx); emit(rv_or(rd, rd, RV_REG_T1), ctx); } - if (!is64) + if (!is64 && aux->no_verifier_zext) emit_zext_32(rd, ctx); break; case BPF_ALU | BPF_XOR | BPF_K: @@ -928,7 +929,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, emit_imm(RV_REG_T1, imm, ctx); emit(rv_xor(rd, rd, RV_REG_T1), ctx); } - if (!is64) + if (!is64 && aux->no_verifier_zext) emit_zext_32(rd, ctx); break; case BPF_ALU | BPF_MUL | BPF_K: @@ -936,7 +937,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, emit_imm(RV_REG_T1, imm, ctx); emit(is64 ? rv_mul(rd, rd, RV_REG_T1) : rv_mulw(rd, rd, RV_REG_T1), ctx); - if (!is64) + if (!is64 && aux->no_verifier_zext) emit_zext_32(rd, ctx); break; case BPF_ALU | BPF_DIV | BPF_K: @@ -944,7 +945,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, emit_imm(RV_REG_T1, imm, ctx); emit(is64 ? rv_divu(rd, rd, RV_REG_T1) : rv_divuw(rd, rd, RV_REG_T1), ctx); - if (!is64) + if (!is64 && aux->no_verifier_zext) emit_zext_32(rd, ctx); break; case BPF_ALU | BPF_MOD | BPF_K: @@ -952,7 +953,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, emit_imm(RV_REG_T1, imm, ctx); emit(is64 ? rv_remu(rd, rd, RV_REG_T1) : rv_remuw(rd, rd, RV_REG_T1), ctx); - if (!is64) + if (!is64 && aux->no_verifier_zext) emit_zext_32(rd, ctx); break; case BPF_ALU | BPF_LSH | BPF_K: @@ -1503,6 +1504,11 @@ static void bpf_flush_icache(void *start, void *end) flush_icache_range((unsigned long)start, (unsigned long)end); } +bool bpf_jit_hardware_zext(void) +{ + return false; +} + struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) { bool tmp_blinded = false, extra_pass = false; From patchwork Wed Apr 10 19:50:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1083576 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="ZRL4nJ5R"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44fZXT5tz8z9s5c for ; Thu, 11 Apr 2019 05:51:05 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726847AbfDJTvE (ORCPT ); Wed, 10 Apr 2019 15:51:04 -0400 Received: from mail-wr1-f66.google.com ([209.85.221.66]:44528 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726794AbfDJTvD (ORCPT ); Wed, 10 Apr 2019 15:51:03 -0400 Received: by mail-wr1-f66.google.com with SMTP id y7so4331766wrn.11 for ; Wed, 10 Apr 2019 12:51:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=qxA1L/gVWESQpZ3Mokilcchz/Sf49i0tM5B88NgcAXI=; b=ZRL4nJ5RCaME6Z0pqy5bAaphIlJKsv1LuR5h+uOFooWG3ek47fyqH52vFfjNRThPIK oEK98zqouP1jSIZfBmO8H63Mzl1y8b3cRioJbBpbrH2wTNOlBkoWAJrRI+7cvY3r6QeG 8FB0qx4vXWLHhS+jJ5Ib+568yOtzhU1UeODbRoQmcLfEWaCrY733xqc6NfshYi6/sPrT L/WFUVIY8bG0eqhWqFzCwVSmj2cajeL0toI3mZpjN2NeX7cFtIXp+g2kjziB92a/JVKF 99gnICxJdEITZhGK5Ovp2PU4nbpdxKs2QELbxIgCBxbKZxZKnHiXrmDtDQuflog5sH6i oUOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=qxA1L/gVWESQpZ3Mokilcchz/Sf49i0tM5B88NgcAXI=; b=SsWCBP+ubC+rY7ndX46erMZMsxu2q+zU1Ew7Fr5sFE3wiQ35T6uV5xjDOBb0xr9zdQ ZfhHHN87OEk1GQurhUaAf3HAZ+yYbUMtWYiWUXFDZ/rcKZO+fYicqxIjvJk+naJLDJkY FbQ0WOzDRRV0VVg/syU1ZTKn7HdlLIOd8AuIymYcZNl0CeDpFjd6tgecy/G7uP1wMHCA mE3V6vMi9EoG2a/gnJCuITrCNsfGuv1+bWvxSX53LFC8Acu/CNhCHvPhlrQhghsfTR3M N8hcVC0uayYaL8QFVMuCebcl1aKdCPjLNq6br2knSy7Gf+kPJXvpZTSlqJSXwDB1gT7P R2sA== X-Gm-Message-State: APjAAAWN0Vk6crjwb3ToZ9m1w/DtXV/Zlp9GsEALyxsYOK4UUKN2uzu3 Kj6Eo3gzmq83qMIsb3ARztsnCw== X-Google-Smtp-Source: APXvYqzjmx3qhJsi+oW7qTQ01LmPzsWpZ1uJaGrs1Bz6QJDXSs5Zkwt7W6ayaB88r3UZz90TazsQcg== X-Received: by 2002:a5d:428c:: with SMTP id k12mr13003555wrq.279.1554925860512; Wed, 10 Apr 2019 12:51:00 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id 4sm2288540wmg.12.2019.04.10.12.50.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Wed, 10 Apr 2019 12:50:59 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang Subject: [PATCH/RFC v2 bpf-next 19/19] nfp: bpf: eliminate zero extension code-gen Date: Wed, 10 Apr 2019 20:50:33 +0100 Message-Id: <1554925833-7333-20-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> References: <1554925833-7333-1-git-send-email-jiong.wang@netronome.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org This patch eliminate zero extension code-gen for instructions except load/store when possible. Elimination for load/store will be supported in up coming patches. Reviewed-by: Jakub Kicinski Signed-off-by: Jiong Wang --- drivers/net/ethernet/netronome/nfp/bpf/jit.c | 119 +++++++++++++--------- drivers/net/ethernet/netronome/nfp/bpf/main.h | 2 + drivers/net/ethernet/netronome/nfp/bpf/verifier.c | 12 +++ 3 files changed, 83 insertions(+), 50 deletions(-) diff --git a/drivers/net/ethernet/netronome/nfp/bpf/jit.c b/drivers/net/ethernet/netronome/nfp/bpf/jit.c index f272247..eb30c52 100644 --- a/drivers/net/ethernet/netronome/nfp/bpf/jit.c +++ b/drivers/net/ethernet/netronome/nfp/bpf/jit.c @@ -612,6 +612,13 @@ static void wrp_immed(struct nfp_prog *nfp_prog, swreg dst, u32 imm) } static void +wrp_zext(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, u8 dst) +{ + if (meta->flags & FLAG_INSN_DO_ZEXT) + wrp_immed(nfp_prog, reg_both(dst + 1), 0); +} + +static void wrp_immed_relo(struct nfp_prog *nfp_prog, swreg dst, u32 imm, enum nfp_relo_type relo) { @@ -847,7 +854,8 @@ static int nfp_cpp_memcpy(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) } static int -data_ld(struct nfp_prog *nfp_prog, swreg offset, u8 dst_gpr, int size) +data_ld(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, swreg offset, + u8 dst_gpr, int size) { unsigned int i; u16 shift, sz; @@ -870,14 +878,15 @@ data_ld(struct nfp_prog *nfp_prog, swreg offset, u8 dst_gpr, int size) wrp_mov(nfp_prog, reg_both(dst_gpr + i), reg_xfer(i)); if (i < 2) - wrp_immed(nfp_prog, reg_both(dst_gpr + 1), 0); + wrp_zext(nfp_prog, meta, dst_gpr); return 0; } static int -data_ld_host_order(struct nfp_prog *nfp_prog, u8 dst_gpr, - swreg lreg, swreg rreg, int size, enum cmd_mode mode) +data_ld_host_order(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, + u8 dst_gpr, swreg lreg, swreg rreg, int size, + enum cmd_mode mode) { unsigned int i; u8 mask, sz; @@ -900,33 +909,34 @@ data_ld_host_order(struct nfp_prog *nfp_prog, u8 dst_gpr, wrp_mov(nfp_prog, reg_both(dst_gpr + i), reg_xfer(i)); if (i < 2) - wrp_immed(nfp_prog, reg_both(dst_gpr + 1), 0); + wrp_zext(nfp_prog, meta, dst_gpr); return 0; } static int -data_ld_host_order_addr32(struct nfp_prog *nfp_prog, u8 src_gpr, swreg offset, - u8 dst_gpr, u8 size) +data_ld_host_order_addr32(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, + u8 src_gpr, swreg offset, u8 dst_gpr, u8 size) { - return data_ld_host_order(nfp_prog, dst_gpr, reg_a(src_gpr), offset, - size, CMD_MODE_32b); + return data_ld_host_order(nfp_prog, meta, dst_gpr, reg_a(src_gpr), + offset, size, CMD_MODE_32b); } static int -data_ld_host_order_addr40(struct nfp_prog *nfp_prog, u8 src_gpr, swreg offset, - u8 dst_gpr, u8 size) +data_ld_host_order_addr40(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, + u8 src_gpr, swreg offset, u8 dst_gpr, u8 size) { swreg rega, regb; addr40_offset(nfp_prog, src_gpr, offset, ®a, ®b); - return data_ld_host_order(nfp_prog, dst_gpr, rega, regb, + return data_ld_host_order(nfp_prog, meta, dst_gpr, rega, regb, size, CMD_MODE_40b_BA); } static int -construct_data_ind_ld(struct nfp_prog *nfp_prog, u16 offset, u16 src, u8 size) +construct_data_ind_ld(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, + u16 offset, u16 src, u8 size) { swreg tmp_reg; @@ -942,10 +952,12 @@ construct_data_ind_ld(struct nfp_prog *nfp_prog, u16 offset, u16 src, u8 size) emit_br_relo(nfp_prog, BR_BLO, BR_OFF_RELO, 0, RELO_BR_GO_ABORT); /* Load data */ - return data_ld(nfp_prog, imm_b(nfp_prog), 0, size); + return data_ld(nfp_prog, meta, imm_b(nfp_prog), 0, size); } -static int construct_data_ld(struct nfp_prog *nfp_prog, u16 offset, u8 size) +static int +construct_data_ld(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, + u16 offset, u8 size) { swreg tmp_reg; @@ -956,7 +968,7 @@ static int construct_data_ld(struct nfp_prog *nfp_prog, u16 offset, u8 size) /* Load data */ tmp_reg = re_load_imm_any(nfp_prog, offset, imm_b(nfp_prog)); - return data_ld(nfp_prog, tmp_reg, 0, size); + return data_ld(nfp_prog, meta, tmp_reg, 0, size); } static int @@ -1193,7 +1205,7 @@ mem_op_stack(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, } if (clr_gpr && size < 8) - wrp_immed(nfp_prog, reg_both(gpr + 1), 0); + wrp_zext(nfp_prog, meta, gpr); while (size) { u32 slice_end; @@ -1294,9 +1306,10 @@ wrp_alu32_imm(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, enum alu_op alu_op) { const struct bpf_insn *insn = &meta->insn; + u8 dst = insn->dst_reg * 2; - wrp_alu_imm(nfp_prog, insn->dst_reg * 2, alu_op, insn->imm); - wrp_immed(nfp_prog, reg_both(insn->dst_reg * 2 + 1), 0); + wrp_alu_imm(nfp_prog, dst, alu_op, insn->imm); + wrp_zext(nfp_prog, meta, dst); return 0; } @@ -1308,7 +1321,7 @@ wrp_alu32_reg(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, u8 dst = meta->insn.dst_reg * 2, src = meta->insn.src_reg * 2; emit_alu(nfp_prog, reg_both(dst), reg_a(dst), alu_op, reg_b(src)); - wrp_immed(nfp_prog, reg_both(meta->insn.dst_reg * 2 + 1), 0); + wrp_zext(nfp_prog, meta, dst); return 0; } @@ -2385,12 +2398,14 @@ static int neg_reg(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) u8 dst = meta->insn.dst_reg * 2; emit_alu(nfp_prog, reg_both(dst), reg_imm(0), ALU_OP_SUB, reg_b(dst)); - wrp_immed(nfp_prog, reg_both(meta->insn.dst_reg * 2 + 1), 0); + wrp_zext(nfp_prog, meta, dst); return 0; } -static int __ashr_imm(struct nfp_prog *nfp_prog, u8 dst, u8 shift_amt) +static int +__ashr_imm(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, u8 dst, + u8 shift_amt) { if (shift_amt) { /* Set signedness bit (MSB of result). */ @@ -2399,7 +2414,7 @@ static int __ashr_imm(struct nfp_prog *nfp_prog, u8 dst, u8 shift_amt) emit_shf(nfp_prog, reg_both(dst), reg_none(), SHF_OP_ASHR, reg_b(dst), SHF_SC_R_SHF, shift_amt); } - wrp_immed(nfp_prog, reg_both(dst + 1), 0); + wrp_zext(nfp_prog, meta, dst); return 0; } @@ -2414,7 +2429,7 @@ static int ashr_reg(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) umin = meta->umin_src; umax = meta->umax_src; if (umin == umax) - return __ashr_imm(nfp_prog, dst, umin); + return __ashr_imm(nfp_prog, meta, dst, umin); src = insn->src_reg * 2; /* NOTE: the first insn will set both indirect shift amount (source A) @@ -2423,7 +2438,7 @@ static int ashr_reg(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) emit_alu(nfp_prog, reg_none(), reg_a(src), ALU_OP_OR, reg_b(dst)); emit_shf_indir(nfp_prog, reg_both(dst), reg_none(), SHF_OP_ASHR, reg_b(dst), SHF_SC_R_SHF); - wrp_immed(nfp_prog, reg_both(dst + 1), 0); + wrp_zext(nfp_prog, meta, dst); return 0; } @@ -2433,15 +2448,17 @@ static int ashr_imm(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) const struct bpf_insn *insn = &meta->insn; u8 dst = insn->dst_reg * 2; - return __ashr_imm(nfp_prog, dst, insn->imm); + return __ashr_imm(nfp_prog, meta, dst, insn->imm); } -static int __shr_imm(struct nfp_prog *nfp_prog, u8 dst, u8 shift_amt) +static int +__shr_imm(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, u8 dst, + u8 shift_amt) { if (shift_amt) emit_shf(nfp_prog, reg_both(dst), reg_none(), SHF_OP_NONE, reg_b(dst), SHF_SC_R_SHF, shift_amt); - wrp_immed(nfp_prog, reg_both(dst + 1), 0); + wrp_zext(nfp_prog, meta, dst); return 0; } @@ -2450,7 +2467,7 @@ static int shr_imm(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) const struct bpf_insn *insn = &meta->insn; u8 dst = insn->dst_reg * 2; - return __shr_imm(nfp_prog, dst, insn->imm); + return __shr_imm(nfp_prog, meta, dst, insn->imm); } static int shr_reg(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) @@ -2463,22 +2480,24 @@ static int shr_reg(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) umin = meta->umin_src; umax = meta->umax_src; if (umin == umax) - return __shr_imm(nfp_prog, dst, umin); + return __shr_imm(nfp_prog, meta, dst, umin); src = insn->src_reg * 2; emit_alu(nfp_prog, reg_none(), reg_a(src), ALU_OP_OR, reg_imm(0)); emit_shf_indir(nfp_prog, reg_both(dst), reg_none(), SHF_OP_NONE, reg_b(dst), SHF_SC_R_SHF); - wrp_immed(nfp_prog, reg_both(dst + 1), 0); + wrp_zext(nfp_prog, meta, dst); return 0; } -static int __shl_imm(struct nfp_prog *nfp_prog, u8 dst, u8 shift_amt) +static int +__shl_imm(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, u8 dst, + u8 shift_amt) { if (shift_amt) emit_shf(nfp_prog, reg_both(dst), reg_none(), SHF_OP_NONE, reg_b(dst), SHF_SC_L_SHF, shift_amt); - wrp_immed(nfp_prog, reg_both(dst + 1), 0); + wrp_zext(nfp_prog, meta, dst); return 0; } @@ -2487,7 +2506,7 @@ static int shl_imm(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) const struct bpf_insn *insn = &meta->insn; u8 dst = insn->dst_reg * 2; - return __shl_imm(nfp_prog, dst, insn->imm); + return __shl_imm(nfp_prog, meta, dst, insn->imm); } static int shl_reg(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) @@ -2500,11 +2519,11 @@ static int shl_reg(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) umin = meta->umin_src; umax = meta->umax_src; if (umin == umax) - return __shl_imm(nfp_prog, dst, umin); + return __shl_imm(nfp_prog, meta, dst, umin); src = insn->src_reg * 2; shl_reg64_lt32_low(nfp_prog, dst, src); - wrp_immed(nfp_prog, reg_both(dst + 1), 0); + wrp_zext(nfp_prog, meta, dst); return 0; } @@ -2566,34 +2585,34 @@ static int imm_ld8(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) static int data_ld1(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) { - return construct_data_ld(nfp_prog, meta->insn.imm, 1); + return construct_data_ld(nfp_prog, meta, meta->insn.imm, 1); } static int data_ld2(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) { - return construct_data_ld(nfp_prog, meta->insn.imm, 2); + return construct_data_ld(nfp_prog, meta, meta->insn.imm, 2); } static int data_ld4(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) { - return construct_data_ld(nfp_prog, meta->insn.imm, 4); + return construct_data_ld(nfp_prog, meta, meta->insn.imm, 4); } static int data_ind_ld1(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) { - return construct_data_ind_ld(nfp_prog, meta->insn.imm, + return construct_data_ind_ld(nfp_prog, meta, meta->insn.imm, meta->insn.src_reg * 2, 1); } static int data_ind_ld2(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) { - return construct_data_ind_ld(nfp_prog, meta->insn.imm, + return construct_data_ind_ld(nfp_prog, meta, meta->insn.imm, meta->insn.src_reg * 2, 2); } static int data_ind_ld4(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta) { - return construct_data_ind_ld(nfp_prog, meta->insn.imm, + return construct_data_ind_ld(nfp_prog, meta, meta->insn.imm, meta->insn.src_reg * 2, 4); } @@ -2632,7 +2651,7 @@ static int mem_ldx_skb(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, return -EOPNOTSUPP; } - wrp_immed(nfp_prog, reg_both(meta->insn.dst_reg * 2 + 1), 0); + wrp_zext(nfp_prog, meta, dst); return 0; } @@ -2658,7 +2677,7 @@ static int mem_ldx_xdp(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, return -EOPNOTSUPP; } - wrp_immed(nfp_prog, reg_both(meta->insn.dst_reg * 2 + 1), 0); + wrp_zext(nfp_prog, meta, dst); return 0; } @@ -2671,7 +2690,7 @@ mem_ldx_data(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, tmp_reg = re_load_imm_any(nfp_prog, meta->insn.off, imm_b(nfp_prog)); - return data_ld_host_order_addr32(nfp_prog, meta->insn.src_reg * 2, + return data_ld_host_order_addr32(nfp_prog, meta, meta->insn.src_reg * 2, tmp_reg, meta->insn.dst_reg * 2, size); } @@ -2683,7 +2702,7 @@ mem_ldx_emem(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, tmp_reg = re_load_imm_any(nfp_prog, meta->insn.off, imm_b(nfp_prog)); - return data_ld_host_order_addr40(nfp_prog, meta->insn.src_reg * 2, + return data_ld_host_order_addr40(nfp_prog, meta, meta->insn.src_reg * 2, tmp_reg, meta->insn.dst_reg * 2, size); } @@ -2744,7 +2763,7 @@ mem_ldx_data_from_pktcache_unaligned(struct nfp_prog *nfp_prog, wrp_reg_subpart(nfp_prog, dst_lo, src_lo, len_lo, off); if (!len_mid) { - wrp_immed(nfp_prog, dst_hi, 0); + wrp_zext(nfp_prog, meta, dst_gpr); return 0; } @@ -2752,7 +2771,7 @@ mem_ldx_data_from_pktcache_unaligned(struct nfp_prog *nfp_prog, if (size <= REG_WIDTH) { wrp_reg_or_subpart(nfp_prog, dst_lo, src_mid, len_mid, len_lo); - wrp_immed(nfp_prog, dst_hi, 0); + wrp_zext(nfp_prog, meta, dst_gpr); } else { swreg src_hi = reg_xfer(idx + 2); @@ -2783,10 +2802,10 @@ mem_ldx_data_from_pktcache_aligned(struct nfp_prog *nfp_prog, if (size < REG_WIDTH) { wrp_reg_subpart(nfp_prog, dst_lo, src_lo, size, 0); - wrp_immed(nfp_prog, dst_hi, 0); + wrp_zext(nfp_prog, meta, dst_gpr); } else if (size == REG_WIDTH) { wrp_mov(nfp_prog, dst_lo, src_lo); - wrp_immed(nfp_prog, dst_hi, 0); + wrp_zext(nfp_prog, meta, dst_gpr); } else { swreg src_hi = reg_xfer(idx + 1); diff --git a/drivers/net/ethernet/netronome/nfp/bpf/main.h b/drivers/net/ethernet/netronome/nfp/bpf/main.h index b25a482..7369bdf 100644 --- a/drivers/net/ethernet/netronome/nfp/bpf/main.h +++ b/drivers/net/ethernet/netronome/nfp/bpf/main.h @@ -249,6 +249,8 @@ struct nfp_bpf_reg_state { #define FLAG_INSN_SKIP_PREC_DEPENDENT BIT(4) /* Instruction is optimized by the verifier */ #define FLAG_INSN_SKIP_VERIFIER_OPT BIT(5) +/* Instruction needs to zero extend to high 32-bit */ +#define FLAG_INSN_DO_ZEXT BIT(6) #define FLAG_INSN_SKIP_MASK (FLAG_INSN_SKIP_NOOP | \ FLAG_INSN_SKIP_PREC_DEPENDENT | \ diff --git a/drivers/net/ethernet/netronome/nfp/bpf/verifier.c b/drivers/net/ethernet/netronome/nfp/bpf/verifier.c index 36f56eb..e92ee51 100644 --- a/drivers/net/ethernet/netronome/nfp/bpf/verifier.c +++ b/drivers/net/ethernet/netronome/nfp/bpf/verifier.c @@ -744,6 +744,17 @@ static unsigned int nfp_bpf_get_stack_usage(struct nfp_prog *nfp_prog) goto continue_subprog; } +static void nfp_bpf_insn_flag_zext(struct nfp_prog *nfp_prog, + struct bpf_insn_aux_data *aux) +{ + struct nfp_insn_meta *meta; + + list_for_each_entry(meta, &nfp_prog->insns, l) { + if (aux[meta->n].zext_dst) + meta->flags |= FLAG_INSN_DO_ZEXT; + } +} + int nfp_bpf_finalize(struct bpf_verifier_env *env) { struct bpf_subprog_info *info; @@ -784,6 +795,7 @@ int nfp_bpf_finalize(struct bpf_verifier_env *env) return -EOPNOTSUPP; } + nfp_bpf_insn_flag_zext(nfp_prog, env->insn_aux_data); return 0; }